Press n or j to go to the next uncovered block, b, p or k for the previous block.
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 | 2x 11x 11x 4x 4x 5x 3x 3x 5x 11x 2x 21x 21x 9x 9x 10x 9x 9x 10x 1x 1x 21x 2x 2x 2x 2x 178x 178x 2x 164x 164x 2x 33x 27x 27x 4x 23x 33x 6x 6x 6x 6x 2x 4x 4x 4x 2x 30x 29x 7x 7x 7x 29x 29x 29x 13x 13x 2x 27x 27x 27x 10x 10x 2x 48x 48x 48x 48x 48x 77x 49x 32x 32x 22x 22x 22x 22x 26x 48x 48x 77x 55x 77x 77x 48x 48x 2x 7x 7x 7x 7x 7x 5x 1x 7x | /**
* The sidecar index beside a file, and the rules for pairing the two.
*
* Its own module, and small on purpose. Three things need these rules and they
* sit in three different layers: the annotations loader, which pairs a drop;
* `app/Session`, which routes one; and `browse/local`, which picks a file out
* of a connected folder and should hand over the index sitting next to it.
* Leaving them in `annotations/loader.ts` meant the folder browser importing
* the loader — and with it the store, the worker client and a UI module — to
* ask what a `.bai` is called.
*
* Nothing here touches the DOM, a worker, or anything but names and `File`s.
*/
/**
* Split `<url>#index=<url>` into the two URLs it names.
*
* The one place a reader can say where an index really is, and it exists
* because the guess cannot always be made: `<url>.bai` is meaningless against a
* presigned URL, against one carrying a query string, and against any bucket
* that publishes its index under a different key. A fragment rather than a
* second input box, because a URL is what gets pasted into chat, into a
* notebook and into this app's own box — so the pair travels as one string or
* it does not travel (§8.13).
*
* The fragment is stripped from the address, since a `#` is never sent to a
* server and leaving it on would only make the name shown for the file wrong.
*/
export function splitIndexFragment(url: string): { address: string; index?: string } {
const at = url.indexOf('#index=')
if (at < 0) return { address: url }
const index = url.slice(at + '#index='.length).trim()
const address = url.slice(0, at)
if (index === '') return { address }
// A malformed percent-escape throws `URIError`, and this runs inside the URL
// box's submit handler where nothing catches it: one stray `%` and the box
// stops responding with no message anywhere. The escaped form is a courtesy
// for URLs carrying their own `&` and `#`; a string that will not decode is
// taken as written, which is what a reader who typed it meant.
try {
return { address, index: decodeURIComponent(index) }
} catch {
return { address, index }
}
}
/**
* Split `<url>#bins=<url>` into the two URLs it names.
*
* The HiC-Pro twin of {@link splitIndexFragment}, and it exists for the same
* reason: the guess cannot always be made. A presigned matrix URL, one carrying
* a query string, or a bucket that publishes the two halves under unrelated
* keys all defeat the convention, and a pair that cannot travel as one string
* cannot be shared at all (§11).
*/
export function splitBinsFragment(url: string): { address: string; bins?: string } {
const at = url.indexOf('#bins=')
if (at < 0) return { address: url }
const bins = url.slice(at + '#bins='.length).trim()
const address = url.slice(0, at)
if (bins === '') return { address }
try {
return { address, bins: decodeURIComponent(bins) }
} catch {
return { address, bins }
}
}
/**
* The sidecar suffixes this app pairs with the file they index.
*
* `.tbi` indexes a bgzipped text file and is opportunistic — the file opens
* without it (§8.11). `.px2` indexes a `.pairs.gz` and is opportunistic in the
* same sense (§3.7): with one the import's cost is known before it starts and
* can be scoped, without one the file is read whole. `.fai` indexes a FASTA
* and is opportunistic in a third sense (§8.8): a small genome is read whole
* with or without one, and a large one is read *by range* with it and refused
* without. `.bai` and `.csi` index a BAM and are a fourth sense (§8.13): a BAM
* has no whole-file fallback at §12's budgets, so with one a window's bytes
* are read off the index, and without one they are *found* — by a search over
* the file's own order, a dozen probes a window, which is slower and exact to
* a block rather than to a record. One regular expression for all five,
* because *pairing* is the same gesture whichever it is; what differs is only
* what happens when the pairing fails.
*/
const INDEX_SUFFIX = /\.(tbi|bai|csi|px2|fai)$/i
/** A FASTA by name: what `samtools faidx` indexes, plain and never gzipped. */
export const FASTA_NAME = /\.(fa|fasta|fna|fas)$/i
/**
* The bin file beside a HiC-Pro matrix (§3.9).
*
* A fourth kind of companion, paired by the same gesture as the three above and
* for the reason the comment on {@link INDEX_SUFFIX} gives: what differs
* between them is only what happens when the pairing fails. This one fails
* hardest — a `.tbi` is an optimisation and a `.bai` is a format's only bounded
* read, but an `_abs.bed` is the matrix's **coordinate system**, so without it
* there is not a slower map or a refused one but no map at all.
*
* Deliberately narrow. An ordinary BED must not be captured by this, or a
* feature file dropped beside a contact map would be silently swallowed as its
* bins; HiC-Pro's own `<prefix>_<res>_abs.bed` is the only spelling matched,
* and a lone one still opens as the track it also is.
*/
const HICPRO_BINS = /_abs\.bed(\.gz)?$/i
/** The matrix half of a HiC-Pro pair, by name. */
export function isHicProMatrix(name: string): boolean {
return /\.matrix(\.gz)?$/i.test(name)
}
/** Whether this file is a sidecar index rather than something to draw. */
export function isIndexName(name: string): boolean {
return INDEX_SUFFIX.test(name) || HICPRO_BINS.test(name)
}
/**
* The names a sidecar could be the index for.
*
* Two, because `samtools index` writes both spellings: `reads.bam.bai` beside
* `reads.bam`, and — with an explicit output path, which is what every pipeline
* that renames its outputs does — `reads.bai`. A reader who drops the pair they
* have must not have to know which convention produced it.
*/
export function basesForIndex(name: string): string[] {
if (HICPRO_BINS.test(name)) return hicProMatrixNames(name)
const stem = name.replace(INDEX_SUFFIX, '')
return /\.(bai|csi)$/i.test(name) && !/\.bam$/i.test(stem)
? [stem, `${stem}.bam`]
: [stem]
}
/**
* The matrices a `<prefix>_<res>_abs.bed` could belong to.
*
* Four spellings, and each is a real HiC-Pro output: the raw matrix beside the
* bed in `raw/<res>/`, the ICE-normalised one in `iced/<res>/` which carries an
* `_iced` before its extension, and either of them gzipped. The bed is written
* once per resolution and both matrices share it, which is why a reader may
* drop the bed with whichever of the two they meant.
*/
function hicProMatrixNames(bins: string): string[] {
const stem = bins.replace(HICPRO_BINS, '')
return [`${stem}.matrix`, `${stem}_iced.matrix`, `${stem}.matrix.gz`, `${stem}_iced.matrix.gz`]
}
/**
* The bin file a HiC-Pro matrix needs, by name, most likely first.
*
* The reverse of {@link hicProMatrixNames}, and the road a matrix dropped or
* linked **alone** takes: the same rule §8.11 uses to find a `<url>.tbi` beside
* the file it indexes, applied to a companion that is not optional.
*/
export function hicProBinsNames(matrix: string): string[] {
const stem = matrix.replace(/\.matrix(\.gz)?$/i, '').replace(/_iced$/i, '')
return [`${stem}_abs.bed`, `${stem}_abs.bed.gz`]
}
/**
* The names a sidecar for `base` could have been remembered under.
*
* The reverse of {@link basesForIndex}, and it exists for one road only: a
* restored row knows the name of its file and nothing about its index, because
* a `File` does not survive session storage. Both suffixes and both spellings
* are offered, most specific first, and the first that a grant answers for is
* the one taken.
*/
export function indexNamesFor(base: string): string[] {
if (isHicProMatrix(base)) return hicProBinsNames(base)
if (/\.bam$/i.test(base)) {
const stem = base.replace(/\.bam$/i, '')
return [`${base}.bai`, `${base}.csi`, `${stem}.bai`, `${stem}.csi`]
}
// A pairs file takes a `.px2` and never a `.tbi`; everything else bgzipped
// takes a `.tbi` and never a `.px2`. Offering both for either would mean a
// restored row fetching an index that cannot exist.
if (/\.pairs\.b?gz$/i.test(base)) return [`${base}.px2`]
if (/\.b?gz$/i.test(base)) return [`${base}.tbi`]
// `samtools faidx genome.fa` writes `genome.fa.fai`, and nothing else does.
if (FASTA_NAME.test(base)) return [`${base}.fai`]
return []
}
/**
* The sidecar URL for a remote file, guessed from its name.
*
* `<url>.tbi` for anything gzipped — as `.gz` or as `.bgz`, which is gnomAD's
* spelling — `<url>.bai` for a BAM and `<url>.fai` for a FASTA: the spellings
* `tabix`, `samtools index` and `samtools faidx` write, which is what every
* bucket that publishes an index publishes it as. Guessed optimistically and
* never probed — a `HEAD` to find out costs the same round trip the miss does,
* and §12 does not have a request to spend on finding out that a file has no
* index. What a miss costs differs by format, and each reader says so where it
* falls back; what is the same for all three is that `#index=<url>` on the
* file's own URL always wins over this guess, because a presigned URL or one
* carrying a query string can never usefully have a suffix appended to it
* (§8.13).
*/
export function guessedIndexUrl(url: string): string | undefined {
if (/\.b?gz$/i.test(url)) return `${url}.tbi`
if (/\.bam$/i.test(url)) return `${url}.bai`
if (FASTA_NAME.test(url)) return `${url}.fai`
return undefined
}
/** A file to open, and the sidecar dropped alongside it if there was one. */
export interface PairedDrop {
file: File
index?: File
}
/**
* Group a dropped batch into the files to open and the indexes that belong to them.
*
* The one implementation of the pairing rule, because there are two roads into
* it and they used to disagree. {@link AnnotationLoader.addFiles} pairs what it
* is handed; `Session.dropped` decides what a single file *is* — a workspace, a
* map, a `chrom.sizes`, a track — and every caller was looping it over the
* batch, so the loader was handed one file at a time and the pairing never
* fired for a plain drop onto the window. A `.tbi` losing that race read a file
* whole that it had an index for; a `.bai` losing it loses the file (§8.13).
*
* A sidecar whose file is not in the batch is returned as its own entry rather
* than dropped, so it reaches the loader alone and is refused by name — more
* use than silently doing nothing.
*/
export function pairIndexes(files: readonly File[]): PairedDrop[] {
const bases = new Map<string, File>()
for (const file of files) if (!isIndexName(file.name)) bases.set(file.name, file)
const indexOf = new Map<File, File>()
const spent = new Set<File>()
for (const file of files) {
if (!isIndexName(file.name)) continue
for (const name of basesForIndex(file.name)) {
const base = bases.get(name)
// First claim wins, in the order `basesForIndex` offers: `reads.bam.bai`
// names one file exactly, so a directory holding both spellings pairs
// each with its own rather than both with the first.
if (base === undefined || indexOf.has(base)) continue
indexOf.set(base, file)
spent.add(file)
break
}
}
const out: PairedDrop[] = []
for (const file of files) {
if (spent.has(file)) continue
const index = indexOf.get(file)
out.push(index === undefined ? { file } : { file, index })
}
return out
}
/**
* Which of a dropped batch opens the session, once the sidecars are paired.
*
* A map first, then anything that can declare a genome, then whatever came
* first — which is the order `Session.openFiles` has always used. What is new
* is that it chooses among **pairs** rather than among files, and that is the
* whole of why it is a function: choosing from the raw list picks `reads.bam`,
* opens it without the `reads.bam.bai` sitting right beside it, and leaves the
* index to be routed on its own. The result is a browser on the BAM's own
* reference table, the BAM refused for want of an index, and an orphan row for
* the index — from the one gesture the feature exists for (§8.13).
*
* `route` is passed in rather than imported so this stays in the annotations
* layer, which does not know about the app's front doors.
*/
export function leadOf(
paired: readonly PairedDrop[],
route: (name: string) => 'map' | 'genome' | 'workspace' | 'unknown',
): PairedDrop | undefined {
return (
paired.find((pair) => route(pair.file.name) === 'map') ??
paired.find((pair) => route(pair.file.name) === 'genome') ??
paired[0]
)
}
|