Twenty-seven analyses on public data, each run by chromosomix and by the package it reproduces
Rendered 1 November 2026, 21:32 UTC · chromosomix 0.19.0 at 997e59c By GitHub Actions, Notebooks run 16, on demand of js2264 On AMD EPYC 9V45 96-Core Processor, 4 cores, 15.6 GB, Linux 7.0.0-1017-azure (x86_64); Python 3.13.17, Quarto 1.10.19
Every command of §§ 1–19 and 24–27 below, run again and timed: the timings. Beside this page: commands.json, every command as it ran; summary.tsv, every verdict; and validation.txt, every number its figures draw.
Each analysis is run on the same public file twice: by chromosomix, the command-line form of the app’s Analysis panel (the same code), and by the reference package it reproduces, called as its authors intend. The two outputs are matched bin by bin or call by call. Every command is printed as it ran, from notebooks/work/validation/, and written to commands.json there, from which timing.qmd runs each again and times it — those of §§ 20–23 aside, which are compared here and not timed.
Every number a figure draws is one line of validation.txt, beside commands.json: a tidy table, one row per mark — a point, a vertex of a line, a hexagon of a density and its count, a cell of a picture, a bar, a band or a line across a panel, a named tick, a text — with the figure’s label and the panel’s letter, the panel’s axes, what the mark stands for (series, as the legend names it) and where it is drawn. It is read off each figure as it is shown, so any figure below can be drawn again from it alone. A density is the hexagons it draws, not the millions of values behind it, which are the outputs of the commands printed above it; a symmetric map’s numbers are written once, its upper triangle; and the metaplot’s heatmaps, tens of thousands of transcripts in a few hundred pixels, draw each line as the mean of 100, which is what is written.
§
Analysis
Data
Reference
1
Diamond insulation
mouse Hi-C, 10 kb
cooltools.insulation
2
Directionality index
mouse Hi-C, 50 kb
domaincaller
3
Compartments
mouse Hi-C, 100 kb
cooltools.eigs_cis
4
P(s)
yeast Micro-C, 800 bp; mouse Hi-C, 10 kb
cooltools.expected_cis
5
Peak calling
mouse ATAC-seq, two replicates, chr19
yapc 0.1 with IDR 2.0.3
6
Loops
yeast Micro-C, 800 bp
chromosight 1.6.3
7
Reproducibility
chicken Hi-C, three maps, 100 kb
hicrep 0.2.6
8
3D model
yeast Micro-C, 12.8 kb
pastis (GitHub 6888ac2)
9
Coverage
mouse Hi-C, 10 kb
cooltools.coverage
10
Virtual 4C
mouse Hi-C, 10 kb
cooltools.virtual4c
11
Aggregate peak analysis
mouse Hi-C, 10 kb
coolpuppy
12
APA, chromosight’s score
mouse Hi-C, 10 kb
chromosight quantify
13
Boost-HiC
mouse Hi-C, 100 kb
Boost-HiC (GitHub a615aff)
14
Metaplot
mouse ATAC-seq, one and two replicates, RefSeq TSS
deepTools computeMatrix, plotProfile, plotHeatmap
15
Interval overlap
ATAC-seq peaks and promoters, chr19
bedtools jaccard, intersect, shuffle
16
Base composition
mm10, 10 kb
bioframe.frac_gc, Biopython GC_skew
17
Motif search
mm10, chr19
seqkit locate
18
Track correlation
mouse ATAC-seq, two replicates, and insulation, chr19, 100 bp to 10 kb
pyBigWig and SciPy; deepTools multiBigwigSummary, plotCorrelation
19
Loop calling
mouse Hi-C, 10 kb
cooltools.dots
20
Cis and trans per sequence
chicken Hi-C, 100 kb
cooltools.expected_cis, expected_trans
21
Saddle
mouse Hi-C, 100 kb, § 3’s E1
cooltools.saddle
22
Fraction of signal in features
mouse ATAC-seq, two replicates, chr19; peaks and promoters
pyBigWig and NumPy
23
Signal by group
mouse ATAC-seq, two replicates, chr19; § 3’s compartments
pyBigWig, NumPy and SciPy
24
CTCF sites in GM12878 (Fig. 2): insulation at four scales, peaks, APA between consecutive peaks, metaplot
GM12878 Hi-C, 10 kb; CTCF ChIP-seq of two labs and ATAC-seq; chr19
pairtools restrict and pairtools stats 1.1.2, cooltools.coverage
28
TAD annotation
K562 Hi-C (.hic), chromosomes 21 and 22, 10 kb, raw and KR
juicer_tools arrowhead 1.22.01
29
Map report
yeast Micro-C, every stored resolution
cooler and numpy
To render.npm run build:cli at the repository root builds this checkout’s chromosomix. yapc needs an environment of its own (numpy < 1.24; the recipe is in the yapc section of scripts/make_analysis_golden.py); Boost-HiC is a checkout, beside the repository or named in BOOSTHIC_DIR; bedtools and seqkit are programs on the PATH (conda, or the system’s packages); the other references share one environment, with SciPy before 1.18, which dropped an argument pastis passes:
notebooks/requirements.txt is that environment at the versions this page was last rendered with, every package pinned, and requirements-yapc.txt yapc’s. .github/workflows/notebooks.yml installs both and renders this page and the timings on every release, publishing them at https://chromosomix.app/validation/ and https://chromosomix.app/timings/; the block under the title says which render this is.
The data (9.8 GB) are fetched once into notebooks/work/validation/data/. Outputs are kept beside them, so a second render reads rather than recomputes (REDO).
Code
from pathlib import Pathimport os#: Run a step again even where its outputs are already on disk.REDO =False#: Processes for the reference packages that take them: one a core, at most four. Four on two cores are no#: faster than two, and on a runner of 8 GB they held more memory than it had.THREADS =min(4, os.cpu_count() or1)#: yapc 0.1's interpreter, relative to the repository or absolute.YAPC_PYTHON = os.environ.get('YAPC_PYTHON', 'yapc-env/bin/python')#: pairtools 1.1.2's interpreter: an environment of its own, on Python 3.12 or older, since pairtools#: imports the `pipes` module 3.13 removed.PAIRTOOLS_PYTHON = os.environ.get('PAIRTOOLS_PYTHON', 'pairtools-env/bin/python')#: A checkout of LeopoldC/Boost-HiC, relative to the repository or absolute.BOOSTHIC_DIR = os.environ.get('BOOSTHIC_DIR', '../Boost-HiC')HERE = Path.cwd().resolve()REPO = HERE.parent if HERE.name =='notebooks'else HERECLI = REPO /'dist-cli'/'chromosomix.mjs'WORK = REPO /'notebooks'/'work'/'validation'DATA = WORK /'data'SCRIPTS = REPO /'notebooks'/'scripts'#: Every command of this notebook, as timing.qmd runs it again.MANIFEST = WORK /'commands.json'YAPC_PYTHON = Path(YAPC_PYTHON) if Path(YAPC_PYTHON).is_absolute() else REPO / YAPC_PYTHONPAIRTOOLS_PYTHON = Path(PAIRTOOLS_PYTHON) if Path(PAIRTOOLS_PYTHON).is_absolute() else REPO / PAIRTOOLS_PYTHON#: The `pairtools` program beside that interpreter.PAIRTOOLS =str(PAIRTOOLS_PYTHON.parent /'pairtools')BOOSTHIC_DIR = Path(BOOSTHIC_DIR) if Path(BOOSTHIC_DIR).is_absolute() else (REPO / BOOSTHIC_DIR).resolve()
Code
import hashlibimport htmlimport jsonimport reimport shleximport shutilimport subprocessimport sysimport timeimport urllib.parseimport warningsfrom importlib.metadata import versionimport coolerimport matplotlib.pyplot as pltimport numpy as npimport pandas as pdimport pyBigWigfrom matplotlib.axis import Axisfrom matplotlib.collections import PathCollection, PolyCollectionfrom matplotlib.colors import LinearSegmentedColormap, LogNormfrom matplotlib.image import AxesImagefrom matplotlib.legend import Legendfrom matplotlib.lines import Line2Dfrom matplotlib.patches import Rectangle, StepPatchfrom matplotlib.spines import Spinefrom matplotlib.text import Textfrom matplotlib.ticker import FixedLocatorfrom scipy import statsfrom download import downloadwarnings.filterwarnings('ignore')ifnot CLI.exists():raiseFileNotFoundError(f'{CLI} is not there: run `npm run build:cli` at the repository root.')ifnot (BOOSTHIC_DIR /'HiCutils.py').exists():raiseFileNotFoundError(f'{BOOSTHIC_DIR} holds no Boost-HiC: clone LeopoldC/Boost-HiC there, or set BOOSTHIC_DIR.')DATA.mkdir(parents=True, exist_ok=True)PYTHON = sys.executabledef tool(name):"""A program on the PATH, or beside this interpreter where pip put it."""return shutil.which(name) orstr(Path(PYTHON).parent / name)CHROMOSIGHT, COMPUTEMATRIX, PLOTPROFILE = tool('chromosight'), tool('computeMatrix'), tool('plotProfile')PLOTHEATMAP = tool('plotHeatmap')MULTIBIGWIGSUMMARY, PLOTCORRELATION = tool('multiBigwigSummary'), tool('plotCorrelation')BEDTOOLS, SEQKIT = tool('bedtools'), tool('seqkit')#: What a reference's command runs with besides this process's environment.YAPC_ENV = {'PATH': f'{YAPC_PYTHON.parent}{os.pathsep}{os.environ["PATH"]}'}BOOSTHIC_ENV = {'BOOSTHIC_DIR': str(BOOSTHIC_DIR)}# pairtools imports matplotlib, through bioframe, and its environment has no matplotlib-inline, the backend# the kernel names in MPLBACKEND for every process it starts: it is given the one every matplotlib has.PAIRTOOLS_ENV = {'MPLBACKEND': 'agg'}def fetch(url, name=None, md5=None):"""A public file, fetched once into work/validation/data — asked again when its server fails, as notebooks/scripts/download.py says — and held to `md5` where there is one: a file of that name from anywhere else is refused rather than compared.""" path = DATA / (name or urllib.parse.unquote(url.rsplit('/', 1)[-1]))ifnot path.exists():print('fetching', url) download(url, path)if md5 isnotNone: digest = hashlib.md5()withopen(path, 'rb') as handle:for block initer(lambda: handle.read(1<<22), b''): digest.update(block)if digest.hexdigest() != md5:raiseValueError(f'{path} is not {url}: its MD5 is {digest.hexdigest()}, not {md5}. ''Delete it, and it is fetched again.')return pathdef rel(path):"""A path as the commands give it: a file of this repository relative to work/validation/, where they run; any other as it is.""" path = Path(path)return os.path.relpath(path, WORK) if path.is_relative_to(REPO) elsestr(path)def arg(value):return rel(value) ifisinstance(value, Path) elsestr(value)def typed(argv):"""A command as a reader would type it: each tool by its installed name.""" names = {PYTHON: 'python', str(YAPC_PYTHON): 'yapc-env/bin/python', PAIRTOOLS: 'pairtools', CHROMOSIGHT: 'chromosight', COMPUTEMATRIX: 'computeMatrix', PLOTPROFILE: 'plotProfile', SEQKIT: 'seqkit', MULTIBIGWIGSUMMARY: 'multiBigwigSummary', PLOTCORRELATION: 'plotCorrelation', PLOTHEATMAP: 'plotHeatmap'}if argv[:2] == ['node', str(CLI)]: argv = ['chromosomix', *argv[2:]]return shlex.join([names.get(a, a) for a in argv])#: Every chromosomix command and every reference command, by name: for the summary and the timing.APP, REFS, NOTES = {}, {}, {}#: Where a reference writes, in its command; it may be the start of a longer argument.OUT ='<out>'#: What each reference last ran, by name: its command and its scripts' content.STAMPS = WORK /'.stamps'def mr(task, *args, out, name, analysis, label='', group=None, bench=True):"""One chromosomix command into `out`, skipped when `out` holds what this same command wrote at this same commit of chromosomix — a pileup cached from an older build is the older build's. `group` pairs it with the reference commands it is timed against; its label by default.""" out = Path(out) argv = ['node', str(CLI), task, *[arg(a) for a in args]] full = argv + ['--out', rel(out)] APP[name] =dict(analysis=analysis, tool='chromosomix', label=label, group=label if group isNoneelse group, argv=argv, bench=bench, threads=1, shown=typed(full))print('$', typed(full)) stamp =f'{typed(full)}\n# chromosomix {APP_VERSION}\n' held = (out /'.command').read_text() if (out /'.command').exists() else''if REDO ornot held.startswith(stamp): out.mkdir(parents=True, exist_ok=True) start = time.time() run = subprocess.run(full, capture_output=True, text=True, cwd=WORK)if run.returncode:raiseRuntimeError(run.stderr[-4000:]) notes = [line.strip()[len('note: '):] for line in re.split(r'[\r\n]', run.stderr)if line.strip().startswith('note:')] (out /'.notes').write_text('\n'.join(notes)) (out /'.command').write_text(f'{stamp}# {time.time() - start:.1f} s\n') NOTES[name] = (out /'.notes').read_text().splitlines()return outdef script(name):"""A reference's script, in notebooks/scripts/: run, printed and timed.""" path = SCRIPTS /f'{name}.py'ifnot path.exists():raiseFileNotFoundError(f'{path} is not there.')return pathdef environment(extra):"""This process's environment with a command's own additions, or None for it as it is."""returnNoneif extra isNoneelsedict(os.environ, **extra)def reference(name, analysis, tool_, argv, out, made=None, env=None, label='', group=None, threads=1, bench=True):"""A reference command, run once for the comparison (skipped when its output is on disk). `argv` holds OUT where the command writes, so the timing can send it elsewhere; `env` is what it needs besides this environment, and `threads` the processes it was given.""" argv = [a ifisinstance(a, str) and OUT in a else arg(a) for a in argv] out = Path(out) real = [a.replace(OUT, rel(out)) for a in argv] REFS[name] =dict(analysis=analysis, tool=tool_, label=label, group=label if group isNoneelse group, argv=argv, out=out, env=env, bench=bench, threads=threads, shown=typed(real), scripts=[a for a in argv if a.endswith('.py')])print('$', typed(real))# Run again when the command or a script it runs has changed since it ran, which its stamp says;# a result from before there were stamps is kept unless one of its scripts is newer than it. target, ran = Path(made or out), STAMPS /f'{name}.command' scripts = [(WORK / s).resolve() for s in REFS[name]['scripts']] stamp =f"{typed(real)}\n# scripts {hashlib.sha256(b''.join(s.read_bytes() for s in scripts)).hexdigest()[:16]}\n"if ran.exists(): stale = ran.read_text() != stampelse: stale = target.exists() andany(s.stat().st_mtime > target.stat().st_mtime for s in scripts)if REDO or stale ornot target.exists(): out.parent.mkdir(parents=True, exist_ok=True) run = subprocess.run(real, capture_output=True, text=True, cwd=WORK, env=environment(env))if run.returncode:raiseRuntimeError(run.stderr[-4000:]) STAMPS.mkdir(parents=True, exist_ok=True) ran.write_text(stamp)return outdef table(path):"""A table chromosomix wrote: its '#' lines are provenance, not data."""return pd.read_csv(path, sep='\t', comment='#')def track_name(name):"""A bigWig as chromosomix names it in a table it writes, as the app names the track's row: its file's name less the `.bw` (`src/annotations/names.ts`). A reference that names it by its file is read through this, so that the two meet."""returnstr(name).strip("'").removesuffix('.bw')def one(folder, pattern):"""The one file of `folder` matching `pattern`.""" hits =sorted(Path(folder).glob(pattern))iflen(hits) !=1:raiseFileNotFoundError(f'{len(hits)} files match {pattern} in {folder}')return hits[0]def cached(path, compute):"""A result of this notebook's own, computed once and kept: a table as TSV, an array as NPY.""" path = Path(path)if path.exists() andnot REDO:return pd.read_csv(path, sep='\t') if path.suffix =='.tsv'else np.load(path) value = compute() path.parent.mkdir(parents=True, exist_ok=True)if path.suffix =='.tsv': value.to_csv(path, sep='\t', index=False)else: np.save(path, value)return valuedef show(frame, precision=4, **formats):"""A table without row numbers: the columns named in `formats` their way, the others to `precision`."""return frame.style.hide(axis='index').format(formats orNone, precision=precision)def measures(values):"""Named numbers as a two-column table."""return pd.DataFrame(list(values.items()), columns=['measure', 'value']).style.hide(axis='index')def pearson(a, b): ok = np.isfinite(a) & np.isfinite(b)return stats.pearsonr(a[ok], b[ok])[0]def relative_difference(a, b):"""|a − b| over the larger of the two, cell by cell."""return np.abs(a - b) / np.maximum(np.maximum(np.abs(a), np.abs(b)), 1e-300)def rounded(decimals):"""How far rounding to `decimals` places moves a number: half the last place, and a hair over it for the float the difference is taken in."""return0.5*10.0**-decimals * (1+1e-6)#: The edges a figure counts differences between two tools in: four to a decade, from 10⁻⁹ to 1.DIFFERENCE_EDGES = np.logspace(-9, 0, 37)def differences(values):"""How many of `values` fall between each two of `DIFFERENCE_EDGES`, as `ax.stairs` draws them: the first counts every one below 10⁻⁹, an exact 0 among them. A figure of tens of thousands of cells draws, and writes to validation.txt, three dozen counts rather than a point a cell.""" counts, _ = np.histogram(np.clip(values, DIFFERENCE_EDGES[0], DIFFERENCE_EDGES[-1]), DIFFERENCE_EDGES)return counts, DIFFERENCE_EDGES# Blue is chromosomix, orange the reference.APP_COLOUR, REF ='#2a78d6', '#eb6834'INK, INK2, GRID, SURFACE, MISSING ='#0b0b0b', '#52514e', '#e7e6e2', '#fcfcfb', '#efeeea'DENSITY = LinearSegmentedColormap.from_list('density', ['#cde2fb', '#86b6ef', '#3987e5', '#1c5cab', '#0d366b'])plt.rcParams.update({'figure.facecolor': SURFACE, 'axes.facecolor': SURFACE, 'savefig.facecolor': SURFACE,'axes.edgecolor': INK2, 'axes.labelcolor': INK, 'xtick.color': INK2, 'ytick.color': INK2,'text.color': INK, 'axes.spines.top': False, 'axes.spines.right': False,'axes.grid': True, 'grid.color': GRID, 'grid.linewidth': 0.6, 'axes.axisbelow': True,'lines.linewidth': 2, 'font.size': 9, 'axes.titlesize': 10, 'axes.titleweight': 'bold','axes.titlelocation': 'left', 'legend.frameon': False, 'figure.dpi': 110,})def identity(ax, lo, hi): ax.plot([lo, hi], [lo, hi], color=INK2, lw=1, ls=(0, (4, 3)), zorder=3, gid='identity')def agreement(ax, x, y, xlabel, ylabel, title, scale='linear'):"""Every pair of values as a density, against the identity line.""" ok = np.isfinite(x) & np.isfinite(y)if scale =='log': ok &= (x >0) & (y >0) x, y = x[ok], y[ok] ax.hexbin(x, y, gridsize=70, cmap=DENSITY, mincnt=1, bins='log', linewidths=0, xscale=scale, yscale=scale) identity(ax, min(x.min(), y.min()), max(x.max(), y.max())) ax.set(xlabel=xlabel, ylabel=ylabel, title=title) ax.text(0.03, 0.97, f'{ok.sum():,} values\nPearson r = {stats.pearsonr(x, y)[0]:.4f}', transform=ax.transAxes, va='top', fontsize=8, color=INK2)def square(ax, matrix, title, cmap=DENSITY, norm=None, label=''):"""A pileup or a map as an image, its colour scale beside it; the scale is returned.""" image = ax.imshow(matrix, cmap=cmap, norm=norm, interpolation='nearest') ax.set(title=title, xticks=[], yticks=[]) ax.grid(False)return plt.colorbar(image, ax=ax, shrink=0.8, label=label)TEXT = {}SUMMARY = []def commands(lines):return'<pre>'+ html.escape('\n'.join(lines)) +'</pre>'def record(task, data, reference_, agreement_, verdict, apps, refs):"""One row of the summary, with the commands as they ran.""" SUMMARY.append({'§': len(SUMMARY) +1, 'Analysis': task, 'Data': data,'chromosomix command': commands([f'$ {APP[name]["shown"]}'for name in apps]),'Reference': reference_,'Reference command': commands([f'$ {REFS[name]["shown"]}'for name in refs]),'Agreement': agreement_, 'Verdict': verdict})#: Every number each figure draws, by the figure's label on this page: the rows of validation.txt.DRAWN = {}#: validation.txt's columns, in order: the figure and its panel, the panel's axes, what is drawn, where,#: and what it says.DRAWN_COLUMNS = ['figure', 'panel', 'x_axis', 'y_axis', 'series', 'mark', 'x', 'x_end', 'y', 'value', 'text']def words(text):"""A label or a text on one line, as a cell of a table holds it."""return' '.join(str(text).split())def floats(values):"""Values as floats, NaN where a masked array hides one: a copy, never the figure's own array."""return np.ma.filled(np.ma.array(values, dtype=float, copy=True), np.nan)def marks(ax):"""What one panel draws, a frame for each thing drawn: see `drawn`."""# A histogram is one thing drawn as many bars, and named once: on its first bar, for the legend. held = {}for container in ax.containers: given = [n for n in (container.get_label(), *(bar.get_label() for bar in container)) if n and n[0] !='_'] held.update(dict.fromkeys(container, given[0] if given else'')) texts =set(ax.texts)def name(artist): given = artist.get_gid() or held.get(artist) or artist.get_label() or''return''if given.startswith('_') else words(given)def frame(mark, series, **columns): arrays = {k: v for k, v in columns.items() if np.ndim(v)} out = pd.DataFrame(arrays) if arrays else pd.DataFrame(index=[0])return out.assign(mark=mark, series=series, **{k: v for k, v in columns.items() ifnot np.ndim(v)})def unread(artist):returnTypeError(f'validation.txt cannot read the {type(artist).__name__} a figure draws: teach marks() to, ''or the table is no longer every number the figures draw.') out = []for artist in ax.get_children():ifisinstance(artist, Line2D): x, y = floats(artist.get_xdata()), floats(artist.get_ydata()) where = artist.get_transform()if where == ax.transData: out.append(frame('line', name(artist), x=x, y=y))elif where == ax.get_yaxis_transform(): out.append(frame('rule', name(artist), y=y[0]))elif where == ax.get_xaxis_transform(): out.append(frame('rule', name(artist), x=x[0]))else:raise unread(artist)elifisinstance(artist, PolyCollection) and artist.get_array() isnotNone:# hexbin's centres are in log10 along a log axis. centre = floats(artist.get_offsets())for k, scale inenumerate((ax.get_xscale(), ax.get_yscale())):if scale =='log': centre[:, k] =10** centre[:, k] out.append(frame('hexagon', name(artist), x=centre[:, 0], y=centre[:, 1], value=floats(artist.get_array())))elifisinstance(artist, PathCollection): at = floats(artist.get_offsets()).reshape(-1, 2) out.append(frame('point', name(artist), x=at[:, 0], y=at[:, 1]))elifisinstance(artist, AxesImage): values = floats(artist.get_array()) i, j = np.indices(values.shape) keep = np.ones(values.shape, bool)if values.shape[0] == values.shape[1] and np.array_equal(values, values.T, equal_nan=True): keep &= i <= jif artist.cmap.get_bad()[3] ==0: keep &= np.isfinite(values) left, right, bottom, top = artist.get_extent() first, last = (top, bottom) if artist.origin =='upper'else (bottom, top) out.append(frame('cell', name(artist), x=left + (j[keep] +0.5) * (right - left) / values.shape[1], y=first + (i[keep] +0.5) * (last - first) / values.shape[0], value=values[keep]))elifisinstance(artist, StepPatch): steps, edges, _ = artist.get_data() out.append(frame('step', name(artist), x=floats(edges[:-1]), x_end=floats(edges[1:]), y=floats(steps)))elifisinstance(artist, Rectangle) and artist isnot ax.patch: where, x = artist.get_data_transform(), artist.get_x()if where == ax.transData: out.append(frame('bar', name(artist), x=x, x_end=x + artist.get_width(), y=artist.get_y() + artist.get_height()))elif where == ax.get_xaxis_transform(): out.append(frame('band', name(artist), x=x, x_end=x + artist.get_width()))else:raise unread(artist)elif artist in texts: out.append(frame('text', name(artist), text=words(artist.get_text())))# What is left draws no number — the axes themselves, their titles, the legend, the background —# or draws numbers this does not know how to read, which is not to be passed over in silence.elifnotisinstance(artist, (Axis, Legend, Spine, Text)) and artist isnot ax.patch:raise unread(artist)# Ticks placed by hand are the ones that name something: a chromosome, a pileup, a track.for axis, key in ((ax.xaxis, 'x'), (ax.yaxis, 'y')):ifisinstance(axis.get_major_locator(), FixedLocator): at = axis.get_ticklocs() out.append(frame('tick', '', **{key: floats(at)}, text=[words(t) for t in axis.get_major_formatter().format_ticks(at)]))return outdef drawn(label):"""Show the figure just drawn, and keep every number it draws under `label`, its label on this page: validation.txt, a row a mark, panel by panel in the order drawn. A mark is a `point`; a vertex of a `line`, in order, a gap where it has no y; a `hexagon` of a density, at its centre, its count the value; a `cell` of a picture, at its centre, its number the value — a symmetric map's upper triangle only, as it draws each of those numbers twice, and no cell it leaves transparent; a `bar` or a `step`, from x to x_end, y high; a `band` from x to x_end, and a `rule` at x or at y, across the panel; a `tick` naming a place on an axis; and a `text` written in the panel. Its series is what it stands for: the legend's name for it, or the name the code gives it (`gid`).""" frames = []for ax in plt.gcf().axes:if ax.get_label() =='<colorbar>':continue title = ax.get_title('left') or ax.get_title() letter = re.match(r'([a-z]){2}', title) frames += [part.assign(figure=label, panel=letter.group(1) if letter else words(title), x_axis=words(ax.get_xlabel()), y_axis=words(ax.get_ylabel())) for part in marks(ax)] DRAWN[label] = pd.concat(frames, ignore_index=True).reindex(columns=DRAWN_COLUMNS) plt.show()
Data and versions
Every file is public, read by its accession and fetched from the URL listed. 4DN files come from the 4DN Data Portal’s open-data bucket on AWS; the mm10 sequence, used to phase compartments by GC content and read for composition and motifs, and the RefSeq genes, for the metaplot’s TSS and the promoters, are UCSC’s as distributed by Illumina iGenomes; the yeast Micro-C map is GEO’s GSE151553, fetched from GEO; § 24’s ChIP-seq and ATAC-seq of GM12878 are ENCODE’s, from its open-data bucket, and its map is 4DN’s, of which only what § 24 reads is fetched, by range; the yeast Hi-C contact list of § 25 is the one the chromosomix demo keeps on its bucket, as pairtools wrote it; and the yeast reference of §§ 26 and 27 is Ensembl’s R64-1-1, whose sequences are the contact list’s, by name and by length. Seven are also held to an MD5, and a file of any of those names that does not match is refused rather than compared: the yeast map, 4DN’s insulation score for § 18, to the MD5 4DN publishes for it, ENCODE’s three tracks, to theirs, the contact list, and the yeast reference, which also matches the checksum Ensembl publishes beside it.
Code
FOURDN ='https://4dn-open-data-public.s3.amazonaws.com/fourfront-webprod/wfoutput'IGENOMES ='https://ngi-igenomes.s3.amazonaws.com/igenomes/Mus_musculus/UCSC/mm10'GEO ='https://ftp.ncbi.nlm.nih.gov/geo/series'ENCODE ='https://encode-public.s3.amazonaws.com'DATASETS = [dict(data='Mouse Hi-C (mm10)', accession='4DNFIOZ1TRUS', source='4DN', url=f'{FOURDN}/727b405d-c9f5-4b13-ae58-1dbf29d19a8f/4DNFIOZ1TRUS.mcool'),dict(data='Yeast Micro-C, G2/M (Costantino et al. 2020)', accession='GSE151553', source='GEO', url=f'{GEO}/GSE151nnn/GSE151553/suppl/GSE151553_A364_merged.mcool', md5='319a04a8d7e41d9eb88e2378ea3ea373'),dict(data='Chicken Hi-C (galGal5)', accession='4DNFIH8L6WON', source='4DN', url=f'{FOURDN}/4e321fa9-1abe-48e6-9920-bcf88ee3b8ce/4DNFIH8L6WON.mcool'),dict(data='Chicken Hi-C (galGal5)', accession='4DNFITBCUX56', source='4DN', url=f'{FOURDN}/f96ea651-4950-41c4-b172-06f33577347d/4DNFITBCUX56.mcool'),dict(data='Chicken Hi-C (galGal5)', accession='4DNFIIML46E5', source='4DN', url=f'{FOURDN}/4bc15ed0-155b-4a16-b433-130c1daf4a0f/4DNFIIML46E5.mcool'),dict(data='Mouse ATAC-seq (mm10), replicate 1', accession='4DNFIIY4QW2D', source='4DN', url=f'{FOURDN}/7fb6d419-a8de-4bc8-9c17-f68f814496a2/4DNFIIY4QW2D.bw'),dict(data='Mouse ATAC-seq (mm10), replicate 2', accession='4DNFITBRHVZE', source='4DN', url=f'{FOURDN}/e567a80e-c7d2-41ff-b12d-67047906c076/4DNFITBRHVZE.bw'),dict(data='Mouse ES-cell Hi-C, insulation score (mm10; Bonev et al. 2017)', accession='4DNFIUJ5CJVN', source='4DN', url=f'{FOURDN}/dfe47343-08e7-48aa-ac1e-7cfbef055ded/4DNFIUJ5CJVN.bw', md5='66c3c8c6e8351aaff6eeab4767a72b66'),dict(data='Mouse genome sequence (Illumina iGenomes copy)', accession='mm10', source='UCSC', name='mm10.fa', url=f'{IGENOMES}/Sequence/WholeGenomeFasta/genome.fa'),dict(data='Mouse RefSeq genes (Illumina iGenomes copy)', accession='refGene', source='UCSC mm10', name='mm10.genes.gtf', url=f'{IGENOMES}/Annotation/Genes/genes.gtf'),dict(data='GM12878 CTCF ChIP-seq, fold change over control (GRCh38; Bernstein lab, ENCSR000AKB)', accession='ENCFF734CUT', source='ENCODE', md5='b68aeffbebeae3d03e734b51f43c0cb8', url=f'{ENCODE}/2020/10/01/a42c46dc-c33f-4906-8469-ba93e2cf30d3/ENCFF734CUT.bigWig'),dict(data='GM12878 CTCF ChIP-seq, fold change over control (GRCh38; Stamatoyannopoulos lab, ENCSR000DRZ)', accession='ENCFF644EEX', source='ENCODE', md5='0404a4c8f2ea76f528bb5306d2431a11', url=f'{ENCODE}/2020/09/26/03eeca15-17a9-47ac-aa63-08d471c2e25d/ENCFF644EEX.bigWig'),dict(data='GM12878 ATAC-seq, fold change over control (GRCh38; ENCSR095QNB)', accession='ENCFF487LOB', source='ENCODE', md5='f95a4eeec1156e2e1dc392ab988ae28a', url=f'{ENCODE}/2021/02/24/4c49a77a-edb2-4766-8328-d44ebe7446eb/ENCFF487LOB.bigWig'),dict(data='Yeast Hi-C contact list, as pairtools wrote it', accession='CH128', source='chromosomix demo', url='https://rsglab-data.s3.eu-west-3.amazonaws.com/demo/yeast/CH128.pairs.gz', md5='e3e37117383283da0c8e57f4e0837699'),dict(data='Yeast genome sequence, S288C (R64-1-1)', accession='R64-1-1', source='Ensembl 112', url='https://ftp.ensembl.org/pub/release-112/fasta/saccharomyces_cerevisiae/dna/''Saccharomyces_cerevisiae.R64-1-1.dna.toplevel.fa.gz', md5='5ae49d2d22cd977abdd6b054b94c544d'),dict(data='Juicer Tools 1.22.01, the reference of § 28', accession='juicer_tools_1.22.01', source='Aiden lab', url='https://s3.amazonaws.com/hicfiles.tc4ga.com/public/juicer/juicer_tools_1.22.01.jar'),]PATHS = {d['accession']: fetch(d['url'], d.get('name'), d.get('md5')) for d in DATASETS}MOUSE, YEAST, MM10, GENES = PATHS['4DNFIOZ1TRUS'], PATHS['GSE151553'], PATHS['mm10'], PATHS['refGene']CH128 = PATHS['CH128']JUICER = PATHS['juicer_tools_1.22.01']#: § 28's map, 4DN's in situ Hi-C of K562 (Rao et al. 2014) as Juicer wrote it: 6.7 GB, of which both#: tools read only chromosomes 21 and 22, by range, so it is named by its address and not fetched.K562_HIC =f'{FOURDN}/dcfcb009-f006-4ab8-a4c7-af72be58c12c/4DNFITUOMFUQ.hic'CHICKEN = [PATHS[a] for a in ('4DNFIH8L6WON', '4DNFITBCUX56', '4DNFIIML46E5')]ATAC = [PATHS[a] for a in ('4DNFIIY4QW2D', '4DNFITBRHVZE')]# mm10's index, by which every tool reads the sequence by range: made once, beside it.import pyfaidxpyfaidx.Faidx(str(MM10)).close()show(pd.DataFrame([{'data': d['data'], 'accession': f"{d['source']}{d['accession']}",'MB': PATHS[d['accession']].stat().st_size /1e6, 'URL': d['url']} for d in DATASETS]), MB='{:,.0f}')
def commit(where): run = subprocess.run(['git', 'rev-parse', '--short', 'HEAD'], capture_output=True, text=True, cwd=where)return run.stdout.strip() or'unknown'def yapc_versions(): probe ='import importlib.metadata as m; print(m.version("yapc"), m.version("idr"), m.version("numpy"))'try: run = subprocess.run([str(YAPC_PYTHON), '-c', probe], capture_output=True, text=True)return run.stdout.split() if run.returncode ==0else ['?'] *3exceptFileNotFoundError:return ['not found'] *3def program_version(argv, pattern=r'v?([\d.]+)', env=None):"""A program's version, as it prints it.""" run = subprocess.run(argv, capture_output=True, text=True, env=environment(env)) found = re.search(pattern, run.stdout + run.stderr)return found.group(1) if found else'?'APP_VERSION =f"{json.loads((REPO /'package.json').read_text())['version']} at {commit(REPO)}"YAPC, IDR, YAPC_NUMPY = yapc_versions()PAIRTOOLS_VERSION = (program_version([PAIRTOOLS, '--version'], r'version ([\d.]+)', env=PAIRTOOLS_ENV)if Path(PAIRTOOLS).exists() else'not found')VERSIONS = {name: version(name) for name in ('cooler', 'cooltools', 'bioframe', 'chromosight', 'hicrep', 'domaincaller', 'pastis', 'coolpuppy','deeptools', 'pyBigWig', 'biopython', 'pyfaidx', 'numpy', 'scipy', 'pandas')}JAVA_VERSION = program_version(['java', '-version'], r'version "([^"]+)"')PROGRAMS = {'bedtools': program_version([BEDTOOLS, '--version']),'seqkit': program_version([SEQKIT, 'version']),'Boost-HiC (GitHub)': commit(BOOSTHIC_DIR)}#: Each reference by the name its commands give, with its version: for the summary and the timing.TOOL_VERSIONS = {'chromosomix': APP_VERSION, 'yapc 0.1': f'{YAPC}, with idr {IDR}','Boost-HiC': PROGRAMS['Boost-HiC (GitHub)'], 'deepTools': VERSIONS['deeptools'],'bioframe, Biopython': f"{VERSIONS['bioframe']}, {VERSIONS['biopython']}",'pyBigWig, NumPy': f"{VERSIONS['pyBigWig']}, {VERSIONS['numpy']}",'pyBigWig, SciPy': f"{VERSIONS['pyBigWig']}, {VERSIONS['scipy']}",'pairtools': PAIRTOOLS_VERSION,'Juicer Tools': f'1.22.01, on Java {JAVA_VERSION}',**VERSIONS, **PROGRAMS}show(pd.DataFrame([{'package': 'chromosomix', 'version': APP_VERSION},*[{'package': k, 'version': v} for k, v in VERSIONS.items()],*[{'package': k, 'version': v} for k, v in PROGRAMS.items()], {'package': 'yapc (own environment)', 'version': f'{YAPC}, with idr {IDR} and numpy {YAPC_NUMPY}'}, {'package': 'pairtools (own environment)', 'version': PAIRTOOLS_VERSION}, {'package': 'Juicer Tools', 'version': f'1.22.01, on Java {JAVA_VERSION}'}]))
package
version
chromosomix
0.19.0 at 997e59c
cooler
0.10.4
cooltools
0.7.1
bioframe
0.8.0
chromosight
1.6.3
hicrep
0.2.6
domaincaller
0.1.0
pastis
0.5.0
coolpuppy
1.1.0
deeptools
4.0.0
pyBigWig
0.3.26
biopython
1.88
pyfaidx
0.9.0.4
numpy
2.5.4
scipy
1.16.3
pandas
2.3.3
bedtools
2.31.1
seqkit
2.3.0
Boost-HiC (GitHub)
a615aff
yapc (own environment)
0.1, with idr 2.0.3 and numpy 1.23.5
pairtools (own environment)
1.1.2
Juicer Tools
1.22.01, on Java 21.0.13
The RefSeq annotation as transcripts: each one’s extent over its exons, on its sequence and strand. The metaplot reads their TSS, the overlap their promoters, and virtual 4C its viewpoints from them.
cooltools’ definition: per bin, the mean of the valid cells of the window × window square on its diagonal pixel, the ignoreDiags nearest diagonals left out, over the chromosome’s median, in log2; boundaries at the minima whose prominence reaches the threshold.
Figure 1: Diamond insulation, 100 kb window. a, every bin; b, every boundary both call; c, chr11, 30–40 Mb (dropped bins shaded).
Over 250,342 bins the largest difference is 5.1e-06 (float32); the same 11,089 boundaries are called, 0 by one only, with strengths within 5.1e-06.
Code
record('Diamond insulation', 'mouse Hi-C, 10 kb, window 100 kb', f"cooltools {VERSIONS['cooltools']}",f"max |difference| {TEXT['ins_max']} over {TEXT['ins_n']} bins; the same {TEXT['ins_calls']} boundaries",'Identical (float32)', apps=['insulation'], refs=['cooltools-insulation'])
2 · Directionality index
Dixon et al. (2012): sign(B − A)·(B − A)²/(A + B), A and B the contacts upstream and downstream within the window. Reference: domaincaller’s Chrom.calDI on the balanced matrix, built as its own script builds it.
chromosomix
domaincaller
window
--window 40 (2 Mb)
calDI(window=2_000_000)
cells
every cell off the diagonal, --ignoreDiags 0
the upper triangle off the diagonal
coverage
--minValid 0.5: no score where either side holds under half its window
Figure 2: Directionality index, 2 Mb window. a, every bin both score (symmetric-log axes); b, chr2, 40–70 Mb.
Over 48,094 bins the relative difference is at most 5.0e-06 (float32), and every sign agrees. The 6,346 bins only domaincaller scores are the app’s coverage rule, at chromosome ends and beside gaps.
cooltools’ preparation in both — observed/expected cis over cooltools’ own expectation, clipped at its 99.9th percentile, less one — and three eigenvectors ranked by eigenvalue and signed by GC content, computed once per 100 kb bin (bioframe.frac_gc, mm10) and given to both. cooltools signs by Spearman correlation, the app by Pearson, and the app leaves the sign unset below |r| = 0.1. chrY, mostly unmappable at 100 kb, is left out of the comparison.
chromosomix
cooltools
components
--components 3
n_eigs=3
phasing
--phaseTrack gc.bedGraph
phasing_track=gc
expectation
each diagonal’s mean, pooled from about the 35th in groups at most 3% wide (default)
phased = cmp_table[cmp_table['phasing'] =='set']TEXT['cmp_min_r'] =f"{phased['r E1'].min():.4f}"TEXT['cmp_gap'] =f"{max(1- phased['r E1'].min(), 1e-16):.0e}"TEXT['cmp_delta'] =f"{phased['E1 largest difference'].max():.0e}"TEXT['cmp_n'] =f'{len(phased)}'TEXT['cmp_eig'] =f"{phased['eigenvalue E1, relative difference'].max():.0e}"TEXT['cmp_ab'] =f"{phased['A/B agreement'].min():.2%}"ambiguous =', '.join(cmp_table.loc[cmp_table['phasing'] =='ambiguous', 'chromosome'])TEXT['cmp_subject'] = (f'the {len(phased)} chromosomes compared, every one phased by GC content'ifnot ambiguous elsef'the {len(phased)} chromosomes whose phasing is set ({ambiguous}: ambiguous, its E1 sign arbitrary)')e23 = pd.concat([phased[f'r E{k}'] for k in (2, 3)])TEXT['cmp_e23'] =f"{int((e23.abs() >=0.98).sum())} of {len(e23)}"fig, axes = plt.subplots(1, 2, figsize=(11, 4.2), layout='constrained', width_ratios=[1.25, 1])ax = axes[0]x = np.arange(len(cmp_table))for k, colour, offset in ((1, APP_COLOUR, -0.2), (2, REF, 0), (3, '#1baf7a', 0.2)): ax.scatter(x + offset, cmp_table[f'r E{k}'].abs(), s=24, color=colour, label=f'E{k}', zorder=3)ax.set_xticks(x, cmp_table['chromosome'].str.replace('chr', ''), fontsize=7)ax.set(ylim=(0, 1.05), xlabel='chromosome', ylabel='|Pearson r|, chromosomix vs cooltools', title='a each component, each chromosome')ax.legend(loc='lower left', ncols=3)ax = axes[1]s = chrom100 =='chr1'xs = ref_vec['start'].to_numpy()[s] /1e6ax.plot(xs, ref_vec['E1'].to_numpy(float)[s], color=REF, lw=3, label='cooltools')ax.plot(xs, app_vec[1][s], color=APP_COLOUR, lw=1.2, label='chromosomix')ax.axhline(0, color=INK2, lw=0.8)ax.set(xlabel='chr1 (Mb)', ylabel='E1', title='b E1 along chr1')ax.legend(loc='lower left', ncols=2)drawn('fig-compartments')
Figure 3: Compartments at 100 kb, both signed by GC content. a, correlation of each component between the two tools, chromosome by chromosome; b, E1 along chr1.
On the 20 chromosomes compared, every one phased by GC content, E1 correlates at r ≥ 1.0000 — 1 − r at most 1e-12 — no bin of it more than 5e-06 from cooltools’, which is the six significant digits the track is written to, eigenvalues agree within 1e-06, relative, and the A/B call on at least 100.00% of bins. E2 and E3 agree up to sign on 40 of 40 chromosome-components (|r| ≥ 0.98) — up to sign because one that follows GC content at |r| < 0.1 keeps the sign the app’s solver gave it, where cooltools signs it all the same. The two decompose the same matrix: the app divides by cooltools’ own expectation, which eigs_cis takes from each chromosome’s square rather than from expected_cis — each diagonal’s mean to about the thirty-fifth and the mean of a group of them, at most 3% wide, beyond. Divided by each diagonal’s own mean, as the app did until version 4 of the task, E1 agreed at r = 0.9985 to 0.99993, and on chr4 and chr15 one tool’s third component was the other’s fourth.
Code
record('Compartments', 'mouse Hi-C, 100 kb, GC-phased', f"cooltools {VERSIONS['cooltools']}",f"E1 1 − r ≤ {TEXT['cmp_gap']} on {TEXT['cmp_n']} chromosomes, no bin more than {TEXT['cmp_delta']} "f"apart; eigenvalues within {TEXT['cmp_eig']}; A/B ≥ {TEXT['cmp_ab']}",'Same eigenvectors'+ (f' (phasing ambiguous on {ambiguous})'if ambiguous else''), apps=['compartments'], refs=['cooltools-eigs'])
4 · P(s)
--quantity expected --scope file: per band of separations (factor 1.1), the balanced contacts of every chromosome over the bin pairs that could hold them, at the band’s pair-weighted mean separation, from the app’s default minimum separation of two bins — the diagonal and the one beside it left out, as cooltools’ own default leaves them out (ignore_diags=2). cooltools’ per-diagonal sums (expected_cis, every diagonal) are pooled over the same bands, each a run of consecutive diagonals from the first the app kept, whose weighted mean is the separation the app writes. The app counts the bin pairs each separation could hold from the bins the balancer dropped, rather than by walking every pair: the same integers, in four hundredths of a second over the mouse map.
Code
PS = WORK /'ps'YEAST_800 =f'{rel(YEAST)}::/resolutions/800'PS_CASES = {'yeast Micro-C, 800 bp': ('yeast', YEAST_800, f'{YEAST}::/resolutions/800', 'yeast 800 bp'),'mouse Hi-C, 10 kb': ('mouse', MOUSE_10K, f'{MOUSE}::/resolutions/10000', 'mouse 10 kb')}EXPECTED = script('cooltools_expected_cis')def bands(n_valid, separations, binsize, first):"""The run of consecutive diagonals behind each separation the app wrote: from `first`, then from where the last ended, the longest run whose valid-pixel-weighted mean matches it to its six written digits.""" k = np.arange(len(n_valid), dtype=float) cn = np.concatenate([[0.0], np.cumsum(n_valid)]) ck = np.concatenate([[0.0], np.cumsum(n_valid * k)]) runs, lo = [], firstfor x in separations: tolerance =max(0.5, abs(x) *5e-6) match =Nonefor hi inrange(lo +1, len(n_valid) +1): w = cn[hi] - cn[lo]if w <=0:continue mean = (ck[hi] - ck[lo]) / w * binsizeifabs(mean - x) <= tolerance: match = hielif mean > x + tolerance:breakif match isNone:raiseValueError(f'no run of diagonals from {lo} has its mean at {x}') runs.append((lo, match)) lo = matchreturn runsdef kept_from(path):"""The first diagonal the app kept: its minimum separation, as the table's header records it, and never the diagonal itself, which the expected leaves out whatever it says."""for line inopen(path):if line.startswith('# param minSeparation:'):returnmax(1, int(float(line.split(':', 1)[1])))raiseValueError(f'{path} does not say its minimum separation')ps = {}for label, (slug, uri, local, short) in PS_CASES.items(): mr('expected', uri, '--quantity', 'expected', '--scope', 'file', out=PS / slug, name=f'expected-{slug}', analysis='P(s)', label=short) reference(f'cooltools-expected-{slug}', 'P(s)', 'cooltools', [PYTHON, EXPECTED, uri, OUT, THREADS], out=PS /f'{slug}-cooltools.tsv', label=short, threads=THREADS) saved = one(PS / slug, '*.expected.ps.tsv') app = table(saved) cvd = pd.read_csv(PS /f'{slug}-cooltools.tsv', sep='\t') cvd['balanced.sum'] = cvd['balanced.sum'].fillna(0) ref = cvd.groupby('dist')[['n_valid', 'balanced.sum']].sum() n = ref['n_valid'].reindex(range(int(ref.index.max()) +1), fill_value=0).to_numpy(float) total = ref['balanced.sum'].reindex(range(len(n)), fill_value=0).to_numpy(float) x, y = app.iloc[:, 0].to_numpy(float), app.iloc[:, 1].to_numpy(float) runs = bands(n, x, cooler.Cooler(local).binsize, kept_from(saved)) pooled = np.array([total[a:b].sum() / n[a:b].sum() for a, b in runs]) ps[label] =dict(x=x, app=y, ref=pooled, rel=np.abs(y / pooled -1), reach=x.max(), first=runs[0][0])show(pd.DataFrame([{'map': k, 'points': len(v['x']), 'first diagonal': v['first'],'furthest separation (Mb)': v['reach'] /1e6,'relative difference, max': v['rel'].max()} for k, v in ps.items()]),**{'furthest separation (Mb)': '{:.1f}', 'relative difference, max': '{:.1e}'})
yapc (Jänes et al. 2018): the concave regions of the pooled signal’s smoothed second derivative, scored in each replicate and tested for reproducibility by IDR. Both tools are given the two replicates cut to chr19, interval for interval (yapc holds a whole genome’s second derivative in memory). Reference: yapc 0.1’s command line with IDR 2.0.3, NumPy’s random seed fixed. yapc breaks score ties at random, so it is run with five seeds, and their spread is the yardstick for IDR.
chromosomix
yapc 0.1
smoothing
--width 150 --times 3
--smoothing-window-width 150 --smoothing-times 3
narrowest region
--minWidth 75
--min-concave-region-width 75
regions given to IDR
--truncate 100000
--truncate-idr-input 100000
IDR threshold
--idr 0.05, and a count at each of yapc’s six
one BED at each of 0.001–0.2
Code
YP = WORK /'yapc'REPS = [DATA /f'{path.stem}.chr19.bw'for path in ATAC]def chr19(source, target):"""The track with chr19 only, interval for interval."""if target.exists() andnot REDO:return handle = pyBigWig.open(str(source)) out = pyBigWig.open(str(target), 'w') out.addHeader([('chr19', handle.chroms()['chr19'])]) entries = handle.intervals('chr19') or () out.addEntries(['chr19'] *len(entries), [int(a) for a, _, _ in entries], ends=[int(b) for _, b, _ in entries], values=[float(v) for _, _, v in entries]) out.close() handle.close()for source, target inzip(ATAC, REPS): chr19(source, target)mr('peaks', '--signal', REPS[0], '--replicate', REPS[1], '--perPeak', '--width', 150, '--times', 3,'--minWidth', 75, '--truncate', 100000, '--idr', 0.05, out=YP /'app', name='peaks', analysis='Peak calling (yapc, IDR)')YAPC_SCRIPT = script('yapc_idr')SEEDS = {seed: YP /f'seed{seed}'for seed inrange(5)}for seed, folder in SEEDS.items(): reference(f'yapc-seed{seed}', 'Peak calling (yapc, IDR)', 'yapc 0.1', [str(YAPC_PYTHON), YAPC_SCRIPT, seed, OUT, 'cond', *REPS, '--smoothing-window-width', 150,'--smoothing-times', 3, '--min-concave-region-width', 75, '--truncate-idr-input', 100000], out=folder /'yapc', made=folder /'yapc_0.05.bed', env=YAPC_ENV, label=f'seed {seed}', bench=seed ==0)THRESHOLDS = [0.001, 0.005, 0.01, 0.05, 0.1, 0.2]peaksall = pd.read_csv(SEEDS[0] /'yapc_peaksall.tsv', sep='\t')ref_tested = pd.read_csv(SEEDS[0] /'yapc.tsv', sep='\t')listing = table(one(YP /'app', '*.peaks-list.tsv')).rename(columns={'sequence': 'chrom'})counts_app = table(one(YP /'app', '*.peaks-idr.tsv'))called_app = pd.read_csv(one(YP /'app', '*.bed'), sep='\t', header=None, comment='#', names=['chrom', 'start', 'end', 'name', 'score', 'strand'])key = ['chrom', 'start', 'end']bounds = ['chrom', 'concave_start', 'concave_end']m = listing.merge(ref_tested.drop(columns=['start', 'end']), left_on=key, right_on=bounds, how='outer', indicator=True)shared = m[m['_merge'] =='both']curvature = np.abs(np.r_[shared['curvature0'] - shared['cond_rep1_score'], shared['curvature1'] - shared['cond_rep2_score']])def bed(th, seed=0): out = pd.read_csv(SEEDS[seed] /f'yapc_{th}.bed', sep='\t', header=None, comment='#').iloc[:, :3] out.columns = keyreturn outcalled_ref = bed(0.05)same_called = called_app[key].merge(called_ref, on=key)idr = called_app.merge(ref_tested, left_on=key, right_on=bounds)# −log10 IDR over the peaks called: the app against each seed, and the seeds against one another.runs = {seed: pd.read_csv(folder /'yapc.tsv', sep='\t').set_index(bounds)['cond_globalIDR']for seed, folder in SEEDS.items()}app_idr = called_app.set_index(key)['score'].rename_axis(bounds)idr_app = [np.abs(runs[seed].reindex(app_idr.index) - app_idr).max() for seed in SEEDS]idr_self = [np.abs(runs[a].reindex(app_idr.index) - runs[b].reindex(app_idr.index)).max()for a in SEEDS for b in SEEDS if a < b]note =next(n for n in NOTES['peaks'] if'concave region'in n)found_app, wide_app = (int(re.search(rf'([\d,]+){w}', note).group(1).replace(',', ''))for w in ('concave region', 'at least'))wide_ref =int(((peaksall['concave_end'] - peaksall['concave_start']) >=75).sum())measures({'concave regions, chromosomix / yapc': f'{found_app:,} / {len(peaksall):,}','at least 75 bp, chromosomix / yapc': f'{wide_app:,} / {wide_ref:,}','given to IDR, chromosomix / yapc': f'{len(listing):,} / {len(ref_tested):,}','same bounds; in one only': f"{len(shared):,}; {int((m['_merge'] !='both').sum())}",'same summit': f"{(shared['summit'] == shared['mode']).mean():.2%}",'curvature, max |difference| (two decimals written)': f'{curvature.max():.4f}','peaks at IDR ≤ 0.05, chromosomix / yapc; same': f'{len(called_app):,} / {len(called_ref):,}; {len(same_called):,}','−log10 IDR of those peaks, max |difference| to each yapc seed': f'{min(idr_app):.1e} to {max(idr_app):.1e}','−log10 IDR, yapc seeds against one another': f'{min(idr_self):.1e} to {max(idr_self):.1e}',})
curvature, max |difference| (two decimals written)
0.0050
peaks at IDR ≤ 0.05, chromosomix / yapc; same
5,666 / 5,666; 5,666
−log10 IDR of those peaks, max |difference| to each yapc seed
5.4e-06 to 8.7e-04
−log10 IDR, yapc seeds against one another
5.9e-04 to 8.7e-04
Code
def counted(th): values =sorted({len(bed(th, seed)) for seed in SEEDS})returnf'{values[0]:,}'iflen(values) ==1elsef'{values[0]:,}–{values[-1]:,}'show(pd.DataFrame([{'IDR ≤': th, f'yapc 0.1, seeds 0–{len(SEEDS) -1}': counted(th),'chromosomix': f"{int(counts_app.loc[np.isclose(counts_app['threshold'], th), 'peaks'].iloc[0]):,}"}for th in THRESHOLDS]), **{'IDR ≤': '{:g}'})
IDR ≤
yapc 0.1, seeds 0–4
chromosomix
0.001
3,279
3,279
0.005
4,126
4,126
0.01
4,538
4,538
0.05
5,666
5,666
0.1
6,386
6,386
0.2
7,584
7,584
Code
TEXT['yapc_n'] =f'{len(shared):,}'TEXT['yapc_curv'] =f'{curvature.max():.4f}'TEXT['yapc_called'] =f'{len(same_called):,}'TEXT['yapc_idr'] =f'{max(idr_app):.1e}'TEXT['yapc_self'] =f'{max(idr_self):.1e}'TEXT['yapc_counts'] ='every one'ifall(len(bed(th, seed)) ==int(counts_app.loc[np.isclose(counts_app['threshold'], th), 'peaks'].iloc[0])for th in THRESHOLDS for seed in SEEDS) else'not every one'# Within the spread of yapc's own runs, the six significant digits the app's BED writes aside.IDR_WITHIN =max(idr_app) <=max(idr_self) +1e-5fig, axes = plt.subplots(1, 2, figsize=(10, 4.4), layout='constrained')agreement(axes[0], np.r_[shared['cond_rep1_score'], shared['cond_rep2_score']], np.r_[shared['curvature0'], shared['curvature1']], 'yapc 0.1', 'chromosomix', 'a curvature')agreement(axes[1], idr['cond_globalIDR'].to_numpy(float), idr['score'].to_numpy(float), 'yapc 0.1','chromosomix', 'b −log10 IDR, called peaks')drawn('fig-yapc')
Figure 5: yapc with IDR on two mouse ATAC-seq replicates, chr19. a, the curvature of every region given to IDR, both replicates; b, −log10 IDR of every peak called at IDR ≤ 0.05.
The same 100,000 regions go to IDR, with the same summits and curvatures within the two decimals written. 5,666 peaks pass IDR ≤ 0.05 in both, the same regions; their −log10 IDR is within 8.7e-04 of every yapc run, whose five seeds differ from one another by up to 8.7e-04. The counts agree with every seed at every one of yapc’s six thresholds.
Code
record('Peak calling (yapc, IDR)', 'mouse ATAC-seq, 2 replicates, chr19', f'yapc {YAPC}, idr {IDR}',f"{TEXT['yapc_n']} regions, same bounds and summits; {TEXT['yapc_called']} peaks at IDR ≤ 0.05; "f"−log10 IDR within {TEXT['yapc_idr']} of 5 yapc seeds (seeds among themselves: up to {TEXT['yapc_self']})",'Identical peaks'+ ('; IDR within yapc\'s seed-to-seed spread'if IDR_WITHIN else''), apps=['peaks'], refs=[f'yapc-seed{seed}'for seed in SEEDS])
6 · Loops (chromosight)
Chromosight’s loop preset (kernels/loops.json) in both; calls matched by bin pair, p- and q-values read from each call’s detail. The app sums four neighbouring windows of the correlation at once, each adding its cells in chromosight’s order, so its scores are those of one window at a time.
Code
LP = WORK /'loops'mr('chromosight', YEAST_800, '--pattern', 'loops', out=LP /'app', name='chromosight', analysis='Loops (chromosight)')reference('chromosight', 'Loops (chromosight)', 'chromosight', [CHROMOSIGHT, 'detect', '--pattern', 'loops', '--threads', THREADS, YEAST_800, OUT], out=LP /'chromosight'/'loops', made=LP /'chromosight'/'loops.tsv', threads=THREADS)ref_loops = pd.read_csv(LP /'chromosight'/'loops.tsv', sep='\t')app_loops = table(one(LP /'app', '*.chromosight-patterns.tsv'))for key_ in ('p', 'q'): app_loops[key_] = app_loops['detail'].str.extract(rf'\b{key_}([0-9.e+-]+)')[0].astype(float)m = app_loops.merge(ref_loops, on=['chrom1', 'start1', 'chrom2', 'start2'], how='outer', indicator=True, suffixes=('_app', '_ref'))shared = m[m['_merge'] =='both']# chromosight writes p and q to ten decimals: compared where that leaves four significant digits.readable = shared[shared['pvalue'] >=1e-6]readable_q = shared[shared['qvalue'] >=1e-6]measures({'loops, chromosomix / chromosight': f'{len(app_loops):,} / {len(ref_loops):,}','same bin pair; in one only': f"{len(shared):,}; {int((m['_merge'] !='both').sum())}",'score, max |difference|': f"{np.abs(shared['score_app'] - shared['score_ref']).max():.1e}",f'p, max relative difference ({len(readable):,} loops with p ≥ 1e-6)':f"{np.abs(readable['p'] / readable['pvalue'] -1).max():.1e}",f'q, max relative difference ({len(readable_q):,} loops with q ≥ 1e-6)':f"{np.abs(readable_q['q'] / readable_q['qvalue'] -1).max():.1e}",})
Figure 6: Loops, yeast Micro-C at 800 bp. a, the score of every loop both call; b, their p-values where chromosight’s ten decimals leave four significant digits.
659 loops at the same bin pairs (0 in one only); scores within 5.1e-07, p-values within 4.2e-03, the three significant digits printed in a call’s detail.
Code
record('Loops (chromosight)', 'yeast Micro-C, 800 bp', f"chromosight {VERSIONS['chromosight']}",f"{TEXT['loops_n']} loops, the same bin pairs; scores within {TEXT['loops_score']}",'Identical', apps=['chromosight'], refs=['chromosight'])
7 · Reproducibility (HiCRep)
The stratum-adjusted correlation (Yang et al. 2017) as hicrep 0.2.6 computes it: each map’s upper triangle, the main diagonal and every diagonal past the furthest compared set to zero, a 2D mean filter of half-width h, a Pearson correlation per diagonal over the positions where either map has a contact, and weights (n + 1)/12. Every pair of the three maps, on every chromosome longer than the furthest diagonal compared.
chromosomix
hicrep
counts
raw, --no-balance
raw (balance=False)
smoothing
--smooth 1
h=1
furthest diagonal
--maxDistance 50 (5 Mb)
dBPMax=5_000_000
depth
as read
bDownSample=False
Code
HR = WORK /'hicrep'RES, STRATA =100_000, 50LABELS = [path.stem for path in CHICKEN]URIS = {label: f'{rel(path)}::/resolutions/{RES}'for label, path inzip(LABELS, CHICKEN)}SEQS = [c for c, length in cooler.Cooler(f'{CHICKEN[0]}::/resolutions/{RES}').chromsizes.items()if length >= (STRATA +1) * RES]PAIRS =f'{len(LABELS)} maps, {len(LABELS) * (len(LABELS) -1) //2} pairs'against = [a for label in LABELS[1:] for a in ('--against', URIS[label])]mr('hicrep', URIS[LABELS[0]], *against, '--maxDistance', STRATA, '--smooth', 1, '--no-balance','--sequences', ','.join(SEQS), out=HR /'app', name='hicrep', analysis='Reproducibility (HiCRep)', label=PAIRS)reference('hicrep', 'Reproducibility (HiCRep)', 'hicrep', [PYTHON, script('hicrep_scc'), OUT, ','.join(SEQS), *[f'{label}={URIS[label]}'for label in LABELS]], out=HR /'hicrep.tsv', label=PAIRS)names = {'This map': LABELS[0], **{path.name: label for label, path inzip(LABELS, CHICKEN)}}app_scc = table(one(HR /'app', '*.hicrep-sequences.tsv'))app_scc['a'], app_scc['b'] = app_scc['a'].map(names), app_scc['b'].map(names)def unordered(frame): order = {label: i for i, label inenumerate(LABELS)} swap = frame['a'].map(order) > frame['b'].map(order) frame.loc[swap, ['a', 'b']] = frame.loc[swap, ['b', 'a']].to_numpy()return framescc = (unordered(app_scc[['a', 'b', 'sequence', 'scc']].rename(columns={'scc': 'chromosomix'})) .merge(unordered(pd.read_csv(HR /'hicrep.tsv', sep='\t')), on=['a', 'b', 'sequence']))scc_diff = scc['chromosomix'] - scc['hicrep']measures({'values (pairs × chromosomes)': f'{len(scc):,}','mean difference': f'{scc_diff.mean():+.1e}','max |difference|': f'{scc_diff.abs().max():.1e}','Pearson r': f"{pearson(scc['chromosomix'].to_numpy(), scc['hicrep'].to_numpy()):.6f}",})
TEXT['hr_n'] =f'{len(scc)}'TEXT['hr_max'] =f'{scc_diff.abs().max():.1e}'fig, axes = plt.subplots(1, 2, figsize=(10, 4.2), layout='constrained')ax = axes[0]ax.scatter(scc['hicrep'], scc['chromosomix'], s=10, color=APP_COLOUR, alpha=0.6, lw=0)identity(ax, 0, 1)ax.set(xlim=(0, 1), ylim=(0, 1), xlabel='hicrep SCC', ylabel='chromosomix SCC', title='a every value')ax.set_aspect('equal')ax = axes[1]ax.hist(scc_diff *1e5, bins=np.linspace(-6, 6, 49), color=APP_COLOUR)for edge in (-5, 5): ax.axvline(edge, color=INK2, lw=1, ls=(0, (4, 3)), gid='the rounding of four decimals')ax.set(xlabel='chromosomix − hicrep (×10⁻⁵)', ylabel='pairs × chromosomes', title='b difference; dashed, the rounding of four decimals')drawn('fig-hicrep')
Figure 7: HiCRep on three chicken Hi-C maps at 100 kb, every pair, every chromosome of at least 5.1 Mb. a, chromosomix against hicrep; b, their difference, value by value.
Over 78 values the two differ by at most 5.0e-05, the rounding of the four decimals the app writes an SCC to.
Counts drawn from a negative binomial of mean bᵢ·bⱼ·d^−α, α = 3, with bᵢ = 1/wᵢ from the file’s balancing weights scaled to a mean of 1, zeros taken as counts, and a dispersion that is a curve against the mean. Reference: pastis (GitHub 6888ac2; Varoquaux, Noble & Vert 2023) through its own functions — the moments at each separation and the dispersion fitted against the mean (compute_mean_variance, ExponentialDispersion(degree=1)), an MDS start from random positions, negative_binomial_structure.estimate_X — on the same beads (bins with a weight and a contact). chromosomix ports each of those functions, draws its random start with numpy’s own generator so a seed places the beads where pastis’ does, and runs its default 150 steps, about where pastis’ own L-BFGS-B stops on this map; pastis runs to its own tolerances. A structure has no unique solution, so each runs from the same three seeds: the two from one seed should be one model, and pastis from three seeds is the yardstick for how alike two models of this map can be.
Code
P3 = WORK /'pastis'YEAST_12K =f'{rel(YEAST)}::/resolutions/12800'for seed in (0, 1, 2): mr('reconstruct3d', YEAST_12K, '--method', 'nb', '--alpha', 3, '--seed', seed, out=P3 /f'app-seed{seed}', name=f'reconstruct3d-seed{seed}', analysis='3D model (pastis-NB)', label=f'seed {seed}', bench=seed ==0)PASTIS = script('pastis_nb')for seed in (0, 1, 2): reference(f'pastis-seed{seed}', '3D model (pastis-NB)', 'pastis', [PYTHON, PASTIS, YEAST_12K, OUT, seed], out=P3 /f'pastis-seed{seed}.npy', label=f'seed {seed}', bench=seed ==0)clr_y = cooler.Cooler(f'{YEAST}::/resolutions/12800')raw_y = clr_y.matrix(balance=False)[:].astype(float)w_y = clr_y.bins()['weight'][:].to_numpy(float)KEEP = np.isfinite(w_y) & (raw_y.sum(axis=0) >0)beads = clr_y.bins()[['chrom', 'start']][:][KEEP].reset_index(drop=True)bead_chrom = beads['chrom'].astype(str).to_numpy()def app_model(folder): bed3d = pd.read_csv(one(folder, '*.bed3d'), sep='\t', comment='#', header=None, names=['chrom', 'start', 'end', 'x', 'y', 'z'])return beads.merge(bed3d, on=['chrom', 'start'], how='left')[['x', 'y', 'z']].to_numpy(float)MODELS = {**{f'chromosomix, seed {s}': app_model(P3 /f'app-seed{s}') for s in (0, 1, 2)},**{f'pastis, seed {s}': np.load(P3 /f'pastis-seed{s}.npy') for s in (0, 1, 2)}}assertall(model.shape == (KEEP.sum(), 3) and np.isfinite(model).all() for model in MODELS.values()), \'a bead is missing from a model'
from scipy.spatial import procrustesfrom scipy.spatial.distance import pdist, squareformsame = squareform(np.equal.outer(bead_chrom, bead_chrom).astype(int), checks=False).astype(bool)dist = {name: pdist(model) for name, model in MODELS.items()}contacts = squareform(raw_y[np.ix_(KEEP, KEEP)] * np.outer(w_y[KEEP], w_y[KEEP]), checks=False)seen = contacts >0def per_chromosome_disparity(a, b):return np.median([procrustes(a[bead_chrom == c], b[bead_chrom == c])[2] for c in pd.unique(bead_chrom)])def kind(a, b): tool_a, seed_a = a.split(', seed ') tool_b, seed_b = b.split(', seed ')if tool_a != tool_b:return'chromosomix vs pastis, same seed'if seed_a == seed_b else'chromosomix vs pastis, other seeds'returnf'{tool_a} vs {tool_b}'rows = []names_ =list(MODELS)for i, a inenumerate(names_):for b in names_[i +1:]: rows.append({'models': f'{a} / {b}', 'kind': kind(a, b),'within chromosomes, Spearman ρ': stats.spearmanr(dist[a][same], dist[b][same])[0],'between chromosomes, Spearman ρ': stats.spearmanr(dist[a][~same], dist[b][~same])[0],'per-chromosome Procrustes disparity': per_chromosome_disparity(MODELS[a], MODELS[b])})models_table = pd.DataFrame(rows)fit_table = pd.DataFrame([{'model': name, 'fit, all pairs': stats.spearmanr(1/ d[seen], contacts[seen])[0],'fit, within chromosomes': stats.spearmanr(1/ d[seen & same], contacts[seen & same])[0]}for name, d in dist.items()])display(show(models_table, precision=3))show(fit_table, precision=3)
models
kind
within chromosomes, Spearman ρ
between chromosomes, Spearman ρ
per-chromosome Procrustes disparity
chromosomix, seed 0 / chromosomix, seed 1
chromosomix vs chromosomix
0.936
0.501
0.118
chromosomix, seed 0 / chromosomix, seed 2
chromosomix vs chromosomix
0.945
0.624
0.092
chromosomix, seed 0 / pastis, seed 0
chromosomix vs pastis, same seed
0.988
0.982
0.016
chromosomix, seed 0 / pastis, seed 1
chromosomix vs pastis, other seeds
0.941
0.503
0.094
chromosomix, seed 0 / pastis, seed 2
chromosomix vs pastis, other seeds
0.942
0.566
0.095
chromosomix, seed 1 / chromosomix, seed 2
chromosomix vs chromosomix
0.953
0.543
0.079
chromosomix, seed 1 / pastis, seed 0
chromosomix vs pastis, other seeds
0.938
0.499
0.117
chromosomix, seed 1 / pastis, seed 1
chromosomix vs pastis, same seed
0.993
0.997
0.002
chromosomix, seed 1 / pastis, seed 2
chromosomix vs pastis, other seeds
0.950
0.517
0.091
chromosomix, seed 2 / pastis, seed 0
chromosomix vs pastis, other seeds
0.943
0.625
0.092
chromosomix, seed 2 / pastis, seed 1
chromosomix vs pastis, other seeds
0.953
0.544
0.079
chromosomix, seed 2 / pastis, seed 2
chromosomix vs pastis, same seed
0.978
0.902
0.036
pastis, seed 0 / pastis, seed 1
pastis vs pastis
0.942
0.501
0.093
pastis, seed 0 / pastis, seed 2
pastis vs pastis
0.940
0.569
0.094
pastis, seed 1 / pastis, seed 2
pastis vs pastis
0.949
0.517
0.085
model
fit, all pairs
fit, within chromosomes
chromosomix, seed 0
0.473
0.871
chromosomix, seed 1
0.480
0.866
chromosomix, seed 2
0.479
0.867
pastis, seed 0
0.473
0.873
pastis, seed 1
0.480
0.868
pastis, seed 2
0.476
0.868
The fit is the Spearman correlation of 1/d with the balanced contacts over every pair of beads the map counted.
Code
cross = models_table[models_table['kind'] =='chromosomix vs pastis, same seed']selfs = models_table[models_table['kind'] =='pastis vs pastis']def spread(frame, column, digits):returnf"{frame[column].min():.{digits}f}–{frame[column].max():.{digits}f}"TEXT['p3_beads'] =f'{KEEP.sum():,}'TEXT['p3_intra_cross'] = spread(cross, 'within chromosomes, Spearman ρ', 3)TEXT['p3_intra_self'] = spread(selfs, 'within chromosomes, Spearman ρ', 3)TEXT['p3_inter_cross'] = spread(cross, 'between chromosomes, Spearman ρ', 2)TEXT['p3_inter_self'] = spread(selfs, 'between chromosomes, Spearman ρ', 2)TEXT['p3_disparity_cross'] = spread(cross, 'per-chromosome Procrustes disparity', 3)TEXT['p3_disparity_self'] = spread(selfs, 'per-chromosome Procrustes disparity', 3)fits = fit_table.set_index('model')['fit, all pairs']within = fit_table.set_index('model')['fit, within chromosomes']app_names = [n for n in fits.index if n.startswith('chromosomix')]pastis_names = [n for n in fits.index if n.startswith('pastis')]TEXT['p3_fit_app'] =f"{fits[app_names].min():.3f}–{fits[app_names].max():.3f}"TEXT['p3_fit_pastis'] =f"{fits[pastis_names].min():.3f}–{fits[pastis_names].max():.3f}"TEXT['p3_fit_app_within'] =f"{within[app_names].min():.3f}–{within[app_names].max():.3f}"TEXT['p3_fit_pastis_within'] =f"{within[pastis_names].min():.3f}–{within[pastis_names].max():.3f}"def versus(app, reference_):return ('better than'if app.min() > reference_.max() else'less well than'if app.max() < reference_.min()else'as well as')TEXT['p3_within_word'] = versus(within[app_names], within[pastis_names])TEXT['p3_all_word'] = versus(fits[app_names], fits[pastis_names])fig, axes = plt.subplots(1, 2, figsize=(11, 4.6), layout='constrained', width_ratios=[1, 1.15])def scaled(model): step = np.median(np.linalg.norm(np.diff(model, axis=0), axis=1)[bead_chrom[1:] == bead_chrom[:-1]])return pdist(model) / stepagreement(axes[0], scaled(MODELS['pastis, seed 0'])[same], scaled(MODELS['chromosomix, seed 0'])[same],'pastis, seed 0', 'chromosomix, seed 0', 'a distances within chromosomes', scale='log')ax = axes[1]for col, metric inenumerate(('within chromosomes, Spearman ρ', 'between chromosomes, Spearman ρ')):for offset, frame, colour, label in ((-0.12, cross, APP_COLOUR, 'chromosomix vs pastis, same seed'), (0.12, selfs, REF, 'pastis vs pastis')): ax.scatter(np.full(len(frame), col + offset), frame[metric], s=36, color=colour, label=label if col ==0elseNone, zorder=3, gid=label)ax.set_xticks([0, 1], ['within chromosomes', 'between chromosomes'])ax.set(xlim=(-0.5, 1.5), ylim=(0, 1.02), ylabel='Spearman ρ of pairwise distances', title='b how alike two runs are')ax.legend(loc='lower left')drawn('fig-pastis')
Figure 8: pastis-NB, yeast Micro-C at 12.8 kb. a, distances within chromosomes, chromosomix against pastis from the same seed, 0, each over its model’s median distance between neighbouring beads; b, how alike two runs are, within and between chromosomes: chromosomix and pastis from one seed, and pastis from two.
On 905 beads, chromosomix and pastis from the same seed give the same model: distances within chromosomes correlate at ρ = 0.978–0.993, against 0.940–0.949 between pastis’ own seeds; between chromosomes at 0.90–1.00 (0.50–0.57); and the median per-chromosome Procrustes disparity is 0.002–0.036 (0.085–0.094). chromosomix’s models fit the contacts within chromosomes as well as pastis’ (0.866–0.871 against 0.868–0.873), and over all pairs, the chromosomes’ arrangement included, as well as pastis’ (0.473–0.480 against 0.473–0.480).
Code
record('3D model (pastis-NB)', f"yeast Micro-C, 12.8 kb, {TEXT['p3_beads']} beads", 'pastis 0.5.0 (GitHub 6888ac2)',f"same seed: within chromosomes ρ = {TEXT['p3_intra_cross']}, between {TEXT['p3_inter_cross']} "f"(pastis vs pastis {TEXT['p3_intra_self']}, {TEXT['p3_inter_self']}); "f"fit within chromosomes {TEXT['p3_fit_app_within']} vs {TEXT['p3_fit_pastis_within']}, "f"all pairs {TEXT['p3_fit_app']} vs {TEXT['p3_fit_pastis']}",f"Same seed, same model; fits within chromosomes {TEXT['p3_within_word']} pastis; "f"all pairs {TEXT['p3_all_word']} pastis", apps=['reconstruct3d-seed0', 'reconstruct3d-seed1', 'reconstruct3d-seed2'], refs=['pastis-seed0', 'pastis-seed1', 'pastis-seed2'])
9 · Coverage
Per bin, the contacts of every pixel that touches it: within its own chromosome (cis) and over the genome (total). cooltools’ coverage adds a pixel at both of its ends, bincount(bin1) + bincount(bin2), so a pixel on the diagonal is counted twice; the app counts it once, a bin’s contact with itself being one contact. cooltools’ value less the bin’s own diagonal pixel is therefore the app’s, and that is what is compared, raw and balanced.
chromosomix
cooltools
counts
raw, the default, and --balance
clr_weight_name=None, and 'weight'
diagonals
every one
ignore_diags=0
outputs
--cis --trans --fraction, with total
cis and total
Code
COV = WORK /'coverage'WANTED = ['--cis', '--trans', '--fraction']mr('coverage', MOUSE_10K, *WANTED, out=COV /'raw', name='coverage-raw', analysis='Coverage', label='raw')mr('coverage', MOUSE_10K, *WANTED, '--balance', out=COV /'balanced', name='coverage-balanced', analysis='Coverage', label='balanced')for kind, extra in (('raw', []), ('balanced', ['weight'])): reference(f'cooltools-coverage-{kind}', 'Coverage', 'cooltools', [PYTHON, script('cooltools_coverage'), MOUSE_10K, OUT, THREADS, *extra], out=COV /f'cooltools-{kind}.tsv', label=kind, threads=THREADS)def self_contacts(clr, chunk=10_000_000):"""Each bin's pixel on the diagonal, copies of it added, and then how many pixels the file stores twice — one array, the count last. Read whole rows at a time, so that a pixel and its copy, which are in one row, are always read together."""with clr.open('r') as group: offsets = group['indexes/bin1_offset'][:] n =len(offsets) -1 diagonal, twice = np.zeros(n), 0 rows = np.unique(np.r_[np.searchsorted(offsets, np.arange(0, offsets[-1], chunk)), n])for a, b inzip(rows[:-1], rows[1:]): px = clr.pixels()[int(offsets[a]):int(offsets[b])] b1, b2 = px['bin1_id'].to_numpy(), px['bin2_id'].to_numpy() on = b1 == b2 diagonal += np.bincount(b1[on], weights=px['count'].to_numpy(float)[on], minlength=n) pairs = b1.astype(np.int64) * n + b2 twice +=len(pairs) -len(np.unique(pairs))return np.append(diagonal, twice)held = cached(COV /'self-contacts.npy', lambda: self_contacts(clr10))diagonal, stored_twice = held[:-1], int(held[-1])weight10 = bins10['weight'].to_numpy(float)coverage = {}for kind, own in (('raw', diagonal), ('balanced', diagonal * weight10 **2)): ref = pd.read_csv(COV /f'cooltools-{kind}.tsv', sep='\t') app = {track: table(one(COV / kind, f'*.coverage-{track}.tsv'))['value'].to_numpy(float)for track in ('cis', 'total', 'trans', 'fraction')} cis, total = ref['cis'].to_numpy(float) - own, ref['total'].to_numpy(float) - ownwith np.errstate(invalid='ignore', divide='ignore'): fraction = np.where(total >0, cis / total, np.nan) both = np.isfinite(app['total']) & np.isfinite(total) coverage[kind] =dict( app=app, ref=ref, cis=cis, total=total, both=both, gaps=(int((~np.isfinite(app['total'])).sum()), int((~np.isfinite(total)).sum())), same_gaps=bool((np.isfinite(app['total']) == np.isfinite(total)).all()), absolute=max(np.abs(app[t][both] - v[both]).max() for t, v in (('cis', cis), ('total', total))), relative=max(relative_difference(app[t][both], v[both])[v[both] !=0].max() for t, v in (('cis', cis), ('total', total))), trans=np.nanmax(np.abs(app['trans'] - (total - cis))), fraction=np.nanmax(np.abs(app['fraction'] - fraction)))raw, balanced = coverage['raw'], coverage['balanced']measures({'bins': f'{len(bins10):,}','raw cis and total, max |difference| (cooltools less the diagonal)': f"{raw['absolute']:g}",'balanced cis and total, max relative difference': f"{balanced['relative']:.1e}",'trans (total less cis) and cis fraction, max |difference|':f"{max(raw['trans'], balanced['trans']):.1e}; {max(raw['fraction'], balanced['fraction']):.1e}",'balanced bins without a value, chromosomix / cooltools':f"{balanced['gaps'][0]:,} / {balanced['gaps'][1]:,} ({'the same bins'if balanced['same_gaps'] else'not the same bins'})",'pixels the file stores twice (both tools add the copies)': f'{stored_twice:,}',})
raw cis and total, max |difference| (cooltools less the diagonal)
0
balanced cis and total, max relative difference
5.0e-06
trans (total less cis) and cis fraction, max |difference|
1.3e-06; 5.3e-07
balanced bins without a value, chromosomix / cooltools
28,572 / 28,572 (the same bins)
pixels the file stores twice (both tools add the copies)
49
Code
TEXT['cov_n'] =f"{raw['both'].sum():,}"TEXT['cov_raw'] =f"{raw['absolute']:g}"TEXT['cov_bal'] =f"{balanced['relative']:.1e}"TEXT['cov_twice'] =f'{stored_twice:,}'TEXT['cov_gaps'] =f"{balanced['gaps'][0]:,}"TEXT['cov_same'] ='the same'if raw['absolute'] ==0else'not the same'TEXT['cov_gap_words'] ='the same'if balanced['same_gaps'] else'not the same'fig = plt.figure(figsize=(11, 7.2), layout='constrained')grid = fig.add_gridspec(2, 2, height_ratios=[1.15, 1])agreement(fig.add_subplot(grid[0, 0]), raw['total'], raw['app']['total'], 'cooltools less the diagonal','chromosomix', 'a raw', scale='log')agreement(fig.add_subplot(grid[0, 1]), balanced['total'], balanced['app']['total'], 'cooltools less the diagonal','chromosomix', 'b balanced', scale='log')ax = fig.add_subplot(grid[1, :])lo, _ = clr10.extent('chr11')stretch =slice(lo +3_000, lo +4_000)x = bins10['start'].to_numpy()[stretch] /1e6ax.plot(x, raw['ref']['total'].to_numpy(float)[stretch], color=INK2, lw=0.8, label='cooltools, as written')ax.plot(x, raw['total'][stretch], color=REF, lw=3, label='cooltools less the diagonal')ax.plot(x, raw['app']['total'][stretch], color=APP_COLOUR, lw=1.2, label='chromosomix')ax.set(xlabel='chr11 (Mb)', ylabel='contacts', title='c chr11, 30–40 Mb, raw total')ax.legend(loc='upper left', ncols=3)drawn('fig-coverage')
Figure 9: Coverage at 10 kb. a, total coverage of every bin, raw; b, balanced; c, chr11, 30–40 Mb, raw: cooltools as it writes it, and less each bin’s own diagonal pixel.
Raw, cis and total are the same on the 272,564 bins (largest difference 0) once cooltools’ second count of the diagonal is taken off; balanced, they agree within 5.0e-06, the six digits written, and the 28,572 bins the app leaves without a value are the same bins as cooltools’. The file stores 49 pixels twice; both tools add the copies.
Code
record('Coverage', 'mouse Hi-C, 10 kb, raw and balanced', f"cooltools {VERSIONS['cooltools']}",f"raw: identical on {TEXT['cov_n']} bins, cooltools' diagonal counted once; balanced within {TEXT['cov_bal']}",'Identical'if raw['absolute'] ==0and balanced['same_gaps'] else'Differs', apps=['coverage-raw', 'coverage-balanced'], refs=['cooltools-coverage-raw', 'cooltools-coverage-balanced'])
10 · Virtual 4C
A viewpoint’s row of the balanced map, over the whole genome. cooltools’ virtual4c takes the stored pixels between the viewpoint and each chromosome and averages them bin by bin; for a viewpoint one bin wide that is the row itself, with no value where the map stores no contact, nor on the viewpoint. The app sums the viewpoint’s rows and writes 0 where nothing is stored; --no-cis reads every chromosome. The viewpoints are the 10 kb bins holding the 5′-most TSS of three genes of pluripotency, Sox2, Nanog and Pou5f1.
chromosomix
cooltools
viewpoint
--anchor, one bin
viewpoint, one bin
scope
--no-cis: the whole genome
the whole genome
counts
balanced (the file’s weights)
clr_weight_name='weight'
Code
V4C = WORK /'virtual4c'def promoter_bin(gene, binsize=10_000):"""The bin holding a gene's 5′-most TSS.""" rows = TRANSCRIPTS[TRANSCRIPTS['gene'] == gene] plus = rows['strand'].iloc[0] =='+' tss =int(rows['start'].min() if plus else rows['end'].max() -1)return rows['chrom'].iloc[0], tss // binsize * binsize, tss // binsize * binsize + binsizeVIEWPOINTS = {gene: promoter_bin(gene) for gene in ('Sox2', 'Nanog', 'Pou5f1')}for gene, (chrom, start, end) in VIEWPOINTS.items():# Timed at Sox2 alone: the other two viewpoints are the same computation from another bin. mr('fourc', MOUSE_10K, '--anchor', f'{chrom}:{start +1:,}-{end:,}', '--no-cis', out=V4C / gene, name=f'fourc-{gene}', analysis='Virtual 4C', label=gene, bench=gene =='Sox2') reference(f'cooltools-virtual4c-{gene}', 'Virtual 4C', 'cooltools', [PYTHON, script('cooltools_virtual4c'), MOUSE_10K, f'{chrom}:{start}-{end}', OUT, THREADS], out=V4C /f'cooltools-{gene}.tsv', label=gene, threads=THREADS, bench=gene =='Sox2')profiles, rows = {}, []for gene, (chrom, start, end) in VIEWPOINTS.items(): a = table(one(V4C / gene, '*.fourc-*.tsv'))['value'].to_numpy(float) b = pd.read_csv(V4C /f'cooltools-{gene}.tsv', sep='\t')['balanced'].to_numpy(float) at = clr10.offset(f'{chrom}:{start}-{end}') both, app_only = np.isfinite(a) & np.isfinite(b), np.isfinite(a) &~np.isfinite(b) zero = app_only & (a ==0) profiles[gene] =dict(app=a, ref=b, at=at, chrom=chrom, both=both) rows.append({'viewpoint': f'{gene}, {chrom}:{start +1:,}-{end:,}', 'bins both report': int(both.sum()),'max relative difference': relative_difference(a[both], b[both]).max(),'only chromosomix: 0': int(zero.sum()),'only chromosomix: the viewpoint': int(app_only[at] andnot zero[at]),'only chromosomix: anything else': int((app_only &~zero).sum() - (app_only[at] andnot zero[at])),'only cooltools': int((np.isfinite(b) &~np.isfinite(a)).sum()),'no value in chromosomix, at a dropped bin': f"{int((~np.isfinite(a) & np.isnan(weight10)).sum()):,} of {int((~np.isfinite(a)).sum()):,}"})v4c_table = pd.DataFrame(rows)show(v4c_table, **{'max relative difference': '{:.1e}'})
TEXT['v4c_n'] =f"{v4c_table['bins both report'].sum():,}"TEXT['v4c_max'] =f"{v4c_table['max relative difference'].max():.1e}"TEXT['v4c_zero'] =f"{v4c_table['only chromosomix: 0'].sum():,}"TEXT['v4c_other'] = ('and nothing else'if v4c_table['only chromosomix: anything else'].sum() ==0elsef"and {v4c_table['only chromosomix: anything else'].sum():,} bins besides")V4C_OK = (v4c_table['only chromosomix: anything else'].sum() ==0) & (v4c_table['only cooltools'].sum() ==0)fig, axes = plt.subplots(1, 4, figsize=(12, 3.6), layout='constrained', width_ratios=[1, 1, 1, 0.9])for ax, (letter, (gene, v)) inzip(axes, zip('abc', profiles.items())): reach =slice(v['at'] -100, v['at'] +101) x = (np.arange(len(v['app']))[reach] - v['at']) *10 a, b = v['app'][reach].copy(), v['ref'][reach].copy() a[100] = np.nan ax.scatter(x[b >0], b[b >0], s=26, color=REF, label='cooltools', zorder=2) ax.scatter(x[a >0], a[a >0], s=6, color=APP_COLOUR, label='chromosomix', zorder=3) ax.set(yscale='log', xlabel=f"from the {gene} viewpoint (kb), {v['chrom']}", title=f'{letter}{gene}')axes[0].set_ylabel('balanced contacts')axes[0].legend(loc='upper left')agreement(axes[3], np.concatenate([v['ref'][v['both']] for v in profiles.values()]), np.concatenate([v['app'][v['both']] for v in profiles.values()]), 'cooltools', 'chromosomix','d every bin', scale='log')drawn('fig-virtual4c')
Figure 10: Virtual 4C at 10 kb, balanced. a–c, each viewpoint’s own chromosome within 1 Mb of it, where either tool reports a contact (the viewpoint itself left out); d, every bin of the three profiles both report, over the whole genome.
On the 1,099 bins cooltools reports, the two agree within 4.5e-06 (float32, six digits written). The bins only the app writes are 730,874 at 0, where the map stores no contact with the viewpoint, the viewpoints themselves, and nothing else; where the app writes no value, balancing dropped the bin.
Code
record('Virtual 4C', 'mouse Hi-C, 10 kb, three viewpoints, whole genome', f"cooltools {VERSIONS['cooltools']}",f"every bin cooltools reports ({TEXT['v4c_n']}) within {TEXT['v4c_max']}; the others 0 or the viewpoint",'Identical (float32)'if V4C_OK else'Differs', apps=[f'fourc-{g}'for g in VIEWPOINTS], refs=[f'cooltools-virtual4c-{g}'for g in VIEWPOINTS])
11 · Aggregate peak analysis (coolpuppy)
The mean window about a set of loops. The loops are chromosight’s own calls on this map (chromosight detect, its loop preset), given to both tools as one BEDPE; each window runs 5 bins either side of a loop, ±50 kb. Both cut a window as coolpuppy does by default: from the upper triangle only — a window that crosses the diagonal holds the same contacts twice there, once above it and once mirrored below — with the main diagonal and the one beside it left out (min_diag=2); and both leave out every loop nearer the diagonal than twice the flank and two bins (mindist='auto'), which is where a window would start to reach those two diagonals. That floor is coolpuppy’s default and not the app’s, whose nearest loop is five bins unless told otherwise, so the app is given it, --minDist 12 — as it is given the window, the diagonals left out and the normalisation, each at coolpuppy’s value rather than at a default of its own. Both run at coolpuppy’s defaults, and then again with every loop (mindist=0, --minDist 0): those are the windows that reach the diagonal.
Six pileups are compared cell for cell, gap for gap: raw and over the expected — each window divided cell by cell by its chromosome’s expected at that cell’s own diagonal before the mean, the expected being cooltools’ expected_cis from § 4, which coolpuppy reads as balanced.avg — at coolpuppy’s defaults and with every loop; and the same two on the diagonal, at every loop anchor. Then the distance-matched normalisation: each pileup over the pileup of controls, five per loop, shifted along the diagonal by 50 to 500 bins either way — against coolpuppy’s own shifted controls over the same range. Both draw their controls at random, so each is run with five seeds.
chromosomix
coolpuppy
window
--mode fixed --window 5 (bins either side)
flank=50_000
near the diagonal
--minDiag 2 --minDist 12: twice --window and 2
nothing: min_diag is 2, and mindist'auto', by default
from itertools import combinations, productAPA = WORK /'apa'FLANK =5reference('chromosight-mouse', 'Aggregate peak analysis', 'chromosight', [CHROMOSIGHT, 'detect', '--pattern', 'loops', '--threads', THREADS, MOUSE_10K, OUT], out=APA /'loops', made=APA /'loops.tsv', label='the loops, called', threads=THREADS, bench=False)called = pd.read_csv(APA /'loops.tsv', sep='\t')apart = ((called['start2'] - called['start1']) // clr10.binsize).to_numpy()LOOPS, ANCHORS = APA /'loops.bedpe', APA /'anchors.bed'called[['chrom1', 'start1', 'end1', 'chrom2', 'start2', 'end2']].to_csv(LOOPS, sep='\t', header=False, index=False)ends = [called[[f'chrom{k}', f'start{k}', f'end{k}']].set_axis(['chrom', 'start', 'end'], axis=1) for k in (1, 2)]anchors = pd.concat(ends).drop_duplicates().sort_values(['chrom', 'start'])anchors.to_csv(ANCHORS, sep='\t', header=False, index=False)# A window reaches the first two diagonals where its loop is nearer than twice the flank and two, which# is also where coolpuppy's default floor is, `mindist='auto'`; the app's own is five bins, so it is# given this one. With every loop, the lower-left corner, nearest the diagonal, comes only from the# loops whose windows do not.MIN_DIAG, MIN_DIST =2, 2* FLANK +2reaching =int((apart < MIN_DIST).sum())cornered =int((apart -2* FLANK >=2).sum())PILEUP = script('coolpuppy_pileup')EXPECTED_MOUSE = PS /'mouse-cooltools.tsv'#: name: (label, chromosomix's arguments, coolpuppy's features and options). The app's window, its#: diagonals left out, its nearest loop and its normalisation are named at coolpuppy's values: the#: app's own defaults are not all coolpuppy's, and a default can move.CUTS = {'raw': ('raw', ['--pairs', LOOPS, '--norm', 'none', '--minDist', MIN_DIST], [LOOPS]),'oe': ('over the expected', ['--pairs', LOOPS, '--norm', 'oe', '--minDist', MIN_DIST], [LOOPS, '--expected', EXPECTED_MOUSE]),'raw-every': ('raw, every loop', ['--pairs', LOOPS, '--norm', 'none', '--minDist', 0], [LOOPS, '--mindist', 0]),'oe-every': ('over the expected, every loop', ['--pairs', LOOPS, '--norm', 'oe', '--minDist', 0], [LOOPS, '--expected', EXPECTED_MOUSE, '--mindist', 0]),'local': ('on the diagonal, raw', ['--anchors', 'features', '--features', ANCHORS, '--norm', 'none'], [ANCHORS, '--local']),'local-oe': ('on the diagonal, over the expected', ['--anchors', 'features', '--features', ANCHORS, '--norm', 'oe'], [ANCHORS, '--local', '--expected', EXPECTED_MOUSE]),}for key, (label, app_args, (features_, *options)) in CUTS.items():# Timed raw and over controls, not over the expected: there coolpuppy is handed § 4's expected,# computed beforehand, and chromosomix computes its own, so the two times are not of one job. timed ='oe'notin key.split('-') mr('apa', MOUSE_10K, *app_args, '--mode', 'fixed', '--window', FLANK, '--minDiag', MIN_DIAG, out=APA / key, name=f'apa-{key}', analysis='Aggregate peak analysis', label=label, bench=timed) reference(f'coolpuppy-{key}', 'Aggregate peak analysis', 'coolpuppy', [PYTHON, PILEUP, MOUSE_10K, features_, FLANK * clr10.binsize, OUT, THREADS, *options], out=APA /f'coolpuppy-{key}.tsv', label=label, threads=THREADS, bench=timed)SHIFTS =range(5)for seed in SHIFTS: mr('apa', MOUSE_10K, '--pairs', LOOPS, '--norm', 'matched', '--controls', 5, '--minShift', 50,'--maxShift', 500, '--seed', seed +1, '--mode', 'fixed', '--window', FLANK, '--minDiag', MIN_DIAG,'--minDist', MIN_DIST, out=APA /f'controls-seed{seed +1}', name=f'apa-controls-seed{seed +1}', analysis='Aggregate peak analysis', label=f'controls, seed {seed +1}', group='controls', bench=seed ==0) reference(f'coolpuppy-controls-seed{seed}', 'Aggregate peak analysis', 'coolpuppy', [PYTHON, PILEUP, MOUSE_10K, LOOPS, FLANK * clr10.binsize, OUT, THREADS, '--nshifts', 5,'--minshift', 50* clr10.binsize, '--maxshift', 500* clr10.binsize + clr10.binsize //2,'--seed', seed], out=APA /f'coolpuppy-controls-seed{seed}.tsv', label=f'controls, seed {seed}', group='controls', threads=THREADS, bench=seed ==0)def pileup_of(folder, pattern='*.apa-0.tsv'):"""A pileup chromosomix wrote, without its row and column names.""" written_ = table(one(folder, pattern))return written_.set_index(written_.columns[0]).to_numpy(float)def p2ll(pileup):"""The app's P2LL: the mean of the centre over that of the lower-left corner, each a fifth of the window.""" quadrant =max(1, min(pileup.shape) //5) half, centre = quadrant //2, pileup.shape[0] //2return (np.nanmean(pileup[centre - half:centre + half +1, centre - half:centre + half +1])/ np.nanmean(pileup[-quadrant:, :quadrant]))cut_app = {k: pileup_of(APA / k) for k in CUTS}cut_ref = {k: np.loadtxt(APA /f'coolpuppy-{k}.tsv') for k in CUTS}same_gaps = {k: bool((np.isnan(cut_app[k]) == np.isnan(cut_ref[k])).all()) for k in CUTS}cut_rel = {k: relative_difference(cut_app[k], cut_ref[k])[np.isfinite(cut_ref[k])] for k in CUTS}controls_app = np.stack([pileup_of(APA /f'controls-seed{s +1}') for s in SHIFTS])controls_ref = np.stack([np.loadtxt(APA /f'coolpuppy-controls-seed{s}.tsv') for s in SHIFTS])kept = np.isfinite(controls_ref[0])welch = stats.ttest_ind(controls_app[:, kept], controls_ref[:, kept], axis=0, equal_var=False).pvaluedef gaps(left, right=None):"""The largest cell difference between two runs, for every pair of runs: one of each stack, or any two of one stack.""" pairs = product(left, right) if right isnotNoneelse combinations(left, 2)return np.array([np.nanmax(np.abs(a - b)) for a, b in pairs])# A typical pair of runs, so that a pair of the two tools and a pair of one tool are read alike.gap_cross, gap_ref, gap_app = (np.median(g) for g in (gaps(controls_app, controls_ref), gaps(controls_ref), gaps(controls_app)))# Cells at p < 0.01 by chance alone: what a binomial gives as often as one time in twenty.by_chance = stats.binom.sf(int((welch <0.01).sum()) -1, welch.size, 0.01) >0.05centre = FLANKmeasures({'loops called; nearer than coolpuppy\'s floor (windows reaching the first two diagonals); anchors':f"{len(called):,}; {reaching:,}; {len(anchors):,}",'with every loop, the loops the lower-left corner cell is a mean of, in both tools': f'{cornered:,}',**{f'{CUTS[k][0]}: max relative difference; cells left as gaps, chromosomix / coolpuppy':f'{cut_rel[k].max():.1e}; {int(np.isnan(cut_app[k]).sum())} / {int(np.isnan(cut_ref[k]).sum())}'for k in CUTS},'over controls, the centre: chromosomix seeds / coolpuppy seeds':f"{controls_app[:, centre, centre].min():.3f}–{controls_app[:, centre, centre].max():.3f} / "f"{controls_ref[:, centre, centre].min():.3f}–{controls_ref[:, centre, centre].max():.3f}",'over controls, P2LL: chromosomix seeds / coolpuppy seeds':f"{min(map(p2ll, controls_app)):.3f}–{max(map(p2ll, controls_app)):.3f} / "f"{min(map(p2ll, controls_ref)):.3f}–{max(map(p2ll, controls_ref)):.3f}",'over controls, a typical pair of runs\' largest cell difference: chromosomix vs coolpuppy; ''two of coolpuppy; two of chromosomix': f'{gap_cross:.3f}; {gap_ref:.3f}; {gap_app:.3f}','over controls, cells whose five seeds differ (Welch, p < 0.01)': f'{int((welch <0.01).sum())} of {welch.size}',})
loops called; nearer than coolpuppy's floor (windows reaching the first two diagonals); anchors
2,515; 1,826; 4,952
with every loop, the loops the lower-left corner cell is a mean of, in both tools
689
raw: max relative difference; cells left as gaps, chromosomix / coolpuppy
3.6e-06; 0 / 0
over the expected: max relative difference; cells left as gaps, chromosomix / coolpuppy
4.0e-06; 0 / 0
raw, every loop: max relative difference; cells left as gaps, chromosomix / coolpuppy
3.9e-06; 0 / 0
over the expected, every loop: max relative difference; cells left as gaps, chromosomix / coolpuppy
4.4e-06; 0 / 0
on the diagonal, raw: max relative difference; cells left as gaps, chromosomix / coolpuppy
3.6e-06; 31 / 31
on the diagonal, over the expected: max relative difference; cells left as gaps, chromosomix / coolpuppy
4.5e-06; 31 / 31
over controls, the centre: chromosomix seeds / coolpuppy seeds
3.829–3.971 / 3.778–3.958
over controls, P2LL: chromosomix seeds / coolpuppy seeds
2.417–2.464 / 2.410–2.450
over controls, a typical pair of runs' largest cell difference: chromosomix vs coolpuppy; two of coolpuppy; two of chromosomix
0.134; 0.151; 0.132
over controls, cells whose five seeds differ (Welch, p < 0.01)
1 of 121
Code
TEXT['apa_loops'] =f'{len(called):,}'TEXT['apa_reaching'] =f'{reaching:,}'TEXT['apa_anchors'] =f'{len(anchors):,}'TEXT['apa_cornered'] =f'{cornered:,}'TEXT['apa_rel'] =f'{max(v.max() for v in cut_rel.values()):.1e}'APA_GAPS =all(same_gaps.values())TEXT['apa_gaps'] ='the same cells left as gaps'if APA_GAPS else'different cells left as gaps'TEXT['apa_cross'] =f'{gap_cross:.3f}'TEXT['apa_self'] =f'{gap_ref:.3f}'TEXT['apa_self_app'] =f'{gap_app:.3f}'TEXT['apa_welch'] =f'{int((welch <0.01).sum())} of {welch.size}'APA_WITHIN = by_chanceTEXT['apa_chance'] = (f'about as many as chance gives ({0.01* welch.size:.1f} expected)'if by_chanceelsef'more than chance gives ({0.01* welch.size:.1f} expected)')fig, axes = plt.subplots(1, 3, figsize=(12, 3.9), layout='constrained', width_ratios=[1, 1, 1.1])grey = plt.get_cmap('RdBu_r').copy()grey.set_bad(MISSING)bar = square(axes[0], cut_app['oe-every'], 'a every loop, over the expected', cmap=grey, norm=LogNorm(1/3, 3), label='observed / expected')bar.set_ticks([1/3, 1/2, 1, 2, 3], labels=['⅓', '½', '1', '2', '3'])bar.minorticks_off()ax = axes[1]for k, key inenumerate(CUTS): values = np.maximum(cut_rel[key], 1e-9) ax.scatter(k + np.linspace(-0.25, 0.25, values.size), values, s=6, color=APP_COLOUR, lw=0, gid=CUTS[key][0])ax.set(yscale='log', ylim=(1e-9, 1e-3), ylabel='|chromosomix / coolpuppy − 1|', title='b every cell, six pileups')ax.set_xticks(range(len(CUTS)), ['raw', 'O/E', 'raw,\nevery\nloop', 'O/E,\nevery\nloop', 'diagonal,\nraw','diagonal,\nO/E'], fontsize=8)ax = axes[2]for k, (name, stack, colour) inenumerate((('chromosomix', controls_app, APP_COLOUR), ('coolpuppy', controls_ref, REF))): ax.scatter(np.full(len(stack), 0+ (k -0.5) *0.25), stack[:, centre, centre], s=30, color=colour, label=name, zorder=3) ax.scatter(np.full(len(stack), 1+ (k -0.5) *0.25), [p2ll(p) for p in stack], s=30, color=colour, zorder=3, gid=name)ax.set_xticks([0, 1], ['centre', 'P2LL'])ax.set(xlim=(-0.6, 1.6), ylabel='over shifted controls', title='c five seeds each')ax.legend(loc='upper right')drawn('fig-apa')
Figure 11: Pileups at chromosight’s loops, mouse Hi-C at 10 kb, ±50 kb. a, every loop over the expected, as chromosomix draws it with --minDist 0; the cells towards the lower left are means of fewer loops, those whose windows stay clear of the diagonal there; b, every cell of the six pileups against coolpuppy’s, raw and over the expected (O/E); c, over shifted controls at coolpuppy’s defaults, the centre and P2LL of each tool’s five seeds.
Raw and over the expected, at coolpuppy’s defaults and with every loop, and on the diagonal both ways, the two tools draw the same cells left as gaps and every other cell within 4.5e-06 (float32, six digits written). At coolpuppy’s floor both leave out the same 1,826 of the 2,515 loops, those near enough that their windows would reach the diagonal; with every loop, a cell near the diagonal is the mean of the loops whose windows have it above the diagonal — the lower-left corner, of 689 of them, in both — and on the diagonal every one of the 4,952 anchors is piled up.
Over shifted controls, a run of each tool and a run of the other differ, in a typical pair, by up to 0.134 in a cell; two runs of coolpuppy by up to 0.151, and two of the app by up to 0.132. Cell by cell, the five seeds of each tool differ at 1 of 121 cells at p < 0.01, about as many as chance gives (1.2 expected).
Code
record('Aggregate peak analysis', f"mouse Hi-C, 10 kb, {TEXT['apa_loops']} loops", f"coolpuppy {VERSIONS['coolpuppy']}",f"raw and over the expected, at coolpuppy's defaults and with every loop, and on the diagonal: "f"{TEXT['apa_gaps']}, every other cell within "f"{TEXT['apa_rel']}; over controls, two runs of the two tools differ as two of coolpuppy do "f"({TEXT['apa_cross']} against {TEXT['apa_self']}), and no more cells than chance gives "f"({TEXT['apa_welch']} at p < 0.01)", ('Identical (float32); controls within the seeds\' spread'if APA_WITHIN else'Identical; controls differ')if APA_GAPS else'Differ', apps=[*[f'apa-{k}'for k in CUTS], *[f'apa-controls-seed{s +1}'for s in SHIFTS]], refs=['chromosight-mouse', *[f'coolpuppy-{k}'for k in CUTS], *[f'coolpuppy-controls-seed{s}'for s in SHIFTS]])
12 · APA, chromosight’s score
chromosight’s own reading of a loop set: the map detrended by its distance law, a 17-bin window about each loop, correlated with the loop template for a score, and averaged into a pileup. apa --norm chromosight is chromosight’s quantification, ported (§18.9); the reference is chromosight quantify on the same map and the same loops, its pileup chromosight’s own pileup_patterns of the windows it writes.
Figure 12: chromosight’s reading of the same loops. a, the pileup chromosight quantify’s windows make; b, chromosomix’s; c, every loop’s score.
The same 2,515 loops are scored, every score within 5.0e-05, the four decimals the app writes, and the pileups agree within 4.8e-06.
Code
record("APA, chromosight's score", f"mouse Hi-C, 10 kb, {TEXT['apa_loops']} loops", f"chromosight {VERSIONS['chromosight']}",f"{TEXT['csq_n']} loops scored, within {TEXT['csq_score']}; pileup within {TEXT['csq_pileup']}",'Identical', apps=['apa-chromosight'], refs=['chromosight-quantify'])
13 · Boost-HiC
Boost-HiC (Carron et al. 2019) reads a contact map as a graph whose edges are contacts turned into distances, 1/cᵅ, and replaces each by the shortest path between its ends: a contact the library was too shallow to see is inferred from the ones around it. It is what an oversample draws a chromosome’s new contacts from (§6.5: --contacts above a map’s own total), and chromosomix boost writes that map as Boost-HiC’s main.py writes its own — the bins its outlier filter keeps, and the boosted map over them in SCN units. Boost-HiC itself is run as main.py runs it, through its own HiCutils, on the same chromosome’s raw counts. Three chromosomes at 100 kb, the smallest, one between and the largest: Floyd–Warshall is cubic in the bins.
chromosomix
Boost-HiC
α
0.2, the default
0.2, main.py’s
outliers
a kernel density of the coverage, 5 % and 99.5 %, bandwidth 2,000
get_outliers
boost
ten rounds of SCN, Floyd–Warshall on 1/SCNᵅ, the decay put back
SCN, fastFloyd, adjustPdS
Code
BH = WORK /'boost-hic'BOOSTED = {chrom: int(np.diff(clr100.extent(chrom))[0]) for chrom in ('chr19', 'chr11', 'chr1')}for chrom, n in BOOSTED.items(): label =f'{chrom}, {n:,} bins'# Timed on chr1 alone, the largest: the other two are the same computation on fewer bins. mr('boost', MOUSE_100K, '--sequences', chrom, out=BH / chrom, name=f'boost-{chrom}', analysis='Boost-HiC', label=label, bench=chrom =='chr1') reference(f'boosthic-{chrom}', 'Boost-HiC', 'Boost-HiC', [PYTHON, script('boosthic'), MOUSE_100K, chrom, OUT], out=BH /f'boosthic-{chrom}', made=BH /f'boosthic-{chrom}-boosted.npy', env=BOOSTHIC_ENV, label=label, bench=chrom =='chr1')boosted, rows = {}, []for chrom, n in BOOSTED.items(): square_ = table(one(BH / chrom, f'*.boost-{chrom}.tsv')) app = square_.iloc[:, 1:].to_numpy(float) kept_app = (square_.iloc[:, 0].to_numpy() // clr100.binsize).astype(int) kept_ref = np.load(BH /f'boosthic-{chrom}-kept.npy') ref = np.load(BH /f'boosthic-{chrom}-boosted.npy') same = np.array_equal(kept_app, kept_ref) boosted[chrom] =dict(app=app, ref=ref, kept=kept_ref, same=same) rows.append({'chromosome': chrom, 'bins': n, 'kept, chromosomix / Boost-HiC': f'{len(kept_app):,} / {len(kept_ref):,}','the same bins': 'yes'if same else'no','cells': app.size if same else np.nan,'max relative difference': relative_difference(app, ref).max() if same else np.nan})boost_table = pd.DataFrame(rows)show(boost_table, **{'max relative difference': '{:.1e}', 'cells': '{:,.0f}', 'bins': '{:,}'})
BOOST_SAME = boost_table['the same bins'].eq('yes').all()TEXT['bh_cells'] =f"{int(boost_table['cells'].sum()):,}"TEXT['bh_max'] =f"{boost_table['max relative difference'].max():.1e}"TEXT['bh_kept'] = ('Both keep the same bins on every chromosome'if BOOST_SAMEelse'The two keep different bins on '+', '.join(boost_table.loc[boost_table['the same bins'] =='no', 'chromosome']))fig, axes = plt.subplots(1, 3, figsize=(12, 3.9), layout='constrained')v = boosted['chr19']counts = np.asarray(clr100.matrix(balance=False).fetch('chr19'), float)[np.ix_(v['kept'], v['kept'])]square(axes[0], np.where(counts >0, counts, np.nan), 'a chr19, raw', norm=LogNorm(), label='contacts')square(axes[1], v['app'], 'b chr19, boosted (chromosomix)', norm=LogNorm(), label='SCN units')agreement(axes[2], np.concatenate([b['ref'].ravel() for b in boosted.values()]), np.concatenate([b['app'].ravel() for b in boosted.values()]), 'Boost-HiC', 'chromosomix','c every cell', scale='log')drawn('fig-boost')
Figure 13: Boost-HiC at 100 kb. a, chr19’s raw counts over the bins Boost-HiC keeps; b, the boosted map chromosomix writes, in SCN units; c, every cell of the three chromosomes, chromosomix against Boost-HiC.
Both keep the same bins on every chromosome, and over the 5,404,076 cells of the chromosomes where they are the same the boosted maps agree within 5.0e-10, the ten digits chromosomix boost writes.
Code
record('Boost-HiC', 'mouse Hi-C, 100 kb, chr19, chr11, chr1', f"Boost-HiC (GitHub {PROGRAMS['Boost-HiC (GitHub)']})",f"the same bins kept; {TEXT['bh_cells']} cells within {TEXT['bh_max']}",'Identical'if BOOST_SAME else'Differs', apps=[f'boost-{c}'for c in BOOSTED], refs=[f'boosthic-{c}'for c in BOOSTED])
14 · Metaplot (deepTools)
The mean of a track about a set of features: replicate 1 of the mouse ATAC-seq, the whole genome, about the TSS of every RefSeq transcript, ±2 kb in 50 bp bins, a minus-strand window read backwards. deepTools’ computeMatrix reference-point reads every base of a window (pyBigWig.values) and averages the ones the track covers bin by bin, as the app does; deepTools’ profile is plotProfile’s, each bin’s mean over the transcripts with a value there. The app ranks its rows, one per transcript, by their mean and deepTools keeps the BED’s order, so rows are paired by that ranking, rows of the same mean in whichever order matches.
chromosomix
deepTools
anchor
--mode reference --at start, stranded (the default)
reference-point --referencePoint TSS
window
--flank 2000 --step 50
-b 2000 -a 2000 --binSize 50
average
the mean (the default), no band (--band none)
--averageTypeBins mean; plotProfile
Code
MP = WORK /'metaplot'MP.mkdir(parents=True, exist_ok=True)signal = pyBigWig.open(str(ATAC[0]))covered = signal.chroms()signal.close()TSS_BED = MP /'transcripts.bed'TRANSCRIPTS[TRANSCRIPTS['chrom'].isin(covered)][BED6].to_csv(TSS_BED, sep='\t', header=False, index=False)mr('metaplot', '--tracks', ATAC[0], '--features', TSS_BED, '--mode', 'reference', '--at', 'start', '--flank', 2000,'--step', 50, '--band', 'none', out=MP /'app', name='metaplot', analysis='Metaplot', group='TSS')reference('computematrix', 'Metaplot', 'deepTools', [COMPUTEMATRIX, 'reference-point', '-S', ATAC[0], '-R', TSS_BED, '--referencePoint', 'TSS', '-b', 2000,'-a', 2000, '--binSize', 50, '-p', THREADS, '-o', OUT], out=MP /'deeptools.mat.gz', label='computeMatrix', group='TSS', threads=THREADS)reference('plotprofile', 'Metaplot', 'deepTools', [PLOTPROFILE, '-m', MP /'deeptools.mat.gz', '-out', f'{OUT}.png', '--outFileNameData', f'{OUT}.tab'], out=MP /'deeptools-profile', made=MP /'deeptools-profile.tab', label='plotProfile', group='TSS')profile_app = table(one(MP /'app', '*.metaplot.tsv'))profile_ref = pd.read_csv(MP /'deeptools-profile.tab', sep='\t', header=None, skiprows=2).iloc[0, 2:].to_numpy(float)offsets = profile_app.iloc[:, 0].to_numpy(float)profile_app = profile_app.iloc[:, 1].to_numpy(float)rows_app = table(one(MP /'app', '*.metaplot-heatmap.tsv')).iloc[:, 1:].to_numpy(float)rows_ref = pd.read_csv(MP /'deeptools.mat.gz', sep='\t', header=None, skiprows=1).iloc[:, 6:].to_numpy(float)def written(ref):"""What two writers' rounding allows: the app's six significant digits, deepTools' six decimals."""return5e-7+5e-6* np.abs(ref)def paired_rows(app, ref):"""For each of the app's rows, strongest first, the deepTools row paired with it: of those whose mean is the same within the rounding, the unpaired one closest to it."""def strength(rows):with np.errstate(invalid='ignore'): means = np.nanmean(rows, axis=1)return np.where(np.isnan(means), -np.inf, means) mine, theirs = strength(app), strength(ref) order = np.argsort(theirs, kind='stable') sorted_ = theirs[order] used, match = np.zeros(len(ref), bool), np.empty(len(app), int)for i, row inenumerate(app): slack =1e-5*abs(mine[i]) +1e-6if np.isfinite(mine[i]) else0 lo, hi = np.searchsorted(sorted_, mine[i] - slack, 'left'), np.searchsorted(sorted_, mine[i] + slack, 'right') candidates = order[lo:hi][~used[order[lo:hi]]]iflen(candidates) ==0: candidates = np.flatnonzero(~used) pick = candidates[int(np.argmin(past(row, ref[candidates]).max(axis=1)))] used[pick], match[i] =True, pickreturn matchdef past(app, ref):"""How far each cell is past the two writers' rounding (≤ 0: within it); two gaps are within it.""" excess = np.abs(app - ref) - written(ref) excess = np.where(np.isnan(app) & np.isnan(ref), -1.0, excess)return np.where(np.isnan(excess), np.inf, excess)match = cached(MP /'paired.npy', lambda: paired_rows(rows_app, rows_ref)).astype(int)row_excess = past(rows_app, rows_ref[match])profile_rel = relative_difference(profile_app, profile_ref).max()measures({'transcripts, chromosomix / deepTools': f'{len(rows_app):,} / {len(rows_ref):,}','profile, max relative difference': f'{profile_rel:.1e}','rows paired one to one': 'every one'iflen(set(match)) ==len(match) ==len(rows_ref) else'not every one','cells past both tools\' rounding': f'{int((row_excess >0).sum()):,} of {row_excess.size:,}',})
TEXT['mp_n'] =f'{len(rows_app):,}'TEXT['mp_profile'] =f'{profile_rel:.1e}'TEXT['mp_cells'] =f'{int((row_excess >0).sum()):,}'METAPLOT_OK =int((row_excess >0).sum()) ==0andlen(set(match)) ==len(rows_ref)TEXT['mp_rows'] = ('every one of the'if METAPLOT_OK else'not every one of the')#: How many of a heatmap's rows each line of its picture is the mean of. A picture a few hundred pixels#: high shows tens of thousands of rows only by leaving most of them out; it draws their means instead.POOL =100def pictured(rows):"""A heatmap as its picture draws it: each run of POOL rows, in order, as their mean."""return np.array([np.nanmean(rows[k:k + POOL], axis=0) for k inrange(0, len(rows), POOL)])fig, axes = plt.subplots(1, 3, figsize=(12, 3.9), layout='constrained', width_ratios=[1.2, 0.8, 1])ax = axes[0]ax.plot(offsets, profile_ref, color=REF, lw=3, label='deepTools plotProfile')ax.plot(offsets, profile_app, color=APP_COLOUR, lw=1.2, label='chromosomix')ax.set(xlabel='from the TSS (bp)', ylabel='mean ATAC signal', title='a profile')ax.legend(loc='upper left')ax = axes[1]ax.imshow(pictured(rows_app), aspect='auto', cmap=DENSITY, norm=LogNorm(0.05, np.nanpercentile(rows_app, 99.5)), interpolation='nearest', extent=[offsets[0] -25, offsets[-1] +25, len(rows_app), 0])ax.set(xlabel='from the TSS (bp)', ylabel='transcripts, strongest first', title='b every transcript')ax.grid(False)agreement(axes[2], rows_ref[match].ravel(), rows_app.ravel(), 'deepTools computeMatrix', 'chromosomix','c every bin of every row', scale='log')drawn('fig-metaplot')
Figure 14: ATAC-seq about RefSeq TSS. a, the profile; b, chromosomix’s rows, strongest first, each line the mean of 100; c, every cell of every row against deepTools’ row paired with it.
The profiles agree within 4.7e-06, the six digits written, and every one of the 35,034 rows has its pair with every bin within the two tools’ rounding (0 cells past it).
Over two tracks the app draws a heatmap for each, and lists the transcripts in one order in all of them: by each one’s mean over every bin of every track, strongest first. That is plotHeatmap‘s order for a matrix of several samples, whose defaults (--sortRegions descend --sortUsing mean) sort by the mean over every sample. So both replicates at once, and the two tools’ rows compared in that order: the app’s two heatmaps side by side against computeMatrix’s rows, in the order of the regions plotHeatmap writes. A transcript neither track covers has no mean at all; the app puts it last and deepTools first, its descending order being NumPy’s ascending one reversed, and those are compared apart.
chromosomix
deepTools
tracks
--tracks, once for each replicate
computeMatrix reference-point -S, both replicates
order
the heatmaps’ own
plotHeatmap --outFileSortedRegions
Code
mr('metaplot', '--tracks', ATAC[0], '--tracks', ATAC[1], '--features', TSS_BED, '--mode', 'reference', '--at','start', '--flank', 2000, '--step', 50, '--band', 'none', out=MP /'app-two', name='metaplot-two', analysis='Metaplot', group='TSS, two tracks')reference('computematrix-two', 'Metaplot', 'deepTools', [COMPUTEMATRIX, 'reference-point', '-S', ATAC[0], ATAC[1], '-R', TSS_BED, '--referencePoint', 'TSS','-b', 2000, '-a', 2000, '--binSize', 50, '-p', THREADS, '-o', OUT], out=MP /'deeptools-two.mat.gz', label='computeMatrix', group='TSS, two tracks', threads=THREADS)reference('plotprofile-two', 'Metaplot', 'deepTools', [PLOTPROFILE, '-m', MP /'deeptools-two.mat.gz', '-out', f'{OUT}.png', '--outFileNameData', f'{OUT}.tab'], out=MP /'deeptools-two-profile', made=MP /'deeptools-two-profile.tab', label='plotProfile', group='TSS, two tracks')reference('plotheatmap-two', 'Metaplot', 'deepTools', [PLOTHEATMAP, '-m', MP /'deeptools-two.mat.gz', '-out', f'{OUT}.png', '--outFileSortedRegions',f'{OUT}.bed'], out=MP /'deeptools-two-heatmap', made=MP /'deeptools-two-heatmap.bed', label='plotHeatmap', group='TSS, two tracks')curves = table(one(MP /'app-two', '*.metaplot.tsv'))profiles_app = curves[[c for c in curves.columns[1:] ifnot c.endswith(('_low', '_high'))]].to_numpy(float).Tprofiles_ref = pd.read_csv(MP /'deeptools-two-profile.tab', sep='\t', header=None, skiprows=2).iloc[:, 2:].to_numpy(float)# The two heatmaps side by side, as computeMatrix writes two samples: a row's bins of the first, then the second's.heatmaps = [table(one(MP /'app-two', f'*.{name}.tsv')).iloc[:, 1:].to_numpy(float)for name in ('metaplot-heatmap', 'metaplot-heatmap-2')]both_app = np.hstack(heatmaps)regions = pd.read_csv(MP /'deeptools-two.mat.gz', sep='\t', header=None, skiprows=1)both_ref = regions.iloc[:, 6:].to_numpy(float)def place(frame):"""A region of a deepTools table by its coordinates, name and strand."""return frame[0].astype(str) +':'+ frame[1].astype(str) +'-'+ frame[2].astype(str) +' '+ frame[3] + frame[5]# plotHeatmap's order: the regions it wrote, top to bottom, each found among computeMatrix's rows.listed = pd.read_csv(MP /'deeptools-two-heatmap.bed', sep='\t', header=None, comment='#')order_ref = pd.Series(np.arange(len(regions)), index=place(regions))[place(listed)].to_numpy()with np.errstate(invalid='ignore'): key_app, key_ref = np.nanmean(both_app, axis=1), np.nanmean(both_ref[order_ref], axis=1)none_app, none_ref = np.isnan(key_app), np.isnan(key_ref)uncovered =int(none_app.sum())# Each tool's covered rows in its own order, held position by position to the other's: the app's uncovered# rows are its last, deepTools' its first.order_same =bool(none_ref.sum() == uncovered and none_app[len(none_app) - uncovered:].all()and none_ref[:uncovered].all()and (np.abs(key_app[~none_app] - key_ref[~none_ref]) <= written(key_ref[~none_ref])).all())match_two = cached(MP /'paired-two.npy', lambda: paired_rows(both_app, both_ref)).astype(int)two_excess = past(both_app, both_ref[match_two])profiles_rel = relative_difference(profiles_app, profiles_ref).max()measures({'transcripts, chromosomix / deepTools': f'{len(both_app):,} / {len(both_ref):,}','profiles, max relative difference': f'{profiles_rel:.1e}','rows paired one to one': 'every one'iflen(set(match_two)) ==len(match_two) ==len(both_ref) else'not every one','cells of both heatmaps past both tools\' rounding': f'{int((two_excess >0).sum()):,} of {two_excess.size:,}','order, against plotHeatmap\'s': 'the same'if order_same else'not the same','transcripts neither track covers': f'{uncovered:,}',})
TEXT['mp2_profiles'] =f'{profiles_rel:.1e}'TEXT['mp2_cells'] =f'{int((two_excess >0).sum()):,}'TEXT['mp2_uncovered'] =f'{uncovered:,}'TEXT['mp2_order'] ='in the same order'if order_same else'not in the same order'METAPLOT_TWO_OK =int((two_excess >0).sum()) ==0andlen(set(match_two)) ==len(both_ref) and order_samefig, axes = plt.subplots(1, 4, figsize=(12, 3.9), layout='constrained', width_ratios=[1.2, 0.6, 0.6, 1])ax = axes[0]for i, (mine, theirs) inenumerate(zip(profiles_app, profiles_ref)): ax.plot(offsets, theirs, color=REF, lw=3, label='deepTools plotProfile'if i ==0elseNone, gid=f'deepTools plotProfile, replicate {i +1}') ax.plot(offsets, mine, color=APP_COLOUR, lw=1.2, label='chromosomix'if i ==0elseNone, gid=f'chromosomix, replicate {i +1}')ax.set(xlabel='from the TSS (bp)', ylabel='mean ATAC signal', title='a profiles')ax.legend(loc='upper left')for ax, rows, title inzip(axes[1:3], heatmaps, ('b replicate 1', 'c replicate 2')): ax.imshow(pictured(rows), aspect='auto', cmap=DENSITY, norm=LogNorm(0.05, np.nanpercentile(rows, 99.5)), interpolation='nearest', extent=[offsets[0] -25, offsets[-1] +25, len(rows), 0]) ax.set(xlabel='from the TSS (bp)', title=title) ax.grid(False)axes[1].set_ylabel('transcripts, strongest over both first')axes[2].set_yticks([])ax = axes[3]ax.plot(key_ref[~none_ref], color=REF, lw=3, label="plotHeatmap's order")ax.plot(key_app[~none_app], color=APP_COLOUR, lw=1.2, label="chromosomix's")ax.set(yscale='log', xlabel='row, top to bottom', ylabel='mean over both tracks', title='d the order')ax.legend(loc='upper right')drawn('fig-metaplot-two')
Figure 15: Both replicates about RefSeq TSS. a, each one’s profile; b and c, chromosomix’s two heatmaps, the transcripts in one order in both, each line the mean of 100; d, each row’s mean over both tracks, top to bottom, in chromosomix’s order and in plotHeatmap’s.
Both profiles agree within 4.7e-06; every row of the two heatmaps has its pair among computeMatrix‘s, both tracks’ bins within the two tools’ rounding (0 cells past it); and the rows are in the same order as plotHeatmap’s, each one’s mean over both tracks within that rounding of the mean of the row deepTools puts there, with the 1 transcripts neither replicate covers at the other end.
Code
record('Metaplot', 'mouse ATAC-seq, one and two replicates, whole genome, RefSeq TSS ±2 kb',f"deepTools {VERSIONS['deeptools']}",f"profiles within {max(profile_rel, profiles_rel):.1e}; {TEXT['mp_n']} rows a track, paired one to one ""within the rounding; over two tracks, plotHeatmap's order",'Identical'if METAPLOT_OK and METAPLOT_TWO_OK else'Differs', apps=['metaplot', 'metaplot-two'], refs=['computematrix', 'plotprofile', 'computematrix-two', 'plotprofile-two', 'plotheatmap-two'])
15 · Interval overlap (bedtools)
How much two interval sets share: in bases, with a Jaccard index over them, and in features, those of each set that touch the other. Each tool merges a set’s own overlaps first. The sets are the ATAC-seq peaks of § 5, yapc’s at IDR ≤ 0.05 on chr19, and the promoters on chr19, 1 kb either side of every RefSeq TSS. Then the null the app sets an overlap against: the peaks moved at random, each within its own chromosome and keeping its length, a thousand times — what bedtools shuffle -chrom does. Two random draws, so the two nulls are compared as distributions.
chromosomix
bedtools
observed
overlap --a peaks --b promoters
jaccard; intersect -u, both ways
null
--shuffle --iterations 1000
shuffle -chrom -seed 1…1000, then the same
lengths
mm10’s, from --sequence
mm10’s, as a genome file
Code
OV = WORK /'overlap'OV.mkdir(parents=True, exist_ok=True)PEAKS_BED = SEEDS[0] /'yapc_0.05.bed'PROMOTERS = OV /'promoters.bed'GENOME = OV /'mm10.genome'promoters = TRANSCRIPTS.assign(start=np.maximum(0, TSS -1000), end=TSS +1000)promoters[promoters['chrom'] =='chr19'].sort_values(['start', 'end'])[BED6].to_csv( PROMOTERS, sep='\t', header=False, index=False)pd.read_csv(f'{MM10}.fai', sep='\t', header=None, usecols=[0, 1]).to_csv(GENOME, sep='\t', header=False, index=False)ITERATIONS =1000mr('overlap', '--sequence', MM10, '--features', PEAKS_BED, '--features', PROMOTERS, '--a', PEAKS_BED,'--b', PROMOTERS, '--shuffle', '--iterations', ITERATIONS, '--seed', 1, out=OV /'app', name='overlap', analysis='Interval overlap')reference('bedtools-overlap', 'Interval overlap', 'bedtools', [PYTHON, script('bedtools_overlap'), PEAKS_BED, PROMOTERS, GENOME, ITERATIONS, OUT], out=OV /'bedtools', made=OV /'bedtools-null.tsv')said = table(one(OV /'app', '*.overlap.tsv')).set_index('measure')hit =lambda measure: int(re.match(r'([\d,]+) of', said.loc[measure, 'note']).group(1).replace(',', ''))observed_app = {'intersection': int(said.loc['Shared bases', 'value']), 'union': int(said.loc['Bases in either', 'value']),'jaccard': float(said.loc['Jaccard (bases)', 'value']), 'a_hit': hit('First set hit'),'b_hit': hit('Second set hit')}observed_ref = pd.read_csv(OV /'bedtools-observed.tsv', sep='\t').iloc[0].to_dict()null_app = table(one(OV /'app', '*.overlap-null.tsv')).set_index('measure')null_ref = pd.read_csv(OV /'bedtools-null.tsv', sep='\t')NULLS = {'Shared bases': 'intersection', 'Features of the first set hit': 'a_hit'}rows = []for measure, column in NULLS.items(): mean_app =float(null_app.loc[measure, 'expected']) sd_app = (float(null_app.loc[measure, 'observed']) - mean_app) /float(null_app.loc[measure, 'z']) mean_ref, sd_ref = null_ref[column].mean(), null_ref[column].std() rows.append({'null of': measure, 'chromosomix, mean ± sd': f'{mean_app:,.1f} ± {sd_app:,.1f}','bedtools, mean ± sd': f'{mean_ref:,.1f} ± {sd_ref:,.1f}','difference of the means, in standard errors': (mean_app - mean_ref) / np.sqrt( (sd_app **2+ sd_ref **2) / ITERATIONS),'ratio of the sds': sd_app / sd_ref})null_table = pd.DataFrame(rows)display(measures({'peaks; promoters': f"{len(pd.read_csv(PEAKS_BED, sep='\t', header=None, comment='#')):,}; {sum(1for _ inopen(PROMOTERS)):,}",**{f'{k}, chromosomix / bedtools': f'{observed_app[k]:,} / {observed_ref[k]:,}'for k in ('intersection', 'union', 'a_hit', 'b_hit')},'Jaccard, chromosomix (four decimals) / bedtools': f"{observed_app['jaccard']} / {observed_ref['jaccard']}",}))show(null_table, precision=2)
OBSERVED_SAME =all(observed_app[k] ==int(observed_ref[k]) for k in ('intersection', 'union', 'a_hit', 'b_hit'))NULL_SAME = ((null_table['difference of the means, in standard errors'].abs() <3).all()and null_table['ratio of the sds'].between(0.9, 1.1).all())TEXT['ov_shared'] =f"{observed_app['intersection']:,}"TEXT['ov_count'] ='count the same'if OBSERVED_SAME else'do not count the same'TEXT['ov_null'] ='are one distribution'if NULL_SAME else'differ'TEXT['ov_union'] =f"{observed_app['union']:,}"TEXT['ov_hit'] =f"{observed_app['a_hit']:,}"TEXT['ov_z'] =f"{null_table['difference of the means, in standard errors'].abs().max():.1f}"TEXT['ov_sd'] =' to '.join(dict.fromkeys(f'{r:.2f}'for r insorted(null_table['ratio of the sds'])))fig, axes = plt.subplots(1, 2, figsize=(10, 3.6), layout='constrained')for ax, (letter, (measure, column)) inzip(axes, zip('ab', NULLS.items())): ax.hist(null_ref[column], bins=40, color=REF, alpha=0.7, label='bedtools, 1,000 shuffles') mean_app =float(null_app.loc[measure, 'expected']) sd_app = (float(null_app.loc[measure, 'observed']) - mean_app) /float(null_app.loc[measure, 'z']) ax.axvspan(mean_app - sd_app, mean_app + sd_app, color=APP_COLOUR, alpha=0.15, lw=0, gid='chromosomix, ± sd') ax.axvline(mean_app, color=APP_COLOUR, lw=2, label='chromosomix, mean ± sd') ax.axvline(observed_app[column], color=INK, lw=1.2, ls=(0, (4, 3)), label='observed') ax.set(xlabel=measure.lower(), ylabel='shuffles', title=f'{letter}{measure.lower()}')axes[0].legend(loc='upper center')drawn('fig-overlap')
Figure 16: The null of each tool for the bases peaks and promoters share on chr19 (a) and for the peaks on a promoter (b): bedtools’ thousand shuffles, chromosomix’s mean ± sd over its own thousand, and what is observed.
The two tools count the same: 191,552 bases shared of 2,693,667, and 851 peaks on a promoter, by the app’s count. Their nulls are one distribution: the means differ by at most 0.3 standard errors and the spreads are in a ratio of 0.99.
Code
record('Interval overlap', 'ATAC-seq peaks (§ 5) and promoters, chr19', f"bedtools {PROGRAMS['bedtools']}",f"the same bases and features counted; the nulls' means within {TEXT['ov_z']} standard errors", ('Identical'if OBSERVED_SAME else'Differs') + ('; the same null'if NULL_SAME else'; nulls differ'), apps=['overlap'], refs=['bedtools-overlap'])
16 · Base composition
GC content and GC skew in 10 kb bins along mm10: chromosomix composition against bioframe’s frac_gc, G and C over the bases that are not N, and Biopython’s GC_skew, (G − C)/(G + C). Biopython writes 0 for a bin with neither G nor C, where the app writes no value. The app reads mm10 through the .fai beside it, a few megabases at a time and none of them kept, and counts the bases a 32-bit word at a time; bioframe and Biopython read it through pyfaidx, bioframe a bin at a time and Biopython a chromosome at a time.
Code
COMP = WORK /'composition'mr('composition', '--sequence', MM10, '--resolution', 10_000, out=COMP /'app', name='composition', analysis='Base composition')reference('composition', 'Base composition', 'bioframe, Biopython', [PYTHON, script('composition'), MM10, 10_000, OUT], out=COMP /'reference.tsv')composition = pd.read_csv(COMP /'reference.tsv', sep='\t')for track, column in (('gc', 'gc_app'), ('gcskew', 'skew_app')): written_ = pd.read_csv(one(COMP /'app', f'*.composition-{track}.bedGraph'), sep='\t', comment='#', header=None, names=['chrom', 'start', 'end', column]) composition = composition.merge(written_[['chrom', 'start', column]], on=['chrom', 'start'], how='left')gc_app, gc_ref = composition['gc_app'].to_numpy(float), composition['GC'].to_numpy(float)skew_app, skew_ref = composition['skew_app'].to_numpy(float), composition['skew'].to_numpy(float)gc_both = np.isfinite(gc_app) & np.isfinite(gc_ref)skew_both = np.isfinite(skew_app) & np.isfinite(skew_ref)unread =~np.isfinite(skew_app) & np.isfinite(skew_ref)measures({'bins': f'{len(composition):,}','GC: bins both read; max |difference|': f'{gc_both.sum():,}; {np.abs(gc_app - gc_ref)[gc_both].max():.1e}','GC: bins neither reads (all N); read by one only':f"{int((~np.isfinite(gc_app) &~np.isfinite(gc_ref)).sum()):,}; {int((np.isfinite(gc_app) != np.isfinite(gc_ref)).sum())}",'GC skew: bins both read; max |difference|': f'{skew_both.sum():,}; {np.abs(skew_app - skew_ref)[skew_both].max():.1e}','GC skew: bins only Biopython writes, and its value there': f"{int(unread.sum()):,}, {', '.join(f'{v:g}'for v in np.unique(skew_ref[unread]))}",})
Figure 17: Composition of mm10 in 10 kb bins. a, GC content; b, GC skew; c, GC content along chr11, 30–40 Mb.
Over the 264,952 bins with sequence in them the GC content agrees within 7.2e-08 and the skew within 6.7e-08, the float32 the app writes; the 7,614 bins of N have no skew in the app and 0 in Biopython.
Code
record('Base composition', 'mm10, 10 kb bins', f"bioframe {VERSIONS['bioframe']}, Biopython {VERSIONS['biopython']}",f"GC within {TEXT['comp_gc']}, skew within {TEXT['comp_skew']} over {TEXT['comp_n']} bins",'Identical (float32)'if COMP_OK else'Differs', apps=['composition'], refs=['composition'])
17 · Motif search (seqkit)
Every place a motif occurs on chr19, on both strands, overlapping sites included: the AP-1 site TGACTCA allowing one mismatch, and the GATA site WGATAR, in IUPAC codes, exactly. Both are given chr19 alone as a FASTA. seqkit locate reports a site that reads the same on both strands twice, one per strand, where the app reports it once, as .: those are counted on both strands.
chromosomix
seqkit
strands
both (the default)
both (the default)
mismatches
--mismatches 1
-m 1
IUPAC codes
always
-d
case
ignored
-i
Code
MOTIF = WORK /'motif'CHR19_FA = DATA /'mm10.chr19.fa'if REDO ornot CHR19_FA.exists(): CHR19_FA.write_text(f">chr19\n{str(pyfaidx.Fasta(str(MM10))['chr19'][:])}\n")MOTIFS = {'TGACTCA': 1, 'WGATAR': 0}for pattern, mismatches in MOTIFS.items(): label =f'{pattern}, ≤ {mismatches} mismatch'if mismatches else pattern mr('motif', '--sequence', CHR19_FA, '--pattern', pattern, '--mismatches', mismatches, out=MOTIF / pattern, name=f'motif-{pattern}', analysis='Motif search', label=label) reference(f'seqkit-{pattern}', 'Motif search', 'seqkit', [SEQKIT, 'locate', '-i', *(['-m', mismatches] if mismatches else ['-d']), '-p', pattern, '--bed','-j', THREADS, CHR19_FA, '-o', OUT], out=MOTIF /f'seqkit-{pattern}.bed', label=label, threads=THREADS)SITE = ['chrom', 'start', 'end', 'strand']found, rows = {}, []for pattern in MOTIFS: app = pd.read_csv(one(MOTIF / pattern, '*.bed'), sep='\t', comment='#', header=None, names=['chrom', 'start', 'end', 'name', 'score', 'strand']) app = pd.concat([app[app['strand'] !='.'], *[app[app['strand'] =='.'].assign(strand=s) for s in'+-']]) ref = pd.read_csv(MOTIF /f'seqkit-{pattern}.bed', sep='\t', header=None, names=['chrom', 'start', 'end', 'name', 'score', 'strand']) m = app[SITE].merge(ref[SITE], how='outer', indicator=True) found[pattern] =dict(app=app, ref=ref) rows.append({'motif': pattern, 'sites, chromosomix / seqkit': f'{len(app):,} / {len(ref):,}','the same site and strand': int((m['_merge'] =='both').sum()),'in one only': int((m['_merge'] !='both').sum())})motif_table = pd.DataFrame(rows)show(motif_table, **{'the same site and strand': '{:,}'})
MOTIF_OK = motif_table['in one only'].eq(0).all()TEXT['motif_sites'] =' and '.join(f"{n:,}"for n in motif_table['the same site and strand'])TEXT['motif_only'] = ('and none in one only'if MOTIF_OKelsef"and {motif_table['in one only'].sum():,} in one only")fig, axes = plt.subplots(1, 2, figsize=(11, 3.4), layout='constrained')edges = np.arange(0, 62e6, 1e6)for ax, (letter, (pattern, v)) inzip(axes, zip('ab', found.items())): ax.stairs(np.histogram(v['ref']['start'], edges)[0], edges /1e6, color=REF, lw=3, label='seqkit') ax.stairs(np.histogram(v['app']['start'], edges)[0], edges /1e6, color=APP_COLOUR, lw=1.2, label='chromosomix') ax.set(xlabel='chr19 (Mb)', ylabel='sites per Mb', title=f'{letter}{pattern}')axes[0].legend(loc='lower center', ncols=2)drawn('fig-motif')
Figure 18: Sites per megabase along chr19, each motif, both strands.
The two find the same 199,858 and 145,681 sites, each on the same strand, and none in one only.
Code
record('Motif search', 'mm10 chr19, two motifs, both strands', f"seqkit {PROGRAMS['seqkit']}",f"the same {TEXT['motif_sites']} sites", 'Identical'if MOTIF_OK else'Differs', apps=[f'motif-{p}'for p in MOTIFS], refs=[f'seqkit-{p}'for p in MOTIFS])
18 · Track correlation (deepTools)
How much every pair of three public mouse tracks agrees, over whole windows of 100 bp, 1 kb and 10 kb, by Pearson’s r and by Spearman’s ρ: the two ATAC-seq replicates of § 5, and the diamond insulation score 4DN computed from mouse ES-cell Hi-C (Bonev et al. 2017), one value every 5 kb and none where the map is too sparse to give one. All three are 4DN’s, on mm10, and every tool is given them cut to chr19, interval for interval, as § 5 cuts them. Genome-wide, the two replicates are 1.7 GB of bigWig, and one deepTools summary of them at 100 bp — 27 million windows — took 23 minutes and 4.8 GB here and wrote 1.5 GB of text: a cost the timings would pay at three window sizes, five times over. A window’s value is the mean over the bases a track has a value for, and a window any of the three lacks — here, where the insulation score has no value — is left out of every pair, as deepTools leaves it out. The reference for exactly that is pyBigWig’s own exact mean, stats(..., exact=True), correlated by SciPy over the windows every track has (notebooks/scripts/pybigwig_correlation.py). Beside it is deepTools as it is usually run, multiBigwigSummary bins then plotCorrelation, which differs in one way, measured below: its window means are pyBigWig’s default, read from a zoom level wherever the file has one fine enough.
chromosomix
pyBigWig and SciPy
deepTools
windows
--binsize B, whole windows
whole windows of B
multiBigwigSummary bins --binSize B
a window’s value
the mean over the bases with a value
stats(..., exact=True)
stats(...), from a zoom level where one is fine enough
a missing window
left out of every pair
left out of every pair
left out of every pair
correlation
--method pearson, spearman
scipy.stats.pearsonr, spearmanr
plotCorrelation -c pearson, spearman
Code
TC = WORK /'trackcorr'# § 5's two replicates as § 5 cut them, and the insulation score cut by the same hand.INSULATION = DATA /f"{PATHS['4DNFIUJ5CJVN'].stem}.chr19.bw"chr19(PATHS['4DNFIUJ5CJVN'], INSULATION)TRACKS = [*REPS, INSULATION]TRACK_NAMES = [path.name for path in TRACKS]SHORT =dict(zip(TRACK_NAMES, ('ATAC, rep. 1', 'ATAC, rep. 2', 'insulation')))SIZES, METHODS = (100, 1_000, 10_000), ('pearson', 'spearman')CORR = script('pybigwig_correlation')for size in SIZES:# Both methods of each tool in one group: chromosomix answers one per run, deepTools both from# one summary of the windows. group =f'{size:,} bp windows'for method in METHODS: mr('trackcorr', '--tracks', ','.join(rel(path) for path in TRACKS), '--binsize', size,'--method', method, out=TC /f'app-{size}-{method}', name=f'trackcorr-{size}-{method}', analysis='Track correlation', label=f'{size:,} bp, {method}', group=group) reference(f'pybigwig-correlation-{size}-{method}', 'Track correlation', 'pyBigWig, SciPy', [PYTHON, CORR, OUT, size, method, *TRACKS], out=TC /f'exact-{size}-{method}.tsv', label=f'{size:,} bp, {method}', group='exact means', bench=False) reference(f'multibigwigsummary-{size}', 'Track correlation', 'deepTools', [MULTIBIGWIGSUMMARY, 'bins', '-b', *TRACKS, '--binSize', size, '-p', THREADS,'-o', OUT +'.npz', '--outRawCounts', OUT +'.tab'], out=TC /f'deeptools-{size}', made=TC /f'deeptools-{size}.npz', label=f'{size:,} bp, the windows', group=group, threads=THREADS)for method in METHODS: reference(f'plotcorrelation-{size}-{method}', 'Track correlation', 'deepTools', [PLOTCORRELATION, '-in', TC /f'deeptools-{size}.npz', '-c', method, '-p', 'heatmap','-o', OUT +'.png', '--outFileCorMatrix', OUT +'.tab'], out=TC /f'deeptools-{size}-{method}', made=TC /f'deeptools-{size}-{method}.tab', label=f'{size:,} bp, {method}', group=group)def square_of(path):"""A correlation matrix, its tracks put in one order whatever order the file wrote them in, each known by its file's name with or without the `.bw`: chromosomix names a track as the app names its row, without it, and the two references with it.""" frame = pd.read_csv(path, sep='\t', comment='#', index_col=0) frame.index = [track_name(label) for label in frame.index] frame.columns = [track_name(label) for label in frame.columns] order = [track_name(label) for label in TRACK_NAMES]return frame.loc[order, order].to_numpy(float)def exact_windows(size):"""Every track's exact mean over each whole window, by pyBigWig itself: one column a track."""def compute(): handles = [pyBigWig.open(str(path)) for path in TRACKS] chroms = handles[0].chroms()return np.stack([np.concatenate([np.array(h.stats(c, 0, (n := chroms[c] // size) * size, nBins=n, exact=True), dtype=float) for c in chroms if chroms[c] // size >0])for h in handles], axis=1)return cached(TC /f'exact-windows-{size}.npy', compute)def correlated(columns, method, together=True):"""Every pair's coefficient: over the windows every track has, or over the pair's own windows.""" measure = stats.pearsonr if method =='pearson'else stats.spearmanr rows = np.isfinite(columns).all(axis=1) if together elseNone out = []for a, b in combinations(range(columns.shape[1]), 2): keep = rows if together else np.isfinite(columns[:, a]) & np.isfinite(columns[:, b]) out.append(measure(columns[keep, a], columns[keep, b])[0])return np.array(out)UPPER = np.triu_indices(len(TRACK_NAMES), 1)tc_rows, tc = [], {}for size in SIZES: exact_cols = exact_windows(size) raw = pd.read_csv(TC /f'deeptools-{size}.tab', sep='\t') raw.columns = [str(c).strip("#'") for c in raw.columns] deep_cols = raw.loc[(raw['end'] - raw['start']) == size, TRACK_NAMES].to_numpy(float)for method in METHODS: app = square_of(one(TC /f'app-{size}-{method}', '*.trackcorr.tsv'))[UPPER] exact = square_of(TC /f'exact-{size}-{method}.tsv')[UPPER] deep = square_of(TC /f'deeptools-{size}-{method}.tab')[UPPER] own = correlated(exact_cols, method, together=False) tc[size, method] =dict(app=app, exact=exact, deep=deep, own=own) tc_rows.append({'windows': f'{size:,} bp', 'method': method, 'pairs': len(app),'left out': f'{(~np.isfinite(exact_cols).all(axis=1)).sum():,} of {len(exact_cols):,}','chromosomix vs exact, max |Δ|': np.abs(app - exact).max(),'deepTools vs exact, max |Δ|': np.abs(deep - exact).max(),'each pair over its own windows, max |Δ|': np.abs(own - exact).max()})TC_TABLE = pd.DataFrame(tc_rows)show(TC_TABLE, **{c: '{:.1e}'for c in TC_TABLE.columns if'Δ'in c or'…'in c})
TEXT['tc_n'] =f"{sum(len(v['app']) for v in tc.values())}"TEXT['tc_app'] =f"{TC_TABLE['chromosomix vs exact, max |Δ|'].max():.1e}"TEXT['tc_deep'] =f"{TC_TABLE['deepTools vs exact, max |Δ|'].max():.3f}"TEXT['tc_deep_100'] =f"{TC_TABLE.loc[TC_TABLE['windows'] =='100 bp', 'deepTools vs exact, max |Δ|'].max():.1e}"TEXT['tc_rule'] =f"{TC_TABLE['each pair over its own windows, max |Δ|'].max():.3f}"TEXT['tc_left'] = TC_TABLE.loc[TC_TABLE['windows'] =='100 bp', 'left out'].iloc[0]# The two replicates, Spearman at 100 bp: the pair the rule moves, over every track's windows and# over their own.REPLICATES =list(combinations(range(len(TRACK_NAMES)), 2)).index((0, 1))TEXT['tc_reps'] =f"{tc[100, 'spearman']['exact'][REPLICATES]:.3f}"TEXT['tc_reps_own'] =f"{tc[100, 'spearman']['own'][REPLICATES]:.3f}"TC_EXACT = TC_TABLE['chromosomix vs exact, max |Δ|'].max() <1e-5fig, axes = plt.subplots(1, 2, figsize=(11, 4.6), layout='constrained', width_ratios=[1.05, 1])ax = axes[0]heat = square_of(one(TC /'app-100-spearman', '*.trackcorr.tsv'))image = ax.imshow(heat, cmap='RdBu_r', vmin=-1, vmax=1)short = [SHORT[name] for name in TRACK_NAMES]ax.set_xticks(range(len(short)), short, rotation=45, ha='right')ax.set_yticks(range(len(short)), short)ax.grid(False)ax.set_title('a Spearman ρ, 100 bp windows')plt.colorbar(image, ax=ax, shrink=0.8, label='ρ')ax = axes[1]exact_all = np.concatenate([v['exact'] for v in tc.values()])ax.scatter(exact_all, np.concatenate([v['deep'] for v in tc.values()]), s=16, color=REF, label='deepTools', zorder=2)ax.scatter(exact_all, np.concatenate([v['app'] for v in tc.values()]), s=10, color=APP_COLOUR, label='chromosomix', zorder=3)identity(ax, exact_all.min(), exact_all.max())ax.set(xlabel='exact window means, pyBigWig and SciPy', ylabel='the tool’s coefficient', title='b every pair, both methods, three window sizes')ax.legend(loc='upper left')drawn('fig-trackcorr')
Figure 19: Three mouse tracks on chr19, correlated. a, Spearman’s ρ over 100 bp windows, as chromosomix draws it; b, every coefficient of both methods at all three window sizes against the exact reference — chromosomix in blue, deepTools in orange.
All 18 coefficients from chromosomix agree with the exact reference within 4.5e-07, the six digits it writes, over the same windows: at 100 bp, 45,715 of 614,315 windows lack a value in at least one track and are left out of every pair. deepTools’ coefficients differ by up to 0.027, all of it its window means, read from a zoom level where a file has one fine enough; at 100 bp no file has, and deepTools is within 2.9e-05, the four digits it writes. Had each pair kept its own windows, as chromosomix once did, a coefficient would have moved by up to 0.055: the two replicates’ ρ at 100 bp would read 0.591 rather than 0.537.
Code
record('Track correlation', 'mouse ATAC-seq, two replicates, and insulation, chr19, 100 bp to 10 kb windows',f"pyBigWig {VERSIONS['pyBigWig']} and SciPy {VERSIONS['scipy']}; deepTools {VERSIONS['deeptools']}",f"{TEXT['tc_n']} coefficients within {TEXT['tc_app']} of the exact means, a window any track "f"lacks left out of every pair as deepTools leaves it; deepTools within {TEXT['tc_deep']}, from "f"its zoom-level means",'Identical to the exact means'if TC_EXACT else'Differs from the exact means', apps=[f'trackcorr-{size}-{method}'for size in SIZES for method in METHODS], refs=[*[f'pybigwig-correlation-{size}-{method}'for size in SIZES for method in METHODS],*[f'multibigwigsummary-{size}'for size in SIZES],*[f'plotcorrelation-{size}-{method}'for size in SIZES for method in METHODS]])
19 · Loop calling (cooltools)
HiCCUPS as cooltools.dots computes it, with its Python API’s defaults in both, over every chromosome of the map: a pixel is tested against four neighbourhoods of its own, each pixel’s chunk of like local expected corrected on its own, and the pixels that pass are clustered and filtered. Calls matched by bin pair; the four locally adjusted expecteds and q-values of each compared to the digits chromosomix writes — four decimals, and six significant digits for a q below 10⁻⁴.
chromosomix
cooltools
kernel
the one for the map’s bins (w 5, p 2 at 10 kb)
recommend_kernels(10_000)
expected
the map’s own, the first two diagonals left out
expected_cis(ignore_diags=2)
furthest from the diagonal
1,000 bins, 10 Mb (default)
max_loci_separation=10_000_000
λ-chunks, FDR
--lambdaBins 40, --fdr 0.1 (defaults)
n_lambda_bins=40, lambda_bin_fdr=0.1
clusters
--clusteringRadius 2 bins, 20 kb, and the filter (defaults)
clustering_radius=20_000, cluster_filtering=True
Code
LC = WORK /'loopcalling'mr('loops', MOUSE_10K, out=LC /'app', name='loops', analysis='Loop calling')reference('cooltools-dots', 'Loop calling', 'cooltools', [PYTHON, script('cooltools_dots'), MOUSE_10K, OUT, THREADS], out=LC /'cooltools.tsv', threads=THREADS)#: Each kernel by cooltools' name, with the word chromosomix's table spells its q-value column with.KERNELS = {'donut': 'Donut', 'vertical': 'Vertical', 'horizontal': 'Horizontal', 'lowleft': 'Lowleft'}#: And as the figures name it.KERNEL_NAMES = {'donut': 'donut', 'vertical': 'vertical', 'horizontal': 'horizontal', 'lowleft': 'lower-left'}ref_dots = pd.read_csv(LC /'cooltools.tsv', sep='\t').rename(columns={'chrom1': 'sequence'})app_dots = table(one(LC /'app', '*.loops-table.tsv'))dots = app_dots.merge(ref_dots, on=['sequence', 'start1', 'start2'], how='outer', indicator=True, suffixes=('_app', '_ref'))both_dots = dots[dots['_merge'] =='both']lam_app = np.concatenate([both_dots[k].to_numpy(float) for k in KERNELS])lam_ref = np.concatenate([both_dots[f'la_exp.{k}.value'].to_numpy(float) for k in KERNELS])q_app = np.concatenate([both_dots[f'q{K}'].to_numpy(float) for K in KERNELS.values()])q_ref = np.concatenate([both_dots[f'la_exp.{k}.qval'].to_numpy(float) for k in KERNELS])# chromosomix writes a q to four decimals, and one below 1e-4 to six significant digits.fixed, small = q_ref >=1e-4, (q_ref <1e-4) & (q_ref >0)lc_lam = np.abs(lam_app - lam_ref).max()lc_q = np.abs(q_app[fixed] - q_ref[fixed]).max()lc_q_small = np.abs(q_app[small] / q_ref[small] -1).max()lc_zero =int((q_ref ==0).sum())LC_EXACT = (len(both_dots) ==len(app_dots) ==len(ref_dots)and (both_dots['count_app'] == both_dots['count_ref']).all()and (both_dots['cluster'] == both_dots['c_size']).all()and lc_lam <=5.0001e-5and lc_q <=5.0001e-5and lc_q_small <=5.0001e-6and (q_app[q_ref ==0] ==0).all())measures({'loops, chromosomix / cooltools': f'{len(app_dots):,} / {len(ref_dots):,}','same bin pair; in one only': f"{len(both_dots):,}; {int((dots['_merge'] !='both').sum())}",'contacts, the same at every pair': f"{int((both_dots['count_app'] == both_dots['count_ref']).sum()):,}",'cluster size, the same at every pair': f"{int((both_dots['cluster'] == both_dots['c_size']).sum()):,}",f'λ, max |difference| ({len(lam_ref):,}, four to a loop)': f'{lc_lam:.1e}',f'q ≥ 1e-4, max |difference| ({int(fixed.sum()):,})': f'{lc_q:.1e}',f'q < 1e-4, max relative difference ({int(small.sum()):,})': f'{lc_q_small:.1e}','q of 0 in cooltools, 0 in chromosomix': f"{int((q_app[q_ref ==0] ==0).sum())} of {lc_zero}",'chromosomix, as it noted': ' '.join(NOTES['loops'][2:]),})
158,920,381 pixels scored over 21 sequences; 998 reached all four thresholds at an FDR of 0.1. 787 clusters at a 20 kb radius; 236 passed the enrichment filter.
Code
TEXT['lc_n'] =f'{len(both_dots):,}'TEXT['lc_only'] =f"{int((dots['_merge'] !='both').sum())}"TEXT['lc_lam'] =f'{lc_lam:.1e}'TEXT['lc_q'] =f'{lc_q:.1e}'TEXT['lc_q_small'] =f'{lc_q_small:.1e}'fig = plt.figure(figsize=(11, 7.6), layout='constrained')grid = fig.add_gridspec(2, 2, height_ratios=[1, 1.25])ax = fig.add_subplot(grid[0, 0])for k, colour inzip(KERNELS, (APP_COLOUR, REF, INK2, '#7c56c2')): ax.scatter(both_dots[f'la_exp.{k}.value'], both_dots[k], s=7, color=colour, alpha=0.6, lw=0, label=KERNEL_NAMES[k])identity(ax, lam_ref.min(), lam_ref.max())ax.set(xscale='log', yscale='log', xlabel='cooltools, λ (raw counts)', ylabel='chromosomix, λ (raw counts)', title='a locally adjusted expected')ax.legend(loc='upper left', markerscale=2)ax = fig.add_subplot(grid[0, 1])seen = q_ref >0ax.scatter(-np.log10(q_ref[seen]), -np.log10(q_app[seen]), s=7, color=APP_COLOUR, alpha=0.6, lw=0)identity(ax, -np.log10(q_ref[seen].max()), -np.log10(q_ref[seen].min()))ax.set(xscale='log', yscale='log', xlabel='cooltools, −log10 q', ylabel='chromosomix, −log10 q', title='b q-values, four kernels')# The 2 Mb that hold the most loops both call, from one of them.LC_SPAN =2_000_000busiest =max(((chrom, start) for chrom, start in both_dots[['sequence', 'start1']].itertuples(index=False)), key=lambda at: ((both_dots['sequence'] == at[0]) & (both_dots['start1'] >= at[1])& (both_dots['start2'] < at[1] + LC_SPAN)).sum())chrom, first = busiest# Drawn with 100 kb either side, so that no loop sits on the frame.lo, hi =max(0, first -100_000), min(int(clr10.chromsizes[chrom]), first + LC_SPAN +100_000)held = clr10.matrix(balance=True).fetch(f'{chrom}:{lo}-{hi}')ax = fig.add_subplot(grid[1, :])ax.imshow(np.where(held >0, held, np.nan), cmap=DENSITY, norm=LogNorm(), interpolation='nearest', extent=(lo /1e6, hi /1e6, hi /1e6, lo /1e6))ax.grid(False)inside =lambda frame: frame[(frame['sequence'] == chrom) & (frame['start1'] >= lo) & (frame['start2'] < hi)]mine, theirs = inside(app_dots), inside(ref_dots)ax.scatter((mine['start2'] +5_000) /1e6, (mine['start1'] +5_000) /1e6, s=90, facecolors='none', edgecolors=APP_COLOUR, lw=1.4, label='chromosomix')ax.scatter((theirs['start2'] +5_000) /1e6, (theirs['start1'] +5_000) /1e6, s=18, marker='x', color=REF, lw=1.2, label='cooltools')ax.set(xlabel=f'{chrom} (Mb)', ylabel=f'{chrom} (Mb)', title=f'c {chrom}, {lo /1e6:.2f}–{hi /1e6:.2f} Mb')ax.set_aspect('equal')ax.legend(loc='lower left')drawn('fig-loopcalling')
Figure 20: Loop calling, mouse Hi-C at 10 kb. a, the four locally adjusted expecteds of every loop both call; b, their q-values; c, the 2 Mb stretch that holds the most of them, with 100 kb either side, both call sets over the balanced map.
236 loops at the same bin pairs (0 in one only), with the same contacts and the same cluster sizes; the locally adjusted expecteds within 5.0e-05 and the q-values within 5.0e-05 — and, below 10⁻⁴, within 3.2e-06 of their own size — which is the rounding of the digits chromosomix writes.
Code
record('Loop calling', 'mouse Hi-C, 10 kb, every chromosome', f"cooltools {VERSIONS['cooltools']}",f"{TEXT['lc_n']} loops, the same bin pairs, contacts and clusters; λ within {TEXT['lc_lam']}, q within "f"{TEXT['lc_q']}, the digits written",'Identical (to the digits written)'if LC_EXACT else'Differs', apps=['loops'], refs=['cooltools-dots'])
20 · Cis and trans per sequence (cooltools)
Per sequence, its contacts with both ends on it (cis) and with one end elsewhere (trans), the cis fraction and the ratio of the two; between every pair of sequences, their contacts, and those over what indifference predicts from the two sequences’ own trans contacts, T_a·T_b/2T, scaled so that the expectation sums to the trans contacts there are (§18.8). cooltools has no such task, but its two summaries hold every one of those sums: expected_cis, added up over every diagonal of a sequence, the main one included, is the sequence’s cis contacts, and expected_trans the contacts between each pair; the ratio to indifference is NumPy’s, over cooltools’ sums. The first of § 7’s chicken maps at 100 kb, its 36 sequences running from the macrochromosomes to microchromosomes a few bins long, raw and balanced. The table per sequence is written to whole contacts and three decimals and the square between sequences to six digits, so the contacts between two sequences are compared in the square, drawn with --normalise raw, those within one in the table, and the ratio to indifference in the square the task draws by default.
chromosomix
cooltools
within a sequence
cistrans, its table
expected_cis(ignore_diags=0), every diagonal of the sequence added up
between two
its square, --normalise raw
expected_trans
counts
raw (--no-balance) and balanced (the default)
clr_weight_name=None, and 'weight'
against indifference
its square, --normalise enrichment (the default)
NumPy over expected_trans’s sums
Code
CT = WORK /'cistrans'CHICKEN_100K = URIS[LABELS[0]]clr_ch = cooler.Cooler(f'{CHICKEN[0]}::/resolutions/{RES}')SEQUENCES = clr_ch.chromnames#: name: (label, chromosomix's arguments).CT_RUNS = {'raw': ('raw counts', ['--no-balance', '--normalise', 'raw']),'balanced': ('balanced', ['--normalise', 'raw']),'indifference': ('balanced, against indifference', [])}for key, (label, extra) in CT_RUNS.items(): mr('cistrans', CHICKEN_100K, *extra, out=CT / key, name=f'cistrans-{key}', analysis='Cis and trans per sequence', label=label, bench=False)for kind, weight in (('raw', []), ('balanced', ['weight'])): reference(f'cooltools-cistrans-{kind}', 'Cis and trans per sequence', 'cooltools', [PYTHON, script('cooltools_cistrans'), CHICKEN_100K, OUT, THREADS, *weight], out=CT /f'cooltools-{kind}.tsv', label=kind, threads=THREADS, bench=False)def sums(kind):"""cooltools' sums as one square, the sequences in the file's order: cis on the diagonal, the pairs off it.""" at = {name: i for i, name inenumerate(SEQUENCES)} held = np.zeros((len(SEQUENCES), len(SEQUENCES)))for a, b, contacts in pd.read_csv(CT /f'cooltools-{kind}.tsv', sep='\t').itertuples(index=False): held[at[a], at[b]] = held[at[b], at[a]] =0if np.isnan(contacts) else contactsreturn helddef against_indifference(held):"""Each pair's contacts over `T_a·T_b/2T`, scaled to sum to the trans contacts there are (§18.8).""" between = held - np.diag(np.diag(held)) trans = between.sum(axis=1) total = trans.sum() /2 expected =4* total **2/ (4* total **2- (trans **2).sum()) * np.outer(trans, trans) / (2* total)with np.errstate(invalid='ignore', divide='ignore'): ratio = np.where(expected >0, between / expected, np.nan) np.fill_diagonal(ratio, np.nan)return ratiodef drawn_square(key):"""The square chromosomix wrote, in the file's order of sequences.""" frame = table(one(CT / key, '*.cistrans-matrix.tsv'))return frame.set_index(frame.columns[0]).loc[SEQUENCES, SEQUENCES].to_numpy(float)BINS_CH = np.array([np.diff(clr_ch.extent(name))[0] for name in SEQUENCES])OFF =~np.eye(len(SEQUENCES), dtype=bool)ct = {}for kind in ('raw', 'balanced'): held = sums(kind) cis = np.diag(held).copy() trans = (held - np.diag(cis)).sum(axis=1) rows = table(one(CT / kind, '*.cistrans.tsv')).set_index('sequence').loc[SEQUENCES]with np.errstate(invalid='ignore', divide='ignore'): shares = {'fraction': np.where(cis + trans >0, cis / (cis + trans), np.nan),'ratio': np.where(cis >0, trans / cis, np.nan), 'perBin': (cis + trans) / BINS_CH} pairs = drawn_square(kind) ct[kind] =dict(held=held, cis=cis, trans=trans, rows=rows, pairs=pairs, fraction=shares['fraction'], bins=bool((rows['bins'].to_numpy() == BINS_CH).all()), sums=max(np.abs(rows[column].to_numpy(float) - value).max()for column, value in (('cis', cis), ('trans', trans), ('total', cis + trans))), shares={column: np.nanmax(np.abs(rows[column].to_numpy(float) - value))for column, value in shares.items()}, shares_gaps=all((np.isfinite(rows[column].to_numpy(float)) == np.isfinite(value)).all()for column, value in shares.items()), between=relative_difference(pairs[OFF], held[OFF]).max())oe_app, oe_ref = drawn_square('indifference'), against_indifference(ct['balanced']['held'])oe_both = np.isfinite(oe_app) & np.isfinite(oe_ref)oe_rel = relative_difference(oe_app[oe_both], oe_ref[oe_both]).max()OE_GAPS =bool((np.isfinite(oe_app) == np.isfinite(oe_ref)).all())raw_, bal_ = ct['raw'], ct['balanced']measures({'sequences; their bins as the file has them': f"{len(SEQUENCES)}; {'yes'if raw_['bins'] and bal_['bins'] else'no'}",'raw: cis, trans and total of every sequence, max |difference|': f"{raw_['sums']:g}",'raw: contacts between two sequences, max relative difference': f"{raw_['between']:.1e}",'balanced: cis, trans and total, max |difference| (whole contacts written)': f"{bal_['sums']:.2f}",'balanced: contacts between two sequences, max relative difference': f"{bal_['between']:.1e}",'cis fraction, trans / cis, contacts per bin: max |difference| (3, 3 and 2 decimals written)':', '.join(f"{max(raw_['shares'][c], bal_['shares'][c]):.1e}"for c in ('fraction', 'ratio', 'perBin')),'against indifference, max relative difference; the same pairs without a value':f"{oe_rel:.1e}; {'yes'if OE_GAPS else'no'}",})
raw: cis, trans and total of every sequence, max |difference|
0
raw: contacts between two sequences, max relative difference
0.0e+00
balanced: cis, trans and total, max |difference| (whole contacts written)
0.50
balanced: contacts between two sequences, max relative difference
4.6e-06
cis fraction, trans / cis, contacts per bin: max |difference| (3, 3 and 2 decimals written)
5.0e-04, 5.0e-04, 5.0e-03
against indifference, max relative difference; the same pairs without a value
4.6e-06; yes
Code
TEXT['ct_n'] =f'{len(SEQUENCES)}'TEXT['ct_pairs'] =f'{len(SEQUENCES) * (len(SEQUENCES) -1) //2:,}'TEXT['ct_raw'] ='the same'if raw_['sums'] ==0and raw_['between'] ==0else'not the same'TEXT['ct_bal_sums'] =f"{bal_['sums']:.2f}"TEXT['ct_bal_pairs'] =f"{bal_['between']:.1e}"TEXT['ct_oe'] =f'{oe_rel:.1e}'TEXT['ct_oe_gaps'] = ('the same pairs left without one'if OE_GAPS else'different pairs left without one')CT_EXACT = (raw_['sums'] ==0and raw_['between'] ==0and bal_['sums'] <= rounded(0) and bal_['between'] <6e-6andall(ct[k]['shares']['fraction'] <= rounded(3) and ct[k]['shares']['ratio'] <= rounded(3)and ct[k]['shares']['perBin'] <= rounded(2) and ct[k]['shares_gaps'] and ct[k]['bins'] for k in ct)and oe_rel <6e-6and OE_GAPS)#: A sequence's name on an axis: `chr` dropped, and an unplaced scaffold's long tail.SHORT_SEQ = [name.removeprefix('chr').split('_')[0] + ('…'if'_'in name else'') for name in SEQUENCES]# The microchromosomes, chr16 on, and the five largest, against indifference.numbered = {int(name[3:]): i for i, name inenumerate(SEQUENCES) if name[3:].isdigit()}MACRO, MICRO = [numbered[k] for k inrange(1, 6)], [i for k, i in numbered.items() if k >=16]TEXT['ct_micro'] =f'{np.nanmedian(oe_app[np.ix_(MICRO, MICRO)]):.2f}'TEXT['ct_macro_micro'] =f'{np.nanmedian(oe_app[np.ix_(MACRO, MICRO)]):.2f}'fig, axes = plt.subplots(1, 3, figsize=(12, 4.3), layout='constrained', width_ratios=[1.25, 1, 1.15])ax = axes[0]x = np.arange(len(SEQUENCES))ax.bar(x, bal_['rows']['fraction'].to_numpy(float), width=0.75, color=APP_COLOUR, label='chromosomix')ax.scatter(x, bal_['fraction'], s=12, color=REF, zorder=3, label='cooltools')ax.set_xticks(x, SHORT_SEQ, rotation=90, fontsize=6)ax.set(ylim=(0, 1), xlabel='sequence', ylabel='cis fraction, balanced', title='a cis fraction')ax.legend(loc='upper left')ax = axes[1]upper = np.triu(OFF)for kind, colour in (('raw', APP_COLOUR), ('balanced', '#1baf7a')): v = ct[kind] ref_all = np.r_[v['cis'], v['trans'], v['held'][upper]] app_all = np.r_[v['rows']['cis'].to_numpy(float), v['rows']['trans'].to_numpy(float), v['pairs'][upper]] seen = (ref_all >0) & (app_all >0) ax.scatter(ref_all[seen], app_all[seen], s=8, color=colour, alpha=0.7, lw=0, label=kind)every_sum = np.concatenate([np.r_[ct[k]['cis'], ct[k]['trans'], ct[k]['held'][upper]] for k in ct])identity(ax, every_sum[every_sum >0].min(), every_sum.max())ax.set(xscale='log', yscale='log', xlabel='cooltools', ylabel='chromosomix', title='b every sum')ax.legend(loc='upper left')ax = axes[2]diverging = plt.get_cmap('RdBu_r').copy()diverging.set_bad(MISSING)bar = square(ax, oe_app, 'c against indifference', cmap=diverging, norm=LogNorm(1/4, 4), label='observed / expected')bar.set_ticks([1/4, 1/2, 1, 2, 4], labels=['¼', '½', '1', '2', '4'])bar.minorticks_off()ax.set_xticks(x, SHORT_SEQ, rotation=90, fontsize=5)ax.set_yticks(x, SHORT_SEQ, fontsize=5)drawn('fig-cistrans')
Figure 21: Cis and trans per sequence, chicken Hi-C at 100 kb, the sequences in the file’s order. a, the cis fraction of every sequence, balanced; b, every sum, raw and balanced — a sequence’s cis and trans contacts and the contacts between every pair — against cooltools’; c, every pair against indifference, as chromosomix draws it.
Raw, the two tools count the same contacts on each of the 36 sequences and between each of their 630 pairs. Balanced, the table’s sums are within 0.50 of cooltools’, the whole contacts it writes, and the pairs within 4.6e-06, the six digits the square is written to; every cis fraction and ratio is within its three decimals; and against indifference every pair is within 4.6e-06 of NumPy’s over cooltools’ sums, with the same pairs left without one. The square is the karyotype’s: the microchromosomes, chr16 on, meet one another at a median of 1.92 times what indifference predicts, and the five largest chromosomes meet them at 0.81.
Code
record('Cis and trans per sequence', f'chicken Hi-C, 100 kb, {len(SEQUENCES)} sequences, raw and balanced',f"cooltools {VERSIONS['cooltools']}",f"raw: {TEXT['ct_raw']} contacts within and between every sequence; balanced: the pairs within "f"{TEXT['ct_bal_pairs']}; against indifference within {TEXT['ct_oe']} of NumPy over cooltools' sums",'Identical (to the digits written)'if CT_EXACT else'Differs', apps=[f'cistrans-{key}'for key in CT_RUNS], refs=['cooltools-cistrans-raw', 'cooltools-cistrans-balanced'])
21 · Saddle (cooltools)
The saddle § 3’s compartment call draws for each chromosome: its bins in 50 groups at the quantiles of E1 over the chromosome, and the mean observed/expected of every pair of groups, the first two diagonals left out as the call leaves them out; and AA, BB and AB, the means of its corners over the fifth of the groups at each end (§18.8). cooltools’ saddle, over each chromosome alone, its expected expected_cis‘s unsmoothed balanced.avg, is handed chromosomix’s own E1 as its track, from the bigWig chromosomix writes, cut into its groups by cooltools’ digitize: what is compared is the saddle, the eigenvector being § 3’s to compare. AA, BB and AB are taken from cooltools’ saddle as chromosomix takes them from its own.
chromosomix
cooltools
groups
--quantiles 50 (the default), E1’s quantiles over the chromosome
digitize(n_bins=50, qrange=(0, 1)), over the chromosome (view_df)
SD = WORK /'saddle'SD.mkdir(parents=True, exist_ok=True)SADDLE_GROUPS =50# chromosomix's E1 from the bigWig it writes, a float32 a bin, where its table keeps six digits: two# bins six digits cannot tell apart may lie either side of a group's edge. As cooltools reads a track,# and under a name that changes with it, so that the reference runs again when the eigenvector does.handle = pyBigWig.open(str(one(CMP /'app', '*.compartment-e1.bw')))e1_held = {(chrom, start): valuefor chrom in handle.chroms() for start, _, value in handle.intervals(chrom) or ()}handle.close()e1_bins = clr100.bins()[['chrom', 'start', 'end']][:]e1_track = e1_bins.assign(E1=[e1_held.get(at, np.nan) for at inzip(e1_bins['chrom'], e1_bins['start'])]).to_csv( sep='\t', index=False)E1_TRACK = SD /f"e1-{hashlib.sha256(e1_track.encode()).hexdigest()[:12]}.tsv"ifnot E1_TRACK.exists(): E1_TRACK.write_text(e1_track)reference('cooltools-saddle', 'Saddle', 'cooltools', [PYTHON, script('cooltools_saddle'), MOUSE_100K, E1_TRACK, SADDLE_GROUPS, OUT, ','.join(COMPARED)], out=SD /'cooltools.npz', bench=False)saddles_ref = np.load(SD /'cooltools.npz')CORNER =max(1, round(SADDLE_GROUPS /5))def corners(cells):"""AA, BB and AB as chromosomix takes them: each corner's mean over the cells that have a value.""" k = CORNERreturn {'AA': np.nanmean(cells[-k:, -k:]), 'BB': np.nanmean(cells[:k, :k]), 'AB': np.nanmean(cells[:k, -k:])}sd, rows = {}, []for chrom in COMPARED: path = one(CMP /'app', f'*.compartment-saddle-{chrom}.tsv') frame = table(path) app = frame.set_index(frame.columns[0]).to_numpy(float) stated = {k: float(v) for k, v in re.findall(r'^# (AA|BB|AB): (\S+)$', path.read_text(), flags=re.M)} ref = saddles_ref[chrom] both = np.isfinite(app) & np.isfinite(ref) expected_corners = corners(ref) sd[chrom] =dict(app=app, stated=stated, corners=expected_corners, rel=relative_difference(app, ref)[both]) rows.append({'chromosome': chrom, 'the same cells without a value': bool((np.isfinite(app) == np.isfinite(ref)).all()),'max relative difference': sd[chrom]['rel'].max(),**{f'{k}, chromosomix / cooltools': f"{stated[k]:.2f} / {v:.4f}"for k, v in expected_corners.items()}})saddle_table = pd.DataFrame(rows)show(saddle_table, **{'max relative difference': '{:.1e}'})
TEXT['sd_n'] =f'{len(COMPARED)}'TEXT['sd_rel'] =f"{saddle_table['max relative difference'].max():.1e}"corner_gap =max(abs(v['stated'][k] - v['corners'][k]) for v in sd.values() for k in ('AA', 'BB', 'AB'))TEXT['sd_corners'] =f'{corner_gap:.1e}'SADDLE_GAPS = saddle_table['the same cells without a value'].all()TEXT['sd_gaps'] ='the same cells'if SADDLE_GAPS else'not the same cells'SADDLE_EXACT = SADDLE_GAPS and saddle_table['max relative difference'].max() <6e-6and corner_gap <= rounded(2)fig, axes = plt.subplots(1, 3, figsize=(12, 3.9), layout='constrained', width_ratios=[1, 1.1, 1])diverging = plt.get_cmap('RdBu_r').copy()diverging.set_bad(MISSING)bar = square(axes[0], sd['chr2']['app'], 'a chr2', cmap=diverging, norm=LogNorm(1/2, 2), label='observed / expected')bar.set_ticks([1/2, 1, 2], labels=['½', '1', '2'])bar.minorticks_off()axes[0].set(xlabel='E1, B to A', ylabel='E1, B to A')ax = axes[1]ax.stairs(*differences(np.concatenate([v['rel'] for v in sd.values()])), color=APP_COLOUR, lw=1.6, label='every cell')ax.set(xscale='log', yscale='log', xlabel='|chromosomix / cooltools − 1|, 0 at 10⁻⁹', ylabel='cells', title='b every cell')ax = axes[2]for name, colour in (('AA', APP_COLOUR), ('BB', REF), ('AB', '#1baf7a')): ax.scatter([v['corners'][name] for v in sd.values()], [v['stated'][name] for v in sd.values()], s=18, color=colour, label=name, zorder=3)every_corner = [v['corners'][k] for v in sd.values() for k in ('AA', 'BB', 'AB')]identity(ax, min(every_corner), max(every_corner))ax.set(xlabel='cooltools', ylabel='chromosomix (two decimals)', title='c AA, BB and AB')ax.legend(loc='upper left')drawn('fig-saddle')
Figure 22: Saddles, mouse Hi-C at 100 kb, by chromosomix’s E1. a, chr2’s, as chromosomix draws it; b, every cell of every chromosome’s against cooltools’; c, the three corners of every chromosome.
Every cell of the 20 chromosomes’ saddles agrees with cooltools’ within 5.0e-06, the six digits chromosomix writes, with the same cells left without a value, and AA, BB and AB within 4.9e-03, their two decimals.
Code
record('Saddle', f"mouse Hi-C, 100 kb, § 3's E1, {len(COMPARED)} chromosomes", f"cooltools {VERSIONS['cooltools']}",f"every cell within {TEXT['sd_rel']}, the corners within {TEXT['sd_corners']}",'Identical (float32)'if SADDLE_EXACT else'Differs', apps=['compartments'], refs=['cooltools-saddle'])
22 · Fraction of signal in features (pyBigWig)
ChIP-seq’s FRiP, over a track: the signal inside a set of intervals over the whole of it, the signal being an area, value × bases, and the intervals merged first so that a base two of them share counts once; beside it, the share of the sequences’ bases the set covers, and the enrichment, their ratio (§18.8). The reference is pyBigWig’s exact sum over each interval, stats(..., type='sum', exact=True), and NumPy. § 5’s two ATAC-seq replicates on chr19, over two sets: § 5’s peaks, yapc’s at IDR ≤ 0.05, and § 15’s promoters, which overlap wherever two transcripts start alike — and every feature of each on its own (--perFeature), its signal and its mean over the bases with a value. chromosomix reads the features finely, at a fiftieth of their median width, and the whole track coarsely, at a two hundred-thousandth of the assembly, from a zoom level where the file has one that fine: away from a boundary, a sum weighted by span is the same area at any resolution, and a whole sequence has almost no boundaries.
chromosomix
pyBigWig and NumPy
features
merged where they overlap or touch
merged likewise, as bedtools merge merges them
signal
Σ value × bases, read at a fiftieth of the features’ median width
stats(..., type='sum', exact=True)
the whole track
read at a two hundred-thousandth of the assembly
the same exact sum, over every sequence the header declares
each feature
--perFeature: its signal, and its mean over the bases with a value
type='sum', type='mean', exact
Code
FR = WORK /'frip'#: name: (label, the feature set).FRIP_SETS = {'peaks': ('ATAC-seq peaks', PEAKS_BED), 'promoters': ('promoters', PROMOTERS)}for key, (label, bed) in FRIP_SETS.items(): mr('frip', '--tracks', ','.join(rel(path) for path in REPS), '--features', bed, '--perFeature', out=FR / key, name=f'frip-{key}', analysis='Fraction of signal in features', label=label, bench=False) reference(f'pybigwig-frip-{key}', 'Fraction of signal in features', 'pyBigWig, NumPy', [PYTHON, script('pybigwig_frip'), bed, OUT, *REPS], out=FR /f'pybigwig-{key}', made=FR /f'pybigwig-{key}-frip.tsv', label=label, bench=False)#: The columns of a FRiP chromosomix writes, and the places it writes each to.FRIP_PLACES = {'frip': 4, 'covered': 4, 'enrichment': 2, 'inFeature': 3}KEYS = ['track', 'contig', 'start', 'end']REPLICATE = {track_name(path.name): f'rep. {k +1}'for k, path inenumerate(REPS)}fr, rows = {}, []for key, (label, bed) in FRIP_SETS.items(): app = table(one(FR / key, '*.frip.tsv')).set_index('track') ref = (pd.read_csv(FR /f'pybigwig-{key}-frip.tsv', sep='\t').assign(track=lambda f: f['track'].map(track_name)) .set_index('track').loc[app.index]) total = relative_difference(app['totalSignal'].to_numpy(float), ref['totalSignal'].to_numpy(float))# Every feature by its track and coordinates; one written twice is the same numbers twice. each = (table(one(FR / key, '*.frip-features.tsv')).drop_duplicates(KEYS) .merge(pd.read_csv(FR /f'pybigwig-{key}-features.tsv', sep='\t') .assign(track=lambda f: f['track'].map(track_name)).drop_duplicates(KEYS), on=KEYS, how='outer', suffixes=('_app', '_ref'), indicator=True)) paired = each[each['_merge'] =='both'] signal_gap = np.abs(paired['signal_app'] - paired['signal_ref']) mean_gap = np.abs(paired['mean_app'] - paired['mean_ref']) gaps = {column: np.abs(app[column].to_numpy(float) - ref[column].to_numpy(float)) for column in FRIP_PLACES} fr[key] =dict(app=app, ref=ref, paired=paired, unpaired=int((each['_merge'] !='both').sum()), inside=np.abs(app['signal'] - ref['signal']).max(), total=total.max(), places={column: gap.max() for column, gap in gaps.items()},# Each within the places it is written to, once the whole track's own difference, which# moves a FRiP and an enrichment in proportion, is allowed for. within=all((gap <= rounded(FRIP_PLACES[c]) + np.abs(ref[c].to_numpy(float)) * total).all()for c, gap in gaps.items()),# Two decimals and four written, beside the float32 a bigWig holds its values in. features=bool((signal_gap <= rounded(2) +1e-6* paired['signal_ref'].abs()).all()and (mean_gap <= rounded(4) +1e-6* paired['mean_ref'].abs()).all()and (np.isnan(paired['mean_app']) == np.isnan(paired['mean_ref'])).all()), signal_gap=signal_gap.max(), mean_gap=mean_gap.max())for track in app.index: rows.append({'set': label, 'track': REPLICATE[track], 'FRiP, chromosomix / exact':f"{app.loc[track, 'frip']:.4f} / {ref.loc[track, 'frip']:.6f}",'covered': f"{app.loc[track, 'covered']:.4f} / {ref.loc[track, 'covered']:.6f}",'enrichment': f"{app.loc[track, 'enrichment']:.2f} / {ref.loc[track, 'enrichment']:.4f}",'signal inside, |difference|': abs(app.loc[track, 'signal'] - ref.loc[track, 'signal']),'signal in all, relative difference': relative_difference(app.loc[track, 'totalSignal'], ref.loc[track, 'totalSignal'])})display(show(pd.DataFrame(rows),**{'signal inside, |difference|': '{:.2f}', 'signal in all, relative difference': '{:.1e}'}))measures({f'{FRIP_SETS[k][0]}: features; unpaired; signal, mean, max |difference| (2 and 4 decimals written)':f"{len(v['paired']) //len(REPS):,}; {v['unpaired']}; {v['signal_gap']:.4f}, {v['mean_gap']:.1e}"for k, v in fr.items()})
ATAC-seq peaks: features; unpaired; signal, mean, max |difference| (2 and 4 decimals written)
5,666; 0; 0.0050, 5.0e-05
promoters: features; unpaired; signal, mean, max |difference| (2 and 4 decimals written)
887; 0; 0.0050, 5.0e-05
Code
TEXT['fr_features'] =f"{sum(len(v['paired']) for v in fr.values()) //len(REPS):,}"TEXT['fr_inside'] =f"{max(v['inside'] for v in fr.values()):.2f}"TEXT['fr_total'] =f"{max(v['total'] for v in fr.values()):.1e}"TEXT['fr_frip'] =f"{max(v['places']['frip'] for v in fr.values()):.1e}"FRIP_EXACT =all(v['inside'] <= rounded(0) and v['features'] and v['unpaired'] ==0and v['within']for v in fr.values())TEXT['fr_places'] ='every one'if FRIP_EXACT else'not every one'fig, axes = plt.subplots(1, 3, figsize=(12, 3.9), layout='constrained', width_ratios=[1, 1, 1.05])every = pd.concat([v['paired'] for v in fr.values()])agreement(axes[0], every['signal_ref'].to_numpy(float), every['signal_app'].to_numpy(float), 'pyBigWig, exact','chromosomix', 'a every feature’s signal', scale='log')agreement(axes[1], every['mean_ref'].to_numpy(float), every['mean_app'].to_numpy(float), 'pyBigWig, exact','chromosomix', 'b every feature’s mean', scale='log')ax = axes[2]at =0ticks, names = [], []for key, (label, _) in FRIP_SETS.items(): v = fr[key]for track in v['app'].index: ax.bar(at, v['app'].loc[track, 'frip'], width=0.7, color=APP_COLOUR, label='chromosomix'if at ==0elseNone, gid='chromosomix') ax.scatter(at, v['ref'].loc[track, 'frip'], s=26, color=REF, zorder=3, label='pyBigWig, exact'if at ==0elseNone, gid='pyBigWig, exact') ticks.append(at) names.append(f"{key},\n{REPLICATE[track]}") at +=1 ax.axhline(v['app']['covered'].iloc[0], xmin=(at -2) /4, xmax=at /4, color=INK2, lw=1, ls=(0, (4, 3)), gid=f'{label}: the share of chr19 covered')ax.set_xticks(ticks, names, fontsize=8)ax.set(xlim=(-0.5, at -0.5), ylabel='FRiP', title='c each set, each track; dashed, chr19 covered')ax.legend(loc='upper right')drawn('fig-frip')
Figure 23: FRiP of two mouse ATAC-seq replicates on chr19. a, every feature’s signal, both sets and both tracks, against pyBigWig’s exact sum; b, every feature’s mean over the bases with a value; c, the FRiP of each track over each set, and the share of chr19 the set covers.
Every one of the 6,553 features has its signal and its mean on both tracks within the rounding of the two and four decimals chromosomix writes them to, and inside the merged features the signal is pyBigWig’s exact sum within 0.34, the rounding of the whole number it is written as. The whole track, read from a zoom level, is within 4.8e-05 of its exact sum, which moves a FRiP by less than the last of its four decimals: every one of the FRiPs, the shares covered and the enrichments is within the rounding of the places it is written to, the FRiP within 4.5e-05.
Code
record('Fraction of signal in features', 'mouse ATAC-seq, 2 replicates, chr19; § 5’s peaks and § 15’s promoters',f"pyBigWig {VERSIONS['pyBigWig']}, NumPy {VERSIONS['numpy']}",f"every feature's signal and mean to the digits written; inside, the exact sum; the whole track, from a "f"zoom level, within {TEXT['fr_total']}; FRiP within {TEXT['fr_frip']}",'The same FRiP to its four decimals and every feature to the digits written, the whole track read from a ''zoom level'if FRIP_EXACT else'Differs', apps=[f'frip-{key}'for key in FRIP_SETS], refs=[f'pybigwig-frip-{key}'for key in FRIP_SETS])
23 · Signal by group (pyBigWig and SciPy)
A track compared between groups of the genome (§18.8): the groups are a feature set’s names and the stretch of the genome it does not cover; by default each group is one value per entry — each run of a name, merged where its features overlap or touch, and each stretch of the rest between two — and with a bin size each run is cut on the bin grid into stretches of at most a bin; a value is the track’s exact mean over the bases it has one for; and the groups are summarised and tested by ranks — Mann–Whitney pair by pair and Kruskal–Wallis over all of them, every P adjusted with every other (Benjamini–Hochberg). The groups are § 3’s compartment call on chr19, A and B, and the rest of chr19, which the call leaves out; the tracks are § 5’s two ATAC-seq replicates; and the run is made both ways, by entry and in 10 kb stretches. The reference cuts the groups by the same rule, reads each one’s mean with pyBigWig, and summarises and tests them with NumPy and SciPy. chromosomix writes every value, a group’s ascending, as the violin it draws, and those are compared with the reference’s rank for rank in each group; its summaries and tests are compared with NumPy’s and SciPy’s, each to the places it is written to.
chromosomix
pyBigWig, NumPy and SciPy
groups
--features, § 3’s call, --split name; the rest of chr19 (--outside, the default)
the BED’s names; what it does not cover
what is compared
one value per entry (the default); stretches of at most a bin, --resolution 10000
the same, a bin size of 0 cutting nothing
a value
its mean, at the track’s own resolution
stats(..., type='mean', exact=True)
summaries
n, mean, SD, median, quartiles, the mean less the others’
GS = WORK /'groupsignal'GS.mkdir(parents=True, exist_ok=True)GS_BIN =10_000# § 3's compartment call, under a name that changes with it, so that the reference runs again when it does.ab_call = one(CMP /'app', '*.compartment-ab.bed').read_text()AB_CALL = GS /f"ab-{hashlib.sha256(ab_call.encode()).hexdigest()[:12]}.bed"ifnot AB_CALL.exists(): AB_CALL.write_text(ab_call)#: The two ways of making a group's values: the default, one value per entry, timed against its reference;#: and stretches of at most 10 kb. Each: its folder's suffix, the flag, the bin size the reference is given.GS_CASES = {'one value per entry': ('-entries', [], 0), '10 kb stretches': ('', ['--resolution', GS_BIN], GS_BIN)}for label, (suffix, flag, binsize) in GS_CASES.items(): mr('groupsignal', '--tracks', ','.join(rel(path) for path in REPS), '--features', AB_CALL, '--split', 'name',*flag, out=GS /f'app{suffix}', name=f'groupsignal{suffix}', analysis='Signal by group', label=label, bench=binsize ==0) reference(f'pybigwig-groupsignal{suffix}', 'Signal by group', 'pyBigWig, SciPy', [PYTHON, script('pybigwig_groupsignal'), AB_CALL, binsize, OUT, *REPS], out=GS /f'reference{suffix}', made=GS /f'reference{suffix}-tests.tsv', label=label, bench=binsize ==0)def violin_values(path):"""A violin as chromosomix writes it: its track, and every value a group, ascending.""" frame = table(path) track = re.search(r'^# output: (.+?) by ', path.read_text(), flags=re.M).group(1)return track, {name: frame[name].dropna().to_numpy(float) for name in frame.columns[1:]}def significant(digits):"""How far rounding to `digits` significant digits moves a number, relative to it, and a hair over."""return0.5*10.0** (1- digits) * (1+1e-6)#: Each column chromosomix writes and how near the reference has to be: the places it is written to; P and#: q, written in scientific notation below 10⁻⁴, to their six significant digits there and to their four#: places above it.GROUP_COLUMNS = {'mean': rounded(4), 'sd': rounded(4), 'median': rounded(4), 'q1': rounded(4), 'q3': rounded(4),'vsRest': rounded(4)}TEST_COLUMNS = {'difference': rounded(4), 'effect': rounded(3), 'statistic': rounded(2), 'p': significant(6),'q': significant(6)}def against(app, other, key, columns):"""Each column's largest difference — P and q relative where they are written with an exponent, the others as written — and whether every one, P and q above 10⁻⁴ too, is near enough.""" paired = app.merge(other, on=key, suffixes=('', '_other')) out = {}for column, near in columns.items(): mine, theirs = paired[column].to_numpy(float), paired[f'{column}_other'].to_numpy(float)if column in ('p', 'q'): exponent = np.abs(mine) <1e-4 gap = np.nan_to_num(np.where(exponent, relative_difference(mine, theirs), np.abs(mine - theirs)), nan=0.0) ok = np.where(exponent, gap <= near, gap <= rounded(4)) out[column] = (np.max(gap[exponent], initial=0.0), bool(ok.all()))continue gap = np.nan_to_num(np.abs(mine - theirs), nan=0.0) out[column] = (np.max(gap), bool((gap <= near).all()))return out, len(paired)def compared(suffix):"""One way of making the groups, chromosomix's against the reference's: its values, rank for rank in each group, and every summary and test.""" groups_app = table(one(GS /f'app{suffix}', '*.groupsignal-groups.tsv')) tests_app = table(one(GS /f'app{suffix}', '*.groupsignal-tests.tsv')).fillna({'b': ''}) groups_ref = (pd.read_csv(GS /f'reference{suffix}-groups.tsv', sep='\t') .assign(track=lambda f: f['track'].map(track_name))) tests_ref = (pd.read_csv(GS /f'reference{suffix}-tests.tsv', sep='\t').fillna({'b': ''}) .assign(track=lambda f: f['track'].map(track_name))) exact = pd.read_csv(GS /f'reference{suffix}-stretches.tsv', sep='\t').rename(columns=track_name) values =dict(violin_values(path) for path insorted((GS /f'app{suffix}').glob('*.groupsignal-[0-9]*.tsv'))) held_groups, n_groups = against(groups_app, groups_ref, ['track', 'group'], GROUP_COLUMNS) held_tests, n_tests = against(tests_app, tests_ref, ['track', 'test', 'a', 'b'], TEST_COLUMNS) held = held_groups | held_testsdef exact_sorted(track, name):"""A group's exact means on one track, ascending, as chromosomix's violins write its own."""return np.sort(exact.loc[(exact['group'] == name) & exact[track].notna(), track].to_numpy()) mine = np.concatenate([v for by_group in values.values() for v in by_group.values()]) theirs = np.concatenate([exact_sorted(track, name) for track, by_group in values.items() for name in by_group]) same_counts =bool((groups_app.merge(groups_ref, on=['track', 'group'], suffixes=('', '_ref')) .eval('n == n_ref and bases == bases_ref')).all()) every =all(ok for _, ok in held.values()) and n_groups ==len(groups_app) and n_tests ==len(tests_app)returndict(n=len(exact), values=values, mine=mine, theirs=theirs, rank_gap=relative_difference(mine, theirs), held=held, same_counts=same_counts, every=every, names=list(dict.fromkeys(groups_app['group'])))GS_RESULTS = {label: compared(suffix) for label, (suffix, _, _) in GS_CASES.items()}display(measures({f'{label}: values; groups': f"{r['n']:,}; {', '.join(r['names'])}"for label, r in GS_RESULTS.items()} | {f'{label}: values and bases of each group, the same': 'yes'if r['same_counts'] else'no'for label, r in GS_RESULTS.items()} | {f'{label}: means, rank for rank: median, max relative difference':f"{np.median(r['rank_gap']):.1e}, {r['rank_gap'].max():.1e}"for label, r in GS_RESULTS.items()}))show(pd.DataFrame([{'column': column,'measured': 'relative below 10⁻⁴, else as written'if column in ('p', 'q') else'as written','near enough': (GROUP_COLUMNS | TEST_COLUMNS)[column],**{f'max difference, {label}': r['held'][column][0] for label, r in GS_RESULTS.items()},'near enough, every one': all(r['held'][column][1] for r in GS_RESULTS.values())}for column in [*GROUP_COLUMNS, *TEST_COLUMNS]]),**{c: '{:.1e}'for c in ('near enough', *[f'max difference, {label}'for label in GS_RESULTS])})
one value per entry: values and bases of each group, the same
yes
10 kb stretches: values and bases of each group, the same
yes
one value per entry: means, rank for rank: median, max relative difference
4.6e-07, 2.7e-06
10 kb stretches: means, rank for rank: median, max relative difference
5.5e-07, 5.0e-06
column
measured
near enough
max difference, one value per entry
max difference, 10 kb stretches
near enough, every one
mean
as written
5.0e-05
3.9e-05
4.2e-05
True
sd
as written
5.0e-05
4.9e-05
3.8e-05
True
median
as written
5.0e-05
5.0e-05
4.0e-05
True
q1
as written
5.0e-05
4.7e-05
2.8e-05
True
q3
as written
5.0e-05
4.2e-05
4.1e-05
True
vsRest
as written
5.0e-05
4.7e-05
4.0e-05
True
difference
as written
5.0e-05
4.8e-05
4.1e-05
True
effect
as written
5.0e-04
4.1e-04
3.4e-04
True
statistic
as written
5.0e-03
4.8e-03
3.3e-03
True
p
relative below 10⁻⁴, else as written
5.0e-06
1.7e-06
2.9e-06
True
q
relative below 10⁻⁴, else as written
5.0e-06
1.7e-06
2.8e-06
True
Code
ENTRIES, STRETCHES = GS_RESULTS['one value per entry'], GS_RESULTS['10 kb stretches']TEXT['gs_n'] =f"{STRETCHES['n']:,}"TEXT['gs_entries'] =f"{ENTRIES['n']:,}"TEXT['gs_rank'] =f"{max(r['rank_gap'].max() for r in GS_RESULTS.values()):.1e}"GS_HELD =all(r['every'] for r in GS_RESULTS.values())TEXT['gs_held'] ='every one'if GS_HELD else'not every one'GS_EXACT = GS_HELD andall(r['same_counts'] and r['rank_gap'].max() <= significant(6) for r in GS_RESULTS.values())fig, axes = plt.subplots(1, 3, figsize=(12, 3.9), layout='constrained', width_ratios=[1.05, 1, 1.25])ax = axes[0]first_track =next(iter(STRETCHES['values']))edges = np.linspace(0, np.percentile(np.concatenate(list(STRETCHES['values'][first_track].values())), 99.5), 41)for name, colour inzip(STRETCHES['names'], ('#1baf7a', '#7c56c2', INK2)): counts, _ = np.histogram(np.clip(STRETCHES['values'][first_track][name], 0, edges[-1]), edges) ax.stairs(counts / counts.sum(), edges, color=colour, lw=1.6, label=f'{name}, 10 kb')# Each entry's mean as a tick, a row a group, under the stretches' steps. entries = np.clip(ENTRIES['values'][first_track][name], 0, edges[-1]) ax.plot(entries, np.full(len(entries), -0.04* (1+ STRETCHES['names'].index(name))), '|', color=colour, ms=8, gid=f'{name}, by entry')top = ax.get_ylim()[1]ax.set_yticks(np.arange(0, top, 0.2))ax.set(xlabel=f'mean ATAC signal, {REPLICATE[first_track]}', ylabel='share of the group’s stretches', title='a each group; ticks, by entry', ylim=(-0.04* (len(STRETCHES['names']) +1), top))ax.legend(loc='upper right')agreement(axes[1], np.concatenate([r['theirs'] for r in GS_RESULTS.values()]), np.concatenate([r['mine'] for r in GS_RESULTS.values()]), 'pyBigWig', 'chromosomix','b every value, rank for rank')ax = axes[2]columns = [*GROUP_COLUMNS, *TEST_COLUMNS]for (label, r), marker inzip(GS_RESULTS.items(), ('o', 's')):for k, column inenumerate(columns): ax.scatter(k, max(r['held'][column][0], 1e-9), s=26, marker=marker, color=APP_COLOUR, zorder=3, facecolor=APP_COLOUR if marker =='o'else'none', label=label if k ==0elseNone, gid=label)for k, column inenumerate(columns): near = (GROUP_COLUMNS | TEST_COLUMNS)[column] ax.plot([k -0.35, k +0.35], [near, near], color=INK2, lw=1, gid='near enough')ax.set_xticks(range(len(columns)), ['mean', 'SD', 'median', 'Q1', 'Q3', 'mean − rest', 'median diff.', 'effect','U or H', 'P, relative', 'q, relative'], fontsize=7, rotation=45, ha='right')ax.set(yscale='log', ylim=(1e-9, 10), ylabel='max difference', title='c every summary and test; ticks, near enough')ax.legend(loc='upper left')drawn('fig-groupsignal')
Figure 24: ATAC-seq on chr19 by § 3’s compartments. a, each group’s means as chromosomix measures them, replicate 1: one value per entry (bars) and in stretches of at most 10 kb (steps); b, every value’s mean, rank for rank in its group, both replicates and both ways, against pyBigWig’s; c, every summary and test against NumPy’s and SciPy’s, both ways — P and q relative where written with an exponent, the others as they are written.
The two tools make the same groups both ways — chr19 as 80 entries by default and as 6,143 stretches of at most 10 kb — the same number in each group and of the same bases, and read the same means from them: rank for rank in each group, chromosomix’s are pyBigWig’s within 5.0e-06, the six digits it writes. NumPy and SciPy give every one of its summaries and tests to the places it writes them — U and H to two decimals, P and q to six significant digits where they are written with an exponent and to four places above 10⁻⁴.
Code
record('Signal by group', "mouse ATAC-seq, 2 replicates, chr19, by § 3's compartments; one value per entry, ""and 10 kb stretches",f"pyBigWig {VERSIONS['pyBigWig']}, NumPy {VERSIONS['numpy']}, SciPy {VERSIONS['scipy']}",f"the same entries and stretches, their means within {TEXT['gs_rank']}; {TEXT['gs_held']} of the ""summaries and tests as written",'Identical (to the digits written)'if GS_EXACT else'Differs', apps=['groupsignal-entries', 'groupsignal'], refs=['pybigwig-groupsignal-entries', 'pybigwig-groupsignal'])
24 · CTCF sites in GM12878 (Fig. 2)
Fig. 2 chains four tasks, and this section runs the chain twice, each link by chromosomix and by the package it reproduces, on the same inputs: in situ Hi-C of GM12878 (Rao et al. 2014; 4DN’s 4DNFIXP4QG5B, every replicate merged, 4,010,478,943 contacts) at 10 kb; CTCF ChIP-seq of two labs as the two replicates — ENCODE’s ENCSR000AKB (Bernstein lab, Broad) and ENCSR000DRZ (Stamatoyannopoulos lab, UW), each the fold change over control of its pooled replicates; and ATAC-seq (ENCSR095QNB). The map is 27 GB, and what both sides read of it is the 10-kb rung of chr19 — and for Fig. 2’s own pileup the 2-kb one — every contact within it with the file’s own weights, read by range from 4DN’s bucket into a .cool (notebooks/scripts/cool_subset.py, which writes a pixel the file stores twice once, as the sum both tools read it as); the tracks are cut to chr19 interval for interval, as § 5 cuts its replicates, and for § 5’s reason: yapc holds the second derivative of everything it is given in memory and writes it out, 700 MB a seed for two chromosomes. Each link is fed what the one before it wrote in chromosomix, so the chain is the app’s and every reference is held to one step of it.
Insulation at four scales: windows of 50, 100, 200 and 400 kb, against cooltools.insulation with the four, and the boundaries called on the narrowest, which is the window the app calls them on.
CTCF’s peaks: the two labs’ tracks as replicates, against yapc 0.1 with IDR 2.0.3 at § 5’s settings, five seeds.
APA between consecutive peaks: each peak of 2 paired with the next along its chromosome — by the app’s own consecutive features, and for coolpuppy by a BEDPE of the same pairs written here — piled up raw and over the expected, at coolpuppy’s defaults and with every pair, against coolpuppy with § 11’s script and settings; and Fig. 2’s own panel, on the 2-kb rung, ten bins either side from coolpuppy’s floor for that window, over the expected. There one peak’s midpoint sits half a base before a bin edge, and the two tools agree only if both round it down. And the TADs between the boundaries of 1, on the 2-kb rung, each with half its length either side, stretched evenly onto 51 cells a side over the expected, against coolpuppy’s rescaled local pileup.
The signal at CTCF’s sites: the metaplot of both CTCF tracks and ATAC-seq about each peak’s centre, ±2 kb in 50-bp bins, and of the insulation at its four scales, ±500 kb in 10-kb bins, against deepTools’ computeMatrix reference-point, plotProfile and plotHeatmap.
chromosomix
reference
insulation
insulation --window 5 --scales 4
cooltools.insulation(…, [50_000, 100_000, 200_000, 400_000]), § 1’s other settings
peaks
peaks --signal ENCFF734CUT --replicate ENCFF644EEX, § 5’s settings
yapc, § 5’s settings, seeds 0–4
pairs
apa --anchors consecutive --features the peaks: each and the next on its chromosome
a BEDPE of each peak and the next, written here
APA
apa --anchors consecutive --mode fixed --window 5 --minDiag 2 --minDist 12 (or 0), --norm none or oe
coolpup.pileup(…, flank=50_000), mindist'auto' (or 0), raw or over § 4’s expected for this map
APA, Fig. 2
the same at 2 kb: --window 10 --minDiag 2 --minDist 22 --norm oe
coolpup.pileup(…, flank=20_000), mindist'auto', over the 2-kb rung’s expected
TADs
apa --anchors features --features the domains --mode scaled --grid 51 --flankFraction 0.5 --minDiag 2 --norm oe, at 2 kb
coolpup.pileup(…, local=True, rescale=True, rescale_flank=0.5, rescale_size=51), over the 2-kb rung’s expected
metaplot
metaplot --at centre --stranded=false --band none, --flank 2000 --step 50 or --flank 500000 --step 10000
computeMatrix reference-point --referencePoint center, -b/-a 2000 --binSize 50 or -b/-a 500000 --binSize 10000
Code
GM = WORK /'gm12878'GM_ANALYSIS ='CTCF sites in GM12878 (Fig. 2)'GM_SEQUENCES = ['chr19']GM_MAP =f'{FOURDN}/d6abea45-b0bb-4154-9854-1d3075b98097/4DNFIXP4QG5B.mcool'GM_COOL = DATA /'4DNFIXP4QG5B_10000.chr19.cool'ifnot GM_COOL.exists(): subprocess.run([PYTHON, str(SCRIPTS /'cool_subset.py'), GM_MAP, '10000', str(GM_COOL), '--sequences',','.join(GM_SEQUENCES)], check=True)GM_10K = rel(GM_COOL)def cut(source, sequences=GM_SEQUENCES):"""The track with these sequences only, interval for interval: § 5's cut, for more than one.""" target = DATA /f"{source.stem}.{'-'.join(sequences)}.bw"if target.exists() andnot REDO:return target handle = pyBigWig.open(str(source)) out = pyBigWig.open(str(target), 'w') out.addHeader([(name, handle.chroms()[name]) for name in sequences])for name in sequences: entries = handle.intervals(name) or () out.addEntries([name] *len(entries), [int(a) for a, _, _ in entries], ends=[int(b) for _, b, _ in entries], values=[float(v) for _, _, v in entries]) out.close() handle.close()return targetCTCF = [cut(PATHS[a]) for a in ('ENCFF734CUT', 'ENCFF644EEX')]GM_ATAC = cut(PATHS['ENCFF487LOB'])clr_gm = cooler.Cooler(str(GM_COOL))gm_bins = clr_gm.bins()[:]measures({'map': f"4DNFIXP4QG5B, {', '.join(GM_SEQUENCES)} at 10 kb: {len(gm_bins):,} bins, "f"{clr_gm.info['nnz']:,} pixels, {clr_gm.info['sum']:,} contacts",'CTCF, the two replicates': ', '.join(rel(path) for path in CTCF), 'ATAC-seq': rel(GM_ATAC)})
50 kb: bins both score; only one; max |difference|
5,536; 0; 4.7e-06
100 kb: bins both score; only one; max |difference|
5,553; 0; 4.8e-06
200 kb: bins both score; only one; max |difference|
5,562; 0; 5.0e-06
400 kb: bins both score; only one; max |difference|
5,562; 0; 4.9e-06
boundaries on 50 kb, chromosomix / cooltools; at the same bin
306 / 306; 306
boundary strength, max |difference|
4.7e-06
Code
TEXT['gm_ins_n'] =f"{sum(gm_both[w].sum() for w in GM_WINDOWS):,}"TEXT['gm_ins_max'] =f'{max(d.max() for d in gm_diff.values()):.1e}'TEXT['gm_ins_one'] =f'{sum(gm_one.values())}'TEXT['gm_calls'] =f'{len(gm_same):,}'TEXT['gm_calls_only'] =f"{int((gm_calls['_merge'] !='both').sum())}"TEXT['gm_strength'] =f'{gm_strength.max():.1e}'fig = plt.figure(figsize=(12, 7.4), layout='constrained')grid = fig.add_gridspec(2, 4, height_ratios=[1, 1.05])for k, w inenumerate(GM_WINDOWS): a, b = gm_ins[w] agreement(fig.add_subplot(grid[0, k]), b[gm_both[w]], a[gm_both[w]], 'cooltools (log2)','chromosomix (log2)'if k ==0else'', f'{"abcd"[k]}{w *10} kb')ax = fig.add_subplot(grid[1, 0])ax.scatter(gm_same['boundary_strength_50000'], gm_same['score'], s=6, color=APP_COLOUR, alpha=0.4, lw=0)identity(ax, 0, gm_same['boundary_strength_50000'].max())ax.set(xlabel='cooltools boundary strength', ylabel='chromosomix prominence', title='e every boundary')ax = fig.add_subplot(grid[1, 1:])lo, _ = clr_gm.extent('chr19')stretch =slice(lo +3_000, lo +3_500)x = gm_bins['start'].to_numpy()[stretch] /1e6for k, w inenumerate(GM_WINDOWS): a, b = gm_ins[w] ax.plot(x, b[stretch] -1.5* k, color=REF, lw=3, label='cooltools'if k ==0elseNone, gid=f'cooltools, {w *10} kb') ax.plot(x, a[stretch] -1.5* k, color=APP_COLOUR, lw=1.2, label='chromosomix'if k ==0elseNone, gid=f'chromosomix, {w *10} kb')ax.set_yticks([-1.5* k for k inrange(len(GM_WINDOWS))], [f'{w *10} kb'for w in GM_WINDOWS])ax.set(xlabel='chr19 (Mb)', ylabel='insulation (log2), offset by window', title='f chr19, 30–35 Mb')ax.legend(loc='lower left', ncols=2)drawn('fig-gm-insulation')
Figure 25: Insulation of GM12878 at 10 kb, chr19. a–d, every bin at each window; e, every boundary both call on the 50-kb window; f, chr19 at 30–35 Mb, the four scales.
At the four windows, 22,213 scores in all differ by at most 5.0e-06, with 0 scored by one tool only; on the 50-kb window the same 306 boundaries are called, 0 by one only, with strengths within 4.7e-06.
CTCF’s peaks
Code
GY = GM /'yapc'mr('peaks', '--signal', CTCF[0], '--replicate', CTCF[1], '--perPeak', '--width', 150, '--times', 3, '--minWidth', 75,'--truncate', 100000, '--idr', 0.05, out=GY /'app', name='gm-peaks', analysis=GM_ANALYSIS, label='peaks', group='peaks')GM_SEEDS = {seed: GY /f'seed{seed}'for seed inrange(5)}for seed, folder in GM_SEEDS.items(): reference(f'gm-yapc-seed{seed}', GM_ANALYSIS, 'yapc 0.1', [str(YAPC_PYTHON), YAPC_SCRIPT, seed, OUT, 'cond', *CTCF, '--smoothing-window-width', 150,'--smoothing-times', 3, '--min-concave-region-width', 75, '--truncate-idr-input', 100000], out=folder /'yapc', made=folder /'yapc_0.05.bed', env=YAPC_ENV, label=f'peaks, seed {seed}', group='peaks', bench=seed ==0)gm_peaksall = pd.read_csv(GM_SEEDS[0] /'yapc_peaksall.tsv', sep='\t')gm_tested = pd.read_csv(GM_SEEDS[0] /'yapc.tsv', sep='\t')gm_listing = table(one(GY /'app', '*.peaks-list.tsv')).rename(columns={'sequence': 'chrom'})gm_counts = table(one(GY /'app', '*.peaks-idr.tsv'))gm_called = pd.read_csv(one(GY /'app', '*.bed'), sep='\t', header=None, comment='#', names=['chrom', 'start', 'end', 'name', 'score', 'strand'])# A peak's bounds as chromosomix lists them and as yapc does: § 5's `key` and `bounds`, named again,# since the loops of §§ 11, 20 and 22 take `key` for their own.gm_key, gm_bounds = ['chrom', 'start', 'end'], ['chrom', 'concave_start', 'concave_end']gm_m = gm_listing.merge(gm_tested.drop(columns=['start', 'end']), left_on=gm_key, right_on=gm_bounds, how='outer', indicator=True)gm_shared = gm_m[gm_m['_merge'] =='both']gm_curvature = np.abs(np.r_[gm_shared['curvature0'] - gm_shared['cond_rep1_score'], gm_shared['curvature1'] - gm_shared['cond_rep2_score']])def gm_bed(th, seed=0): out = pd.read_csv(GM_SEEDS[seed] /f'yapc_{th}.bed', sep='\t', header=None, comment='#').iloc[:, :3] out.columns = gm_keyreturn outgm_called_ref = gm_bed(0.05)gm_same_called = gm_called[gm_key].merge(gm_called_ref, on=gm_key)gm_idr = gm_called.merge(gm_tested, left_on=gm_key, right_on=gm_bounds)gm_runs = {seed: pd.read_csv(folder /'yapc.tsv', sep='\t').set_index(gm_bounds)['cond_globalIDR']for seed, folder in GM_SEEDS.items()}gm_app_idr = gm_called.set_index(gm_key)['score'].rename_axis(gm_bounds)gm_idr_app = [np.abs(gm_runs[seed].reindex(gm_app_idr.index) - gm_app_idr).max() for seed in GM_SEEDS]gm_idr_self = [np.abs(gm_runs[a].reindex(gm_app_idr.index) - gm_runs[b].reindex(gm_app_idr.index)).max()for a in GM_SEEDS for b in GM_SEEDS if a < b]gm_counts_agree =all(len(gm_bed(th, seed)) ==int(gm_counts.loc[np.isclose(gm_counts['threshold'], th), 'peaks'].iloc[0])for th in THRESHOLDS for seed in GM_SEEDS)measures({'given to IDR, chromosomix / yapc': f'{len(gm_listing):,} / {len(gm_tested):,}','same bounds; in one only': f"{len(gm_shared):,}; {int((gm_m['_merge'] !='both').sum())}",'same summit': f"{(gm_shared['summit'] == gm_shared['mode']).mean():.2%}",'curvature, max |difference| (two decimals written)': f'{gm_curvature.max():.4f}','peaks at IDR ≤ 0.05, chromosomix / yapc; same': f'{len(gm_called):,} / {len(gm_called_ref):,}; {len(gm_same_called):,}','−log10 IDR of those peaks, max |difference| to each yapc seed': f'{min(gm_idr_app):.1e} to {max(gm_idr_app):.1e}','−log10 IDR, yapc seeds against one another': f'{min(gm_idr_self):.1e} to {max(gm_idr_self):.1e}','peaks at each of yapc\'s six thresholds, against every seed': 'the same'if gm_counts_agree else'not the same',})
curvature, max |difference| (two decimals written)
0.0050
peaks at IDR ≤ 0.05, chromosomix / yapc; same
2,106 / 2,106; 2,106
−log10 IDR of those peaks, max |difference| to each yapc seed
5.5e-06 to 1.9e-05
−log10 IDR, yapc seeds against one another
3.6e-06 to 2.3e-05
peaks at each of yapc's six thresholds, against every seed
the same
APA between consecutive peaks
Code
GP = GM /'apa'GP.mkdir(parents=True, exist_ok=True)# The peaks chromosomix called, each with the next one along its chromosome: the app pairs them itself# (`--anchors consecutive`), and coolpuppy is handed the same pairs as a BEDPE written here.GM_PEAKS_BED = rel(one(GY /'app', '*.bed'))gm_sorted = gm_called.sort_values(['chrom', 'start', 'end']).reset_index(drop=True)gm_next = gm_sorted.shift(-1)gm_pairs = gm_sorted[gm_sorted['chrom'] == gm_next['chrom']].assign( chrom2=gm_next['chrom'], start2=gm_next['start'].astype('Int64'), end2=gm_next['end'].astype('Int64'))GM_PAIRS = GP /'consecutive.bedpe'gm_pairs[['chrom', 'start', 'end', 'chrom2', 'start2', 'end2']].to_csv(GM_PAIRS, sep='\t', header=False, index=False)gm_apart = ((gm_pairs['start2'].astype(int) - gm_pairs['start'].astype(int)) // clr_gm.binsize).to_numpy()reference('gm-expected', GM_ANALYSIS, 'cooltools', [PYTHON, EXPECTED, GM_10K, OUT, THREADS], out=GP /'expected.tsv', label='the expected, for coolpuppy', group='APA', threads=THREADS, bench=False)GM_CUTS = {'raw': ('raw', ['--norm', 'none', '--minDist', MIN_DIST], []),'oe': ('over the expected', ['--norm', 'oe', '--minDist', MIN_DIST], ['--expected', GP /'expected.tsv']),'raw-every': ('raw, every pair', ['--norm', 'none', '--minDist', 0], ['--mindist', 0]),'oe-every': ('over the expected, every pair', ['--norm', 'oe', '--minDist', 0], ['--expected', GP /'expected.tsv', '--mindist', 0]),}for cut_key, (label, app_args, options) in GM_CUTS.items():# Timed raw, not over the expected, for § 11's reason: coolpuppy is handed the expected and the app# computes its own. timed ='oe'notin cut_key.split('-') mr('apa', GM_10K, '--anchors', 'consecutive', '--features', GM_PEAKS_BED, *app_args, '--mode', 'fixed','--window', FLANK, '--minDiag', MIN_DIAG, out=GP / cut_key, name=f'gm-apa-{cut_key}', analysis=GM_ANALYSIS, label=f'APA, {label}', group='APA', bench=timed) reference(f'gm-coolpuppy-{cut_key}', GM_ANALYSIS, 'coolpuppy', [PYTHON, PILEUP, GM_10K, GM_PAIRS, FLANK * clr_gm.binsize, OUT, THREADS, *options], out=GP /f'coolpuppy-{cut_key}.tsv', label=f'APA, {label}', group='APA', threads=THREADS, bench=timed)# Fig. 2's own panel: the same pairs on the 2-kb rung, ten bins either side, from coolpuppy's floor for# that window (22 bins), over that rung's expected. Not timed, for the reason above.GM_COOL_2K = DATA /'4DNFIXP4QG5B_2000.chr19.cool'ifnot GM_COOL_2K.exists(): subprocess.run([PYTHON, str(SCRIPTS /'cool_subset.py'), GM_MAP, '2000', str(GM_COOL_2K), '--sequences',','.join(GM_SEQUENCES)], check=True)GM_2K, GM_HALF_2K = rel(GM_COOL_2K), 10reference('gm-expected-2kb', GM_ANALYSIS, 'cooltools', [PYTHON, EXPECTED, GM_2K, OUT, THREADS], out=GP /'expected-2kb.tsv', label='the expected at 2 kb, for coolpuppy', group='APA', threads=THREADS, bench=False)mr('apa', GM_2K, '--anchors', 'consecutive', '--features', GM_PEAKS_BED, '--norm', 'oe', '--minDist',2* GM_HALF_2K +2, '--mode', 'fixed', '--window', GM_HALF_2K, '--minDiag', MIN_DIAG, out=GP /'oe-2kb', name='gm-apa-oe-2kb', analysis=GM_ANALYSIS, label='APA at 2 kb, over the expected', group='APA', bench=False)reference('gm-coolpuppy-oe-2kb', GM_ANALYSIS, 'coolpuppy', [PYTHON, PILEUP, GM_2K, GM_PAIRS, GM_HALF_2K *2000, OUT, THREADS, '--expected', GP /'expected-2kb.tsv'], out=GP /'coolpuppy-oe-2kb.tsv', label='APA at 2 kb, over the expected', group='APA', threads=THREADS, bench=False)# The TADs: each domain of the app's insulation, from one boundary's bin to the next's, on the 2-kb# rung, scaled about its middle by half its length each side and stretched evenly onto 51 cells a side,# over the expected — the one scaled pileup that draws nothing at random, so the one coolpuppy and the# app can agree on cell for cell. Both read the same domains, written here as a plain BED.GM_DOMAINS = GP /'domains.bed'pd.read_csv(one(GI /'app', '*.insulation-domains.bed'), sep='\t', header=None, comment='#', usecols=[0, 1, 2]).to_csv( GM_DOMAINS, sep='\t', header=False, index=False)GM_GRID, GM_TAD_FLANK =51, 0.5mr('apa', GM_2K, '--anchors', 'features', '--features', GM_DOMAINS, '--mode', 'scaled', '--grid', GM_GRID,'--flankFraction', GM_TAD_FLANK, '--minDiag', MIN_DIAG, '--norm', 'oe', out=GP /'tads-oe', name='gm-apa-tads-oe', analysis=GM_ANALYSIS, label='TADs, scaled, over the expected', group='APA', bench=False)reference('gm-coolpuppy-tads-oe', GM_ANALYSIS, 'coolpuppy', [PYTHON, PILEUP, GM_2K, GM_DOMAINS, 100_000, OUT, THREADS, '--local', '--expected', GP /'expected-2kb.tsv','--rescale-flank', GM_TAD_FLANK, '--rescale-size', GM_GRID], out=GP /'coolpuppy-tads-oe.tsv', label='TADs, scaled, over the expected', group='APA', threads=THREADS, bench=False)#: Every pileup compared, by its label.GM_PILEUPS = {**{k: v[0] for k, v in GM_CUTS.items()}, 'oe-2kb': 'at 2 kb, over the expected (Fig. 2)','tads-oe': 'TADs at 2 kb, scaled, over the expected (Fig. 2)'}# A peak whose midpoint sits half a base before a bin edge: rounded to the nearest base it is in the next# bin, and coolpuppy floors it.gm_mid = gm_called['start'] + gm_called['end']gm_edge = {res: int(((gm_mid %2==1) & ((gm_mid //2+1) % res ==0)).sum()) for res in (10_000, 2000)}gm_cut_app = {k: pileup_of(GP / k) for k in GM_PILEUPS}gm_cut_ref = {k: np.loadtxt(GP /f'coolpuppy-{k}.tsv') for k in GM_PILEUPS}gm_same_gaps = {k: bool((np.isnan(gm_cut_app[k]) == np.isnan(gm_cut_ref[k])).all()) for k in GM_PILEUPS}gm_cut_rel = {k: relative_difference(gm_cut_app[k], gm_cut_ref[k])[np.isfinite(gm_cut_ref[k])] for k in GM_PILEUPS}measures({'consecutive pairs; nearer than coolpuppy\'s floor; median separation':f'{len(gm_pairs):,}; {int((gm_apart < MIN_DIST).sum()):,}; {np.median(gm_apart) * clr_gm.binsize /1e3:,.0f} kb','peaks whose midpoint is half a base before a bin edge, at 10 kb; at 2 kb': f'{gm_edge[10_000]}; {gm_edge[2000]}',**{f'{GM_PILEUPS[k]}: max relative difference; cells left as gaps, chromosomix / coolpuppy':f'{gm_cut_rel[k].max():.1e}; {int(np.isnan(gm_cut_app[k]).sum())} / {int(np.isnan(gm_cut_ref[k]).sum())}'for k in GM_PILEUPS},'over the expected, at coolpuppy\'s floor: centre; P2LL': f"{gm_cut_app['oe'][FLANK, FLANK]:.3f}; {p2ll(gm_cut_app['oe']):.3f}",})
2,106 sites, rows paired one to one: yes; profiles within 4.8e-06; cells past both tools' rounding 0 of 505,440; order plotHeatmap’s; 0 sites no track covers
insulation
2,106 sites, rows paired one to one: yes; profiles within 4.8e-06; cells past both tools' rounding 0 of 842,400; order plotHeatmap’s; 0 sites no track covers
Code
TEXT['gm_pairs'] =f'{len(gm_pairs):,}'TEXT['gm_apa_rel'] =f'{max(v.max() for v in gm_cut_rel.values()):.1e}'TEXT['gm_edge'] =f'{gm_edge[2000]:,}'TEXT['gm_tads'] =f"{len(pd.read_csv(GM_DOMAINS, sep='\t', header=None)):,}"GM_APA_GAPS =all(gm_same_gaps.values())TEXT['gm_apa_gaps'] ='the same cells left as gaps'if GM_APA_GAPS else'different cells left as gaps'TEXT['gm_peaks'] =f'{len(gm_called):,}'TEXT['gm_peaks_same'] =f'{len(gm_same_called):,}'TEXT['gm_idr'] =f'{max(gm_idr_app):.1e}'TEXT['gm_idr_self'] =f'{max(gm_idr_self):.1e}'GM_IDR_WITHIN =max(gm_idr_app) <=max(gm_idr_self) +1e-5GM_PEAKS_SAME =len(gm_same_called) ==len(gm_called) ==len(gm_called_ref) and gm_counts_agreeTEXT['gm_mp_profile'] =f"{max(v['profile'] for v in gm_plot.values()):.1e}"TEXT['gm_mp_past'] =f"{sum(v['past'] for v in gm_plot.values()):,}"TEXT['gm_mp_sites'] =f"{gm_plot['signal']['rows']:,}"GM_METAPLOT_OK =all(v['past'] ==0and v['one_to_one'] and v['order'] for v in gm_plot.values())fig, axes = plt.subplots(2, 3, figsize=(12, 7.6), layout='constrained')ax = axes[0, 0]grey_gm = plt.get_cmap('RdBu_r').copy()grey_gm.set_bad(MISSING)bar = square(ax, gm_cut_app['oe-every'], 'a consecutive peaks, over the expected', cmap=grey_gm, norm=LogNorm(1/2, 2), label='observed / expected')bar.set_ticks([1/2, 1, 2], labels=['½', '1', '2'])bar.minorticks_off()ax = axes[0, 1]for k, cut_key inenumerate(GM_PILEUPS): values = np.maximum(gm_cut_rel[cut_key], 1e-9) ax.scatter(k + np.linspace(-0.25, 0.25, values.size), values, s=6, color=APP_COLOUR, lw=0, gid=GM_PILEUPS[cut_key])ax.set(yscale='log', ylim=(1e-9, 1e-3), ylabel='|chromosomix / coolpuppy − 1|', title='b every cell, six pileups')ax.set_xticks(range(len(GM_PILEUPS)), ['raw', 'O/E', 'raw,\nevery\npair', 'O/E,\nevery\npair', 'O/E,\n2 kb', 'TADs,\nO/E'], fontsize=8)agreement(axes[0, 2], gm_idr['cond_globalIDR'].to_numpy(float), gm_idr['score'].to_numpy(float), 'yapc 0.1','chromosomix', 'c −log10 IDR, called peaks')names_signal = ['CTCF · Broad', 'CTCF · UW', 'ATAC-seq']for ax, plot_key, names_, unit in ((axes[1, 0], 'signal', names_signal, 'from the peak (bp)'), (axes[1, 1], 'insulation', [f'{w *10} kb'for w in GM_WINDOWS], 'from the peak (kb)')): v = gm_plot[plot_key] x = v['offsets'] / (1e3if plot_key =='insulation'else1)for i, (mine_, theirs_, name_) inenumerate(zip(v['mine'], v['theirs'], names_)): ax.plot(x, theirs_, color=REF, lw=3, label='deepTools plotProfile'if i ==0elseNone, gid=f'deepTools plotProfile, {name_}') ax.plot(x, mine_, color=APP_COLOUR, lw=1.2, label='chromosomix'if i ==0elseNone, gid=f'chromosomix, {name_}')# The signal's three curves meet at their ends and part at the peak: each is named beside its summit. at =int(np.nanargmax(mine_)) if plot_key =='signal'else-1 ax.annotate(name_, (x[at], mine_[at]), xytext=(12if plot_key =='signal'else2, 0), textcoords='offset points', fontsize=7, color=INK2, va='center') ax.set(xlabel=unit, ylabel='fold change over control'if plot_key =='signal'else'insulation (log2)', title='d CTCF and ATAC-seq'if plot_key =='signal'else'e insulation, four scales')# One key for both, where d has room for it.axes[1, 0].legend(loc='center right', fontsize=7, handlelength=1.5)agreement(axes[1, 2], np.r_[gm_plot['signal']['heat_theirs'], gm_plot['insulation']['heat_theirs']], np.r_[gm_plot['signal']['heat_mine'], gm_plot['insulation']['heat_mine']], 'deepTools computeMatrix','chromosomix', 'f every bin of every row')drawn('fig-gm-fig2')
Figure 26: The rest of Fig. 2’s chain on GM12878, chr19. a, the pileup over consecutive CTCF peaks, every pair, over the expected, as chromosomix draws it; b, every cell of the six pileups against coolpuppy’s, the last two Fig. 2’s own at 2 kb, its consecutive peaks and its TADs; c, −log10 IDR of every peak both call at IDR ≤ 0.05; d, CTCF’s two tracks and ATAC-seq about the peaks; e, the insulation at its four scales about them; f, every cell of both metaplots’ heatmaps against computeMatrix’s.
The chain agrees link by link. The same 2,106 of the 2,106 CTCF peaks pass IDR ≤ 0.05 in both, their −log10 IDR within 1.9e-05 of every yapc run, whose seeds differ from one another by up to 2.3e-05. Over the 2,105 pairs of consecutive peaks the five pileups draw the same cells left as gaps and every other cell within 4.9e-06 of coolpuppy’s — Fig. 2’s at 2 kb too, where 1 of the peaks has its midpoint half a base before a bin edge and both tools round it down — and so do the 305 TADs between the boundaries, stretched evenly over the expected. About the 2,106 peaks, every profile is within 4.8e-06 of plotProfile‘s, and 0 cells of the heatmaps lie past the two tools’ rounding.
Code
GM_OK = (max(d.max() for d in gm_diff.values()) <1e-5andint((gm_calls['_merge'] !='both').sum()) ==0and GM_PEAKS_SAME and GM_APA_GAPS andmax(v.max() for v in gm_cut_rel.values()) <1e-5and GM_METAPLOT_OK)record(GM_ANALYSIS, 'GM12878 Hi-C, 10 and 2 kb; CTCF ChIP-seq of two labs and ATAC-seq; chr19',f"cooltools {VERSIONS['cooltools']}, yapc {YAPC}, coolpuppy {VERSIONS['coolpuppy']}, deepTools {VERSIONS['deeptools']}",f"insulation at four scales within {TEXT['gm_ins_max']}, the same {TEXT['gm_calls']} boundaries; "f"the same {TEXT['gm_peaks_same']} peaks at IDR ≤ 0.05, −log10 IDR within {TEXT['gm_idr']} "f"(seeds among themselves: up to {TEXT['gm_idr_self']}); APA over {TEXT['gm_pairs']} consecutive pairs within "f"{TEXT['gm_apa_rel']}; metaplots within {TEXT['gm_mp_profile']}, {TEXT['gm_mp_past']} cells past the rounding", ('Identical (float32)'+ ('; IDR within yapc\'s seed-to-seed spread'if GM_IDR_WITHIN else'')) if GM_OKelse'Differs', apps=[name for name, app in APP.items() if app['analysis'] == GM_ANALYSIS], refs=[name for name, ref in REFS.items() if ref['analysis'] == GM_ANALYSIS])
25 · Pairs statistics (pairtools)
What pairtools stats counts over a contact list — records mapped on both sides, on one and on neither, duplicates, cis and trans and cis past 1 to 40 kb, every pair type, the four strand orientations by separation and where they converge, and the library’s size — counted by the pass the app’s import makes of a .pairs file, through chromosomix stats, against pairtools stats itself. The library is a yeast Hi-C contact list as pairtools wrote it, deduplicated, each pair with its type and its two SAM records. pairtools has an environment of its own, on Python 3.12: 1.1.2 imports the pipes module 3.13 removed.
chromosomix
pairtools
command
stats
stats
compared
every key pairtools writes, and its value as the file spells it
Code
PS = WORK /'pairstats'PS.mkdir(parents=True, exist_ok=True)mr('stats', CH128, out=PS /'app', name='pairstats', analysis='Pairs statistics')reference('pairtools-stats', 'Pairs statistics', 'pairtools', [PAIRTOOLS, 'stats', CH128, '-o', OUT], out=PS /'pairtools.stats', env=PAIRTOOLS_ENV)def stats_file(path):"""A `.stats` file as pairtools reads one back: a key and a value a line."""withopen(path) as handle:returndict(line.rstrip('\n').split('\t', 1) for line in handle if line.strip())ours, theirs = stats_file(one(PS /'app', '*.stats')), stats_file(PS /'pairtools.stats')only =sorted(set(ours) ^set(theirs))differ = {key: (ours.get(key), value) for key, value in theirs.items() if ours.get(key) != value}STRANDS = ['++', '--', '-+', '+-']#: The separation bins' left edges, as pairtools' keys name them: `dist_freq/<lo>-<hi>/<strands>`, the#: last `dist_freq/<lo>+/<strands>`.edges =sorted({int(key.split('/')[1].split('-')[0].rstrip('+')) for key in theirs if key.startswith('dist_freq/')})ranges = [f'{lo}-{hi}'for lo, hi inzip(edges, edges[1:])] + [f'{edges[-1]}+']def binned(table, strands):"""One orientation's cis pairs per separation bin, in the bins' order."""return np.array([int(table[f'dist_freq/{span}/{strands}']) for span in ranges])TEXT['ps_records'] =f"{int(theirs['total']):,}"TEXT['ps_keys'] =f'{len(theirs):,}'TEXT['ps_same'] ='every one of them spelled the same'ifnot differ andnot only else\f'{len(differ)} spelled differently and {len(only)} in one file only'TEXT['ps_types'] =', '.join(key.split('/')[1] for key in theirs if key.startswith('pair_types/'))TEXT['ps_cis'] =f"{float(theirs['summary/frac_cis']):.1%}"TEXT['ps_meet'] = theirs['summary/dist_freq_convergence/convergence_dist']display(measures({'records': TEXT['ps_records'],'keys, chromosomix / pairtools': f'{len(ours):,} / {len(theirs):,}','keys in one file only': ', '.join(only) or'none','values spelled differently': ', '.join(f'{k}: {a} / {b}'for k, (a, b) in differ.items()) or'none','mapped, duplicates (pairtools)': f"{int(theirs['total_mapped']):,}, {int(theirs['total_dups']):,}",'cis, of the unique (pairtools)': TEXT['ps_cis'],'pair types': TEXT['ps_types'],'strands converge past (pairtools), bp': TEXT['ps_meet'],'library size (pairtools)': theirs['summary/complexity_naive'],}))
fig, ax = plt.subplots(figsize=(7, 4), layout='constrained')# Each bin's width, the last as wide as an eighth of a decade past its edge; drawn from 1 bp, on a log axis.widths = np.diff(np.r_[edges, edges[-1] *10** (1/8)])starts = np.maximum(edges, 1)# An orientation a dash, pairtools' in orange and chromosomix's points over it in blue.DASHES = {'++': '-', '--': (0, (5, 2)), '-+': (0, (1, 1.5)), '+-': (0, (6, 2, 1, 2))}for strands in STRANDS: reference_curve, app_curve = binned(theirs, strands), binned(ours, strands) seen = reference_curve >0 ax.plot(starts[seen], (reference_curve / widths)[seen], color=REF, lw=1.3, ls=DASHES[strands], label=f'pairtools, {strands}') ax.plot(starts[seen], (app_curve / widths)[seen], ls='none', marker='o', ms=2.4, color=APP_COLOUR, label='chromosomix, each orientation'if strands =='++'elseNone, gid=f'chromosomix {strands}')meet = theirs['summary/dist_freq_convergence/convergence_dist']if meet !='0': ax.axvline(int(meet), color=INK, lw=1, ls=(0, (4, 3)), gid='convergence') ax.text(int(meet), 0.97, ' converged', transform=ax.get_xaxis_transform(), va='top', ha='right', fontsize=7, color=INK2, rotation=90)ax.set(xscale='log', yscale='log', xlabel='separation, bp', ylabel='cis pairs per bp of separation', title='Cis pairs by separation and strand orientation')ax.legend(ncol=2, fontsize=7)drawn('fig-pairstats')PAIRSTATS_SAME =not differ andnot only
Figure 27: Cis pairs by separation in each of the four strand orientations, per base pair of each bin: pairtools’ (lines) and chromosomix’s (points), and the separation past which pairtools finds every orientation within 5 % of their mean.
pairtools wrote 509 keys for the 17,583,821 pairs, and the app’s import counted every one of them spelled the same: the totals, the pair types (UU), the cis fraction of 97.1%, every separation bin of every orientation, and the convergence past 1778279 bp. A library with no duplicates has no estimate of its size, and both write nan for it: scipy’s Lambert W at -exp(-1), which lies a hair past -1/e, and the port’s.
Code
record('Pairs statistics', 'yeast Hi-C contact list, deduplicated', f'pairtools {PAIRTOOLS_VERSION}','every key, and every value as the file spells it','Identical'if PAIRSTATS_SAME else'Differs', apps=['pairstats'], refs=['pairtools-stats'])
26 · Restriction sites (bioframe)
Where an enzyme cuts a reference. chromosomix restriction finds the sites of DpnII (^GATC) along the yeast reference R64-1-1 and counts them in each bin of a 5 kb map; bioframe.digest finds them with Biopython’s Bio.Restriction. § 27 reads § 25’s contact list against these sites.
The enzyme. The enzyme CH128 was made with is not recorded. DpnII is chosen to check the computation, not the biology: § 27’s classes say what the library looks like if DpnII cut it, and nothing in the file says that it did. The reads themselves point to a GATC cutter (§ 27), but that is the reads’ evidence, not a record.
The cut, 2 bp apart. For a site whose first base is at s (0-based), cut k bases in (0 for ^GATC), chromosomix cuts at c = s + k, the 0-based index of the first base after the cut, and a fragment runs from one cut to the next, [Bi, Bi+1). bioframe’s fragments end 2 bp later, at c + 2. Biopython’s search gives the 1-based position of the first base after the cut, c + 1, and bioframe.digest adds 1 to it (np.array(cut_finder(seq)) + 1), where converting it to 0-based would take 1 away. cooler digest is the same code, so a fragment BED from cooler or bioframe is 2 bp late, at every site of every enzyme. The check is therefore that bioframe’s boundaries are chromosomix’s cuts plus 2, every one. The fragments pairtools restrict is given are bioframe’s moved back by 2 (bioframe_digest.py’s SHIFT), chromosomix’s own, so that it puts ends on the boundaries the app puts them on — for the top strand; § 27 has the bottom strand’s.
chromosomix
reference
cuts
restriction --enzyme DpnII, its sites
bioframe.digest(genome, 'DpnII'), each boundary less 2
cuts per bin
restriction, its density on the 5 kb map’s bins
the same cuts, counted per bin
Code
import gzipRS = WORK /'restriction'RS.mkdir(parents=True, exist_ok=True)RS_ANALYSIS, PQ_ANALYSIS ='Restriction sites', 'Pairs QC'ENZYME, RS_BIN ='DpnII', 5_000#: R64-1-1 as one plain FASTA beside its index, by which bioframe reads it, and its sizes, by which cooler#: bins: written once from the gzip Ensembl serves.R64 = PATHS['R64-1-1']R64_FA, R64_SIZES = DATA /'R64-1-1.fa', DATA /'R64-1-1.chrom.sizes'if REDO ornot R64_FA.exists():with gzip.open(R64, 'rt') as packed, open(R64_FA, 'w') as plain: shutil.copyfileobj(packed, plain) Path(f'{R64_FA}.fai').unlink(missing_ok=True)pyfaidx.Faidx(str(R64_FA)).close()R64_LENGTHS = pd.read_csv(f'{R64_FA}.fai', sep='\t', header=None, usecols=[0, 1], names=['chrom', 'length'])if REDO ornot R64_SIZES.exists(): R64_LENGTHS.to_csv(R64_SIZES, sep='\t', header=False, index=False)def pairs_sizes(path):"""A contact list's `#chromsize:` lines: its sequences and their lengths, in its order.""" sizes = {}with gzip.open(path, 'rt') as handle:for line in handle:ifnot line.startswith('#'):breakif line.startswith('#chromsize:'): name, length = line.split()[1:3] sizes[name] =int(length)return sizesCH128_SIZES = pairs_sizes(CH128)R64_SAME =list(zip(R64_LENGTHS['chrom'], R64_LENGTHS['length'])) ==list(CH128_SIZES.items())ifnot R64_SAME:raiseValueError("R64-1-1's sequences are not the contact list's, by name, length and order.")measures({'sequences, R64-1-1 and the contact list': f"{len(R64_LENGTHS)}: {', '.join(R64_LENGTHS['chrom'])}",'names, lengths and order': 'the same','bases': f"{R64_LENGTHS['length'].sum():,}"})
measure
value
sequences, R64-1-1 and the contact list
17: I, II, III, IV, V, VI, VII, VIII, IX, X, XI, XII, XIII, XIV, XV, XVI, Mito
names, lengths and order
the same
bases
12,157,105
The reference’s commands: the digest, and the 5 kb map cooler cload pairs bins from § 25’s contact list, whose bins the cuts are counted on. The map is also the binning Pairs QC does itself, and its coverage is § 27’s, so it is listed with § 27’s commands and timed against Pairs QC.
Code
reference('bioframe-digest', RS_ANALYSIS, 'bioframe, Biopython', [PYTHON, script('bioframe_digest'), R64_FA, ENZYME, RS_BIN, OUT, -2], out=RS /'digest', made=RS /'digest.sites.tsv')# cooler's program, beside the interpreter pip put it with. Pairs QC's reference, made here for its bins.reference('cooler-cload', PQ_ANALYSIS, 'cooler', ['cooler', 'cload', 'pairs', '-c1', 2, '-p1', 3, '-c2', 4, '-p2', 5, f'{rel(R64_SIZES)}:{RS_BIN}', CH128, OUT], out=RS /f'CH128.{RS_BIN}.cool', env={'PATH': f'{Path(PYTHON).parent}{os.pathsep}{os.environ["PATH"]}'})
#: Each output §§ 26 and 27 read, by where chromosomix writes it: `<map>.<task>.<output id>.<ext>`, as#: `writeOutputs` (src/cli/main.ts) names its files. The readers below put each into the reference's shape;#: this cell and § 27's are the only places an output's name or columns are spelled.APP_OUT = {# The cuts, a point [c, c + 1) each, and their density on the map's bins: `restriction-<the enzyme>`.'cuts': (RS /'app-sites', '*.restriction.restriction-*.bed'),'density': (RS /'app-sites', '*.restriction.restriction-*-density.bedGraph'),}def app_out(key):return one(*APP_OUT[key])def merged(app, path, on, value):"""chromosomix's numbers beside the reference's, keyed, every key of either kept: a missing number is 0.""" ref = pd.read_csv(path, sep='\t', dtype={'chrom': str}) both = app.merge(ref, on=on, how='outer', suffixes=('_app', '_ref')) both[[f'{value}_app', f'{value}_ref']] = both[[f'{value}_app', f'{value}_ref']].fillna(0)return bothdef app_cuts():"""chromosomix's cuts, a row each (chrom, cut): the start of each point in the BED it writes."""return pd.read_csv(app_out('cuts'), sep='\t', comment='#', header=None, usecols=[0, 1], names=['chrom', 'cut'], dtype={'chrom': str})def app_density():"""Its cuts per bin (chrom, start, end, sites), from the density's bedGraph."""return pd.read_csv(app_out('density'), sep='\t', comment='#', header=None, names=['chrom', 'start', 'end', 'sites'], dtype={'chrom': str})
Code
rs_lengths =dict(zip(R64_LENGTHS['chrom'], R64_LENGTHS['length']))# bioframe's boundaries between fragments, as it writes them: every end but each sequence's last.rs_bioframe = pd.read_csv(RS /'digest.bioframe.bed', sep='\t', header=None, names=['chrom', 'start', 'end'], dtype={'chrom': str})rs_bioframe = rs_bioframe[rs_bioframe['end'] != rs_bioframe['chrom'].map(rs_lengths)]rs_app_cuts = app_cuts()def offsets(app, bioframe):"""bioframe's boundaries less chromosomix's cuts, sequence by sequence, the first less the first, the second less the second and so on — or, where the two find a different number, each cut's nearest.""" out = []for chrom, ours in app.groupby('chrom'): theirs = np.sort(bioframe.loc[bioframe['chrom'] == chrom, 'end'].to_numpy()) at = np.sort(ours['cut'].to_numpy())iflen(theirs) ==len(at): out.append(theirs - at)continue k = np.clip(np.searchsorted(theirs, at), 1, len(theirs) -1) out.append(np.where(np.abs(theirs[k -1] - at) <= np.abs(theirs[k] - at), theirs[k -1], theirs[k]) - at)return np.concatenate(out)rs_offsets = pd.Series(offsets(rs_app_cuts, rs_bioframe)).value_counts().sort_index()rs_cut_match = rs_app_cuts.assign(end=rs_app_cuts['cut'] +2).merge(rs_bioframe[['chrom', 'end']], how='outer', indicator=True)['_merge'].value_counts()rs_density = merged(app_density(), RS /'digest.sites.tsv', ['chrom', 'start', 'end'], 'sites')CUTS_SAME = rs_cut_match.get('left_only', 0) ==0and rs_cut_match.get('right_only', 0) ==0DENSITY_SAME =bool((rs_density['sites_app'] == rs_density['sites_ref']).all())measures({"cuts, chromosomix / bioframe's boundaries": f"{len(rs_app_cuts):,} / {len(rs_bioframe):,}","bioframe's boundary less chromosomix's cut, in order": ', '.join(f'{k:+d} bp at {n:,}'for k, n in rs_offsets.items()),'cuts in one only, once 2 is added to chromosomix\'s': f"{rs_cut_match.get('left_only', 0):,} / "f"{rs_cut_match.get('right_only', 0):,}",f'{RS_BIN //1000} kb bins with the same cuts':f"{int((rs_density['sites_app'] == rs_density['sites_ref']).sum()):,} of {len(rs_density):,}",})
measure
value
cuts, chromosomix / bioframe's boundaries
35,896 / 35,896
bioframe's boundary less chromosomix's cut, in order
+2 bp at 35,896
cuts in one only, once 2 is added to chromosomix's
0 / 0
5 kb bins with the same cuts
2,441 of 2,441
Code
fig, axes = plt.subplots(1, 2, figsize=(8.4, 3.6), layout='constrained')agreement(axes[0], rs_density['sites_ref'].to_numpy(float), rs_density['sites_app'].to_numpy(float),'bioframe, boundaries less 2', 'chromosomix', f'a cuts per {RS_BIN //1000} kb bin')ax = axes[1]ax.bar(rs_offsets.index, rs_offsets.to_numpy(), width=0.6, color=INK2, gid='cuts')ax.set_xticks(range(min(-1, rs_offsets.index.min() -1), max(4, rs_offsets.index.max() +2)))ax.set(xlabel="bioframe's boundary less chromosomix's cut, bp", ylabel='cuts', title=f'b {len(rs_app_cuts):,} cuts, each against bioframe')drawn('fig-restriction')
Figure 28: DpnII on R64-1-1, chromosomix against bioframe. a, cuts per 5 kb bin, bioframe’s boundaries less 2; b, bioframe’s boundary less chromosomix’s cut, at every cut.
Code
RS_SAME = CUTS_SAME andset(rs_offsets.index) == {2} and DENSITY_SAMETEXT['rs_cuts'] =f'{len(rs_bioframe):,}'TEXT['rs_offset'] = ('each 2 bp past the app’s cut'ifset(rs_offsets.index) == {2}else'at '+', '.join(f'{k:+d} bp from the app’s cut at {n:,}'for k, n in rs_offsets.items()))TEXT['rs_bins'] =f'{len(rs_density):,}'TEXT['rs_density'] ='the same number of cuts in every one'if DENSITY_SAME else\f"different numbers in {int((rs_density['sites_app'] != rs_density['sites_ref']).sum()):,}"
bioframe finds the 35,896 sites chromosomix finds, each 2 bp past the app’s cut: bioframe’s + 1 on top of Biopython’s 1-based position. In the 2,441 bins of 5 kb the two count the same number of cuts in every one.
Code
record(RS_ANALYSIS, 'yeast R64-1-1, DpnII, 5 kb bins',f"bioframe {VERSIONS['bioframe']}, Biopython {VERSIONS['biopython']}",f"the same {TEXT['rs_cuts']} cuts, bioframe's 2 bp later; {TEXT['rs_density']} of the {TEXT['rs_bins']} bins",'Identical'if RS_SAME else'Differs', apps=['restriction'], refs=['bioframe-digest'])
27 · Pairs QC (pairtools, cooltools)
What a contact list says about the fragments an enzyme cut. The restriction half of chromosomix pairsqc reads § 25’s contact list against § 26’s sites: the fragment each end lies in, the class of each pair, each end’s distance to the cut it reads towards, how many of the fragments the pairs reach, and each 5 kb bin’s coverage against its cuts. Its reference is three tools: pairtools restrict, which puts each end in a fragment; a short pandas script, restriction_reference.py, which counts from pairtools restrict’s columns by the definitions the task states (they are in the script’s docstring, below); and cooltools.coverage, over the same pairs binned by cooler cload pairs (§ 26). The rest of Pairs QC draws § 25’s counts. Its table of them is held to pairtools stats here too.
An end’s fragment, by its own strand’s cut. A .pairs position is the 1-based 5′ end of the read, x = pos − 1. Given a BED of fragments, pairtools puts it in fragment searchsorted([0, end + 1, …], pos, 'right') − 1, which is searchsorted(B, x, 'right') − 1: an end on a boundary lies in the fragment that starts there. That suits a read on the + strand, whose 5′ end is the top strand’s. A read on the − strand starts at the bottom strand’s 5′ end, and the bottom strand is cut elsewhere. DpnII cuts each strand before its own GATC. On the top strand’s coordinates that is before the G (c = s) and after the C (c′ = s + 4), a four-base overhang that the library’s fill-in makes double-stranded. So a fragment’s bottom strand runs to c′ − 1, where c′ = s + L − k is L − 2k bases past c; its last base is the site’s C. Placed by the top strand’s cuts, the − read of an unligated fragment lies 3 bp into the next fragment, and a dangling end counts as a re-ligation.
chromosomix therefore places a + end among the top strand’s boundaries, B = [0, …c, N], and a − end among the bottom strand’s, B′ = [0, …c′, N]. Fragment i is the same fragment in both. Each end’s distance is measured on its own strand’s boundaries: B[i+1] − x on +, x − B′[i] + 1 on −, so a dangling end’s two reads both give the fragment’s length. The reference does the same with pairtools. pairtools restrict runs twice, on the top strand’s fragments and on the bottom strand’s (bioframe’s boundaries moved by −2, and by −2 + L − 2k, which is +2 for DpnII), and the script reads each end’s fragment from the run on its strand. Everything counted below follows from where each end was put, so it checks chromosomix’s placement end by end.
chromosomix
reference
each end’s fragment
pairsqc --enzyme DpnII
pairtools restrict on each strand’s fragments, a + end read from the one and a − end from the other
classes, distances, fragments seen
pairsqc
restriction_reference.py, from pairtools restrict’s columns
coverage against cuts
pairsqc --bin-size 5000
§ 26’s cooler cload pairs at 5 kb, cooltools.coverage(ignore_diags=0) less each bin’s diagonal (§ 9), SciPy’s spearmanr
the pairs’ statistics
pairsqc, its table
pairtools stats (§ 25’s)
The reference’s commands: each end’s fragment and what is counted from it, and the coverage of § 26’s map. pairtools stats is § 25’s command and output; it is listed here again so that the timing weighs Pairs QC against every reference command it stands for, cooler cload among them.
Code
# Once on the top strand's fragments, for the + ends, and once on the bottom strand's, for the − ends.for strand, suffix in (('top', ''), ('bottom', '-bottom')): reference(f'pairtools-restrict-{strand}', PQ_ANALYSIS, 'pairtools', [PAIRTOOLS, 'restrict', '-f', RS /f'digest.frags{suffix}.bed', CH128, '-o', OUT], out=RS /f'CH128.restricted{suffix}.pairs.gz', env=PAIRTOOLS_ENV)reference('restriction-reference', PQ_ANALYSIS, 'pandas', [PYTHON, script('restriction_reference'), R64_FA, ENZYME, RS /'CH128.restricted.pairs.gz', RS /'CH128.restricted-bottom.pairs.gz', OUT, -2], out=RS /'pairtools', made=RS /'pairtools.summary.tsv')reference('cooltools-coverage-sites', PQ_ANALYSIS, 'cooltools', [PYTHON, script('cooltools_coverage_sites'), RS /f'CH128.{RS_BIN}.cool', RS /'digest.sites.tsv', OUT, THREADS], out=RS /'coverage', made=RS /'coverage.summary.tsv', threads=THREADS)reference('pairsqc-pairtools-stats', PQ_ANALYSIS, 'pairtools', [PAIRTOOLS, 'stats', CH128, '-o', OUT], out=PS /'pairtools.stats', env=PAIRTOOLS_ENV)
#: Pairs QC's outputs, each written as a TSV, read as § 26's cell reads the cuts.RS_CLASSES = ['trans', 'dangling', 'self-circle', 'same-strand', 're-ligation', 'valid']RS_ORIENTATIONS = ['++', '+-', '-+', '--']APP_OUT.update({'classes': (RS /'app-pairsqc', '*.pairsqc.pairsqc-classes-table.tsv'),'distance': (RS /'app-pairsqc', '*.pairsqc.pairsqc-distance.tsv'),'fragments': (RS /'app-pairsqc', '*.pairsqc.pairsqc-fragments.tsv'),'sites': (RS /'app-pairsqc', '*.pairsqc.pairsqc-sites.tsv'),'stats': (RS /'app-pairsqc', '*.pairsqc.pairsqc-stats.tsv'),})def ascii_minus(names):"""Strands as the reference spells them, should a figure's − have reached a column's name."""return [str(n).replace('\u2212', '-') for n in names]def app_classes():"""Its pairs per class and orientation (class, orientation, pairs), from its table of the classes: a row a class, its pairs and share, and a column an orientation.""" held = table(app_out('classes')) held.columns = ascii_minus(held.columns)return held[['class', *RS_ORIENTATIONS]].melt(id_vars='class', var_name='orientation', value_name='pairs')def app_distance():"""Its ends per distance bin and strand (strand, start, ends): a curve, each bin at its left edge, a column a strand.""" held = table(app_out('distance')) held.columns = ['start', *ascii_minus(held.columns[1:])]return held.melt(id_vars='start', value_vars=['+', '-'], var_name='strand', value_name='ends')def app_fragments():"""Its fragments per number of ends (label, fragments): a bar a range, named as the range."""return table(app_out('fragments')).set_axis(['label', 'fragments'], axis=1)def app_seen():"""The fragments seen and of how many, as the fragments' label says: `… · seen / total fragments seen`."""withopen(app_out('fragments')) as handle: label =next(line for line in handle if line.startswith('# output:'))return [int(n.replace(',', '')) for n in re.search(r'([\d,]+)\s*/\s*([\d,]+)', label).groups()]def app_sites():"""Its points, a bin each (sites, coverage), in its order — by sites, then along the genome — and its mean line (sites, coverage), one row per number of sites: a curve whose two series share the points' x.""" held = table(app_out('sites')).set_axis(['sites', 'coverage', 'mean'], axis=1)# A number of cuts too few bins share has its points and no mean: the column is blank there.return (held[['sites', 'coverage']], held[['sites', 'mean']].dropna(subset=['mean']).drop_duplicates('sites') .rename(columns={'mean': 'coverage'}))def app_rho():"""Spearman's ρ as Pairs QC writes it in its note, as written."""for line in NOTES['pairsqc']: found = re.search(r'ρ\s*=\s*(-?[\d.]+)', line)if found:return found.group(1)raiseValueError(f'no note of pairsqc gives ρ: {NOTES["pairsqc"]}')def app_stats():"""Its table of the pairs' statistics: pairtools' key, and the value as written.""" held = pd.read_csv(app_out('stats'), sep='\t', comment='#', dtype=str, keep_default_na=False)returndict(zip(held['statistic'], held['value']))
Code
rs_summary = pd.read_csv(RS /'pairtools.summary.tsv', sep='\t').set_index('measure')['value']rs_orientation = merged(app_classes(), RS /'pairtools.orientation.tsv', ['class', 'orientation'], 'pairs')rs_distance = merged(app_distance(), RS /'pairtools.distance.tsv', ['strand', 'start'], 'ends')rs_fragments = merged(app_fragments(), RS /'pairtools.fragments.tsv', ['label'], 'fragments')# The bins kept, in the order chromosomix writes its points: by their cuts, then along the genome.rs_kept = pd.read_csv(RS /'coverage.coverage.tsv', sep='\t', dtype={'chrom': str})rs_kept = rs_kept[(rs_kept['coverage'] >0) | (rs_kept['sites'] >0)].sort_values('sites', kind='stable')app_bins, app_means = app_sites()rs_coverage = pd.concat([rs_kept[['chrom', 'start', 'sites', 'coverage']].reset_index(drop=True).add_suffix('_ref'), app_bins.reset_index(drop=True).add_suffix('_app')], axis=1)# Pairs QC draws the mean only where at least this many bins share a number of cuts, so a repeat's one# bright bin does not draw the line; the reference's means are held to the same bins.MEAN_MIN_BINS =5rs_means_ref = pd.read_csv(RS /'coverage.means.tsv', sep='\t')rs_means = (app_means.merge(rs_means_ref.loc[rs_means_ref['bins'] >= MEAN_MIN_BINS, ['sites', 'coverage']], on='sites', how='outer', suffixes=('_app', '_ref')).fillna(0))rs_cov_summary = pd.read_csv(RS /'coverage.summary.tsv', sep='\t').set_index('measure')['value']RHO_REF, RHO_APP =float(rs_cov_summary['spearman']), app_rho()SEEN_APP, TOTAL_APP = app_seen()rs_stats_app, rs_stats_ref = app_stats(), stats_file(PS /'pairtools.stats')rs_stats_differ = {k: (v, rs_stats_ref.get(k)) for k, v in rs_stats_app.items() if rs_stats_ref.get(k) != v}def same(frame, value):returnint((frame[f'{value}_app'] != frame[f'{value}_ref']).sum())RS_HELD = {'classes by orientation': same(rs_orientation, 'pairs'),'distance bins, both strands': same(rs_distance, 'ends'),'fragments per number of ends': same(rs_fragments, 'fragments'),f'{RS_BIN //1000} kb bins kept, in order: coverage': same(rs_coverage, 'coverage'),f'{RS_BIN //1000} kb bins kept, in order: cuts': same(rs_coverage, 'sites'),}RS_MEANS_GAP =float(relative_difference(rs_means['coverage_app'].to_numpy(float), rs_means['coverage_ref'].to_numpy(float)).max())RS_RHO_GAP =abs(float(RHO_APP) - RHO_REF)RS_RHO_NEAR = rounded(len(RHO_APP.split('.')[-1]))measures({'pairs counted (pairtools restrict, pandas)': f"{int(rs_summary['counted']):,} of {int(rs_summary['records']):,}","ends pairtools puts on fragments other than the digest's": f"{int(rs_summary['ends off the digest']):,}",**{f'{name}, differing': f'{n:,}'for name, n in RS_HELD.items()},'fragments seen, chromosomix / pandas':f"{SEEN_APP:,} of {TOTAL_APP:,} / {int(rs_summary['fragments seen']):,} of {int(rs_summary['fragments']):,}",'mean coverage per number of cuts, max relative difference': f'{RS_MEANS_GAP:.1e}',"Spearman's ρ, chromosomix (as written) / SciPy": f'{RHO_APP} / {RHO_REF:.6f}','statistics in its table, written as pairtools writes them':f"{len(rs_stats_app) -len(rs_stats_differ):,} of {len(rs_stats_app):,}",})
measure
value
pairs counted (pairtools restrict, pandas)
17,583,821 of 17,583,821
ends pairtools puts on fragments other than the digest's
0
classes by orientation, differing
0
distance bins, both strands, differing
0
fragments per number of ends, differing
0
5 kb bins kept, in order: coverage, differing
0
5 kb bins kept, in order: cuts, differing
0
fragments seen, chromosomix / pandas
34,289 of 35,913 / 34,289 of 35,913
mean coverage per number of cuts, max relative difference
3.9e-06
Spearman's ρ, chromosomix (as written) / SciPy
0.482 / 0.482321
statistics in its table, written as pairtools writes them
51 of 51
Code
rs_classes = rs_orientation.groupby('class', sort=False)[['pairs_app', 'pairs_ref']].sum().reindex(RS_CLASSES)rs_counted =int(rs_summary['counted'])# Every closed bin from 1 bp, no end being 0 from its cut, to the last any end is in on either strand.rs_reached = rs_distance.loc[(rs_distance[['ends_app', 'ends_ref']] >0).any(axis=1), 'start'].max()rs_closed = rs_distance[rs_distance['end'].notna() & rs_distance['start'].between(1, rs_reached)].sort_values( ['strand', 'start'])rs_frag_order = pd.read_csv(RS /'pairtools.fragments.tsv', sep='\t')['label']rs_frag_bars = rs_fragments.set_index('label').reindex(rs_frag_order).fillna(0)RS_DASH = {'+': '-', '-': (0, (4, 2))}fig, axes = plt.subplots(2, 3, figsize=(12, 7.6), layout='constrained')ax = axes[0, 0]at = np.arange(len(RS_CLASSES))ax.bar(at, rs_classes['pairs_ref'] / rs_counted, width=0.7, color=REF, label='pairtools restrict, pandas')ax.bar(at, rs_classes['pairs_app'] / rs_counted, width=0.28, color=APP_COLOUR, label='chromosomix')ax.set_xticks(at, RS_CLASSES, rotation=30, ha='right')ax.set(yscale='log', ylabel='share of the pairs counted', title='a classes')ax.legend(loc='upper left')ax = axes[0, 1]shown = rs_orientation[(rs_orientation['pairs_ref'] >0) | (rs_orientation['pairs_app'] >0)]ax.scatter(shown['pairs_ref'], shown['pairs_app'], s=22, color=APP_COLOUR, zorder=3, gid='class and orientation')identity(ax, shown[['pairs_ref', 'pairs_app']].min().min(), shown[['pairs_ref', 'pairs_app']].max().max())# A name a point, those of a class within 2 % of one another named once.for name, rows in shown.sort_values('pairs_ref').groupby(['class', np.round(np.log10(shown['pairs_ref']) /0.01)], sort=False): ax.annotate(f"{name[0]}{', '.join(rows['orientation'])}", (rows['pairs_ref'].iloc[0], rows['pairs_app'].iloc[0]), fontsize=6, color=INK2, xytext=(4, -3), textcoords='offset points')ax.set(xscale='log', yscale='log', xlabel='pairtools restrict, pandas', ylabel='chromosomix', title='b classes by orientation')ax = axes[0, 2]for strand, rows in rs_closed.groupby('strand'): edges_ = np.r_[rows['start'].to_numpy(float), rows['end'].to_numpy(float)[-1]] width = np.diff(edges_) ax.stairs(rows['ends_ref'] / width, edges_, color=REF, lw=3, ls=RS_DASH[strand], label=f'pairtools restrict, pandas, {strand}') ax.stairs(rows['ends_app'] / width, edges_, color=APP_COLOUR, lw=1.1, ls=RS_DASH[strand], label=f'chromosomix, {strand}')ax.set(xscale='log', yscale='log', xlabel='distance to the cut, bp', ylabel='ends per bp', title='c distance to the cut, by strand')ax.legend(loc='lower left', fontsize=7)ax = axes[1, 0]at = np.arange(len(rs_frag_bars))ax.bar(at, rs_frag_bars['fragments_ref'], width=0.7, color=REF, label='pairtools restrict, pandas')ax.bar(at, rs_frag_bars['fragments_app'], width=0.28, color=APP_COLOUR, label='chromosomix')ax.set_xticks(at, rs_frag_bars.index, rotation=60, ha='right', fontsize=7)ax.set(yscale='log', xlabel='ends in the fragment', ylabel='fragments', title=f"d ends per fragment; {int(rs_summary['fragments seen']):,} of {int(rs_summary['fragments']):,} seen")ax.legend(loc='upper left')ax = axes[1, 1]ax.scatter(rs_coverage['sites_ref'], rs_coverage['coverage_ref'], s=9, color=REF, alpha=0.5, lw=0, label='cooltools, bioframe')ax.scatter(rs_coverage['sites_app'], rs_coverage['coverage_app'], s=2, color=APP_COLOUR, lw=0, label='chromosomix')ax.plot(rs_means['sites'], rs_means['coverage_ref'], color=REF, lw=3, label='mean, pandas')ax.plot(rs_means['sites'], rs_means['coverage_app'], color=APP_COLOUR, lw=1.1, label='mean, chromosomix')ax.text(0.97, 0.68, f'Spearman ρ\nSciPy {RHO_REF:.4f}\nchromosomix {RHO_APP}', transform=ax.transAxes, ha='right', va='top', fontsize=7, color=INK2)ax.set(xlabel=f'cuts in the {RS_BIN //1000} kb bin', ylabel='contacts, the diagonal once', title='e coverage against cuts')ax.legend(loc='upper right', fontsize=7)ax = axes[1, 2]# Its statistics written as numbers, each against pairtools stats', those above 0 on the log axes.rs_stats_both = pd.DataFrame([(pd.to_numeric(rs_stats_ref[k], errors='coerce'), pd.to_numeric(v, errors='coerce'))for k, v in rs_stats_app.items() if k in rs_stats_ref], columns=['ref', 'app'])rs_stats_both = rs_stats_both[(rs_stats_both['ref'] >0) & (rs_stats_both['app'] >0)]ax.scatter(rs_stats_both['ref'], rs_stats_both['app'], s=12, color=APP_COLOUR, zorder=3, gid='statistic')identity(ax, rs_stats_both.min().min(), rs_stats_both.max().max())ax.set(xscale='log', yscale='log', xlabel='pairtools stats', ylabel='chromosomix', title=f'f {len(rs_stats_both):,} statistics above 0')drawn('fig-pairsqc')
Figure 29: The contact list’s pairs against § 26’s DpnII sites: pairtools restrict, pandas, pairtools stats and cooltools (orange) and chromosomix (blue). a, the pairs in each class, of those counted; b, each class in each orientation; c, the distance from each end to the cut it reads towards, per bp of each bin, on each strand; d, the fragments with each number of ends; e, each 5 kb bin’s coverage against its cuts, and the mean coverage per number of cuts; f, the statistics in Pairs QC’s table against pairtools stats’, each above 0.
Code
PQ_SAME = (notany(RS_HELD.values()) andint(rs_summary['ends off the digest']) ==0and (SEEN_APP, TOTAL_APP) == (int(rs_summary['fragments seen']), int(rs_summary['fragments']))and RS_RHO_GAP <= RS_RHO_NEAR and RS_MEANS_GAP <= significant(6) andnot rs_stats_differ)rs_fraction = rs_classes['pairs_ref'] / rs_countedTEXT['rs_counted'] =f'{rs_counted:,}'TEXT['rs_held'] = ('the same, key for key'ifnotany(RS_HELD.values())else'; '.join(f'{name}: {n:,} differ'for name, n in RS_HELD.items() if n))TEXT['rs_seen'] =f"{int(rs_summary['fragments seen']):,}"TEXT['rs_total'] =f"{int(rs_summary['fragments']):,}"TEXT['rs_kept'] =f"{int(rs_cov_summary['kept']):,}"TEXT['rs_rho'] =f'{RHO_REF:.4f}'TEXT['rs_rho_app'] = RHO_APPTEXT['rs_means'] =f'{RS_MEANS_GAP:.1e}'TEXT['rs_stats'] = ('every one'ifnot rs_stats_differelsef'all but {len(rs_stats_differ)} of the {len(rs_stats_app)}')for name in RS_CLASSES: TEXT[f'rs_{name}'] =f'{rs_fraction[name]:.1%}'# Reads that start where DpnII cut, on each strand, over those a base away.TEXT['rs_plus'] =f"{rs_summary['+ ends on a cut'] / rs_summary['+ ends a base past a cut']:.0f}"TEXT['rs_minus'] =f"{rs_summary['- ends on a cut'] / rs_summary['- ends a base before a cut']:.0f}"# And the classes had every end been placed by the top strand's cuts.rs_top_only = pd.read_csv(RS /'pairtools.classes.tsv', sep='\t').set_index('class')['top_strand_only']TEXT['rs_top_dangling'] =f"{rs_top_only['dangling'] / rs_counted:.1%}"TEXT['rs_top_relig'] =f"{rs_top_only['re-ligation'] / rs_counted:.1%}"
Of the 17,583,821 pairs counted, the pandas script’s counts from pairtools restrict’s fragments and Pairs QC’s are the same, key for key: the six classes in their four orientations, every distance bin on both strands, the fragments with each number of ends (34,289 of 35,913 reached), and each bin’s coverage and cuts. The mean coverage at each number of cuts that at least five bins share — the only ones Pairs QC draws a mean for, so a repeat’s one bright bin does not draw the line — agrees within 3.9e-06, the six digits written. Over the 2,441 bins with coverage or a cut, Spearman’s ρ is 0.4823 by SciPy and 0.482 as chromosomix writes it, to three decimals. Of the statistics in Pairs QC’s table, every one is written as pairtools stats writes it.
If DpnII cut it, CH128 has 75.8% valid pairs, 8.9% dangling ends, 11.2% re-ligations, 2.9% trans, 0.7% same-strand and 0.6% self-circles. The reads point to a GATC cutter. + ends start on a site’s G, the top strand’s cut, 41 times as often as a base past it. − ends start on its C, the bottom strand’s cut, 38 times as often as a base before it. Those are the two ends of a fragment DpnII cut and the library filled in, and they are why each end is placed by its own strand’s cut. With every end placed by the top strand’s cuts, as one run of pairtools restrict places them, the dangling ends would have been 8.2% and the re-ligations 11.0%.
Code
record(PQ_ANALYSIS, "§ 25's contact list against § 26's DpnII sites, 5 kb",f"pairtools {PAIRTOOLS_VERSION}, cooler {VERSIONS['cooler']}, cooltools {VERSIONS['cooltools']}",f"classes, distances, fragments seen and coverage {TEXT['rs_held']}; ρ {TEXT['rs_rho_app']} against "f"{TEXT['rs_rho']}; {TEXT['rs_stats']} of its statistics as pairtools writes it",'Identical'if PQ_SAME else'Differs', apps=['pairsqc'], refs=['cooler-cload', 'pairtools-restrict-top', 'pairtools-restrict-bottom', 'restriction-reference','cooltools-coverage-sites', 'pairsqc-pairtools-stats'])
28 · TAD annotation (Juicer Arrowhead)
Contact domains as juicer_tools arrowhead calls them, by the port of its BlockBuster in chromosomix arrowhead: each window of 2,000 bins along the diagonal turned into the arrowhead transform, every corner scored by its two triangles’ means, signs and variances, a strict pass and a lenient one, and the corners near one another merged. The map is 4DN’s in situ Hi-C of K562 (Rao et al. 2014, 4DNFITUOMFUQ) as Juicer wrote it, a .hic both read by range from 4DN’s bucket; chromosomes 21 and 22 at 10 kb, which is the resolution Juicer picks for a map of this depth, and each longer than one of Arrowhead’s 2,000-bin windows. Twice: on the raw counts, which both read as the same float32 counts, and balanced, where Juicer is given the KR vector and chromosomix reads the file’s own preferred vector — SCALE, then KR (src/hic/HicFile.ts), and 4DN’s pipeline wrote no SCALE. Juicer Tools 1.22.01 runs on Java 21: one step walks a HashSet whose order changed in JDK 19, and the port follows a current JDK.
chromosomix
Juicer Tools
window
--matrixSize 2000 (default)
-m 2000
thresholds
variance 0.2, sign 0.5; lenient 0.4 to 0 by 0.1, wider than 60 bins (defaults)
its BlockBuster defaults
counts
--no-balance; balanced (default)
-k NONE; -k KR
compared
each domain’s corners, and its five scores, to the digits Juicer prints
Code
AH = WORK /'arrowhead'AH_RES =10_000AH_CHROMS = ['21', '22']#: Each variant: Juicer's normalisation and chromosomix's flags for the same counts.AH_VARIANTS = {'raw': ('NONE', ['--no-balance']), 'balanced': ('KR', [])}for variant, (norm, flags) in AH_VARIANTS.items(): mr('arrowhead', f'{K562_HIC}::/resolutions/{AH_RES}', '--sequences', ','.join(AH_CHROMS), *flags, out=AH /f'app-{variant}', name=f'arrowhead-{variant}', analysis='TAD annotation', label=variant) reference(f'juicer-arrowhead-{variant}', 'TAD annotation', 'Juicer Tools', ['java', '-Xmx8g', '-jar', JUICER, 'arrowhead', '-c', ','.join(AH_CHROMS), '-m', '2000','-r', str(AH_RES), '-k', norm, '--threads', '1', K562_HIC, OUT], out=AH /f'juicer-{variant}', made=AH /f'juicer-{variant}'/f'{AH_RES}_blocks.bedpe', label=variant)AH_SCORES = ['score', 'uVarScore', 'lVarScore', 'upSign', 'loSign']def juicer_blocks(path):"""Juicer's `_blocks.bedpe`, by position: its header names `score` twice, and the five scores are always a row's last five.""" rows = []for line in Path(path).read_text().splitlines():if line.startswith('#') ornot line.strip():continue fields = line.split('\t') rows.append({'sequence': fields[0], 'start': int(fields[1]), 'end_juicer': int(fields[2]),**{key: float(value) for key, value inzip(AH_SCORES, fields[-5:])}})return pd.DataFrame(rows)AH_RESULTS = {}for variant in AH_VARIANTS:# A sequence's name is a name, `21` included, on both sides of the merge. mine = table(one(AH /f'app-{variant}', '*.arrowhead-table.tsv')).astype({'sequence': str}) theirs = juicer_blocks(AH /f'juicer-{variant}'/f'{AH_RES}_blocks.bedpe')# Juicer writes a domain to the start of its last corner bin, and chromosomix clamps that to the# sequence: an end is the same, or chromosomix's is short of Juicer's by less than a bin, which only a# domain ending on a sequence's last bin can be. Nested domains share starts, so a pair is matched on# its start and its end both. pairs = mine.merge(theirs, on=['sequence', 'start'], suffixes=('_app', '_juicer')) pairs = pairs[(pairs['end'] == pairs['end_juicer'])| ((pairs['end'] < pairs['end_juicer']) & (pairs['end_juicer'] - pairs['end'] < AH_RES))] paired = pairs.drop_duplicates(['sequence', 'start', 'end']) ends = pd.Series(True, index=paired.index) only = (len(mine) -len(paired)) + (len(theirs) - paired[['sequence', 'start', 'end_juicer']].drop_duplicates().shape[0]) worst = {key: float(np.abs(paired[f'{key}_app'] - paired[f'{key}_juicer']).max()) iflen(paired) else np.nanfor key in AH_SCORES} AH_RESULTS[variant] =dict(mine=mine, theirs=theirs, paired=paired, only=int(only), ends=int(ends.sum()), worst=worst)# chromosomix writes a score to six decimals.AH_EXACT =all(r['only'] ==0and r['ends'] ==len(r['paired']) andlen(r['paired']) >0andmax(r['worst'].values()) <= rounded(6) for r in AH_RESULTS.values())measures({**{f'{v}: domains, chromosomix / Juicer': f"{len(r['mine']):,} / {len(r['theirs']):,}"for v, r in AH_RESULTS.items()},**{f'{v}: same corners; in one only': f"{len(r['paired']):,}; {r['only']}"for v, r in AH_RESULTS.items()},**{f'{v}: max |difference|, five scores': ', '.join(f'{k}{x:.1e}'for k, x in r['worst'].items())for v, r in AH_RESULTS.items()},'chromosomix, as it noted (balanced)': ' '.join(NOTES['arrowhead-balanced']),'Java': JAVA_VERSION,})
200 domains on 2 of 2 sequences, 84 of them inside another.
Java
21.0.13
Code
balanced = AH_RESULTS['balanced']TEXT['ah_n'] =f"{len(balanced['paired']):,}"TEXT['ah_raw_n'] =f"{len(AH_RESULTS['raw']['paired']):,}"TEXT['ah_only'] =f"{sum(r['only'] for r in AH_RESULTS.values())}"TEXT['ah_worst'] =f"{max(max(r['worst'].values()) for r in AH_RESULTS.values()):.1e}"fig = plt.figure(figsize=(11, 6.4), layout='constrained')grid = fig.add_gridspec(2, 2, width_ratios=[1, 2.2])ax = fig.add_subplot(grid[:, 0])ax.scatter(balanced['paired']['score_juicer'], balanced['paired']['score_app'], s=10, color=APP_COLOUR, alpha=0.7, lw=0)identity(ax, balanced['paired']['score_juicer'].min(), balanced['paired']['score_juicer'].max())ax.set(xlabel='Juicer, corner score', ylabel='chromosomix, corner score', title='a corner scores')for row, (who, frame, colour, sign) inenumerate([('chromosomix', balanced['mine'], APP_COLOUR, 1), ('Juicer', balanced['theirs'], REF, -1)]): ax = fig.add_subplot(grid[row, 1]) on = frame[frame['sequence'] == AH_CHROMS[0]] stop = on['end'if who =='chromosomix'else'end_juicer']for start, end inzip(on['start'], stop): ax.plot([start /1e6, (start + end) /2e6, end /1e6], [0, sign * (end - start) /2e6, 0], color=colour, lw=1) ax.axhline(0, color=INK2, lw=0.8) ax.set(xlabel=f'{AH_CHROMS[0]} (Mb)', ylabel='half the span (Mb)', title=f"{'b'if row ==0else''}{who}, {len(on):,} domains")drawn('fig-arrowhead')
Figure 30: TAD annotation, K562 Hi-C at 10 kb, balanced. a, the corner score of every domain both call; b, the domains of chromosome 21 as the triangles they span on the map, chromosomix’s above the axis and Juicer’s below it.
200 domains balanced and 177 raw at the same corners (0 in one only), and every one of the five scores within 5.0e-07, which is the six decimals chromosomix writes.
Code
record('TAD annotation', 'K562 Hi-C (.hic), chromosomes 21 and 22, 10 kb, raw and KR', 'Juicer Tools 1.22.01',f"{TEXT['ah_n']} domains (balanced) and {TEXT['ah_raw_n']} (raw), the same corners; scores within "f"{TEXT['ah_worst']}, the digits written",'Identical (to the digits written)'if AH_EXACT else'Differs', apps=['arrowhead-raw', 'arrowhead-balanced'], refs=['juicer-arrowhead-raw', 'juicer-arrowhead-balanced'])
29 · Map report: the resolution ladder (cooler)
The map report’s own numbers. At each resolution, how a map’s contacts spread over its bins — how many bins, the mean and the median contacts per bin, the count 80 % of bins reach, and the empty and nearly empty shares — with a contact counted at both its ends, as Juicer’s calculate_map_resolution.sh counts it. chromosomix counts the ends once, at the file’s finest resolution, and sums them to every coarser one. The reference reads each resolution the .mcool stores, as cooler zoomify coarsened it, and counts each one’s ends from its own pixel table with numpy (map_resolution.py, below). The map is the yeast Micro-C of § 4.
chromosomix
cooler and numpy
command
mapreport, the ladder from the file’s finest resolution (default)
every stored resolution’s own pixels
compared
at each resolution the file stores: bins, mean, median, the 80 % count and the two shares
Code
MR = WORK /'mapreport'mr('mapreport', f'{rel(YEAST)}::/resolutions/800', out=MR /'app', name='mapreport', analysis='Map report')reference('map-resolution', 'Map report', 'cooler', [PYTHON, script('map_resolution'), YEAST, OUT], out=MR /'cooler.tsv')mr_app = table(one(MR /'app', '*.mapreport.report-resolutions.tsv'))mr_ref = pd.read_csv(MR /'cooler.tsv', sep='\t')mr_both = mr_app.merge(mr_ref, on='binsize', suffixes=('_app', '_ref'))MR_COLUMNS = ['bins', 'mean', 'median', 'reached80', 'empty', 'belowTenthMedian']#: chromosomix writes the mean and the median to one decimal, the shares to four, the counts whole.MR_DIGITS = {'bins': 0, 'mean': 1, 'median': 1, 'reached80': 0, 'empty': 4, 'belowTenthMedian': 4}mr_worst = {key: float(np.abs(mr_both[f'{key}_app'] - mr_both[f'{key}_ref']).max()) for key in MR_COLUMNS}MR_EXACT =len(mr_both) ==len(mr_ref) andall(mr_worst[k] <= rounded(MR_DIGITS[k]) for k in MR_COLUMNS)TEXT['mr_n'] =f'{len(mr_both)}'TEXT['mr_sizes'] =', '.join(f'{b:,}'for b insorted(mr_both['binsize']))measures({'resolutions the file stores, compared': f"{len(mr_both)} of {len(mr_ref)} ({TEXT['mr_sizes']} bp)",'rows chromosomix reports': f'{len(mr_app)} (the file’s and its finest doubled)',**{f'{k}, max |difference|': f'{x:.1e}'for k, x in mr_worst.items()},})show(mr_both[['binsize', *[f'{k}_{s}'for k in MR_COLUMNS for s in ('app', 'ref')]]])
At each of the 10 resolutions the file stores, chromosomix’s sums of its finest bins are the counts cooler’s own coarsening gives, to the digits written.
Code
record('Map report', 'yeast Micro-C, every stored resolution', f"cooler {VERSIONS['cooler']}, numpy",f"{TEXT['mr_n']} resolutions, bins, mean, median, 80 % count and shares, the digits written",'Identical (to the digits written)'if MR_EXACT else'Differs', apps=['mapreport'], refs=['map-resolution'])
Summary
Commands as run, from notebooks/work/validation/: chromosomix is this checkout’s node dist-cli/chromosomix.mjs, python the reference packages’ interpreter, yapc-env/bin/python yapc’s. The reference scripts, in notebooks/scripts/, are listed below.
Code
WIDTHS = {'§': 3, 'Analysis': 8, 'Data': 9, 'chromosomix command': 24, 'Reference': 8, 'Reference command': 24,'Agreement': 13, 'Verdict': 11}summary = pd.DataFrame(SUMMARY)[list(WIDTHS)]# Every verdict as a table of its own, published beside the page (.github/workflows/notebooks.yml).VERDICTS_TSV = WORK /'summary.tsv'summary.drop(columns=['chromosomix command', 'Reference command']).replace(r'<[^>]+>', '', regex=True).to_csv( VERDICTS_TSV, sep='\t', index=False)# And every number the figures draw, a row a mark (`drawn`), published beside it too: to ten significant# digits, as many as chromosomix writes its finest table to (`boost`) and more than any figure shows.DRAWN_TXT = WORK /'validation.txt'pd.concat(DRAWN.values(), ignore_index=True).to_csv(DRAWN_TXT, sep='\t', index=False, float_format='%.10g')summary.style.hide(axis='index').set_table_styles([ {'selector': '', 'props': [('table-layout', 'fixed'), ('width', '100%'), ('font-size', '0.85em')]}, {'selector': 'td, th', 'props': [('vertical-align', 'top'), ('text-align', 'left')]}, {'selector': 'td pre', 'props': [('font-size', '0.85em'), ('white-space', 'pre-wrap'), ('overflow-wrap', 'anywhere'), ('margin', '0'), ('padding', '0'), ('background', 'transparent')]},*[{'selector': f'th.col{i}', 'props': [('width', f'{w}%')]} for i, w inenumerate(WIDTHS.values())],])
same seed: within chromosomes ρ = 0.978–0.993, between 0.90–1.00 (pastis vs pastis 0.940–0.949, 0.50–0.57); fit within chromosomes 0.866–0.871 vs 0.868–0.873, all pairs 0.473–0.480 vs 0.473–0.480
Same seed, same model; fits within chromosomes as well as pastis; all pairs as well as pastis
raw and over the expected, at coolpuppy's defaults and with every loop, and on the diagonal: the same cells left as gaps, every other cell within 4.5e-06; over controls, two runs of the two tools differ as two of coolpuppy do (0.134 against 0.151), and no more cells than chance gives (1 of 121 at p < 0.01)
Identical (float32); controls within the seeds' spread
12
APA, chromosight's score
mouse Hi-C, 10 kb, 2,515 loops
$ chromosomix apa data/4DNFIOZ1TRUS.mcool::/resolutions/10000 --pairs apa/loops.bedpe --norm chromosight --out chromosight-quantify/app
18 coefficients within 4.5e-07 of the exact means, a window any track lacks left out of every pair as deepTools leaves it; deepTools within 0.027, from its zoom-level means
raw: the same contacts within and between every sequence; balanced: the pairs within 4.6e-06; against indifference within 4.6e-06 of NumPy over cooltools' sums
insulation at four scales within 5.0e-06, the same 306 boundaries; the same 2,106 peaks at IDR ≤ 0.05, −log10 IDR within 1.9e-05 (seeds among themselves: up to 2.3e-05); APA over 2,105 consecutive pairs within 4.9e-06; metaplots within 4.8e-06, 0 cells past the rounding
Identical (float32); IDR within yapc's seed-to-seed spread
10 resolutions, bins, mean, median, 80 % count and shares, the digits written
Identical (to the digits written)
Code
def manifest():"""Every command, for timing.qmd: in the summary's order, each analysis's chromosomix commands before its references, each with what the timing needs to run it again and to say what it was.""" data = {row['Analysis']: (row['§'], row['Data']) for row in SUMMARY} out = []for analysis, (section, about) in data.items():for name, app in APP.items():if app['analysis'] == analysis: out.append(dict(name=name, section=section, analysis=analysis, data=about, tool='chromosomix', version=APP_VERSION, variant=app['label'], group=app['group'], threads=1, bench=app['bench'], argv=app['argv'], out=None, env=None, command=app['shown']))for name, ref in REFS.items():if ref['analysis'] == analysis: out.append(dict(name=name, section=section, analysis=analysis, data=about, tool=ref['tool'], version=TOOL_VERSIONS.get(ref['tool'], '?'), variant=ref['label'], group=ref['group'], threads=ref['threads'], bench=ref['bench'], argv=ref['argv'], out=ref['out'].name, env=ref['env'], command=ref['shown']))return {'placeholder': OUT, 'chromosomix': APP_VERSION, 'threads': THREADS, 'commands': out}MANIFEST.write_text(json.dumps(manifest(), indent=1, ensure_ascii=False))listed = json.loads(MANIFEST.read_text())['commands']measures({'commands written for timing.qmd': f"{len(listed)} ({sum(c['bench'] for c in listed)} timed)",'to': rel(MANIFEST), 'verdicts to': rel(VERDICTS_TSV),'numbers the figures draw': f"{sum(map(len, DRAWN.values())):,} rows from {len(DRAWN)} figures, "f"{DRAWN_TXT.stat().st_size /1e6:.1f} MB, to {rel(DRAWN_TXT)}"})
measure
value
commands written for timing.qmd
166 (103 timed)
to
commands.json
verdicts to
summary.tsv
numbers the figures draw
551,585 rows from 30 figures, 35.6 MB, to validation.txt
Code
VERDICTS = summary.set_index('Analysis')['Verdict']def named(analysis):"""An analysis's name inside a sentence."""return analysis if analysis.startswith(('P(s)', 'APA', 'Boost-HiC', '3D')) else analysis[0].lower() + analysis[1:]EXACT = [a for a, v in VERDICTS.items() if v.startswith('Identical')]REST = [a for a in VERDICTS.index if a notin EXACT andnot a.startswith('3D')]TEXT['identical'] ='; '.join(map(named, EXACT))TEXT['n_identical'] =f'{len(EXACT)}'TEXT['others'] ='; '.join(f'{named(a)}: {VERDICTS[a][0].lower() + VERDICTS[a][1:]}'for a in REST) or'none'
26 of the 29 analyses reproduce their reference to the float32 the app computes in or the digits it writes: diamond insulation; directionality index; P(s); peak calling (yapc, IDR); loops (chromosight); reproducibility (HiCRep); coverage; virtual 4C; aggregate peak analysis; APA, chromosight’s score; Boost-HiC; metaplot; interval overlap; base composition; motif search; track correlation; loop calling; cis and trans per sequence; saddle; signal by group; cTCF sites in GM12878 (Fig. 2); pairs statistics; restriction sites; pairs QC; tAD annotation; map report. Where a reference draws at random — the controls of a pileup, the shuffles of an overlap, IDR’s ties — the app’s answer lies within the reference’s own spread from one seed to the next. Besides those: compartments: same eigenvectors; fraction of signal in features: the same FRiP to its four decimals and every feature to the digits written, the whole track read from a zoom level. The 3D model fits the contacts within chromosomes as well as pastis, and over all pairs as well as pastis. P(s) as a density (HiContacts’ distanceLaw), the app’s default, is held to HiContacts by the unit tests (test/golden/analysis/hicontacts_ps.json). Each command’s time, but those of §§ 20–23, is measured by timing.qmd.
Reference scripts
The scripts the references are called through, from notebooks/scripts/:
Code
for path, analysis in {p: ref['analysis'] for ref in REFS.values() for p in ref['scripts']}.items():file= Path(os.path.normpath(WORK / path))print(f'**{analysis}**, `{file.relative_to(REPO)}`\n\n```python\n{file.read_text().rstrip()}\n```\n')
"""Dixon et al.'s directionality index with domaincaller, 2 Mb window (validation.qmd § 2).Each chromosome balanced and sparse, as domaincaller's own script builds it; one value per bin of thefile, NaN where domaincaller gives none.usage: python domaincaller_di.py MAP_URI OUT_NPY"""import sysimport coolerimport numpy as npfrom domaincaller.chromLev import Chromuri, out = sys.argv[1:]clr = cooler.Cooler(uri)di = np.full(clr.info['nbins'], np.nan)for chrom in clr.chromnames: matrix = clr.matrix(balance='weight', sparse=True).fetch(chrom).tocsr() work = Chrom(chrom, clr.binsize, matrix) work.calDI(window=2_000_000) lo = clr.extent(chrom)[0] di[lo:lo +len(work.DIs)] = work.DIsnp.save(out, di)
CTCF sites in GM12878 (Fig. 2), notebooks/scripts/yapc_idr.py
"""yapc 0.1's command line, with NumPy's random state seeded (validation.qmd § 5).yapc breaks score ties at random, through IDR's ranks; the seed makes a run repeatable. Everythingafter SEED is yapc's own arguments. Run with yapc's interpreter, its `yapc` on the PATH.usage: python yapc_idr.py SEED OUTPUT_PREFIX CONDITION REP1.bw REP2.bw [yapc options]"""import runpyimport shutilimport sysimport numpy as npnp.random.seed(int(sys.argv[1]))sys.argv = ['yapc', *sys.argv[2:]]runpy.run_path(shutil.which('yapc'), run_name='__main__')
"""HiCRep's stratum-adjusted correlation with hicrep, every pair of maps (validation.qmd § 7).Raw counts, h = 1, diagonals to 5 Mb, no subsampling; one row per pair and sequence.usage: python hicrep_scc.py OUT_TSV SEQ1,SEQ2,... LABEL=MAP_URI LABEL=MAP_URI [...]"""import sysimport coolerimport pandas as pdfrom hicrep import hicrepSCCout, seqs = sys.argv[1], sys.argv[2].split(',')maps =dict(m.split('=', 1) for m in sys.argv[3:])labels, rows =list(maps), []for i, a inenumerate(labels):for b in labels[i +1:]: scc = hicrepSCC(cooler.Cooler(maps[a]), cooler.Cooler(maps[b]), h=1, dBPMax=5_000_000, bDownSample=False, chrNames=seqs) rows += [dict(a=a, b=b, sequence=s, hicrep=v) for s, v inzip(seqs, scc)]pd.DataFrame(rows).to_csv(out, sep='\t', index=False)
3D model (pastis-NB), notebooks/scripts/pastis_nb.py
"""A pastis-NB structure with pastis' own functions, from one MDS seed (validation.qmd § 8).The beads are the bins with a balancing weight and a contact, as chromosomix models them; thebiases are the inverse weights. Writes the coordinates, one row per bead, as a .npy.usage: python pastis_nb.py MAP_URI OUT_NPY SEED"""import sysimport coolerimport numpy as npimport pandas as pdfrom pastis import dispersionfrom pastis.optimization import mds, negative_binomial_structure as nbsuri, out, seed = sys.argv[1], sys.argv[2], int(sys.argv[3])clr = cooler.Cooler(uri)raw = clr.matrix(balance=False)[:].astype(float)w = clr.bins()['weight'][:].to_numpy(float)keep = np.isfinite(w) & (raw.sum(axis=0) >0)chroms = clr.bins()['chrom'][:].to_numpy()[keep]lengths = np.array([np.sum(chroms == c) for c in pd.unique(chroms)])counts, bias = raw[np.ix_(keep, keep)], 1/ w[keep]bias /= bias.mean()_, mean, variance, weights = dispersion.compute_mean_variance(counts, lengths, bias=bias)fitted = dispersion.ExponentialDispersion(degree=1)fitted.fit(mean, variance, sample_weights=weights **0.5)start = mds.estimate_X(counts / np.outer(bias, bias), alpha=-3., beta=1., random_state=np.random.RandomState(seed))model = nbs.estimate_X(counts, alpha=-3., beta=1., ini=start.flatten(), dispersion=fitted, bias=bias, lengths=lengths, use_zero_entries=True, random_state=np.random.RandomState(seed))np.save(out, model.reshape(-1, 3))
Coverage, notebooks/scripts/cooltools_coverage.py
"""Cis and total coverage per bin with cooltools, every diagonal kept (validation.qmd § 9).Raw counts, or balanced by the weight column named; one row per bin of the file.usage: python cooltools_coverage.py MAP_URI OUT_TSV NPROC [WEIGHT]"""import sysimport coolerimport cooltoolsuri, out, nproc = sys.argv[1], sys.argv[2], int(sys.argv[3])weight = sys.argv[4] iflen(sys.argv) >4elseNoneclr = cooler.Cooler(uri)cis, total = cooltools.coverage(clr, ignore_diags=0, clr_weight_name=weight, nproc=nproc)table = clr.bins()[['chrom', 'start', 'end']][:]table['cis'], table['total'] = cis, totaltable.to_csv(out, sep='\t', index=False)
"""A virtual 4C profile with cooltools, balanced, genome-wide (validation.qmd § 10).One row per bin of the file; the viewpoint's own bins, and the bins it has nostored contact with, are NaN as cooltools leaves them.usage: python cooltools_virtual4c.py MAP_URI VIEWPOINT OUT_TSV NPROC"""import sysimport coolerimport cooltoolsuri, viewpoint, out, nproc = sys.argv[1], sys.argv[2], sys.argv[3], int(sys.argv[4])table = cooltools.virtual4c(cooler.Cooler(uri), viewpoint, clr_weight_name='weight', nproc=nproc)table.to_csv(out, sep='\t', index=False)
CTCF sites in GM12878 (Fig. 2), notebooks/scripts/coolpuppy_pileup.py
"""A pileup with coolpuppy (validation.qmd § 11).Balanced counts, each window ±FLANK bp about each loop, cut as coolpuppy cuts one by default:from the upper triangle, the first MIN_DIAG diagonals left out, and every loop nearer thediagonal than coolpuppy's `mindist='auto'` — twice the flank and two bins — left out, unless--mindist says otherwise (0 keeps every loop). With --expected, every window is divided bythe expected — cooltools' `expected_cis`, read as `balanced.avg` — before the mean(`ooe=True`). With --local, FEATURES is a BED and the windows sit on the diagonal; with--rescale-size too, each feature is scaled by 1 + 2 × RESCALE_FLANK about its middle andresampled to RESCALE_SIZE cells a side (`rescale=True`), and FLANK is not used. With--nshifts, the pileup is divided by that of NSHIFTS controls per loop, each moved along thediagonal by MINSHIFT to MAXSHIFT bp either way, with coolpuppy's seed. Writes the pileup as aTSV, one line per row of the window.usage: python coolpuppy_pileup.py MAP_URI FEATURES FLANK OUT_TSV NPROC [--min-diag N] [--mindist BP] [--expected TSV] [--local [--rescale-flank F --rescale-size N]] [--nshifts N --minshift BP --maxshift BP --seed S]"""import argparseimport coolerimport numpy as npimport pandas as pdfrom coolpuppy import coolpupparser = argparse.ArgumentParser()for name in ('uri', 'features'): parser.add_argument(name)parser.add_argument('flank', type=int)parser.add_argument('out')parser.add_argument('nproc', type=int)parser.add_argument('--min-diag', type=int, default=2)parser.add_argument('--mindist', type=int)parser.add_argument('--expected')parser.add_argument('--local', action='store_true')parser.add_argument('--rescale-flank', type=float)parser.add_argument('--rescale-size', type=int)for name in ('nshifts', 'minshift', 'maxshift', 'seed'): parser.add_argument(f'--{name}', type=int)args = parser.parse_args()if args.local: features = pd.read_csv(args.features, sep='\t', header=None, usecols=range(3), names=['chrom', 'start', 'end']) options =dict(features_format='bed', local=True)if args.rescale_size isnotNone: options.update(rescale=True, rescale_flank=args.rescale_flank, rescale_size=args.rescale_size)else: features = pd.read_csv(args.features, sep='\t', header=None, usecols=range(6), names=['chrom1', 'start1', 'end1', 'chrom2', 'start2', 'end2']) options =dict(features_format='bedpe')if args.mindist isnotNone: options.update(mindist=args.mindist)if args.expected isnotNone: options.update(expected_df=pd.read_csv(args.expected, sep='\t'), ooe=True)else: options.update(ooe=False)if args.nshifts: options.update(nshifts=args.nshifts, minshift=args.minshift, maxshift=args.maxshift, seed=args.seed)pileup = coolpup.pileup(cooler.Cooler(args.uri), features, flank=args.flank, min_diag=args.min_diag, nproc=args.nproc, **options)np.savetxt(args.out, np.asarray(pileup['data'].iloc[0], dtype=float), delimiter='\t', fmt='%.10g')
Boost-HiC, notebooks/scripts/boosthic.py
"""Boost-HiC on one sequence of a cooler, as its main.py runs it (validation.qmd § 12).The sequence's own block of raw counts, dense and symmetric as main.py loads aHiC-Pro matrix; `get_outliers`, then main.py's `BoostHiC` — `SCN`, `fastFloyd` on`1/SCN**alpha`, `adjustPdS` — at main.py's alpha of 0.2, through Boost-HiC's own`HiCutils`. Writes OUT_PREFIX-kept.npy, the bins the outlier filter kept (indicesinto the sequence), and OUT_PREFIX-boosted.npy, `boostedmat` over those bins inSCN units, which is what main.py writes.`HiCutils` is read from a checkout of LeopoldC/Boost-HiC, at BOOSTHIC_DIR or`../Boost-HiC` beside the repository. It does not import as committed: the firststatement of `clean_out`, which main.py never calls, sits at column zero. Thatline is put back inside its function, in memory, and nothing else is touched.usage: python boosthic.py MAP_URI SEQUENCE OUT_PREFIX [ALPHA]"""import osimport sysimport typesfrom pathlib import Pathimport coolerimport numpy as npREPO = Path(__file__).resolve().parents[2]BOOSTHIC = Path(os.environ.get('BOOSTHIC_DIR', REPO.parent /'Boost-HiC'))sys.path.insert(0, str(BOOSTHIC))sys.dont_write_bytecode =Truedef hicutils(): path = BOOSTHIC /'HiCutils.py' source = path.read_text().replace('\ndata[pos_out] = 0\n', '\n\tdata[pos_out] = 0\n') module = types.ModuleType('HiCutils') module.__file__=str(path)exec(compile(source, str(path), 'exec'), module.__dict__)return moduleHiCutils = hicutils()uri, sequence, out = sys.argv[1:4]alpha =float(sys.argv[4]) iflen(sys.argv) >4else0.2basemat = np.asarray(cooler.Cooler(uri).matrix(balance=False).fetch(sequence), dtype=float)pos_out = HiCutils.get_outliers(basemat)basematfilter = np.copy(basemat[np.ix_(~pos_out, ~pos_out)])with np.errstate(divide='ignore'): normmat = HiCutils.SCN(np.copy(basematfilter)) FFmat = np.power(HiCutils.fastFloyd(1/ np.power(normmat.copy(), alpha)), -1/ alpha)boostedmat = HiCutils.adjustPdS(normmat, FFmat)np.save(f'{out}-kept.npy', np.flatnonzero(~pos_out))np.save(f'{out}-boosted.npy', boostedmat)
"""Overlap of two interval sets with bedtools, and its shuffled null (validation.qmd § 15).`bedtools jaccard` for the bases the two share and the bases either covers, and`bedtools intersect -u` both ways for the intervals of each that touch the other.Then ITERATIONS times `bedtools shuffle -chrom` of the first set — each intervalmoved within its own sequence, its length kept, `-seed` the iteration — and thesame numbers for each shuffle. Writes OUT_PREFIX-observed.tsv and OUT_PREFIX-null.tsv.usage: python bedtools_overlap.py A_BED B_BED GENOME ITERATIONS OUT_PREFIX"""import subprocessimport sysimport pandas as pda, b, genome, iterations, out = sys.argv[1], sys.argv[2], sys.argv[3], int(sys.argv[4]), sys.argv[5]def bedtools(*argv, given=None):return subprocess.run(['bedtools', *argv], input=given, capture_output=True, text=True, check=True).stdoutdef measured(first):"""The numbers for one version of the first set, handed over sorted, as BED text.""" intersection, union, jaccard, _ = bedtools('jaccard', '-a', 'stdin', '-b', b, given=first).splitlines()[1].split('\t')return {'intersection': int(intersection), 'union': int(union), 'jaccard': float(jaccard),'a_hit': len(bedtools('intersect', '-u', '-a', 'stdin', '-b', b, given=first).splitlines()),'b_hit': len(bedtools('intersect', '-u', '-a', b, '-b', 'stdin', given=first).splitlines())}observed = bedtools('sort', '-i', a)pd.DataFrame([measured(observed)]).to_csv(f'{out}-observed.tsv', sep='\t', index=False)null = [measured(bedtools('sort', '-i', 'stdin', given=bedtools('shuffle', '-chrom', '-seed', str(seed), '-i', a,'-g', genome)))for seed inrange(1, iterations +1)]pd.DataFrame(null).to_csv(f'{out}-null.tsv', sep='\t', index=False)
Base composition, notebooks/scripts/composition.py
"""GC content per bin with bioframe, and GC skew with Biopython (validation.qmd § 16).Bins of BINSIZE along each sequence of the FASTA's index, the last one short:`bioframe.frac_gc` counts G and C over the bases that are not N, and Biopython's`GC_skew` is (G − C)/(G + C) over consecutive windows of the same width.usage: python composition.py FASTA BINSIZE OUT_TSV"""import sysimport bioframeimport pandas as pdfrom Bio.SeqUtils import GC_skewfasta, binsize, out = sys.argv[1], int(sys.argv[2]), sys.argv[3]genome = bioframe.load_fasta(fasta, engine='pyfaidx')sizes = pd.read_csv(f'{fasta}.fai', sep='\t', header=None, usecols=[0, 1], index_col=0).iloc[:, 0]table = bioframe.frac_gc(bioframe.binnify(sizes, binsize), genome)table['skew'] = [skew for chrom in sizes.index for skew in GC_skew(str(genome[chrom][:]), window=binsize)]table.to_csv(out, sep='\t', index=False)
"""Track correlation from pyBigWig's exact window means and SciPy (validation.qmd § 18).Every sequence the tracks share is cut into whole windows of BINSIZE bases from its start, eachtrack's value in a window is pyBigWig's `stats(..., exact=True)` mean — the mean over the basesit has a value for — and every pair of tracks is correlated over the windows where every track hasone, as deepTools' plotCorrelation drops a row with a missing value before it correlates anything,by `scipy.stats.pearsonr` or `scipy.stats.spearmanr`. Writes the square matrix as a TSV, thetracks named by their file names.usage: python pybigwig_correlation.py OUT_TSV BINSIZE METHOD TRACK [TRACK ...]"""import osimport sysimport numpy as npimport pyBigWigfrom scipy import statsout, size, method, paths = sys.argv[1], int(sys.argv[2]), sys.argv[3], sys.argv[4:]handles = [pyBigWig.open(path) for path in paths]shared = [name for name in handles[0].chroms() ifall(name in h.chroms() for h in handles[1:])]columns = []for handle in handles: parts = []for name in shared: whole = handle.chroms()[name] // sizeif whole >0: parts.append(np.array(handle.stats(name, 0, whole * size, nBins=whole, exact=True), dtype=float)) columns.append(np.concatenate(parts))measure = {'pearson': stats.pearsonr, 'spearman': stats.spearmanr}[method]every = np.all([np.isfinite(column) for column in columns], axis=0)n =len(paths)matrix = np.eye(n)for a inrange(n):for b inrange(a +1, n): matrix[a, b] = matrix[b, a] = measure(columns[a][every], columns[b][every])[0]names = [os.path.basename(path) for path in paths]withopen(out, 'w') as handle: handle.write('\t'+'\t'.join(names) +'\n')for name, row inzip(names, matrix): handle.write(name +'\t'+'\t'.join(f'{v:.10g}'for v in row) +'\n')
Loop calling, notebooks/scripts/cooltools_dots.py
"""Loops with cooltools.dots, as its Python API calls them by default (validation.qmd § 19).The kernel is the one cooltools recommends for the map's bins, the expected `expected_cis` with itsdefault two diagonals left out, and every other setting `dots()`'s own: 10 Mb from the diagonal, 40λ-chunks, an FDR of 0.1, a 20 kb clustering radius and the HiCCUPS filter. The view is thechromosomes named, whole, in the cooler's own order, or every one of them.usage: python cooltools_dots.py MAP_URI OUT_TSV NPROC [CHROM ...]"""import sysimport coolerimport cooltoolsimport pandas as pduri, out, nproc, wanted = sys.argv[1], sys.argv[2], int(sys.argv[3]), set(sys.argv[4:])clr = cooler.Cooler(uri)chroms = [name for name in clr.chromnames ifnot wanted or name in wanted]view = pd.DataFrame({'chrom': chroms, 'start': 0, 'end': [int(clr.chromsizes[c]) for c in chroms], 'name': chroms})expected = cooltools.expected_cis(clr, view_df=view, ignore_diags=2, clr_weight_name='weight', nproc=nproc)dots = cooltools.dots(clr, expected, view_df=view, clr_weight_name='weight', nproc=nproc)dots.to_csv(out, sep='\t', index=False)
Cis and trans per sequence, notebooks/scripts/cooltools_cistrans.py
"""Contacts within each sequence and between every pair with cooltools (validation.qmd § 20).Within a sequence, `expected_cis` summed over every one of its diagonals, the main one included(`ignore_diags=0`); between two, `expected_trans`. Raw counts, or balanced by the weight columnnamed. One row per pair of sequences, each sequence with itself among them: its two names and thecontacts.usage: python cooltools_cistrans.py MAP_URI OUT_TSV NPROC [WEIGHT]"""import sysimport coolerimport cooltoolsimport pandas as pduri, out, nproc = sys.argv[1], sys.argv[2], int(sys.argv[3])weight = sys.argv[4] iflen(sys.argv) >4elseNonecolumn ='count.sum'if weight isNoneelse'balanced.sum'clr = cooler.Cooler(uri)cis = cooltools.expected_cis(clr, ignore_diags=0, smooth=False, aggregate_smoothed=False, clr_weight_name=weight, nproc=nproc)within = cis.groupby(['region1', 'region2'], observed=True, sort=False)[column].sum().reset_index()between = cooltools.expected_trans(clr, clr_weight_name=weight, nproc=nproc)[['region1', 'region2', column]]table = pd.concat([within, between], ignore_index=True).rename(columns={column: 'contacts'})table.to_csv(out, sep='\t', index=False)
Saddle, notebooks/scripts/cooltools_saddle.py
"""The saddle of each chromosome with cooltools (validation.qmd § 21).cooltools' `expected_cis` — `balanced.avg`, every diagonal from the third (`ignore_diags=2`), notsmoothed — and its `saddle`, a chromosome at a time, `contact_type='cis'`, over the track given: thetrack cut by cooltools' `digitize` into N groups at its quantiles over the chromosome(`qrange=(0, 1)`), the bin holding the largest value in the last, and handed to `saddle` digitized,the first two diagonals left out (`min_diag=2`). Writes each chromosome's N × N meanobserved/expected, as CHROM, to one NPZ.usage: python cooltools_saddle.py MAP_URI TRACK_TSV N OUT_NPZ CHROM[,CHROM...]TRACK_TSV holds chrom, start, end and the value, one row per bin of the map, with a header."""import sysimport coolerimport cooltoolsimport numpy as npimport pandas as pdfrom cooltools.api.saddle import digitizefrom cooltools.lib.common import align_track_with_cooleruri, track_path, n, out, named = sys.argv[1], sys.argv[2], int(sys.argv[3]), sys.argv[4], sys.argv[5].split(',')clr = cooler.Cooler(uri)track = pd.read_csv(track_path, sep='\t')view = pd.DataFrame({'chrom': clr.chromnames, 'start': 0, 'end': clr.chromsizes.to_numpy(), 'name': clr.chromnames})expected = cooltools.expected_cis(clr, view_df=view, ignore_diags=2, smooth=False, aggregate_smoothed=False)saddles = {}for chrom in named: region = view[view['chrom'] == chrom].reset_index(drop=True) decay = expected[expected['region1'] == chrom].reset_index(drop=True) own = track[track['chrom'] == chrom].reset_index(drop=True) aligned = align_track_with_cooler(own, clr, view_df=region, clr_weight_name='weight', mask_clr_bad_bins=True) digitized, edges = digitize(aligned.iloc[:, :4], n, qrange=(0.0, 1.0)) column = digitized.columns[3]# The bin holding the largest value in group N, where `digitize` says it puts it. codes = digitized[column].astype(int).to_numpy() codes[(aligned.iloc[:, 3] == edges[-1]).to_numpy()] = n digitized[column] = pd.Categorical(codes, categories=digitized[column].cat.categories, ordered=True) total, count = cooltools.saddle(clr, decay, digitized, 'cis', n_bins=None, view_df=region, min_diag=2)with np.errstate(invalid='ignore', divide='ignore'): saddles[chrom] = (total / count)[1:-1, 1:-1]np.savez(out, **saddles)
Fraction of signal in features, notebooks/scripts/pybigwig_frip.py
"""Fraction of signal in features from pyBigWig's exact sums (validation.qmd § 22).A track's signal over an interval is the sum, base by base, of the value it has there —pyBigWig's `stats(..., type='sum', exact=True)` — and its whole signal is that sum over everysequence its header declares. The features are merged where they overlap or touch, as `bedtoolsmerge` merges them, before the FRiP is taken: the signal inside them over the whole, beside the shareof the sequences' bases they cover and the share of their own bases the track has a value for. Writesone row a track to OUT_PREFIX-frip.tsv, and one a feature, unmerged as the BED gives them, with itssignal and its mean over the bases the track has a value for, to OUT_PREFIX-features.tsv.usage: python pybigwig_frip.py FEATURES_BED OUT_PREFIX TRACK [TRACK ...]"""import osimport sysimport numpy as npimport pandas as pdimport pyBigWigfeatures_path, prefix, paths = sys.argv[1], sys.argv[2], sys.argv[3:]features = pd.read_csv(features_path, sep='\t', header=None, comment='#', usecols=[0, 1, 2], names=['chrom', 'start', 'end'])def merged(frame):"""The intervals merged where they overlap or touch, sequence by sequence.""" runs = []for chrom, rows in frame[frame['end'] > frame['start']].sort_values(['chrom', 'start']).groupby('chrom', sort=False): start, end =None, Nonefor a, b inzip(rows['start'], rows['end']):if start isnotNoneand a <= end: end =max(end, b)continueif start isnotNone: runs.append((chrom, start, end)) start, end = a, b runs.append((chrom, start, end))return pd.DataFrame(runs, columns=['chrom', 'start', 'end'])runs = merged(features)totals, each = [], []for path in paths: handle = pyBigWig.open(path) sizes = handle.chroms()def stat(chrom, start, end, kind): value = handle.stats(chrom, int(start), int(end), type=kind, exact=True)[0]return np.nan if value isNoneelse value total =sum(stat(chrom, 0, length, 'sum') for chrom, length in sizes.items()) held = runs[runs['chrom'].isin(sizes)] widths = (held['end'] - held['start']).to_numpy() inside =sum(stat(*run, 'sum') for run in held.itertuples(index=False)) with_data =sum(stat(*run, 'coverage') * width for run, width inzip(held.itertuples(index=False), widths)) covered = widths.sum() /sum(sizes.values()) name = os.path.basename(path) totals.append({'track': name, 'frip': inside / total, 'covered': covered, 'enrichment': inside / total / covered,'signal': inside, 'totalSignal': total, 'inFeature': with_data / widths.sum()})for chrom, start, end in features.itertuples(index=False):if chrom notin sizes:continue signal = stat(chrom, start, end, 'sum') each.append({'track': name, 'contig': chrom, 'start': start, 'end': end,'signal': 0.0if np.isnan(signal) else signal, 'mean': stat(chrom, start, end, 'mean'),'share': (0.0if np.isnan(signal) else signal) / total}) handle.close()pd.DataFrame(totals).to_csv(f'{prefix}-frip.tsv', sep='\t', index=False, float_format='%.10g')pd.DataFrame(each).to_csv(f'{prefix}-features.tsv', sep='\t', index=False, float_format='%.10g')
Signal by group, notebooks/scripts/pybigwig_groupsignal.py
"""Signal by group from pyBigWig's exact means and SciPy's rank tests (validation.qmd § 23).The groups are the names a BED writes, in their natural order, and the rest of the sequences thetracks' headers declare: each name's intervals merged where they overlap or touch, what none of themcovers a group of its own, and every run cut into stretches on the bin grid as chromosomix cuts them(§18.8) — one stretch where it is a bin long or shorter, else at each multiple of the bin, a piece ofless than half a bin at either end joined to the one beside it. A BINSIZE of 0 cuts nothing, aschromosomix's default does not: each merged run of a name, and each stretch of the rest between two,is one entry. A track's value on a stretch ispyBigWig's exact mean over the bases it has a value for, `stats(..., exact=True)`. Each group issummarised by NumPy (the SD over n − 1, the quartiles interpolated), and the groups are compared by`scipy.stats.kruskal` over all of them and `scipy.stats.mannwhitneyu` pair by pair — two-sided,asymptotic, with the continuity correction — every P adjusted with every other of the run by`scipy.stats.false_discovery_control` (Benjamini–Hochberg). Writes OUT_PREFIX-stretches.tsv, a row astretch with its group and each track's mean; OUT_PREFIX-groups.tsv; and OUT_PREFIX-tests.tsv.usage: python pybigwig_groupsignal.py FEATURES_BED BINSIZE OUT_PREFIX TRACK [TRACK ...]"""import osimport reimport sysimport numpy as npimport pandas as pdimport pyBigWigfrom scipy import statsfeatures_path, binsize, prefix, paths = sys.argv[1], int(sys.argv[2]), sys.argv[3], sys.argv[4:]REST ='rest of the genome'handles = [pyBigWig.open(path) for path in paths]lengths = {}for handle in handles:for chrom, length in handle.chroms().items(): lengths.setdefault(chrom, length)features = pd.read_csv(features_path, sep='\t', header=None, comment='#', usecols=[0, 1, 2, 3], names=['chrom', 'start', 'end', 'name'], keep_default_na=False, dtype={'name': str})features['name'] = features['name'].replace('', '(no name)')def natural(text):return [int(part) if part.isdigit() else part.casefold() for part in re.split(r'(\d+)', text)]def merged(intervals): out = []for start, end insorted(intervals):if out and start <= out[-1][1]: out[-1][1] =max(out[-1][1], end)else: out.append([start, end])return outdef cut(start, end):ifnot end > start:return []if binsize <=0or end - start <= binsize:return [[start, end]] pieces, at = [], startwhile at < end: to =min(end, (at // binsize +1) * binsize) pieces.append([at, to]) at = toiflen(pieces) >1and pieces[0][1] - pieces[0][0] < binsize /2: pieces[1][0] = pieces[0][0] pieces.pop(0)iflen(pieces) >1and pieces[-1][1] - pieces[-1][0] < binsize /2: pieces[-2][1] = pieces[-1][1] pieces.pop()return piecesheld = features[features['chrom'].isin(lengths)].copy()held['start'] = held['start'].clip(lower=0)held['end'] = np.minimum(held['end'], held['chrom'].map(lengths))held = held[held['end'] > held['start']]names =sorted(held['name'].unique(), key=natural) + [REST]stretches = []for chrom, length in lengths.items(): here = held[held['chrom'] == chrom]for group, name inenumerate(names[:-1]): rows = here[here['name'] == name]for start, end in merged(zip(rows['start'], rows['end'])): stretches += [(chrom, a, b, group) for a, b in cut(start, end)] at =0for start, end in merged(zip(here['start'], here['end'])): stretches += [(chrom, a, b, len(names) -1) for a, b in cut(at, start)] at =max(at, end) stretches += [(chrom, a, b, len(names) -1) for a, b in cut(at, length)]table = pd.DataFrame(stretches, columns=['chrom', 'start', 'end', 'group'])# A group no stretch fell in is left out, as chromosomix leaves it out.present = [g for g inrange(len(names)) if (table['group'] == g).any()]table['group'] = table['group'].map({g: names[g] for g in present})names = [names[g] for g in present]summaries, tests = [], []for path, handle inzip(paths, handles): track = os.path.basename(path) means = [handle.stats(chrom, int(start), int(end), type='mean', exact=True)[0]for chrom, start, end in table[['chrom', 'start', 'end']].itertuples(index=False)] table[track] = np.array([np.nan if value isNoneelse value for value in means], dtype=float) groups = [table.loc[(table['group'] == name) & table[track].notna(), track].to_numpy() for name in names] every = np.concatenate(groups)for name, values inzip(names, groups): others =len(every) -len(values) rest = (every.sum() - values.sum()) / others if others else np.nan q1, median, q3 = np.quantile(values, [0.25, 0.5, 0.75]) iflen(values) else (np.nan,) *3 summaries.append({'track': track, 'group': name, 'n': len(values),'bases': int((table.loc[table['group'] == name, 'end'] - table.loc[table['group'] == name, 'start']).sum()),'mean': values.mean() iflen(values) else np.nan,'sd': values.std(ddof=1) iflen(values) >1else np.nan,'median': median, 'q1': q1, 'q3': q3,'vsRest': values.mean() - rest iflen(values) else np.nan}) kept = [(name, values) for name, values inzip(names, groups) iflen(values)]iflen(kept) >2: h = stats.kruskal(*[values for _, values in kept]) tests.append({'track': track, 'test': 'Kruskal–Wallis H', 'a': f'all {len(kept)} groups', 'b': '','difference': np.nan, 'effect': np.nan, 'statistic': h.statistic, 'p': h.pvalue})for i inrange(len(kept)):for j inrange(i +1, len(kept)): (a, x), (b, y) = kept[i], kept[j] u = stats.mannwhitneyu(x, y, alternative='two-sided', method='asymptotic', use_continuity=True) tests.append({'track': track, 'test': 'Mann–Whitney U', 'a': a, 'b': b,'difference': np.median(x) - np.median(y), 'effect': 2* u.statistic / (len(x) *len(y)) -1,'statistic': u.statistic, 'p': u.pvalue})tests = pd.DataFrame(tests)tests['q'] = stats.false_discovery_control(tests['p'].to_numpy(), method='bh')table.to_csv(f'{prefix}-stretches.tsv', sep='\t', index=False, float_format='%.10g')pd.DataFrame(summaries).to_csv(f'{prefix}-groups.tsv', sep='\t', index=False, float_format='%.10g')tests.to_csv(f'{prefix}-tests.tsv', sep='\t', index=False, float_format='%.10g')
CTCF sites in GM12878 (Fig. 2), notebooks/scripts/cooltools_insulation_scales.py
"""Diamond insulation and boundaries with cooltools at several windows at once (validation.qmd § 24).usage: python cooltools_insulation_scales.py MAP_URI OUT_TSV NPROC WINDOW_BP [WINDOW_BP ...]"""import sysimport coolerimport cooltoolsuri, out, nproc = sys.argv[1], sys.argv[2], int(sys.argv[3])windows = [int(w) for w in sys.argv[4:]]table = cooltools.insulation(cooler.Cooler(uri), windows, ignore_diags=2, clr_weight_name='weight', min_frac_valid_pixels=0.5, threshold=0.2, nproc=nproc)table.to_csv(out, sep='\t', index=False)
"""A reference's restriction sites with bioframe and Biopython (validation.qmd § 26).`bioframe.digest` cuts each sequence of the FASTA where Biopython's `Bio.Restriction` finds theenzyme. Its boundaries are 2 bp past the cut: Biopython's `search` gives the 1-based position of thefirst base after the cut, and bioframe adds 1 to it rather than taking 1 away. SHIFT (-2 by default)brings them back to c = s + k, the 0-based index of the first base after the top strand's cut, for asite whose first base is s and which is cut k bases in: the cut as chromosomix writes it.Each cut has a place on the bottom strand too, c' = s + L - k for a site of L bases, L - 2k past c:where a read on the - strand starts, its 5' end being the bottom strand's, once the overhang theenzyme leaves is filled in (for DpnII, ^GATC, c = s and c' = s + 4). A + end is placed among thefragments by the top strand's cuts and a - end by the bottom strand's; fragment i is the samefragment on both.Writes, beside OUT_PREFIX: .bioframe.bed the fragments as bioframe.digest gives them (chrom, start, end), untouched; .frags.bed the fragments between the top strand's cuts, for `pairtools restrict --frags`; .frags-bottom.bed the same fragments between the bottom strand's; .cuts.tsv every cut, a row each (chrom, cut, bottom): c and c', the cuts whose c is at 0 or at the sequence's end dropped; .sites.tsv the cuts in each bin of BINSIZE along each sequence, the last bin short (chrom, start, end, sites): a cut is in the bin start <= c < end.usage: python bioframe_digest.py FASTA ENZYME BINSIZE OUT_PREFIX [SHIFT]"""import sysimport bioframeimport numpy as npimport pandas as pdfrom Bio import Restrictionfasta, enzyme, binsize, prefix = sys.argv[1], sys.argv[2], int(sys.argv[3]), sys.argv[4]shift =int(sys.argv[5]) iflen(sys.argv) >5else-2# The site's length and where in it the top strand is cut: c' is L - 2k past c.site =getattr(Restriction, enzyme)apart = site.size -2* site.fst5genome = bioframe.load_fasta(fasta, engine='pyfaidx')raw = bioframe.digest(genome, enzyme)raw.to_csv(f'{prefix}.bioframe.bed', sep='\t', header=False, index=False)cuts, top, bottom, sites = [], [], [], []for chrom in genome: length =len(genome[chrom])# bioframe's boundaries are [0, every site, the length]: the sites are the ones in between. ends = raw.loc[raw['chrom'] == chrom, 'end'].to_numpy(np.int64)[:-1] + shift at = np.unique(ends[(ends >0) & (ends < length)]) cuts.append(pd.DataFrame({'chrom': chrom, 'cut': at, 'bottom': at + apart}))for frags, bounds in ((top, np.r_[0, at, length]), (bottom, np.r_[0, at + apart, length])): frags.append(pd.DataFrame({'chrom': chrom, 'start': bounds[:-1], 'end': bounds[1:]})) starts = np.arange(0, length, binsize) sites.append(pd.DataFrame({'chrom': chrom, 'start': starts, 'end': np.minimum(starts + binsize, length),'sites': np.bincount(at // binsize, minlength=len(starts))}))pd.concat(cuts).to_csv(f'{prefix}.cuts.tsv', sep='\t', index=False)pd.concat(top).to_csv(f'{prefix}.frags.bed', sep='\t', header=False, index=False)pd.concat(bottom).to_csv(f'{prefix}.frags-bottom.bed', sep='\t', header=False, index=False)pd.concat(sites).to_csv(f'{prefix}.sites.tsv', sep='\t', index=False)
"""Restriction-fragment statistics of a contact list, from `pairtools restrict`'s columns (validation.qmd § 27).The fragments are `bioframe.digest`'s, as bioframe_digest.py writes them: between the top strand'scuts, c, its boundaries moved by SHIFT (-2 by default), and between the bottom strand's, c', L - 2kpast them. `pairtools restrict` was run once on each, over the same contact list, and has put eachend of each pair in a fragment (`rfrag1`, `rfrag2`, 0-based along its sequence) and written thatfragment's ends, 1-based (`rfrag_start`, `rfrag_end`: the fragment's first base, 0 for a sequence'sfirst fragment, and the first base of the next). A + end is read from the run on the top strand'sfragments and a - end from the run on the bottom strand's: a - read's 5' end is the bottom strand's,which once the enzyme's overhang is filled in ends at c' - 1. From those columns alone, read inchunks, this counts what chromosomix's Pairs QC counts, by the definitions it states:- a pair counts when both ends are mapped (the chromosome not `!`, the strand `+` or `-`), both sequences are the reference's and it is not a duplicate (`pair_type` DD); a mapped pair that is no duplicate, with an end on a sequence the reference lacks, is `unassigned`;- `trans` across sequences; within one, the two ends ordered by x = pos - 1 (the file's order on a tie), a then b, in fragments fa and fb: `dangling` in one fragment as (+, -), `self-circle` as (-, +), `same-strand` as (+, +) or (-, -); `re-ligation` in neighbouring fragments as (+, -); `valid` otherwise; each class by the strands of (a, b) too, a trans pair's in the file's order;- the distance from each end of a counted pair to the cut it reads towards, on its own strand's boundaries, B for + and B' for -: B[i+1] - x on +, x - B'[i] + 1 on -, counted on pairtools stats' separation bins (eight a decade), a value d in the bin searchsorted(edges, d, 'right') - 1, each strand apart;- the ends of counted pairs in each fragment, the fragments seen (with one end or more) of all the reference's, and how many fragments have 0, 1, 2, 3-4, 5-8, 9-16, ... ends.Writes, beside OUT_PREFIX: .summary.tsv measure, value: records read, counted, unassigned, not counted, cis, trans, fragments, fragments seen, cuts, and the ends whose fragment pairtools placed off the digest's boundaries for their strand (0 when the two were given the same); and, what a read starting where the enzyme cut looks like, the + ends on a cut (x = c) and a base past one (c + 1), and the - ends on a cut (x = c' - 1, the bottom strand's first base) and a base before one (c' - 2); .classes.tsv class, pairs, fraction of the counted, and the pairs it would have had with every end placed by the top strand's cuts (`top_strand_only`), as one run of `pairtools restrict` places them; .orientation.tsv class, orientation, pairs: every class by each of ++ +- -+ --; .distance.tsv strand, start, end, ends: each separation bin [start, end), the last open; .fragments.tsv start, end, label, fragments: fragments with start to end ends, inclusive; .ends.tsv chrom, start, end, ends: every fragment, 0-based and half-open between the top strand's cuts, and its ends.usage: python restriction_reference.py FASTA ENZYME TOP_PAIRS BOTTOM_PAIRS OUT_PREFIX [SHIFT] (TOP_PAIRS and BOTTOM_PAIRS: `pairtools restrict`'s output on each strand's fragments)"""import csvimport gzipimport sysimport bioframeimport numpy as npimport pandas as pdfrom Bio import Restrictionfasta, enzyme, top_pairs, bottom_pairs, prefix = sys.argv[1:6]shift =int(sys.argv[6]) iflen(sys.argv) >6else-2# The fragments on both strands, as bioframe_digest.py writes them for pairtools restrict.site =getattr(Restriction, enzyme)apart = site.size -2* site.fst5genome = bioframe.load_fasta(fasta, engine='pyfaidx')digest = bioframe.digest(genome, enzyme)TOP, BOTTOM = {}, {}for chrom in genome: length =len(genome[chrom]) ends = digest.loc[digest['chrom'] == chrom, 'end'].to_numpy(np.int64)[:-1] + shift at = np.unique(ends[(ends >0) & (ends < length)]) TOP[chrom], BOTTOM[chrom] = np.r_[0, at, length], np.r_[0, at + apart, length]CHROMS =list(TOP)#: Each sequence's first fragment in one numbering of every fragment, sequence after sequence.OFFSET = np.cumsum([0] + [len(b) -1for b in TOP.values()])TOTAL =int(OFFSET[-1])#: Each fragment's two ends, by that numbering, on each strand.STARTS = {'+': np.concatenate([b[:-1] for b in TOP.values()]), '-': np.concatenate([b[:-1] for b in BOTTOM.values()])}ENDS = {'+': np.concatenate([b[1:] for b in TOP.values()]), '-': np.concatenate([b[1:] for b in BOTTOM.values()])}#: Whether a fragment starts at a cut, and ends at one, rather than at its sequence's ends.CUT_BEFORE, CUT_AFTER = np.ones(TOTAL, bool), np.ones(TOTAL, bool)CUT_BEFORE[OFFSET[:-1]], CUT_AFTER[OFFSET[1:] -1] =False, False# pairtools stats' separation bins: 0, then 10 ** (k / 8) rounded, to 10⁹.EDGES = np.unique(np.r_[0, np.round(10** np.arange(0, 9.001, 0.125)).astype(np.int64)])CLASSES = ['trans', 'dangling', 'self-circle', 'same-strand', 're-ligation', 'valid']ORIENTATIONS = ['++', '+-', '-+', '--']FRAGMENT = ['rfrag', 'rfrag_start', 'rfrag_end']def chunks(path, wanted):"""`pairtools restrict`'s output, the columns `wanted`, a million records at a time. Nothing in a .pairs is quoted, and a quality string may hold a quote."""with (gzip.openif path.endswith('.gz') elseopen)(path, 'rt') as handle: header =0for line in handle:ifnot line.startswith('#'):break header +=1if line.startswith('#columns:'): columns = line.split()[1:] wanted = [c for c in wanted if c in columns] types = {c: (np.int64 if c.startswith(('pos', 'rfrag')) elsestr) for c in wanted}return columns, pd.read_csv(path, sep='\t', header=None, names=columns, skiprows=header, quoting=csv.QUOTE_NONE, usecols=wanted, dtype=types, chunksize=1_000_000)PLACED = [f'{c}{k}'for k in'12'for c in FRAGMENT]columns, tops = chunks(top_pairs, ['chrom1', 'pos1', 'chrom2', 'pos2', 'strand1', 'strand2', 'pair_type', *PLACED])TYPED ='pair_type'in columns_, bottoms = chunks(bottom_pairs, ['pos1', 'pos2', *PLACED])read = counted = unassigned = cis = off =0#: The ends on a cut, and a base from one, by strand: see the docstring.at_cut = {'+ ends on a cut': 0, '+ ends a base past a cut': 0, '- ends on a cut': 0, '- ends a base before a cut': 0}by_class = np.zeros(len(CLASSES) *4, np.int64)top_only = np.zeros(len(CLASSES), np.int64)distance = {'+': np.zeros(len(EDGES), np.int64), '-': np.zeros(len(EDGES), np.int64)}ends = np.zeros(TOTAL, np.int64)for chunk, below inzip(tops, bottoms):ifnot (chunk['pos1'].equals(below['pos1']) and chunk['pos2'].equals(below['pos2'])):raiseValueError(f'{top_pairs} and {bottom_pairs} do not hold the same records in the same order') read +=len(chunk) c1, c2 = chunk['chrom1'].to_numpy(str), chunk['chrom2'].to_numpy(str) s1, s2 = chunk['strand1'].to_numpy(str), chunk['strand2'].to_numpy(str) mapped = (c1 !='!') & (c2 !='!') & np.isin(s1, ['+', '-']) & np.isin(s2, ['+', '-']) unique = chunk['pair_type'].to_numpy(str) !='DD'if TYPED else np.ones(len(chunk), bool) known = np.isin(c1, CHROMS) & np.isin(c2, CHROMS) unassigned +=int((mapped & unique &~known).sum()) keep = mapped & unique & known counted +=int(keep.sum()) c1, c2, s1, s2 = c1[keep], c2[keep], s1[keep], s2[keep]def placed(k, chrom, strand):"""End k's fragment, numbered across sequences, read from the run on its strand's fragments, and whether pairtools' ends of it are not the digest's.""" f = np.where(strand =='+', chunk[f'rfrag{k}'].to_numpy()[keep], below[f'rfrag{k}'].to_numpy()[keep]) g = OFFSET[pd.Categorical(chrom, categories=CHROMS).codes] + f lo = np.maximum(np.where(strand =='+', chunk[f'rfrag_start{k}'].to_numpy()[keep], below[f'rfrag_start{k}'].to_numpy()[keep]) -1, 0) hi = np.where(strand =='+', chunk[f'rfrag_end{k}'].to_numpy()[keep], below[f'rfrag_end{k}'].to_numpy()[keep]) -1 wrong = (lo != np.where(strand =='+', STARTS['+'][g], STARTS['-'][g])) |\ (hi != np.where(strand =='+', ENDS['+'][g], ENDS['-'][g]))return f, g, int(wrong.sum()) (f1, g1, off1), (f2, g2, off2) = placed(1, c1, s1), placed(2, c2, s2) off += off1 + off2 x1, x2 = chunk['pos1'].to_numpy()[keep] -1, chunk['pos2'].to_numpy()[keep] -1 trans = c1 != c2 cis +=int((~trans).sum()) swap = x2 < x1 sa, sb = np.where(swap, s2, s1), np.where(swap, s1, s2)# A trans pair's orientation is its ends' strands in the file's order. sa, sb = np.where(trans, s1, sa), np.where(trans, s2, sb) plus_minus, minus_plus = (sa =='+') & (sb =='-'), (sa =='-') & (sb =='+')def classes(f1, f2):"""Each pair's class, its ends in fragments f1 and f2.""" fa, fb = np.where(swap, f2, f1), np.where(swap, f1, f2) same = fa == fb kind = np.full(len(c1), CLASSES.index('valid')) kind[~trans & (np.abs(fb - fa) ==1) & plus_minus] = CLASSES.index('re-ligation') kind[~trans & same & (sa == sb)] = CLASSES.index('same-strand') kind[~trans & same & minus_plus] = CLASSES.index('self-circle') kind[~trans & same & plus_minus] = CLASSES.index('dangling') kind[trans] = CLASSES.index('trans')return kind orientation = (sa =='-') *2+ (sb =='-') by_class += np.bincount(classes(f1, f2) *4+ orientation, minlength=len(by_class))# And had every end been placed by the top strand's cuts, as pairtools restrict places any end. top_only += np.bincount(classes(chunk['rfrag1'].to_numpy()[keep], chunk['rfrag2'].to_numpy()[keep]), minlength=len(CLASSES))for x, g, s in ((x1, g1, s1), (x2, g2, s2)): d = np.where(s =='+', ENDS['+'][g] - x, x - STARTS['-'][g] +1)for strand in'+-': distance[strand] += np.bincount(np.searchsorted(EDGES, d[s == strand], 'right') -1, minlength=len(EDGES)) ends += np.bincount(g, minlength=TOTAL) plus, minus = (s =='+') & CUT_BEFORE[g], (s =='-') & CUT_AFTER[g] at_cut['+ ends on a cut'] +=int((plus & (x == STARTS['+'][g])).sum()) at_cut['+ ends a base past a cut'] +=int((plus & (x == STARTS['+'][g] +1)).sum()) at_cut['- ends on a cut'] +=int((minus & (x == ENDS['-'][g] -1)).sum()) at_cut['- ends a base before a cut'] +=int((minus & (x == ENDS['-'][g] -2)).sum())counts = by_class.reshape(len(CLASSES), 4)seen =int((ends >0).sum())pd.DataFrame([('records', read), ('counted', counted), ('unassigned', unassigned), ('not counted', read - counted - unassigned), ('cis', cis), ('trans', counted - cis), ('fragments', TOTAL), ('fragments seen', seen), ('cuts', TOTAL -len(TOP)), ('ends off the digest', off), *at_cut.items()], columns=['measure', 'value']).to_csv(f'{prefix}.summary.tsv', sep='\t', index=False)pd.DataFrame({'class': CLASSES, 'pairs': counts.sum(1), 'fraction': counts.sum(1) /max(counted, 1),'top_strand_only': top_only}).to_csv(f'{prefix}.classes.tsv', sep='\t', index=False)pd.DataFrame([(name, o, counts[k, j]) for k, name inenumerate(CLASSES) for j, o inenumerate(ORIENTATIONS)], columns=['class', 'orientation', 'pairs']).to_csv(f'{prefix}.orientation.tsv', sep='\t', index=False)pd.DataFrame([(strand, EDGES[k], EDGES[k +1] if k +1<len(EDGES) else pd.NA, distance[strand][k])for strand in'+-'for k inrange(len(EDGES))], columns=['strand', 'start', 'end', 'ends']).to_csv(f'{prefix}.distance.tsv', sep='\t', index=False)# 0, 1, 2, 3-4, 5-8, 9-16, ...: a bin of its own for 0 and for 1, then up to each power of two.most =int(ends.max()) if TOTAL else0powers =max(1, int(np.ceil(np.log2(max(most, 2)))))spans = [(0, 0), (1, 1)] + [(2** (k -1) +1, 2** k) for k inrange(1, powers +1)]pd.DataFrame([(lo, hi, f'{lo}'if lo == hi elsef'{lo}–{hi}', int(((ends >= lo) & (ends <= hi)).sum()))for lo, hi in spans], columns=['start', 'end', 'label', 'fragments']).to_csv(f'{prefix}.fragments.tsv', sep='\t', index=False)pd.DataFrame({'chrom': np.repeat(CHROMS, np.diff(OFFSET)), 'start': STARTS['+'], 'end': ENDS['+'],'ends': ends}).to_csv(f'{prefix}.ends.tsv', sep='\t', index=False)
"""Coverage against restriction sites, bin by bin (validation.qmd § 27).Each bin's raw total coverage, cis and trans with every diagonal kept, as `cooltools.coverage` givesit, beside the cuts in the bin, as bioframe_digest.py counts them (a cut c in the binstart <= c < end). cooltools adds a pixel at both its ends, so a bin's own pixel on the diagonal iscounted twice; chromosomix counts it once (validation.qmd § 9), and the coverage set against thecuts here is cooltools' total less that pixel. A bin with neither coverage nor a cut is left out ofthe correlation and of the means.Writes, beside OUT_PREFIX: .coverage.tsv chrom, start, end, total, diagonal, coverage, sites: every bin of the map, `total` cooltools' as it gives it, `diagonal` the bin's own pixel, `coverage` the one less the other; .means.tsv sites, bins, coverage: the mean coverage of the bins kept with that many cuts; .summary.tsv measure, value: the bins, those kept, and Spearman's rho over them (SciPy).usage: python cooltools_coverage_sites.py MAP_URI SITES_TSV OUT_PREFIX NPROC"""import sysimport coolerimport cooltoolsimport numpy as npimport pandas as pdfrom scipy import statsuri, sites_tsv, prefix, nproc = sys.argv[1], sys.argv[2], sys.argv[3], int(sys.argv[4])clr = cooler.Cooler(uri)_, total = cooltools.coverage(clr, ignore_diags=0, nproc=nproc)table = clr.bins()[['chrom', 'start', 'end']][:]table['chrom'] = table['chrom'].astype(str)table['total'] = totalpixels = clr.pixels()[:]own = pixels[pixels['bin1_id'] == pixels['bin2_id']]table['diagonal'] = np.bincount(own['bin1_id'], weights=own['count'], minlength=len(table)).astype(np.int64)table['coverage'] = table['total'] - table['diagonal']sites = pd.read_csv(sites_tsv, sep='\t', dtype={'chrom': str})table = table.merge(sites[['chrom', 'start', 'end', 'sites']], on=['chrom', 'start', 'end'], how='left', validate='one_to_one')if table['sites'].isna().any():raiseValueError(f'{sites_tsv} has no count for {int(table["sites"].isna().sum())} bins of {uri}')table['sites'] = table['sites'].astype(int)table.to_csv(f'{prefix}.coverage.tsv', sep='\t', index=False)kept = table[(table['coverage'] >0) | (table['sites'] >0)](kept.groupby('sites').agg(bins=('coverage', 'size'), coverage=('coverage', 'mean')).reset_index() .to_csv(f'{prefix}.means.tsv', sep='\t', index=False))rho = stats.spearmanr(kept['sites'], kept['coverage']).statisticpd.DataFrame([('bins', len(table)), ('kept', len(kept)), ('spearman', float(rho))], columns=['measure', 'value'], dtype=object).to_csv(f'{prefix}.summary.tsv', sep='\t', index=False)
Map report, notebooks/scripts/map_resolution.py
"""The map report's resolution ladder, from cooler's own rungs (validation.qmd § 29).At every resolution an .mcool stores, each bin's contact ends — a pixel counted once at each of itstwo bins, so a contact within one bin twice for it, as Juicer's calculate_map_resolution.sh counts —summed from that rung's own pixel table with numpy, and what the map report says of them: how manybins, the mean and the median, the count at least 80 % of bins reach (the order statistic atfloor(0.2 · bins)), and the share of bins that are empty and that hold under a tenth of the median.Each rung is read as `cooler zoomify` wrote it, so a row is the file's coarsening and not a sum theapp made; every bin of every sequence counts, a sequence's short last bin included.usage: python map_resolution.py MCOOL OUT_TSV"""import sysimport coolerimport numpy as npimport pandas as pdpath, out = sys.argv[1], sys.argv[2]rows = []for uri in cooler.fileops.list_coolers(path): clr = cooler.Cooler(f'{path}::{uri}') ends = np.zeros(clr.info['nbins'])for lo inrange(0, clr.info['nnz'], 10_000_000): chunk = clr.pixels()[lo:lo +10_000_000] np.add.at(ends, chunk['bin1_id'].to_numpy(), chunk['count'].to_numpy(dtype=float)) np.add.at(ends, chunk['bin2_id'].to_numpy(), chunk['count'].to_numpy(dtype=float)) values = np.sort(ends) n =len(values) median =float(np.median(values)) rows.append({'binsize': clr.binsize,'bins': n,'mean': float(values.mean()),'median': median,'reached80': float(values[int(np.floor(0.2* n))]),'empty': float((values ==0).mean()),'belowTenthMedian': float((values < median /10).mean()), })pd.DataFrame(rows).sort_values('binsize', ascending=False).to_csv(out, sep='\t', index=False)
Rendered in 54 min 40 s, from 1 November 2026, 21:32 UTC to 22:27 UTC, by chromosomix 0.19.0 at 997e59c.
