Press n or j to go to the next uncovered block, b, p or k for the previous block.
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 1660 1661 1662 1663 1664 1665 1666 1667 1668 1669 1670 1671 1672 1673 1674 1675 1676 1677 1678 1679 1680 1681 1682 1683 1684 1685 1686 1687 1688 1689 1690 1691 1692 1693 1694 1695 1696 1697 1698 1699 1700 1701 1702 1703 1704 1705 1706 1707 1708 1709 1710 1711 1712 1713 1714 1715 1716 1717 1718 1719 1720 1721 1722 1723 1724 1725 1726 1727 1728 1729 1730 1731 1732 1733 1734 1735 1736 1737 1738 1739 1740 1741 1742 1743 1744 1745 1746 1747 1748 1749 1750 1751 1752 1753 1754 1755 1756 1757 1758 1759 1760 1761 1762 1763 1764 1765 1766 1767 1768 1769 1770 1771 1772 1773 1774 1775 1776 1777 1778 1779 1780 1781 1782 1783 1784 1785 1786 1787 1788 1789 1790 1791 1792 1793 1794 1795 1796 1797 1798 1799 1800 1801 1802 1803 1804 1805 1806 1807 1808 1809 1810 1811 1812 1813 1814 1815 1816 1817 1818 1819 1820 1821 1822 1823 1824 1825 1826 1827 1828 1829 1830 1831 1832 1833 1834 1835 1836 1837 1838 1839 1840 1841 1842 1843 1844 1845 1846 1847 1848 1849 1850 1851 1852 1853 1854 1855 1856 1857 1858 1859 1860 1861 1862 1863 1864 1865 1866 1867 1868 1869 1870 1871 1872 1873 1874 1875 1876 1877 1878 1879 1880 1881 1882 1883 1884 1885 1886 1887 1888 1889 1890 1891 1892 1893 1894 1895 1896 1897 1898 1899 1900 1901 1902 1903 1904 1905 1906 1907 1908 1909 1910 1911 1912 1913 1914 1915 1916 1917 1918 1919 1920 1921 1922 1923 1924 1925 1926 1927 1928 1929 1930 1931 1932 1933 1934 1935 1936 1937 1938 1939 1940 1941 1942 1943 1944 1945 1946 1947 1948 1949 1950 1951 1952 1953 1954 1955 1956 1957 1958 1959 1960 1961 1962 1963 1964 1965 1966 1967 1968 1969 1970 1971 1972 1973 1974 1975 1976 1977 1978 1979 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025 2026 2027 2028 2029 2030 2031 2032 2033 2034 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2051 2052 2053 2054 2055 2056 2057 2058 2059 2060 2061 2062 2063 2064 2065 2066 2067 2068 2069 2070 2071 2072 2073 2074 2075 2076 2077 2078 2079 2080 2081 2082 2083 2084 2085 2086 2087 2088 2089 2090 2091 2092 2093 2094 2095 2096 2097 2098 2099 2100 2101 2102 2103 2104 2105 2106 2107 2108 2109 2110 2111 2112 2113 2114 2115 2116 2117 2118 2119 2120 2121 2122 2123 2124 2125 2126 2127 2128 2129 2130 2131 2132 2133 2134 2135 2136 2137 2138 2139 2140 2141 2142 2143 2144 2145 2146 2147 2148 2149 2150 2151 2152 2153 2154 2155 2156 2157 2158 2159 2160 2161 2162 2163 2164 2165 2166 2167 2168 2169 2170 2171 2172 2173 2174 2175 2176 2177 2178 2179 2180 2181 2182 2183 2184 2185 2186 2187 2188 2189 2190 2191 2192 2193 2194 2195 2196 2197 2198 2199 2200 2201 2202 2203 2204 2205 2206 2207 2208 2209 2210 2211 2212 2213 2214 2215 2216 2217 2218 2219 2220 2221 2222 2223 2224 2225 2226 2227 2228 2229 2230 2231 2232 2233 2234 2235 2236 2237 2238 2239 2240 2241 2242 2243 2244 2245 2246 2247 2248 2249 2250 2251 2252 2253 2254 2255 2256 2257 2258 2259 2260 2261 2262 2263 2264 2265 2266 2267 2268 2269 2270 2271 2272 2273 2274 2275 2276 2277 2278 2279 2280 2281 2282 2283 2284 2285 2286 2287 2288 2289 2290 2291 2292 2293 2294 2295 2296 | 2x 2x 2x 2x 2x 2x 2x 2x 2x 2x 2x 2x 2x 2x 2x 2x 2x 2x 2x 2x 2x 2x 2x 2x 2x 2x 2x 2x 2x 2x 2x 2x 2x 2x 2x 2x 2x 2x 2x 855x 855x 855x 855x 855x 855x 855x 855x 855x 855x 855x 855x 855x 855x 855x 1029x 1029x 1029x 855x 5855x 5855x 5855x 855x 34x 34x 34x 34x 34x 855x 855x 855x 855x 1657x 1657x 1657x 7x 7x 7x 1657x 855x 855x 855x 1903x 1903x 855x 11x 11x 855x 855x 855x 855x 855x 855x 855x 855x 855x 855x 855x 855x 855x 2x 695x 695x 695x 695x 695x 695x 691x 7x 7x 7x 7x 7x 7x 7x 7x 7x 7x 7x 582x 695x 695x 1x 1x 1x 1x 1x 1x 1x 685x 2x 2x 2x 2x 2x 2x 2x 579x 579x 579x 578x 684x 9x 9x 9x 9x 695x 2x 180x 180x 180x 180x 179x 179x 179x 179x 179x 179x 179x 179x 179x 179x 179x 179x 179x 179x 179x 180x 2x 33x 33x 33x 33x 33x 33x 33x 33x 33x 33x 33x 33x 33x 33x 33x 33x 33x 33x 33x 33x 33x 33x 33x 2x 32x 32x 32x 32x 32x 32x 32x 32x 32x 32x 96x 96x 32x 32x 32x 32x 32x 32x 32x 32x 32x 32x 32x 32x 32x 32x 2x 19x 1x 1x 1x 1x 1x 1x 18x 18x 18x 18x 18x 18x 18x 18x 18x 19x 2x 591x 591x 591x 591x 591x 586x 123x 123x 123x 123x 123x 123x 123x 123x 123x 470x 458x 458x 458x 458x 458x 458x 458x 458x 458x 458x 591x 2x 10x 10x 10x 10x 10x 2x 10x 10x 10x 10x 2x 855x 855x 855x 855x 855x 855x 855x 855x 855x 855x 855x 855x 855x 855x 855x 855x 855x 855x 855x 855x 855x 855x 855x 855x 855x 855x 855x 855x 855x 855x 2x 71x 71x 2x 396x 396x 2x 4x 4x 2x 1859x 1859x 2x 3803x 3803x 2x 4662624x 4662624x 2x 31x 31x 2x 122x 122x 2x 2370x 2370x 2x 1482x 1482x 1x 1x 1x 1x 1x 1x 1481x 1482x 2x 3485x 3485x 3485x 3485x 2x 312x 312x 312x 2x 2x 2x 2x 1531x 1531x 1531x 2x 1532x 1532x 1532x 1531x 1531x 1531x 1532x 1532x 2x 1531x 1531x 1531x 1531x 1531x 1531x 1531x 1531x 2x 3624x 3624x 3624x 3624x 2x 94x 94x 1003x 1003x 94x 94x 2x 2x 12x 12x 12x 12x 12x 12x 12x 12x 12x 12x 12x 12x 12x 12x 12x 12x 2x 248x 248x 248x 248x 248x 2x 248x 248x 2x 2x 1490x 1490x 2x 135x 135x 135x 2x 224x 224x 224x 224x 224x 2x 2x 40x 40x 40x 40x 40x 40x 40x 40x 40x 40x 40x 34x 6x 40x 40x 40x 40x 40x 40x 40x 40x 40x 40x 40x 40x 40x 40x 40x 60x 408737x 408737x 408737x 408737x 407076x 1661x 1661x 1661x 408737x 408737x 60x 40x 40x 40x 40x 40x 40x 40x 40x 40x 40x 2x 11x 11x 11x 11x 11x 11x 11x 11x 11x 11x 11x 11x 11x 11x 11x 11x 11x 11x 11x 11x 11x 11x 11x 11x 11x 11x 13x 185667x 185667x 185667x 185667x 185667x 185667x 185667x 13x 11x 11x 11x 11x 11x 11x 11x 11x 11x 11x 11x 2x 6x 6x 6x 6x 6x 6x 6x 6x 6x 6x 6x 6x 6x 6x 6x 6x 6x 6x 6x 6x 6x 2x 8x 1x 1x 8x 8x 6x 6x 6x 6x 6x 6x 6x 6x 6x 6x 6x 8x 8x 8x 8x 8x 8x 2x 4x 4x 4x 4x 4x 4x 4x 4x 4x 4x 4x 4x 4x 4x 4x 4x 4x 4x 4x 4x 4x 4x 4x 4x 4x 2x 2x 10x 10x 2x 2x 2x 2x 2x 2x 8x 8x 10x 4x 4x 4x 4x 10x 2x 5x 5x 5x 5x 5x 5x 5x 5x 2x 310x 310x 310x 310x 310x 310x 310x 310x 2x 14x 14x 2x 5x 5x 2x 5x 1x 1x 1x 1x 1x 1x 4x 4x 5x 2x 2x 4x 4x 4x 4x 4x 4x 4x 4x 4x 4x 52x 52x 52x 12x 52x 8x 52x 4x 4x 12x 12x 4x 4x 2x 25x 25x 2x 55x 55x 2x 2x 855x 855x 855x 1710x 1710x 855x 855x 855x 458x 458x 1832x 1832x 458x 458x 458x 458x 458x 855x 855x 855x 397x 458x 855x 855x 855x 855x 3023x 3023x 855x 2x 13x 13x 13x 13x 13x 1x 1x 1x 12x 12x 13x 12x 13x 13x 12x 12x 12x 12x 12x 12x 12x 12x 12x 12x 12x 12x 12x 12x 12x 12x 13x 2x 10x 10x 10x 10x 2x 9x 9x 9x 9x 9x 9x 9x 9x 9x 9x 9x 9x 9x 9x 9x 9x 2x 2x 2x 2x 2x 2x 2x 21x 21x 21x 21x 2x 746x 746x 746x 746x 2x 7x 7x 7x 7x 8x 2x 2x 2x 2x 2x 2x 2x 2x 6x 8x 8x 7x 7x 855x 855x 855x 4x 4x 4x 4x 4x 458x 458x 458x 458x 458x 458x 458x 40x 40x 40x 80x 80x 80x 80x 80x 3322x 3322x 3322x 3322x 3322x 3322x 3322x 3322x 750x 750x 578x 578x 18x 1x 1x 1x 1x 1x 1x 1x 1x 2x 581x 581x 583x 581x 583x 583x | /**
* The reader (spec §6): a read-only JS mirror of the Python `MetaCooler`.
*
* Same names, same semantics, same errors, no new concepts. Where this file
* and `metacooler/api.py` disagree, this file is wrong.
*
* What `open` reads, and nothing more (§3.2): the root attributes **once**, the
* `/genomes` and `/contigs` columns the *reader* uses, and the two small offset
* arrays. Never `/pixels/*`, never `indexes/bin1_offset` (one entry per bin —
* sliced on demand), and never `/bins` at all (§3.3).
*
* "The columns the reader uses" is a narrowing of §3.2's "all columns", and it
* is a large one: on the reference file the optional `/contigs` metadata is 72%
* of the whole open's HDF5 bytes, for a table view the user may never open.
* {@link MetaCooler.fullContigs} and {@link MetaCooler.fullGenomes} read the
* rest when that view actually opens, which is the only place it is shown.
*/
import { UNBINNED_NAME } from '../constants'
import type { StorageMode } from '../constants'
import { MetaCoolerFormatError, truncateNames, ViewerError } from '../errors'
import { pyRepr } from '../format'
import { fmt } from '../units'
import type { AttrValue, Hdf5Handle } from '../h5/api'
import { NameIndex } from '../genome/naming'
import { openHdf5 } from '../h5/Hdf5File'
import { dtypeKind } from '../h5/typed'
import { makeTable } from '../h5/readTable'
import type {
ByteSource,
Column,
DenseMatrix,
Extent,
Info,
Layout,
MatrixQuery,
PixelChunk,
Table,
} from '../types'
import {
aggregateMatrix,
pixelAccumulator,
sparseAggregator,
type AggregatedMatrix,
type AggregationLevel,
} from './aggregate'
import { Coords } from './coords'
import {
poolBins,
POOLED_CELL_CAP,
selectionPooling,
type BinPooling,
type PoolPlan,
} from './pooling'
import {
assertAcceptable,
cellPath,
detectLayout,
extractMetadata,
normaliseInfo,
scoolGeometry,
willListCells,
} from './detect'
import {
DEFAULT_WEIGHT_COLUMN,
DENSE_HINTS,
type SpanScanOptions,
planRead,
readDenseMatrix,
resolveCountKind,
} from './matrix'
import { CoolerStore, type PixelStore, type ScanAllOptions } from './store'
import {
pyramid,
readPicture,
type GatherOptions,
type PictureContacts,
type PictureLayout,
type PictureLevel,
type Quantizer,
} from './picture'
import { BinLens, LensStore, lensFor, lensedHandle, sequenceTable } from './lens'
import { CoarseGrid } from './coarsen'
import { CoarseStore, coarsenedHandle } from './coarseMap'
import type { ReferenceEdit } from '../genome/edit'
import {
ResampledStore,
takeCensus,
type Census,
type ResamplePlan,
type ResampleSpec,
} from './resample'
import { resolveCells, ScoolStore, type PerCellScanOptions } from './scool'
import { HicFile } from '../hic/HicFile'
import { isHicMagic } from '../hic/format'
import { hicHandle } from '../hic/handle'
import { HicStore } from '../hic/store'
import type { BinnedPixels } from '../pairs/bin'
import { isPairsFile, readPairsHeader, type PairsHeader } from '../pairs/format'
import { isGzip as isGzipMagic } from '../pairs/stream'
import { pairsHandle } from '../pairs/handle'
import { PairsStore } from '../pairs/store'
import { hicProHandle, type HicProFile } from '../hicpro/handle'
import { parseRegion, regionToExtent } from './range'
import { parseUri } from '../uri'
/** Columns `schema.validate` requires; absence is a format error, not a crash. */
const REQUIRED_GENOME_COLUMNS = ['name', 'length', 'n_contigs', 'n_bins'] as const
/** What any cooler's chrom/contig table must carry. */
const REQUIRED_COOLER_COLUMNS = ['name', 'length'] as const
/**
* The `/contigs` columns the *reader* uses, as against the ones a creator may
* have joined in for the table view.
*
* `genome_id` is here for the metacool path and simply absent from a cooler's
* `/chroms`, which is the same shape of absence the required-column check
* already handles.
*/
const READER_CONTIG_COLUMNS = ['name', 'length', 'genome_id'] as const
/** The `/genomes` columns the reader uses; the rest are the table view's. */
const READER_GENOME_COLUMNS = ['name', 'length', 'n_contigs', 'n_bins'] as const
/**
* The attributes an scool's root may leave to its cells.
*
* Geometry only, and it is a whitelist rather than a merge for the reason
* {@link MetaCooler.fromScool} gives: a cell's `nnz` and `sum` are that cell's,
* and letting them stand as the container's would have the light validation
* check one cell's pixel count against the whole selection's.
*/
const SCOOL_GEOMETRY_ATTRS = ['bin-size', 'bin-type', 'storage-mode'] as const
/** Cell-table columns that lead, and are shown even when every cell agrees. */
const CELL_TABLE_LEADING: readonly string[] = ['nnz', 'sum']
/**
* Everything `MetaCooler` needs, already read.
*
* Split out from the constructor so a synthesised view — a plain cooler
* built from something other than a metacool's four tables — a plain cooler,
* which has two — can be constructed without this class growing a second
* constructor.
*/
export interface MetaCoolerInit {
uri: string
handle: Hdf5Handle
layout: Layout
attrs: Record<string, AttrValue>
contigs: Table
/**
* Where the pixels come from, when they do not come from this handle.
*
* Absent for every HDF5 file, which is the overwhelming majority and takes
* {@link CoolerStore} by default. Supplied by {@link MetaCooler.fromHic},
* which is the one caller whose pixels are indexed some other way.
*/
store?: PixelStore
/**
* Where the *tables* come from, when they do not come from `handle`.
*
* One container needs this and it is the `.scool`: its root holds a set of
* coolers rather than being one, so `/pixels` is not there to take a dtype
* from and — in the flat layout — neither is `/chroms` or `/bins`. Those
* questions are answered by a representative cell, which shares its bin space
* with every other cell by construction.
*
* Deliberately **not** where weights come from. `bins/weight` stays on
* `handle`, so a cell set is balanced by the *shared* bins or not at all
* (`metacooler/scool.ts`). Absent for every other layout, where the two
* handles are the same object.
*/
tables?: Hdf5Handle
/** `indexes/chrom_offset`, length `nContigs + 1`. */
chromOffset: Float64Array
/**
* `/genomes`, or `undefined` for a plain cooler.
*
* Undefined rather than a synthesised one-row table: a cooler has no genomes,
* and inventing one named after the assembly would put a row in the UI that
* is not in the file, then require every genome path to special-case it. The
* genome layer is simply absent, and the accessors say so.
*/
genomes?: Table
/** `indexes/genome_offset`, length `nGenomes + 1`; absent for a cooler. */
genomeOffset?: Float64Array
/**
* The map this one is an edit of (§27.7), where it is drawn through one —
* {@link MetaCooler.withLens} sets it, and the one read that must see the
* file's own order, the editing stage's picture (§27.9), reads through it.
*/
unlensed?: MetaCooler
}
/** Anything `MetaCooler.open` needs that is not the file or its bytes. */
export interface OpenOptions {
/**
* The reader has already agreed to what listing this file's cells costs.
*
* Set on the *second* open of a container over the network — the one made
* with their answer to the §12 gate on the source — and it is what stops that
* gate from asking the same question again forever. Choosing *download it
* whole* changes the source's own answer to `isRemote` and would pass
* regardless; choosing *small pieces* does not, and did not, which is the bug
* this exists to have fixed rather than a defensive flag.
*/
listingAgreed?: boolean
/**
* Whether the multi-genome containers are read (§29) — `false` while the
* *dev features* switch is off, and absent on the command line, which reads
* everything. See `DetectOptions.metacool`.
*/
metacool?: boolean
}
/** The §7.4 QC statistic: how much of the file binning failed to assign. */
export interface UnassignedStats {
/** True genome index of `__unbinned__` — never renumbered away (§7.4). */
genomeIndex: number
contigs: number
bins: number
totalContigs: number
totalBins: number
/** Fraction in `[0, 1]`, or `0` when the denominator is 0. */
contigFraction: number
binFraction: number
}
/** Callbacks for {@link MetaCooler.scanPixels}. */
export interface ScanOptions {
/** Pixels per chunk; `DEEP_PIXEL_CHUNK` in Python. */
chunkSize?: number
/** Called per chunk. Returning `false` stops the scan — this is the cancel. */
onChunk(chunk: PixelChunk): boolean | void
/** Cumulative and total pixel counts after each chunk. */
onProgress?(done: number, total: number): void
}
/** Outcome of {@link MetaCooler.validateDeep}. */
export interface DeepValidation {
ok: boolean
problems: string[]
/** What was actually checked — and, just as importantly, what was not. */
note: string
}
export class MetaCooler {
/** The URI this map was opened from, `file` or `file::group`. */
readonly uri: string
/** Which of the four layouts the file turned out to be. */
readonly layout: Layout
/**
* Sibling names when this map lives inside a collection: the other maps of an
* mmetacool (§3.4), or the other resolutions of an mcool (§9).
*/
readonly entries: string[]
/** The group those siblings live under — `entries` or `resolutions`. */
readonly entryGroup: string
/** The handle that answers table and pixel-shape questions — see `MetaCoolerInit.tables`. */
private readonly tables: Hdf5Handle
/** Normalised root attributes (`MetaCooler.info`). */
readonly info: Info
/** The optional user `metadata` object, `{}` when absent. */
readonly metadata: Record<string, unknown>
/**
* `/genomes` with every column the creator wrote (spec §5.1), or `null` for a
* plain cooler, which has no such table and is not pretending to.
*/
readonly genomes: Table | null
/** `/contigs` with every column the creator wrote. */
readonly contigs: Table
/** Uniform bin size in bp. Never `null` here — §3.2 refuses variable bins. */
readonly binsize: number
/** The map this one is an edit of, or `null` for a map that is the file (§27.7). */
readonly unlensed: MetaCooler | null
/** `symmetric-upper` or `square`, read from the attribute, never assumed. */
readonly storageMode: StorageMode
/** Dtype kind of `pixels/count`, resolved once (correction #2). */
readonly countKind: 'i' | 'u' | 'f'
/** Fixed-bin arithmetic over `chrom_offset` + `contigs/length` (§3.3). */
readonly coords: Coords
/** Contig names, in file order — cooler's `chromnames`. */
readonly chromnames: string[]
/**
* Contig name → length in bp — cooler's `chromsizes`. Built on first use.
*
* Lazy because it is only ever consulted by name, and only by a *region*
* selection: on a 154 k-contig file it is 7.3 MB of map built during the open,
* before the landing table has painted, for a lookup many sessions never make.
*/
get chromsizes(): Map<string, number> {
this.chromsizeCache ??= new Map(this.chromnames.map((name, i) => [name, this.contigLengths[i]]))
return this.chromsizeCache
}
/** Contig name → index. Lazy for the same reason as {@link chromsizes}. */
private get contigIds(): Map<string, number> {
this.contigIdCache ??= new Map(this.chromnames.map((name, i) => [name, i]))
return this.contigIdCache
}
/**
* This file's sequences, indexed by every convention that might name one.
*
* Lazy for {@link chromsizes}' reason and consulted for a narrower one: it is
* only ever reached where a name has *already failed* to resolve strictly
* (`parseMetacoolerRange`), which is to say when a second file with its own
* spellings is in the room. A session of one file never builds it.
*/
get names(): NameIndex {
this.nameIndexCache ??= NameIndex.of(
this.chromnames.map((name, i) => ({ name, length: this.contigLengths[i] })),
)
return this.nameIndexCache
}
/** Genome names, in file order, `__unbinned__` included; `[]` for a cooler. */
readonly genomeNames: string[]
/**
* `contigs/genome_id`, one entry per contig, exactly as written. Public
* because the aggregation path needs contig → genome and must never invent
* its own mapping (§7.4: nothing is renumbered).
*
* Empty for a plain cooler: there is no mapping to carry, and a filled-in one
* would be an invention the aggregation path could not tell from real data.
*/
readonly contigGenomeIds: Int32Array
/**
* The file, as opened — what {@link handle} answers from when nothing is
* resampled, and what closes.
*/
private readonly rawHandle: Hdf5Handle
/**
* The file's groups and datasets, with its weights on the resampled map's
* scale when there is one (§6.5).
*
* Every balanced read multiplies by `bins/weight` through here — the dense
* block, the pooled and aggregated roll-ups, the tasks' own reads — so scaling
* the one column here is scaling it everywhere, and a subsample balanced with
* the file's weights cannot read `N/T` times fainter in one place and not in
* another. Everything else passes through untouched.
*/
get handle(): Hdf5Handle {
const scale = this.resampled?.weightScale ?? 1
if (scale === 1) return this.rawHandle
if (this.weighted?.scale !== scale) {
this.weighted = { scale, handle: scaledWeights(this.rawHandle, scale) }
}
return this.weighted.handle
}
/** The weight-scaling view of {@link rawHandle}, for the scale it was made at. */
private weighted: { scale: number; handle: Hdf5Handle } | null = null
/**
* The pixels, behind the one interface both containers implement.
*
* Public because `analysis/run/` walks pixels directly and must go through
* the same seam the views do; see `metacooler/store.ts` for why the seam is
* at the operation rather than at the dataset.
*/
/**
* Where the pixels come from.
*
* Behind a getter because one container replaces it: an `.scool` swaps the
* store when the cell set changes ({@link selectCells}), and every reader of
* it — `readDenseMatrix` through `MatrixSource`, the scans, the pooled
* aggregations — must see the current one rather than a captured reference.
*/
private pixelStore: PixelStore
get store(): PixelStore {
return this.resampled ?? this.pixelStore
}
/**
* The pixels exactly as the file stores them, whatever is drawn over them.
*
* What the deep check walks (§6.3): it reconciles the stored counts against
* `attrs['sum']`, and a resampled map would fail it by construction.
*/
get storedPixels(): PixelStore {
return this.pixelStore
}
/** The map drawn to another total, or `null` for the file as stored (§6.5). */
private resampled: ResampledStore | null = null
/**
* Every sequence pair's contacts, taken once and kept: it does not depend on
* the total, so a second number typed costs no second walk.
*/
private census: Promise<Census> | null = null
/** A resample is waiting on the census, which then no longer pauses (§12). */
private censusWanted = false
private readonly chromOffset: Float64Array
private readonly genomeOffset: Float64Array
private readonly genomeIds: Map<string, number>
private closed = false
/** Memoised full-width tables; see {@link fullContigs}. */
private fullContigTable: Table | null = null
private fullGenomeTable: Table | null = null
/** Built on the first lookup by name; see {@link contigIds}. */
private contigIdCache: Map<string, number> | null = null
private chromsizeCache: Map<string, number> | null = null
private nameIndexCache: NameIndex | null = null
/** `indexes/bin1_offset`, read whole on first use; see {@link binOffsets}. */
private readonly contigLengths: Float64Array
/**
* Open a map from a byte source.
*
* The source is the only thing that differs between a local drag-and-drop
* and a remote range-request file (§6.1); everything below this line is
* identical for both.
*/
static async open(
uri: string,
source: ByteSource,
options: OpenOptions = {},
): Promise<MetaCooler> {
// Mounted first so the sniff below has a byte accessor in every backing —
// a path-backed source resolves its filesystem at mount and has nowhere
// else to await one. Idempotent, and `openHdf5` reuses the same mount.
await source.mount()
// Four bytes decide the container. Asked before HDF5 is asked anything at
// all, because a `.hic` is not an HDF5 file and h5wasm's refusal of one is
// a stack trace rather than the sentence §10 requires.
if (isHicMagic(source.readBytes(0, 4))) return MetaCooler.fromHic(uri, source)
// A `.pairs` is a *third* container and the only one that cannot be opened
// from its own bytes alone: it declares no resolution, so there is nothing
// to open it *at* until a reader has chosen one. The refusal carries what
// the choice needs — see `pairs/import.ts` — and the worker turns it into a
// dialog. Asked here rather than in the worker so every entry point (the
// CLI, a test) gets the same sentence instead of an HDF5 stack trace.
if (await isPairsFile(source)) {
const header = await readPairsHeader(source)
throw new ViewerError(
'pairs-binning-needed',
'This is a .pairs file: a list of contacts with no resolution of its own. Choose a ' +
'bin size to build a map at.',
{
contigs: header.contigs.length,
assembly: header.assembly,
sorted: header.sorted,
},
)
}
// A gzip that is *not* a pairs file is refused here rather than handed on.
// HDF5 cannot read a compressed file in any case, and what it does instead
// is worth pre-empting: h5wasm answers a request for the root attributes of
// a non-HDF5 file with `error - name not defined!`, which is neither the
// sentence §10 requires nor recognisably about the file the reader dropped.
// The containers a refusal may name: the multi-genome one only while it
// is read (§29), since a sentence naming a format this page refuses would
// be the one place the reader was told it exists.
const containers =
options.metacool === false ? '.cool, .mcool or .scool' : '.cool, .mcool, .scool or .metacool'
if (isGzipMagic(source.readBytes(0, 2))) {
throw new ViewerError(
'not-hdf5',
'This file is gzip-compressed. HDF5 is read in place and cannot be decompressed on ' +
`the fly, so a ${containers} must be decompressed first. A .pairs.gz ` +
'is read compressed — but this one carries no `## pairs format` line, so it is not ' +
'one.',
{},
)
}
// And a file that is not HDF5 at all is refused by its first bytes, for
// the same reason the gzip above is: a file named `.mcool` that is really
// an HTML error page, a text export or a download that stopped short
// fails inside h5wasm with `error - name not defined!`, which names
// neither the file nor the fact. The superblock may sit behind a user
// block at a power-of-two offset, so the first few are looked at.
if (!hasHdf5Signature(source)) {
throw new ViewerError(
'not-hdf5',
`${parseUri(uri).path} does not start with the HDF5 signature, so it is not a ` +
`${containers} whatever its name says. A download that stopped ` +
'short, an HTML page saved under the file’s name, or a text matrix look like this.',
{},
)
}
// The group comes from the URI, so `file.mmetacool::entries/rep1` and a
// plain `file.metacool` reach the handle already resolved (§3.5).
const handle = await openHdf5(uri, source, parseUri(uri).group)
try {
if (options.listingAgreed !== true) assertListingAffordable(handle, source)
return MetaCooler.fromHandle(handle, options)
} catch (error) {
handle.close()
// h5wasm's own failures are a sentence about nothing — `error - name
// not defined!` — and reach the screen as "something went wrong". They
// are about *this file*, so they are said about it: which file, which
// group, and the library's words after, for whoever reads them.
if (!(error instanceof ViewerError) && error instanceof Error) {
const group = parseUri(uri).group
throw new ViewerError(
'format',
`${parseUri(uri).path}${group === '' ? '' : `::${group}`} could not be read as a ` +
`cooler container. HDF5 said: ${error.message}. The file opens as HDF5, so this is ` +
'a group, dataset or attribute the reader expected and did not find.',
{ hdf5: error.message },
)
}
throw error
}
}
/**
* Open a `.hic` as one of these.
*
* The three lines of translation are all there is, and that is the design
* rather than an accident: `hic/handle.ts` answers the metadata questions a
* cooler's HDF5 groups answer, `hic/store.ts` answers the pixel ones, and
* this constructor is the only place in the repo that knows both exist. The
* resulting object is a `MetaCooler` in every sense the rest of the
* application uses the word.
*
* The resolution is named the way an mcool's is — `file.hic::/resolutions/5000`
* — so the entry picker, the URL hash, the refine ladder (§3.4) and a deck
* descending to its lead's rung all work on a `.hic` without a line of their
* own. A URI with no group opens the rung `landingRung` names — a third of
* the way up from the finest, which is what dropping one on the page means, as
* it is for an `.mcool` addressed at its root.
*/
static fromHic(uri: string, source: ByteSource): MetaCooler {
const { group } = parseUri(uri)
const named = /^\/resolutions\/(\d+)$/.exec(group)
const file = HicFile.open(source, named === null ? {} : { binSize: Number(named[1]) })
if (!file.hasAnyMatrix()) {
throw new ViewerError(
'unsupported',
`This .hic carries no base-pair matrix at ${file.binSize} bp. Its resolutions are ` +
`${file.resolutions.join(', ')}.`,
{ binSize: file.binSize, resolutions: file.resolutions },
)
}
const handle = hicHandle(uri, file)
const attrs = handle.attrs()
return new MetaCooler({
uri,
handle,
layout: {
kind: 'hic',
group,
entries: file.resolutions.map((size) => String(size)),
},
attrs,
contigs: handle.readTable('chroms'),
chromOffset: handle.readNumeric('indexes/chrom_offset'),
store: new HicStore(file),
})
}
/**
* Open one rung of an imported `.pairs` as one of these.
*
* The same three lines of translation a `.hic` gets, and for the same reason:
* `pairs/handle.ts` answers the metadata questions, `pairs/store.ts` answers
* the pixel ones, and this constructor is the only place in the repo that
* knows both exist.
*
* Unlike every other constructor here, this one reads no bytes. The file was
* read once, at import; what arrives is a pixel table and the header it came
* from, and the resulting object is a `MetaCooler` in every sense the rest of
* the application uses the word — including the fourteen analysis tasks,
* which cannot tell the difference and are not told.
*
* The rung list is spelled the way an mcool's resolutions are, so the entry
* picker, the URL hash, §3.4's refine ladder and a deck descending to its
* lead's rung all work with no code of their own.
*/
static fromPairs(init: {
uri: string
header: PairsHeader
pixels: BinnedPixels
rungs: number[]
indexed: boolean
declaredRecords?: number | null
}): MetaCooler {
const handle = pairsHandle({
uri: init.uri,
header: init.header,
pixels: init.pixels,
rungs: init.rungs,
indexed: init.indexed,
declaredRecords: init.declaredRecords ?? null,
})
return new MetaCooler({
uri: init.uri,
handle,
layout: {
kind: 'pairs',
group: `/resolutions/${init.pixels.binSize}`,
entries: init.rungs.map((size) => String(size)),
},
attrs: handle.attrs(),
contigs: handle.readTable('chroms'),
chromOffset: handle.readNumeric('indexes/chrom_offset'),
store: new PairsStore(init.pixels),
})
}
/**
* Open an `.scool` as one cooler over a set of its cells (§3.8).
*
* The file is a container of coolers sharing one bin space, so the object
* this returns is a `MetaCooler` in the same full sense a `.hic` is: the
* coordinate system comes from the cells' shared tables and the pixels come
* from {@link ScoolStore}, which sums the selected set. Every view above it —
* the chromosome table, a region, the whole map, the axis runs, the pooled
* reads — works with no knowledge that a cell exists.
*
* `cells` is the ticked set, or `null` for **all of them**. All is what a file
* dropped and pressed on through means: an scool with nothing chosen is the
* pseudo-bulk of everything in it. It is also the expensive one, which is why
* the count is on {@link OpenResult} and gated in the caller rather than here
* — the reader refuses nothing it was asked for.
*
* ## Where each table comes from
*
* A **representative cell**, in both layouts and for both tables. The
* standard container hard-links `/chroms` and `/bins` into every cell from
* the root, so reading a cell's is reading the root's; the flat one has
* nothing at its root to read. One path, and it is correct for both.
*
* The attributes are the other way round: the root's win, and the cell's fill
* the gaps. A standard root states the geometry it has a `/bins` to describe;
* a flat root states only provenance, and the geometry is inside the cells.
* Whitelisted rather than merged wholesale, because a cell's `nnz` and `sum`
* describe *that cell* and would be validated here against the whole set.
*/
static fromScool(
handle: Hdf5Handle,
layout: Layout,
attrs: Record<string, AttrValue>,
cells: readonly string[] | null,
): MetaCooler {
const names = resolveCells(layout.entries ?? [], cells)
// The representative is the file's first cell, not the selection's: it
// answers questions about the *shared* bin space, which every cell agrees
// on, and pinning it to the file rather than the selection keeps the
// coordinate system identical as the ticks change.
const representative = handle.at(cellPath(layout, (layout.entries ?? [])[0]))
const geometry = scoolGeometry(handle, layout, attrs).attrs
const merged: Record<string, AttrValue> = { ...attrs }
for (const key of SCOOL_GEOMETRY_ATTRS) {
if (merged[key] === undefined && geometry[key] !== undefined) merged[key] = geometry[key]
}
const chromOffset = representative.readNumeric('indexes/chrom_offset')
const countKind = resolveCountKind(representative.dtype('pixels/count'))
const nBins = Number(chromOffset[chromOffset.length - 1] ?? 0)
return new MetaCooler({
uri: handle.uri,
handle,
tables: representative,
layout,
attrs: merged,
contigs: representative.readTable('chroms', undefined, undefined, READER_CONTIG_COLUMNS),
chromOffset,
store: new ScoolStore(handle, layout, names, nBins, countKind),
})
}
/**
* Open a HiC-Pro pair as one of these (§3.9).
*
* Shorter than {@link fromHic} by a whole module, and the reason is worth
* keeping: a HiC-Pro matrix has no index, so it was read whole before this
* point, and what the parser hands back is what a cooler stores — sorted
* triplets and a cumulative `bin1_offset`. So the handle serves the pixel
* datasets honestly and the **default {@link CoolerStore} reads them**, which
* makes the pixel path for this container the cooler path exactly rather than
* a second implementation of it.
*
* The refusal below is §3.3's, applied where the evidence is: a HiC-Pro map
* binned by restriction fragment (`GENOME_FRAGMENT`) is a variable-bin file,
* and the only place that is visible is the widths in the BED.
*/
static fromHicPro(uri: string, file: HicProFile): MetaCooler {
if (!file.bins.fixed || file.bins.binsize === null) {
throw new ViewerError(
'variable-bins',
`The viewer supports fixed-bin files only, and ${file.binsName} is not one: its ` +
`intervals are of differing widths, which is what a fragment-level map ` +
`(\`GENOME_FRAGMENT\` in HiC-Pro's config) looks like. Re-run HiC-Pro with a ` +
'`BIN_SIZE`, or open the fixed-bin matrix beside this one.',
{ 'bin-type': 'variable', 'bin-size': null, bins: file.binsName },
)
}
const handle = hicProHandle(uri, file)
return new MetaCooler({
uri,
handle,
layout: { kind: 'hicpro', group: '/' },
attrs: handle.attrs(),
contigs: handle.readTable('chroms'),
chromOffset: file.bins.chromOffset,
})
}
/**
* Build a map from an already-open handle: detect, gate, then read the four
* small things §3.2 allows.
*
* @throws ViewerError from {@link detectLayout} (`not-a-cooler`) or
* {@link assertAcceptable} (`mmetacool-root`, `mcool-root`,
* `bad-format-version`, `bad-ordering`, `variable-bins`), and
* MetaCoolerFormatError for a metacool missing `indexes/genome_offset`.
*/
static fromHandle(handle: Hdf5Handle, options: OpenOptions = {}): MetaCooler {
// Read **once**. Three enumerations of the root attribute block cost three
// wasm round trips per fixed-length string attribute, for a set of values
// that cannot change between them.
const attrs = handle.attrs()
const gate = options.metacool === undefined ? {} : { metacool: options.metacool }
const layout = detectLayout(handle, attrs, gate)
assertAcceptable(handle, layout, attrs, gate)
// A plain cooler — including one resolution of an mcool — is opened as
// itself: its `/chroms` is the contig table, and there is no genome layer
// to read. Everything below this point is bin arithmetic over
// `chrom_offset` and contig lengths, which is the same in both files.
if (layout.kind === 'scool') return MetaCooler.fromScool(handle, layout, attrs, null)
if (layout.kind !== 'metacool') {
return new MetaCooler({
uri: handle.uri,
handle,
layout,
attrs,
contigs: handle.readTable('chroms', undefined, undefined, READER_CONTIG_COLUMNS),
chromOffset: handle.readNumeric('indexes/chrom_offset'),
})
}
if (!handle.has('indexes/genome_offset')) {
throw new MetaCoolerFormatError('indexes/genome_offset is missing')
}
return new MetaCooler({
uri: handle.uri,
handle,
layout,
attrs,
genomes: handle.readTable('genomes', undefined, undefined, READER_GENOME_COLUMNS),
contigs: handle.readTable('contigs', undefined, undefined, READER_CONTIG_COLUMNS),
chromOffset: handle.readNumeric('indexes/chrom_offset'),
genomeOffset: handle.readNumeric('indexes/genome_offset'),
})
}
/**
* `/contigs` (or `/chroms`) with **every** column the creator wrote — the
* table the contigs *view* shows, read when that view opens.
*
* Opening a file does not need them. On the reference file the optional
* metadata columns are 72% of the whole open's HDF5 bytes, decoded before the
* landing table has painted, for a view the user may never reach. Memoised,
* because the view is reached by navigation and comes back.
*/
fullContigs(): Table {
this.fullContigTable ??= this.tables.readTable(
this.layout.kind === 'metacool' ? 'contigs' : 'chroms',
)
return this.fullContigTable
}
/** `/genomes` with every column the creator wrote; `null` for a cooler. */
fullGenomes(): Table | null {
if (this.genomes === null) return null
this.fullGenomeTable ??= this.handle.readTable('genomes')
return this.fullGenomeTable
}
constructor(init: MetaCoolerInit) {
this.uri = init.uri
this.unlensed = init.unlensed ?? null
this.rawHandle = init.handle
this.tables = init.tables ?? init.handle
this.layout = init.layout
this.entries = init.handle.rootEntries
this.entryGroup = init.handle.rootEntryGroup
this.genomes = init.genomes ?? null
this.contigs = init.contigs
this.chromOffset = init.chromOffset
this.genomeOffset = init.genomeOffset ?? new Float64Array(0)
this.info = normaliseInfo(init.attrs)
this.metadata = extractMetadata(init.attrs)
this.storageMode = this.info['storage-mode']
const binsize = this.info['bin-size']
if (binsize === null || binsize <= 0) {
// Unreachable through `fromHandle` — the §3.2 gate ran first — but a
// synthesised init could get here, and a zero bin size would turn every
// coordinate formula into a division by zero rather than an error.
throw new MetaCoolerFormatError(
`attrs['bin-size'] must be positive, got ${pyRepr(binsize)}`,
{ 'bin-size': binsize },
)
}
this.binsize = binsize
this.chromnames = this.contigs.strings('name')
this.genomeNames = this.genomes === null ? [] : this.genomes.strings('name')
// `numbers()` materialises a plain JS array first — 1.5 MB of throwaway per
// call at 154 k contigs, twice. The column is already a `Float64Array` off
// the decoder, so the typed-array constructor converts it in one pass with
// nothing in between.
const contigLengths = toFloat64(this.contigs, 'length')
this.contigGenomeIds =
this.genomes === null ? new Int32Array(0) : toInt32(this.contigs, 'genome_id')
this.coords = new Coords(this.binsize, this.chromOffset, contigLengths)
this.contigLengths = contigLengths
this.genomeIds = new Map()
for (let i = 0; i < this.genomeNames.length; i++) this.genomeIds.set(this.genomeNames[i], i)
this.countKind = resolveCountKind(this.tables.dtype('pixels/count'))
// After `countKind` and after `chromOffset`, because the default store is
// built from both. A supplied one already knows its own answers.
this.pixelStore = init.store ?? new CoolerStore(this.rawHandle, this.nBins, this.countKind)
this.validateLight()
}
// -------------------------------------------------------------------------
// Shape and identity
// -------------------------------------------------------------------------
/**
* `true` when this file has a genome layer — a metacool.
*
* The one thing callers branch on. A plain cooler answers `0`, `[]` and
* `null` from every genome accessor below rather than throwing, so the
* difference shows up as an empty list in the UI and not as an error screen.
*/
get hasGenomes(): boolean {
return this.genomes !== null
}
/**
* `true` when `/bins/weight` exists — the file has been balanced.
*
* One `has` on a dataset path, no read. The UI offers the balance toggle only
* when this is true, so "balanced" is never an option that then fails.
*/
get hasWeights(): boolean {
return this.handle.has(`bins/${DEFAULT_WEIGHT_COLUMN}`)
}
/**
* The `/pixels` value columns this file carries, `count` first.
*
* One `keys` call on a group, no read. What it is *for* is the score picker:
* cooler fixes three columns and lets a file carry more, and a viewer that
* assumed `count` could not show the others at all — so a file that had gone
* to the trouble of storing an expected value, a second normalisation or a
* per-pixel significance had no way of saying so.
*/
get scoreColumns(): string[] {
return this.store.scoreColumns()
}
/** Number of genomes, `__unbinned__` included; `0` for a plain cooler. */
get nGenomes(): number {
return this.genomes?.length ?? 0
}
/** Number of contigs. */
get nContigs(): number {
return this.contigs.length
}
/** Total bins — `chrom_offset`'s terminator, never `len(/bins)`. */
get nBins(): number {
return this.coords.nBins
}
/** `(n_bins, n_bins)`, as cooler reports it. */
get shape(): [number, number] {
return [this.nBins, this.nBins]
}
/** `true` if *name* is a genome in this file (`__unbinned__` counts). */
hasGenome(name: string): boolean {
return this.genomeIds.has(name)
}
/** `true` if *name* is a contig in this file. */
hasContig(name: string): boolean {
return this.contigIds.has(name)
}
/**
* Genome index by name.
*
* @throws ViewerError `unknown-label`, mirroring the Python `KeyError`.
*/
genomeId(name: string): number {
const id = this.genomeIds.get(name)
if (id === undefined) {
throw new ViewerError(
'unknown-label',
`Genome ${pyRepr(name)} not found. Available: ${truncateNames(this.displayGenomes().map((i) => this.genomeNames[i]))}`,
{ label: name },
)
}
return id
}
/**
* Contig index by name.
*
* @throws ViewerError `unknown-label`, mirroring the Python `KeyError`.
*/
contigId(name: string): number {
const id = this.contigIds.get(name)
if (id === undefined) {
throw new ViewerError(
'unknown-label',
`Contig ${pyRepr(name)} not found. Available: ${truncateNames(this.chromnames)}`,
{ label: name },
)
}
return id
}
// -------------------------------------------------------------------------
// Extents
// -------------------------------------------------------------------------
/**
* Half-open bin range of a `contig` or `contig:start-end` region, exactly as
* `cooler.Cooler.extent` computes it. Genome names are **not** accepted here
* — that is `parseMetacoolerRange`'s job, and cooler has no concept of one.
*/
extent(region: string): Extent {
const [chrom, start, end] = parseRegion(region, this.chromsizes)
return regionToExtent(this.chromOffset, this.contigId(chrom), start, end, this.binsize)
}
/** First bin of a region — `cooler.Cooler.offset`. */
offset(region: string): number {
return this.extent(region)[0]
}
/** Half-open contig range `[c0, c1)` of a genome, from `genome_offset`. */
genomeContigExtent(genome: string | number): Extent {
const gid = typeof genome === 'number' ? genome : this.genomeId(genome)
return [this.genomeOffset[gid], this.genomeOffset[gid + 1]]
}
/**
* Half-open **bin** range of a genome.
*
* Under the accepted `genome-then-length` ordering this always succeeds: the
* genome's contigs are `genome_offset[g] … genome_offset[g+1]` and its bins
* are the corresponding `chrom_offset` span. The contiguity assertion is
* kept anyway — it is O(1) and free, and it catches a file whose
* `contig-ordering` attribute lies about its actual layout, which no other
* check in the open path would notice.
*
* @returns `[0, 0]` for a genome with no bins, matching Python.
* @throws ViewerError when the genome's contig block is not homogeneous.
*/
genomeExtent(genome: string | number): Extent {
const gid = typeof genome === 'number' ? genome : this.genomeId(genome)
const [c0, c1] = this.genomeContigExtent(gid)
if (c1 <= c0) return [0, 0]
this.assertContiguous(gid, c0, c1)
const lo = this.chromOffset[c0]
const hi = this.chromOffset[c1]
return hi <= lo ? [0, 0] : [lo, hi]
}
/**
* The free invariant check behind {@link genomeExtent}.
*
* `genome_id` is non-decreasing under a genome-grouped ordering, so proving
* that the block `[c0, c1)` is exactly genome *gid* needs only its two
* endpoints and its two neighbours — four lookups, no scan.
*/
private assertContiguous(gid: number, c0: number, c1: number): void {
const ids = this.contigGenomeIds
const bad =
ids[c0] !== gid ||
ids[c1 - 1] !== gid ||
(c0 > 0 && ids[c0 - 1] === gid) ||
(c1 < ids.length && ids[c1] === gid)
if (bad) {
throw new ViewerError(
'bad-ordering',
`Genome ${pyRepr(this.genomeNames[gid])} is not contiguous in bin space under ` +
`contig-ordering ${pyRepr(this.info.ordering)}; a single bin range cannot represent ` +
`it, so the file's ordering attribute does not describe its actual contig layout. ` +
`Re-create the file with a genome-grouped ordering.`,
{ genome: this.genomeNames[gid], genomeId: gid, contigs: [c0, c1] },
)
}
}
/**
* Half-open bin range of a contig, straight from `chrom_offset`.
*
* @returns `[0, 0]` for a contig with no bins, matching Python.
*/
contigExtent(contig: string | number): Extent {
const cid = typeof contig === 'number' ? contig : this.contigId(contig)
const [lo, hi] = this.coords.contigBinExtent(cid)
return hi <= lo ? [0, 0] : [lo, hi]
}
// -------------------------------------------------------------------------
// `__unbinned__` (spec §7.4)
// -------------------------------------------------------------------------
/**
* Genomes the UI may show: everything except `__unbinned__`.
*
* Filtered by **name**, never by index — `create(drop_unassigned=True)`
* omits the row entirely, which makes genome 0 an ordinary MAG in exactly
* the files that were binned most cleanly. Each entry keeps its true index,
* because nothing is renumbered or compacted to close the gap: `genome_id`,
* `genome_offset` and every bin index stay as written, and exclusion happens
* at selection and display time only.
*/
displayGenomes(): number[] {
const out: number[] = []
for (let i = 0; i < this.genomeNames.length; i++) {
if (this.genomeNames[i] !== UNBINNED_NAME) out.push(i)
}
return out
}
/**
* The one number `__unbinned__` is allowed to contribute (§7.4): how many
* contigs and what share of the bins binning failed to assign.
*
* It is a QC statistic, not a view — it opens nothing and selects nothing —
* and it is read straight off the `/genomes` row, so no code path reaches
* `__unbinned__` through it.
*/
/**
* The bins a picture of "the whole file" is allowed to draw.
*
* `__unbinned__` is not a genome and is never displayed (§7.4) — it is absent
* from the genomes table, unselectable, and excluded from the overview. The
* whole-file map read `[0, nBins)` and drew it anyway, which on a real
* metagenome is the largest thing in the picture: on one 1.9 M-bin file it is
* 56% of the bins, so it dominated a band *and*, being in the drawn pixels, set
* the auto `vmax` for everything else (§7.1).
*
* Under `genome-then-length` every genome is one contiguous bin range and
* `__unbinned__` holds genome index 0, so leaving it out is a range and not a
* gap. Where it is not at either end — which no file this app accepts can
* produce — the whole range is kept rather than a wrong half of it.
*/
displayBinExtent(): Extent {
const whole: Extent = [0, this.nBins]
const gid = this.genomeIds.get(UNBINNED_NAME)
if (gid === undefined) return whole
const [lo, hi] = this.genomeExtent(gid)
if (hi <= lo) return whole
if (lo === 0) return [hi, this.nBins]
if (hi === this.nBins) return [0, lo]
return whole
}
unassignedStats(): UnassignedStats | null {
const totalContigs = this.nContigs
const totalBins = this.nBins
const gid = this.genomeIds.get(UNBINNED_NAME)
// Absence is normal, not an error: `create(drop_unassigned=True)` omits the
// row, and `null` is what tells the info panel there is nothing to report.
if (gid === undefined || this.genomes === null) return null
const contigs = Number(this.genomes.get('n_contigs', gid) ?? 0)
const bins = Number(this.genomes.get('n_bins', gid) ?? 0)
return {
genomeIndex: gid,
contigs,
bins,
totalContigs,
totalBins,
contigFraction: totalContigs > 0 ? contigs / totalContigs : 0,
binFraction: totalBins > 0 ? bins / totalBins : 0,
}
}
// -------------------------------------------------------------------------
// Density proxy (spec §4)
// -------------------------------------------------------------------------
/**
* Non-zero pixels whose **row** bin falls inside genome *g*:
* `bin1_offset[hi] - bin1_offset[lo]`, two lookups into an index the file
* already carries.
*
* **This is not a contact count and must never be labelled as one.** It
* counts stored pixels, not contacts; it ignores everything stored under a
* lower row bin (i.e. this genome's contacts with genomes before it); and it
* is exactly as cheap as its two array reads. As a per-MAG *density*
* indicator at zero cost it earns its sortable column in the genomes table,
* and nothing more.
*/
genomeDensity(genome: string | number): number {
const [lo, hi] = this.genomeExtent(genome)
if (hi <= lo) return 0
const offsets = this.binOffsets()
return offsets[hi] - offsets[lo]
}
/**
* `indexes/bin1_offset` in full, read once and kept.
*
* Two one-element HDF5 reads look cheaper than one whole-array read, and are
* not: each is a `file.get`, a metadata fetch, a `malloc`, a hyperslab read
* and a `free` — measured at 72 µs against an in-memory mount, the best case
* there is. The landing view asks for one per genome, so the reference file
* spent 8,788 of them (635 ms) before the table painted, and the overview's
* top-N sort asked ~190,000 (13.7 s) before its scan even started. The whole
* array is one read of 3.4 MB on that file, and every query after it is two
* array lookups.
*
* Held for the session: it is the index the density column, the size guards
* and every bounded read consult, so it is re-asked for constantly.
*/
binOffsets(): Float64Array {
return this.store.binOffsets()
}
/**
* The same proxy for every displayed genome, in one pass.
*
* Indexed by *genome id*, not by position in the result — nothing is
* renumbered here any more than anywhere else (§7.4).
*/
genomeDensities(): Float64Array {
const out = new Float64Array(this.nGenomes)
if (this.genomes === null) return out
const offsets = this.binOffsets()
for (const genome of this.displayGenomes()) {
const [lo, hi] = this.genomeExtent(genome)
if (hi > lo) out[genome] = offsets[hi] - offsets[lo]
}
return out
}
// -------------------------------------------------------------------------
// Matrix
// -------------------------------------------------------------------------
/**
* Load one dense block (§5.2, §5.3). Async only because the byte source may
* be; the read itself is four slices and no scan.
*/
async matrix(q: MatrixQuery, hint: string = DENSE_HINTS.range): Promise<DenseMatrix> {
return readDenseMatrix(this, q, hint)
}
/** The whole intra-genome matrix of one MAG — the §5.2 genome view. */
async genomeMatrix(genome: string | number): Promise<DenseMatrix> {
const extent = this.genomeExtent(genome)
return this.matrix({ rows: extent }, DENSE_HINTS.binSelection)
}
/**
* The rectangular block between two genomes — the §5.3 pair view.
*
* Both genomes are contiguous under the accepted ordering, so this is one
* bounded query however far apart they sit in bin space; the transpose is
* handled inside {@link readDenseMatrix}.
*/
async genomePairMatrix(row: string | number, col: string | number): Promise<DenseMatrix> {
return this.matrix(
{ rows: this.genomeExtent(row), cols: this.genomeExtent(col) },
DENSE_HINTS.binSelection,
)
}
/**
* Roll a block already in memory up to contig or genome level — free, no
* I/O (§4). The bin → contig mapping comes from `chrom_offset`, never from
* `/bins/chrom`.
*
* `storageMode` is passed through because it decides whether an on-diagonal
* block has to be folded before it is summed: under `symmetric-upper` the
* block is a mirror image of a half-stored triangle, and summing it whole
* would double every within-group off-diagonal pixel. The result then
* matches `metacooler aggregate` and `show --level contig` exactly.
*/
/**
* Roll a block up to contig or genome level **without materialising it**.
*
* The dense route is right for what its docstring describes — a MAG already on
* screen, whose contig view is then free — and wrong for a selection whose
* output is small and whose input is not. A whole yeast genome at 500 bp is
* 2,366 MB of `Float32Array` and 295 million iterations to produce a 17 × 17
* answer; this reads the same pixels in bounded windows and accumulates into
* `Float64Array(17²)`. Measured: 2,090 ms and 2.4 GB against 139 ms and a few
* kilobytes, with output identical to the last bit.
*
* That is also what takes the roll-up out from behind the dense guard. The
* refusal for an over-large block used to offer "or switch to contig level" as
* the escape hatch, and switching hit the same guard for the same reason,
* because the roll-up read the same block first.
*/
async aggregateSparse(
q: MatrixQuery,
level: AggregationLevel,
options: SpanScanOptions = {},
): Promise<AggregatedMatrix & { contigsPerCell: number }> {
const rows = clampExtent(q.rows, this.nBins)
const cols = clampExtent(q.cols ?? q.rows, this.nBins)
const plan = planRead(rows, cols, this.storageMode)
const aggregator = sparseAggregator(
rows,
cols,
this.coords,
this.contigGenomeIds,
level,
plan.mode,
this.storageMode === 'symmetric-upper',
// Bounded at what the renderer draws, for the same reason the bin grid is:
// an axis past it is pooled a second time on the way to the screen, flat
// and blind to the sequences it is collapsing.
POOLED_CELL_CAP,
)
// Weights are read once per axis for the bins on screen, exactly as the
// dense path does — the one bounded read of `/bins` in the reader (§6.4).
const weights = q.balance === true ? this.weightSlices(rows, cols) : null
// Watched for the same reason as the pooled read above: a roll-up of the
// whole file walks every pixel of it, and the level toggle is one click away
// from the view that does.
await this.store.forEachWindow(
rows,
cols,
plan,
(chunk) => {
for (let i = 0; i < chunk.length; i++) {
const b1 = chunk.bin1[i]
const b2 = chunk.bin2[i]
const value =
weights === null
? chunk.count[i]
: chunk.count[i] *
weightAt(weights, rows, cols, b1) *
weightAt(weights, rows, cols, b2)
aggregator.place(b1, b2, value)
}
},
scanFor(options, q.value),
)
return { ...aggregator.finish(), contigsPerCell: aggregator.perCell }
}
/**
* A block read at a **coarser grid** than the file stores: consecutive bins of
* one contig, or whole contigs, summed into one cell.
*
* The same pass as {@link aggregateSparse} over the same pixels, into a
* different set of slots — see `metacooler/pooling.ts` for why the slots are
* bins rather than contigs, and why that makes the result an ordinary block
* one bin size up rather than a new kind of object. An identity plan is not
* pooling; the caller reads such a block the direct way.
*
* The output is bounded by construction: `cap` cells per axis, whatever the
* file's bin count. That is what lets a whole-genome view of a large cooler
* exist at all — the dense block it would otherwise ask for is refused by the
* browser's 2 GiB ceiling on one array, and was refused *after* the user had
* been asked to wait for it. The bound is on the cells the renderer will draw,
* not near them, so nothing downstream pools a second time.
*
* `aim` is the grid the caller has already labelled its axis with. It is a
* floor, never a target: the read may coarsen it where the sequences fall
* badly, and reports what it used.
*/
async pooledMatrix(
q: MatrixQuery,
aim: PoolPlan,
options: SpanScanOptions = {},
): Promise<
Omit<AggregatedMatrix, 'level' | 'values'> & {
/** `Float32` where the block was folded on the diagonal; see `pixelAccumulator`. */
values: Float64Array | Float32Array
binsPerCell: number
contigsPerCell: number
rowMembers: BinPooling
colMembers: BinPooling
}
> {
const rows = clampExtent(q.rows, this.nBins)
const cols = clampExtent(q.cols ?? q.rows, this.nBins)
const plan = planRead(rows, cols, this.storageMode)
const onDiagonal = rows[0] === cols[0] && rows[1] === cols[1]
// The contig → genome map goes in so that a cell of whole sequences is a
// cell of one genome's sequences (`pooling.ts`). Empty for a plain cooler,
// where there is no coarser structure and a chromosome is the only unit.
const groups = this.contigGenomeIds
const rowPool = poolBins(this.coords, rows[0], rows[1], aim, POOLED_CELL_CAP, groups)
const colPool = onDiagonal
? rowPool
: poolBins(this.coords, cols[0], cols[1], aim, POOLED_CELL_CAP, groups)
// Narrowing: this block goes to the wire as `Float32` whatever it is summed
// in, and on the diagonal — the whole-file read — the triangle form is what
// keeps the default level inside the memory the other two levels use.
const acc = pixelAccumulator(
rows,
cols,
rowPool,
colPool,
plan.mode,
this.storageMode === 'symmetric-upper',
true,
)
// Weights are read once per axis for the bins on screen, exactly as the
// dense and roll-up paths do — the one bounded read of `/bins` (§6.4).
const weights = q.balance === true ? this.weightSlices(rows, cols) : null
// Watched rather than raced through: this is the read the whole-genome view
// makes, its span is the whole file, and a person waiting on it is owed a
// bar and a way out (`forEachPixelSpanAsync`).
await this.store.forEachWindow(
rows,
cols,
plan,
(chunk) => {
for (let i = 0; i < chunk.length; i++) {
const b1 = chunk.bin1[i]
const b2 = chunk.bin2[i]
const value =
weights === null
? chunk.count[i]
: chunk.count[i] *
weightAt(weights, rows, cols, b1) *
weightAt(weights, rows, cols, b2)
acc.place(b1, b2, value)
}
},
scanFor(options, q.value),
)
return {
...acc.finish(),
binsPerCell: rowPool.stride,
contigsPerCell: Math.max(rowPool.contigsPerCell, colPool.contigsPerCell),
rowMembers: rowPool,
colMembers: colPool,
}
}
/**
* One map of **several chosen sequences**, and nothing else.
*
* The picture a reader asks for by ticking four MAGs out of two thousand:
* their own maps on the diagonal, everything between them off it, in the
* order they were given. It is not a range — the four are scattered through
* the file — so it is built the way the overview is, by walking pixels and
* dropping the ones whose bins are not on either axis
* (`selectionPooling`).
*
* Which makes it the whole file's worth of reading whatever is selected, and
* that is why it is watched: the same bar and the same cancel as the
* whole-genome read, through the same options.
*
* @param units Selected sequences as `(id, bin extent)`, in display order.
*/
async selectionMatrix(
units: ReadonlyArray<{ id: number; extent: readonly [number, number] }>,
aim: PoolPlan,
options: SpanScanOptions & {
balance?: boolean
cellCap?: number
/**
* Keep every pooled cell inside one genome (`selectionPooling`).
*
* Set where the selection is *of* genomes, because that picture is drawn
* with the genome boundaries as its dividers and a cell holding the tail
* of one MAG beside the head of the next puts a divider a cell late. A
* selection of contigs asks for no such thing: its axis is the choice.
*/
byGenome?: boolean
} = {},
): Promise<
Omit<AggregatedMatrix, 'level'> & {
binsPerCell: number
contigsPerCell: number
ids: Int32Array
pooling: BinPooling
}
> {
// `cellCap` is the caller's ceiling on the *picture*, which is what a memory
// budget resolves to here: the block is square, so a byte budget divides
// into a cells-per-axis cap and nothing else has to change. Absent means
// `selectionPooling`'s own cap, which is the raster's.
const groups = options.byGenome === true ? this.contigGenomeIds : undefined
const pooling = selectionPooling(
this.nBins,
units,
aim,
options.cellCap === undefined ? POOLED_CELL_CAP : options.cellCap,
groups,
)
// On the diagonal by construction: one axis, used twice. `planRead` decides
// what that means for the store rather than this function assuming it — a
// `square` file holds both triangles and each stored pixel is its own cell,
// an upper-triangular one holds half and the result is mirrored at the end.
const whole: Extent = [0, this.nBins]
const acc = pixelAccumulator(
whole,
whole,
pooling,
pooling,
planRead(whole, whole, this.storageMode).mode,
this.storageMode === 'symmetric-upper',
)
const weights = options.balance === true ? this.weightSlices(whole, whole) : null
await this.store.forEachBands(
units.map(({ extent }) => [extent[0], extent[1]] as Extent),
(chunk) => {
for (let i = 0; i < chunk.length; i++) {
const b1 = chunk.bin1[i]
const b2 = chunk.bin2[i]
const value =
weights === null
? chunk.count[i]
: chunk.count[i] * weights.rows[b1] * weights.cols[b2]
acc.place(b1, b2, value)
}
},
options,
)
const out = acc.finish()
return {
...out,
binsPerCell: pooling.stride,
contigsPerCell: pooling.contigsPerCell,
ids: pooling.ids,
pooling,
}
}
/**
* This map through an edit of its reference (§27.2, §27.7): the edited
* sequences as its own, and every pixel carried to the edited bins it lands
* at — see `metacooler/lens.ts`. Everything above sees an ordinary map.
*
* A new map over the same handle and the same store, so the file is read
* exactly as it was; closing it closes the file, and the map it was made
* from is not closed again. The map as it was where the edit changes
* nothing, and a refusal, with the map as it was, where it cannot be drawn
* here: an edit made on another assembly, or a container whose sequences
* are not one list — a metagenome's genome layer and a single-cell file's
* cells are not yet drawn through one.
*/
static withLens(mc: MetaCooler, edit: ReferenceEdit): { map: MetaCooler; refusal: string | null } {
// One genome's edit is a map of that genome alone, drawn through it
// (§27.10); the whole file's, on a metagenome, is not drawn yet.
const ofGenome = edit.original.genome !== undefined
if (mc.hasGenomes && !ofGenome) return { map: mc, refusal: 'An edit is not yet drawn on a map of several genomes.' }
if (mc.layout.kind === 'scool') return { map: mc, refusal: 'An edit is not yet drawn on a single-cell file.' }
const found = lensFor(edit, mc.chromnames, mc.contigLengths, { always: ofGenome })
if (found === null) return { map: mc, refusal: null }
if ('refusal' in found) return { map: mc, refusal: found.refusal }
const lens = BinLens.of(mc.coords, mc.chromnames, found.assembly)
const table = sequenceTable(lens)
const attrs = { ...mc.rawHandle.attrs(), nchroms: lens.names.length, nbins: lens.nBins }
const map = new MetaCooler({
uri: mc.uri,
handle: lensedHandle(mc.rawHandle, lens, table, attrs),
layout: mc.layout,
attrs,
contigs: table,
chromOffset: lens.chromOffset,
store: new LensStore(mc.store, lens, mc.storageMode === 'symmetric-upper'),
unlensed: mc,
})
return { map, refusal: null }
}
/**
* This map at a resolution the reader added (§3.4): every `factor` of its bins
* along each sequence summed into one, as the map is drawn there — see
* `metacooler/coarseMap.ts`. Everything above sees an ordinary map at the
* coarser bin size, which is what lets a task measure the map the reader is
* looking at rather than the rung the file stores (§18.4).
*
* A new map over the same handle and the same pixels, as {@link withLens}
* makes one, and made after it and after a resample, so that it sums the map
* as drawn: closing it closes the file. The map as it was for a factor of one,
* and a refusal, with the map as it was, for a container of cells, whose cell
* set is chosen on the map that reads it rather than through one.
*/
static coarsened(mc: MetaCooler, factor: number): { map: MetaCooler; refusal: string | null } {
if (!Number.isInteger(factor) || factor < 1) {
return { map: mc, refusal: `A resolution is a whole number of this map's bins, not ${factor} of them.` }
}
if (factor === 1) return { map: mc, refusal: null }
if (mc.layout.kind === 'scool') {
return { map: mc, refusal: 'A single-cell file is not yet measured at a resolution you added; measure it at one it stores.' }
}
const grid = CoarseGrid.of(mc.chromOffset, factor)
// `nnz` is the fine table's length, and the coarse map's is not known
// without summing every block: left out, as a file that does not say is.
const { nnz: _fine, ...kept } = mc.handle.attrs()
void _fine
const attrs = { ...kept, 'bin-size': mc.binsize * factor, nbins: grid.nBins }
const map = new MetaCooler({
uri: mc.uri,
handle: coarsenedHandle(mc.handle, grid, attrs),
layout: mc.layout,
attrs,
contigs: mc.contigs,
chromOffset: grid.chromOffset,
...(mc.genomes === null ? {} : { genomes: mc.genomes, genomeOffset: mc.genomeOffset }),
store: new CoarseStore(mc.store, grid, mc.storageMode),
...(mc.unlensed === null ? {} : { unlensed: mc.unlensed }),
})
return { map, refusal: null }
}
/**
* The stage's whole-genome picture of this map (§27.9): every cell of it,
* `stride` bins a side, one byte each, with the pyramid over it.
*
* A pass over every stored pixel, a band of rows at a time, so the floats in
* hand never outnumber one band — see `metacooler/picture.ts`. Watched like
* the whole-genome read, with the same bar and the same way out.
*
* @returns `null` where the caller cancelled.
*/
async picture(
layout: PictureLayout,
options: SpanScanOptions & { balance?: boolean; gather?: GatherOptions } = {},
): Promise<{
levels: PictureLevel[]
quantizer: Quantizer
histogram: Float64Array
pixels: number
contacts?: PictureContacts
} | null> {
const whole: Extent = [0, this.nBins]
const weights = options.balance === true ? this.weightSlices(whole, whole).rows : null
const read = await readPicture({
store: this.store,
coords: this.coords,
layout,
weights,
...(options.value === undefined ? {} : { value: options.value }),
...(options.gather === undefined ? {} : { gather: options.gather }),
...(options.cancelled === undefined ? {} : { cancelled: options.cancelled }),
...(options.onProgress === undefined ? {} : { onProgress: options.onProgress }),
})
if (read === null) return null
const levels = pyramid({ n: layout.n, cells: read.cells }, read.quantizer)
return {
levels,
quantizer: read.quantizer,
histogram: read.histogram,
pixels: read.pixels,
...(read.contacts === undefined ? {} : { contacts: read.contacts }),
}
}
/**
* The `/bins` columns this file carries, with the kind of each — names and
* dtypes, never a value (§8.12).
*
* One group listing and one `dtype` per column, which is what makes it safe
* to ask at open time for a drop-down that may never be opened: §3.3's rule
* is that `/bins` is never *read*, and this reads nothing. The weight column
* is in the list like any other, because to a reader it is exactly what this
* feature is for — the balancing decision, drawn beside the map it was made
* for.
*/
binColumns(): Array<{ name: string; text: boolean }> {
if (!this.tables.has('bins')) return []
const out: Array<{ name: string; text: boolean }> = []
for (const name of this.tables.keys('bins')) {
// A column whose dtype this reader cannot interpret — a compound, a
// reference, an enum — is left out rather than reported as a number.
// Nothing downstream could draw it, and the honest place to decline is
// the list rather than the lane.
const text = columnIsText(this.tables.dtype(`bins/${name}`))
if (text !== null) out.push({ name, text })
}
return out
}
/**
* One `/bins` column over one half-open bin range.
*
* Bounded exactly as the weight read is, and for the same reason: the block
* on screen is at most a few thousand cells however large the file is, so a
* lane over it never needs more of the column than the picture covers. A
* whole-file column read would be as long as the file's bin count, for a
* picture that is going to be four thousand pixels wide.
*/
binColumn(column: string, lo: number, hi: number): Float64Array | string[] {
const path = `bins/${column}`
if (!this.tables.has(path)) {
throw new ViewerError(
'unsupported',
`This file has no /${path} column.`,
{ column },
)
}
const from = Math.max(0, Math.min(lo, this.nBins))
const to = Math.max(from, Math.min(hi, this.nBins))
if (column === DEFAULT_WEIGHT_COLUMN && this.tables === this.rawHandle) {
// Through the handle, so a resampled map's weights are on its scale.
return this.handle.readNumeric(path, from, to)
}
return columnIsText(this.tables.dtype(path)) === true
? this.tables.readStrings(path, from, to)
: this.tables.readNumeric(path, from, to)
}
/** The two bounded weight slices a balanced roll-up multiplies by. */
private weightSlices(rows: Extent, cols: Extent): { rows: Float64Array; cols: Float64Array } {
const path = `bins/${DEFAULT_WEIGHT_COLUMN}`
if (!this.handle.has(path)) {
throw new ViewerError(
'unsupported',
`This file has no /${path} column, so there are no weights to balance with. ` +
'Run `cooler balance` on it, or view the raw counts.',
{ column: DEFAULT_WEIGHT_COLUMN },
)
}
const rowWeights = this.handle.readNumeric(path, rows[0], rows[1])
const colWeights =
cols[0] === rows[0] && cols[1] === rows[1]
? rowWeights
: this.handle.readNumeric(path, cols[0], cols[1])
return { rows: rowWeights, cols: colWeights }
}
aggregate(m: DenseMatrix, level: AggregationLevel): AggregatedMatrix {
return aggregateMatrix(
m,
this.coords,
this.contigGenomeIds,
level,
this.storageMode === 'symmetric-upper',
)
}
// -------------------------------------------------------------------------
// Cells (spec §3.8)
// -------------------------------------------------------------------------
/** Every cell of an `.scool`, in file order; `[]` for any other container. */
get cells(): string[] {
return this.layout.kind === 'scool' ? (this.layout.entries ?? []) : []
}
/**
* The cells currently summed into this map.
*
* Equal to {@link cells} until something ticks a subset, and `[]` on a file
* that has none — which is what lets every caller ask without branching on
* the container first.
*/
get activeCells(): string[] {
return this.pixelStore instanceof ScoolStore ? this.pixelStore.names : []
}
/**
* Sum a different set of cells, in place. `null` is every cell.
*
* A swap rather than a re-open, and the difference is what makes ticking
* boxes feel like ticking boxes: the mount, the chrom table, the coordinate
* system and the bin space are all shared by every cell, so what actually
* changes is which pixel tables get walked. The cost is one small index read
* per newly selected cell.
*/
selectCells(names: readonly string[] | null): void {
if (this.layout.kind !== 'scool') {
throw new ViewerError(
'unsupported',
`${this.uri} is not a single-cell container, so it has no cells to choose.`,
{ layout: this.layout.kind },
)
}
const resolved = resolveCells(this.cells, names)
this.pixelStore = new ScoolStore(this.handle, this.layout, resolved, this.nBins, this.countKind)
}
/**
* Every selected cell's pixels, one cell at a time (§3.8).
*
* The read the per-cell QC task is built on, and it is here rather than in the
* task because {@link ScoolStore} is private to this class: a runner asking
* `mc.pixelStore instanceof ScoolStore` would be reaching through the seam
* `store.ts` exists to draw.
*
* Refused by name on any other container, rather than answered with an empty
* walk. A cooler has no cells, and a task that asked one for them and got
* silence would report a clean table of nothing.
*/
async scanCells(options: PerCellScanOptions): Promise<void> {
if (!(this.pixelStore instanceof ScoolStore)) {
throw new ViewerError(
'unsupported',
`${this.uri} is not a single-cell container, so it has no cells to walk.`,
{ layout: this.layout.kind },
)
}
await this.pixelStore.scanPerCell(options)
}
/**
* One row per cell, with the columns its attributes actually distinguish.
*
* `name`, `nnz` and `sum` first, because every cooler writes the latter two
* and they are what a reader ranks cells by — a single-cell library is a
* distribution of depths, and the useful ones are at one end of it. After
* them, any other attribute that **differs** across cells, discovered rather
* than listed: a writer that records a phase, a batch or a QC call per cell
* gets a sortable column for free.
*
* A constant attribute is deliberately not a column — see {@link CellsResult}.
* `metadata` is dropped whatever it does, being a JSON blob rather than a
* cell of a table.
*/
cellTable(): Table {
const names = this.cells
const rows = names.map((name) => this.handle.attrsAt(cellPath(this.layout, name)))
const columns: string[] = ['name']
const data: Record<string, Column> = { name: names }
const discovered = new Set<string>()
for (const attrs of rows) for (const key of Object.keys(attrs)) discovered.add(key)
// Order: the two guaranteed numbers, then whatever else varies, by name, so
// the table is stable across files rather than following HDF5's iteration.
const rest = [...discovered]
.filter((key) => !CELL_TABLE_LEADING.includes(key) && key !== 'metadata')
.sort()
for (const key of [...CELL_TABLE_LEADING, ...rest]) {
if (!discovered.has(key)) continue
const values = rows.map((attrs) => attrs[key])
// A column every row agrees on describes the file, not the cell.
if (!CELL_TABLE_LEADING.includes(key) && values.every((v) => v === values[0])) continue
const numeric = values.every((v) => typeof v === 'number' || typeof v === 'boolean')
if (numeric) {
data[key] = Float64Array.from(values, (v) => Number(v))
} else {
data[key] = values.map((v) => (v === null || v === undefined ? '' : String(v)))
}
columns.push(key)
}
return makeTable(names.length, columns, data)
}
// -------------------------------------------------------------------------
// The full scan — gated, never automatic (spec §4, §7.3)
// -------------------------------------------------------------------------
/** Total stored pixels, from the dataset shape. No pixel is read. */
get nnz(): number {
return this.store.nnz ?? 0
}
/**
* Stream every non-zero pixel, in chunks.
*
* **This is the one operation that costs the whole file.** Summing all
* genome pairs is inherently a full pixel scan — there is no index that
* shortcuts it — which is why the genome × genome overview is gated rather
* than being the landing view. Returning `false` from `onChunk` stops the
* scan, and that is what makes the gate cancellable.
*/
async scanPixels(options: ScanOptions): Promise<void> {
await this.store.scanAll(options as ScanAllOptions)
}
/**
* The deep integrity check, offered as an explicit debug action only (§6.3).
*
* This is `check --deep`'s pixel scan (`_deep_pixel_problems`) plus the
* offset-array invariants of `_check_offset_array`: every pixel in range,
* upper-triangular, sorted by `(bin1_id, bin2_id)`, owned by the
* `bin1_offset` slot that claims it, and totalling `attrs['sum']`.
*
* It is deliberately **not** a port of the whole Python `validate()`: that
* function reads three `/bins` columns end to end, and this reader never
* reads `/bins` (§3.3). `note` says so, so a user running the debug action
* knows exactly which invariants went unchecked rather than assuming a
* clean report means the file is perfect.
*/
async validateDeep(options?: {
/** Polled between chunks; `true` abandons the scan with an `AbortError`. */
cancelled?: () => boolean
onProgress?: (done: number, total: number) => void
}): Promise<DeepValidation> {
const problems: string[] = []
const reported = new Set<string>()
const once = (kind: string, message: string): void => {
if (!reported.has(kind)) {
reported.add(kind)
problems.push(message)
}
}
const nBins = this.nBins
const square = this.storageMode === 'square'
const bin1Offset = this.binOffsets()
const total = this.nnz
if (bin1Offset.length !== nBins + 1) {
problems.push(`bin1_offset length (${bin1Offset.length}) != ${nBins + 1}`)
} else if (bin1Offset[0] !== 0 || bin1Offset[nBins] !== total) {
problems.push(
`bin1_offset endpoints (${bin1Offset[0]}, ${bin1Offset[nBins]}) do not bracket the ` +
`table (0, ${total})`,
)
}
for (let i = 1; i < bin1Offset.length; i++) {
if (bin1Offset[i] < bin1Offset[i - 1]) {
once(
'monotone',
`bin1_offset is not non-decreasing: bin1_offset[${i - 1}]=${bin1Offset[i - 1]} > ` +
`bin1_offset[${i}]=${bin1Offset[i]}`,
)
break
}
}
let sum = 0
const integral = this.countKind !== 'f'
// Scalars, not a two-element array per pixel: at a reference-scale `nnz` of
// 3e8 that was 300 million short-lived objects, forcing a continuous
// scavenge through a loop that is otherwise allocation-free. Measured
// 14.2 ns/pixel against 4.6.
let prev1 = -1
let prev2 = -1
// A *walking* owner pointer rather than a binary search per pixel. Rows
// arrive in order, so the slot only ever advances; the search re-derived
// from scratch what a single incrementing counter already knows, at
// 33.5 ns/pixel against 7.3.
const indexed = bin1Offset.length === nBins + 1
let owner = 0
let row = 0
const cancelled = options?.cancelled
// The stored pixels, never a resampled draw of them: this reconciles the
// file against its own `attrs['sum']`.
await this.pixelStore.scanAll({
onProgress: options?.onProgress,
// The rows as `pixels/bin1_id` stores them: the index is what they are
// checked against below, so they cannot be taken from it.
storedRows: true,
onChunk: (chunk) => {
if (cancelled?.() === true) return false
for (let i = 0; i < chunk.length; i++, row++) {
const b1 = chunk.bin1[i]
const b2 = chunk.bin2[i]
const v = chunk.count[i]
if (integral) sum += v
if (b1 < 0 || b2 < 0 || b1 >= nBins || b2 >= nBins) {
once('range', `pixel bin ids out of range [0, ${nBins}) at row ${row}`)
}
if (!square && b1 > b2) {
once(
'triangle',
`pixels are not upper-triangular (bin1_id <= bin2_id): row ${row} is (${b1}, ${b2})`,
)
}
if (prev1 >= 0 && !(prev1 < b1 || (prev1 === b1 && prev2 <= b2))) {
once('order', `pixels are not sorted by (bin1_id, bin2_id) at row ${row}`)
}
if (indexed) {
while (owner < nBins && bin1Offset[owner + 1] <= row) owner++
if (owner !== b1) {
once(
'index',
`bin1_offset disagrees with pixels/bin1_id at row ${row}: the index places it in ` +
`bin ${owner}, the pixel says ${b1}`,
)
}
}
prev1 = b1
prev2 = b2
}
return true
},
})
if (cancelled?.() === true) {
const err = new Error('cancelled')
err.name = 'AbortError'
throw err
}
// Only integer counts are totalled: attrs['sum'] is an integer contact
// count, so there is nothing to reconcile a float count column against.
const declaredSum = this.info.sum
if (integral && declaredSum !== null && declaredSum !== sum) {
problems.push(`attrs['sum'] (${declaredSum}) != the sum of pixels/count (${sum})`)
}
return {
ok: problems.length === 0,
problems,
note:
`Scanned ${total} pixels for range, upper-triangle, sort order and bin1_offset ` +
`agreement${integral ? ", and reconciled attrs['sum']" : ' (float counts: sum not reconciled)'}. ` +
`The /bins geometry checks of the Python validate() are not run — this reader never ` +
`reads /bins.`,
}
}
// -------------------------------------------------------------------------
// Validation (spec §6.3) and teardown
// -------------------------------------------------------------------------
/**
* The light checks: attributes, group presence, table lengths, the `/chroms`
* soft link, and declared-vs-actual counts. All O(1) or O(n_columns).
*
* The full Python `validate()` reads whole `bins` columns, the whole
* `genome_id` column, both offset arrays and three pixel windows — fine on
* local disk, unacceptable over HTTP — so it belongs behind an explicit
* debug action, not in the open path.
*/
private validateLight(): void {
const fail = (message: string, detail: Record<string, unknown> = {}): never => {
throw new MetaCoolerFormatError(message, detail)
}
// Shared with every cooler: a contig table, its columns, and an offset
// array that brackets it.
if (this.nContigs === 0) fail('/contigs table is empty')
for (const column of REQUIRED_COOLER_COLUMNS) {
if (!this.contigs.columns.includes(column)) fail(`Missing column in /contigs: ${column}`)
}
if (this.chromOffset.length !== this.nContigs + 1) {
fail(`chrom_offset length (${this.chromOffset.length}) != ${this.nContigs + 1}`)
}
if (this.chromOffset[0] !== 0) {
fail(`chrom_offset endpoints (${this.chromOffset[0]}, ${this.nBins}) do not bracket the table`)
}
// The genome layer, only where there is one. A plain cooler fails the
// soft-link check by construction — its `/chroms` *is* the real group, and
// the check exists to prove a metacool is readable as one of these.
if (this.genomes !== null) {
if (this.nGenomes === 0) fail('/genomes table is empty')
for (const column of REQUIRED_GENOME_COLUMNS) {
if (!this.genomes.columns.includes(column)) fail(`Missing column in /genomes: ${column}`)
}
if (!this.contigs.columns.includes('genome_id')) fail('Missing column in /contigs: genome_id')
if (!this.handle.isSoftLink('chroms')) {
fail('/chroms must be a soft link to /contigs for cooler compatibility')
}
if (this.genomeOffset.length !== this.nGenomes + 1) {
fail(`genome_offset length (${this.genomeOffset.length}) != ${this.nGenomes + 1}`)
}
if (this.genomeOffset[0] !== 0 || this.genomeOffset[this.nGenomes] !== this.nContigs) {
fail(
`genome_offset endpoints (${this.genomeOffset[0]}, ${this.genomeOffset[this.nGenomes]}) ` +
`do not bracket the table (0, ${this.nContigs})`,
)
}
}
// Declared counts must agree with what is actually there. `nnz` is read
// from the dataset *shape*, which costs nothing and never touches a pixel.
const declared: [string, number | null, number, string][] = [
['nchroms', this.info.ncontigs, this.nContigs, 'len(/contigs)'],
...(this.genomes === null
? []
: ([['ngenomes', this.info.ngenomes, this.nGenomes, 'len(/genomes)']] as [
string,
number | null,
number,
string,
][])),
['nbins', this.info.nbins, this.nBins, 'len(/bins)'],
['nnz', this.info.nnz, this.tables.datasetLength('pixels/bin1_id'), 'len(/pixels)'],
]
for (const [name, value, actual, label] of declared) {
if (value !== null && value !== actual) {
fail(`attrs[${pyRepr(name)}] (${value}) != ${label} (${actual})`)
}
}
}
// -------------------------------------------------------------------------
// Resampling (§6.5)
// -------------------------------------------------------------------------
/**
* Draw this map to `spec.contacts`, or put back the file as stored (`null`).
*
* The resampled pixels replace the stored ones behind {@link store} at once,
* before the census is taken, so a read already queued behind this waits for
* the draw rather than reading the file as stored in between. A refusal —
* counts that are not contacts, a file that stores both triangles, a
* container of cells, a sequence Boost-HiC cannot walk — stays installed and
* is what every read then answers with: a map drawn from the stored counts
* under a header that says otherwise would be the one wrong answer.
*
* `census` is a census another mount of this same map already took; it is
* checked against this map's bins and used rather than walking every pixel
* again.
*/
async resample(
spec: ResampleSpec | null,
options: {
census?: Census
onProgress?(done: number, total: number): void
cancelled?(): boolean
} = {},
): Promise<ResamplePlan | null> {
if (spec === null) {
this.resampled = null
return null
}
const refusal = this.resampleRefusal()
// Waited on now, so a census begun in the background stops standing aside.
this.censusWanted = true
if (options.census !== undefined) this.census = Promise.resolve(options.census)
let census: Promise<Census>
if (refusal !== null) {
census = Promise.reject(refusal)
} else {
this.census ??= takeCensus(this.pixelStore, this.chromOffset, {
...(options.onProgress === undefined ? {} : { onProgress: options.onProgress }),
cancelled: () => this.closed || options.cancelled?.() === true,
})
census = this.census
// A census that was called off is not an answer about the map: the next
// request walks it again rather than inheriting the cancellation.
census.catch((error: unknown) => {
if (error instanceof DOMException && error.name === 'AbortError' && this.census === census) {
this.census = null
}
})
}
const store = new ResampledStore(
this.pixelStore,
{ starts: this.chromOffset, names: this.chromnames },
spec,
census,
)
this.resampled = store
return store.ready()
}
/**
* The contacts this map says it holds, before anything is drawn from it:
* cooler's `attrs['sum']`, a pairs import's records. `null` where the file
* does not say — an old cooler written before `sum` was an attribute, a
* container of cells — and **for a `.hic`**, whose per-matrix `sumCounts`
* counts every contact off the diagonal of a chromosome twice (the matrix both
* ways round) and so is not the number of contacts at all. A census counts
* those ({@link countContacts}).
*/
get declaredContacts(): number | null {
const sum = this.info.sum
if (typeof sum === 'number' && Number.isFinite(sum) && sum >= 0) return sum
return null
}
/**
* The contacts this map holds as stored: as the file says where it says so
* exactly, and counted by the census otherwise — the one a resample takes,
* kept for it. `null` for a map whose stored values are not contacts.
*/
async countContacts(
options: {
onProgress?(done: number, total: number): void
/**
* What the census waits on between steps while nothing else needs it
* (§12); handed a test that says when something has come to (a resample),
* which must end the wait.
*/
pause?(wanted: () => boolean): Promise<void>
} = {},
): Promise<number | null> {
const declared = this.declaredContacts
if (declared !== null) return declared
if (this.resampleRefusal() !== null) return null
const pause = options.pause
this.census ??= takeCensus(this.pixelStore, this.chromOffset, {
...(options.onProgress === undefined ? {} : { onProgress: options.onProgress }),
...(pause === undefined ? {} : { pause: () => pause(() => this.censusWanted) }),
cancelled: () => this.closed,
})
try {
return (await this.census).total
} catch {
return null
}
}
/** The census, where one has been taken and has finished. */
async takenCensus(): Promise<Census | null> {
if (this.census === null) return null
try {
return await this.census
} catch {
return null
}
}
/** Why this map cannot be resampled at all, or `null`. */
private resampleRefusal(): ViewerError | null {
if (this.layout.kind === 'scool') {
return new ViewerError(
'unsupported',
'A single-cell container’s map is a sum of its cells rather than one library, so it is not ' +
'resampled as one. Open one cell, or a pseudo-bulk written as a cooler, to resample it.',
{ resample: true, layout: 'scool' },
)
}
if (this.storageMode !== 'symmetric-upper') {
return new ViewerError(
'unsupported',
'This map stores both triangles, so each contact would be drawn twice. Resampling reads ' +
'upper-triangular files — cooler’s default.',
{ resample: true, storageMode: this.storageMode },
)
}
return null
}
/** Release the handle and its byte source. Idempotent. */
close(): void {
if (this.closed) return
this.closed = true
this.rawHandle.close()
}
}
/**
* `handle` with `bins/weight` multiplied by `scale`, and nothing else touched.
*
* A proxy rather than a wrapper class because a handle is an interface with a
* dozen methods and one of them differs; every other call is forwarded to the
* handle itself, bound to it.
*/
function scaledWeights(handle: Hdf5Handle, scale: number): Hdf5Handle {
const path = `bins/${DEFAULT_WEIGHT_COLUMN}`
return new Proxy(handle, {
get(target, property) {
if (property === 'readNumeric') {
return (at: string, lo?: number, hi?: number): Float64Array => {
const values = target.readNumeric(at, lo, hi)
if (at !== path) return values
const out = new Float64Array(values.length)
for (let i = 0; i < values.length; i++) out[i] = values[i] * scale
return out
}
}
const value: unknown = Reflect.get(target, property, target)
return typeof value === 'function' ? (value as (...args: unknown[]) => unknown).bind(target) : value
},
})
}
/**
* A numeric column as a `Float64Array`, without the intermediate JS array.
*
* `Table.numbers` returns `number[]`, which is the right shape for the handful
* of small columns the UI formats — and the wrong one for a 154,000-row index
* the reader is about to convert anyway. Every numeric column arrives from the
* decoder as a `Float64Array`, so the common case is a straight hand-back.
*/
function toFloat64(table: Table, column: string): Float64Array {
const raw = table.data[column]
if (raw instanceof Float64Array) return raw
return Float64Array.from(table.numbers(column))
}
/**
* Whether a dataset holds text, or `null` where its dtype is not a scalar one.
*
* A total function over a partial one: {@link dtypeKind} throws for a compound
* or reference datatype, which is the right answer for a required column being
* validated and the wrong one for a *discovery* pass over whatever a creator
* happened to put in `/bins` (§8.12). One odd column should cost that column,
* not the listing.
*/
function columnIsText(dtype: string): boolean | null {
try {
return 'SOU'.includes(dtypeKind(dtype))
} catch {
return null
}
}
/** The same, narrowing to `Int32Array` — for id columns, which are indices. */
function toInt32(table: Table, column: string): Int32Array {
const raw = table.data[column]
if (ArrayBuffer.isView(raw) && !(raw instanceof DataView) && !(raw instanceof BigInt64Array)) {
const out = new Int32Array(raw.length)
out.set(raw as unknown as ArrayLike<number>)
return out
}
return Int32Array.from(table.numbers(column))
}
/**
* Scan options carrying the query's own value column.
*
* The streaming reads take their column through `SpanScanOptions` and the dense
* read takes it through `MatrixQuery`; this is the one line that joins them, and
* it exists so that no caller of the pooled or rolled-up path can forget. An
* absent column is left absent rather than defaulted here, so the default stays
* in one place (`DEFAULT_COUNT_COLUMN`).
*/
function scanFor(options: SpanScanOptions, value: string | undefined): SpanScanOptions {
return value === undefined ? options : { ...options, value }
}
/** A bin range clamped onto the file, matching `readDenseMatrix`'s validation. */
function clampExtent(extent: Extent, nBins: number): Extent {
const lo = Math.min(Math.max(Math.trunc(extent[0]), 0), nBins)
const hi = Math.min(Math.max(Math.trunc(extent[1]), lo), nBins)
return [lo, hi]
}
/**
* The weight of an absolute bin, from whichever axis slice covers it.
*
* A pixel contributes through both of its bins, and the two can fall on
* different axes of a rectangular block. Outside both slices the weight is `1`,
* which cannot happen for a pixel the aggregator will place — every placement is
* range-checked first — and is the identity if it ever did.
*/
function weightAt(
weights: { rows: Float64Array; cols: Float64Array },
rows: Extent,
cols: Extent,
bin: number,
): number {
if (bin >= rows[0] && bin < rows[1]) return weights.rows[bin - rows[0]]
if (bin >= cols[0] && bin < cols[1]) return weights.cols[bin - cols[0]]
return 1
}
/**
* Refuse to list a container's cells over a source that cannot afford it (§12).
*
* Every other open in this application is a bounded read, and this is the one
* that is not: a container's cell names live in HDF5 link storage that writers
* interleave with the cells' own data, so reading them is a thousand small
* requests spread over the whole object. Measured on a 3,882-cell file: 1,015
* reads averaging 512 bytes, uniformly distributed across 327 MB. Neither
* chunk size wins — at the default 1 MiB the listing fetches the entire object,
* and small enough to avoid that is small enough to need a thousand round trips.
*
* So the reader is asked, which is what this refusal is: a question with the
* numbers in it, thrown where the cost is about to be incurred and caught by
* the shell, which re-opens with the answer on the source (`SourceSpec.reading`).
* A source that already holds the whole object in memory — a dropped file, or a
* download the reader has just agreed to — has nothing to be asked about and
* passes straight through.
*
* @throws ViewerError `scool-listing-cost`, carrying the object's size.
*/
function assertListingAffordable(handle: Hdf5Handle, source: ByteSource): void {
if (!source.isRemote) return
if (!willListCells(handle, handle.attrs())) return
const { path } = parseUri(handle.uri)
throw new ViewerError(
'scool-listing-cost',
`${path} is a single-cell container, and listing its cells is not a bounded read: the ` +
`names live in HDF5 link records spread across the whole object. Reading them over the ` +
`network costs either one download of ${fmt.bytes(source.size)} or a few percent of that in ` +
`about a thousand separate requests. Choose which to pay, or open a local copy, where ` +
`neither cost applies.`,
{ path, size: source.size },
)
}
/** `\x89HDF\r\n\x1a\n`, the eight bytes every HDF5 file opens with. */
const HDF5_SIGNATURE = [0x89, 0x48, 0x44, 0x46, 0x0d, 0x0a, 0x1a, 0x0a]
/**
* Whether a source carries an HDF5 superblock where the library will look.
*
* At offset 0 for every file this application writes or reads in practice;
* the format also allows a user block, in which case the signature sits at
* 512, 1024, 2048… up to the file's size. The first few are enough — a user
* block past 8 KiB is a file nobody has met — and on a remote source they are
* all inside the chunk the first read already fetched.
*/
function hasHdf5Signature(source: ByteSource): boolean {
for (const at of [0, 512, 1024, 2048, 4096, 8192]) {
if (at + 8 > source.size) return false
const head = source.readBytes(at, 8)
if (head.length === 8 && HDF5_SIGNATURE.every((byte, i) => head[i] === byte)) return true
}
return false
}
|