0views0forks
MAPS predicts whether a single amino-acid substitution is pathogenic by reading one number out of a protein language model's internals: it embeds the wild-type and the mutant sequence with ESM-C 6B, subtracts them residue by residue, and hands the 256 most-disturbed positions to a small covariance probe. On 12,565 held-out ClinVar variants that share no protein and no homology cluster with training, the two-seed ensemble of the largest training arm reaches 0.9482 AUROC. Recomputing each of the six published runs from its stored per-variant scores reproduces the recorded value to within 4.0e-05, so the numbers below are the recorded ones, checked. Two things change how they read. Resampling homology clusters rather than variants widens the mean 95% interval from 0.0077 to 0.0167, because variants cluster inside proteins and the recorded variant-level intervals treat them as independent. And the pooled figure hides the per-protein spread: among the 161 proteins with at least 20 test variants, the median AUROC is 0.9659, but 19 fall below 0.8.
Three choices carry the prediction. The first is the difference field: a single substitution barely moves a 2,560-dimensional representation, so subtracting the wild-type at the same layer is what isolates the move. The second is position selection: rather than a fixed window around the substitution, MAPS ranks residues by how far their wild-type and mutant states diverge in angle and keeps the top 256, which lets the field follow a perturbation that spreads beyond the mutated site. The third is covariance pooling: the signal is a dispersed second-order pattern across that field, and averaging the positions throws it away.
The differences are kept raw. Normalizing each position would discard the magnitude of the move, which is itself signal; the probe's matrix square root handles the conditioning instead. Splits are cut at homology clusters, because the dominant shortcut in missense pathogenicity is gene identity: pathogenic fraction varies enormously between genes, so a probe that recognizes the protein scores well without learning anything about substitutions.
Those choices were measured on an earlier, smaller benchmark: 9,841 ClinVar missense variants over 60 genes, five-fold gene-disjoint cross-validation, with ESM-C at layer 78 and Evo 2 at layer 26. Covariance pooling beat mean pooling by 0.0506 AUROC on the identical representation (0.9326 against 0.8820), and the difference field beat scoring the mutant embedding alone by 0.0423. Both gaps also appear for the genomic model, which is what makes them properties of the recipe rather than of one substrate.
Two results in the same table set the ceiling honestly. AlphaMissense (0.9629) and REVEL (0.9486) sat above every freshly trained probe on that benchmark, so "competitive with the published meta-predictors" is the defensible phrasing rather than "better than". And combining the protein-space and genome-space probes bought nothing: concatenation came in -0.0017 and ensembling -0.0063 against the better single model, both with intervals spanning zero. On coding missense the two model families largely carry the same signal.
Pooled out-of-fold AUROC with gene-bootstrap 95% intervals on 9,841 variants over 60 genes. Recorded, not recomputed: this benchmark's data was deleted, and it uses a different split and a different ESM-C layer from the held-out table below.
Covariance pooling gains 0.0506 AUROC over mean pooling and the difference field gains 0.0423 over the mutant embedding alone. The stored score at the top (0.9676) has an unknown train/test relationship to these variants, so it is not a reproduction target.
Six published runs cover three training arms crossed with two seeds. The arms differ in how much data they see: the baseline pool is 49,719 variants from the original MAPS set, while the scaled arms add the ClinVar expansion for 188,348. All six evaluate on the same 12,565 variants. Test AUROC spans 0.9442 to 0.9463.
| Arm | Seed | Training variants | Validation AUROC | Validation variants | Selected epoch | Test AUROC | 95% interval, clusters | Recorded |
|---|---|---|---|---|---|---|---|---|
| baseline | 42 | 42,122 | 0.9452 | 7,597 | 2 | 0.9448 | [0.9359, 0.9525] | 0.9448 |
| baseline | 43 | 39,736 | 0.9494 | 9,983 | 1 | 0.9447 | [0.9360, 0.9524] | 0.9447 |
| scaled | 42 | 163,772 | 0.9583 | 24,576 | 7 | 0.9454 | [0.9370, 0.9531] | 0.9454 |
| scaled | 43 | 157,619 | 0.9639 | 30,729 | 3 | 0.9455 | [0.9364, 0.9537] | 0.9455 |
| scaled + balanced | 42 | 91,180 | 0.9591 | 24,576 | 9 | 0.9463 | [0.9380, 0.9541] | 0.9463 |
| scaled + balanced | 43 | 84,122 | 0.9648 | 30,729 | 7 | 0.9442 | [0.9349, 0.9524] | 0.9442 |
| baseline | 42 and 43 | 0.9475 | [0.9390, 0.9551] | not recorded | ||||
| scaled | 42 and 43 | 0.9482 | [0.9398, 0.9559] | not recorded | ||||
| scaled + balanced | 42 and 43 | 0.9480 | [0.9396, 0.9558] | not recorded |
The two-seed ensemble is the shipped estimate and is not recorded anywhere: averaging the two seeds' pathogenic probabilities lifts the scaled arm to 0.9482, a gain of 0.0028 over its seed-42 member with a cluster-level interval of [0.0017, 0.0038] that excludes zero. Ensembling two seeds of the same recipe is therefore a real, if small, improvement.
Nearly quadrupling the training pool is not. The scaled ensemble beats the baseline ensemble by 0.0007, interval [-0.0014, 0.0029], which spans zero. The validation numbers tell the opposite story, rising from 0.9452 to 0.9648 across the arms while held-out AUROC stays flat, and that is the signal worth remembering: validation groups are carved out of training clusters, so they reward extra data from familiar protein families in a way the cluster-disjoint test set does not.
Each point is one run's AUROC on the same 12,565 held-out variants; bars are 95% intervals from resampling the 494 homology clusters. The x axis starts at 0.930, well above chance (0.5).
The three ensembles (filled) sit above their own seeds, and the arms are indistinguishable from one another: every interval overlaps every other.
The dot is the value at the epoch chosen on validation AUROC, which is the reportable number; the bar spans what test AUROC did across all 12 training epochs. Those intermediate values were logged as diagnostics, not used for selection, which the recorded histories confirm for all six runs.
Within one run, test AUROC wanders over a range of 0.0032 to 0.0155 between epochs, as wide as the differences between training arms. A single run's headline number carries that much incidental variation.
Distribution of the scaled arm's two-seed probability for 6,470 benign and 6,095 pathogenic held-out variants, in 40 bins.
Benign variants have a median score of 0.0459 and pathogenic ones 0.9492. At the natural 0.5 threshold, 8.67% of benign variants are called pathogenic and 14.63% of pathogenic ones are missed, so this is a ranking tool, not a clinical classifier.
The recorded intervals resample variants. Variants are not independent: the test set's 12,565 variants sit in 581 proteins and 494 homology clusters, a median of 11 variants per protein, and a protein either suits the probe or does not. Resampling clusters, the unit the split itself was built on, more than doubles the interval width, from a mean of 0.0077 to 0.0167. Resampling proteins gives essentially the same answer as clusters (0.0167), so the widening comes from within-protein correlation rather than from homology between proteins.
This does not move any point estimate, and it does not change which arm is best, because no arm was distinguishable to begin with. It changes how far apart two MAPS-shaped numbers have to be before the difference means anything: about 0.017 rather than 0.008.
Each run appears twice: once with the recorded variant-level interval, once with the interval from resampling homology clusters. 1,000 resamples, percentile method, seed 42.
Cluster-level intervals average 0.0167 wide against 0.0077 for variant-level, a factor of 2.16. The recomputed variant-level intervals also match the recorded ones closely (mean width 0.0079).
Pooling 12,565 variants across 581 proteins hides where the probe works. Restricting to the 161 proteins with at least 20 test variants and both labels present (8,912 variants in total), the median within-protein AUROC is 0.9659 with quartiles [0.9074, 0.9934], and 19 proteins fall below 0.8. The three largest panels straddle the pooled number: KCNQ2 scores 0.9942 over 271 variants, PAH 0.9156 over 399, and MSH2 0.8805 over 287. The failures are concentrated in proteins whose label mix is lopsided: MBD5 scores 0.6504 over 135 variants, and TNC scores 0.0 because its 57 test variants include exactly one pathogenic variant, which the probe ranks last of all 57.
One point per protein for the 161 proteins with at least 20 test variants and both labels. The x axis is log-scaled. The dashed line is chance; the dotted line is the pooled ensemble AUROC.
Most proteins sit near or above the pooled number, and the extremes at both ends belong to proteins with few variants of one class. A per-variant score is not equally trustworthy gene by gene, which matters for anyone reading a single panel.
The benchmark is ClinVar missense with review status of at least one star and a pathogenic-or-benign label, mapped onto canonical RefSeq proteins.
| Assembly stage | Variants |
|---|---|
| Missense candidates in the ClinVar expression tables | 11,050,882 |
| Distinct missense variation records | 2,518,181 |
| Labelled pathogenic or benign | 937,493 |
| Review status at least one star, distinct records | 206,005 |
| On a canonical protein with a sequence | 205,768 |
| Final labelled set | 203,742 |
| Harvested with a stored layer-80 difference field | 200,913 |
Of the 203,742 labelled variants (58,835 pathogenic, 144,907 benign) over 15,610 genes and 17,845 proteins, 200,913 have a stored layer-80 difference field. Review quality is thin at the top: 139,579 variants at one star, 59,499 at two, 4,656 at three and 8 at four.
| Split | Variants | Pathogenic | Benign | Proteins | Homology clusters |
|---|---|---|---|---|---|
| test | 12,565 | 6,095 | 6,470 | 581 | 494 |
| train, original pool | 49,719 | 25,604 | 24,115 | 1,922 | 1,603 |
| train, ClinVar expansion | 138,629 | 25,509 | 113,120 | 16,540 | 10,484 |
Three checks were rerun rather than quoted. The six score files carry identical variant sets, and that set equals the manifest's test split exactly in both directions, with labels agreeing on all 12,565 rows. Train and test share 0 proteins and 0 homology clusters. The test split is close to balanced, 6,095 pathogenic against 6,470 benign, so AUROC and average precision tell similar stories here.
One join is imperfect and worth stating, because it decided how the intervals above are computed. 7,940 harvested variants are absent from the labelled ClinVar table that carries gene symbols, which leaves 1,537 test variants without one; and 386 of the 581 test proteins carry more than one symbol string across their variants, including compound entries for overlapping genes. Gene symbol is therefore not a usable grouping key, and the resampling units here are the protein and the homology cluster, with each protein given one readable symbol for the figure above.
A pathogenicity score on its own says nothing about mechanism. The second half of MAPS is a battery of per-residue probes read at the same layer as the pathogenicity probe, on the wild-type and on the mutant sequence; the difference between the two readings, per channel, is the disruption profile that explains a score. The registry behind the readouts used here holds 153 channels: 114 InterPro family and domain detectors, 15 structural tracks, 9 post-translational-modification channels, 7 functional-site channels and 7 topology channels. Each was fitted on 22,152,685 residue positions across roughly 55,000 proteins, apart from the conservation channel's 38,279 human-only positions.
The shipped channel artifacts are final fits over all selected rows, so their own training performance says nothing about accuracy. Quality comes from separate grouped cross-validation, where folds are cut by protein and by homology cluster.
Five-fold cross-validation grouped by homology cluster on 35,609 variant positions, with the fold-level 95% interval. These runs use ESM-C layer 78, one layer below the readouts they qualify.
All nine channels are decodable, from 0.7813 for modified residue to 0.9957 for signal peptide. Decodable is not the same as informative, which the next figure separates.
Cross-validated R² for a probe on ESM-C states against the better of two model-free baselines, on the same grouped folds over 80,000 residue positions. Both baselines see only the amino acid at the position.
Every structural and conservation channel gains substantially from the model's states: contact number goes from 0.1115 to 0.8067 and conservation from 0.0472 to 0.6391. These channels read context, not chemistry.
The same comparison run over the binary channels, in an earlier pass on a different backbone (ESM2-650M, layer 33), sorts them into tiers that the disruption profile has to respect. A one-hot amino-acid baseline alone reaches 0.919 on the active-site channel against the probe's 0.945, so that channel mostly reads "is this a histidine, cysteine or aspartate". A physicochemical baseline reaches 0.959 on transmembrane spans and 0.995 on signal peptides. Six sequence-derived channels have zero or negative lift, meaning the label is simply computable from the sequence.
| Tier | Channels | Lift over the better model-free baseline | Reading |
|---|---|---|---|
| model-derived | 7 | +0.234 to +0.441 | the model's states add the signal; safe to read as a disruption channel |
| identity-leaning | 3 | +0.015 to +0.097 | mostly reads which amino acid sits there, not its role |
| recoverable from chemistry | 2 | +0.005 to +0.031 | a hydrophobicity baseline already reaches the same score |
| deterministic | 6 | -0.242 to -0.014 | the label is computable from the sequence; the probe adds nothing |
Those tiers are recorded on a different backbone and layer from the shipped ESM-C readouts, so treat them as the direction of the effect rather than exact numbers for this battery. The consequence for reading a profile is the same either way: a large move on an identity-leaning channel can be manufactured by the substitution itself, while a move on relative solvent accessibility, pLDDT, secondary structure, burial, domain membership or contact geometry reflects something the model computed about context.
Per-channel mutant-minus-wild-type change at the substituted position, divided by that channel's own standard deviation of site-level disruption. Restricted to the 19 channels that have grouped cross-validation evidence, excluding the InterPro family detectors. This illustrates the readout; it is not evidence about any channel's accuracy.
GNAI1 T48K (score 0.9999, 2 ClinVar stars) moves the burial and contact channels coherently and by 6 to 8 standard deviations: solvent accessibility up, contact number, half-sphere exposure and long-range contact fraction down. The benign GNAS A467S (score 0.0002) moves conservation and pLDDT by about 5 and leaves the geometry channels nearly untouched.
One recorded result bounds how much weight a profile can carry. In the same earlier ESM2-layer-33 pass, using the named channels' deltas as features to predict pathogenic against benign reached 0.645 AUROC for the informative channels and 0.669 for all 18, against 0.873 for a zero-shot likelihood ratio from the same model family, and adding the whole profile on top of that ratio moved it to 0.879. Projecting the raw difference onto named structural channels is lossy: the profile is a readable description of what changed at the site, not the thing that makes the prediction accurate.
The fields, probes and inputs are public, so the analysis on this page can be redone
without cluster access. All three join on variant_id.
ching-goodfire/MAPS-ClinVar-VKS-Embeddings-L80The layer-80 mutant-minus-wild-type difference fields for all 200,913 variants: the whole 12,565-variant test split and the whole 188,348-variant training pool, 262 parquet shards, 154.0 GB on disk.
ching-goodfire/MAPS-ESMC-ProbesThe six layer-80 pathogenicity probes with their metrics, 12-epoch histories and per-variant test scores, plus a 159-channel layer-78 annotation battery; manifest.csv indexes all 165 probes and load_probes.py carries the input contract.
ching-goodfire/MAPS-ClinVar-VKS-InputsSequences, labels, splits, homology cluster ids and ClinVar identifiers for the same 200,913 variants, in two normalized tables.
MAPS is the protein-space half of a pair. EVEE applies the same shape to DNA: a covariance probe on frozen Evo 2 embeddings for the prediction, a per-residue annotation battery for the mechanism, and a frontier-model step that turns the top disruptions into a written explanation grounded in named annotations and directions. MAPS borrows that last step when the deliverable includes prose rather than a score. The redundancy result above is the reason to think of them as siblings rather than as competitors: on coding missense they carry much the same signal, so the interesting question is which substrate explains a variant better, not which one scores it higher.