MAPS: a covariance probe on ESM-C difference fields, and the published artifacts to build on

Scope: same-length single amino-acid substitutions

The probe's input is the aligned difference between the wild-type and the mutant residue field, so the method is defined only where the two sequences have the same length and differ at one position. Indels, frameshifts, stop gains, splice variants and synonymous changes are out of scope, and widening the window does not bring them in: a length change breaks the position-by-position alignment the difference field is built from. The dataset cards state the same restriction.

Labels are ClinVar clinical significance collapsed to pathogenic against benign, so they carry ClinVar's submission and ascertainment biases. The review-status star rating is the only quality signal shipped alongside them, and it is null for 7,940 carried-over rows from the earlier ProteinGym-derived clinical set.

Three design choices, two of them unmeasured

The difference field rather than the mutant embedding. One substitution moves a 6-billion-parameter model's representation slightly. Subtracting the wild-type field at the same layer isolates that move instead of asking the probe to find it inside the absolute embedding. The differences are kept raw: no per-position normalisation, because the magnitude of each mutant-minus-wild-type step is part of the signal and the probe's Newton-Schulz matrix square root handles the conditioning.

Covariance pooling rather than mean pooling. The probe forms two linear views of the field, takes their cross-covariance, and reads a factored linear head off that matrix, so it sees co-occurrence between representation dimensions rather than per-dimension averages. That is the case covariance pooling exists for. It is also the claim with no published support in this artifact set: the mean-pool version of the same probe, on the same fields, is not among the shipped runs.

Top-256 positions by cosine divergence rather than a fixed window. Positions are ranked by 1 - cosine(wild-type, mutant) and the top 256 are kept in ascending position order, so the field is a variable-length slice of the mutation's ripple through the sequence rather than a neighbourhood of the substituted residue. A protein shorter than 256 residues keeps all of them. The alternative, a fixed window, changes what "the field" means, so the choice has to be stated with any result.

Gene identity is the shortcut the split has to break

Pathogenic fraction varies enormously between genes, so a probe that recognises "this looks like this protein" scores well on missense pathogenicity without learning anything about substitutions. A random variant-level split hands it exactly that, because the same protein then appears on both sides. MAPS splits by MMseqs2 sequence-identity cluster instead, so a protein and its homologs land in one split only.

The test split shares zero proteins and zero clusters with either training pool, checked in both directions from the published tables rather than taken from an upstream report. Its label balance is 6,095 pathogenic against 6,470 benign.

Two things the split does not give. The published metric files record a 95% interval but not the unit it was resampled over, so those intervals cannot be read as gene-level intervals, and variants cluster inside genes, which makes a variant-level interval too narrow. And no per-gene AUROC distribution is published, so the pooled number cannot be checked against the possibility that a few large gene families carry it.

Published pathogenicity results at layer 80

Six probes ship: three training arms crossed with two seeds. They differ in training-pool size as well as seed, from 39,736 to 163,772 variants, so the arms are not a clean seed comparison. Held-out AUROC moves by 0.0021 across all six, inside 95% intervals each about 0.0082 wide, so nothing here separates the arms: the largest training pool does not produce the best held-out number.

All six are evaluated on the same 12,565 variants with 6,095 pathogenic, and each probe has 344,450 trainable parameters. The published model card also re-derives every one of the six AUROCs from the shipped per-variant score files and reproduces the recorded value, so the metric files and the scores agree.

The earlier layer-78 pair recorded in the repository

The MAPS repository records an older pair of covariance probes at layer 78, evaluated on the same 12,565-variant test split. They belong to a different harvest with different training pools, and no run pairs the two layers under one protocol, so these numbers stand on their own rather than as a layer comparison.

The annotation battery: 159 channels at layer 78

The mechanism half is 159 per-residue probes at layer 78: 144 binary, 14 regression and 1 multiclass channel, each a logistic or ridge estimator over a standardised residue vector. They cover InterPro domains (115 channels), functional sites (17), structure (11), post-translational modifications (8) and topology (8).

The shipped artifacts are final fits on all selected rows. They are inference tools, not held-out checkpoints, so a channel's accuracy comes from separate grouped cross-validation, never from the artifact itself. The manifest is explicit about which evidence each channel has: 18 channels carry protein- and cluster-grouped 5-fold cross-validation, 139 carry a metric from the old-versus-new comparison table, and 2 carry no evaluation entry at all. Every classifier's decision threshold is 0.5, which is scikit-learn's own rule rather than a per-channel tuned value.

Those are means over 5 homology-grouped folds on 35,609 residue rows at variant positions. Three more channels of other types land in the same table: 3-state secondary structure at 0.7505 macro F1, AlphaFold pLDDT at 0.8576 R-squared, and relative solvent accessibility at 0.7898 R-squared. A rare-positive channel such as modified residue reaches 0.7813 AUROC but only 0.0832 average precision, which is the number that matters when positives are scarce.

Six channels decode well and add nothing over chemistry

An earlier generation of the same battery was audited head by head against one-hot amino acid and physicochemistry on a held-out 30%-identity split. Those probes read layer 33 of the earlier, smaller ESM-2 model rather than layer 78 of ESM-C 6B, so the numbers below are not the shipped channels' scores. The pattern they establish is what carries over: raw accuracy and embedding-attributable signal are different quantities, and six channels that decode well add nothing over chemistry.

Three specifics worth carrying into any use of the battery. Active site decodes at 0.945 while one-hot amino-acid identity alone reaches 0.919, and at layer 0 of that model, before any contextual computation, it already scores 0.885 against a best-layer 0.891. That is what reading "is this a histidine, cysteine or aspartate" looks like. Transmembrane and signal peptide decode near-perfectly, but a physicochemistry-only baseline reaches 0.959 and 0.995, so most of that is hydrophobicity. And the disulfide channel's headline is inflated by conditioning: its unconditioned one-hot baseline is 0.543, but restricted to cysteines the same baseline sits at 0.500, exactly chance, so the channel has to be re-scored within cysteines to mean anything.

The disruption profile is a lossy explainer, not a predictor

The existential check for the mechanism half is whether the named-channel deltas separate pathogenic from benign on their own. In the recorded benchmark they do not, by a wide margin. The 10 informative channel deltas reach 0.645 AUROC and all 18 reach 0.669, while an ESM-1v zero-shot log-likelihood ratio, which needs no training at all, reaches 0.873 on the same variants under the same grouped protocol. Adding the whole profile on top of the zero-shot ratio moves it by +0.006, to 0.879. The best single channels are relative solvent accessibility at 0.60, domain membership at 0.59 and binding site at 0.59.

The useful contrast comes from the annotation cross-validation table: a probe trained on the mutant-minus-wild-type difference, on ESM-C layer-78 states under the same grouped cross-validation, reaches 0.9089 AUROC (5 folds, 38,960 variants). The raw difference predicts pathogenicity far better than its decomposition into named structural channels, which makes the decomposition lossy: projecting the difference onto solvent accessibility, secondary structure and site membership discards most of what separates the classes.

That does not disqualify the profile as a mechanism lens. It forbids selling it as a predictor, and it points at the missing axes: pathogenic missense are separated mainly by evolutionary constraint and folding stability, and neither conservation nor folding-stability sensitivity is in the profile. One caveat on the whole comparison: it was run on the earlier, shallower embedding generation, and no equivalent benchmark of the shipped layer-78 battery's deltas is published.

Limitations

• No published-predictor comparison. The artifact set contains no paired evaluation against AlphaMissense, REVEL, CADD, PolyPhen-2 or SIFT on these variants. MAPS's own methodology asks for one, so the correct phrasing today is "not yet evaluated against the field's tools".
• No pooling or representation ablation. Neither the mean-pool probe nor the mutant-embedding-only probe is published, so "covariance pooling helped" and "the difference field helped" are design rationales here, not measured results.
• Interval unit unrecorded, no per-gene breakdown. The metric files carry a 95% interval without the unit it was resampled over, and no per-gene AUROC distribution is published. Variants cluster inside genes, so a variant-level interval would be too narrow.
• The layer-80 harvest driver does not survive. The published difference fields are the only copy of that computation; they cannot be regenerated from code in the source tree. The layer-78 path runs a different driver.
• The ClinVar release is unidentified. Variant identifiers and review stars are preserved, but which weekly release the new pull came from is not recorded anywhere in the surviving tree.
• The harvested population is not a random sample of labelled ClinVar. Among the 10,769 labelled variants with no manifest row, 40.25% are pathogenic against 28.24% in the harvested population, and 76.69% are 1-star against 68.05%. Prevalence, base rates and absolute calibration computed on this set inherit that skew; the cluster-disjoint test split with its own known balance does not depend on it.
• Annotation training-pool membership is unrecorded, as above.
• Two bookkeeping mismatches. Both scaledbal metric files record arm: scaled while the run name says scaledbal, and a few published cross-validation intervals for annotation channels exceed 1.0 because they are normal-approximation bounds across folds. Treat both as recording artifacts, and the annotation intervals as indicative.

Loading the three published artifacts

All three are public Hugging Face repositories joined by variant_id, of the form {protein_id}_{wt_aa}{pos1}{mut_aa}.

Probes: ching-goodfire/MAPS-ESMC-Probes

165 rows in manifest.csv, one per probe: 6 pathogenicity checkpoints at layer 80 and 159 annotation channels at layer 78. Read the manifest first, because the two families take different inputs and carry different kinds of evaluation, and its metric_basis column is what tells them apart. The pathogenicity checkpoints hold weights only, so rebuilding one needs goodfire_core.probes.covariance.SequenceCovarianceProbe; the annotation battery has a safetensors export (161 coefficient rows, the multiclass channel occupying three) that loads with NumPy alone and agrees with scikit-learn's own output to 7.0e-06. The original joblibs were written under scikit-learn 1.7.2 and warn under later versions.

from huggingface_hub import snapshot_download
import sys

local = snapshot_download("ching-goodfire/MAPS-ESMC-Probes")
sys.path.insert(0, local)
from load_probes import AnnotationBattery, build_diff_field, score_variant

battery = AnnotationBattery.open(
    f"{local}/annotations/l78/annotation_probes_l78.safetensors")
tracks = battery.score(embeddings_l78)          # per channel, one value per residue

field, mask = build_diff_field(wt_l80, mut_l80)  # top 256 by 1 - cosine
probability = score_variant(field, [
    f"{local}/pathogenicity/l80/scaled_L80_cov_s42.pt",
    f"{local}/pathogenicity/l80/scaled_L80_cov_s43.pt"])

Difference fields: ching-goodfire/MAPS-ClinVar-VKS-Embeddings-L80

Every stored layer-80 difference field for all 200,913 variants: the whole 12,565-variant test split and the whole 188,348-variant training pool, 262 parquet shards (17 test, 245 train), 154.0 GB on disk from 260.8 GB of raw fp16, row groups of 64 rows. Each row is self-describing: the field as a raw fp16 blob plus its shape, the kept positions, the mean difference vector, and the variant's identity, label, cluster and split. The training pool is 27.1% pathogenic, its source pools' own rate rather than a balanced draw. An earlier revision shipped a 20,000-variant sample of that pool; that sample now survives as a membership list, so anything computed on it stays reproducible.

import numpy as np
from datasets import load_dataset

ds = load_dataset("ching-goodfire/MAPS-ClinVar-VKS-Embeddings-L80", split="test", streaming=True)
row = next(iter(ds))
X = np.frombuffer(row["X"], dtype=np.float16).reshape(row["X_rows"], row["X_cols"])
idx = np.asarray(row["idx"], dtype=np.int32)

Inputs: ching-goodfire/MAPS-ClinVar-VKS-Inputs

200,913 variant rows and 17,231 wild-type sequences, with identity, position, label, split, homology cluster and ClinVar identifiers. Mutant sequences are not stored because every variant is a single substitution: splice the mutant residue into the wild-type sequence at pos1 - 1, a rule verified against the stored mutant sequences on all 200,913 rows before that column was dropped. Protein lengths run 32 to 35,991 residues, median 490, and 2,595 proteins exceed 1,022 residues, covering 91,221 variants.

EVEE, the genomic sibling

MAPS is the protein-space counterpart of EVEE (Evo Variant Effect Explorer, Goodfire with Mayo Clinic, bioRxiv 2026.04.10.717844), and the shape is the same: a covariance probe on frozen foundation-model embeddings for the prediction, supervised annotation-probe deltas for the mechanism, homology-aware evaluation for the verdict. Two differences matter for anyone porting work between them. EVEE's substrate is a genomic model (Evo 2) over a DNA window, so its scope includes variant classes MAPS cannot represent. And EVEE adds a step MAPS's published set does not have: a frontier model writes the explanation from the top disruptions, and that explanation is scored against a metadata-only baseline instead of shipped unevaluated. The figures in that work (0.997 AUROC on 839K ClinVar variants, explanations rated 3.8 against 2.8 out of 5 on a 154-variant held-out set) are that paper's, on genomic variants, and are not comparable to the protein-space numbers on this page. When a MAPS deliverable includes a written explanation, that scored-explanation step is the piece to borrow.

Three things to do next

Replicate, then run the missing ablation. The training pool, the test split and the difference fields are all public, and the probe config is fully recorded, so retraining the covariance probe from the published train split and checking the held-out number is a closed loop that needs no harvest. The same script gives the mean-pool ablation on identical inputs, which is the one measurement that would turn "covariance pooling is the right choice here" into a result. Keep the inputs raw in both arms, or the comparison is not clean.

Extend where the artifacts are silent. Three gaps are ready to close: a paired comparison against AlphaMissense and REVEL on these 12,565 variants, using their precomputed tables joined by variant; per-gene AUROC with a gene-level bootstrap, which the pooled number cannot substitute for; and the two channels the disruption profile lacks, per-position conservation and folding-stability sensitivity, which is where the analysis of the negative result points.

Apply it, within scope. Scoring your own variants needs paired wild-type and mutant layer-80 embeddings, the difference-field rule, and the two seed checkpoints of one arm averaged as probabilities. Two guards before trusting the output: restrict the input to same-length single substitutions, and check your proteins against the published homology clusters, because a protein sitting in a training cluster is not a held-out case for these probes.