DNA & genomes
Sequence variation, splicing and gene regulation.
6 models · 3 test familiesBrowse models and tests →Open source
Benchmarks
A map of biological models and the tests that fit them, alongside independently measured rewire.it results and a separate collection of scores reported in research papers.
Explore by biology
Browse models and evaluation tasks by area of biology.
RNA & transcriptomes
RNA structure and transcript-level function.
4 models · 4 test familiesBrowse models and tests →Proteins & complexes
Mutation effects, folding and protein design.
5 models · 3 test familiesBrowse models and tests →Cells & tissues
Single-cell reference mapping, integration and perturbation.
5 models · 3 test familiesBrowse models and tests →Microbes & communities
Microbial sequences and metagenomic classification.
5 models · 4 test familiesBrowse models and tests →Molecular interactions
Docking, affinity and multi-molecule structure.
5 models · 3 test familiesBrowse models and tests →Find a model
Find a model, explore the tests that apply to it and see what each evaluation requires.
27 candidate models and comparators
- AlphaFold 3 Server hosted serverfoundation model
- AlphaGenome API / released weightsfoundation model
- AutoDock Vina 1.2.7baseline
- Boltz-2 released weightsfoundation model
- Chai-1 released weightsfoundation model
- DiffDock-L 2024 releasespecialist
- DNABERT-2 117Mfoundation model
- ESM-2 8Mfoundation model
- ESMFold v1foundation model
- Evo 2 7Bfoundation model
- GEARS published implementationspecialist
- Geneformer published checkpointsfoundation model
- Kraken2 current database pinned at run timebaseline
- METAGENE-1 6Bfoundation model
- MetaPhlAn current marker database pinned at run timespecialist
- MIMIC 1.0foundation model
- mRNA-FM codon-tokenisedfoundation model
- Nucleotide Transformer v2 50M multi-speciesfoundation model
- Pangolin published checkpointsspecialist
- ProkBERT minifoundation model
- ProteinMPNN v_48_020specialist
- RhoFold+ pretrainedspecialist
- RNA-FM ncRNAfoundation model
- scFoundation 100Mfoundation model
- scGPT whole-humanfoundation model
- scVI scvi-toolsbaseline
- SpliceAI 1.3.1specialist
Results reported in papers
Results from 100 papers, with datasets, evaluation methods and links to the source.
Our measured runs
See how DNABERT-2 and specialist splicing tools compare with a simple baseline on MFASS.
For rewire.it runs, the aim is to publish a floor that anyone can measure against and re-run. These six requirements guide each independent result and its published protocol.
- A trivial baseline always runs, with a fair tuning budget. A leaderboard without a floor is misleading, and in this field the floor frequently wins.
- The grouping rule and the independent-group count are published with every result. Splits leak in ways that never show up in the output file.
- Contamination is stated for every pretrained method: what was checked, and what is unknown. “Unknown” is an acceptable answer. Silence is not.
- Coverage reconciles against the original denominator. A method that cannot score an input has a coverage problem, not a negative prediction.
- Throughput sits beside accuracy, measured end to end.
- Configuration is recorded: checkpoint revision, pooling, context in bases and tokens, precision, batch size.
mfass-v1 · historical report
Superseded by MFASS v2. The original baseline centred sequence windows incorrectly for 7,770 eligible variants. Its score and comparisons below are the preserved v1 record, not current evidence for choosing a model. Read the corrected v2 result →
Splice-variant prioritisation against a functional assay. Does a model improve the ranking of splice-region variants over the tools a diagnostic laboratory already runs?
Built on MFASS, from Chong and colleagues, Molecular Cell 2018: 27,733 ExAC variants assayed for exon recognition in a minigene reporter, with 1,050 splice-disrupting at 3.8% prevalence. The labels come from a functional assay rather than clinical assertions. About 17% of the disrupting variants sit at canonical splice sites.
Results
Held-out set of 8,324 variants across 463 independent groups, grouped by connected exon and gene components. Primary metric is precision at a 100-variant review capacity.
| Method | Family | P@100 | AP | AUROC | Coverage | s/variant |
|---|---|---|---|---|---|---|
| baseline-kmer-position | trivial baseline | 0.620 | 0.286 | 0.768 | 8324/8324 | 0.00002 |
| spliceai-1.3.1 | specialist | 0.640 | 0.299 | 0.806 | 8194/8324 | 0.54 |
| pangolin (mask=False) | specialist | 0.650 | 0.389 | 0.876 | 8301/8324 | 1.64 |
Paired comparisons
Observed difference on the variants both methods scored, with a 95% interval from resampling whole groups. Bold means the interval excludes zero.
| Candidate minus reference | P@100 | AP | AUROC |
|---|---|---|---|
| SpliceAI minus baseline | +0.020 [−0.090, +0.105] | +0.009 [−0.040, +0.056] | +0.037 [+0.002, +0.075] |
| Pangolin minus baseline | +0.030 [−0.054, +0.102] | +0.102 [+0.061, +0.138] | +0.108 [+0.081, +0.134] |
| Pangolin minus SpliceAI | +0.010 [−0.039, +0.076] | +0.092 [+0.061, +0.122] | +0.070 [+0.043, +0.094] |
Original v1 interpretation, superseded by the corrected v2 result: Across the scored variants, Pangolin leads on AUROC, followed by SpliceAI and the baseline. Among the first hundred variants an analyst would actually review, the three put 62, 64 and 65 confirmed disruptions in the queue and cannot be told apart.
By distance to the exon boundary
AUROC on the 8,194 variants all three methods scored.
| Band | Variants | SDVs | baseline | SpliceAI | Pangolin |
|---|---|---|---|---|---|
| canonical, 0 to 2 | 443 | 41 | 0.825 | 0.902 | 0.925 |
| near, 3 to 10 | 1,671 | 77 | 0.745 | 0.798 | 0.857 |
| mid, 11 to 30 | 4,166 | 159 | 0.744 | 0.785 | 0.868 |
| distal, over 30 | 1,914 | 31 | 0.786 | 0.744 | 0.844 |
These bands are descriptive and carry no comparison interval. The distal band holds only 31 positives, and the band variable is itself a fitted feature of the baseline, so a reversal involving the baseline there is confounded with its supervision rather than simply imprecise.
What this does not establish
MFASS measures exon recognition in a minigene construct. These are not predictions of splicing in patient RNA. The baseline is supervised on this assay’s training split while both specialists are zero-shot, so that comparison measures in-domain training against a specialist prior rather than the standalone quality of either tool. SpliceAI ran on its bundled GENCODE v24 annotation and Pangolin on v44, so the gap between them carries an annotation difference as well as a model difference. A matched-annotation run is needed before attributing the size or direction of that gap to the models alone. A frozen DNABERT-2 protocol is now reported separately in mfass-v2.
Reproduce the preserved v1 baseline
The benchmark code is MIT licensed; the MFASS source data remains the authors’ work. The baseline path uses pinned benchmark and MFASS revisions, with no genome retrieval, because every assayed sequence is a self-contained 170-base window.
git clone https://github.com/rewire-bio/rewire-benchmarks
cd rewire-benchmarks
git checkout edf5b5c0b83bec27975e8c8a30be88e7a3581a52
uv sync
curl -L -o benchmarks/mfass/data/snv_data_clean.txt \
https://raw.githubusercontent.com/KosuriLab/MFASS/9a8e4f27106be52aeb11acad27f95f5cded663a8/processed_data/snv/snv_data_clean.txt
curl -L -o benchmarks/mfass/data/snv_func_annot.txt \
https://raw.githubusercontent.com/KosuriLab/MFASS/9a8e4f27106be52aeb11acad27f95f5cded663a8/processed_data/snv/snv_func_annot.txt
uv run mfass-build
uv run mfass-split --key ensembl_id --key ensembl_gene_id \
--out benchmarks/mfass/splits/split-v2.tsv
uv run mfass-baselineThe specialist runs additionally need a GRCh38 primary assembly and, for Pangolin, a GENCODE database. Both runners pin every flag explicitly, including the ones already at their defaults, because SpliceAI and Pangolin disagree about masking by default and that difference is easy to mistake for a difference between the models.
Next
A matched-annotation SpliceAI run to close the confound above, Pangolin at its own masking default to measure what that setting is worth, and further pretrained encoders with declared scoring protocols. Further benchmarks will follow the same six rules.
If there is a task or dataset you would like evaluated this way, or you think a result here is wrong, the repository takes issues and so do I.