MFASS v2 evaluates splice-variant prioritisation using a corrected sequence baseline and a fixed grouped holdout.
Explanatory profile: limited source coverage · Automated source review, 2026-09-16. Review applies to the cited claims; unresolved fields are listed below. Numerical results retain their own review status.
| Property | Description and evidence |
|---|---|
| Datasets | Functional exon-recognition labels from the MFASS assay; validated assay-oriented reference/mutant sequence pairs define the sequence-model inputs.SourcesMFASS computational protocol README at bee9133b83f3aedaf2bbb9013f1875515845607e · Pinned README: correction notice; Dataset; Cohort reconciliation; Split; MFASS-v2 results; Paired comparisons; Limits; Archived mfass-v1 snapshot |
| Splits | The canonical split assigns whole connected exon/gene groups to training or test. Prevalence matching is selected before model runs and uses no predictions.SourcesMFASS computational protocol README at bee9133b83f3aedaf2bbb9013f1875515845607e · Pinned README: correction notice; Dataset; Cohort reconciliation; Split; MFASS-v2 results; Paired comparisons; Limits; Archived mfass-v1 snapshot |
| Metrics | Precision at a fixed review capacity, average precision and AUROC. Each method’s point estimates use its scored subset; paired comparisons use common scored variants.SourcesMFASS computational protocol README at bee9133b83f3aedaf2bbb9013f1875515845607e · Pinned README: correction notice; Dataset; Cohort reconciliation; Split; MFASS-v2 results; Paired comparisons; Limits; Archived mfass-v1 snapshot |
| Baselines | Corrected k-mer/position baseline, frozen DNABERT-2 pair embeddings with a fixed logistic head, and SpliceAI/Pangolin genomic-context specialists.SourcesMFASS computational protocol README at bee9133b83f3aedaf2bbb9013f1875515845607e · Pinned README: correction notice; Dataset; Cohort reconciliation; Split; MFASS-v2 results; Paired comparisons; Limits; Archived mfass-v1 snapshot |
| Leakage controls | Baseline and logistic-head fitting use training labels only. Exon/gene connected components prevent linked variants crossing arms; exact DNABERT-2 pretraining overlap has not been checked.SourcesMFASS computational protocol README at bee9133b83f3aedaf2bbb9013f1875515845607e · Pinned README: correction notice; Dataset; Cohort reconciliation; Split; MFASS-v2 results; Paired comparisons; Limits; Archived mfass-v1 snapshot |
| Uncertainty | Paired intervals resample whole connected exon/gene groups; precision intervals account for fixed review-list fraction during group resampling.SourcesMFASS computational protocol README at bee9133b83f3aedaf2bbb9013f1875515845607e · Pinned README: correction notice; Dataset; Cohort reconciliation; Split; MFASS-v2 results; Paired comparisons; Limits; Archived mfass-v1 snapshot |
| Entity type | Paper-specific computational evaluation protocol.SourcesMFASS computational protocol README at bee9133b83f3aedaf2bbb9013f1875515845607e · Pinned README: correction notice; Dataset; Cohort reconciliation; Split; MFASS-v2 results; Paired comparisons; Limits; Archived mfass-v1 snapshot |
| Organisms | Human (Homo sapiens): naturally occurring ExAC variants in or beside human exons, measured in the MFASS minigene assay.SourcesMFASS original assay accession GSE120695 · GSE120695 SOFT: Series_summary and Sample_organism_ch1 fields |
| Assays | MFASS functional exon-recognition labels.SourcesMFASS computational protocol README at bee9133b83f3aedaf2bbb9013f1875515845607e · Pinned README: correction notice; Dataset; Cohort reconciliation; Split; MFASS-v2 results; Paired comparisons; Limits; Archived mfass-v1 snapshot |
| Allowed inputs | Validated assay-oriented sequence pairs for baseline/DNABERT-2; specialists use genomic context.SourcesMFASS computational protocol README at bee9133b83f3aedaf2bbb9013f1875515845607e · Pinned README: correction notice; Dataset; Cohort reconciliation; Split; MFASS-v2 results; Paired comparisons; Limits; Archived mfass-v1 snapshot |
| Adaptation | Baseline and logistic head use MFASS training labels; specialists are zero-shot on the assay.SourcesMFASS computational protocol README at bee9133b83f3aedaf2bbb9013f1875515845607e · Pinned README: correction notice; Dataset; Cohort reconciliation; Split; MFASS-v2 results; Paired comparisons; Limits; Archived mfass-v1 snapshot |
Conceptual summary of the cited evaluation; exact task configuration and source version remain part of the protocol.
Functional exon-recognition labels from the MFASS assay; validated assay-oriented reference/mutant sequence pairs define the sequence-model inputs. The canonical split assigns whole connected exon/gene groups to training or test. Prevalence matching is selected before model runs and uses no predictions. Precision at a fixed review capacity, average precision and AUROC. Each method’s point estimates use its scored subset; paired comparisons use common scored variants. Corrected k-mer/position baseline, frozen DNABERT-2 pair embeddings with a fixed logistic head, and SpliceAI/Pangolin genomic-context specialists. Baseline and logistic-head fitting use training labels only. Exon/gene connected components prevent linked variants crossing arms; exact DNABERT-2 pretraining overlap has not been checked. Paired intervals resample whole connected exon/gene groups; precision intervals account for fixed review-list fraction during group resampling.
Benchmarks bring together tasks and protocols. A task describes the biological question; a protocol defines a particular test.
These source-backed links do not make different protocols or scores interchangeable.
Each evaluation records what was tested and under which conditions.
Run the corrected MFASS v2 cohort validation and baseline, with an optional DNABERT-2 resource pilot. Keep this protocol separate from the superseded v1 run.
Checked against the official instructions on 2026-09-17. These commands have not been executed by rewire. Running them does not automatically reproduce the published scores.
Repository checkout wrapper: the detached revision selects the exact official source inspected for this guide.
git clone https://github.com/rewire-bio/rewire-benchmarks.git
cd rewire-benchmarks
git checkout --detach 68d1ee53a0fcd1104d9f427d68dedffdccf2c47fOfficial download commands. These upstream master URLs are mutable; the source README records the expected input hashes.
mkdir -p benchmarks/mfass/data
curl -L -o benchmarks/mfass/data/snv_data_clean.txt \
https://raw.githubusercontent.com/KosuriLab/MFASS/master/processed_data/snv/snv_data_clean.txt
curl -L -o benchmarks/mfass/data/snv_func_annot.txt \
https://raw.githubusercontent.com/KosuriLab/MFASS/master/processed_data/snv/snv_func_annot.txtPreserves the official v2 command sequence. The baseline and encoder head are supervised on the predeclared training split.
uv sync --package mfass --extra dnabert2-pilot
uv run --package mfass --extra dnabert2-pilot mfass-build --check
uv run --package mfass --extra dnabert2-pilot mfass-build
uv run --package mfass --extra dnabert2-pilot mfass-baselineThe pilot measures feasibility and validates checkpoint provenance; it does not calculate accuracy. The README gates any subsequent full encoder run on projected runtime and free disk; the full run is intentionally a separate decision.
uv run --package mfass --extra dnabert2-pilot mfass-dnabert2-pilotRelease 2026-09-17-d277315f7d76 · 4 evaluations · 12 metric rows. Different protocols are not a single leaderboard.
| Metric and finding | Coverage and uncertainty | Evidence |
|---|---|---|
| Corrected k-mer / position baseline on MFASS v2 Configuration: Corrected k-mer / position baselineProtocol: MFASS v2Dataset: MFASS v2 eligible assay cohort Assay-oriented 21 bp k-mer window, exon position, allele identity and conservation features; gradient-boosted trees trained on the MFASS training split. Rewire evaluation · Evaluation metadata: reproduced | ||
| 0.7779498064677238 auroc Unit: fraction · Direction: higher | Uncertainty: Point estimate; see paired-comparison artifacts. Coverage (scored/eligible): 8324/8324 | reproducedMFASS v2 pinned rewire artifacts · benchmarks/mfass/results/baseline-kmer-position-v2.json :: auroc Source checking is not independent reproduction. |
| 0.28641674595892375 average_precision Unit: fraction · Direction: higher | Uncertainty: Point estimate; see paired-comparison artifacts. Coverage (scored/eligible): 8324/8324 | reproducedMFASS v2 pinned rewire artifacts · benchmarks/mfass/results/baseline-kmer-position-v2.json :: average_precision Source checking is not independent reproduction. |
| 0.61 precision_at_100 Unit: fraction · Direction: higher | Uncertainty: Point estimate; see paired-comparison artifacts. Coverage (scored/eligible): 8324/8324 | reproducedMFASS v2 pinned rewire artifacts · benchmarks/mfass/results/baseline-kmer-position-v2.json :: precision_at_100 Source checking is not independent reproduction. |
| DNABERT-2 117M · frozen pair embeddings on MFASS v2 Pipeline: DNABERT-2 117M · frozen pair embeddingsProtocol: MFASS v2Dataset: MFASS v2 eligible assay cohort Masked mean of frozen last hidden states for 170 bp reference and mutant sequences; concatenate reference and mutant-minus-reference embeddings; fixed balanced L2 logistic head trained only on the MFASS training split. Rewire evaluation · Evaluation metadata: reproduced | ||
| 0.5500324040216661 auroc Unit: fraction · Direction: higher | Uncertainty: Point estimate; see paired-comparison artifacts. Coverage (scored/eligible): 8324/8324 | reproducedMFASS v2 pinned rewire artifacts · benchmarks/mfass/results/dnabert2-117m-frozen-pair-logreg.json :: auroc Source checking is not independent reproduction. |
| 0.04508654312652131 average_precision Unit: fraction · Direction: higher | Uncertainty: Point estimate; see paired-comparison artifacts. Coverage (scored/eligible): 8324/8324 | reproducedMFASS v2 pinned rewire artifacts · benchmarks/mfass/results/dnabert2-117m-frozen-pair-logreg.json :: average_precision Source checking is not independent reproduction. |
| 0.03 precision_at_100 Unit: fraction · Direction: higher | Uncertainty: Point estimate; see paired-comparison artifacts. Coverage (scored/eligible): 8324/8324 | reproducedMFASS v2 pinned rewire artifacts · benchmarks/mfass/results/dnabert2-117m-frozen-pair-logreg.json :: precision_at_100 Source checking is not independent reproduction. |
| Pangolin · mask=False on MFASS v2 Unchanged specialist run in genomic context with GENCODE v44; zero-shot on MFASS assay labels. Point metrics use the scored subset. Rewire evaluation · Evaluation metadata: reproduced | ||
| 0.8756851300560864 auroc Unit: fraction · Direction: higher | Uncertainty: Point estimate; see paired-comparison artifacts. Coverage (scored/eligible): 8301/8324 | reproducedMFASS v2 pinned rewire artifacts · benchmarks/mfass/results/pangolin-maskFalse.json :: auroc Source checking is not independent reproduction. |
| 0.3887617543064248 average_precision Unit: fraction · Direction: higher | Uncertainty: Point estimate; see paired-comparison artifacts. Coverage (scored/eligible): 8301/8324 | reproducedMFASS v2 pinned rewire artifacts · benchmarks/mfass/results/pangolin-maskFalse.json :: average_precision Source checking is not independent reproduction. |
| 0.65 precision_at_100 Unit: fraction · Direction: higher | Uncertainty: Point estimate; see paired-comparison artifacts. Coverage (scored/eligible): 8301/8324 | reproducedMFASS v2 pinned rewire artifacts · benchmarks/mfass/results/pangolin-maskFalse.json :: precision_at_100 Source checking is not independent reproduction. |
| SpliceAI 1.3.1 on MFASS v2 Unchanged specialist run in genomic context with bundled annotation; zero-shot on MFASS assay labels. Point metrics use the scored subset. Rewire evaluation · Evaluation metadata: reproduced | ||
| 0.8055241740253153 auroc Unit: fraction · Direction: higher | Uncertainty: Point estimate; see paired-comparison artifacts. Coverage (scored/eligible): 8194/8324 | reproducedMFASS v2 pinned rewire artifacts · benchmarks/mfass/results/spliceai-1.3.1.json :: auroc Source checking is not independent reproduction. |
| 0.2986855472760137 average_precision Unit: fraction · Direction: higher | Uncertainty: Point estimate; see paired-comparison artifacts. Coverage (scored/eligible): 8194/8324 | reproducedMFASS v2 pinned rewire artifacts · benchmarks/mfass/results/spliceai-1.3.1.json :: average_precision Source checking is not independent reproduction. |
| 0.64 precision_at_100 Unit: fraction · Direction: higher | Uncertainty: Point estimate; see paired-comparison artifacts. Coverage (scored/eligible): 8194/8324 | reproducedMFASS v2 pinned rewire artifacts · benchmarks/mfass/results/spliceai-1.3.1.json :: precision_at_100 Source checking is not independent reproduction. |
Last literature check: 2026-09-17. Dated primary-source discovery and protocol/table screening. Source checking does not mean experimental reproduction. Only separately extracted and independently reviewed numeric batches are publishable.
| Paper or primary resource | Version | Reference |
|---|---|---|
| MFASS benchmark: corrected v2 protocol and reproducibility instructions | Primary full-text snapshot retrieved 2026-09-17; exact bytes pinned by SHA-256 | Read source |
historical results preserved
Task-specific computational methodology and field context checked in the cited primary-source artifact. Source-backed fields, inapplicable evaluator dimensions and unresolved details are distinguished. Numerical results were not reproduced. Original assay organism checked against GEO SOFT metadata; v1 interval procedure checked against its immutable historical JSON without rehabilitating superseded comparisons.
Stable record: rewire-mfass-v2Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.
One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.
24 evidence rows matching the loaded filters
| Property and statement | Original source and location | Review and provenance |
|---|---|---|
| Diagram caption Conceptual summary of the cited evaluation; exact task configuration and source version remain part of the protocol. Individual claims | MFASS computational protocol README at bee9133b83f3aedaf2bbb9013f1875515845607e Pinned README: correction notice; Dataset; Cohort reconciliation; Split; MFASS-v2 results; Paired comparisons; Limits; Archived mfass-v1 snapshot Version: bee9133b83f3aedaf2bbb9013f1875515845607e | source checked automated source review · 2026-09-16 Audit detailsTask-specific computational methodology and field context checked in the cited primary-source artifact. Source-backed fields, inapplicable evaluator dimensions and unresolved details are distinguished. Numerical results were not reproduced. Original assay organism checked against GEO SOFT metadata; v1 interval procedure checked against its immutable historical JSON without rehabilitating superseded comparisons. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Diagram steps ["Input: Validated assay-oriented sequence pairs for baseline/DNABERT-2; specialists use genomic context.","Evaluation: Baseline and logistic head use MFASS training labels; specialists are zero-shot on the assay.","Readout: Precision at a fixed review capacity, average precision and AUROC. Each method’s point estimates use its scored subset; paired comparisons use common scored variants."] Individual claims | MFASS computational protocol README at bee9133b83f3aedaf2bbb9013f1875515845607e Pinned README: correction notice; Dataset; Cohort reconciliation; Split; MFASS-v2 results; Paired comparisons; Limits; Archived mfass-v1 snapshot Version: bee9133b83f3aedaf2bbb9013f1875515845607e | source checked automated source review · 2026-09-16 Audit detailsTask-specific computational methodology and field context checked in the cited primary-source artifact. Source-backed fields, inapplicable evaluator dimensions and unresolved details are distinguished. Numerical results were not reproduced. Original assay organism checked against GEO SOFT metadata; v1 interval procedure checked against its immutable historical JSON without rehabilitating superseded comparisons. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Diagram title Computational evaluation flow Individual claims | MFASS computational protocol README at bee9133b83f3aedaf2bbb9013f1875515845607e Pinned README: correction notice; Dataset; Cohort reconciliation; Split; MFASS-v2 results; Paired comparisons; Limits; Archived mfass-v1 snapshot Version: bee9133b83f3aedaf2bbb9013f1875515845607e | source checked automated source review · 2026-09-16 Audit detailsTask-specific computational methodology and field context checked in the cited primary-source artifact. Source-backed fields, inapplicable evaluator dimensions and unresolved details are distinguished. Numerical results were not reproduced. Original assay organism checked against GEO SOFT metadata; v1 interval procedure checked against its immutable historical JSON without rehabilitating superseded comparisons. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Datasets Functional exon-recognition labels from the MFASS assay; validated assay-oriented reference/mutant sequence pairs define the sequence-model inputs. Individual claims | MFASS computational protocol README at bee9133b83f3aedaf2bbb9013f1875515845607e Pinned README: correction notice; Dataset; Cohort reconciliation; Split; MFASS-v2 results; Paired comparisons; Limits; Archived mfass-v1 snapshot Version: bee9133b83f3aedaf2bbb9013f1875515845607e | source checked automated source review · 2026-09-16 Audit detailsTask-specific computational methodology and field context checked in the cited primary-source artifact. Source-backed fields, inapplicable evaluator dimensions and unresolved details are distinguished. Numerical results were not reproduced. Original assay organism checked against GEO SOFT metadata; v1 interval procedure checked against its immutable historical JSON without rehabilitating superseded comparisons. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Splits The canonical split assigns whole connected exon/gene groups to training or test. Prevalence matching is selected before model runs and uses no predictions. Individual claims | MFASS computational protocol README at bee9133b83f3aedaf2bbb9013f1875515845607e Pinned README: correction notice; Dataset; Cohort reconciliation; Split; MFASS-v2 results; Paired comparisons; Limits; Archived mfass-v1 snapshot Version: bee9133b83f3aedaf2bbb9013f1875515845607e | source checked automated source review · 2026-09-16 Audit detailsTask-specific computational methodology and field context checked in the cited primary-source artifact. Source-backed fields, inapplicable evaluator dimensions and unresolved details are distinguished. Numerical results were not reproduced. Original assay organism checked against GEO SOFT metadata; v1 interval procedure checked against its immutable historical JSON without rehabilitating superseded comparisons. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Adaptation Baseline and logistic head use MFASS training labels; specialists are zero-shot on the assay. Individual claims | MFASS computational protocol README at bee9133b83f3aedaf2bbb9013f1875515845607e Pinned README: correction notice; Dataset; Cohort reconciliation; Split; MFASS-v2 results; Paired comparisons; Limits; Archived mfass-v1 snapshot Version: bee9133b83f3aedaf2bbb9013f1875515845607e | source checked automated source review · 2026-09-16 Audit detailsTask-specific computational methodology and field context checked in the cited primary-source artifact. Source-backed fields, inapplicable evaluator dimensions and unresolved details are distinguished. Numerical results were not reproduced. Original assay organism checked against GEO SOFT metadata; v1 interval procedure checked against its immutable historical JSON without rehabilitating superseded comparisons. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Metrics Precision at a fixed review capacity, average precision and AUROC. Each method’s point estimates use its scored subset; paired comparisons use common scored variants. Individual claims | MFASS computational protocol README at bee9133b83f3aedaf2bbb9013f1875515845607e Pinned README: correction notice; Dataset; Cohort reconciliation; Split; MFASS-v2 results; Paired comparisons; Limits; Archived mfass-v1 snapshot Version: bee9133b83f3aedaf2bbb9013f1875515845607e | source checked automated source review · 2026-09-16 Audit detailsTask-specific computational methodology and field context checked in the cited primary-source artifact. Source-backed fields, inapplicable evaluator dimensions and unresolved details are distinguished. Numerical results were not reproduced. Original assay organism checked against GEO SOFT metadata; v1 interval procedure checked against its immutable historical JSON without rehabilitating superseded comparisons. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Baselines Corrected k-mer/position baseline, frozen DNABERT-2 pair embeddings with a fixed logistic head, and SpliceAI/Pangolin genomic-context specialists. Individual claims | MFASS computational protocol README at bee9133b83f3aedaf2bbb9013f1875515845607e Pinned README: correction notice; Dataset; Cohort reconciliation; Split; MFASS-v2 results; Paired comparisons; Limits; Archived mfass-v1 snapshot Version: bee9133b83f3aedaf2bbb9013f1875515845607e | source checked automated source review · 2026-09-16 Audit detailsTask-specific computational methodology and field context checked in the cited primary-source artifact. Source-backed fields, inapplicable evaluator dimensions and unresolved details are distinguished. Numerical results were not reproduced. Original assay organism checked against GEO SOFT metadata; v1 interval procedure checked against its immutable historical JSON without rehabilitating superseded comparisons. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Leakage controls Baseline and logistic-head fitting use training labels only. Exon/gene connected components prevent linked variants crossing arms; exact DNABERT-2 pretraining overlap has not been checked. Individual claims | MFASS computational protocol README at bee9133b83f3aedaf2bbb9013f1875515845607e Pinned README: correction notice; Dataset; Cohort reconciliation; Split; MFASS-v2 results; Paired comparisons; Limits; Archived mfass-v1 snapshot Version: bee9133b83f3aedaf2bbb9013f1875515845607e | source checked automated source review · 2026-09-16 Audit detailsTask-specific computational methodology and field context checked in the cited primary-source artifact. Source-backed fields, inapplicable evaluator dimensions and unresolved details are distinguished. Numerical results were not reproduced. Original assay organism checked against GEO SOFT metadata; v1 interval procedure checked against its immutable historical JSON without rehabilitating superseded comparisons. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Uncertainty Paired intervals resample whole connected exon/gene groups; precision intervals account for fixed review-list fraction during group resampling. Individual claims | MFASS computational protocol README at bee9133b83f3aedaf2bbb9013f1875515845607e Pinned README: correction notice; Dataset; Cohort reconciliation; Split; MFASS-v2 results; Paired comparisons; Limits; Archived mfass-v1 snapshot Version: bee9133b83f3aedaf2bbb9013f1875515845607e | source checked automated source review · 2026-09-16 Audit detailsTask-specific computational methodology and field context checked in the cited primary-source artifact. Source-backed fields, inapplicable evaluator dimensions and unresolved details are distinguished. Numerical results were not reproduced. Original assay organism checked against GEO SOFT metadata; v1 interval procedure checked against its immutable historical JSON without rehabilitating superseded comparisons. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
Release 2026-09-17-d277315f7d76 · Record review: source checked
Supersedes MFASS v1 (superseded)
Download this releaseStable ID: rewire-mfass-v2