ProteinGym v1.3 AMFR substitution assay
Score all 2,972 variants in AMFR_HUMAN_Tsuboyama_2023_4G3O with ESM-2 8M masked marginals. This single-assay result is not a full ProteinGym track or suite score.
Overview
Score all 2,972 variants in AMFR_HUMAN_Tsuboyama_2023_4G3O with ESM-2 8M masked marginals. This single-assay result is not a full ProteinGym track or suite score.
Consult the linked sources for architecture or protocol details. Missing evidence is not evidence of a missing capability.
Results
Results are available, but no reviewed comparison panel is linked in this release.
All evaluations
1 evaluation · 5 metric rows. Different protocols are not a single leaderboard.
Filter evaluations
Applied filters: All linked evaluations
Source checking is not independent reproduction. Release 2026-09-22-f58a0f1d267f.
Methods and evaluation design
Procedure, tasks and evaluated configurations
Evaluation design
Benchmarks bring together tasks and protocols. A task describes the biological question; a protocol defines a particular test.
Benchmarks
These source-backed links do not make different protocols or scores interchangeable.
Recorded evaluations
Each evaluation records what was tested and under which conditions.
Run this benchmark
Repeat the local AMFR evaluation
Score all 2,972 variants in AMFR_HUMAN_Tsuboyama_2023_4G3O with ESM-2 8M masked marginals. This single-assay result is not a full ProteinGym track or suite score.
Generate predictions and evaluate them. This recipe does not establish reproduction of a particular published score.
- Dataset access
- Obtain the permitted source files and verify hashes following the cited instructions before running.
- Model and weights
- esm2_t6_8M_UR50D with the documented SHA-256
- Licences
- Follow original dataset and software reuse terms. No dataset or weight redistribution is included.
- Software
- Pinned runner checkout; Python 3.11.13 and rewirebench 0.4.0 wheel. Install the exact executed dependency list linked in the instructions.
- Hardware
- Executed on macOS arm64 CPU, one thread. No accelerator required; memory and cross-platform performance unreported.
Required inputs and expected outputs
Inputs
- Prepared official AMFR assay CSV and pinned ESM-2 checkpoint
Outputs
- Private predictions, coverage report, metrics and sanitized contribution bundle
Execution steps
1. Run from the pinned runner checkout after preparation (Command line)
Source reviewed; these instructions have not been executed by rewire.
ProteinGym v1.3 AMFR substitution assay: reproduction instructions · Reproduce section; replace paths with prepared local inputsOMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 python research/local-runs-2026-09-20/proteingym-esm2/reproduce.py \ --data /data/DMS_ProteinGym_substitutions \ --checkpoint /weights/esm2_t6_8M_UR50D.pt \ --output /new/private/amfr-run
Use your own model
Run your model locally and return predictions keyed by the input IDs. The evaluator supplies biological inputs without test labels and owns scoring. This interface is not a sandbox for model code.
Pass your existing prediction function into this adapter. Its output direction must match the selected protocol.
class MyModelAdapter:
def __init__(self, score):
self.score = score
def predict(self, inputs):
return {row["id"]: float(self.score(row)) for row in inputs}
# adapter = MyModelAdapter(your_prediction_function)
# report = rewirebench.run(prepared, adapter, output="runs/my-model")Alternatively, generate a keyed prediction file in your existing model environment and use the score-only recipe. Your model code and weights do not need to be shared.
ProteinGym v1.3 AMFR substitution assay: reproduction instructions · Reproduction instructions and execution scopeScope and limitations
- The script was executed locally, but these portable path examples have not been rerun verbatim.
- Only the selected assay or selected sequence controls are evaluated; no suite aggregate.
- Use new output directories. Raw predictions and private inputs are not uploaded.
- Matching a fresh local result is distinct from reproducing an earlier paper score.
Contribute a result for review. The library can submit an exported evaluation for private review when intake is open. Check the contribution page for access and sign-in.
Original repository instructions
Run instructions
No runnable recipe has been reviewed for this protocol. Dataset access, model requirements, licences and compute requirements must be checked against its sources before execution.
Strengths, limitations and unresolved questions
Evidence
Source checking verifies the cited claim or transcription. It does not establish independent reproduction.
Evidence table
Inspect claims, sources and review details
Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.
One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.
7 evidence rows matching the loaded filters
| Property and statement | Original source and location | Review and provenance |
|---|---|---|
| Run recipe: inputs ["Prepared official AMFR assay CSV and pinned ESM-2 checkpoint"] Individual claims | ProteinGym v1.3 AMFR substitution assay: reproduction instructions Reproduction instructions and execution scope Version: ca73fa47136d182f2d4ddb083d084712198fc0e2 | source checked implementation review Audit detailsRecipe metadata reviewed against pinned sources; not experimental reproduction. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Run recipe: Run from the pinned runner checkout after preparation OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 python research/local-runs-2026-09-20/proteingym-esm2/reproduce.py \ --data /data/DMS_ProteinGym_substitutions \ --checkpoint /weights/esm2_t6_8M_UR50D.pt \ --output /new/private/amfr-run Individual claims | ProteinGym v1.3 AMFR substitution assay: reproduction instructions Reproduce section; replace paths with prepared local inputs Version: ca73fa47136d182f2d4ddb083d084712198fc0e2 | source checked implementation review Audit detailsExecution status: source_reviewed_not_executed. Execution receipts apply only to their stated scope; not experimental reproduction. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Run recipe: limitations ["The script was executed locally, but these portable path examples have not been rerun verbatim.","Only the selected assay or selected sequence controls are evaluated; no suite aggregate.","Use new output directories. Raw predictions and private inputs are not uploaded.","Matching a fresh local result is distinct from reproducing an earlier paper score."] Individual claims | ProteinGym v1.3 AMFR substitution assay: reproduction instructions Reproduction instructions and execution scope Version: ca73fa47136d182f2d4ddb083d084712198fc0e2 | source checked implementation review Audit detailsRecipe metadata reviewed against pinned sources; not experimental reproduction. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Run recipe: outputs ["Private predictions, coverage report, metrics and sanitized contribution bundle"] Individual claims | ProteinGym v1.3 AMFR substitution assay: reproduction instructions Reproduction instructions and execution scope Version: ca73fa47136d182f2d4ddb083d084712198fc0e2 | source checked implementation review Audit detailsRecipe metadata reviewed against pinned sources; not experimental reproduction. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Run recipe: requirements {"data":"Obtain the permitted source files and verify hashes following the cited instructions before running.","weights":"esm2_t6_8M_UR50D with the documented SHA-256","licence":"Follow original dataset and software reuse terms. No dataset or weight redistribution is included.","software":"Pinned runner checkout; Python 3.11.13 and rewirebench 0.4.0 wheel. Install the exact executed dependency list linked in the instructions.","hardware":"Executed on macOS arm64 CPU, one thread. No accelerator required; memory and cross-platform performance unreported."} Individual claims | ProteinGym v1.3 AMFR substitution assay: reproduction instructions Reproduction instructions and execution scope Version: ca73fa47136d182f2d4ddb083d084712198fc0e2 | source checked implementation review Audit detailsRecipe metadata reviewed against pinned sources; not experimental reproduction. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Run recipe: summary Score all 2,972 variants in AMFR_HUMAN_Tsuboyama_2023_4G3O with ESM-2 8M masked marginals. This single-assay result is not a full ProteinGym track or suite score. Individual claims | ProteinGym v1.3 AMFR substitution assay: reproduction instructions Reproduction instructions and execution scope Version: ca73fa47136d182f2d4ddb083d084712198fc0e2 | source checked implementation review Audit detailsRecipe metadata reviewed against pinned sources; not experimental reproduction. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Relationship: part of discovery-benchmark-proteingym Individual claims | ESM-2 8M masked-marginal scoring: local execution report (20 September 2026) protocol_id; protocol_configuration; provenance Version: ca73fa47136d182f2d4ddb083d084712198fc0e2 | source checked automated execution evidence review · 2026-09-20T20:54:03Z Audit detailsField: Claim: rewire-protocol-proteingym-amfr-v13-part-of-discovery-benchmark-proteingym Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
Sources and history
View linked audit checks and correction history
Release 2026-09-22-f58a0f1d267f · Record review: source checked
2 source records and release history
- ESM-2 8M masked-marginal scoring: local execution report (20 September 2026) · Original source · ca73fa47136d182f2d4ddb083d084712198fc0e2
- ProteinGym v1.3 AMFR substitution assay: reproduction instructions · Original source · ca73fa47136d182f2d4ddb083d084712198fc0e2
Technical metadata and extraction receipts
Stable ID: rewire-protocol-proteingym-amfr-v13
- areas
- proteins-complexes
- protocol id
- proteingym-v1.3-dms-substitutions
- version
- 1.3
- procedure
- aggregation: round_assay_3dp_then_mean_by_UniProt_and_selection_then_equal_category_mean; assays: AMFR HUMAN Tsuboyama 2023 4G3O: MSA Neff L category: Medium; UniProt ID: AMFR_HUMAN; expected count: 2972; prepared count: 2972; selection type: Stability; taxon: Human; binary labels: official_DMS_score_bin; official assay count: 217; official variant count: 2465767; score direction: higher_is_better
- source locator
- protocol_configuration; protocol_results; provenance
- access note
- See the pinned execution report and source retrieval evidence. Dataset reuse terms are source-specific.
- reproducibility note
- Fresh local evaluation; reference metric agreement is not reproduction of a paper score.
- reproduction url
- https://github.com/rewire-bio/rewire-benchmarks/blob/ca73fa47136d182f2d4ddb083d084712198fc0e2/research/local-runs-2026-09-20/README.md
- run recipes
- id: local-20260920-proteingym-repeat; protocol id: rewire-protocol-proteingym-amfr-v13; version: ca73fa47136d182f2d4ddb083d084712198fc0e2; title: Repeat the local AMFR evaluation; purpose: generate_and_evaluate; summary: Score all 2,972 variants in AMFR_HUMAN_Tsuboyama_2023_4G3O with ESM-2 8M masked marginals. This single-assay result is not a full ProteinGym track or suite score.; inputs: Prepared official AMFR assay CSV and pinned ESM-2 checkpoint; outputs: Private predictions, coverage report, metrics and sanitized contribution bundle; requirements: data: Obtain the permitted source files and verify hashes following the cited instructions before running.; weights: esm2_t6_8M_UR50D with the documented SHA-256; licence: Follow original dataset and software reuse terms. No dataset or weight redistribution is included.; software: Pinned runner checkout; Python 3.11.13 and rewirebench 0.4.0 wheel. Install the exact executed dependency list linked in the instructions.; hardware: Executed on macOS arm64 CPU, one thread. No accelerator required; memory and cross-platform performance unreported.; instructions: runtime: command_line; title: Run from the pinned runner checkout after preparation; code: OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 python research/local-runs-2026-09-20/proteingym-esm2/reproduce.py \ --data /data/DMS_ProteinGym_substitutions \ --checkpoint /weights/esm2_t6_8M_UR50D.pt \ --output /new/private/amfr-run; status: source_reviewed_not_executed; source ids: rewire-local-20260920-instructions-proteingym; source locator: Reproduce section; replace paths with prepared local inputs; limitations: The script was executed locally, but these portable path examples have not been rerun verbatim.; Only the selected assay or selected sequence controls are evaluated; no suite aggregate.; Use new output directories. Raw predictions and private inputs are not uploaded.; Matching a fresh local result is distinct from reproducing an earlier paper score.; source ids: rewire-local-20260920-instructions-proteingym; source locator: Reproduction instructions and execution scope