A documented local ESM-2 masked-marginal evaluation of all 2,972 variants in the ProteinGym v1.3 AMFR stability assay. It is one complete assay out of 217, with subset scope and no suite-level score or uncertainty interval.
The preserved per-assay report contains Spearman, AUC, MCC, NDCG and Top_recall, rounded to three decimals. Higher is better; the reported negative correlation is retained. The local documentation reports agreement with upstream formulas; this review neither recomputed nor changed these result values.
limited source coverage · Automated source review, 2026-09-23. All specifications and missing details
Results are available, but no reviewed comparison panel is linked in this release.
1 evaluation · 5 results. Different protocols are not a single leaderboard.
Applied filters: All linked evaluations
Source checking is not independent reproduction. Release 2026-09-25-8af07e960e5f.
The pinned reference identifies a 47-residue human AMFR construct measured by cDNA-display proteolysis for stability. It lists 820 single and 2,152 multiple substitutions. This assay concerns the measured construct and experimental stability readout; it does not establish clinical variant effects or whole-protein performance.
The report identifies esm2_t6_8M_UR50D with a checkpoint digest and masked-marginal log odds in wild-type context, summed across substitutions. It reports no fitting and no assay labels, MSA or structure in the adapter inputs. Those observations apply to this reported configuration only.
All 2,972 selected variants are reported scored, with zero unscored. The assay status is complete, while the enclosing report remains subset/partial_track, one of 217 assays, with empty suite metrics. The selected-assay denominator must not be interpreted as the denominator of the whole substitution track.
This review retrieved and checked the public report, pinned reference, documentation and source code. The original documentation reports saved-prediction rescoring and agreement with five upstream metric formulas, but not execution of the complete upstream aggregation program. No predictions or weights were downloaded and no model was run in this review. The report explicitly says independently_reproduced=false.
Benchmarks bring together tasks and protocols. A task describes the biological question; a protocol defines a particular test.
These source-backed links do not make different protocols or scores interchangeable.
Each evaluation records what was tested and under which conditions.
Reference methods help show what a model adds beyond simple controls. We track a null control and a conventional method for each protocol.
0 of 2 active baseline roles have published Rewire measurements in this release. Measurements on a selected protocol do not establish coverage of an entire suite.
1 execution recipes Recipe availability does not establish a completed evaluation.
Literature evidence is not a Rewire measurement. Executed but unpublished runs and private review status are not included.
Proposed control: requires review
Select a task-valid null control after reviewing inputs and metric
Protocol-specific applicability, permitted inputs, access, split, evaluator and execution requirements need review before implementation or execution.
This is a suggested selection rule, not a validated method or a measured score.
Proposed control: requires review
Select an upstream conventional reference after reviewing the full protocol
Protocol-specific applicability, permitted inputs, access, split, evaluator and execution requirements need review before implementation or execution.
This is a suggested selection rule, not a validated method or a measured score.
Protocol coverage CSV · Model evaluation matrix · Source table · Release and checksums
Coverage is derived from release 2026-09-25-8af07e960e5f. Source citations describe the original records; they do not validate an unreviewed baseline proposal. No results have been generated by this audit.
Score all 2,972 variants in AMFR_HUMAN_Tsuboyama_2023_4G3O with ESM-2 8M masked marginals. This single-assay result is not a full ProteinGym track or suite score.
Generate predictions and evaluate them. This recipe does not establish reproduction of a particular published score.
Source reviewed; these instructions have not been executed by rewire.
OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 python research/local-runs-2026-09-20/proteingym-esm2/reproduce.py \
--data /data/DMS_ProteinGym_substitutions \
--checkpoint /weights/esm2_t6_8M_UR50D.pt \
--output /new/private/amfr-runRun your model locally and return predictions keyed by the input IDs. The evaluator supplies biological inputs without test labels and owns scoring. This interface is not a sandbox for model code.
Pass your existing prediction function into this adapter. Its output direction must match the selected protocol.
class MyModelAdapter:
def __init__(self, score):
self.score = score
def predict(self, inputs):
return {row["id"]: float(self.score(row)) for row in inputs}
# adapter = MyModelAdapter(your_prediction_function)
# report = rewirebench.run(prepared, adapter, output="runs/my-model")Alternatively, generate a keyed prediction file in your existing model environment and use the score-only recipe. Your model code and weights do not need to be shared.
ProteinGym v1.3 AMFR substitution assay: reproduction instructions · Reproduction instructions and execution scopeContribute a result for review. The library can submit an exported evaluation for private review when intake is open. Check the contribution page for access and sign-in.
No runnable recipe has been reviewed for this protocol. Dataset access, model requirements, licences and compute requirements must be checked against its sources before execution.
Bounded review of pinned protocol sources and an existing public report. No model execution, numerical-result re-review, independent reproduction or named human scientific review was performed.
Stable record: rewire-protocol-proteingym-amfr-v13Explanatory profile: limited source coverage · Automated source review, 2026-09-23. Review applies to the cited claims; unresolved fields are listed below. Numerical results retain their own review status.
| Property | Description and evidence |
|---|---|
| Assay and dataset | AMFR_HUMAN_Tsuboyama_2023_4G3O in ProteinGym v1.3 DMS substitutions: human AMFR, 47 residues, Stability selection category, cDNA-display proteolysis; 820 single plus 2,152 multiple substitutions.SourcesProtocol metadata evidence: proteingym-reference_files-DMS_substitutions.csv · DMS_substitutions.csv: AMFR row, taxon, seq_len, selection_assay, selection_type, DMS_number_single_mutants, DMS_number_multiple_mutants |
| Version and reference | Report preserved at runner commit ca73fa47136d182f2d4ddb083d084712198fc0e2. Report provenance pins upstream revision 144fe22b07dfaeec2b366f2346203a9838a55b4c and reference SHA-256 a8f498011532a74aa9fe556a50555a75e928c5837d19c06a87592ae04049b308.SourcesESM-2 8M masked-marginal scoring: local execution report (20 September 2026) · source record: version, artifact_sha256; report.json: /provenance/upstream_revision, /provenance/reference_sha256 |
| Checkpoint and scoring | The report names esm2_t6_8M_UR50D and checkpoint SHA-256 46f002a9870c9bdecd0ea887acb1f9a38a6b561e8f8bf8a6990b679b9d31b928, with wild-type-context masked marginals summed over substitutions. This is reported provenance, not a newly verified weight download or a checkpoint assignment to other result rows.SourcesESM-2 8M masked-marginal scoring: local execution report (20 September 2026) · report.json: /execution/adapter_provenance/model, /checkpoint_sha256, /strategy (all under /execution/adapter_provenance) |
| Inputs and fitting | Wild-type protein sequence and amino-acid substitutions; no MSA, structure or assay labels in adapter inputs. The execution reports fitting=none. Official DMS_score_bin labels and higher-is-better target orientation are retained by the evaluator.Sources (2)ESM-2 8M masked-marginal scoring: local execution report (20 September 2026); ProteinGym v1.3 AMFR substitution assay: reproduction instructions · report.json: /input_information, /execution/fitting, /protocol_configuration/binary_labels, /protocol_configuration/score_direction; local README.md: Data and method |
| Splits | All variants of this selected assay are evaluated with no new supervised train/test split or assay-label fitting. Selection before observing scores is reported in the local documentation; this review did not independently verify that chronology.Sources (2)ProteinGym v1.3 AMFR substitution assay: reproduction instructions; ESM-2 8M masked-marginal scoring: local execution report (20 September 2026) · local README.md: opening paragraph and Data and method; report.json: /execution/fitting |
| Coverage | 2,972/2,972 selected variants scored, zero unscored; one complete assay out of 217. Report scope=subset, completion=partial and protocol status=partial_track. Both top-level and protocol suite metrics are empty.SourcesESM-2 8M masked-marginal scoring: local execution report (20 September 2026) · report.json: /coverage, /scope, /completion, /metrics, /protocol_results/metrics, /protocol_results/complete_assays, /protocol_results/total_assays |
| Metrics | The preserved per-assay report contains Spearman, AUC, MCC, NDCG and Top_recall, rounded to three decimals. Higher is better; the reported negative correlation is retained. The local documentation reports agreement with upstream formulas; this review neither recomputed nor changed these result values.Sources (3)ESM-2 8M masked-marginal scoring: local execution report (20 September 2026); ProteinGym v1.3 AMFR substitution assay: reproduction instructions; Pinned ProteinGym protocol implementation · report.json: /protocol_results/per_assay/AMFR_HUMAN_Tsuboyama_2023_4G3O/metrics; local README.md: metric table and Execution and verification; pinned protocols/proteingym.py lines 157–184 |
| Aggregation | The report records assay rounding followed by UniProt/selection-type and equal-category means. Its AMFR and Stability summaries contain only this selected assay, so they are not full-track estimates or independent observations.Sources (2)ESM-2 8M masked-marginal scoring: local execution report (20 September 2026); Pinned ProteinGym protocol implementation · report.json: /protocol_configuration/aggregation, /protocol_results/by_protein_selection_type, /protocol_results/by_selection_type, /protocol_results/metrics; protocols/proteingym.py lines 221–249 |
| Uncertainty | No uncertainty interval was estimated in this report. Upstream cross-model bootstrap differences are not an absolute confidence interval for this assay; no seed-variability estimate is supplied.SourcesESM-2 8M masked-marginal scoring: local execution report (20 September 2026) · report.json: /protocol_results/uncertainty, /model_configuration/seed |
| Data verification | The report records local AMFR CSV SHA-256 dd911d925eca79329a496eb6cb7b181ab448e15db244daf1c1f9c01e8a704c2a and a successful expected-hash check, while explicitly retaining local_bytes_hashed_not_independently_source_verified. Matching those bytes does not independently authenticate the official archive.SourcesESM-2 8M masked-marginal scoring: local execution report (20 September 2026) · report.json: /provenance/assay_sha256/AMFR_HUMAN_Tsuboyama_2023_4G3O, /provenance/expected_hashes_checked, /provenance/data_verification |
| Execution environment | The report records rewirebench 0.4.0, Python 3.11.13, fair-esm 2.0.0, macOS arm64, CPU, seed 0 and one thread. Container digest/runtime and SIF SHA-256 are unreported. No accelerator or container execution is established.SourcesESM-2 8M masked-marginal scoring: local execution report (20 September 2026) · report.json: /environment, /execution/adapter_provenance, /model_configuration |
| Training overlap | The report explicitly leaves training overlap unreported and notes that UniRef50 pretraining may overlap benchmark proteins. This review does not establish absence or extent of overlap. · Not reported in inspected sourcesSourcesESM-2 8M masked-marginal scoring: local execution report (20 September 2026) · report.json: /model/training_overlap, /execution/adapter_provenance/training_overlap |
| Reproduction status | The report states independently_reproduced=false and published_result_reproduction=false. Source review and the original local formula checks do not establish independent reproduction of a published model result or named human scientific review.Sources (2)ESM-2 8M masked-marginal scoring: local execution report (20 September 2026); ProteinGym v1.3 AMFR substitution assay: reproduction instructions · report.json: /independently_reproduced, /execution/adapter_provenance/published_result_reproduction, /review_status; local README.md: Execution and verification |
| Organisms | Not extracted or verified for this record. |
| Baselines | Not extracted or verified for this record. |
Source checking verifies the cited claim or transcription. It does not establish independent reproduction.
Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.
One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.
40 evidence rows matching the loaded filters
| Property and statement | Original source and location | Review and provenance |
|---|---|---|
| Assay and dataset AMFR_HUMAN_Tsuboyama_2023_4G3O in ProteinGym v1.3 DMS substitutions: human AMFR, 47 residues, Stability selection category, cDNA-display proteolysis; 820 single plus 2,152 multiple substitutions. Individual claims | Protocol metadata evidence: proteingym-reference_files-DMS_substitutions.csv DMS_substitutions.csv: AMFR row, taxon, seq_len, selection_assay, selection_type, DMS_number_single_mutants, DMS_number_multiple_mutants Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c | source checked automated source review · 2026-09-23 Audit detailsBounded review of pinned protocol sources and an existing public report. No model execution, numerical-result re-review, independent reproduction or named human scientific review was performed. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Version and reference Report preserved at runner commit ca73fa47136d182f2d4ddb083d084712198fc0e2. Report provenance pins upstream revision 144fe22b07dfaeec2b366f2346203a9838a55b4c and reference SHA-256 a8f498011532a74aa9fe556a50555a75e928c5837d19c06a87592ae04049b308. Individual claims | ESM-2 8M masked-marginal scoring: local execution report (20 September 2026) source record: version, artifact_sha256; report.json: /provenance/upstream_revision, /provenance/reference_sha256 Version: ca73fa47136d182f2d4ddb083d084712198fc0e2 | source checked automated source review · 2026-09-23 Audit detailsBounded review of pinned protocol sources and an existing public report. No model execution, numerical-result re-review, independent reproduction or named human scientific review was performed. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Execution environment The report records rewirebench 0.4.0, Python 3.11.13, fair-esm 2.0.0, macOS arm64, CPU, seed 0 and one thread. Container digest/runtime and SIF SHA-256 are unreported. No accelerator or container execution is established. Individual claims | ESM-2 8M masked-marginal scoring: local execution report (20 September 2026) report.json: /environment, /execution/adapter_provenance, /model_configuration Version: ca73fa47136d182f2d4ddb083d084712198fc0e2 | source checked automated source review · 2026-09-23 Audit detailsBounded review of pinned protocol sources and an existing public report. No model execution, numerical-result re-review, independent reproduction or named human scientific review was performed. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Training overlap The report explicitly leaves training overlap unreported and notes that UniRef50 pretraining may overlap benchmark proteins. This review does not establish absence or extent of overlap. Individual claims | ESM-2 8M masked-marginal scoring: local execution report (20 September 2026) report.json: /model/training_overlap, /execution/adapter_provenance/training_overlap Version: ca73fa47136d182f2d4ddb083d084712198fc0e2 | unreported automated source review · 2026-09-23 Audit detailsBounded review of pinned protocol sources and an existing public report. No model execution, numerical-result re-review, independent reproduction or named human scientific review was performed. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Reproduction status The report states independently_reproduced=false and published_result_reproduction=false. Source review and the original local formula checks do not establish independent reproduction of a published model result or named human scientific review. Individual claims | ProteinGym v1.3 AMFR substitution assay: reproduction instructions report.json: /independently_reproduced, /execution/adapter_provenance/published_result_reproduction, /review_status; local README.md: Execution and verification Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: ca73fa47136d182f2d4ddb083d084712198fc0e2 | source checked automated source review · 2026-09-23 Audit detailsBounded review of pinned protocol sources and an existing public report. No model execution, numerical-result re-review, independent reproduction or named human scientific review was performed. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Reproduction status The report states independently_reproduced=false and published_result_reproduction=false. Source review and the original local formula checks do not establish independent reproduction of a published model result or named human scientific review. Individual claims | ESM-2 8M masked-marginal scoring: local execution report (20 September 2026) report.json: /independently_reproduced, /execution/adapter_provenance/published_result_reproduction, /review_status; local README.md: Execution and verification Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: ca73fa47136d182f2d4ddb083d084712198fc0e2 | source checked automated source review · 2026-09-23 Audit detailsBounded review of pinned protocol sources and an existing public report. No model execution, numerical-result re-review, independent reproduction or named human scientific review was performed. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Checkpoint and scoring The report names esm2_t6_8M_UR50D and checkpoint SHA-256 46f002a9870c9bdecd0ea887acb1f9a38a6b561e8f8bf8a6990b679b9d31b928, with wild-type-context masked marginals summed over substitutions. This is reported provenance, not a newly verified weight download or a checkpoint assignment to other result rows. Individual claims | ESM-2 8M masked-marginal scoring: local execution report (20 September 2026) report.json: /execution/adapter_provenance/model, /checkpoint_sha256, /strategy (all under /execution/adapter_provenance) Version: ca73fa47136d182f2d4ddb083d084712198fc0e2 | source checked automated source review · 2026-09-23 Audit detailsBounded review of pinned protocol sources and an existing public report. No model execution, numerical-result re-review, independent reproduction or named human scientific review was performed. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Inputs and fitting Wild-type protein sequence and amino-acid substitutions; no MSA, structure or assay labels in adapter inputs. The execution reports fitting=none. Official DMS_score_bin labels and higher-is-better target orientation are retained by the evaluator. Individual claims | ProteinGym v1.3 AMFR substitution assay: reproduction instructions report.json: /input_information, /execution/fitting, /protocol_configuration/binary_labels, /protocol_configuration/score_direction; local README.md: Data and method Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: ca73fa47136d182f2d4ddb083d084712198fc0e2 | source checked automated source review · 2026-09-23 Audit detailsBounded review of pinned protocol sources and an existing public report. No model execution, numerical-result re-review, independent reproduction or named human scientific review was performed. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Inputs and fitting Wild-type protein sequence and amino-acid substitutions; no MSA, structure or assay labels in adapter inputs. The execution reports fitting=none. Official DMS_score_bin labels and higher-is-better target orientation are retained by the evaluator. Individual claims | ESM-2 8M masked-marginal scoring: local execution report (20 September 2026) report.json: /input_information, /execution/fitting, /protocol_configuration/binary_labels, /protocol_configuration/score_direction; local README.md: Data and method Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: ca73fa47136d182f2d4ddb083d084712198fc0e2 | source checked automated source review · 2026-09-23 Audit detailsBounded review of pinned protocol sources and an existing public report. No model execution, numerical-result re-review, independent reproduction or named human scientific review was performed. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Splits All variants of this selected assay are evaluated with no new supervised train/test split or assay-label fitting. Selection before observing scores is reported in the local documentation; this review did not independently verify that chronology. Individual claims | ProteinGym v1.3 AMFR substitution assay: reproduction instructions local README.md: opening paragraph and Data and method; report.json: /execution/fitting Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: ca73fa47136d182f2d4ddb083d084712198fc0e2 | source checked automated source review · 2026-09-23 Audit detailsBounded review of pinned protocol sources and an existing public report. No model execution, numerical-result re-review, independent reproduction or named human scientific review was performed. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
View linked audit checks and correction history
Release 2026-09-25-8af07e960e5f · Record review: source checked
Stable ID: rewire-protocol-proteingym-amfr-v13