rewire.it
Protocol

ProteinGym v1.3 zero-shot DMS substitutions

The zero-shot DMS substitution protocol in ProteinGym v1.3. This protocol record does not reassign historical ProteinGym results whose exact release is unverified.

0 evaluations · 0 metric rows

Overview

The zero-shot DMS substitution protocol in ProteinGym v1.3. This protocol record does not reassign historical ProteinGym results whose exact release is unverified.

Consult the linked sources for architecture or protocol details. Missing evidence is not evidence of a missing capability.

Evaluation design

Benchmarks bring together tasks and protocols. A task describes the biological question; a protocol defines a particular test.

Benchmarks

These source-backed links do not make different protocols or scores interchangeable.

Run this benchmark

ProteinGym v1.3: score mutation predictions

Score supplied zero-shot substitution predictions using the pinned ProteinGym evaluator definitions and report explicit per-assay coverage.

Recompute metrics from supplied predictions. This recipe does not establish reproduction of a particular published score.

Dataset access
Download and extract the official ProteinGym v1.3 DMS substitutions archive. The packaged reference metadata are pinned to upstream 144fe22b07dfaeec2b366f2346203a9838a55b4c; input assay bytes are hashed during preparation.
Model and weights
No model or weights are needed to score existing mutation predictions.
Licences
Runner code is MIT and retains upstream evaluator notices. Dataset and checkpoint reuse terms must be checked at their original sources.
Software
Python 3.11 with the pinned core environment; no GPU framework required for scoring.
Hardware
CPU scoring. Full-track storage, memory and runtime must be assessed locally; no universal resource estimate is established.
Required inputs and expected outputs

Inputs

  • Extracted official v1.3 DMS substitution assay CSV files.
  • Keyed prediction files; use the IDs from prepared input rather than constructing ambiguous mutation IDs.

Outputs

  • Local report.json with metrics, coverage, protocol and artifact hashes.
  • Local predictions.json and unscored.json; neither is submitted automatically.

Prepare a selected assay and score predictions

Source reviewed; these instructions have not been executed by rewire.

import rewirebench
prepared = rewirebench.prepare(
    "proteingym-v1.3-dms-substitutions", source="data/DMS_ProteinGym_substitutions",
    output="prepared", assay_ids=["AMFR_HUMAN_Tsuboyama_2023_4G3O"],
)
report = rewirebench.evaluate(
    prepared, "predictions/predictions.tsv", output="runs/proteingym-rescore",
    model={"name": "My protein model", "training_overlap": "Unreported"},
)
# Omit assay_ids only when preparing all assays for an explicitly requested full-track run.

rewirebench: Library guide; rewirebench: ProteinGym guide; rewirebench: ProteinGym scoring code; rewirebench: ESM-2 adapter code · docs/proteingym.md: preparation, selected assays and evaluation
Use your own model

Run your model locally and return predictions keyed by the input IDs. The evaluator supplies biological inputs without test labels and owns scoring. This interface is not a sandbox for model code.

Pass your existing prediction function into this adapter. Its output direction must match the selected protocol.

class MyModelAdapter:
    def __init__(self, score):
        self.score = score

    def predict(self, inputs):
        return {row["id"]: float(self.score(row)) for row in inputs}

# adapter = MyModelAdapter(your_prediction_function)
# report = rewirebench.run(prepared, adapter, output="runs/my-model")

Alternatively, generate a keyed prediction file in your existing model environment and use the score-only recipe. Your model code and weights do not need to be shared.

rewirebench: Library guide; rewirebench: ProteinGym guide; rewirebench: ProteinGym scoring code; rewirebench: ESM-2 adapter code · Pinned runner documentation, protocol implementation and adapter implementation
Scope and limitations
  • Only zero-shot DMS substitutions are supported. Indels, supervised tracks and clinical tasks are excluded.
  • A selected assay or incomplete coverage is not a full-suite score.
  • Preserve mutation identities and upstream score direction and aggregation.
  • Input hashes record local bytes; they do not independently establish source authenticity.

Contribute a result for review. The library can prepare a private submission; production submissions remain disabled until verified email delivery is enabled.

Original repository instructions

Run this benchmark

Prepare official v1.3 substitution assays, generate or supply model scores, and evaluate the selected track. Full-suite scores require complete track coverage.

A maintained rewire runner has not been verified for this benchmark. Check data access, weights, licences, dependencies and hardware in the linked official documentation; requirements have not been fully extracted.

rewirebench: Library guide; rewirebench: ProteinGym guide; rewirebench: ProteinGym scoring code; rewirebench: ESM-2 adapter code · docs/proteingym.md and pinned protocols/proteingym.py

Tested entities and results

Release 2026-09-17-b9bc163c8ab1 · 0 evaluations · 0 metric rows. Different protocols are not a single leaderboard.

No evaluations linked in this release.

Evidence table

Inspect claims, sources and review details

Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.

One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.

63 evidence rows matching the loaded filters

Claims, original sources and review scope · Release 2026-09-17-b9bc163c8ab1
Property and statementOriginal source and locationReview and provenance
Run recipe: inputs

["Extracted official v1.3 DMS substitution assay CSV files.","Keyed prediction files; use the IDs from prepared input rather than constructing ambiguous mutation IDs."]

Individual claims		rewirebench: ProteinGym guide

Original source ↗

Pinned runner documentation, protocol implementation and adapter implementation

Shared locator for this statement’s cited sources; not a separate locator for each citation.

Version: f80cef7f818bec33e51b7f43ad499eb5078c8d87
Retrieved: 2026-09-17

source checked

implementation review

Audit details

Recipe metadata reviewed against pinned sources; not experimental reproduction.

Field: attributes.run_recipes.0.inputs

Source artifact SHA-256: d2b1fc009acc590b82ca125d886f3d52b07c2f36bbe0e9cb0d30776618166b5f

Hash scope: complete file bytes

Format: text

Run recipe: inputs

["Extracted official v1.3 DMS substitution assay CSV files.","Keyed prediction files; use the IDs from prepared input rather than constructing ambiguous mutation IDs."]

Individual claims		rewirebench: Library guide

Original source ↗

Pinned runner documentation, protocol implementation and adapter implementation

Shared locator for this statement’s cited sources; not a separate locator for each citation.

Version: f80cef7f818bec33e51b7f43ad499eb5078c8d87
Retrieved: 2026-09-17

source checked

implementation review

Audit details

Recipe metadata reviewed against pinned sources; not experimental reproduction.

Field: attributes.run_recipes.0.inputs

Source artifact SHA-256: 798cc447f50b7a3c84e67d8eabe1a0dc148be3ae2ec2c9986f0181494c14da45

Hash scope: complete file bytes

Format: text

Run recipe: inputs

["Extracted official v1.3 DMS substitution assay CSV files.","Keyed prediction files; use the IDs from prepared input rather than constructing ambiguous mutation IDs."]

Individual claims		rewirebench: ESM-2 adapter code

Original source ↗

Pinned runner documentation, protocol implementation and adapter implementation

Shared locator for this statement’s cited sources; not a separate locator for each citation.

Version: f80cef7f818bec33e51b7f43ad499eb5078c8d87
Retrieved: 2026-09-17

source checked

implementation review

Audit details

Recipe metadata reviewed against pinned sources; not experimental reproduction.

Field: attributes.run_recipes.0.inputs

Source artifact SHA-256: 4ff38cfbb91d718b52a44da53aa5594fe73d9d1d4d0ce43724807597c3929889

Hash scope: complete file bytes

Format: text

Run recipe: inputs

["Extracted official v1.3 DMS substitution assay CSV files.","Keyed prediction files; use the IDs from prepared input rather than constructing ambiguous mutation IDs."]

Individual claims		rewirebench: ProteinGym scoring code

Original source ↗

Pinned runner documentation, protocol implementation and adapter implementation

Shared locator for this statement’s cited sources; not a separate locator for each citation.

Version: f80cef7f818bec33e51b7f43ad499eb5078c8d87
Retrieved: 2026-09-17

source checked

implementation review

Audit details

Recipe metadata reviewed against pinned sources; not experimental reproduction.

Field: attributes.run_recipes.0.inputs

Source artifact SHA-256: 2d62959db5611b1c7c1c7e8e44056e96c90b3f52975b2080bc36da576f0ac871

Hash scope: complete file bytes

Format: text

Run recipe: Prepare a selected assay and score predictions

import rewirebench prepared = rewirebench.prepare( "proteingym-v1.3-dms-substitutions", source="data/DMS_ProteinGym_substitutions", output="prepared", assay_ids=["AMFR_HUMAN_Tsuboyama_2023_4G3O"], ) report = rewirebench.evaluate( prepared, "predictions/predictions.tsv", output="runs/proteingym-rescore", model={"name": "My protein model", "training_overlap": "Unreported"}, ) # Omit assay_ids only when preparing all assays for an explicitly requested full-track run.

Individual claims		rewirebench: ProteinGym guide

Original source ↗

docs/proteingym.md: preparation, selected assays and evaluation

Shared locator for this statement’s cited sources; not a separate locator for each citation.

Version: f80cef7f818bec33e51b7f43ad499eb5078c8d87
Retrieved: 2026-09-17

source checked

implementation review

Audit details

Execution status: source_reviewed_not_executed. Execution receipts apply only to their stated scope; not experimental reproduction.

Field: attributes.run_recipes.0.instructions.0.code

Source artifact SHA-256: d2b1fc009acc590b82ca125d886f3d52b07c2f36bbe0e9cb0d30776618166b5f

Hash scope: complete file bytes

Format: text

Run recipe: Prepare a selected assay and score predictions

import rewirebench prepared = rewirebench.prepare( "proteingym-v1.3-dms-substitutions", source="data/DMS_ProteinGym_substitutions", output="prepared", assay_ids=["AMFR_HUMAN_Tsuboyama_2023_4G3O"], ) report = rewirebench.evaluate( prepared, "predictions/predictions.tsv", output="runs/proteingym-rescore", model={"name": "My protein model", "training_overlap": "Unreported"}, ) # Omit assay_ids only when preparing all assays for an explicitly requested full-track run.

Individual claims		rewirebench: Library guide

Original source ↗

docs/proteingym.md: preparation, selected assays and evaluation

Shared locator for this statement’s cited sources; not a separate locator for each citation.

Version: f80cef7f818bec33e51b7f43ad499eb5078c8d87
Retrieved: 2026-09-17

source checked

implementation review

Audit details

Execution status: source_reviewed_not_executed. Execution receipts apply only to their stated scope; not experimental reproduction.

Field: attributes.run_recipes.0.instructions.0.code

Source artifact SHA-256: 798cc447f50b7a3c84e67d8eabe1a0dc148be3ae2ec2c9986f0181494c14da45

Hash scope: complete file bytes

Format: text

Run recipe: Prepare a selected assay and score predictions

import rewirebench prepared = rewirebench.prepare( "proteingym-v1.3-dms-substitutions", source="data/DMS_ProteinGym_substitutions", output="prepared", assay_ids=["AMFR_HUMAN_Tsuboyama_2023_4G3O"], ) report = rewirebench.evaluate( prepared, "predictions/predictions.tsv", output="runs/proteingym-rescore", model={"name": "My protein model", "training_overlap": "Unreported"}, ) # Omit assay_ids only when preparing all assays for an explicitly requested full-track run.

Individual claims		rewirebench: ESM-2 adapter code

Original source ↗

docs/proteingym.md: preparation, selected assays and evaluation

Shared locator for this statement’s cited sources; not a separate locator for each citation.

Version: f80cef7f818bec33e51b7f43ad499eb5078c8d87
Retrieved: 2026-09-17

source checked

implementation review

Audit details

Execution status: source_reviewed_not_executed. Execution receipts apply only to their stated scope; not experimental reproduction.

Field: attributes.run_recipes.0.instructions.0.code

Source artifact SHA-256: 4ff38cfbb91d718b52a44da53aa5594fe73d9d1d4d0ce43724807597c3929889

Hash scope: complete file bytes

Format: text

Run recipe: Prepare a selected assay and score predictions

import rewirebench prepared = rewirebench.prepare( "proteingym-v1.3-dms-substitutions", source="data/DMS_ProteinGym_substitutions", output="prepared", assay_ids=["AMFR_HUMAN_Tsuboyama_2023_4G3O"], ) report = rewirebench.evaluate( prepared, "predictions/predictions.tsv", output="runs/proteingym-rescore", model={"name": "My protein model", "training_overlap": "Unreported"}, ) # Omit assay_ids only when preparing all assays for an explicitly requested full-track run.

Individual claims		rewirebench: ProteinGym scoring code

Original source ↗

docs/proteingym.md: preparation, selected assays and evaluation

Shared locator for this statement’s cited sources; not a separate locator for each citation.

Version: f80cef7f818bec33e51b7f43ad499eb5078c8d87
Retrieved: 2026-09-17

source checked

implementation review

Audit details

Execution status: source_reviewed_not_executed. Execution receipts apply only to their stated scope; not experimental reproduction.

Field: attributes.run_recipes.0.instructions.0.code

Source artifact SHA-256: 2d62959db5611b1c7c1c7e8e44056e96c90b3f52975b2080bc36da576f0ac871

Hash scope: complete file bytes

Format: text

Run recipe: Prepare a selected assay and score predictions

git clone https://github.com/rewire-bio/rewire-benchmarks.git cd rewire-benchmarks git checkout --detach f80cef7f818bec33e51b7f43ad499eb5078c8d87 uv sync --locked --package rewirebench --python 3.11 uv run --package rewirebench rewirebench prepare proteingym-v1.3-dms-substitutions \ --source data/DMS_ProteinGym_substitutions --output prepared \ --options '{"assay_ids":["AMFR_HUMAN_Tsuboyama_2023_4G3O"]}' uv run --package rewirebench rewirebench evaluate --prepared prepared \ --predictions predictions/predictions.tsv --output runs/proteingym-rescore \ --model-name "My protein model"

Individual claims		rewirebench: ProteinGym guide

Original source ↗

docs/proteingym.md: CLI preparation and scoring

Shared locator for this statement’s cited sources; not a separate locator for each citation.

Version: f80cef7f818bec33e51b7f43ad499eb5078c8d87
Retrieved: 2026-09-17

source checked

implementation review

Audit details

Execution status: source_reviewed_not_executed. Execution receipts apply only to their stated scope; not experimental reproduction.

Field: attributes.run_recipes.0.instructions.1.code

Source artifact SHA-256: d2b1fc009acc590b82ca125d886f3d52b07c2f36bbe0e9cb0d30776618166b5f

Hash scope: complete file bytes

Format: text

Run recipe: Prepare a selected assay and score predictions

git clone https://github.com/rewire-bio/rewire-benchmarks.git cd rewire-benchmarks git checkout --detach f80cef7f818bec33e51b7f43ad499eb5078c8d87 uv sync --locked --package rewirebench --python 3.11 uv run --package rewirebench rewirebench prepare proteingym-v1.3-dms-substitutions \ --source data/DMS_ProteinGym_substitutions --output prepared \ --options '{"assay_ids":["AMFR_HUMAN_Tsuboyama_2023_4G3O"]}' uv run --package rewirebench rewirebench evaluate --prepared prepared \ --predictions predictions/predictions.tsv --output runs/proteingym-rescore \ --model-name "My protein model"

Individual claims		rewirebench: Library guide

Original source ↗

docs/proteingym.md: CLI preparation and scoring

Shared locator for this statement’s cited sources; not a separate locator for each citation.

Version: f80cef7f818bec33e51b7f43ad499eb5078c8d87
Retrieved: 2026-09-17

source checked

implementation review

Audit details

Execution status: source_reviewed_not_executed. Execution receipts apply only to their stated scope; not experimental reproduction.

Field: attributes.run_recipes.0.instructions.1.code

Source artifact SHA-256: 798cc447f50b7a3c84e67d8eabe1a0dc148be3ae2ec2c9986f0181494c14da45

Hash scope: complete file bytes

Format: text

All evidence (CSV)All evidence (JSONL)How to read the evidence

Sources and history

Release 2026-09-17-b9bc163c8ab1 · Record review: source checked

5 source records and release historyDownload this release
Technical metadata and extraction receipts

Stable ID: rewire-proteingym-v1-3-dms-substitutions

domain
proteomics
entity level
protocol
protocol version
v1.3
upstream revision
144fe22b07dfaeec2b366f2346203a9838a55b4c
run documentation
record id: rewire-proteingym-v1-3-dms-substitutions; status: source_reviewed_not_executed; summary: Prepare official v1.3 substitution assays, generate or supply model scores, and evaluate the selected track. Full-suite scores require complete track coverage.; source ids: runner-recipe-docs-sdk-md-f80cef7f; runner-recipe-docs-proteingym-md-f80cef7f; runner-recipe-packages-rewirebench-src-rewirebench-protocols-proteingym-py-f80cef7f; runner-recipe-packages-rewirebench-src-rewirebench-adapters-esm-py-f80cef7f; source locator: docs/proteingym.md and pinned protocols/proteingym.py
run recipes
id: proteingym-v1-3-rescore; protocol id: proteingym-v1.3-dms-substitutions; version: f80cef7f818bec33e51b7f43ad499eb5078c8d87; title: ProteinGym v1.3: score mutation predictions; purpose: rescore_predictions; summary: Score supplied zero-shot substitution predictions using the pinned ProteinGym evaluator definitions and report explicit per-assay coverage.; inputs: Extracted official v1.3 DMS substitution assay CSV files.; Keyed prediction files; use the IDs from prepared input rather than constructing ambiguous mutation IDs.; outputs: Local report.json with metrics, coverage, protocol and artifact hashes.; Local predictions.json and unscored.json; neither is submitted automatically.; requirements: data: Download and extract the official ProteinGym v1.3 DMS substitutions archive. The packaged reference metadata are pinned to upstream 144fe22b07dfaeec2b366f2346203a9838a55b4c; input assay bytes are hashed during preparation.; weights: No model or weights are needed to score existing mutation predictions.; licence: Runner code is MIT and retains upstream evaluator notices. Dataset and checkpoint reuse terms must be checked at their original sources.; software: Python 3.11 with the pinned core environment; no GPU framework required for scoring.; hardware: CPU scoring. Full-track storage, memory and runtime must be assessed locally; no universal resource estimate is established.; instructions: runtime: python; title: Prepare a selected assay and score predictions; code: import rewirebench prepared = rewirebench.prepare( "proteingym-v1.3-dms-substitutions", source="data/DMS_ProteinGym_substitutions", output="prepared", assay_ids=["AMFR_HUMAN_Tsuboyama_2023_4G3O"], ) report = rewirebench.evaluate( prepared, "predictions/predictions.tsv", output="runs/proteingym-rescore", model={"name": "My protein model", "training_overlap": "Unreported"}, ) # Omit assay_ids only when preparing all assays for an explicitly requested full-track run.; status: source_reviewed_not_executed; source ids: runner-recipe-docs-sdk-md-f80cef7f; runner-recipe-docs-proteingym-md-f80cef7f; runner-recipe-packages-rewirebench-src-rewirebench-protocols-proteingym-py-f80cef7f; runner-recipe-packages-rewirebench-src-rewirebench-adapters-esm-py-f80cef7f; source locator: docs/proteingym.md: preparation, selected assays and evaluation; runtime: command_line; title: Prepare a selected assay and score predictions; code: git clone https://github.com/rewire-bio/rewire-benchmarks.git cd rewire-benchmarks git checkout --detach f80cef7f818bec33e51b7f43ad499eb5078c8d87 uv sync --locked --package rewirebench --python 3.11 uv run --package rewirebench rewirebench prepare proteingym-v1.3-dms-substitutions \ --source data/DMS_ProteinGym_substitutions --output prepared \ --options '{"assay_ids":["AMFR_HUMAN_Tsuboyama_2023_4G3O"]}' uv run --package rewirebench rewirebench evaluate --prepared prepared \ --predictions predictions/predictions.tsv --output runs/proteingym-rescore \ --model-name "My protein model"; status: source_reviewed_not_executed; source ids: runner-recipe-docs-sdk-md-f80cef7f; runner-recipe-docs-proteingym-md-f80cef7f; runner-recipe-packages-rewirebench-src-rewirebench-protocols-proteingym-py-f80cef7f; runner-recipe-packages-rewirebench-src-rewirebench-adapters-esm-py-f80cef7f; source locator: docs/proteingym.md: CLI preparation and scoring; runtime: podman; title: Score in a local Podman image; code: # Build the pinned core image as described in the linked HPC guide first. mkdir -p runs podman run --rm --userns=keep-id --network=none \ -v "$PWD/prepared:/prepared:ro" \ -v "$PWD/predictions:/predictions:ro" \ -v "$PWD/runs:/outputs:rw" \ localhost/rewirebench:core evaluate \ --prepared /prepared --predictions /predictions/predictions.tsv \ --output /outputs/proteingym-v1-3-rescore; status: source_reviewed_not_executed; source ids: runner-recipe-docs-hpc-md-f80cef7f; runner-recipe-docs-sdk-md-f80cef7f; source locator: docs/hpc.md: Podman build, read-only input mounts and offline execution; docs/sdk.md: evaluate; runtime: apptainer; title: Score in an Apptainer image; code: # Build or obtain the pinned SIF as described in the linked HPC guide first. mkdir -p runs apptainer run --cleanenv --containall \ --bind "$PWD/prepared:/prepared:ro" \ --bind "$PWD/predictions:/predictions:ro" \ --bind "$PWD/runs:/outputs:rw" \ rewirebench-core.sif evaluate \ --prepared /prepared --predictions /predictions/predictions.tsv \ --output /outputs/proteingym-v1-3-rescore; status: source_reviewed_not_executed; source ids: runner-recipe-docs-hpc-md-f80cef7f; runner-recipe-docs-sdk-md-f80cef7f; source locator: docs/hpc.md: Apptainer execution and source image identity; docs/sdk.md: evaluate; runtime: slurm; title: Submit the prepared scoring job to Slurm; code: #!/bin/bash set -euo pipefail # Set these for your cluster; no performance or resource estimate is implied. : "${REWIRE_ACCOUNT:?Set your Slurm account}" : "${REWIRE_PARTITION:?Set your Slurm partition}" : "${REWIRE_CPUS:?Set the requested CPU count}" : "${REWIRE_MEMORY:?Set the requested memory}" : "${REWIRE_WALLTIME:?Set the requested time limit}" : "${REWIRE_JOB_ROOT:?Set a shared absolute directory with prepared data and SIF}" export REWIRE_JOB_ROOT sbatch --account="$REWIRE_ACCOUNT" --partition="$REWIRE_PARTITION" \ --cpus-per-task="$REWIRE_CPUS" --mem="$REWIRE_MEMORY" \ --time="$REWIRE_WALLTIME" --export=ALL <<'REWIRE_JOB' #!/bin/bash set -euo pipefail cd "$REWIRE_JOB_ROOT" mkdir -p runs apptainer run --cleanenv --containall \ --bind "$PWD/prepared:/prepared:ro" \ --bind "$PWD/predictions:/predictions:ro" \ --bind "$PWD/runs:/outputs:rw" \ rewirebench-core.sif evaluate \ --prepared /prepared --predictions /predictions/predictions.tsv \ --output /outputs/proteingym-v1-3-rescore REWIRE_JOB; status: source_reviewed_not_executed; source ids: runner-recipe-docs-hpc-md-f80cef7f; runner-recipe-docs-sdk-md-f80cef7f; source locator: docs/hpc.md: Slurm template, preparation outside compute nodes and site-specific resources; limitations: Only zero-shot DMS substitutions are supported. Indels, supervised tracks and clinical tasks are excluded.; A selected assay or incomplete coverage is not a full-suite score.; Preserve mutation identities and upstream score direction and aggregation.; Input hashes record local bytes; they do not independently establish source authenticity.; source ids: runner-recipe-docs-sdk-md-f80cef7f; runner-recipe-docs-proteingym-md-f80cef7f; runner-recipe-packages-rewirebench-src-rewirebench-protocols-proteingym-py-f80cef7f; runner-recipe-packages-rewirebench-src-rewirebench-adapters-esm-py-f80cef7f; source locator: Pinned runner documentation, protocol implementation and adapter implementation; id: proteingym-v1-3-esm2; protocol id: proteingym-v1.3-dms-substitutions; version: f80cef7f818bec33e51b7f43ad499eb5078c8d87; title: ProteinGym v1.3: ESM-2 8M example; purpose: generate_and_evaluate; summary: Generate zero-shot substitution scores with the public ESM-2 8M checkpoint and score a limited assay smoke input.; inputs: Extracted ProteinGym v1.3 assay files.; The local official esm2_t6_8M_UR50D.pt checkpoint; its SHA-256 is verified before loading.; outputs: Local report.json with metrics, coverage, protocol and artifact hashes.; Local predictions.json and unscored.json; neither is submitted automatically.; requirements: data: Download and extract the official ProteinGym v1.3 DMS substitutions archive. The packaged reference metadata are pinned to upstream 144fe22b07dfaeec2b366f2346203a9838a55b4c; input assay bytes are hashed during preparation.; weights: ESM-2 esm2_t6_8M_UR50D checkpoint from the official fair-esm distribution; download before the offline job.; licence: Runner code is MIT and retains upstream evaluator notices. Dataset and checkpoint reuse terms must be checked at their original sources.; software: Python 3.11 with rewirebench[esm], fair-esm 2.0.0 and PyTorch.; hardware: CPU example. Sequences longer than 1,022 residues are explicitly unscored and remain in coverage; this example cannot produce complete predictions for tracks containing those inputs.; instructions: runtime: python; title: Run a limited ESM-2 example; code: import rewirebench from rewirebench.adapters.esm import ESM2Adapter prepared = rewirebench.prepare( "proteingym-v1.3-dms-substitutions", source="data/DMS_ProteinGym_substitutions", output="prepared-smoke", assay_ids=["AMFR_HUMAN_Tsuboyama_2023_4G3O"], limit=10, ) report = rewirebench.run( prepared, ESM2Adapter(checkpoint="weights/esm2_t6_8M_UR50D.pt"), output="runs/esm2-smoke", model={"name": "ESM-2 8M masked marginals", "training_overlap": "Unreported"}, ); status: source_reviewed_not_executed; source ids: runner-recipe-docs-sdk-md-f80cef7f; runner-recipe-docs-proteingym-md-f80cef7f; runner-recipe-packages-rewirebench-src-rewirebench-protocols-proteingym-py-f80cef7f; runner-recipe-packages-rewirebench-src-rewirebench-adapters-esm-py-f80cef7f; source locator: docs/proteingym.md: public ESM-2 example; adapters/esm.py: ESM2Adapter; runtime: command_line; title: Run a limited ESM-2 example; code: git clone https://github.com/rewire-bio/rewire-benchmarks.git cd rewire-benchmarks git checkout --detach f80cef7f818bec33e51b7f43ad499eb5078c8d87 uv sync --locked --package rewirebench --extra esm --python 3.11 uv run --package rewirebench --extra esm rewirebench prepare proteingym-v1.3-dms-substitutions --source data/DMS_ProteinGym_substitutions --output prepared-smoke --options '{"assay_ids":["AMFR_HUMAN_Tsuboyama_2023_4G3O"],"limit":10}' uv run --package rewirebench --extra esm rewirebench run --prepared prepared-smoke --adapter rewirebench.adapters.esm:ESM2Adapter --adapter-options '{"checkpoint":"weights/esm2_t6_8M_UR50D.pt"}' --output runs/esm2-smoke --model-name "ESM-2 8M masked marginals"; status: source_reviewed_not_executed; source ids: runner-recipe-docs-sdk-md-f80cef7f; runner-recipe-docs-proteingym-md-f80cef7f; runner-recipe-packages-rewirebench-src-rewirebench-protocols-proteingym-py-f80cef7f; runner-recipe-packages-rewirebench-src-rewirebench-adapters-esm-py-f80cef7f; source locator: docs/sdk.md: run; docs/proteingym.md: public ESM example; limitations: Only zero-shot DMS substitutions are supported. Indels, supervised tracks and clinical tasks are excluded.; A selected assay or incomplete coverage is not a full-suite score.; Preserve mutation identities and upstream score direction and aggregation.; Input hashes record local bytes; they do not independently establish source authenticity.; Masked-marginal log odds are calculated in wild-type context and summed over substitutions.; This is an example adapter, not reproduction of a published ESM ProteinGym result.; source ids: runner-recipe-docs-sdk-md-f80cef7f; runner-recipe-docs-proteingym-md-f80cef7f; runner-recipe-packages-rewirebench-src-rewirebench-protocols-proteingym-py-f80cef7f; runner-recipe-packages-rewirebench-src-rewirebench-adapters-esm-py-f80cef7f; source locator: Pinned runner documentation, protocol implementation and adapter implementation
Related records

Suggest a correction