ProteinGym mutation-effect evaluation compares variant scores with measurements from individual functional assays.
Explanatory profile: source reviewed · Automated source review, 2026-09-16. Review applies to the cited claims; unresolved fields are listed below. Numerical results retain their own review status.
| Property | Description and evidence |
|---|---|
| Entity type | Task guide; concrete protocol identities remain separate.SourcesOATML-Markslab/ProteinGym official source · README: Overview, Results, benchmark baselines and contribution notes on aggregation |
| Datasets | Select a particular ProteinGym release, assay subset and evaluation track; the suite contains separate substitution and indel resources.SourcesOATML-Markslab/ProteinGym official source · README: Overview, Results, benchmark baselines and contribution notes on aggregation |
| Organisms | No shared organism population is defined at this guide level. Record it for each selected dataset. · Not applicableSourcesOATML-Markslab/ProteinGym official source · README: Overview, Results, benchmark baselines and contribution notes on aggregation |
| Assays | Deep mutational scanning measurements for molecular effects; assay metadata and functional categories remain distinct.SourcesOATML-Markslab/ProteinGym official source · README: Overview, Results, benchmark baselines and contribution notes on aggregation |
| Splits | No executable split is attached to this task identity. Use the selected protocol’s split manifest. · Not applicableSourcesOATML-Markslab/ProteinGym official source · README: Overview, Results, benchmark baselines and contribution notes on aggregation |
| Allowed inputs | Protein variants, permitted sequence or structure information, and the selected assay’s reference measurements.SourcesOATML-Markslab/ProteinGym official source · README: Overview, Results, benchmark baselines and contribution notes on aggregation |
| Adaptation | No common fitting regime is imposed here. Keep pretrained, frozen, probed, fine-tuned and conventional methods distinct where applicable. · Not applicableSourcesOATML-Markslab/ProteinGym official source · README: Overview, Results, benchmark baselines and contribution notes on aggregation |
| Metrics | DMS zero-shot tracks include Spearman, NDCG, AUC, MCC and top-k recall; supervised tracks include Spearman and MSE. Choose the track-specific definition.SourcesOATML-Markslab/ProteinGym official source · README: Overview, Results, benchmark baselines and contribution notes on aggregation |
| Baselines | ProteinGym supplies single-sequence, alignment-based and other comparator scores; their extra information and supervision must remain visible.SourcesOATML-Markslab/ProteinGym official source · README: Overview, Results, benchmark baselines and contribution notes on aggregation |
Conceptual task guide. Dataset preparation, parameters and scoring must come from a separately identified protocol.
Select the suite release, assay subset and zero-shot or supervised track. ProteinGym reports within-assay metrics and further aggregation by protein and functional category. Use the published aggregation rules; a simple mean across all assays is not automatically the suite’s reported score. Preserve the distinction between molecular assay effects and clinical labels.
This is a task guide, not a single versioned benchmark protocol. The connected resources provide examples or concrete procedures. A candidate method or proposed control is not evidence that an evaluation has been completed.
Benchmarks bring together tasks and protocols. A task describes the biological question; a protocol defines a particular test.
These source-backed links do not make different protocols or scores interchangeable.
Release 2026-09-17-d277315f7d76 · 0 evaluations · 0 metric rows. Different protocols are not a single leaderboard.
No evaluations linked in this release.
Last literature check: 2026-09-17. Dated primary-source discovery and protocol/table screening. Source checking does not mean experimental reproduction. Only separately extracted and independently reviewed numeric batches are publishable.
| Paper or primary resource | Version | Reference |
|---|---|---|
| ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design | Primary full-text snapshot retrieved 2026-09-17; exact bytes pinned by SHA-256 | Read source |
primary protocol reviewed
Reviewed named primary documentation and existing protocol records. Editorial task scope is distinguished from source-reported procedures. No model execution or new numerical result; a reviewed guide is not a complete runnable protocol.
Stable record: catalog-task-proteingym-effectsTrace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.
One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.
17 evidence rows matching the loaded filters
| Property and statement | Original source and location | Review and provenance |
|---|---|---|
| Diagram caption Conceptual task guide. Dataset preparation, parameters and scoring must come from a separately identified protocol. Individual claims | OATML-Markslab/ProteinGym official source README: Overview, Results, benchmark baselines and contribution notes on aggregation Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c | source checked automated source review · 2026-09-16 Audit detailsReviewed named primary documentation and existing protocol records. Editorial task scope is distinguished from source-reported procedures. No model execution or new numerical result; a reviewed guide is not a complete runnable protocol. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Diagram steps ["Pin release, track and assays","Score the eligible variants","Compute per-assay metrics","Apply the declared aggregation"] Individual claims | OATML-Markslab/ProteinGym official source README: Overview, Results, benchmark baselines and contribution notes on aggregation Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c | source checked automated source review · 2026-09-16 Audit detailsReviewed named primary documentation and existing protocol records. Editorial task scope is distinguished from source-reported procedures. No model execution or new numerical result; a reviewed guide is not a complete runnable protocol. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Diagram title Conceptual evaluation workflow Individual claims | OATML-Markslab/ProteinGym official source README: Overview, Results, benchmark baselines and contribution notes on aggregation Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c | source checked automated source review · 2026-09-16 Audit detailsReviewed named primary documentation and existing protocol records. Editorial task scope is distinguished from source-reported procedures. No model execution or new numerical result; a reviewed guide is not a complete runnable protocol. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Entity type Task guide; concrete protocol identities remain separate. Individual claims | OATML-Markslab/ProteinGym official source README: Overview, Results, benchmark baselines and contribution notes on aggregation Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c | source checked automated source review · 2026-09-16 Audit detailsReviewed named primary documentation and existing protocol records. Editorial task scope is distinguished from source-reported procedures. No model execution or new numerical result; a reviewed guide is not a complete runnable protocol. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Datasets Select a particular ProteinGym release, assay subset and evaluation track; the suite contains separate substitution and indel resources. Individual claims | OATML-Markslab/ProteinGym official source README: Overview, Results, benchmark baselines and contribution notes on aggregation Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c | source checked automated source review · 2026-09-16 Audit detailsReviewed named primary documentation and existing protocol records. Editorial task scope is distinguished from source-reported procedures. No model execution or new numerical result; a reviewed guide is not a complete runnable protocol. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Organisms No shared organism population is defined at this guide level. Record it for each selected dataset. Individual claims | OATML-Markslab/ProteinGym official source README: Overview, Results, benchmark baselines and contribution notes on aggregation Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c | inapplicable automated source review · 2026-09-16 Audit detailsReviewed named primary documentation and existing protocol records. Editorial task scope is distinguished from source-reported procedures. No model execution or new numerical result; a reviewed guide is not a complete runnable protocol. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Assays Deep mutational scanning measurements for molecular effects; assay metadata and functional categories remain distinct. Individual claims | OATML-Markslab/ProteinGym official source README: Overview, Results, benchmark baselines and contribution notes on aggregation Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c | source checked automated source review · 2026-09-16 Audit detailsReviewed named primary documentation and existing protocol records. Editorial task scope is distinguished from source-reported procedures. No model execution or new numerical result; a reviewed guide is not a complete runnable protocol. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Splits No executable split is attached to this task identity. Use the selected protocol’s split manifest. Individual claims | OATML-Markslab/ProteinGym official source README: Overview, Results, benchmark baselines and contribution notes on aggregation Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c | inapplicable automated source review · 2026-09-16 Audit detailsReviewed named primary documentation and existing protocol records. Editorial task scope is distinguished from source-reported procedures. No model execution or new numerical result; a reviewed guide is not a complete runnable protocol. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Allowed inputs Protein variants, permitted sequence or structure information, and the selected assay’s reference measurements. Individual claims | OATML-Markslab/ProteinGym official source README: Overview, Results, benchmark baselines and contribution notes on aggregation Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c | source checked automated source review · 2026-09-16 Audit detailsReviewed named primary documentation and existing protocol records. Editorial task scope is distinguished from source-reported procedures. No model execution or new numerical result; a reviewed guide is not a complete runnable protocol. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Adaptation No common fitting regime is imposed here. Keep pretrained, frozen, probed, fine-tuned and conventional methods distinct where applicable. Individual claims | OATML-Markslab/ProteinGym official source README: Overview, Results, benchmark baselines and contribution notes on aggregation Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c | inapplicable automated source review · 2026-09-16 Audit detailsReviewed named primary documentation and existing protocol records. Editorial task scope is distinguished from source-reported procedures. No model execution or new numerical result; a reviewed guide is not a complete runnable protocol. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
Release 2026-09-17-d277315f7d76 · Record review: discovered
Stable ID: catalog-task-proteingym-effects