Task

Zero-shot variant effect prediction

Zero-shot variant-effect assessment compares structural protein representations with their sequence-model backbones.

Sources

Endowing protein language models with structural knowledge · Results: zero-shot variant prediction; implementation; cached text lines 65, 84

2 evaluations · 2 metric rows

At a glance

Inputs, training, access and other details

Explanatory profile: limited source coverage · Automated source review, 2026-09-16. Review applies to the cited claims; unresolved fields are listed below. Numerical results retain their own review status.

Data, procedure and scoring
Property	Description and evidence
Datasets	Variant-effect measurements for 38 proteins from the Riesselman/Meier DMS collections, assessed without assay-specific fine-tuning. The supplementary task section points to the original assay inventory. Sources (2) Endowing protein language models with structural knowledge; pst journal Supplementary Information — pinned PDF · Main §4.1.3; Supplementary Appendix C.6
Splits	No additional task fine-tuning for the reported zero-shot comparison. Sources Endowing protein language models with structural knowledge · Results: zero-shot variant prediction; implementation; cached text lines 65, 84
Metrics	Average Spearman rank correlation. Sources Endowing protein language models with structural knowledge · Results: zero-shot variant prediction; implementation; cached text lines 65, 84
Baselines	PST and ESM-2. Sources Endowing protein language models with structural knowledge · Results: zero-shot variant prediction; implementation; cached text lines 65, 84
Leakage controls	Variant-effect scoring uses no assay-label fine-tuning. The main variant-effect section and Supplementary Appendix C.6 do not report exclusion of the evaluated proteins or homologues from ESM-2 or structure pretraining; the identity/family exclusions in Appendix C.5 apply to other downstream tasks. · Not reported in inspected sources Sources (2) Endowing protein language models with structural knowledge; pst journal Supplementary Information — pinned PDF · Main §4.1.3–4.1.4; Supplementary Appendix C.5–C.6, pp11–12
Uncertainty	The cited text-accessible evaluation sections give no confidence-interval, resampling or repeat-run error-bar specification. Image-only tables and uninspected supplements are outside this absence claim. · Not reported in inspected sources Sources Endowing protein language models with structural knowledge · Results: zero-shot variant prediction; implementation; cached text lines 65, 84
Entity type	Paper-specific computational evaluation protocol. Sources Endowing protein language models with structural knowledge · Results: zero-shot variant prediction; implementation; cached text lines 65, 84
Organisms	The variant-effect benchmark comprises 38 proteins from the Riesselman/Meier DMS collections. The main task section and supplementary variant-effect section do not enumerate their taxa; Supplementary C.6 instead points to the original DMS assay inventory. · Not reported in inspected sources Sources (2) Endowing protein language models with structural knowledge; pst journal Supplementary Information — pinned PDF · Main §4.1.3; Supplementary Appendix C.6, pp11–12
Assays	Protein variant-effect measurements. Sources Endowing protein language models with structural knowledge · Results: zero-shot variant prediction; implementation; cached text lines 65, 84
Allowed inputs	Protein representations under PST and ESM-2 scoring. Sources Endowing protein language models with structural knowledge · Results: zero-shot variant prediction; implementation; cached text lines 65, 84
Adaptation	No additional task fine-tuning in the reported zero-shot comparison. Sources Endowing protein language models with structural knowledge · Results: zero-shot variant prediction; implementation; cached text lines 65, 84

How it works

How it worksComputational evaluation flow

Conceptual summary of the cited evaluation; exact task configuration and source version remain part of the protocol.

Sources

Endowing protein language models with structural knowledge · Results: zero-shot variant prediction; implementation; cached text lines 65, 84

Evaluation methodology

Variant-effect measurements for 38 proteins from the Riesselman/Meier DMS collections, assessed without assay-specific fine-tuning. The supplementary task section points to the original assay inventory. Average Spearman rank correlation. PST and ESM-2.

Sources (2)

Endowing protein language models with structural knowledge; pst journal Supplementary Information — pinned PDF · Main §4.1.3; Supplementary Appendix C.6; Results: zero-shot variant prediction; implementation; cached text lines 65, 84; Results: zero-shot variant prediction; implementation; cached text lines 65, 84

Recorded evaluations

Each evaluation records what was tested and under which conditions.

Tested entities and results

Release 2026-09-17-d277315f7d76 · 2 evaluations · 2 metric rows. Different protocols are not a single leaderboard.

Evaluation setupMetricEvidence origin

Results grouped by the exact reported evaluation
Metric and finding	Coverage and uncertainty	Evidence
PST: Zero-shot variant effect prediction Configuration: PSTTask: Zero-shot variant effect predictionDataset: ProteinShake VEP datasets Zero-shot VEP; paper averages absolute Spearman correlations. Author-reported evaluation · Evaluation metadata: needs review
0.501 Mean \|Spearman rho\| Unit: unitless · Direction: unknown	Uncertainty: not reported in legacy extract Scored: Not reported · Eligible: Not reported	source checkedEndowing protein language models with structural knowledge · Table 2, PST row, Zero-shot VEP Mean \|ρ\| column Source checking is not independent reproduction.
ESM-2: Zero-shot variant effect prediction Configuration: ESM-2Task: Zero-shot variant effect predictionDataset: ProteinShake VEP datasets Zero-shot VEP; paper averages absolute Spearman correlations. Independent external evaluation · Evaluation metadata: needs review
0.489 Mean \|Spearman rho\| Unit: unitless · Direction: unknown	Uncertainty: not reported in legacy extract Scored: Not reported · Eligible: Not reported	source checkedEndowing protein language models with structural knowledge · Table 2, ESM-2 row, Zero-shot VEP Mean \|ρ\| column Source checking is not independent reproduction.

Papers and result coverage

Last literature check: 2026-09-17. Primary-source discovery and table/protocol screening; source checked is not independently reproduced. Raw acquisitions not automatically numerical publication approval.

Paper or primary resource	Version	Reference
Endowing protein language models with structural knowledge	PMC12603367.1	Read source DOI: 10.1093/bioinformatics/btaf582

What is still missing

Table contains two zero-shot mean absolute Spearman values already in catalogue; no new independent observations warranted. Other Table 2 tasks are different benchmarks. Full raw table and source receipt retained.

Search and extraction details

existing observations cover named comparison

Searches

Endowing protein language models with structural knowledge primary paper benchmark results

Evidence locations

Table2 zero-shotVEP lastcolumn; structural augmentation evaluations

Strengths and limitations

Strengths and considerations

No source-reviewed explanatory claims are recorded here yet.

Limitations and conditions

Structure-pretraining sources and downstream assay overlap require a separate audit. Structure-based splits described for ProteinShake are not assumed to define these variant-effect tasks.
Sources
Endowing protein language models with structural knowledge · Results: zero-shot variant prediction; implementation; cached text lines 65, 84

What remains unknown

Leakage controls: Variant-effect scoring uses no assay-label fine-tuning. The main variant-effect section and Supplementary Appendix C.6 do not report exclusion of the evaluated proteins or homologues from ESM-2 or structure pretraining; the identity/family exclusions in Appendix C.5 apply to other downstream tasks.
Uncertainty: The cited text-accessible evaluation sections give no confidence-interval, resampling or repeat-run error-bar specification. Image-only tables and uninspected supplements are outside this absence claim.
Organisms: The variant-effect benchmark comprises 38 proteins from the Riesselman/Meier DMS collections. The main task section and supplementary variant-effect section do not enumerate their taxa; Supplementary C.6 instead points to the original DMS assay inventory.

Profile review details

Relevant full-paper computational evaluation sections, tables/captions and cited supplementary task passages were reviewed. Reporting omissions are scoped to the inspected sources. Original numerical results are unchanged.

Stable record: reported-task-a5141363b0ee45

Evidence table

Inspect claims, sources and review details

Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.

One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.

21 evidence rows matching the loaded filters

Claims, original sources and review scope · Release 2026-09-17-d277315f7d76
Property and statement	Original source and location	Review and provenance
Diagram caption Conceptual summary of the cited evaluation; exact task configuration and source version remain part of the protocol. Individual claims	Endowing protein language models with structural knowledge Original source ↗ Results: zero-shot variant prediction; implementation; cached text lines 65, 84 Version: PMC12603367.1 Retrieved: 2026-09-16T10:41:16.527633+00:00	source checked automated source review · 2026-09-16 Audit details Relevant full-paper computational evaluation sections, tables/captions and cited supplementary task passages were reviewed. Reporting omissions are scoped to the inspected sources. Original numerical results are unchanged. Field: `attributes.profile.diagram.caption` Source artifact SHA-256: `c21ad593de589a7188ca86a8b7ce617e301d48dd939ecc7da03efb22cbe8d7a3` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Diagram steps ["Input: Protein representations under PST and ESM-2 scoring.","Evaluation: No additional task fine-tuning in the reported zero-shot comparison.","Readout: Average Spearman rank correlation."] Individual claims	Endowing protein language models with structural knowledge Original source ↗ Results: zero-shot variant prediction; implementation; cached text lines 65, 84 Version: PMC12603367.1 Retrieved: 2026-09-16T10:41:16.527633+00:00	source checked automated source review · 2026-09-16 Audit details Relevant full-paper computational evaluation sections, tables/captions and cited supplementary task passages were reviewed. Reporting omissions are scoped to the inspected sources. Original numerical results are unchanged. Field: `attributes.profile.diagram.steps` Source artifact SHA-256: `c21ad593de589a7188ca86a8b7ce617e301d48dd939ecc7da03efb22cbe8d7a3` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Diagram title Computational evaluation flow Individual claims	Endowing protein language models with structural knowledge Original source ↗ Results: zero-shot variant prediction; implementation; cached text lines 65, 84 Version: PMC12603367.1 Retrieved: 2026-09-16T10:41:16.527633+00:00	source checked automated source review · 2026-09-16 Audit details Relevant full-paper computational evaluation sections, tables/captions and cited supplementary task passages were reviewed. Reporting omissions are scoped to the inspected sources. Original numerical results are unchanged. Field: `attributes.profile.diagram.title` Source artifact SHA-256: `c21ad593de589a7188ca86a8b7ce617e301d48dd939ecc7da03efb22cbe8d7a3` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Datasets Variant-effect measurements for 38 proteins from the Riesselman/Meier DMS collections, assessed without assay-specific fine-tuning. The supplementary task section points to the original assay inventory. Individual claims	pst journal Supplementary Information — pinned PDF Original source ↗ Main §4.1.3; Supplementary Appendix C.6 Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: Published supplementary PDF btaf582_supplementary_data.pdf; sha256:f8c3375644f10aeb50c76f99071ec64b2b0dceadcee7f70a809adbcc8985d936 Retrieved: 2026-09-16T21:06:10.146345+00:00	source checked automated source review · 2026-09-16 Audit details Relevant full-paper computational evaluation sections, tables/captions and cited supplementary task passages were reviewed. Reporting omissions are scoped to the inspected sources. Original numerical results are unchanged. Field: `attributes.profile.facts.0.value` Source artifact SHA-256: `f8c3375644f10aeb50c76f99071ec64b2b0dceadcee7f70a809adbcc8985d936` Hash scope: Hash scope not separately documented; inspect source record Archive member: btaf582_supplementary_data.pdf Inspected artifact
Datasets Variant-effect measurements for 38 proteins from the Riesselman/Meier DMS collections, assessed without assay-specific fine-tuning. The supplementary task section points to the original assay inventory. Individual claims	Endowing protein language models with structural knowledge Original source ↗ Main §4.1.3; Supplementary Appendix C.6 Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: PMC12603367.1 Retrieved: 2026-09-16T10:41:16.527633+00:00	source checked automated source review · 2026-09-16 Audit details Relevant full-paper computational evaluation sections, tables/captions and cited supplementary task passages were reviewed. Reporting omissions are scoped to the inspected sources. Original numerical results are unchanged. Field: `attributes.profile.facts.0.value` Source artifact SHA-256: `c21ad593de589a7188ca86a8b7ce617e301d48dd939ecc7da03efb22cbe8d7a3` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Splits No additional task fine-tuning for the reported zero-shot comparison. Individual claims	Endowing protein language models with structural knowledge Original source ↗ Results: zero-shot variant prediction; implementation; cached text lines 65, 84 Version: PMC12603367.1 Retrieved: 2026-09-16T10:41:16.527633+00:00	source checked automated source review · 2026-09-16 Audit details Relevant full-paper computational evaluation sections, tables/captions and cited supplementary task passages were reviewed. Reporting omissions are scoped to the inspected sources. Original numerical results are unchanged. Field: `attributes.profile.facts.1.value` Source artifact SHA-256: `c21ad593de589a7188ca86a8b7ce617e301d48dd939ecc7da03efb22cbe8d7a3` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Adaptation No additional task fine-tuning in the reported zero-shot comparison. Individual claims	Endowing protein language models with structural knowledge Original source ↗ Results: zero-shot variant prediction; implementation; cached text lines 65, 84 Version: PMC12603367.1 Retrieved: 2026-09-16T10:41:16.527633+00:00	source checked automated source review · 2026-09-16 Audit details Relevant full-paper computational evaluation sections, tables/captions and cited supplementary task passages were reviewed. Reporting omissions are scoped to the inspected sources. Original numerical results are unchanged. Field: `attributes.profile.facts.10.value` Source artifact SHA-256: `c21ad593de589a7188ca86a8b7ce617e301d48dd939ecc7da03efb22cbe8d7a3` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Metrics Average Spearman rank correlation. Individual claims	Endowing protein language models with structural knowledge Original source ↗ Results: zero-shot variant prediction; implementation; cached text lines 65, 84 Version: PMC12603367.1 Retrieved: 2026-09-16T10:41:16.527633+00:00	source checked automated source review · 2026-09-16 Audit details Relevant full-paper computational evaluation sections, tables/captions and cited supplementary task passages were reviewed. Reporting omissions are scoped to the inspected sources. Original numerical results are unchanged. Field: `attributes.profile.facts.2.value` Source artifact SHA-256: `c21ad593de589a7188ca86a8b7ce617e301d48dd939ecc7da03efb22cbe8d7a3` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Baselines PST and ESM-2. Individual claims	Endowing protein language models with structural knowledge Original source ↗ Results: zero-shot variant prediction; implementation; cached text lines 65, 84 Version: PMC12603367.1 Retrieved: 2026-09-16T10:41:16.527633+00:00	source checked automated source review · 2026-09-16 Audit details Relevant full-paper computational evaluation sections, tables/captions and cited supplementary task passages were reviewed. Reporting omissions are scoped to the inspected sources. Original numerical results are unchanged. Field: `attributes.profile.facts.3.value` Source artifact SHA-256: `c21ad593de589a7188ca86a8b7ce617e301d48dd939ecc7da03efb22cbe8d7a3` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Leakage controls Variant-effect scoring uses no assay-label fine-tuning. The main variant-effect section and Supplementary Appendix C.6 do not report exclusion of the evaluated proteins or homologues from ESM-2 or structure pretraining; the identity/family exclusions in Appendix C.5 apply to other downstream tasks. Individual claims	pst journal Supplementary Information — pinned PDF Original source ↗ Main §4.1.3–4.1.4; Supplementary Appendix C.5–C.6, pp11–12 Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: Published supplementary PDF btaf582_supplementary_data.pdf; sha256:f8c3375644f10aeb50c76f99071ec64b2b0dceadcee7f70a809adbcc8985d936 Retrieved: 2026-09-16T21:06:10.146345+00:00	unreported automated source review · 2026-09-16 Audit details Relevant full-paper computational evaluation sections, tables/captions and cited supplementary task passages were reviewed. Reporting omissions are scoped to the inspected sources. Original numerical results are unchanged. Field: `attributes.profile.facts.4.value` Source artifact SHA-256: `f8c3375644f10aeb50c76f99071ec64b2b0dceadcee7f70a809adbcc8985d936` Hash scope: Hash scope not separately documented; inspect source record Archive member: btaf582_supplementary_data.pdf Inspected artifact

All evidence (CSV)All evidence (JSONL)How to read the evidence

Sources and history

Release 2026-09-17-d277315f7d76 · Record review: needs review

3 source records and release history

Endowing protein language models with structural knowledge · Original source · PMC12603367.1
Endowing protein language models with structural knowledge · Original source · PMC12603367.1
pst journal Supplementary Information — pinned PDF · Original source · Published supplementary PDF btaf582_supplementary_data.pdf; sha256:f8c3375644f10aeb50c76f99071ec64b2b0dceadcee7f70a809adbcc8985d936

Download this release

Technical metadata and extraction receipts

Stable ID: reported-task-a5141363b0ee45

areas: proteins-complexes
tasks: Zero-shot variant effect prediction
entity level: task
version: Not reported
task: Zero-shot variant effect prediction
scope note: Paper-specific evaluation task; protocol completeness requires further extraction.
benchmark research: review date: 2026-09-17; status: existing_observations_cover_named_comparison; primary sources: evidence-expansion-p2-pst-2025-c21ad593de58; inspected locators: Table2 zero-shotVEP lastcolumn; structural augmentation evaluations; searched queries: Endowing protein language models with structural knowledge primary paper benchmark results; gaps: Table contains two zero-shot mean absolute Spearman values already in catalogue; no new independent observations warranted. Other Table 2 tasks are different benchmarks. Full raw table and source receipt retained.; claim scope: Primary-source discovery and table/protocol screening; source checked is not independently reproduced. Raw acquisitions not automatically numerical publication approval.
historical missing metadata: protocol version: not_reported_in_legacy_extract; split: not_reported_in_legacy_extract
metadata review scope: historical_missing_metadata preserves the original discovery state. Current descriptive evidence and missingness are recorded in profile.facts; numerical-result review is separate.
legacy kinds: benchmark
entity classification: review date: 2026-09-17; rationale: This source-scoped record identifies the biological prediction task and holds its paper context. Preserve the existing task identity; exact split, model adaptation and scoring remain in linked evaluations or separate protocol records.; source ids: pst-2025; source locator: Results: zero-shot variant prediction; implementation; cached text lines 65, 84; ambiguities: A paper- or suite-specific task may constrain some inputs or metrics; that alone does not make it interchangeable with a complete versioned protocol. No protocol equivalence is inferred.; Some legacy profile Entity type facts use the generic phrase computational evaluation protocol. That boilerplate is not sufficient to establish a single fixed protocol identity or to merge this task with another protocol record.

Related records

dataset: ProteinShake VEP datasets
benchmark: PST: Zero-shot variant effect prediction
benchmark: ESM-2: Zero-shot variant effect prediction

Suggest a correction