Evaluator

OPAL

OPAL evaluates taxon presence and abundance profiles against a reference community.

Sources

CAMI-challenge/OPAL official source · Pinned README: introduction; Computed metrics; input format; example evaluations

At a glance

Inputs, training, access and other details

Explanatory profile: source reviewed · Automated source review, 2026-09-16. Review applies to the cited claims; unresolved fields are listed below. Numerical results retain their own review status.

Data, procedure and scoring
Property	Description and evidence
Datasets	Predicted taxonomic abundance profiles and a gold-standard profile in supported formats. Sources CAMI-challenge/OPAL official source · Pinned README: introduction; Computed metrics; input format; example evaluations
Splits	The evaluator scores supplied outputs; predictor train/test partitions belong to the dataset/run being evaluated. · Not applicable Sources CAMI-challenge/OPAL official source · Pinned README: introduction; Computed metrics; input format; example evaluations
Metrics	Precision, recall, F1, Jaccard, L1 error, UniFrac, Bray–Curtis and diversity measures. Sources CAMI-challenge/OPAL official source · Pinned README: introduction; Computed metrics; input format; example evaluations
Baselines	Multiple profiling tools can be compared; published example reports use CAMI and mock-community data. Sources CAMI-challenge/OPAL official source · Pinned README: introduction; Computed metrics; input format; example evaluations
Leakage controls	OPAL scores taxonomic profiles against a supplied gold standard; it does not define or audit predictor training data. Reference-database cutoffs and novelty controls must accompany the evaluated challenge. · Not applicable Sources opal primary benchmark evidence · Implementation: input data and evaluation metrics
Uncertainty	Uncertainty across samples, datasets or training runs must be defined by the evaluation study; this evaluator entry does not fix one experiment. · Not applicable Sources CAMI-challenge/OPAL official source · Pinned README: introduction; Computed metrics; input format; example evaluations
Entity type	Evaluator for taxonomic abundance profiling. Sources CAMI-challenge/OPAL official source · Pinned README: introduction; Computed metrics; input format; example evaluations
Organisms	The evaluator is not restricted to a single organism; profiles describe microbial communities. · Not applicable Sources CAMI-challenge/OPAL official source · Pinned README: introduction; Computed metrics; input format; example evaluations
Assays	Taxonomic abundance profiles derived from metagenomic analyses. Sources CAMI-challenge/OPAL official source · Pinned README: introduction; Computed metrics; input format; example evaluations
Allowed inputs	Predicted and gold-standard taxon abundances in supported formats. Sources CAMI-challenge/OPAL official source · Pinned README: introduction; Computed metrics; input format; example evaluations
Adaptation	OPAL evaluates profiles and does not define model adaptation. · Not applicable Sources CAMI-challenge/OPAL official source · Pinned README: introduction; Computed metrics; input format; example evaluations
Implementation	Not extracted or verified for this record.

How it works

How it worksEvaluation procedure

Conceptual procedure. Task variants and protocol versions retain their separate scoring conditions.

Sources

CAMI-challenge/OPAL official source · Pinned README: introduction; Computed metrics; input format; example evaluations

Evaluation methodology

Predicted taxonomic abundance profiles and a gold-standard profile in supported formats. The evaluator scores supplied outputs; predictor train/test partitions belong to the dataset/run being evaluated. Precision, recall, F1, Jaccard, L1 error, UniFrac, Bray–Curtis and diversity measures. Multiple profiling tools can be compared; published example reports use CAMI and mock-community data. Uncertainty across samples, datasets or training runs must be defined by the evaluation study; this evaluator entry does not fix one experiment.

Sources

CAMI-challenge/OPAL official source · Pinned README: introduction; Computed metrics; input format; example evaluations

Tested entities and results

Release 2026-09-17-d277315f7d76 · 0 evaluations · 0 metric rows. Different protocols are not a single leaderboard.

No evaluations linked in this release.

Papers and result coverage

Last literature check: 2026-09-17. Primary-source discovery and table/protocol screening; source checked is not independently reproduced. Raw acquisitions not automatically numerical publication approval.

Paper or primary resource	Version	Reference
Assessing taxonomic metagenome profilers with OPAL	PMC6398228	Read source DOI: 10.1186/s13059-019-1646-y

What is still missing

OPAL is an evaluator rather than a single dataset. Numerical comparisons require selected community dataset, taxonomic rank and profiling output version. No complete main-text score matrix; source data remains pending.

Search and extraction details

source found structured extraction pending

Searches

OPAL primary paper benchmark results

Evidence locations

Methods and benchmark Results; figure captions and source availability

Strengths and limitations

Strengths supported by sources

Separates abundance agreement from presence/absence detection.
Sources
CAMI-challenge/OPAL official source · Pinned README: introduction; Computed metrics; input format; example evaluations

Limitations and conditions

Taxonomic profiles must use compatible taxonomy and abundance conventions. A low profiling error does not establish independence from the reference genomes.
Sources
opal primary benchmark evidence · Implementation: input data and evaluation metrics

Profile review details

Primary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction.

Stable record: discovery-benchmark-opal

Evidence table

Inspect claims, sources and review details

Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.

One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.

18 evidence rows matching the loaded filters

Claims, original sources and review scope · Release 2026-09-17-d277315f7d76
Property and statement	Original source and location	Review and provenance
Diagram caption Conceptual procedure. Task variants and protocol versions retain their separate scoring conditions. Individual claims	CAMI-challenge/OPAL official source Original source ↗ Pinned README: introduction; Computed metrics; input format; example evaluations Version: 98120c326eef08e391899e4bd3a362e0e6558b4a Retrieved: 2026-09-16T10:30:23.429372+00:00	source checked automated source review · 2026-09-16 Audit details Primary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Field: `attributes.profile.diagram.caption` Source artifact SHA-256: `d123b202234d3ed520911982dcf88e2fcfd888c83e65378effb2d2e269d0abb1` Hash scope: Hash scope not separately documented; inspect source record
Diagram steps ["Allowed inputs: Predicted and gold-standard taxon abundances in supported formats.","Splits: The evaluator scores supplied outputs; predictor train/test partitions belong to the dataset/run being evaluated.","Metrics: Precision, recall, F1, Jaccard, L1 error, UniFrac, Bray–Curtis and diversity measures."] Individual claims	CAMI-challenge/OPAL official source Original source ↗ Pinned README: introduction; Computed metrics; input format; example evaluations Version: 98120c326eef08e391899e4bd3a362e0e6558b4a Retrieved: 2026-09-16T10:30:23.429372+00:00	source checked automated source review · 2026-09-16 Audit details Primary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Field: `attributes.profile.diagram.steps` Source artifact SHA-256: `d123b202234d3ed520911982dcf88e2fcfd888c83e65378effb2d2e269d0abb1` Hash scope: Hash scope not separately documented; inspect source record
Diagram title Evaluation procedure Individual claims	CAMI-challenge/OPAL official source Original source ↗ Pinned README: introduction; Computed metrics; input format; example evaluations Version: 98120c326eef08e391899e4bd3a362e0e6558b4a Retrieved: 2026-09-16T10:30:23.429372+00:00	source checked automated source review · 2026-09-16 Audit details Primary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Field: `attributes.profile.diagram.title` Source artifact SHA-256: `d123b202234d3ed520911982dcf88e2fcfd888c83e65378effb2d2e269d0abb1` Hash scope: Hash scope not separately documented; inspect source record
Datasets Predicted taxonomic abundance profiles and a gold-standard profile in supported formats. Individual claims	CAMI-challenge/OPAL official source Original source ↗ Pinned README: introduction; Computed metrics; input format; example evaluations Version: 98120c326eef08e391899e4bd3a362e0e6558b4a Retrieved: 2026-09-16T10:30:23.429372+00:00	source checked automated source review · 2026-09-16 Audit details Primary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Field: `attributes.profile.facts.0.value` Source artifact SHA-256: `d123b202234d3ed520911982dcf88e2fcfd888c83e65378effb2d2e269d0abb1` Hash scope: Hash scope not separately documented; inspect source record
Splits The evaluator scores supplied outputs; predictor train/test partitions belong to the dataset/run being evaluated. Individual claims	CAMI-challenge/OPAL official source Original source ↗ Pinned README: introduction; Computed metrics; input format; example evaluations Version: 98120c326eef08e391899e4bd3a362e0e6558b4a Retrieved: 2026-09-16T10:30:23.429372+00:00	inapplicable automated source review · 2026-09-16 Audit details Primary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Field: `attributes.profile.facts.1.value` Source artifact SHA-256: `d123b202234d3ed520911982dcf88e2fcfd888c83e65378effb2d2e269d0abb1` Hash scope: Hash scope not separately documented; inspect source record
Adaptation OPAL evaluates profiles and does not define model adaptation. Individual claims	CAMI-challenge/OPAL official source Original source ↗ Pinned README: introduction; Computed metrics; input format; example evaluations Version: 98120c326eef08e391899e4bd3a362e0e6558b4a Retrieved: 2026-09-16T10:30:23.429372+00:00	inapplicable automated source review · 2026-09-16 Audit details Primary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Field: `attributes.profile.facts.10.value` Source artifact SHA-256: `d123b202234d3ed520911982dcf88e2fcfd888c83e65378effb2d2e269d0abb1` Hash scope: Hash scope not separately documented; inspect source record
Metrics Precision, recall, F1, Jaccard, L1 error, UniFrac, Bray–Curtis and diversity measures. Individual claims	CAMI-challenge/OPAL official source Original source ↗ Pinned README: introduction; Computed metrics; input format; example evaluations Version: 98120c326eef08e391899e4bd3a362e0e6558b4a Retrieved: 2026-09-16T10:30:23.429372+00:00	source checked automated source review · 2026-09-16 Audit details Primary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Field: `attributes.profile.facts.2.value` Source artifact SHA-256: `d123b202234d3ed520911982dcf88e2fcfd888c83e65378effb2d2e269d0abb1` Hash scope: Hash scope not separately documented; inspect source record
Baselines Multiple profiling tools can be compared; published example reports use CAMI and mock-community data. Individual claims	CAMI-challenge/OPAL official source Original source ↗ Pinned README: introduction; Computed metrics; input format; example evaluations Version: 98120c326eef08e391899e4bd3a362e0e6558b4a Retrieved: 2026-09-16T10:30:23.429372+00:00	source checked automated source review · 2026-09-16 Audit details Primary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Field: `attributes.profile.facts.3.value` Source artifact SHA-256: `d123b202234d3ed520911982dcf88e2fcfd888c83e65378effb2d2e269d0abb1` Hash scope: Hash scope not separately documented; inspect source record
Leakage controls OPAL scores taxonomic profiles against a supplied gold standard; it does not define or audit predictor training data. Reference-database cutoffs and novelty controls must accompany the evaluated challenge. Individual claims	opal primary benchmark evidence Original source ↗ Implementation: input data and evaluation metrics Version: PMC6398228 Retrieved: 2026-09-16T21:04:56.586502+00:00	inapplicable automated source review · 2026-09-16 Audit details Primary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Field: `attributes.profile.facts.4.value` Source artifact SHA-256: `a4503ef32e99127d55d0af8ebf0c39baa298b320e4032fcacdb9aa5d33821a09` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Uncertainty Uncertainty across samples, datasets or training runs must be defined by the evaluation study; this evaluator entry does not fix one experiment. Individual claims	CAMI-challenge/OPAL official source Original source ↗ Pinned README: introduction; Computed metrics; input format; example evaluations Version: 98120c326eef08e391899e4bd3a362e0e6558b4a Retrieved: 2026-09-16T10:30:23.429372+00:00	inapplicable automated source review · 2026-09-16 Audit details Primary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Field: `attributes.profile.facts.5.value` Source artifact SHA-256: `d123b202234d3ed520911982dcf88e2fcfd888c83e65378effb2d2e269d0abb1` Hash scope: Hash scope not separately documented; inspect source record

All evidence (CSV)All evidence (JSONL)How to read the evidence

Sources and history

Release 2026-09-17-d277315f7d76 · Record review: discovered

3 source records and release history

CAMI-challenge/OPAL official source · Original source · 98120c326eef08e391899e4bd3a362e0e6558b4a
Assessing taxonomic metagenome profilers with OPAL · Original source · PMC6398228
opal primary benchmark evidence · Original source · PMC6398228

Download this release

Technical metadata and extraction receipts

Stable ID: discovery-benchmark-opal

areas: microbiome
entity level: evaluator
scope note: Specialist molecular or omics evaluation; protocol details require review before numerical comparison.
task: Metagenomic taxonomic profile evaluation
version: Not reported
benchmark research: review date: 2026-09-17; status: source_found_structured_extraction_pending; primary sources: evidence-expansion-p2-evidence-discovery-final-opal-a4503ef32e99; inspected locators: Methods and benchmark Results; figure captions and source availability; searched queries: OPAL primary paper benchmark results; gaps: OPAL is an evaluator rather than a single dataset. Numerical comparisons require selected community dataset, taxonomic rank and profiling output version. No complete main-text score matrix; source data remains pending.; claim scope: Primary-source discovery and table/protocol screening; source checked is not independently reproduced. Raw acquisitions not automatically numerical publication approval.
historical missing metadata: dataset release: unextracted; metric implementation: unextracted; split manifest: unextracted; version: unextracted
metadata review scope: historical_missing_metadata preserves the original discovery state. Current descriptive evidence and missingness are recorded in profile.facts; numerical-result review is separate.
legacy kinds: benchmark
entity classification: review date: 2026-09-17; rationale: The cited profile describes scoring or assessment software/procedures applied to predictions, rather than the biological target or the underlying dataset.; source ids: src-discovery-cami-challenge-opal; source locator: Pinned README: introduction; Computed metrics; input format; example evaluations; ambiguities: None recorded

Related records

Suggest a correction