Strengths supported by sources
- Abundance agreement and taxon detection are separate evaluation dimensions.
Sources
CAMI-challenge/OPAL official source · Pinned README: introduction; Computed metrics; input format; example evaluations
Task
CAMI taxonomic profiling
The CAMI taxonomic-profiling task can be understood through its documented assessment tool; this guide does not identify a challenge-specific run.
Sources
CAMI-challenge/OPAL official source · Pinned README: introduction; Computed metrics; input format; example evaluationsAt a glance
Inputs, training, access and other details
Explanatory profile: source reviewed · Automated source review, 2026-09-16. Review applies to the cited claims; unresolved fields are listed below. Numerical results retain their own review status.
| Property | Description and evidence |
|---|---|
| Datasets | A selected CAMI challenge dataset and matching gold standard are required; this entry does not fix the edition.SourcesCAMI-challenge/OPAL official source · Pinned README: introduction; Computed metrics; input format; example evaluations |
| Splits | CAMI II supplied public-genome practice datasets with ground truth before its blinded challenge. Challenge datasets were marine, strain-madness and plant-associated communities; these are challenge conditions, not a standard supervised train/validation/test partition.Sourcescami2 primary benchmark evidence · Methods: Challenge datasets, Challenge organization, Evaluation metrics; Figures 2–4; Table 1 |
| Metrics | Precision, recall, F1, Jaccard, L1 error, UniFrac, Bray–Curtis and diversity measures.SourcesCAMI-challenge/OPAL official source · Pinned README: introduction; Computed metrics; input format; example evaluations |
| Baselines | Submitted programs are compared under the same data condition. Gold-standard assemblies and MEGAHIT assemblies separate binning performance from upstream assembly error; published method identities and versions are listed in Table 1.Sourcescami2 primary benchmark evidence · Methods: Challenge datasets, Challenge organization, Evaluation metrics; Figures 2–4; Table 1 |
| Leakage controls | Challenge genome data and metadata were kept confidential until the challenge ended. Public reference collections dated 8 January 2019 were supplied for reference-based methods. CAMI II also includes public genomes, so novelty is stratified rather than assumed for every organism.Sourcescami2 primary benchmark evidence · Methods: Challenge datasets, Challenge organization, Evaluation metrics; Figures 2–4; Table 1 |
| Uncertainty | Uncertainty is task-specific: taxonomic binning Figure 3 uses standard errors across bins; taxonomic profiling Figure 4 reports means across samples with standard deviations. These are not a common seed-based interval for every CAMI metric.Sourcescami2 primary benchmark evidence · Figure 3 and Figure 4 captions: standard error across taxonomic bins versus standard deviation across samples |
| Entity type | Constituent benchmark task: CAMI taxonomic profilingSourcesCAMI-challenge/OPAL official source · Pinned README: introduction; Computed metrics; input format; example evaluations |
| Organisms | Taxa are supplied by the selected reference community. · Not applicableSourcesCAMI-challenge/OPAL official source · Pinned README: introduction; Computed metrics; input format; example evaluations |
| Assays | Challenge-specific metagenomic sequence data and reference composition.SourcesCAMI-challenge/OPAL official source · Pinned README: introduction; Computed metrics; input format; example evaluations |
| Allowed inputs | Predicted and reference abundance profiles.SourcesCAMI-challenge/OPAL official source · Pinned README: introduction; Computed metrics; input format; example evaluations |
| Adaptation | OPAL scores profiles; it does not train the submitting method. · Not applicableSourcesCAMI-challenge/OPAL official source · Pinned README: introduction; Computed metrics; input format; example evaluations |
How it works
How it worksEvaluation procedure
Conceptual procedure. Task variants and protocol versions retain their separate scoring conditions.
Sources (2)
CAMI-challenge/OPAL official source; cami2 primary benchmark evidence · Pinned README: introduction; Computed metrics; input format; example evaluations; Methods: Challenge datasets, Challenge organization, Evaluation metrics; Figures 2–4; Table 1Evaluation methodology
CAMI is a series of blinded metagenomic software challenges. In CAMI II, participants received simulated short and long reads from defined communities and submitted assemblies, genome bins, taxonomic assignments or abundance profiles. Reference truth was used only for scoring, and software versions, input read types and community conditions were kept distinct.
Sources
cami2 primary benchmark evidence · Methods: Challenge datasets, Challenge organization, Evaluation metrics; Figures 2–4; Table 1Evaluation design
Benchmarks bring together tasks and protocols. A task describes the biological question; a protocol defines a particular test.
Benchmarks
These source-backed links do not make different protocols or scores interchangeable.
Tested entities and results
Release 2026-09-17-d277315f7d76 · 0 evaluations · 0 metric rows. Different protocols are not a single leaderboard.
No evaluations linked in this release.
Papers and result coverage
Last literature check: 2026-09-17. Primary-source discovery and table/protocol screening; source checked is not independently reproduced. Raw acquisitions not automatically numerical publication approval.
| Paper or primary resource | Version | Reference |
|---|---|---|
| Critical Assessment of Metagenome Interpretation: the second round of challenges | PMC9007738 | Read source DOI: 10.1038/s41592-022-01431-4 |
What is still missing
- Main Table 1 gives best-ranked software, not a complete numerical score table. Assembly, genome binning, taxonomic binning and taxonomic profiling are separate tasks; strain diversity and sample cohorts must be separated. Further source-data extraction remains before new score publication.
Search and extraction details
source found structured extraction pending
Searches
- CAMI II metagenome benchmarking 2022 supplementary results table
Evidence locations
- Table1 rankings; main figures and linked supplementary material
Strengths and limitations
Limitations and conditions
- This profile documents CAMI II as a concrete protocol example. Other CAMI rounds may use different genomes, reference databases and metrics; strain diversity and input assembly quality materially change the task.
Sources
cami2 primary benchmark evidence · Methods: Challenge datasets, Challenge organization, Evaluation metrics; Figures 2–4; Table 1
Profile review details
Primary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Independent automated spot review corrected interval terminology and sequence-identity scope against the original figure captions and methods.
Stable record: discovery-benchmark-cami-taxonomic-profilingApplicable tests and references
Applicability is distinct from a completed evaluation.
- Marker-based microbial profiling · proposed
Evidence table
Inspect claims, sources and review details
Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.
One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.
22 evidence rows matching the loaded filters
| Property and statement | Original source and location | Review and provenance |
|---|---|---|
| Diagram caption Conceptual procedure. Task variants and protocol versions retain their separate scoring conditions. Individual claims | cami2 primary benchmark evidence Pinned README: introduction; Computed metrics; input format; example evaluations; Methods: Challenge datasets, Challenge organization, Evaluation metrics; Figures 2–4; Table 1 Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: PMC9007738 | source checked automated source review · 2026-09-16 Audit detailsPrimary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Independent automated spot review corrected interval terminology and sequence-identity scope against the original figure captions and methods. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Diagram caption Conceptual procedure. Task variants and protocol versions retain their separate scoring conditions. Individual claims | CAMI-challenge/OPAL official source Pinned README: introduction; Computed metrics; input format; example evaluations; Methods: Challenge datasets, Challenge organization, Evaluation metrics; Figures 2–4; Table 1 Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: 98120c326eef08e391899e4bd3a362e0e6558b4a | source checked automated source review · 2026-09-16 Audit detailsPrimary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Independent automated spot review corrected interval terminology and sequence-identity scope against the original figure captions and methods. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Diagram steps ["Allowed inputs: Predicted and reference abundance profiles.","Splits: CAMI II supplied public-genome practice datasets with ground truth before its blinded challenge. Challenge datasets were marine, strain-madness and plant-associated communities; these are challenge conditions, not a standard supervised train/validation/test partition.","Metrics: Precision, recall, F1, Jaccard, L1 error, UniFrac, Bray–Curtis and diversity measures."] Individual claims | cami2 primary benchmark evidence Pinned README: introduction; Computed metrics; input format; example evaluations; Methods: Challenge datasets, Challenge organization, Evaluation metrics; Figures 2–4; Table 1 Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: PMC9007738 | source checked automated source review · 2026-09-16 Audit detailsPrimary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Independent automated spot review corrected interval terminology and sequence-identity scope against the original figure captions and methods. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Diagram steps ["Allowed inputs: Predicted and reference abundance profiles.","Splits: CAMI II supplied public-genome practice datasets with ground truth before its blinded challenge. Challenge datasets were marine, strain-madness and plant-associated communities; these are challenge conditions, not a standard supervised train/validation/test partition.","Metrics: Precision, recall, F1, Jaccard, L1 error, UniFrac, Bray–Curtis and diversity measures."] Individual claims | CAMI-challenge/OPAL official source Pinned README: introduction; Computed metrics; input format; example evaluations; Methods: Challenge datasets, Challenge organization, Evaluation metrics; Figures 2–4; Table 1 Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: 98120c326eef08e391899e4bd3a362e0e6558b4a | source checked automated source review · 2026-09-16 Audit detailsPrimary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Independent automated spot review corrected interval terminology and sequence-identity scope against the original figure captions and methods. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Diagram title Evaluation procedure Individual claims | cami2 primary benchmark evidence Pinned README: introduction; Computed metrics; input format; example evaluations; Methods: Challenge datasets, Challenge organization, Evaluation metrics; Figures 2–4; Table 1 Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: PMC9007738 | source checked automated source review · 2026-09-16 Audit detailsPrimary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Independent automated spot review corrected interval terminology and sequence-identity scope against the original figure captions and methods. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Diagram title Evaluation procedure Individual claims | CAMI-challenge/OPAL official source Pinned README: introduction; Computed metrics; input format; example evaluations; Methods: Challenge datasets, Challenge organization, Evaluation metrics; Figures 2–4; Table 1 Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: 98120c326eef08e391899e4bd3a362e0e6558b4a | source checked automated source review · 2026-09-16 Audit detailsPrimary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Independent automated spot review corrected interval terminology and sequence-identity scope against the original figure captions and methods. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Datasets A selected CAMI challenge dataset and matching gold standard are required; this entry does not fix the edition. Individual claims | CAMI-challenge/OPAL official source Pinned README: introduction; Computed metrics; input format; example evaluations Version: 98120c326eef08e391899e4bd3a362e0e6558b4a | source checked automated source review · 2026-09-16 Audit detailsPrimary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Independent automated spot review corrected interval terminology and sequence-identity scope against the original figure captions and methods. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Splits CAMI II supplied public-genome practice datasets with ground truth before its blinded challenge. Challenge datasets were marine, strain-madness and plant-associated communities; these are challenge conditions, not a standard supervised train/validation/test partition. Individual claims | cami2 primary benchmark evidence Methods: Challenge datasets, Challenge organization, Evaluation metrics; Figures 2–4; Table 1 Version: PMC9007738 | source checked automated source review · 2026-09-16 Audit detailsPrimary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Independent automated spot review corrected interval terminology and sequence-identity scope against the original figure captions and methods. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Adaptation OPAL scores profiles; it does not train the submitting method. Individual claims | CAMI-challenge/OPAL official source Pinned README: introduction; Computed metrics; input format; example evaluations Version: 98120c326eef08e391899e4bd3a362e0e6558b4a | inapplicable automated source review · 2026-09-16 Audit detailsPrimary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Independent automated spot review corrected interval terminology and sequence-identity scope against the original figure captions and methods. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Metrics Precision, recall, F1, Jaccard, L1 error, UniFrac, Bray–Curtis and diversity measures. Individual claims | CAMI-challenge/OPAL official source Pinned README: introduction; Computed metrics; input format; example evaluations Version: 98120c326eef08e391899e4bd3a362e0e6558b4a | source checked automated source review · 2026-09-16 Audit detailsPrimary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Independent automated spot review corrected interval terminology and sequence-identity scope against the original figure captions and methods. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
Sources and history
Release 2026-09-17-d277315f7d76 · Record review: discovered
4 source records and release history
- cami official source · Original source · Not reported
- Critical Assessment of Metagenome Interpretation: the second round of challenges · Original source · PMC9007738
- CAMI-challenge/OPAL official source · Original source · 98120c326eef08e391899e4bd3a362e0e6558b4a
- cami2 primary benchmark evidence · Original source · PMC9007738
Technical metadata and extraction receipts
Stable ID: discovery-benchmark-cami-taxonomic-profiling
- areas
- microbiome
- entity level
- task
- scope note
- Specialist molecular or omics evaluation; protocol details require review before numerical comparison.
- task
- taxonomic profiling
- version
- Not reported
- benchmark research
- review date: 2026-09-17; status: source_found_structured_extraction_pending; primary sources: evidence-expansion-p2-evidence-discovery-final-cami2-da932c1cde8b; inspected locators: Table1 rankings; main figures and linked supplementary material; searched queries: CAMI II metagenome benchmarking 2022 supplementary results table; gaps: Main Table 1 gives best-ranked software, not a complete numerical score table. Assembly, genome binning, taxonomic binning and taxonomic profiling are separate tasks; strain diversity and sample cohorts must be separated. Further source-data extraction remains before new score publication.; claim scope: Primary-source discovery and table/protocol screening; source checked is not independently reproduced. Raw acquisitions not automatically numerical publication approval.
- historical missing metadata
- dataset release: unextracted; metric implementation: unextracted; split manifest: unextracted; version: unextracted
- metadata review scope
- historical_missing_metadata preserves the original discovery state. Current descriptive evidence and missingness are recorded in profile.facts; numerical-result review is separate.
- legacy kinds
- benchmark
- entity classification
- review date: 2026-09-17; rationale: This record identifies the biological prediction question or a suite-specific task, rather than a uniquely fixed evaluated procedure. Preserve its task identity and leave split, model adaptation and scoring details on linked protocols/evaluations.; source ids: src-discovery-cami-challenge-opal; source locator: Pinned README: introduction; Computed metrics; input format; example evaluations; ambiguities: A paper- or suite-specific task may constrain some inputs or metrics; that alone does not make it interchangeable with a complete versioned protocol. No protocol equivalence is inferred.
Related records
- parent: CAMI
- part of: CAMI
- applicable to: Marker-based microbial profiling
- subject: CAMI taxonomic profiling: part of CAMI