Strengths supported by sources
- Purity and completeness expose different binning errors.
Sources
CAMI-challenge/AMBER official source · Pinned README: introduction; Metrics computed per bin/per sample; input formats
Task
CAMI genome binning
The CAMI genome-binning task can be understood through its documented assessment tool; this guide does not identify a challenge-specific run.
Sources
CAMI-challenge/AMBER official source · Pinned README: introduction; Metrics computed per bin/per sample; input formatsAt a glance
Inputs, training, access and other details
Explanatory profile: source reviewed · Automated source review, 2026-09-16. Review applies to the cited claims; unresolved fields are listed below. Numerical results retain their own review status.
| Property | Description and evidence |
|---|---|
| Datasets | A selected CAMI challenge dataset and matching gold standard are required; this entry does not fix the edition.SourcesCAMI-challenge/AMBER official source · Pinned README: introduction; Metrics computed per bin/per sample; input formats |
| Splits | CAMI II supplied public-genome practice datasets with ground truth before its blinded challenge. Challenge datasets were marine, strain-madness and plant-associated communities; these are challenge conditions, not a standard supervised train/validation/test partition.Sourcescami2 primary benchmark evidence · Methods: Challenge datasets, Challenge organization, Evaluation metrics; Figures 2–4; Table 1 |
| Metrics | Bin purity/completeness; sample accuracy, contamination, adjusted Rand index, binned fraction and recovered-genome counts; UniFrac for taxonomic binning.SourcesCAMI-challenge/AMBER official source · Pinned README: introduction; Metrics computed per bin/per sample; input formats |
| Baselines | Submitted programs are compared under the same data condition. Gold-standard assemblies and MEGAHIT assemblies separate binning performance from upstream assembly error; published method identities and versions are listed in Table 1.Sourcescami2 primary benchmark evidence · Methods: Challenge datasets, Challenge organization, Evaluation metrics; Figures 2–4; Table 1 |
| Leakage controls | Challenge genome data and metadata were kept confidential until the challenge ended. Public reference collections dated 8 January 2019 were supplied for reference-based methods. CAMI II also includes public genomes, so novelty is stratified rather than assumed for every organism.Sourcescami2 primary benchmark evidence · Methods: Challenge datasets, Challenge organization, Evaluation metrics; Figures 2–4; Table 1 |
| Uncertainty | Uncertainty is task-specific: taxonomic binning Figure 3 uses standard errors across bins; taxonomic profiling Figure 4 reports means across samples with standard deviations. These are not a common seed-based interval for every CAMI metric.Sourcescami2 primary benchmark evidence · Figure 3 and Figure 4 captions: standard error across taxonomic bins versus standard deviation across samples |
| Entity type | Constituent benchmark task: CAMI genome binningSourcesCAMI-challenge/AMBER official source · Pinned README: introduction; Metrics computed per bin/per sample; input formats |
| Organisms | AMBER accepts community gold standards; the selected CAMI dataset supplies organism membership. · Not applicableSourcesCAMI-challenge/AMBER official source · Pinned README: introduction; Metrics computed per bin/per sample; input formats |
| Assays | Metagenomic sequence assignments and their gold-standard bins/taxa.SourcesCAMI-challenge/AMBER official source · Pinned README: introduction; Metrics computed per bin/per sample; input formats |
| Allowed inputs | Predicted sequence-to-bin or sequence-to-taxon assignments and the matched gold standard.SourcesCAMI-challenge/AMBER official source · Pinned README: introduction; Metrics computed per bin/per sample; input formats |
| Adaptation | AMBER assesses assignments; predictor-training conditions are outside the evaluator. · Not applicableSourcesCAMI-challenge/AMBER official source · Pinned README: introduction; Metrics computed per bin/per sample; input formats |
How it works
How it worksEvaluation procedure
Conceptual procedure. Task variants and protocol versions retain their separate scoring conditions.
Sources (2)
CAMI-challenge/AMBER official source; cami2 primary benchmark evidence · Pinned README: introduction; Metrics computed per bin/per sample; input formats; Methods: Challenge datasets, Challenge organization, Evaluation metrics; Figures 2–4; Table 1Evaluation methodology
CAMI is a series of blinded metagenomic software challenges. In CAMI II, participants received simulated short and long reads from defined communities and submitted assemblies, genome bins, taxonomic assignments or abundance profiles. Reference truth was used only for scoring, and software versions, input read types and community conditions were kept distinct.
Sources
cami2 primary benchmark evidence · Methods: Challenge datasets, Challenge organization, Evaluation metrics; Figures 2–4; Table 1Evaluation design
Benchmarks bring together tasks and protocols. A task describes the biological question; a protocol defines a particular test.
Benchmarks
These source-backed links do not make different protocols or scores interchangeable.
Tested entities and results
Release 2026-09-17-d277315f7d76 · 0 evaluations · 0 metric rows. Different protocols are not a single leaderboard.
No evaluations linked in this release.
Papers and result coverage
Last literature check: 2026-09-17. Primary-source discovery and table/protocol screening; source checked is not independently reproduced. Raw acquisitions not automatically numerical publication approval.
| Paper or primary resource | Version | Reference |
|---|---|---|
| Critical Assessment of Metagenome Interpretation: the second round of challenges | PMC9007738 | Read source DOI: 10.1038/s41592-022-01431-4 |
What is still missing
- Main Table 1 gives best-ranked software, not a complete numerical score table. Assembly, genome binning, taxonomic binning and taxonomic profiling are separate tasks; strain diversity and sample cohorts must be separated. Further source-data extraction remains before new score publication.
Search and extraction details
source found structured extraction pending
Searches
- CAMI II metagenome benchmarking 2022 supplementary results table
Evidence locations
- Table1 rankings; main figures and linked supplementary material
Strengths and limitations
Limitations and conditions
- This profile documents CAMI II as a concrete protocol example. Other CAMI rounds may use different genomes, reference databases and metrics; strain diversity and input assembly quality materially change the task.
Sources
cami2 primary benchmark evidence · Methods: Challenge datasets, Challenge organization, Evaluation metrics; Figures 2–4; Table 1
Profile review details
Primary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Independent automated spot review corrected interval terminology and sequence-identity scope against the original figure captions and methods.
Stable record: discovery-benchmark-cami-genome-binningEvidence table
Inspect claims, sources and review details
Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.
One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.
22 evidence rows matching the loaded filters
| Property and statement | Original source and location | Review and provenance |
|---|---|---|
| Diagram caption Conceptual procedure. Task variants and protocol versions retain their separate scoring conditions. Individual claims | cami2 primary benchmark evidence Pinned README: introduction; Metrics computed per bin/per sample; input formats; Methods: Challenge datasets, Challenge organization, Evaluation metrics; Figures 2–4; Table 1 Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: PMC9007738 | source checked automated source review · 2026-09-16 Audit detailsPrimary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Independent automated spot review corrected interval terminology and sequence-identity scope against the original figure captions and methods. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Diagram caption Conceptual procedure. Task variants and protocol versions retain their separate scoring conditions. Individual claims | CAMI-challenge/AMBER official source Pinned README: introduction; Metrics computed per bin/per sample; input formats; Methods: Challenge datasets, Challenge organization, Evaluation metrics; Figures 2–4; Table 1 Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: f8b3a601043d13fc4227d5691c13561eb4490e50 | source checked automated source review · 2026-09-16 Audit detailsPrimary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Independent automated spot review corrected interval terminology and sequence-identity scope against the original figure captions and methods. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Diagram steps ["Allowed inputs: Predicted sequence-to-bin or sequence-to-taxon assignments and the matched gold standard.","Splits: CAMI II supplied public-genome practice datasets with ground truth before its blinded challenge. Challenge datasets were marine, strain-madness and plant-associated communities; these are challenge conditions, not a standard supervised train/validation/test partition.","Metrics: Bin purity/completeness; sample accuracy, contamination, adjusted Rand index, binned fraction and recovered-genome counts; UniFrac for taxonomic binning."] Individual claims | cami2 primary benchmark evidence Pinned README: introduction; Metrics computed per bin/per sample; input formats; Methods: Challenge datasets, Challenge organization, Evaluation metrics; Figures 2–4; Table 1 Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: PMC9007738 | source checked automated source review · 2026-09-16 Audit detailsPrimary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Independent automated spot review corrected interval terminology and sequence-identity scope against the original figure captions and methods. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Diagram steps ["Allowed inputs: Predicted sequence-to-bin or sequence-to-taxon assignments and the matched gold standard.","Splits: CAMI II supplied public-genome practice datasets with ground truth before its blinded challenge. Challenge datasets were marine, strain-madness and plant-associated communities; these are challenge conditions, not a standard supervised train/validation/test partition.","Metrics: Bin purity/completeness; sample accuracy, contamination, adjusted Rand index, binned fraction and recovered-genome counts; UniFrac for taxonomic binning."] Individual claims | CAMI-challenge/AMBER official source Pinned README: introduction; Metrics computed per bin/per sample; input formats; Methods: Challenge datasets, Challenge organization, Evaluation metrics; Figures 2–4; Table 1 Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: f8b3a601043d13fc4227d5691c13561eb4490e50 | source checked automated source review · 2026-09-16 Audit detailsPrimary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Independent automated spot review corrected interval terminology and sequence-identity scope against the original figure captions and methods. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Diagram title Evaluation procedure Individual claims | cami2 primary benchmark evidence Pinned README: introduction; Metrics computed per bin/per sample; input formats; Methods: Challenge datasets, Challenge organization, Evaluation metrics; Figures 2–4; Table 1 Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: PMC9007738 | source checked automated source review · 2026-09-16 Audit detailsPrimary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Independent automated spot review corrected interval terminology and sequence-identity scope against the original figure captions and methods. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Diagram title Evaluation procedure Individual claims | CAMI-challenge/AMBER official source Pinned README: introduction; Metrics computed per bin/per sample; input formats; Methods: Challenge datasets, Challenge organization, Evaluation metrics; Figures 2–4; Table 1 Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: f8b3a601043d13fc4227d5691c13561eb4490e50 | source checked automated source review · 2026-09-16 Audit detailsPrimary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Independent automated spot review corrected interval terminology and sequence-identity scope against the original figure captions and methods. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Datasets A selected CAMI challenge dataset and matching gold standard are required; this entry does not fix the edition. Individual claims | CAMI-challenge/AMBER official source Pinned README: introduction; Metrics computed per bin/per sample; input formats Version: f8b3a601043d13fc4227d5691c13561eb4490e50 | source checked automated source review · 2026-09-16 Audit detailsPrimary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Independent automated spot review corrected interval terminology and sequence-identity scope against the original figure captions and methods. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Splits CAMI II supplied public-genome practice datasets with ground truth before its blinded challenge. Challenge datasets were marine, strain-madness and plant-associated communities; these are challenge conditions, not a standard supervised train/validation/test partition. Individual claims | cami2 primary benchmark evidence Methods: Challenge datasets, Challenge organization, Evaluation metrics; Figures 2–4; Table 1 Version: PMC9007738 | source checked automated source review · 2026-09-16 Audit detailsPrimary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Independent automated spot review corrected interval terminology and sequence-identity scope against the original figure captions and methods. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Adaptation AMBER assesses assignments; predictor-training conditions are outside the evaluator. Individual claims | CAMI-challenge/AMBER official source Pinned README: introduction; Metrics computed per bin/per sample; input formats Version: f8b3a601043d13fc4227d5691c13561eb4490e50 | inapplicable automated source review · 2026-09-16 Audit detailsPrimary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Independent automated spot review corrected interval terminology and sequence-identity scope against the original figure captions and methods. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Metrics Bin purity/completeness; sample accuracy, contamination, adjusted Rand index, binned fraction and recovered-genome counts; UniFrac for taxonomic binning. Individual claims | CAMI-challenge/AMBER official source Pinned README: introduction; Metrics computed per bin/per sample; input formats Version: f8b3a601043d13fc4227d5691c13561eb4490e50 | source checked automated source review · 2026-09-16 Audit detailsPrimary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Independent automated spot review corrected interval terminology and sequence-identity scope against the original figure captions and methods. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
Sources and history
Release 2026-09-17-d277315f7d76 · Record review: discovered
4 source records and release history
- cami official source · Original source · Not reported
- Critical Assessment of Metagenome Interpretation: the second round of challenges · Original source · PMC9007738
- CAMI-challenge/AMBER official source · Original source · f8b3a601043d13fc4227d5691c13561eb4490e50
- cami2 primary benchmark evidence · Original source · PMC9007738
Technical metadata and extraction receipts
Stable ID: discovery-benchmark-cami-genome-binning
- areas
- microbiome
- entity level
- task
- scope note
- Specialist molecular or omics evaluation; protocol details require review before numerical comparison.
- task
- genome binning
- version
- Not reported
- benchmark research
- review date: 2026-09-17; status: source_found_structured_extraction_pending; primary sources: evidence-expansion-p2-evidence-discovery-final-cami2-da932c1cde8b; inspected locators: Table1 rankings; main figures and linked supplementary material; searched queries: CAMI II metagenome benchmarking 2022 supplementary results table; gaps: Main Table 1 gives best-ranked software, not a complete numerical score table. Assembly, genome binning, taxonomic binning and taxonomic profiling are separate tasks; strain diversity and sample cohorts must be separated. Further source-data extraction remains before new score publication.; claim scope: Primary-source discovery and table/protocol screening; source checked is not independently reproduced. Raw acquisitions not automatically numerical publication approval.
- historical missing metadata
- dataset release: unextracted; metric implementation: unextracted; split manifest: unextracted; version: unextracted
- metadata review scope
- historical_missing_metadata preserves the original discovery state. Current descriptive evidence and missingness are recorded in profile.facts; numerical-result review is separate.
- legacy kinds
- benchmark
- entity classification
- review date: 2026-09-17; rationale: This record identifies the biological prediction question or a suite-specific task, rather than a uniquely fixed evaluated procedure. Preserve its task identity and leave split, model adaptation and scoring details on linked protocols/evaluations.; source ids: src-discovery-cami-challenge-amber; source locator: Pinned README: introduction; Metrics computed per bin/per sample; input formats; ambiguities: A paper- or suite-specific task may constrain some inputs or metrics; that alone does not make it interchangeable with a complete versioned protocol. No protocol equivalence is inferred.
Related records
- parent: CAMI
- part of: CAMI
- subject: CAMI genome binning: part of CAMI