Evaluator

AMBER

AMBER assesses metagenomic genome bins and taxonomic assignments against gold-standard assignments.

Sources

CAMI-challenge/AMBER official source · Pinned README: introduction; Metrics computed per bin/per sample; input formats

At a glance

Inputs, training, access and other details

Explanatory profile: source reviewed · Automated source review, 2026-09-16. Review applies to the cited claims; unresolved fields are listed below. Numerical results retain their own review status.

Data, procedure and scoring
Property	Description and evidence
Datasets	User-supplied bins plus sample-matched gold-standard assignments; example CAMI datasets are linked. Sources CAMI-challenge/AMBER official source · Pinned README: introduction; Metrics computed per bin/per sample; input formats
Splits	The evaluator scores supplied outputs; predictor train/test partitions belong to the dataset/run being evaluated. · Not applicable Sources CAMI-challenge/AMBER official source · Pinned README: introduction; Metrics computed per bin/per sample; input formats
Metrics	Bin purity/completeness; sample accuracy, contamination, adjusted Rand index, binned fraction and recovered-genome counts; UniFrac for taxonomic binning. Sources CAMI-challenge/AMBER official source · Pinned README: introduction; Metrics computed per bin/per sample; input formats
Baselines	Multiple programs or parameter settings can be compared using the same reference. Sources CAMI-challenge/AMBER official source · Pinned README: introduction; Metrics computed per bin/per sample; input formats
Leakage controls	AMBER compares submitted bin assignments with a user-provided gold standard. It does not construct model-training partitions or certify reference-database independence; those controls belong to the evaluated study. · Not applicable Sources amber primary benchmark evidence · Implementation and benchmarking: gold-standard mapping, input formats and metrics
Uncertainty	Uncertainty across samples, datasets or training runs must be defined by the evaluation study; this evaluator entry does not fix one experiment. · Not applicable Sources CAMI-challenge/AMBER official source · Pinned README: introduction; Metrics computed per bin/per sample; input formats
Entity type	Evaluator for genome and taxonomic binning. Sources CAMI-challenge/AMBER official source · Pinned README: introduction; Metrics computed per bin/per sample; input formats
Organisms	The evaluator accepts any community with compatible gold-standard assignments; organism scope belongs to the input dataset. · Not applicable Sources CAMI-challenge/AMBER official source · Pinned README: introduction; Metrics computed per bin/per sample; input formats
Assays	Metagenomic sequence bins and reference assignments. Sources CAMI-challenge/AMBER official source · Pinned README: introduction; Metrics computed per bin/per sample; input formats
Allowed inputs	Predicted sequence-to-bin assignments and sample-matched gold standards. Sources CAMI-challenge/AMBER official source · Pinned README: introduction; Metrics computed per bin/per sample; input formats
Adaptation	AMBER scores submitted assignments; it does not prescribe predictor training. · Not applicable Sources CAMI-challenge/AMBER official source · Pinned README: introduction; Metrics computed per bin/per sample; input formats
Implementation	Not extracted or verified for this record.

How it works

How it worksEvaluation procedure

Conceptual procedure. Task variants and protocol versions retain their separate scoring conditions.

Sources

CAMI-challenge/AMBER official source · Pinned README: introduction; Metrics computed per bin/per sample; input formats

Evaluation methodology

User-supplied bins plus sample-matched gold-standard assignments; example CAMI datasets are linked. The evaluator scores supplied outputs; predictor train/test partitions belong to the dataset/run being evaluated. Bin purity/completeness; sample accuracy, contamination, adjusted Rand index, binned fraction and recovered-genome counts; UniFrac for taxonomic binning. Multiple programs or parameter settings can be compared using the same reference. Uncertainty across samples, datasets or training runs must be defined by the evaluation study; this evaluator entry does not fix one experiment.

Sources

CAMI-challenge/AMBER official source · Pinned README: introduction; Metrics computed per bin/per sample; input formats

Tested entities and results

Release 2026-09-17-d277315f7d76 · 0 evaluations · 0 metric rows. Different protocols are not a single leaderboard.

No evaluations linked in this release.

Papers and result coverage

Last literature check: 2026-09-17. Primary-paper discovery and source inspection. Source-checked results are not independently reproduced experiments.

Paper or primary resource	Version	Reference
AMBER: Assessment of Metagenome BinnERs	PMC6022608	Read source DOI: 10.1093/gigascience/giy069

What is still missing

Do not rank AMBER as a predictor; AMBER is the scoring implementation.
Gold-standard 596 is a reference ceiling, not a competing method.
These are recovered-genome counts, not bin-level precision or seed-averaged means; no uncertainty in Table 1.

Search and extraction details

primary comparison table screened

Searches

AMBER metagenome binning assessment 2018 PMC6022608

Evidence locations

Table 1
Methods: Metrics and accompanying visualizations
Results: CAMI binning evaluation

Strengths and limitations

Strengths supported by sources

Reports purity and completeness separately, revealing over-splitting versus contamination.
Sources
CAMI-challenge/AMBER official source · Pinned README: introduction; Metrics computed per bin/per sample; input formats

Limitations and conditions

Results depend on the supplied gold-standard assignments and taxonomy version. AMBER evaluation alone does not establish independence of the predictor from those references.
Sources
amber primary benchmark evidence · Implementation and benchmarking: gold-standard mapping, input formats and metrics

Profile review details

Primary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction.

Stable record: discovery-benchmark-amber

Evidence table

Inspect claims, sources and review details

Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.

One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.

18 evidence rows matching the loaded filters

Claims, original sources and review scope · Release 2026-09-17-d277315f7d76
Property and statement	Original source and location	Review and provenance
Diagram caption Conceptual procedure. Task variants and protocol versions retain their separate scoring conditions. Individual claims	CAMI-challenge/AMBER official source Original source ↗ Pinned README: introduction; Metrics computed per bin/per sample; input formats Version: f8b3a601043d13fc4227d5691c13561eb4490e50 Retrieved: 2026-09-16T10:30:23.152881+00:00	source checked automated source review · 2026-09-16 Audit details Primary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Field: `attributes.profile.diagram.caption` Source artifact SHA-256: `5908389c2f1f5797ffb73d45776684ff0b34c0fc61a69a2b5b9dbd88df58dcec` Hash scope: Hash scope not separately documented; inspect source record
Diagram steps ["Allowed inputs: Predicted sequence-to-bin assignments and sample-matched gold standards.","Splits: The evaluator scores supplied outputs; predictor train/test partitions belong to the dataset/run being evaluated.","Metrics: Bin purity/completeness; sample accuracy, contamination, adjusted Rand index, binned fraction and recovered-genome counts; UniFrac for taxonomic binning."] Individual claims	CAMI-challenge/AMBER official source Original source ↗ Pinned README: introduction; Metrics computed per bin/per sample; input formats Version: f8b3a601043d13fc4227d5691c13561eb4490e50 Retrieved: 2026-09-16T10:30:23.152881+00:00	source checked automated source review · 2026-09-16 Audit details Primary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Field: `attributes.profile.diagram.steps` Source artifact SHA-256: `5908389c2f1f5797ffb73d45776684ff0b34c0fc61a69a2b5b9dbd88df58dcec` Hash scope: Hash scope not separately documented; inspect source record
Diagram title Evaluation procedure Individual claims	CAMI-challenge/AMBER official source Original source ↗ Pinned README: introduction; Metrics computed per bin/per sample; input formats Version: f8b3a601043d13fc4227d5691c13561eb4490e50 Retrieved: 2026-09-16T10:30:23.152881+00:00	source checked automated source review · 2026-09-16 Audit details Primary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Field: `attributes.profile.diagram.title` Source artifact SHA-256: `5908389c2f1f5797ffb73d45776684ff0b34c0fc61a69a2b5b9dbd88df58dcec` Hash scope: Hash scope not separately documented; inspect source record
Datasets User-supplied bins plus sample-matched gold-standard assignments; example CAMI datasets are linked. Individual claims	CAMI-challenge/AMBER official source Original source ↗ Pinned README: introduction; Metrics computed per bin/per sample; input formats Version: f8b3a601043d13fc4227d5691c13561eb4490e50 Retrieved: 2026-09-16T10:30:23.152881+00:00	source checked automated source review · 2026-09-16 Audit details Primary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Field: `attributes.profile.facts.0.value` Source artifact SHA-256: `5908389c2f1f5797ffb73d45776684ff0b34c0fc61a69a2b5b9dbd88df58dcec` Hash scope: Hash scope not separately documented; inspect source record
Splits The evaluator scores supplied outputs; predictor train/test partitions belong to the dataset/run being evaluated. Individual claims	CAMI-challenge/AMBER official source Original source ↗ Pinned README: introduction; Metrics computed per bin/per sample; input formats Version: f8b3a601043d13fc4227d5691c13561eb4490e50 Retrieved: 2026-09-16T10:30:23.152881+00:00	inapplicable automated source review · 2026-09-16 Audit details Primary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Field: `attributes.profile.facts.1.value` Source artifact SHA-256: `5908389c2f1f5797ffb73d45776684ff0b34c0fc61a69a2b5b9dbd88df58dcec` Hash scope: Hash scope not separately documented; inspect source record
Adaptation AMBER scores submitted assignments; it does not prescribe predictor training. Individual claims	CAMI-challenge/AMBER official source Original source ↗ Pinned README: introduction; Metrics computed per bin/per sample; input formats Version: f8b3a601043d13fc4227d5691c13561eb4490e50 Retrieved: 2026-09-16T10:30:23.152881+00:00	inapplicable automated source review · 2026-09-16 Audit details Primary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Field: `attributes.profile.facts.10.value` Source artifact SHA-256: `5908389c2f1f5797ffb73d45776684ff0b34c0fc61a69a2b5b9dbd88df58dcec` Hash scope: Hash scope not separately documented; inspect source record
Metrics Bin purity/completeness; sample accuracy, contamination, adjusted Rand index, binned fraction and recovered-genome counts; UniFrac for taxonomic binning. Individual claims	CAMI-challenge/AMBER official source Original source ↗ Pinned README: introduction; Metrics computed per bin/per sample; input formats Version: f8b3a601043d13fc4227d5691c13561eb4490e50 Retrieved: 2026-09-16T10:30:23.152881+00:00	source checked automated source review · 2026-09-16 Audit details Primary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Field: `attributes.profile.facts.2.value` Source artifact SHA-256: `5908389c2f1f5797ffb73d45776684ff0b34c0fc61a69a2b5b9dbd88df58dcec` Hash scope: Hash scope not separately documented; inspect source record
Baselines Multiple programs or parameter settings can be compared using the same reference. Individual claims	CAMI-challenge/AMBER official source Original source ↗ Pinned README: introduction; Metrics computed per bin/per sample; input formats Version: f8b3a601043d13fc4227d5691c13561eb4490e50 Retrieved: 2026-09-16T10:30:23.152881+00:00	source checked automated source review · 2026-09-16 Audit details Primary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Field: `attributes.profile.facts.3.value` Source artifact SHA-256: `5908389c2f1f5797ffb73d45776684ff0b34c0fc61a69a2b5b9dbd88df58dcec` Hash scope: Hash scope not separately documented; inspect source record
Leakage controls AMBER compares submitted bin assignments with a user-provided gold standard. It does not construct model-training partitions or certify reference-database independence; those controls belong to the evaluated study. Individual claims	amber primary benchmark evidence Original source ↗ Implementation and benchmarking: gold-standard mapping, input formats and metrics Version: PMC6022608 Retrieved: 2026-09-16T21:04:55.738801+00:00	inapplicable automated source review · 2026-09-16 Audit details Primary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Field: `attributes.profile.facts.4.value` Source artifact SHA-256: `2a75501cbe44396ec0d103d896026d19d823336d131e1fa2aada8555f495a7d0` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Uncertainty Uncertainty across samples, datasets or training runs must be defined by the evaluation study; this evaluator entry does not fix one experiment. Individual claims	CAMI-challenge/AMBER official source Original source ↗ Pinned README: introduction; Metrics computed per bin/per sample; input formats Version: f8b3a601043d13fc4227d5691c13561eb4490e50 Retrieved: 2026-09-16T10:30:23.152881+00:00	inapplicable automated source review · 2026-09-16 Audit details Primary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Field: `attributes.profile.facts.5.value` Source artifact SHA-256: `5908389c2f1f5797ffb73d45776684ff0b34c0fc61a69a2b5b9dbd88df58dcec` Hash scope: Hash scope not separately documented; inspect source record

All evidence (CSV)All evidence (JSONL)How to read the evidence

Sources and history

Release 2026-09-17-d277315f7d76 · Record review: discovered

3 source records and release history

CAMI-challenge/AMBER official source · Original source · f8b3a601043d13fc4227d5691c13561eb4490e50
AMBER: Assessment of Metagenome BinnERs · Original source · PMC6022608
amber primary benchmark evidence · Original source · PMC6022608

Download this release

Technical metadata and extraction receipts

Stable ID: discovery-benchmark-amber

areas: microbiome
entity level: evaluator
scope note: Specialist molecular or omics evaluation; protocol details require review before numerical comparison.
task: Genome reconstruction and taxonomic assignment evaluation
version: Not reported
benchmark research: review date: 2026-09-17; status: primary_comparison_table_screened; primary sources: expansion-p3-amber; inspected locators: Table 1; Methods: Metrics and accompanying visualizations; Results: CAMI binning evaluation; searched queries: AMBER metagenome binning assessment 2018 PMC6022608; gaps: Do not rank AMBER as a predictor; AMBER is the scoring implementation.; Gold-standard 596 is a reference ceiling, not a competing method.; These are recovered-genome counts, not bin-level precision or seed-averaged means; no uncertainty in Table 1.; claim scope: Primary-paper discovery and source inspection. Source-checked results are not independently reproduced experiments.
historical missing metadata: dataset release: unextracted; metric implementation: unextracted; split manifest: unextracted; version: unextracted
metadata review scope: historical_missing_metadata preserves the original discovery state. Current descriptive evidence and missingness are recorded in profile.facts; numerical-result review is separate.
legacy kinds: benchmark
entity classification: review date: 2026-09-17; rationale: The cited profile describes scoring or assessment software/procedures applied to predictions, rather than the biological target or the underlying dataset.; source ids: src-discovery-cami-challenge-amber; source locator: Pinned README: introduction; Metrics computed per bin/per sample; input formats; ambiguities: None recorded

Related records

Suggest a correction