ChartQA

We show this table for reference; we do not rank on it.

ChartQA tests visual and logical reasoning about charts. These own-provider reports keep Anthropic's relaxed-accuracy results separate from Zyphra scores whose metric and scorer are not disclosed.

Evaluation results published by Anthropic and Zyphra. ChartQA test reports: four Anthropic relaxed-accuracy rows and two Zyphra scores with undisclosed metrics.

How to read these ChartQA reports

Read with care: the benchmark is not marked current.

Saturation Different source configurations: Separate original-provider cohorts are shown below. Undisclosed metrics and different report setups do not support a shared performance ranking or a pooled score.
Who produced the rows Mirrored table: Selected rows come from the original reports linked in each source cohort. BenchLM did not rerun these evaluations.
Freshness and weight Display only: Separate historical provider cohorts, refreshed reviewed pinned original-provider artifacts. Display only; it does not move rankings. Original upstream task; no task contents mirrored.

Row statuses come from the verification registry; saturation and freshness from the benchmark lifecycle. A provider self-report is displayable evidence, not an independent run.

Anthropic: Claude 3, March 2024

Test; relaxed accuracy; zero-shot CoT; temperature 0; scorer revision unknown.

Report publication: 2024-03-04; individual evaluation date not disclosed.

Anthropic: Claude 3, March 2024: own-provider aggregate scores
Source model	Reported score	Metric/scorer	Split	Original provider source	Parameters (B)	Open / closed
Claude 3 Opus	80.8%	Relaxed accuracy	test	Anthropic reportClaude 3 model card, Table 3, PDF page 8; sampling statement on page 7	Not reported	Unknown
Claude 3 Sonnet	81.1%	Relaxed accuracy	test	Anthropic reportClaude 3 model card, Table 3, PDF page 8; sampling statement on page 7	Not reported	Unknown
Claude 3 Haiku	81.7%	Relaxed accuracy	test	Anthropic reportClaude 3 model card, Table 3, PDF page 8; sampling statement on page 7	Not reported	Unknown

Anthropic: initial Sonnet 3.5, June 2024

Test; relaxed accuracy; zero-shot CoT; temperature and scorer revision unknown.

Report publication: 2024-06-21; individual evaluation date not disclosed.

Anthropic: initial Sonnet 3.5, June 2024: own-provider aggregate scores
Source model	Reported score	Metric/scorer	Split	Original provider source	Parameters (B)	Open / closed
Claude 3.5 Sonnet (initial June 2024)Source column: Claude 3.5 Sonnet	90.8%	Relaxed accuracy	test	Anthropic reportClaude 3.5 Sonnet addendum, Table 2, PDF page 44	Not reported	Unknown

Zyphra: ZAYA1-VL-8B model card

Zyphra harness based on VLMEvalKit; metric/scorer and run settings not disclosed.

Source repository modified: 2026-05-17; this is not an evaluation date.

Zyphra: ZAYA1-VL-8B model card: own-provider aggregate scores
Source model	Reported score	Metric/scorer	Split	Original provider source	Parameters (B)	Open / closed
ZAYA1-VL-8BSource column: ZAYA1-VL-8B(0.7B / 8B)	82.2	Metric/scorer not disclosed	test	Zyphra reportPinned official model-card Performance table; own-model column only	Not reported	Unknown

Zyphra: Zamba2-VL-7B model card

Zyphra harness based on VLMEvalKit; metric/scorer and run settings not disclosed.

Source repository modified: 2026-05-24; this is not an evaluation date.

Zyphra: Zamba2-VL-7B model card: own-provider aggregate scores
Source model	Reported score	Metric/scorer	Split	Original provider source	Parameters (B)	Open / closed
Zamba2-VL-7B	85.3	Metric/scorer not disclosed	test	Zyphra reportPinned official model-card Performance table; own-model column only	Not reported	Unknown

Only each provider's own model rows are included. Undisclosed metrics keep the source's printed number without an accuracy, ANLS, or percentage interpretation. Scorer revisions, individual run dates, exact evaluated checkpoints, image processing, and evaluated question counts remain unknown. Separate cohorts do not establish a shared protocol or a cross-cohort rank. These reports do not affect overall or category scores.

How we show these ChartQA reports

This page retains 6 selected rows across original provider reports about their own models. Comparison-model columns from those reports are excluded. Source model identities remain separate from catalog profiles.

Each cohort keeps its original split, printed score, source, and disclosed setup. The source revision or report publication date is separate from an evaluation date. Unknown settings remain unreported, and the scores do not enter overall or category rankings.

Snapshot

Own-provider rowsSeparate cohortsDisplay only

Original Anthropic model card Zyphra ZAYA1-VL-8B model card Zyphra Zamba2-VL-7B model card Original benchmark task

Separate original-provider cohorts are shown below. Undisclosed metrics and different report setups do not support a shared performance ranking or a pooled score. The table retains each source configuration and its reported value.

6 own-provider reports are shown for ChartQA. The benchmark falls in the Multimodal & Grounded category. We keep external benchmark mirrors separate from the weighted global scoring system, so these results remain source-specific evidence. ChartQA is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.

About ChartQA

Year

2022

Format

Provider-reported aggregates; separate source cohorts

The Anthropic test rows use zero-shot chain-of-thought prompting. March 2024 results use temperature 0; the June report does not disclose temperature. Zyphra names the test split and its VLMEvalKit-based harness but does not identify the metric, scorer version, or generation settings. The reports are display only.

ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning Public benchmark source

Freshness and provenance

Version

Selected 2024 Anthropic reports and May 2026 Zyphra model cards

Refresh cadence

Reviewed pinned original-provider artifacts

Staleness state

Separate historical provider cohorts

Question availability

Original upstream task; no task contents mirrored

Separate historical provider cohortsDisplay only

BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.

Questions

What does ChartQA measure?

ChartQA tests visual and logical reasoning about charts. These own-provider reports keep Anthropic's relaxed-accuracy results separate from Zyphra scores whose metric and scorer are not disclosed.

Can these source results be compared directly?

Separate original-provider cohorts are shown below. Undisclosed metrics and different report setups do not support a shared performance ranking or a pooled score. BenchLM shows this benchmark for display only and does not use it in overall rankings.

How many source reports are shown?

This snapshot shows 6 selected source reports. Each row retains its original source and disclosed setup.

Are all ChartQA values relaxed accuracy percentages?

Anthropic explicitly labels its four selected rows as relaxed accuracy percentages on the test split. The two Zyphra cards identify ChartQA test scores but leave the metric and scorer unspecified. We keep their printed values without adding percent signs, and separate the report cohorts rather than creating a shared ranking.

Last updated: 2024 Anthropic reports and May 2026 Zyphra source revisions; checked October 1, 2026 · retained from pinned original source reports

Know when it’s worth switching models

The model to choose, the cheaper alternative, and the release we would wait on.

Read a sample issue

Join 5,600+ readers.

One email each week. Unsubscribe anytime.