# ChartQA > ChartQA tests visual and logical reasoning about charts. These own-provider reports keep Anthropic's relaxed-accuracy results separate from Zyphra scores whose metric and scorer are not disclosed. Canonical page: https://benchlm.ai/benchmarks/chartqa - Category: [Multimodal & Grounded](/multimodal-grounded) - Last updated: 2024 Anthropic reports and May 2026 Zyphra source revisions; checked October 1, 2026 Evaluation results published by Anthropic and Zyphra. ChartQA test reports: four Anthropic relaxed-accuracy rows and two Zyphra scores with undisclosed metrics. This page retains 6 selected rows across original provider reports about their own models. Comparison-model columns from those reports are excluded. Source model identities remain separate from catalog profiles. Each cohort keeps its original split, printed score, source, and disclosed setup. The source revision or report publication date is separate from an evaluation date. Unknown settings remain unreported, and the scores do not enter overall or category rankings. Separate original-provider cohorts are shown below. Undisclosed metrics and different report setups do not support a shared performance ranking or a pooled score. - [Original Anthropic model card](https://www-cdn.anthropic.com/c6a80a657af445f40e31afac050f3bf76d3b1404/Claude%203%20Model%20Card.pdf) - [Zyphra ZAYA1-VL-8B model card](https://huggingface.co/Zyphra/ZAYA1-VL-8B/blob/2967258fbdfdf8c9358dc52b3451db24fe45dfcd/README.md) - [Zyphra Zamba2-VL-7B model card](https://huggingface.co/Zyphra/Zamba2-VL-7B/blob/060812b62a6eb91820274cf2ec7abc83297534fe/README.md) - [Original benchmark task](https://arxiv.org/abs/2203.10244) ## About ChartQA - Year: 2022 - Format: Provider-reported aggregates; separate source cohorts - Paper: [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://arxiv.org/abs/2203.10244) The Anthropic test rows use zero-shot chain-of-thought prompting. March 2024 results use temperature 0; the June report does not disclose temperature. Zyphra names the test split and its VLMEvalKit-based harness but does not identify the metric, scorer version, or generation settings. The reports are display only. ChartQA is currently displayed on BenchLM for reference, but it is excluded from the weighted scoring formula. ## Leaderboard (6 own-provider reports) ### Anthropic: Claude 3, March 2024 Test; relaxed accuracy; zero-shot CoT; temperature 0; scorer revision unknown. Report publication: 2024-03-04; individual evaluation date not disclosed. | Source model | Reported score | Metric/scorer | Split | Original provider source | |---|---|---|---|---| | [Claude 3 Opus](https://www-cdn.anthropic.com/c6a80a657af445f40e31afac050f3bf76d3b1404/Claude%203%20Model%20Card.pdf) | 80.8% | Relaxed accuracy | test | [Anthropic report](https://www-cdn.anthropic.com/c6a80a657af445f40e31afac050f3bf76d3b1404/Claude%203%20Model%20Card.pdf); Claude 3 model card, Table 3, PDF page 8; sampling statement on page 7 | | [Claude 3 Sonnet](https://www-cdn.anthropic.com/c6a80a657af445f40e31afac050f3bf76d3b1404/Claude%203%20Model%20Card.pdf) | 81.1% | Relaxed accuracy | test | [Anthropic report](https://www-cdn.anthropic.com/c6a80a657af445f40e31afac050f3bf76d3b1404/Claude%203%20Model%20Card.pdf); Claude 3 model card, Table 3, PDF page 8; sampling statement on page 7 | | [Claude 3 Haiku](https://www-cdn.anthropic.com/c6a80a657af445f40e31afac050f3bf76d3b1404/Claude%203%20Model%20Card.pdf) | 81.7% | Relaxed accuracy | test | [Anthropic report](https://www-cdn.anthropic.com/c6a80a657af445f40e31afac050f3bf76d3b1404/Claude%203%20Model%20Card.pdf); Claude 3 model card, Table 3, PDF page 8; sampling statement on page 7 | ### Anthropic: initial Sonnet 3.5, June 2024 Test; relaxed accuracy; zero-shot CoT; temperature and scorer revision unknown. Report publication: 2024-06-21; individual evaluation date not disclosed. | Source model | Reported score | Metric/scorer | Split | Original provider source | |---|---|---|---|---| | [Claude 3.5 Sonnet (initial June 2024)](https://www-cdn.anthropic.com/c6a80a657af445f40e31afac050f3bf76d3b1404/Claude%203%20Model%20Card.pdf) | 90.8% | Relaxed accuracy | test | [Anthropic report](https://www-cdn.anthropic.com/c6a80a657af445f40e31afac050f3bf76d3b1404/Claude%203%20Model%20Card.pdf); Claude 3.5 Sonnet addendum, Table 2, PDF page 44 | ### Zyphra: ZAYA1-VL-8B model card Zyphra harness based on VLMEvalKit; metric/scorer and run settings not disclosed. Source repository modified: 2026-05-17; this is not an evaluation date. | Source model | Reported score | Metric/scorer | Split | Original provider source | |---|---|---|---|---| | [ZAYA1-VL-8B](https://huggingface.co/Zyphra/ZAYA1-VL-8B/blob/2967258fbdfdf8c9358dc52b3451db24fe45dfcd/README.md) | 82.2 | Metric/scorer not disclosed | test | [Zyphra report](https://huggingface.co/Zyphra/ZAYA1-VL-8B/blob/2967258fbdfdf8c9358dc52b3451db24fe45dfcd/README.md); Pinned official model-card Performance table; own-model column only | ### Zyphra: Zamba2-VL-7B model card Zyphra harness based on VLMEvalKit; metric/scorer and run settings not disclosed. Source repository modified: 2026-05-24; this is not an evaluation date. | Source model | Reported score | Metric/scorer | Split | Original provider source | |---|---|---|---|---| | [Zamba2-VL-7B](https://huggingface.co/Zyphra/Zamba2-VL-7B/blob/060812b62a6eb91820274cf2ec7abc83297534fe/README.md) | 85.3 | Metric/scorer not disclosed | test | [Zyphra report](https://huggingface.co/Zyphra/Zamba2-VL-7B/blob/060812b62a6eb91820274cf2ec7abc83297534fe/README.md); Pinned official model-card Performance table; own-model column only | Only each provider's own model rows are included. Unknown metric/scorer means the printed value is retained without interpreting it as accuracy, ANLS, or a percentage. Cohorts do not establish a shared run or protocol. No pooled score or cross-cohort rank is calculated. ## FAQ ### What does ChartQA measure? ChartQA tests visual and logical reasoning about charts. These own-provider reports keep Anthropic's relaxed-accuracy results separate from Zyphra scores whose metric and scorer are not disclosed. ### Can these source results be compared directly? Separate original-provider cohorts are shown below. Undisclosed metrics and different report setups do not support a shared performance ranking or a pooled score. Results stay outside weighted overall and category scoring. ### How many source reports are shown? This snapshot shows 6 selected source reports. Each row retains its original source and disclosed setup. ### Does ChartQA affect BenchLM's overall score? Not directly. ChartQA is still displayed on BenchLM for reference, but it is excluded from the weighted scoring formula.