Generative AI, and large language models (LLMs) more specifically, have taken the world by storm. And yet, the technology is not new by any means.
The famous paper that started it all was published in 2017. And it took 5 more years of painstaking effort before the technology matured enough to be useful. In 2022, with the release of ChatGPT, LLMs have become the decade’s technology.
Since that fated day, many new companies have entered the generative AI market, launching a worldwide arms race to build the best models. But despite the incredible investments in the field, translation specifically has remained a contested use case in our industry—promised repeatedly, delivered unevenly
One of the potential use cases with the most promise is translation, at least according to these AI labs. But, From our perspective in the language industry, the pattern of new technologies disappointing is all too familiar.
With that in mind, we will investigate Anthropic’s claims and assess Claude’s translation capabilities.
What is Claude (and which version matters for translation)?
Claude is a family of AI language models developed by Anthropic, designed to understand and generate human-like text across a wide range of tasks. When we first published this analysis, the flagship was the Claude 3 family. Anthropic has since shipped several generations, Claude 4, then 4.5, and on through the 4.x line.
As of this update, the current public flagship is Claude Opus 4.8, and Anthropic has introduced a new top tier above it, Claude Fable 5, which it describes as its highest-capability model available for general use.
The three-tier structure that mattered for translation in 2024 still holds today, and it’s the practical lens an LSP should use:
- Opus — the highest-capability tier (currently Opus 4.8). Best where translation demands sustained precision across long, high-stakes documents: contracts, research papers, technical specs where terminology has to stay consistent across thousands of words.
- Sonnet — the balanced tier (currently Sonnet 4.6). For most professional translation work, this is the sensible default: near-Opus quality at materially lower cost per token.
- Haiku — the fast, cheap tier (currently Haiku 4.5). For high-throughput pipelines where speed and cost matter more than nuance and quality is reviewed downstream.
The names will keep moving (Anthropic’s release cadence has accelerated sharply) but that Opus/Sonnet/Haiku logic is stable, and it’s the part worth internalizing.
Which Claude for which job?
- For long, high-stakes documents, Fable or Opus are ideal.
- For most professional translation, Sonnet is more than enough.
- For bulk, speed-first, reviewed-after, Haiku is decent.
Why did we pick Claude instead of ChatGPT?
We originally chose Anthropic’s Claude family on the strength of its benchmark performance, at the time, Claude 3.5 Sonnet ranked among the top three models publicly available, and Claude 3 Opus sat comfortably in the top ten. That pattern has held across generations: Claude’s flagships have stayed consistently at or near the frontier, trading the top spot with OpenAI’s and Google’s latest models rather than being dominated by them.
But here’s the honest 2026 caveat, and it matters for how you read everything below: there is no longer a single “best” model, and that’s true for translation specifically. The current frontier models trade the lead depending on the language pair and the task.
So treat “we picked Claude” as “Claude is a credible frontier contender worth testing for your pairs” (not “Claude wins everything”). We’ll show you exactly where it wins and where it doesn’t.
Can you use Claude for translation? Anthropic's take
Anthropic’s old paper claims that Claude 3 Opus boasts significantly improved multilingual capabilities compared to its predecessors. The leap is so large that it unlocks translation as a viable use case.
This isn’t my assertion, but Anthropic’s 👇
The key phrase to note is “translation services.” It implies capabilities beyond casual, non-professional translation, thus inviting more rigorous scrutiny. You can see further translation claims here and here.
Despite these claims, I’ve yet to find any test or benchmark directly assessing the model’s translation capabilities. That said, let’s dive into Claude 3’s multilingual capabilities.
The paper first addresses multilingual capabilities in section 5. According to the researchers, Claude was trained and tested on multilingual tasks including:
- Translation
- Summarization
- Multilingual reasoning
But it’s in section 5.6 that we can get access to some specifics. Anthropic mainly used two benchmarks here:
- A multilingual math benchmark (MGSM).
- A multilingual general reasoning benchmark (MMLU).
That was the claim in 2024. The difference now is that we no longer have to take Anthropic’s word for it, independent, translation-specific evaluation has since caught up, and we’ll get to it below.
The problem here is clear. These benchmarks are good enough to assess a model’s ability to communicate in different languages (i.e., to be a good chatbot). But they are not enough to test Claude’s translation capabilities.
In the MGSM benchmark, Claude scored over 90% in eight-plus languages, including French, Russian, Simplified Chinese, Spanish, Bengali, Thai, German, and Japanese. In the multilingual MMLU, Claude scored slightly above 80% in German, Spanish, French, Italian, Dutch, Russian, and several other languages.
Beyond these standardized benchmarks, Anthropic employs human feedback as an evaluation tool. Their approach mirrors the Elo rating system I discussed in the leaderboard section. While admittedly limited, I agree with the authors that it’s a solid framework. It assesses an elusive metric that benchmarks struggle to capture: user perception.
Language, unlike many other fields, is inherently flexible. In our industry, what should matter is how end-users perceive a given written piece, not whether it adheres to an arbitrary benchmark. This user-centric approach aligns more closely with real-world translation applications.
But it’s worth noting that Anthropic’s testing, while comprehensive in scope, lacks depth. This limitation leaves us with an incomplete picture. So, it’s time to turn our attention to academia for a more nuanced answer to our question: Are Claude’s translation skills truly relevant in our industry?
What does the research say?
When we first ran this analysis, a search of the arXiv library surfaced 2,595 Claude-related papers, and exactly one that directly investigated Claude’s translation capabilities. That scarcity was the whole problem: Anthropic was marketing “translation services,” but almost nobody had independently tested the claim.
Re-running that search today tells a different story.
As of July of 2026, there are 5,146 otentially Claude-related papers. And yet, of these, 28 investigate Claude in a translation context.
The field has caught up. Beyond the original Enis & Hopkins paper, there’s now work applying Claude to genuinely low-resource settings (for example, a 2025 study on the 13 low-resource national languages of Mali) and a line of research building directly on the paper’s most interesting idea: using Claude to generate synthetic parallel data to improve smaller, cheaper neural MT models.
But the most important development isn’t on arXiv at all. It’s that the field’s flagship competition started benchmarking LLMs head-to-head against dedicated translation engines, under conditions specifically designed to defeat the contamination problem we flagged in 2024.
When we wrote the original version of this article, our most honest complaint was that nobody had directly benchmarked Claude on translation , every number we had tested multilingual reasoning (can it function as a chatbot in other languages?), not translation quality.
That gap has since been filled by the most authoritative source in the field: the Conference on Machine Translation (WMT), the annual shared task that has been the industry’s benchmark of record for two decades. Starting in 2024, WMT began formally including large language models in its evaluation, scored by professional human annotators alongside dedicated MT engines.
WMT24 — Claude wins. In the 2024 General MT task, the organizers evaluated 8 LLMs and 4 online translation providers across 11 language pairs, using a human-annotation protocol called Error Span Annotation. The headline result: the single best-performing system overall was Claude 3.5 Sonnet, winning 9 of 11 language pairs, ahead of GPT-4 and ahead of every online provider.
It’s worth noting for context: dedicated, fine-tuned MT systems were right behind it (one constrained system won 10 pairs within its own category, and Unbabel’s Tower-70B won 8). So Claude was best-in-class, but not in a class of its own. Crucially, WMT24 used freshly collected January-2024 source data specifically to limit contamination, the exact concern we raised about FLORES-200. On clean, unseen data, Claude still came out on top.
WMT25 — the frontier moves. Then came the 2025 edition, and this is where it gets interesting for anyone tempted to crown a permanent winner. The organizers titled their findings paper “Time to Stop Evaluating on Easy Test Sets,” and they meant it. They built deliberately harder test sets using a difficulty-sampling technique, moved to full document-level translation, and expanded to 30 language pairs and 60 systems. Under those tougher conditions, the best system overall was no longer Claude. It was Gemini 2.5 Pro, in the top cluster for 14 of 15 human-evaluated pairs. A dedicated, constrained translation system (Shy-hunyuan-MT) also matched or challenged the general-purpose LLMs.
Read those two results together and you get the single most useful takeaway in this article: the lead changes hands. Claude was the best translator in the field in 2024; a year later, on harder data, it wasn’t.
This is exactly why we keep cautioning against picking a model and assuming it stays best. For an LSP, the implication is operational, benchmark your language pairs, on your content, on a recurring basis, because the answer has a shelf life measured in months.
What professional translators said
Two independent evaluations are worth knowing about, because both rely on human judgment rather than automatic scores:
- In a blind evaluation by professional translators (Lokalise, 2025), Claude 3.5 received the highest “good” rating of any LLM tested, translators preferred its output without knowing which system produced it. This tracks with what practitioners report anecdotally: Claude tends to preserve register, voice, and idiom rather than producing literal, foreign-sounding output.
- Intento’s State of Translation Automation 2025, the most rigorous independent benchmark (dozens of engines across 11 pairs), found Claude and the other frontier models clustering at the top but leading on different language pairs, Claude strong on German, Japanese, Korean, Dutch, Italian; competitors leading elsewhere.
How did they test Claude 3 Opus's translation capabilities?
The researchers evaluated Claude’s translation skills across 37 languages. They used three datasets and two robust evaluation methods. They also compared Claude’s results with those of two “traditional” MT engines: Google Translate and NLLB-54B.
| High-Resource Languages | Low-Resource Languages | Very-Low-Resource Languages |
|---|---|---|
| 1. English | 1. Amharic | 1. Azerbaijani |
| 2. Arabic | 2. Oromo | 2. Nepali |
| 3. French | 3. Gujarati | 3. Pashto |
| 4. Spanish | 4. Hausa | 4. Somali |
| 5. Japanese | 5. Igbo | |
| 6. Hindi | 6. Kinyarwanda | |
| 7. Indonesian | 7. Kyrgyz | |
| 8. Korean | 8. Marathi | |
| 9. Persian | 9. Burmese | |
| 10. Portuguese | 10. Sinhala | |
| 11. Russian | 11. Swahili | |
| 12. Turkish | 12. Tamil | |
| 13. Ukrainian | 13. Telegu | |
| 14. Uzbek | 14. Thai | |
| 15. Vietnamese | 15. Tigrinya | |
| 16. Bengali | 16. Urdu | |
| 17. Yoruba |
The datasets and metrics
This is arguably my favorite part of the study. The researchers leverage three datasets, of which two are completely new ensuring a comprehensive evaluation. Here are the datasets used:
- FLORES-200: This high-quality, robust, human-curated translation dataset from English to 204 different languages. It has largely become an industry standard.
- Maltese speech: A new dataset created from transcriptions and translations for the IWSLT 2024.
- BBC News: Created by the paper’s authors, this dataset consists of BBC News excerpts in 36 languages published after the release date of Claude 3 Opus.
The BBC News dataset is a particularly great addition to their testing. It ensures Claude wouldn’t have seen these strings in its training data, addressing a recurring criticism of LLMs (“learning” vs. memorization).
As for the evaluation metrics, the researchers used widely accepted standards:
- The Bilingual Evaluation Understudy (BLEU): This metric compares a machine translation to one or more reference human translations. It’s based on the overlap of words and phrases between the machine and human translations.
- chrF++: An enhancement of the character n-gram F-score, this metric operates on the character level, making it particularly useful for morphologically rich languages.
These datasets and metrics combined offer a robust framework for assessing Claude’s translation capabilities. In the next section, I’ll dive into the results and what they mean for the future of AI in translation.
Claude 3 Opus's translation performance
Overall, Claude’s translation performance is good. It outperformed Google Translate and NLLB-54B on 25% of language pairs. But this figure is an approximation since Claude’s performance varies significantly depending on the translation direction and the dataset used.
As is common with LLMs, Claude performs substantially better when translating into English rather than from English.
Let’s take a closer look 👇
FLORES-200: Translating into English
When translating the FLORES-200 data into English, Claude significantly outperformed traditional machine translation systems. Here’s a breakdown of its performance:
- Using the chrF++ metric, Claude outperformed in 21 languages.
- When assessed using spBLEU, it outperformed in 20 languages.
In other words, Claude outperformed traditional MT systems 56.9% of the time.
FLORES-200: Translating from English
As you can see in the table below, Claude’s translation performance drops significantly when translating from English into other languages. It only surpasses traditional MTs in the following languages:
- Azerbaijani, Japanese, and Yoruba (using spBLEU).
- Azerbaijani, Burmese, Persian, and Ukrainian (using chrF++).
In other words, it outperforms in 9.65% of instances.
So far, the conclusion seems straightforward. You can start leveraging Claude for your translation projects where the target language is English while avoiding it when translating from English, right?
Well, there’s a catch: Claude’s great performance is unique to the FLORES-200 data.
Why? 👇
BBC: Translating into English
When tested on the BBC dataset, Claude’s performance deteriorates substantially. Regardless of metrics, it only outperforms traditional MTs in 9 languages—that’s a 31.9% drop compared to its performance on FLORES-200 data
BBC: Translating from English
Once again, Claude’s performance decreases when translating from English using the BBC dataset. Regardless of metrics, Claude outperforms Google Translate or the NLLB-54B in only two languages: Korean and Thai.
That’s 2 out of 36 languages! Or 5.56%, which is a 4.09 percentage point drop from its 9.65% performance on the FLORES-200 dataset.
As a side note, the authors also investigated the resource efficiency of Claude relative to 8 other LLMs (including GPT-4 and LLAMA) based on this research. They showed that Claude is more efficient than other LLMs, making it an excellent choice for projects dealing with low-resource languages.
Why does Claude’s performance drop so egregiously?
Claude’s performance drops due to (likely) contamination. In simpler terms, Claude was probably trained on the FLORES-200 data.
While there is no definitive way of knowing, both Google Translate and the NLLB-54B maintain their performance more or less consistently across all datasets (FLORES-200, BBC, and Maltese speech data). But Claude’s performance varies substantially between new and pre-existing datasets, which is an indicator Claude may have been trained on this data.
Of course, unless Anthropic releases Claude’s training data, we won’t know for sure.
It’s worth noting the field came around to this exact concern. When WMT built its 2024 and 2025 LLM benchmarks, it explicitly used freshly collected, unseen source data to limit contamination, and by 2025 titled its findings “Time to Stop Evaluating on Easy Test Sets.” The worry we raised about FLORES-200 in 2024 is now mainstream evaluation practice.
Key takeaways
- Opus excels in translations into English. If your target language is English, it’s a strong contender.
- For translations from English to other languages, traditional machine translation tools still hold an edge. Stick with them for these tasks.
- Opus shines with very-low or low-resource languages, thanks to its superior resource efficiency. It’s an excellent choice for projects dealing with less common languages.
- The model lineup has moved several generations (Opus 4.8 is now the flagship, with Fable 5 above it). Test the current Sonnet tier first for cost-efficiency, Opus for high-stakes documents.
- That being said, since the current translation performance of LLMs may be due to contamination, future research should focus on creating novel tests to assess their translation capabilities.
- WMT24: Claude 3.5 Sonnet was the single best system overall on clean, contamination-controlled data — the strongest independent validation of Anthropic’s translation claim to date.
- WMT25: but on harder 2025 test sets, Gemini 2.5 Pro led and Claude wasn’t on top. The best translator is a moving target —benchmark your own pairs on a recurring basis.
- Independent blind evaluations by professional translators have rated Claude’s output favorably, especially for register and idiom.
I’d like to leave you with the authors’ thought-provoking closing remark: “Our results point toward a future era of LLM-powered machine translation.”
While LLMs undoubtedly have limitations, the current stream of investments in ever-increasing model sizes almost guarantees performance improvements in the next generation of models.
The question isn’t if, but how much improvement will we see? Whether we see an incremental uptick or a quantum leap in capabilities is hard to predict, but one thing’s certain: 2024 levels are unlikely to be the ceiling.
If LLMs are indeed shaping the future of machine translation, it behooves us to take the lead. We must embrace this technology proactively, all while maintaining a critical eye. We need to rigorously assess and test these models in real-world scenarios.
Remember, staying ahead in our field isn’t just about adopting new tools—it’s about understanding them deeply. Are you ready to dive into the LLM-powered future of translation?
FAQ
-
They collected English BBC articles.
-
They made the (good) assumption that an article and its translation will use the same images. So, they used Google Reverse Image Search to find potential translations in other languages.
-
They split these articles into sentences
-
They used Facebook’s LASER tool to align matching sentences across languages.
-
They filtered the data, removing low-quality matches.