We measured Orchestra on six benchmarks. Here are the results, and the method
When we say Orchestra is a small Arabic model that runs inside your organisation, the natural question follows: how small, and at what cost in quality? This post answers with numbers, explains how we measured them, and says where the numbers stop being useful.
What we measured
We ran Orchestra, a 14B-parameter model with 4B active per token, on six widely used tests: MMLU-Pro for general knowledge, GPQA Diamond for hard science questions, LiveCodeBench v6 for coding, HLE without tools, Tau2-Bench for agent tasks, and OALL v2 for Arabic. We then set the results beside the published figures of three larger open models: Gemma 4 at 31B, Qwen3.5 at 27B and DeepSeek-V3.2 at 671B.
The full table, with the method, is on the Orchestra results page. In short: in our own run in November 2026 Orchestra led all three models on four of the six tests and came close on the other two. In Arabic it scored 78.41 on OALL v2.
Why we publish the method with the numbers
A number without a method is worth nothing. A model with 4B active parameters leading a 671B model is a claim that deserves checking, and the CTO who buys from us will check it. So the page lists the model size, serving precision and settings, and we will add the harness and raw-log links when they are published. Re-run it, and if you get a different number, tell us.
One more point of candour: the other models' figures come from their vendors' publications, not from our run, and their settings may differ. The only fair comparison is to run all models with the same settings on your hardware, which is what we help with in any trial.
What the benchmarks do not say
- They do not say how the model will read your contracts and policies. That is measured on your own documents, in a live demonstration.
- They say nothing about dialects. OALL mostly measures Modern Standard Arabic.
- They do not say how much hardware you need. That is why the sizing guide is a separate post.
If you are evaluating Arabic models now, start with how to evaluate an Arabic LLM, then ask us to run Orchestra on a sample of your documents.