Every vendor shows scores on published tests, and almost all of them measure carefully written formal Arabic. But your customers write in their dialects, mix Arabic with English, and ask about your products. A published score does not tell you how the model will handle that. The only reliable way is to test it on your own data before you buy. These are six practical steps to do that, and they work for any vendor, us included.

  1. 1Build the test set from your real conversations
  2. 2Set the pass criteria before you see results
  3. 3Grade blind, with native speakers of each dialect
  4. 4Try to break it on purpose
  5. 5Test on your hardware, at your volume
  6. 6Run it in parallel before cut-over
Does it pass the thresholds you set?
Yes: cut over
No: fix it before your customers see it
Six steps on your own data, decided against criteria written before the test.

1. Build the test set from your real conversations

Collect a few hundred real messages from your channels, with personal data removed, in every dialect your customers actually write, including mixed-language messages and typos. Add the hard questions: what the system should refuse, and what has no answer in your documents. Keep this set yourself and do not hand it to the vendor before the test.

2. Decide what “good” means before you start

Write the pass criteria before you see any result: is the answer correct according to your documents? Does it cite the source? Does it refuse what it should? Does it reply in the right dialect, or in formal Arabic, according to your policy? Set a pass threshold for each criterion. Criteria written after seeing the results tend to justify what you saw.

3. Grade blind, with native speakers

Show the answers to graders who speak the dialects concerned, without telling them which model wrote which. A grader from Riyadh notices things in Najdi that others miss, and a grader from Cairo does the same for Egyptian. If you use a model as an automatic grader, check its judgements against a human sample first.

4. Try to break it on purpose

Before launch, run a session dedicated to pushing the system off its policy: out-of-scope requests, attempts to make it reveal data the asker is not allowed to see, and provocative questions in different dialects. Record what worked, and ask the vendor to fix it before you sign.

5. Test on your hardware and at your volume

A model that shines in a demo can be slow under a real contact centre’s load. Run the test on the infrastructure it will actually run on, with the expected number of concurrent users, and measure the response time the customer sees.

6. Run it in parallel before cut-over

Before the system answers customers directly, run it alongside the current process for a set period: it suggests the answer, and a person decides. Compare the results against your criteria from step two. If it passes the thresholds, cut over. If it does not, you found out before your customers did.

A checklist for any vendor

  1. Will they accept a test on a set of our data they have not seen?
  2. Will they agree pass criteria and thresholds with us before the test?
  3. Does it support the dialects our customers write, not only formal Arabic?
  4. Does it cite the source of each answer from our documents?
  5. Can it run on our hardware or in-country, with a fixed version until we decide to update?
  6. Will they accept a parallel run before cut-over, with a log of every action?

Where we stand

This is how we work with every client: Discover starts by agreeing pass criteria on your data, and Deploy goes through a parallel run before cut-over. We apply the same method to Orchestra in its internal dialect evaluation. Ask us to test on your data, and ask every other vendor too. More in evaluated before launch.

Related pages