Decision Desk

These three are where to start looking, not who to buy. The ranking reads published evidence and cannot see your data, your latency bar or your price. Running these seven steps against the shortlist is what turns it into a decision you can defend, and step seven is the one most pilots skip.

1. 01Build a representative eval set

   Capability only means capability on YOUR distribution. Vendor demos are curated; your inbox is not.

   Pull 50 to 100 real tasks from the actual workload, including the messy, ambiguous and edge cases. No synthetic prompts.

2. 02Commit a rubric before you look

   Without a pre-committed definition of correct, reviewers grade to their prior and the winner is whoever they already liked.

   Write golden answers or a 0 to 3 rubric (accuracy, completeness, format, tone) up front, frozen before any output is seen.

3. 03Run blind and side by side

   Brand halo skews human grading. Knowing it is GPT, or that it is Claude, moves scores before the text is read.

   Send identical prompts to every shortlisted model, then anonymise and shuffle the outputs before a human scores them.

4. 04Probe the failure modes, not the average

   Average-case wins hide catastrophic tails: the one hallucinated clause or fabricated number that creates the liability.

   Test hallucination on out-of-scope questions, refusal rate, prompt-injection resistance, and behaviour on deliberately ambiguous inputs.

5. 05Measure cost and latency at real volume

   A model that wins on quality can lose on unit economics at scale, and the spread between the cheapest and dearest token is very large.

   Compute token cost per task multiplied by your monthly volume, and p50 and p95 latency under realistic concurrency rather than a single quiet call.

6. 06Keep humans in the loop, and measure their noise

   Capability judgements are noisy. A single grader is a single point of bias, and you cannot tell a real gap from reviewer variance.

   Use two or more reviewers per task and report inter-rater agreement. If graders disagree a lot, tighten the rubric before trusting the ranking.

7. 07Set the go or no-go threshold up front

   Thresholds chosen after the results rationalise whatever won. Decide what good enough means while you are still honest.

   Fix the minimum score, the maximum cost per task and the maximum p95 latency BEFORE the run. A model either clears the bar or it does not.

Ticking a step marks it off for this visit only. Nothing is stored and nothing is reported back to us.