Platform · Judge
Judge agents
Your generative and optimization runs already produce more good designs than any reviewer can rank by hand. Judge agents turn that pile of viable candidates into a ranked, defensible shortlist, with the rubric, the evidence and the rationale attached.
01The problem
LMArena ranks frontier language models without ever scoring one in isolation: it shows two anonymized outputs side by side, randomizes their order, and folds millions of pairwise human votes into a single ranking that survives scrutiny. Engineering teams hit the same wall in reverse. A generative or topology-optimization run returns dozens of candidate designs that all clear the constraints, and choosing the best one is a multi-criteria judgment with an audit trail attached, not a number a spreadsheet can hand you.
02How we solve it
Score, compare, and de-bias every candidate.
Agentriq scores each candidate against an explicit rubric, then settles the close calls with order-randomized pairwise comparisons and explicit bias controls.
Order-randomized pairwise comparison
Every close pair is judged in both orders, A against B and B against A, then averaged, so the ranking never depends on which candidate was shown first.
Criterion-wise rubric scoring
Each candidate is graded against an explicit, weighted rubric one criterion at a time, in the direct-assessment style of Prometheus-2, rather than collapsed into a single opaque number.
Evidence-anchored two-stage evaluation
The judge first extracts and cites the concrete evidence, measurements, simulation outputs and retrieved facts, then grades against it, so verdicts are grounded rather than impressionistic.
Bias mitigation and calibration
Position, self-preference and verbosity biases are actively controlled through order swapping, identity masking and cross-model consensus, the failure modes JudgeBench was built to expose.
03What it produces
A ranking you can defend.
Published work on MT-Bench reports that a strong LLM judge agrees with human evaluators more than 80 percent of the time, about the same rate at which the human evaluators agree with one another, and modern open evaluators reach a 72 to 85 percent agreement-with-human band. LLM judges can still show position, verbosity and self-preference bias, so Agentriq controls for them with structured rubrics, independent checks and, where it matters, several judges rather than one. The judge returns a ranked, rubric-scored shortlist with a written rationale instead of one reviewer's opaque call.
04The agentic loop
Judgment as a closed loop.
Agentriq wraps the judge in a closed agent loop that gathers candidates, scores and ranks them, verifies the winners, and writes the decision back to the design record.
- 01Gather the candidate set and the acceptance criteria from the upstream generative or optimization run.
- 02Score and rank every candidate against the weighted rubric, settling the close pairs with order-randomized pairwise comparisons.
- 03Verify the leading candidates by re-running the decisive tests and confirming the evidence holds under swapped order and an independent judge.
- 04On disagreement or a failed criterion, trace the root cause and pull supporting evidence with Agentic RAG, then re-judge.
- 05Commit the judge's ranked decision, its rubric scores and the written rationale back to the design record.
05Tooling
Simulation, solvers & frameworks.
Get started