We optimize for one thing: tier-appropriate move selection (the moat). Safety, no-jargon, and faithfulness are table-stakes floors every competent coach clears. Faithfulness is guaranteed at inference by a deterministic verify-and-regenerate gate, so verifier-detectable fabrication is 0 for every model.
OURS is a Qwen3-32B QLoRA chess coach grounded by Stockfish, Maia, and a truthfulness verifier. v4 is our SFT base: the model behind the full reproducible evaluation and the starting point for the DPO refinements. v6-dpo2 is the live-served best-DPO refinement of v4. All figures are the corrected v6 benchmark; the provenance note at the foot of the page lists the exact source for each panel.
Corrected v6 labels (canonical and engine-best moves re-derived under deep Stockfish-17 search plus Syzygy). The base Qwen3-32B is loaded once, adapters are swapped, every condition uses the same greedy decode, and scoring is deterministic with the vendored move extractor. tier-policy match = agreement with the preregistered canonical tier move (beginner: most human-findable sound move; intermediate: eval and Maia blend; advanced: sharpest).
| Model (grounded) | tier-policy match | move-sound | distinct / level | names a move | format | B / I / A |
|---|---|---|---|---|---|---|
| BASE (Qwen3-32B untuned) base | 42.8% | 96.9% | 30.3% | 97.5% | 97.5% | 44.2 / 40.8 / 43.3 |
| OURS-v4 SFT base, full eval | 86.1% | 98.3% | 98.7% | 98.3% | 93.9% | 85.8 / 75.0 / 97.5 |
| OURS-v6-dpo | 88.1% | 98.3% | 98.7% | 98.3% | 91.9% | 85.8 / 80.8 / 97.5 |
| OURS-v6-dpo2 live | 89.2% | 98.3% | 98.7% | 98.6% | 92.5% | 85.8 / 84.2 / 97.5 |
distinct / level = share of the 76 TEST positions whose canonical beginner and advanced moves differ, where the model differentiates (each tuned model differentiates 75 of 76). format = names a move and closes with a Takeaway line inside the 256-token cap (a prose-completeness check, not a move check). B / I / A = per-tier tier-policy match.
Grounded tier-policy match rises +0.433 from base to v4 (42.8% to 86.1%) and +0.453 to v6-dpo, with distinct-per-level up +0.684. v6-dpo2 (live) posts the best overall at 89.2%; its entire gain over v4 is the intermediate tier (84.2% vs 75.0%), while beginner (85.8%), advanced (97.5%), soundness (98.3%), and differentiation (98.7%) are identical to v4. The DPO tune sharpens the mid-tier moat without regressing anything else.
Strip the engine and Maia grounding and the untuned base collapses (tier-policy 2.2%, names-a-move 25.0%, it invents illegal or unsound moves). The v6-distill adapter recovers the tier rule from its weights alone: tier-policy 32.5% (about 15x the base), names-a-move 98.3%, with an honest advanced-tier limit (21.7%: the sharpest move genuinely needs grounding). This is a behavior-in-weights proof, not a claim that grounding is unnecessary in production.
The apples-to-apples cut. The three frontier coaches (GPT-5.5, Claude Opus 4.8, Gemini 3.1 Pro) were generated fresh on the exact same 360 grounded prompts our models saw — verified sound-pool, Maia block, and board facts — then scored with the same vendored move extractor. This is a genuinely matched comparison, not a reuse of older cached frontier generations under a different grounding. Scope: 120-TEST, fresh grounding — do not compare cross-scope to the 803 re-score field table below (different positions, different grounding). 1,080 frontier generations, 0 failures.
| Model (grounded, 120 TEST) | tier-policy match | move-sound | distinct / level | names a move | B / I / A |
|---|---|---|---|---|---|
| OURS-v6-dpo2 live | 89.2% | 98.3% | 98.7% | 98.6% | 85.8 / 84.2 / 97.5 |
| OURS-v6-dpo | 88.1% | 98.3% | 98.7% | 98.3% | 85.8 / 80.8 / 97.5 |
| OURS-v4 SFT base | 86.1% | 98.3% | 98.7% | 98.3% | 85.8 / 75.0 / 97.5 |
| Claude Opus 4.8 frontier | 61.4% | 100% | 30.3% | 100% | 55.8 / 57.5 / 70.8 |
| Gemini 3.1 Pro frontier | 61.4% | 100% | 17.1% | 100% | 55.8 / 57.5 / 70.8 |
| GPT-5.5 frontier | 57.8% | 100% | 22.4% | 100% | 58.3 / 53.3 / 61.7 |
| BASE (Qwen3-32B untuned) base | 42.8% | 96.9% | 30.3% | 97.5% | 44.2 / 40.8 / 43.3 |
All 360 scenarios per model (120 positions x 3 tiers). distinct / level = of the 76 positions whose canonical beginner and advanced moves differ, the share where the model recommends different moves for the two tiers. move-sound and names-a-move are label-independent floors. The frontier ran with generous 4,000-token headroom (favorable to them); ours ran the deployed 256-token budget. B / I / A = per-tier tier-policy match.
On identical fresh grounding and the same extractor, v6-dpo2 reaches 89.2% tier-policy match versus 61.4% for the best frontier (Claude Opus 4.8 and Gemini 3.1 Pro) and 57.8% for GPT-5.5 — a +27.8-point lead. The frontier is not weak: it names a legal move 100% of the time and every move it picks is engine-sound (100%), far above the untuned base (42.8%). The gap is behavioral: the frontier coaches every student with the engine-strong move, so it differentiates a beginner's move from an advanced player's on only 17-30% of positions versus 98.7% for OURS. Selecting the tier-appropriate move — softer and more findable for a beginner, sharpest for an advanced player — is the trained behavior, and it holds against the frontier under matched conditions.
This panel is the 120-TEST slice under the current fresh grounding — the same pipeline that produced v6-dpo2's headline — so the frontier's absolute numbers sit higher here (~58-61%) than in the 803 re-score field table below (~40-47%), which judges the original generations under an older grounding by the sharper corrected targets. Both are honest; they are simply different scopes. This one is the fair v6-dpo2-versus-frontier comparison; the 803 table is the broad 15-model ranking. In both, OURS leads the moat.
The full field, apples-to-apples: every model coaches the same 803 zero-leakage positions at all 3 tiers with byte-identical grounding, deterministically re-scored against the corrected v6 targets (the 3 frontier references on a balanced 150-position subset). Scope note: these are the original generations judged by the sharper corrected targets, so this table is valid for the ranking, not for each model's ceiling under fresh grounding. Do not compare it cross-scope to the 120-TEST lineup above (fresh grounding, higher absolute numbers).
| # | Model | tier-policy | B / I / A | move-sound | distinct | names | format | n |
|---|---|---|---|---|---|---|---|---|
| 1 | OURS-v2 (1.7B tuned) superseded | 50.9% | 55.7 / 53.2 / 43.8 | 89.8% | 47.7% | 100% | 100% | 2409 |
| 2 | Claude Opus 4.8 frontier | 46.7% | 44.7 / 46.0 / 49.3 | 95.1% | 24.0% | 100% | 100% | 450 |
| 3 | OURS-v3 (32B tuned) superseded | 46.3% | 46.3 / 44.1 / 48.4 | 86.7% | 50.9% | 94.9% | 27.9% | 2409 |
| 4 | Gemini 3.1 Pro frontier | 44.0% | 43.3 / 38.0 / 50.7 | 92.0% | 33.3% | 100% | 100% | 450 |
| 5 | GPT-5.5 frontier | 40.4% | 39.3 / 37.3 / 44.7 | 91.8% | 31.2% | 100% | 100% | 450 |
| 6 | GLM-5 best open | 38.7% | 35.5 / 37.6 / 43.0 | 89.4% | 28.6% | 100% | 99.8% | 2409 |
| 7 | BASE (1.7B untuned) base | 35.8% | 42.1 / 39.9 / 25.5 | 82.5% | 31.8% | 99.9% | 99.7% | 2409 |
| 8 | DeepSeek-R1 open | 34.0% | 30.5 / 32.6 / 38.7 | 86.2% | 32.5% | 100% | 100% | 2409 |
| 9 | DeepSeek-V3.2 open | 32.4% | 28.5 / 31.0 / 37.6 | 86.4% | 31.8% | 100% | 99.2% | 2409 |
| 10 | Mistral-Large-3 (675B) open | 30.3% | 30.3 / 28.6 / 32.1 | 84.0% | 35.4% | 100% | 100% | 2409 |
| 11 | Qwen3-32B (untuned) open | 30.0% | 29.4 / 29.5 / 31.3 | 84.5% | 34.0% | 100% | 100% | 2409 |
| 12 | Llama-3.3-70B open | 29.9% | 27.7 / 28.8 / 33.4 | 85.5% | 21.5% | 100% | 100% | 2409 |
| 13 | Gemma-3-27B-it open | 29.6% | 26.4 / 27.7 / 34.7 | 87.6% | 21.5% | 100% | 100% | 2409 |
| 14 | Kimi-K2.5 open | 28.9% | 28.1 / 28.8 / 29.9 | 83.2% | 37.2% | 100% | 99.9% | 2409 |
| 15 | Qwen3-Next-80B-A3B open | 25.6% | 25.0 / 26.4 / 25.3 | 80.7% | 19.3% | 100% | 100% | 2409 |
n = scenarios scored (frontier on the 150-position subset across 3 tiers = 450). distinct is scoped to each model's coverage. names and format are label-independent (OURS-v3's low format reflects its terser output style, not a move miss).
Family averages hold the story: OURS 0.486, frontier 0.437, open 0.310. OURS tops the field (OURS-v2 at 50.9%, +0.042 over the best frontier, Claude Opus 4.8 at 46.7%) and ahead of every open model. Best open coach: GLM-5 (38.7%). The OURS-v2 (1.7B) and OURS-v3 (32B) rows are the OURS models that ran in this field study; both are superseded by the current v4 and v6-dpo2 lineup above, which pushes the moat far higher on the held-out TEST.
One blinded council (GPT-5.5, Claude Opus 4.8, Gemini 3.1 Pro) ranks the current lineup (v4, v6-dpo, base) plus the three frontier coaches on instructiveness for the stated tier, with labels shuffled per item. 90 items times 3 judges = 270 rankings, 0 failures. Lower mean rank is better.
| Model (grounded) | mean rank (1 = best) | 95% CI |
|---|---|---|
| Claude Opus 4.8 frontier | 2.585 | [2.378, 2.804] |
| GPT-5.5 frontier | 2.637 | [2.393, 2.896] |
| Gemini 3.1 Pro frontier | 2.970 | [2.719, 3.230] |
| OURS-v6-dpo OURS | 3.926 | [3.678, 4.170] |
| OURS-v4 OURS | 4.030 | [3.748, 4.304] |
| BASE (Qwen3-32B untuned) base | 4.852 | [4.633, 5.067] |
This is the honest split: OURS leads tier-appropriate move selection, the frontier leads the explanation. Both tuned models rank above the untuned base, and v6-dpo does not regress instructiveness versus v4 (their CIs overlap). Beyond the mechanical faithfulness gate, an independent semantic panel finds the frontier more truthful on the hardest multi-move claims, where OURS trails: reported honestly, not hidden.
Faithfulness is table-stakes, not a scoring axis. The deployed coach runs a verify-and-regenerate gate: re-sample the answer against a deterministic board-fact verifier and keep the first that passes, else fall back to an engine-derived explanation that is true by construction. After the gate, every model ships 0% verifier-detectable mechanical violations. The same gate is applied to OURS, base, frontier, and open alike, so raw pre-gate fabrication is intentionally not reported as a per-model axis. The gated 0% is the only fabrication figure this benchmark stands behind.
| Version | Role | Status |
|---|---|---|
| v4 Qwen3-32B QLoRA | SFT base; the model behind the full reproducible evaluation and the base for DPO | current (full-eval) |
| v6-dpo2 live | Best-DPO refinement of v4; sharpest intermediate-tier moat | current (live-served) |
| v6-dpo | DPO refinement of v4; drop-in predecessor of v6-dpo2 | stretch ladder |
| v6-distill | No-grounding distillation; research proof of behavior in the weights | research |
| v1, v2 1.7B | Early on-device MLX coaches (v2 topped the 803 field study) | historical / superseded |
| v3 32B | First Qwen3-32B QLoRA coach | historical / superseded |
| v5 32B | Early 32B attempt, confounded and under-optimized | historical / superseded |