Closing the smallest loop first — a game AI pilot on Black and White

Why the chess AI stack never produced a model

The chess-ai stack had 94 source files, 104 tests, a 24M-parameter Transformer, a service, a worker, a dashboard, and k8s manifests. As the RAM-limits post hinted, training/chess-ai/weights/ contained nothing but a .gitkeep, the neural inference tests were conditionally skipped behind it.if(modelExists), and the promotion logic had never once run. The problem was never model quality or missing infrastructure — the loop had never been closed, not even once.

That left one conclusion: before improving anything about chess, close the loop end to end on the smallest possible game.

Why Black and White

We audited nine candidate games. Black and White (흑백게임) won on every “close the loop fast” axis:

• Two players, zero-sum, sequential turns — a one-to-one match with the textbook AlphaZero setup.
• An action space of 9 (tiles 1–9, minus the ones you’ve used) — incomparably faster to debug than Augmented Chess’s 4,708.
• At most 18 plies per game; generating 100 games takes 7ms — iteration time is dominated by training, not data generation.
• There is hidden information (the opponent’s tile number), but the engine already ships a ViewerState projection for it.
• The terminal value is trivial: win +1, draw 0, loss −1.

Fruit Shop was kept as Pilot B, and YINSH was earmarked as the first MCTS target once the loop closes.

Slices 0–4: the skeleton of the loop

The work was cut into thin slices (services, MongoDB, and k8s are slice 6+ — deliberately out of scope).

Slice 0 — packages/black-and-white-ai-core: a GameAIAdapter (init/actor/terminal/legal/apply), a seeded RandomEngine (mulberry32), and playGame with a JSONL serializer. 100 games complete in 7ms, a fixed-seed replay test guarantees reproducibility, 20 tests green.

Slice 1 — a 41-float state encoder: my remaining/used tiles (9+9), the opponent’s submission color per completed round (9, ±1), round outcomes (9, ±1), score difference over target (1), round progress (1), whether I’m second this round (1), and a one-hot of the opponent’s pending first submission color (2). A property test guards the hidden-information boundary — if the opponent’s colors and outcomes are the same, different tile numbers must produce identical encodings.

Slice 2 — the Python mirror (encoding.py), an MLP 41→64→ReLU→64→ReLU→policy(9)+value(1, tanh), and a single-file ONNX export at opset 17. Four Python fixture-parity tests green, plus a Bun-side ONNX inference wrapper with masked softmax.

Slice 3 — the arena script, which caught the first fairness bug during its own verification: the seat-alternation logic (index % 2) was perfectly correlated with the seed-parity first-mover assignment (seed = baseSeed + index), so the challenger played second in all 200 games. The winrate read 0.560 before the fix and 0.502 after. The corrected number is the honest one.

Slice 4 — scripts/iterate.ts: self-play → Python training → ONNX → dual arena → promotion gate → one metrics line, with registry.json tracking the current champion. Verification caught the second design flaw: a greedy-vs-greedy arena collapses to exactly two deterministic games in this title (one per first mover), so the winrate was binary and the 0.55 gate meaningless. Fixed by sampling both sides at --eval-temperature 0.25.

Slice 5: the 52-generation automation session

The iterate command was handed to an automation driver for 48 unattended iterations (gens 005–052, roughly 3–12 seconds per generation), changing one variable per batch.

Batch 1 — signed REINFORCE diverges. Cross-entropy weighted by the ±1 game outcome (REINFORCE) collapsed the policy: scripts/probe.ts showed 100% probability on tile 8 in every state. The cause: ±1 weighting is unbounded below — the term that pushes lost actions toward zero probability grows without limit, and trainPolicyLoss diverged to −1412. Fix: non-negative weights (win-conditioned imitation).

Game-level win conditioning → still uniform. Credit spread across nine rounds is too dilute. Switching to round-AND-game win conditioning (round-outcome), with per-round winners reconstructed in the dataset loader, produced the first real structure in the probe: BLACK→9 (76%) and the “1 snipes 9” sacrifice play (WHITE→1). The signal appeared only when learning from actions that won both their round and their game.

Self-play cycling. Promotions fired but the winrate against random stayed flat — challengers were learning to exploit their predecessor (0.7+ against the champion), not to play better. This is the classic self-play trap (self-play: the model generates training data by playing against itself). Fix: mix a fraction of games against the random baseline (--mix-random), and add a vsRandom anchor gate to the registry — promotion records the champion’s winrate against random, and a successor that can’t hold that number is refused. An initial 0.02 tolerance let the anchor ratchet down 0.563→0.535 across promotions, so it was removed.

Eight generations without promotion → alarm condition. Diagnosis: the fixed point of one-step imitation — imitating the champion’s behavior cannot exceed the champion. Fix: a ValueGreedyEngine, a 1-ply lookahead expert (for each of the nine legal actions, run applyAction → encodeState(next, me) → value and take the best), used as the training teacher. Learning from positions generated by a stronger expert is called expert iteration.

Final result — champion: gen-037. The bare policy head scores 0.543 greedy / 0.583 at T=0.25 sampling against random (1000/400 games). The deployable engine, ValueGreedy(gen-037): 0.923 vs random, 1.000 vs max-tile (1000 games each). The registry holds gen-037 at anchor 0.5825, with a 52-line metrics trail in runs/metrics.jsonl.

The distillation gap, and what comes next

If one finding from this pilot matters most, it is the distillation gap.

While the ValueGreedy engine (with its 1-ply lookahead) scored 0.923 against random, the bare policy head trained on the same data plateaued at 0.55–0.58 — and 24 epochs didn’t change that. The policy softmax simply cannot absorb the lookahead. Seen through the AlphaZero frame, this is the expected conclusion: the deployable engine is the network with its search, not the policy head alone.

Directions from here:

• Soft targets / a bigger policy head. Distilling the expert’s value distribution as a soft target instead of its action index may shrink the gap; so may widening the 64→64 MLP.
• A 2-ply or MCTS-lite expert. The current expert searches a single ply. A deeper teacher means a better learning signal.
• Warm-start training. Every generation currently trains from scratch; starting from the previous champion’s weights should speed convergence and reduce forgetting.
• Pilot B: Fruit Shop. The same slice structure applied to a 3+ player game with hidden bids — the generalization test beyond two-player zero-sum.
• The chess comparison lane. An augmentation-disabled augmented-chess-engine plus the Stockfish anchor, to calibrate this loop against published results.
• Wiring into board-game-client. The end goal is using the trained AI in real solo play — connecting TinyNeuralEngine and ValueGreedyEngine as a computer player in the multiplayer client.

Close the small loop first

The chess AI stack had 94 files and 104 tests and never produced a trained model. This pilot started from the opposite end — the smallest game, a file-based registry, no service, no MongoDB, no k8s. The loop closed in a day.

What the 52-generation automation session taught is worth more than the code: signed REINFORCE divergence, the binary nature of greedy-vs-greedy arenas, self-play cycling, the fixed point of one-step imitation, the distillation gap — flaws that would have cost multiples to debug inside a large model. Because the loop closed on a small game first, six-plus design flaws were diagnosed and fixed from its own metrics trail.

Correction — 0.923 included cheating (2026-06-11)

The day after publishing, deployment planning revealed that the ValueGreedy 0.923 in this post included oracle peeking: the 1-ply lookahead’s applyAction resolved rounds using the opponent’s hidden tile whenever the engine moved second. The encoder’s hidden-information rule and its property test held throughout — the leak entered through the game dynamics path instead, invisible to the probe and to every arena. What caught it was a deployment question: what inputs would this engine receive in a real room?

The honest numbers (1000 games vs random): policy head 0.596, fair belief expectimax 0.592. The ~0.33 gap is the size of the cheating. The incident produced a declared information set on the engine contract (fair/oracle), an information-set invariance test (same visible information + different hidden tiles ⇒ identical behavior), a fair expectimax engine, and a Hidden-Information Fairness Checklist in the guide. The lesson of the distillation-gap section above should read not “ship net + search” but “verify your teacher’s information set first.” Full analysis: docs/task-log/20260610-bw-solo-ai/02-oracle-leak-rootcause.md.

The next step is putting this AI into the actual game. The moment the trained model sits across from a human in a board-game-client room, this loop becomes a product.