Testing an Audit of the Jieqi Engine
In short
- An outside review of the jieqi engine behind Mistboard’s bot found a real evaluation bug: one colour’s dark pieces were never counted.
- Fixing it scored 0.475 against the shipped engine; retuning the 43 terms around the fix, 0.507; replacing the identity averaging the review criticises, 0.370.
- The bot stays as it is. A hand-tuned evaluation absorbs its own bugs, a 200-game gate cannot see +10 Elo, and nothing derived from the engine can beat it.
iwestlin/jieqi-ai vendors Pikafish’s jieqi_old branch, the engine behind the jieqi bot on Mistboard, and ships a source review of it, ai分析.md. Of its five findings, two are known limitations, one is a rules misread, and two were worth a match: a real bug in the evaluation, and the claim that averaging over hidden identities in the search is unsound.
jieqi_old is alpha-beta with a hand-written evaluation and no net. It had already survived ten nets distilled from its own evaluations (best 0.43) and fifteen times the search time on a game a human won (same moves). Three more tests follow, all run the same way: changed engine against shipped, colours alternating, equal time per move, a referee dealing identities so neither engine sees a hidden piece.
The bug: one colour’s dark pieces are never counted
Evaluation::initialize<Us>() runs once for white and once for black. Each call clears the whole DarkPieces array and refills only its own side, so by the time the threat terms read it, white’s dark pieces are gone:
memset(DarkPieces, 0, sizeof DarkPieces); // both colours, every call
for (PieceType i = ROOK; i <= BISHOP; ++i) {
Bitboard b = pos.pieces(Us, i); // only Us is refilled
...
The engine’s eval trace confirms it: with a black pawn attacking a red dark piece, the Threats row is identical for both sides before the fix and the mirror of the red-attacks-black case after. Red was credited for attacking hidden pieces; black never was.
Test 1: fix it
Black had been playing without a term red had, so giving it to black should help.
200 games at 500 ms a move: 91-101-8, score 0.475 ± 0.068. As red 0.525, as black 0.425.
The side that newly receives the credit plays worse, so those weights are wrong as well as miscounted. The odd values in the source (S(-11, 2), S(39, -26)) say they were tuned by play once, with the bug in place. A correct formula with the old weights is a different, untuned engine.
Test 2: retune around the fix
If the weights were tuned for the buggy formula, retuning them for the fixed one should recover what test 1 lost and find what the bug hid.
The 43 terms the bug touches, exposed as UCI options through Pikafish’s TUNE facility, then fishtest-style SPSA on a laptop: 10,000 game pairs at 100 ms a move, six hours. 41 of 43 moved, the clamp on dark-move scores by the most (2862 to 2745). Tuned against shipped, 200 games at 500 ms: 96-93-11, score 0.507 ± 0.067.
Inside noise of even, from the other side this time. Fix-only at 0.475 and fix-plus-retune at 0.507 say these terms carry little strength either way, not that they are under-tuned. A 50,000-pair run on rented CPUs would cost a few dollars; the two results do not argue for it.
Test 3: replace the identity averaging
When the search moves a dark piece it branches once per identity left in that side’s pool and combines the results: a clamped weighted average that falls back to the worst case when the average sits far above it. The review is right that this is not information-set search, and I had my own reason to doubt it: in July the formula chose a dark-piece move with a true expected winning chance of 64% over a safe move at 97%.
If the formula misprices gambles, pricing them properly should pick better moves: when the engine’s top choice moves its own dark piece, take its top four, search every identity of every gamble separately, average the winning chances, play the best average.
Built as a wrapper. 100 games at 400 ms root time with 100 ms per identity: 30-56-14, score 0.370, at 2.2 times the plain engine’s time. 50 games with each identity given the full 400 ms: 17-26-7, score 0.410, at seven times.
It loses at every budget, so the technique is wrong, not underfunded. Separate searches disagree with each other more than one tree disagrees with itself, and the worst-case fallback the review objects to is doing useful work. The July position was a real error; this cure makes more of them than it removes.
Why a hand-tuned evaluation resists piecewise repair
The evaluation is tuned as a whole. Every weight was set by play against every other weight, bug included, so fixing one term regresses and retuning one group recovers only what the fix lost. Improving it means retuning all of it, the work the original author did once.
The gate is coarse. A 200-game match is ± 0.07 at 95%, roughly ± 25 Elo: a change worth +10 is invisible, one worth +30 is a coin flip. A finer gate is 1,000 games, a day per candidate on a laptop.
There is no outside yardstick. Banqi had an open engine to measure against; jieqi has none, so anything distilled from the shipped engine or tuned against it lands at the shipped engine by construction. The signal a teacher lacks is self-play at depth, which is what a net for the jieqi branch would need.
What changes
Nothing in the shipped bot. Moving its pin means a new browser build, a new engine id and a recompute of every cached jieqi review, and nothing here earned that. The bug fix waits for whatever replaces the evaluation it lives in.
The pinned build now ships a Linux binary and a browser build, and the trainer, feature set, referee, match harness and tuner behind every number above are public. A net that scores 0.55 over 200 games against the shipped bot becomes the bot.