When you ask CUBE Backgammon’s coach for a hint, it compares the positions that different legal moves would leave on the board. A neural network helps estimate how promising those positions are. It learns from examples analysed by GNU Backgammon, an established backgammon engine, so it can produce quick estimates on your phone.
During development, we tried giving that network more examples to learn from. The resulting analysis chose the same move as GNU Backgammon more often, and we briefly released the change. But when we checked the moves by simulating how the games might continue, the new versions performed worse overall. We returned to the previous version.
What the agreement score counts
We explain learning from another engine in our article about how the coach works. GNU Backgammon supplies evaluations of backgammon positions, and a smaller neural network learns to approximate them. We call GNU the teacher in this comparison.
To check the result, we counted how often the analysis using our trained network preferred the same move as GNU. If GNU is usually right, a higher agreement score looks like a reason to expect better advice.
There is a limit to that reasoning: even a strong engine’s evaluation can be wrong. Learning to match its answers more closely can copy its errors too. The agreement score cannot distinguish a good answer from a mistake shared by both engines.
The test that made us revert the release
The version we initially released had gained 2.2 percentage points on that agreement score. We then tested six related trained versions against the existing version on a separate set of positions evaluated by rollout.
A rollout compares moves by playing out many possible continuations from each one. It takes longer than a quick evaluation and still has uncertainty, but it gives us another estimate of which move is best. We could then count how often each tested version chose that move.
Across the six trained versions, average agreement with the rollout-preferred move fell by 2.6 to 3.4 percentage points, depending on how the trained network was used to choose moves. Those figures are averages from the follow-up tests, distinct from the initial version’s 2.2-point gain. Individual results varied, and one trained version improved in one of the tests; taken together, the results did not support keeping the release.
points of agreement with GNU for the released version
points of rollout agreement on average across six versions, depending on the test configuration
Changes in percentage points. The first figure is for the released version; the others are averages from follow-up tests. These are not player ratings.
The comparison covered a finite set of positions. It could not tell us how the software would play every possible game, but it showed why the higher agreement score was not enough to justify the release.
An agreement score treats small and costly errors alike
Imagine two legal moves that are almost equally good. Our analysis chooses the second and GNU chooses the first. A simple agreement score counts a disagreement, even if the difference is too small to matter in practice.
Now imagine another position where one move leaves a costly shot and the other avoids it. Choosing the wrong one also counts as one disagreement. The score has treated two very different decisions alike.
If GNU’s preferred move is itself slightly wrong, matching it can raise the agreement score while giving up a better choice. That is a possible limitation of learning from a teacher; our experiment did not prove that it caused every worse choice we observed.
For a player asking for a hint, the size of an error matters. Choosing between two near-equal moves is different from recommending a move that substantially reduces the chance of winning. A count of matching answers does not measure that cost. In our test of coaching advice on 47 tournament matches, we checked whether CUBE Backgammon’s suggested corrections actually improved the moves already played.
What we changed about the test
We stopped accepting agreement with the teaching engine as sufficient evidence to release a change. We also need to compare the selected moves with a separate reference, estimate the cost of errors and check the result against the version people already use.
The test positions must be kept apart from the learning examples. Comparing the new and existing versions under the same conditions also helps us avoid attributing a difference to the network when the test setup has changed.
Our experiment does not establish that more training data is generally harmful. It establishes that, in this case, a higher score for copying GNU’s answers did not translate into better choices on the rollout-based test.
What these scores say about playing strength
Choosing the rollout-preferred move more often is useful evidence about checker play. It does not, on its own, give CUBE Backgammon an Elo rating or show how it compares with professional players. Our article about engine strength explains what those comparisons would require.
The software running the test needs checking too. In a separate investigation, we supplied a reference engine with the wrong player to roll on half the positions. Correcting that error changed which version the comparison favoured.
This account comes from CUBE Backgammon’s recorded development experiments and their later corrections. The figures describe one historical experiment, not a rating of the current app. The teaching engine discussed here is GNU Backgammon; we credit its contributors for their work. Rollout estimates also have sampling uncertainty and assumptions about subsequent play.