Testing CUBE Backgammon’s coaching advice on 47 tournament matches

During CUBE Backgammon’s development, we revisited recorded tournament positions to check the moves suggested by its coach. Whenever CUBE Backgammon flagged an error, we compared its proposed correction with the player’s move using Open Sage, another backgammon engine.

The check found two problems: some recommendations made the position worse, and some sound moves had been labelled mistakes. The figures below describe that development experiment, rather than the reliability of today’s app.

47 recorded matches for the test

We used Hardy Hübener’s match archive for an offline test. The files we could import gave us 47 matches, 930 games and 27,934 checker decisions, spanning 1973 to 2011. The collection includes World Championship matches and other major tournaments.

The recorded dice and moves let us reconstruct positions from matches played decades ago. We owe that record to the players, match recorders and the people who have kept the archive available.

Our question concerned the software: when it disagreed with a strong player, did its proposed move improve on the one already played? The players and the archive were providing a difficult test, with no involvement in or endorsement of CUBE Backgammon.

A better move and the best move are different claims

When a coach suggests a different move, it may improve your position, leave it almost unchanged or even make it worse.

There may also be a third move that neither of you chose, and that is better than both. A coach can help you without finding it. Equally, finding a move it likes does not establish that yours was a mistake.

Here is a small example where we can count the difference exactly.

Three ways to play with two checkers left

You play Ivory with 13 checkers already off and two left on points 6 and 3 when you roll 2–1.

Try the three plays below. For this comparison, ask just one question: how many of the 36 possible next rolls would let you bear off every remaining checker?

This position was created for the article and does not come from a recorded match. The count concerns your next turn only, assuming the game is still going; it is not your probability of winning the whole game.

Play the 2–1

Choose a play to see the remaining checkers and the next-roll count.

Ivory moves toward 1; Red toward 24. This count assumes you get another turn.

Read the position

Ivory, point: number of checkers. 3: 1, 6: 1. Red: 19: 2, 20: 2, 21: 2, 22: 3, 23: 3, 24: 3. On the bar: Ivory 0, Red 0. Borne off: Ivory 13, Red 0.

Although all three arrangements have six pips left to travel, they let you use the next dice in different ways.

With two checkers on 3, you usually need both dice to be at least three. Double two also works: two moves for each checker. That makes 17 rolls.

With one checker on 6, the two dice can work together on the same checker. A 4–2 finishes, as does a 5–1. Double two works here too, because a double gives four moves. Altogether, 27 rolls finish.

If you chose two checkers on 3 and a coach suggested leaving 4 and 2, it would improve this next-turn count from 17 to 23. That would help, but a third play can still reach 27 finishing rolls. That is why “better than my move” and “best available” need separate checks.

You can explore the same idea of giving more rolls something useful to do in Can you give yourself better dice? If the 36-roll count is new to you, our dice guide shows where it comes from.

We asked a second engine to check

Most positions involve more than this bear-off count: a hit now might expose a checker later, while keeping an anchor can cost ground in the race. You need to weigh those consequences too.

For the tournament experiment, we compared the player’s move and our recommendation using a separate analysis engine, Open Sage, at its three-ply setting. It was an independent reference, with its own limitations. A close disagreement between engines still calls for care.

We paid particular attention to the moves the coach marked as mistakes. A confident correction of a sound move can leave you doubting something you had understood correctly.

We checked whether our suggested move was worse than the player’s, and whether we had called a near-best move a mistake.

Using a separate engine also means checking that we send it the right game state. In another development experiment, a turn-order bug distorted an engine comparison: our code supplied the wrong player to roll on half the positions.

Some corrections were worse than the moves played

We had the second engine assess 3,495 of those decisions. Our review rule proposed showing 376 corrections.

According to the reference engine, 14 of those recommendations were worse than the move already played. In 17 cases, the player’s move was close enough to the reference’s best that it should not have been labelled a mistake under our test’s definition. All 14 were among those 17 cases, so the two counts should not be added together.

Most recommendations improved the recorded play, but the rule failed the reliability requirements set before the test: it sometimes recommended a worse move or flagged a sound move as an error.

This is a historical development result, rather than an accuracy figure for today’s app. It also does not tell us how reliable the same rule would be for a club player: that group was not measured in this experiment.

Why a strong player is a harder test

Consider a coach that occasionally misreads a position. When a player makes plenty of clear mistakes, many corrections can still help. When the player makes very few, the coach’s own mistakes become a larger part of what it flags.

Two analyses using the same engine can repeat the same error. For this test, we used Open Sage instead of simply asking CUBE Backgammon to re-examine its own recommendations.

For your own game, the useful habit is to ask what the suggested move achieves. It might complete a prime, preserve a shot or make more rolls useful. The five game plans help connect those moves to the way you are trying to win the game.