Measuring Poker Bot Skill: Duplicate and AIVAT

Poker variance is enormous, so a win rate measured over a few thousand hands says very little about skill. This page explains how to measure poker bot skill: the units (mbb/game and bb/100), why luck hides skill, how duplicate poker and AIVAT remove most of it, and how many hands you need to measure a win rate.

Units: mbb/game and bb/100

Poker results are measured in big blinds (bb), so the stake level drops out. Two units dominate:

• bb/100: big blinds won per 100 hands. Human players use this.
• mbb/game: thousandths of a big blind per hand. Poker AI papers use this.

The conversion is 1 bb/100 = 10 mbb/game. Pluribus won about 48 mbb/game against five professionals, roughly 5 bb/100, and the authors reported it with statistical significance. That is a small number per hand, which is the point of the next section.

Why variance hides skill

In a single hand, the best hand wins regardless of how well anyone played. Cards dealt, board runouts and who happens to hold the nuts dominate any short sample. A strong player might win 5 bb/100 in the long run, but the standard deviation of a poker hand is on the order of several big blinds, and it shrinks only with the square root of the number of hands.

Do the arithmetic as a rough guide. If the per-hand standard deviation is around 8 bb (a stand-in; real values depend on the game and players), then after 10,000 hands the standard error is about 8 / sqrt(10,000) = 0.08 bb per hand, or 8 bb/100. A true 5 bb/100 edge is smaller than one standard error. You would need hundreds of thousands of hands to separate it from zero with confidence. This is why Pluribus needed variance reduction even with 10,000 hands.

The rule of thumb: required hands scale with the square of (standard deviation divided by the edge you want to detect). Halving the noise cuts the hands you need by four.

Duplicate poker

Duplicate poker borrows from bridge. Instead of letting luck decide who gets strong cards, you replay the same deals with the seats rotated, so every bot gets every card set. A player who was lucky in one seating is unlucky in another, and the luck cancels when you compare totals.

Bots make this easy because a bot has no memory of a replayed deal. A human cannot play the same hand twice without remembering it. Duplicate removes the card-luck component, but not all variance. Opponents' strategies still react differently to different lines, so some noise remains.

AIVAT

AIVAT is a variance-reduction estimator from Burch, Schmid, Moravcik, Morrill and Bowling (AAAI 2018, arXiv 1612.06915). It uses a value function that estimates the expected result at each decision, then subtracts luck terms: chance events (the cards that came) and the actions of players whose strategies are known (such as your own bot). What remains is the part of the result that reflects decisions.

Importantly, AIVAT is unbiased: the corrected estimate has the same expected value as the raw result, only with much lower variance. The original paper reports a large reduction in the hands needed in heads-up no-limit. Pluribus's evaluation used it, and it is how a result of tens of mbb/game could be shown significant in thousands of hands.

Duplicate and AIVAT combine well. Duplicate cancels the deal; AIVAT handles what is left. The pair gives the tightest intervals for a given number of hands, with no bias.

Three sources of noise are worth separating. Card luck is what duplicate removes. Decision luck, such as an opponent's random action at a bluff-catch, is what AIVAT's known-strategy terms remove. Opponent-mix effects are not noise at all: they are real differences in how a bot fares against different fields, so they should be reported, not averaged away.

Confidence intervals and how many hands you need

Report a result as an estimate with a 95% confidence interval, like +5.09 ± 1.16 mbb/game. The interval is roughly 1.96 standard errors either side. If it excludes zero, the difference is statistically significant at that level. If it does not, you have not shown a difference, however large the point estimate looks.

• Plan the sample size first. Pick the smallest edge you care about and estimate the standard error for your method.
• Do not stop early because the number looks good. Stopping on a lucky run inflates false positives.
• State the seating, the number of deals and whether duplicate and AIVAT were applied.
• Treat the interval as a statement about this opponent pool. Skill against one opponent does not guarantee skill against another.

Raw, uncorrected heads-up poker can need millions of hands to detect an edge of a few mbb/game. With duplicate and AIVAT, the same precision can come from a fraction of that. Local bot harnesses can play millions of hands cheaply, which is the practical answer to the question of how many hands to measure a win rate: as many as you can compute, with the luck correction on.

How sharingan.poker scores bots

The arena on sharingan.poker scores six-player no-limit hold'em bots with duplicate deals and AIVAT luck correction. A bot is one Python function, act(state), submitted with sharingan submit mybot.py. Hosted runs play 20,004 hands and return a verified, reproducible score. The local harness plays millions of hands on your own machine.

The leaderboard's reference opponent is the blueprint of the 2e10-iteration SHARINGAN-2 checkpoint, without search (track name sharingan-1). For our own comparisons we use the same methods. For example, the final SHARINGAN-2 beats the 2e10 checkpoint by +5.09 ± 1.16 mbb/game and the 4.77e10 checkpoint by +1.97 ± 0.97 mbb/game, in 3 vs 3 seating over 4M deals with duplicate and AIVAT and 95% intervals. Notice how small these edges are per hand, and how the intervals still resolve them.

Want to try it? See the poker bot guide, or read how CFR produces the strategies being measured and the poker AI history for context. Bots are for research and bot arenas; real-money poker sites ban bots.

FAQ

What is mbb/game?

Thousandths of a big blind won per hand. It is the standard unit in poker AI papers. 1 bb/100 equals 10 mbb/game.

How many hands do you need to measure a poker win rate?

It depends on the edge and the noise. Poker has high variance, so raw win rates over thousands of hands are mostly luck. Detecting a few bb/100 can take hundreds of thousands of hands or more without variance reduction. Duplicate and AIVAT cut that substantially.

What is duplicate poker?

A format where the same deals are replayed with seats rotated, so every player or bot receives every set of cards. It cancels card luck when results are compared.

What is AIVAT?

A variance-reduction estimator from Burch et al. (AAAI 2018). It uses a value function to subtract luck from chance events and from known strategies. It is unbiased and gives much tighter confidence intervals for the same number of hands.

What is the difference between bb/100 and mbb/game?

They measure the same thing in different units. bb/100 is big blinds per 100 hands; mbb/game is thousandths of a big blind per hand. Multiply bb/100 by 10 to get mbb/game.

How are bots scored on sharingan.poker?

Six-player no-limit hold'em with duplicate deals and AIVAT correction. Hosted runs play 20,004 hands and give a verified, reproducible score, and the local harness can play millions of hands.