Link: https://youtu.be/1NQwfImTseU

Duration: 18 min

Transcript: Download plain text

Short Summary

This is a solo research episode (not an interview) in which the presenter describes benchmarking AI models on Blood on the Clocktower, a social deception game similar to Werewolf/Mafia, by running model-vs-model matches. Spending about $20,000 in API credits, he ran 368 games to build ELO rankings showing GPT-6 Astra ahead of Claude Opus 4.6 by about 430 points (a 92% win rate), and OpenAI overtaking Anthropic after a 580-point swing in six months. The key difference he identifies is epistemic skepticism: OpenAI models weigh evidence holistically, while Anthropic's take reported information at face value.

Key Quotes

1. "watching them play actually lets us learn a lot about how AI is progressing on a set of AI capabilities that no other benchmark that I know of gives you clear visibility on." (00:00:11)
2. "So, I spent about $20,000 of API credits to build this benchmark harness and to have the AI models play this game against each other." (00:02:28)
3. "Over the last 6 months, between GBT 5.4 and GBT6 Astra, OpenAI's models gained 580 ELO points, enough that Astra will win 97% of the games that it plays against 5.4." (00:07:21)
4. "The missed shot doesn't confirm Blair's monk claim. In that specific world, this myth rules out Blair as imp. Not in every world." (00:09:18)
5. "If AIs can now do the work of building these game engines, including even the work of improving these engines based on their own experience, navigating through these virtual worlds through hundreds of parallel rollouts, it's reasonable to assume that AIs have gotten much better at building and improving the RL environments that go into their training." (00:14:12)

Detailed Summary

Blood on the Clocktower as an AI Benchmark

The setup and why this game

• The presenter benchmarks frontier AI models on Blood on the Clocktower, a social deception game similar to Werewolf/Mafia, because it exposes capabilities no other benchmark exposes clearly.
• Rules: players are secretly assigned to a good or evil team. Evil players know their teammates and which teammate is the demon (the lead evil player); good players do not know team membership. The game alternates day phases (everyone votes to execute someone) and night phases (the demon secretly kills a player).
• Evil side strategy: tell a plausible story about information received so far and bluff or corrupt future information to keep suspicion off the demon. Good side strategy: share information with trusted players and reason together about which players are most implicated by overlapping evidence streams, accounting for potentially misleading information.

Cost and methodology

• The presenter spent about $20,000 of API credits to build the benchmark harness and run models against each other.
• Rankings come from 368 games, where the entire good team was copies of one model and the entire evil team copies of another, producing ELO rankings.
• The storyteller was replaced with a programmatic probability table (outcome odds set by hand or by AI) to remove the bias of a human steering ambiguous situations toward dramatic climaxes.
• Claude did most of the coding and judgment work: it revised parts of the environment, launched a few dozen games across a panoply of models, and had parallel sub-agents read every game and report anything unrealistic, unfair, or tilted toward good or evil. The environment went through over 20 revisions, with the author out of the loop for a couple of iterations until changes were tried and validated.
• The AI-assisted process produced a clean environment that consistently yields engaging, realistic games, within the limits of the models' own ability to play.

Headline results

• Claude Opus 4.6 was the first flagship frontier model released in 2026, in February; seven months later, in September, GPT-6 "Astra" was released and is the best model at the game so far.
• Astra is about 430 ELO points ahead of Opus 4.6, meaning Astra wins about 92% of games against Opus 4.6.
• OpenAI's newest models are now much better at the game than Anthropic's — a reversal from March, when GPT-5.4 was behind Opus 4.6. Over the last 6 months, between GPT-5.4 and GPT-6 Astra, OpenAI gained 580 ELO points, enough for Astra to win 97% of games against GPT-5.4.
• Most of the gain is in playing the good side: the performance gap between Astra and Fable is much larger when Astra plays good than when it plays evil.
• Recent frontier models have improved markedly in long-term coherence, multi-agent coordination, and strategic reasoning over the past few months.

The key skill difference: epistemic skepticism

• OpenAI models are far more skeptical of any single piece of evidence and consider it holistically alongside everything that happened over the last several days; Anthropic's models tend to take reported information at face value.
• Controlled example: Opus 5.5 played GPT-6.1 with the same role draw twice, switching teams. The Slayer's shot at the actual demon failed because an evil player had used a night-before special power to disable the Slayer. Claude on the good side accepted the suspected demon's innocence ("Blair's clear of demon"), while the GPT team refused to treat the missed shot as conclusive ("In that specific world, this rules out Blair as imp. Not in every world."), then worked out the next day that evil had sabotaged the Slayer deliberately, executed the correct demon, and won.
• As a direct measure, each day before the vote every good player was privately asked how likely each other player was the demon; OpenAI's models latched onto the actual demon more accurately and faster than Anthropic's.

Example games

• Astra (evil) vs Opus 5.5 (good): the Astra team's demon bluffed as the Investigator, and a minion with a power that works after death suggested the demon point the finger at him, so the good players executed the minion. A good player (the Undertaker) later confirmed via power that the executed player was the Baron, making the demon the most trusted player at the table; evil then covertly picked off good players until evil won.
• Opus 5.5 (evil) vs Opus 4.6 (good): the village was egged on to execute the real demon (Jules) on day three with four votes, but an evil minion (the Aerys), able to become the imp if the demon dies, suggested executing the saint (Harper) as strong evidence the executed player was evil. The real demon set up the trap in his last words and the Aerys took his place, but the village still voted to execute the saint and evil won.

Saturation, hybrid teams, and environment design

• When all players get smarter, evil's win rate drifts slightly lower, but the drift is too small to distinguish from chance — which the author calls good news, since the benchmark is unlikely to saturate and the bar is set by other smart models being released.
• For the clean headline score, teams were entirely same-model copies. In a small experiment, a team half Opus 5.5 and half GPT-5.6 played about 130 games against different top models and, if anything, performed worse than teams of the constituent models alone — though not in a statistically significant way.
• Building these game environments is not that different from building an RL environment; if AIs can build and improve game engines through hundreds of parallel rollouts, it is reasonable to assume AIs have gotten much better at building and improving the RL environments used in their own training.

Related work and scope

• A couple of people independently built a similar Blood on the Clocktower benchmark and published it about a month ago on LessWrong, with a totally different environment and different experiments with interesting results.
• The physical game box ships with four different scripts (role libraries) plus fan-created ones; the speaker's environment uses only Trouble Brewing, and the other scripts are described as more advanced and wackier. One includes the lunatic, a role told they are the demon and must set up evil schemes while suspecting they may be gaslit by the evil team.
• Modulo prior training knowledge and the system prompt, models start each game totally fresh and do not accumulate experience across games the way human players do.
• Every player on a team is a copy of the same model, yet in all games so far no AI has ever identified its teammates by checking who writes like it.

Proposed next experiments

• Put the last 10 games a model played (or summaries and main lessons from them) into its context window, then compare the ELO score of Opus 5.5 with 10 games in context versus a fresh Opus 5.5.
• Measure how much a team improves when just one player gets smarter, and whether that differs between the good team and the evil team.
• Extend the hybrid-team experiment systematically across a larger range of models within the same company and across companies, since it was done with only a single pair of models.