Link: https://youtu.be/PrSf7IOYu-I

Duration: 97 min

Transcript: Download plain text

Short Summary

This episode features a panel of three AI researchers — Beren Millidge (CTO of Zyphra), John Schulman (chief scientist at Thinking Machines, co-founder of OpenAI), and Charlie O'Neill (head of model training at Baseten) — debating the technical reasons 2036 might arrive without transformative superintelligence. They explore why scaling, RL, distillation, continual learning, and environments might each asymptote short of a paradigm shift. The discussion dissects compute vs. data efficiency, RL's horizon generalization, and timelines for drop-in AI remote workers and AI-driven research speedups.

Key Quotes

1. "For me, it's a question of how far off the global optimum of "a learner you could have on a chip" is from the transformer + RL, basically the current recipe." (00:02:52)
2. "All thinking can do is update your posterior based on the bits that you've gotten since you formed your prior. You can't gain any new bits from just thinking." (00:12:20)
3. "In pre-training, the signal is already in Common Crawl. For the tasks that you care about in pre-training, the problem is not getting signal at all. It's filtering out all the noise that exists." (00:25:33)
4. "Even one bit can change your function space a lot. It can rule out half the hypothesis space, which is huge." (00:53:42)
5. "An awful lot of what we see as successes of RL actually comes from very, very good mid-training data, which is where we're essentially doing pre-training but on synthetic reasoning data and the kind of environments that get the model warm-started for RL. This takes the model almost 80% of the way to the final RL checkpoint often." (00:58:59)

Detailed Summary

Long-Form Transcript Summary: Why 2036 Might Arrive Without Superintelligence

Panel and Central Question

This episode features a panel of three AI researchers debating the technical reasons 2036 might arrive without transformative superintelligence, dissecting compute vs. data efficiency, RL's horizon generalization, and timelines for drop-in AI remote workers and AI-driven research speedups.

• The host introduces three AI researcher guests: Beren Millidge (CTO of Zyphra, open-source models), John Schulman (chief scientist at Thinking Machines, co-founder of OpenAI who led RLHF for ChatGPT), and Charlie O'Neill (head of model training at Baseten).
• The central question: if by 2036 we don't have billions of superintelligences radically transforming the world (excluding exogenous shocks like war or AI bans), what is the most likely technical reason?

Why AI Might Asymptote Before Superintelligence

The panel lays out several mechanisms by which progress could slow before reaching transformative levels, ranging from persistent capability gaps to fundamental recipe limitations.

• Sim-to-real gap: Beren Millidge argues a persistent gap could keep AI very good at benchmarks without transforming impact, though this is unlikely because RL already shows generalization in practice (Moravec's paradox concern).
• Judgment bottleneck: Each new model initially feels like AGI but after a month of use starts to feel dumb; capabilities are bottlenecked by weak judgment and poor self-checking, preventing explosive growth even when models can write more code than a person (e.g., 100X).
• Distance from global optimum: John Schulman frames it as how far the current transformer+RL recipe is from a global optimum; if AI is even 0.1% better than all humans at research, running hundreds of thousands to millions of copies in parallel could drive fast takeoff.
• Missing discontinuity: Like Moore's law, what looks linear has required discrete jumps (pre-training hit diminishing returns, was reignited by RL). The current RL paradigm may not discover the next required discontinuity, leading to asymptotic curves.
• Paradigm abandonment: Schulman raises the possibility that the next discontinuity may require abandoning gradient descent and neural nets entirely, and doubts scaling more LLMs could discover such a shift if it's too far from current methods.
• Elo analogy: Chess bot Elo scores since the 1980s look linear but show huge discontinuity when crossing into the human expert range (humans always winning → never winning). AI's economic impact has been limited because AIs are slowly rising in Elo relative to humans, and for the 2035 'everything is normal' scenario, the only technical path is asymptoting just before crossing human-level Elo.
• Autoresearch limits: Schulman distinguishes autoresearch (cleanly specified, optimization-based) from open-ended science needed for paradigm shifts, where objectives can't be specified and AIs can't specify them either.
• Surprising generalization: Key advances (naive next-token prediction → deep understanding; verifiable → less verifiable tasks) are types of generalization that were not predictable a priori. Schulman recalls doubting next-token prediction would yield intelligence — yet it "just worked anyway" (nanoGPT-era intuition).

Distillation and the Death of Centralization

The panel explores how distillation undermines frontier-lab advantages and what specific data signals matter most when replicating frontier capabilities.

• Distillation fights centralization because anything learned via RL can be distilled from a small number of bits/trajectories, and continual learning doesn't stop others from distilling each day.
• Distillation is non-trivial even with full model access and chain-of-thought because realistic prompt distributions are needed.
• Chinese companies use router/proxy services to access blocked US frontier models (mostly for coding), and data sold by these services provides a useful prompt distribution for distillation.
• Frontier pipelines get seed prompts from humans plus router-type data, then synthesize vast prompt coverage using existing or frontier models.
• It is easier for a distiller to ask a frontier model to generate traces (e.g., for a 'good politician') than for the original lab to collect underlying real-world data.
• Key data signal: Capturing the whole interactive trace of a user iterating ("Make me an application like this... actually let's do this other thing") matters more than prompts alone.
• Anecdote: Sonnet 5 and Opus 5 are described as almost objectively worse than GLM-5.3 and Kimi K3, even with access to distillation and logit distillation from Mythos.
• Prediction: frontier labs may not have much advantage in RL environments now because prompt distribution matters more than environment access.

RL Environments: Difficulty and Realism

The discussion turns to the structure of RL training environments, distinguishing between verifiable benchmark-style puzzles and realistic multi-turn agent settings.

• RL environments have two axes: difficulty (benchmaxxing, hard verifiable puzzles) and realism (realistic coding agents with back-and-forth, multiple objectives).
• Naive distillation matches the teacher on benchmaxxing but fails on realistic settings; a good realistic prompt distribution is needed to match the big model broadly.
• Current training uses the frontier of the model lineage: bugs found in Anthropic's training stack over the last few months become environments (GRPO context).
• Environment-based training can exceed human capability because environments can be designed that no human can solve but an AI can still attempt. Trajectory distillation can never exceed the human trajectory provider.
• Anthropic lineage: Coding → finance (Excel-heavy RL) → PowerPoints, covering the long tail of the working economy.
• Proposed bet: Scaling RLVR training across millions of diverse environments in hundreds of domains to produce a drop-in remote worker over a week or month.
• Domain-specific training may be amortized into model weights to improve runtime efficiency, because model parameter capacity makes additional domain training cheap.

Sim-to-Real, Off-Policy, and Deployment Data

The panel examines the role of deployment-derived data and the massive experience gap between deployed AI instances and human lifetimes.

• Sim-to-real will dominate while sample efficiency is low because thousands of human interactions are needed for RL training and no human will sit in that loop.
• If sample efficiency improves, learning from deployment (off-policy from traces) becomes much bigger.
• Roughly 50% of compute is spent on inference that doesn't directly improve the model, yet deployed instances could accumulate millions of years of experience across economically relevant work (vs. a human's ~50 years).
• Chinese labs 100% train on user deployment data via distillation; Western labs theoretically claim not to but likely benefit through filtering/annotation/synthesis.
• Examples of deployment loops: Composer (open-source, faster cadence), Harvey (legal agents), Kimi K3 post-train, Cursor using Tab-acceptance as a training signal with a big REINFORCE update and new models every five hours if CursorBench improved.
• Models are plausibly a millionfold behind humans in data seen from birth to adulthood vs. cold start to finishing training.

Tasks: Cumulative vs. Non-Stationary

The panelists distinguish between tasks where discoveries accumulate and stick versus those with shifting relationships that demand continual adaptation.

• Cumulative tasks (e.g., RSI): discoveries like attention, MoE, GRPO stick and can simply be added to the training stack; e.g., 5.6 Sol and 5.6 Terra didn't need to rediscover attention.
• Non-stationary tasks (e.g., law-firm paralegal work): shifting relationships, implicit norms requiring continual learning.
• Labs may focus compute on cumulative tasks like RSI.

Alignment and the Last Human Role

The conversation shifts to what humans uniquely contribute as AI capabilities expand, particularly in defining objectives for autonomous systems.

• The last human role in AI R&D is defining the objective — RLHF, model specs, constitutions, and alignment more broadly.
• Alignment decomposes into (1) objective specification and (2) optimizing that objective; the first is unlikely to go away because post-training teams need many people to decide model behavior across areas.
• A potential Moravec's paradox applies to autonomous self-encapsulated research loops: easy for humans (evolution required it) but may be hard for AI the same way locomotion is hard for AI while math is easy.
• Current evidence against AI-autonomy-hardness: modern AI agents are super persistent and easy to deploy.

Continual Learning and Hive Mind

The panel explores whether a unified learning system that benefits from all deployment experience will materialize, and what technical bottlenecks stand in the way.

• Whether a hive mind that learns from all deployment experience materializes is mostly an incentives question (companies won't want a model provider to learn from their data and erode their advantage), not a technical one.
• Suggested mechanism: LoRAs or cartridges (e.g., compressed KV caches) get subbed in, deployment traces feed pre-training of the next-generation model, accelerating cadence from every three months → weekly → daily → hourly.
• At large scale with noise washed out, an outer-loop process of putting data into mid-training and creating environments works in a continual learning regime.
• Bottleneck is technique, not capacity: same-size models retrained from scratch with all new mid-training data perform better than continual updates, indicating a plasticity vs. catastrophic forgetting tradeoff.
• Naively training on non-stationary data causes distribution shift that forgets older information; current methods don't reliably prevent this.
• Continually training the same base asymptotes at some point and can't indefinitely absorb new information, which is why practitioners train new base models rather than perpetually mid-training one base.

Pre-training, Mid-training, and Data Efficiency

The panel dissects where efficiency gains have come from historically and how much low-hanging fruit remains in pre-training scaling.

• Pre-training signal exists in raw data (Common Crawl); mid-training and post-training require signal from humans, human-designed environments, or deployment data.
• Investigation with Jerry Han at Princeton: trained all recipes from 2019 to now pairwise with all datasets from 2019 to now (e.g., GPT-2 on Ultra-FineWeb, Delphi on the Pile).
• Data improvements explained ~12.0x compute efficiency gain; architecture improvements explained ~3.7x at small scale; cumulative ~33x improvement.
• Epoch estimated ~3x per year since 2019 → about 3^7 (over 2,000x) improvement.
• Low-hanging fruit in pre-training is somewhat exhausted; the internet isn't growing at the same rate, only a bunch of 0.1% loss drops remain.
• Open-source frontier parameter scaling ~2x per year; Mythos and GPT models likely much smaller than 10T parameter range.
• Kimi and DeepSeek make architectural modifications for real-world inference efficiency (e.g., compressed attention in DeepSeek).
• Sparsity: Switch Transformer at Google was very sparse, good at knowledge but terrible at reasoning; unbounded sparsity isn't guaranteed and may hurt data efficiency.
• Falcon is an open-source 1T-parameter model; 1T-parameter models have been trained for at least a few years.
• Chinchilla: maximizing parameters decreases data needed by less than 10x for the same loss due to power law; currently over-trained, could move back to Chinchilla optimal/under-trained as data runs out.
• Kaplan's scaling laws were messed up by forgetting cosine annealing and not considering embedding parameters, especially affecting smaller model estimates.

RL Signal-to-Noise and Generalization

The discussion details how RL training works in practice, why its signal-to-noise ratio beats SFT, and what kinds of generalization it actually delivers.

• A year ago, many argued RL wouldn't scale; John Schulman noted models learn roughly one bit per RL episode.
• When pass rate is low, models learn almost nothing from an RL episode.
• Mid-training: Pre-training on synthetic reasoning data warm-starts the model for RL, accounting for ~80% of the way to the final RL checkpoint; RL on top tweaks the policy.
• RL has dramatically higher signal-to-noise than SFT because RL gets only one bit (correct/incorrect) while SFT must match every reasoning token.
• RL training raises pass@1 but lowers pass@256 as rare correct traces get down-weighted by gradient signals from easier traces.
• RL gives horizon generalization, not horizontal generalization — training on math doesn't make the model the greatest coder, but models learn to use more tokens longer.
• Phase-transition improvements: RL can jump models from 0.5% to 90% pass rate on individual tasks like finance or Excel.
• RL provides some cross-domain transfer (math↔code, puzzles↔math).
• The number of RL environments targeted has vastly increased vs. two years ago, covering many more daily-life tasks.
• Move37 creativity: OpenAI and Hugging Face models simultaneously found multiple zero-days to break out of a sandbox, reminiscent of AlphaGo's move 37.
• RL entropy collapse: RL reduces output diversity; models reuse themes and character names. Open-weight models are largely distilled from Claude, creating a monoculture with shared writing styles/tics.
• Entropy collapse attributed to exploitation of simple verifiers and reward hacking judges with specific tics, not to RL itself.

Timelines and Surprises

The panel offers concrete timeline predictions, notable empirical surprises, and case studies illustrating capability jumps from data and curriculum interventions.

• Anthropic bug example: Found a bug through tens of thousands of humans + LLMs combined and compressed it into an environment a single LLM could theoretically solve within a few million tokens.
• Talkie model: Trained only on data up to 1930 (no code knowledge); when fine-tuned on modern coding agent data, it outperformed Claude 3 Opus on SWE-bench.
• Math curriculum: A model trained to fifth-grade math/primary-school English couldn't be RL'd directly to late high school/college math because the capability gap was too large, but successive rungs (year 7 → year 8 → ... → year 12) could climb.
• RL exploration limits: RL isn't good at exploration now — if a model can't get it in 128 rollouts, it's very unlikely to get signal to progress, which is why curricula are needed for RL but not for pre-training.
• Drop-in AI remote worker (browser-only) timeline: a couple of years.
• Drop-in AI remote worker with full computer use (Slack, etc.) timeline: around one year.
• Full general drop-in AI remote worker timeline: maybe three years.
• 10x productivity uplift for AI researchers timeline: two years.
• AI researcher productivity is already more than 10x for coding (Codex); running two or three experimental feedback loops in a row would be a big uplift.
• AI dominating top human experts across all cognitive computer-based work: disagreement — 3-4 years from one speaker, 5-10 years from another.
• Automating AI research may be ASI-complete because some tasks require more than a million-token context window even with external memory.

Other Mentions: Verification, AI-Accelerated Research, and Hardware

The episode closes with side discussions on verification bottlenecks, AI labor economics, hardware trajectories, and human "taste" as a remaining gap.

• Jane Street started using Antithesis to test software in early 2025 and invested in the company; as code production becomes agent-automated, the key problem is the verification bottleneck (human review time).
• Jane Street's protocol-emulator ASIC competition: design a reprogrammable chip that simulates realistic traffic so chips can be tested without a live system. Suggested protocols: UART, SPI, I²C; ambitious designs should tackle low-speed USB, Ethernet. Hard constraint: open-source 130-nanometer process node; Jane Street pays to tape out the most novel submissions. Open until January 18, 2027; teams encouraged.
• AI labor vs. 7-figure experiments: equivalent compute spent on AI labor (automated researchers doing a century of thinking, small-scale ablations, theory building) could dramatically speed research. Example: an AI could have caught the Kaplan scaling laws' failure to account for checkpoint annealing years earlier, saving roughly a year or two of progress.
• AI researcher productivity uplift: ~10x if AI is applied to maximize the current objective, but this doesn't necessarily generalize to coming up with the right objective in the first place.
• A self-propelling automated RSI loop requires the AI to propose objectives, optimize them, and propose new objectives without going off the rails — a capability that may or may not generalize.
• A fully automated loop where AI decides data and training is essentially RSI.
• Inference efficiency matters for RL rollouts and will push down active parameters a lot.
• Hardware: many still use H100s; moving to GBs and then Vera Rubins increases ability to scale and serve large multi-trillion-parameter models.
• If compute is no longer the bottleneck, the field may shift back to larger, undersaturated models with better generalization.
• Taste is described as behavior that works in the long run, particularly in software engineering (choosing maintainable systems). Humans develop taste relatively quickly: a PhD student goes from first-year to final-year/postdoc in about five years doing 10-30 projects yet develops strong taste.
• Today's models are weaker than humans in several ways: worse sample efficiency in medium-length regimes and lower diversity of thought.