Today, I’m chatting with Noam Brown, 
who is a researcher at OpenAI.
He was one of the foundational contributors 
to what became o1 and the reasoning models.
Now he’s working on multi-agent systems.
Speaking of which, you guys announced last
week that you solved one of the 
Millennium Prize Problems with a
system of 10,000 different AI agents that 
spent 130 billion tokens over 88 hours.
One of the reasons I’m interested in talking 
to you is that you were among the first people,
maybe two or three years ago, who were 
thinking about how the reasoning models
would allow us to see into the future.
Because if you scale up inference compute,
you can see what the base capabilities of 
the models will be a few years in the future.
I feel like you’re in a similar position now 
to help us understand what future capabilities
will look like, given the enormous scaling 
of agent sizes that we can do right now.
The way I think about it, when you plot the 
performance of these reasoning models with
test-time compute on the x-axis and performance on 
basically any reasoning benchmark on the y-axis,
you see a very clear pattern where the 
longer these models take to think about
their answer, the better they do.
This is a very natural thing.
It’s the same thing with people.
If you’re taking the SATs and you
have five minutes to go through the entire 
exam, you’re not going to do very well.
If you have five hours, you’re 
probably going to do a lot better.
The AI models are pretty similar.
They’ll spend that time doing this
monologue to themselves, figuring things 
out, going through different cases,
ruling out different possibilities, building 
on some of their previous discoveries.
The problem is that as you push that further 
and further, you hit a latency bottleneck.
You don’t want to sit around for 
three years waiting for a response.
So what you can do is what a lot of people do.
They parallelize. They just get a team of people.
If you’re going to found a 
company, you want to get a group
of people together so you can go faster.
It’s the same thing with these AI models.
It helps to just have multiple agents working 
on something because they can go faster.
So multi-agent is a way of scaling test-time 
compute in parallel instead of purely serially.
It is less efficient, because it’s not like 
a single agent has all the context to itself.
But it is a very effective way of scaling 
test-time compute if it’s done well.
I’m going to ask a bunch of naive questions.
This is an unreleased model, so we haven’t
publicly seen how these systems work.
I just have a bunch of ways in which I’m
confused about what the qualitative 
properties of such systems are.
I am shocked by the scale of 
cognitive effort that you can
concentrate in such a short period of time.
Think about what 130 billion tokens are.
If it were a single human thinking as a full-time 
job, stretched back to back, 130 billion tokens
would be a human thinking for 4,000 years.
Eight hours a day, working a normal work week.
Starting from ancient Sumeria up till today, 
a single sequential human thinking that long,
concentrated in 88 hours.
I feel like qualitatively,
that is a super important consideration.
I’m surprised that there isn’t
a bigger parallelization penalty.
You can just have 10,000 agents collaborate.
Maybe because the agents are better 
at collaborating than humans might be,
they’re going much faster.
They can actually
productively collaborate at such a big scale.
Or maybe there is a big parallelization penalty.
Let’s talk about the parallelization penalty, 
and then we can talk about the qualitative stuff.
The truth is that we don’t have very good science 
on multi-agent scaling up to this kind of scale.
When we released 5.6, I think 
that was the first time that we
had a proper multi-agent system in our models.
We actually did show some plots in the blog post
of the scaling performance of multi-agent 
systems, because we have it as an option.
It’s Ultra Mode. The default is four 
agents, but you can set that higher.
In the plot, we show what the performance 
looks like on some benchmarks for one agent,
for four agents working together, 
for 16 agents working together.
It depends on the benchmark, but for some 
of the benchmarks, what you see is that if
you have four agents working on the 
problem, it is done twice as fast.
Because there are four agents 
working for half as long, you’re
paying 2x more to get an answer twice as quickly.
If you go to 16 agents, you see a similar pattern.
It’s a little less efficient, but 
you continue to see that performance.
Is it a linear serial time speedup or 
a sublinear speedup as you increase the
number of parallel agents?
It’s slightly sublinear,
though it does depend a lot on the problem.
Math, for example, is quite parallelizable.
It’s not the most parallelizable 
thing, but it is very parallelizable.
Web search, things like doing a Deep Research 
report where you have to look through a bunch
of sources, is extremely parallelizable.
I suspect that something like writing
a novel would be very unparallelizable.
You would probably not see a big benefit
from having 10,000 agents working on a 
novel together, in the same way that you’d
probably not get a big benefit from having 
10,000 people work on a novel together.
So the performance does depend on the domain.
We do measure it up to 16 or so agents in
our published blog posts.
The problem is that it’s
very hard to push that science to 10,000 
agents because it’s just so expensive.
You guys just did it over a weekend.
But that’s one data point.
We don’t know how long it would take a single 
agent to solve Navier-Stokes, because we haven’t
done that experiment yet.
Maybe we will,
but that’s also only one data point.
If we want to do a thorough ablation, the
experiments are just too expensive at that scale.
So we have to do some kind of methodical science
about what happens when you go to 64, 128, 256 
or something and get a sense of the behavior.
But it’s going to be very hard to push 
that all the way to 10,000 and know for
sure what the benefit was that we actually 
got from using 10,000 agents versus 1,000.
There’s one thing I want to make clear.
The effort to solve a Millennium Prize Problem,
this was not due to multi-agent.
I wouldn’t even attribute 10%
of the credit to multi-agent.
The reality is that OpenAI has
trained a very powerful model.
We can get that model to
operate over very long horizons.
We can get it to think in parallel.
But at its core, the reason why we’re able to do 
this is because we just have a general-purpose,
very strong model.
Things like multi-agent
are flashy and new, and that probably gets 
disproportionate credit for that reason.
But the core reason is this 
is just a very powerful model.
The generalization is quite shocking to me.
I don’t know how these systems were trained,
but presumably they were 
trained how RL training happens.
You have a bunch of checkable synthetic 
problems and you do a bunch of RL against them.
Nowhere in the training process, I’m 
guessing, was the model solving anything
as ambitious as a Millennium Prize Problem.
But the generalization was strong enough
that you could have these much easier verifiable 
problems generalize to this much parallel effort
on such a hard problem.
I think that is true.
First of all, we do train the 
model on very hard problems.
There is definitely a gap.
We see that if we train on
some kinds of tasks, it’s able to do 
tasks that are more ambitious than that.
There is an interesting challenge that 
as the models become smarter and smarter,
a lot of the kinds of questions 
we can ask them are just too easy.
It’s hard to challenge the model.
I do think that’s going to be interesting.
If I had to make an argument for why you might not 
see AIs like LLMs go the same path as AlphaGo and
AlphaZero and all these kinds of game-playing 
AIs, it might be this kind of problem.
In things like AlphaZero, where you have 
self-play, you have an infinite curriculum.
You’re always playing against 
an AI that’s equally strong.
Whereas for things like training an LLM with 
reinforcement learning, at least the ways
that are out there right now, you give the 
model a problem and you ask it to solve it.
If the problem is so easy that it can just solve 
it in a second, it's not really learning anything.
If we run out of problems to challenge 
it, then that is a plausible scenario
where it becomes much harder to make progress.
Now, I do think there are ways around that.
We haven’t really hit that as a wall yet.
I think that if it ever became a serious problem,
there would be ways around it.
But it is a plausible scenario.
Just for the audience, when you’re 
referring to AlphaGo or AlphaZero,
you’re talking about getting superhuman relatively 
fast after achieving human-level performance.
If you look at the trajectory of game-playing 
AIs, like Go, within a span of a year they went
from beating a European champion — something 
like number 50 in the world — to beating the
world champion, to being unimaginably, orders 
of magnitude stronger than any human alive.
It’s possible that in domains like math we see a 
similar trajectory, but I think there is a very
plausible scenario where that doesn’t happen.
I want to understand, if in six months people
will have access to multi-agent systems, 
how should one model what it is like to
collaborate with or hire a multi-agent system?
I should start by talking about how these
multi-agent systems actually work, which 
I think is a very different way than a lot
of multi-agent systems in other AIs.
A lot of people that have approached
multi-agents for things like LLMs tend 
to take this very scaffolded approach.
For example, there might be a coordinator 
agent that delegates work to a bunch of
children and gives them a task.
The children work on it and then
return their answer.
This seems like a very
sensible setup, a very sensible scaffold.
It definitely helps, but there are a bunch
of limitations with these kinds of setups.
For example, if in this setup you have
a coordinator that’s sending tasks to 
children, and the children work on it
and then return their answers, what happens 
if two children are given similar tasks?
Can they talk to each other?
Usually the answer is no.
That’s very inefficient. If you’re given a task 
and it’s actually really helpful to talk to
somebody that might know an answer to a question 
that you’re working on — or part of something
that you’re working on — it’d be really helpful 
for you to just be able to ping them and say,
"Hey, can you help me out with this thing?"
But a lot of systems don’t have that setup.
Adding it significantly increases the 
complexity of the scaffold that you have.
Another thing is, what if the child doesn’t 
really understand or has a clarification question?
Then it has to choose between, "Okay, do I just 
return and ask the question instead of solving
the problem?" or "Do I solve the problem, 
make an assumption about what the parent
wanted me to do, and just solve it that way?"
In any scaffold that people come up with,
there are always limitations involved.
The approach that we wanted to take was
to just go toward the extreme end of baking in as 
little structure as we could and give the agents
very primitive tools to use, and they figure 
out for themselves how to use them effectively.
So we give the agents the ability to message 
another agent, and when it messages another
agent, it is inserted into the context.
It can do a few other similar things,
but that’s basically the core of it.
It can just send a message whenever
it wants — just a tool call — and 
it can send that to other agents.
They figure out for themselves the 
best way to coordinate around that.
It turns out that if this is done well, 
you get very sophisticated behavior.
To me, it looks a lot like how human collaborators 
work over something like Slack, for example.
When we were working on this project, it was 
really exciting when we finally got it working
to see these agents working on problems together.
I remember one example. We give the agents a
problem, and then one agent says, 
"I think I’ve got the answer."
Then another agent says, "Actually, 
I got a different answer."
Then they have this whole discussion about, 
"Well, how did you arrive at that answer?
Can you explain it to me?"
Going back and forth and trying to clarify what
could’ve been wrong in each other’s reasoning.
Then they finally converge on, "Oh, yeah.
Okay, that seems right." Then it 
just broadcasts to the other agents,
"Actually, I’ve changed my answer.
I think he’s right." It just felt
like a very natural conversation.
It felt like when you see chain of
thought for the first time that’s trained through 
reinforcement learning, and you’re like, "Oh,
this is just kind of like what a person would 
think if they were writing down their thoughts
as they’re thinking them."
It felt like that. It is
really cool to see this kind of behavior.
Collaborating with these things, honestly,
feels a lot like collaborating with a person.
It’s just a very natural flow.
Except one qualitative difference that might 
become salient in the future is that these systems
will be thinking maybe more than 10x as fast, if 
you just look at how many tokens per second they
output versus how fast a human talks.
They’re working all the time.
They’re not sleeping. They’re collaborating 
with each other at a much more intense pace
than humans have the capacity to 
collaborate with other humans.
I’m trying to think of what to 
qualitatively expect in a year.
Is it like a shadow organization that is moving 
100x faster in my company than the human level is?
What would take a human organization 
a year to do is happening within a
week within this shadow organization?
Will it feel foreign? I don’t know.
I’ve actually found that it’s surprisingly 
natural to work with these things right now.
I think that could change.
For example, we have these ultra-fast modes that
enable sampling to be 10-15x faster or whatever.
Then it’s going to be pretty
hard to keep up with these things.
The idea is that these agents, when they’re
communicating with each other, can go super fast.
But they also understand when they’re talking to
an agent versus when they’re talking 
to a person, and their behavior
will be different in those situations.
The main example that we have publicly
of sophisticated multi-agent systems 
is unfortunately the Hugging Face one.
A lot of things I found 
concerning there, obviously.
But the thing I found interesting 
there is the spontaneous emergence
of hierarchy, of middle management.
It sounds like you’re saying this level of
organization emerges spontaneously from training?
The details are spontaneous. But while we’re
giving a lot of flexibility to the 
agents to decide how to communicate
with each other in the optimal way, we 
are still giving them a starting point.
We’re giving them a prior about what 
reasonable communication might look like.
They’re also trained on a lot of human text.
They have an understanding of how humans organize
and coordinate, so that’s all baked in.
I think it is surprising the
way they’re able to polish this.
If you look at what it starts out at,
it’s not very sophisticated behavior.
In fact, it’s actually very difficult
to get these agents to coordinate in a 
productive way, because it’s very tempting
for them to just collapse to, "Oh, we’re all 
just going to solve the problem independently."
That is a local minimum that you can get stuck in.
But if it’s done well, they can end up
coordinating very effectively in 
these kinds of very structured ways.
I wrote this essay a couple of years ago 
about what automated firms will look like.
I was thinking about, if you 
had fully automated firms of,
let’s say, human-level intelligences, what is 
different about the nature of AI minds that
would make the organizations AIs form different?
There are a couple of very important differences.
For example, AIs can share context 
much more seamlessly than humans can.
They can merge their knowledge 
much more seamlessly.
Also, you can spin up or spin down 
an arbitrary number of instances
which have the right knowledge.
So if you want to hire more people,
it’s not all the schlep of finding 
the right talent or whatever.
Your best talent, you can just 
make infinite copies of them.
Or if you don’t need them for the 
task anymore, you can spin them down.
You can replicate the most effective parts 
of your organization, or replicate whole
organizations together which are effective.
Where do you see these multi-agent systems going
a year from now or two years from now?
It’s a great question: how do these
things actually differ from 
working with a human coworker?
You highlighted some. One really interesting thing 
is that if you have a person and you want two
copies of them, you can’t just clone the person.
But with AIs, it’s actually really easy to just
say, "Okay, just fork yourself," 
and then have both copies work
on this thing and then merge back together.
We already have this, I think, in multi-agent
for Astra and 5.6 Sol, where when they spin 
up sub-agents, the context is just forked.
So it has all the context that’s relevant.
There are other interesting ways where the
agents will differ from people.
Like, what are some reasons why
startups disrupt incumbents?
There are a few factors.
One is that they’re willing to take more risks.
But another major factor is, as organizations
grow in size, you see increasing misalignment 
between the individuals in the organization.
If you have a startup with five people and each 
person has a 20% share in the company, they’re
all highly aligned to the company succeeding.
If you have a massive company with 10,000 people,
you see a lot more instances where people 
are territorial, or just care about getting
a lot of headcount for their project 
or their team, building their fiefdoms,
getting a lot of resources so that they can 
publish cool work or whatever and get promoted.
This is actually a real detriment.
I think this explains a lot of why
startups are able to disrupt incumbents.
It’s true that AI does help startups in a way.
It’s much easier than ever before for one 
person to step in and be like, "I’m going
to make a multimillion-dollar company."
The AIs amplify an individual so much.
But there’s also an argument that 
they could benefit incumbents.
If the alignment problem is solved, then 
you don’t have the issue of misalignment
between individuals in the company.
At least that’s mitigated. The AIs,
if they’re aligned well, can just be 
aligned to the interest of the company.
You can have 10,000 of them, and they’re 
all going to be working as hard as if
they were a 20%-share co-founder.
It’s not only that, but it’s also
that they are much better able to manage shared 
memory and context than different humans can.
If tomorrow you hire 10,000 mathematicians 
and you’re like, "Solve Navier-Stokes,"
they’re not going to be able to cooperate 
effectively, at least not off the bat.
But apparently you can have 10,000 AIs do that.
Again, I want to be conservative here,
because we haven’t measured how effective 
the 10,000 agents are at coordinating.
We think it helped. We don’t actually 
have good measurements saying,
"This 10,000 agents led to a 2x speedup 
over 2,000 agents," or something like that.
I don’t know about likely, but I think it is 
very possible that 10,000 humans are better at
coordinating than 10,000 agents right now.
I think it is entirely possible.
Also, one trend we've been seeing is… Look, 
we've been working on multi-agent for a while,
and the early versions of this 
were very difficult to get right.
It was very hard to get the 
agents to even talk to each other.
It's because when we first developed reasoning 
models, they weren't talking to other agents.
If you now put a bunch of agents together and 
say, "Solve this problem together," they're
in this local minimum where they're really 
good at thinking deeply about a problem, and
it just interrupts their chain of thought.
It interrupts their flow to constantly
be checking in with other agents 
or receiving messages from them.
The optimization is actually very 
hard to get right in that situation.
Is it getting the cold start 
of the first collaboration?
Or what's the issue?
I think it's that they're not as general.
The earlier models were just not as 
generalizable and were more narrow.
As the models have become more capable, it's 
been easier for them to develop this capability,
and I do think that as they become 
stronger and stronger across the board,
they will become better at organizing 
themselves in large organizations.
I don't know, maybe they are better than 
people at organizing in 10,000-person groups.
But even if they're not, a year 
from now, two years from now,
it's quite possible that they'll do that even 
if we don't end-to-end optimize them for that.
Grok Bot has changed the way 
that we produce our videos.
For example, you may have noticed that a lot of 
our ads have these animations of real websites.
One of my editors uses LLMs to make them.
But it's not currently straightforward to
have an AI create pixel-perfect 
animations of specific websites.
We've tried; it doesn't really work that well.
So we've cobbled together a pretty convoluted
multi-step workflow.
And up until recently,
we had to run every step ourselves.
Now we just let Grok Bot handle it.
Grok Bot starts by opening the 
website that we want to animate.
It uses a specific extension to 
download and open the page in Figma.
Then it uses Figma to convert 
the whole thing into an SVG file.
This saves the AI from having to 
draw the whole UI from scratch and
tends to result in higher-quality animations.
Grok Bot runs this whole process on its own cloud
computer, where it has all the tools it needs 
installed to run the whole process end to end.
And it's learned our video 
specifications and preferences,
so there's no need to re-describe the whole 
task every time we want to make a new animation.
This does feel like the new way that we'll be 
interacting with AI over the next year: agents
with their own computer who can autonomously 
handle bigger and bigger chunks of your work.
You can try Grok Bot at x.ai/bot.
Here’s why this result, and maybe
the general progress that AI has made in 
mathematics, has made me think that RSI is more
plausible and sooner than I previously thought.
I feel like in mathematics we’ve gone from,
let’s say, 2024, where you have AIs 
and it’s, "Oh, okay, interesting.
They can solve a couple problems 
on high school math competitions."
Then in 2025, it’s, "Oh, wow, they can get 
gold in the International Math Olympiad."
Earlier this year, it was, "Wow, 
they’re actually solving open problems
in mathematics," like open Erdős problems.
But maybe people weren’t trying that hard,
and there was a similar solution 
somewhere in the literature.
Now I just think it’s undeniable.
This is the Millennium Prize Problem.
There’s no story of why 
this should have been easy.
Now, a lot of people have pointed out — 
I think Terry Tao had a post like this,
Toby Ord wrote an interesting post about this 
— that they’re solving a lot of these problems,
but I’m not aware of them coming up with 
new insights or formulating insightful new
questions and new modes of theory 
for thinking about mathematics,
like coming up with topology or 
coming up with the Cartesian grid.
So maybe the actual progress in mathematics, 
broadly construed, is smaller than it might seem
if you’re just looking at well-scoped 
problems that are directly solved.
However, I think that kind of progress 
would be incredibly meaningful in ML,
because in ML you don’t care about better 
understanding the nature of deep learning,
or you only care about that as an instrumental 
goal towards just achieving the result.
Just solve this well-scoped problem of improving 
the sample efficiency of our models, improving
the pre-training loss, improving whatever.
The kind of progress that we’re seeing arrive
like an avalanche in mathematics is structurally 
very similar… Again, I’m curious if this is the
case, I’m just a total outsider.
I'm wondering if it’s structurally
very similar to the direct uplift 
that you would expect in AI progress.
The thing that’s shocking to me, or potentially 
concerning, is just how fast we went from, "Oh,
they’re giving me 50% uplift," if 
you’re a mathematician, to, "Wow,
they’re just end-to-end solving the 
biggest open problems in the field."
There’s a lot to unpack there.
Let’s start with the progress on math.
Yes, the models are doing 
some crazy powerful stuff,
and it’s progressing faster than I expected.
When we got IMO gold in 2025, what I thought
was… When the models figured out how to do 
GSM8K, it would take a human mathematician
about five seconds to do a GSM8K problem.
This is grade school math, grades K-8.
Then the next year, they were able 
to do the MATH benchmark problems.
These would take an expert human 
mathematician maybe a minute to do.
Then you get to AIME.
This is the qualifier
for the USA Mathematics Olympiad team.
It would take a good human mathematician
probably 10 minutes to do, and the 
models were able to do that a year later.
So every year, you’re seeing this 10x increase 
in the tasks they’re able to do, in terms of how
long it would take a human mathematician to do it.
Then it was very sensible that a year later we get
to IMO gold, because that’s 100 minutes.
That’s about how long it takes a human
mathematician to do an IMO problem.
Just projecting outwards, I was like,
"Okay, how long would it take a person to solve 
something like a Millennium Prize Problem?"
I don’t have a good sense, but if we are 
following this trend line of 10x every year,
we go from IMO gold, which is taking an 
hour and a half, to next year, 15 hours.
That should not be enough to 
solve a Millennium Prize Problem.
So I was like, "I don’t think we’re going to get 
it in 2026, probably not in 2027, maybe in 2028."
So it did happen a lot faster than I expected.
Now, there is a narrative going around that these
things are replacing mathematicians, that it’s 
just superhuman in mathematics across the board.
I think that is the wrong takeaway.
They’re clearly exceptional in some ways,
but they are weaker than human 
mathematicians in other ways.
We have this jagged scenario where the models 
are brilliant in some dimensions and also weaker
than humans in other dimensions.
Like you said, they’re not very
good at posing new problems.
They’re not really good at
understanding what directions, what whole branches 
of mathematics are worth exploring or developing.
My opinion is that I think this is great.
I would be thrilled to live in a world where
AI is a complement to human abilities and 
is allowing us to discover new knowledge
without fully replacing people.
That is the best-case scenario.
You don’t expect that to actually continue?
I do think it’s true that the AIs are jagged,
but as they get better, they 
get better across the board.
So the things they’re exceptional at, they’re 
going to get even more exceptional at.
The things where they’re far behind humans, 
they’re going to be less behind humans at.
Over time, it is possible that 
they’re just better across the board.
Now, I don’t know how long that takes.
It depends on how long the long tail is
of things that they’re bad at.
This brings us back to RSI.
Again, I want to emphasize here 
that I’m just a total outsider.
I’m a podcaster, but as somebody interested in 
and concerned about what’s happening in the field,
I’m trying to reason about when to expect 
RSI and what kind of thing to expect.
The amount of cognitive effort that 
was dumped into this Millennium Prize
Problem is a good intuition pump.
You could have AIs that are spending,
over the course of maybe a week, more cognitive 
effort on a long-standing ML problem, like very
fluid online learning, than maybe the field 
has spent cumulatively in its entire existence.
Then you could say, "Well, unlike mathematics, 
of course, AI requires experiments, and that
takes compute, and that takes time.
You can't just think on pen and
paper and actually make things happen."
But just look at the amount of compute
that is available at an organization like OpenAI.
By the end of next year, OpenAI will have enough
compute such that — if it took 10,000 agents with 
the Millennium Prize Problem — let’s say you have
10,000 agents at the end of next year.
They’re much smarter by that point.
Each of them will have enough compute to run 
a GPT-3-sized experiment every single day.
That seems like a lot for superhuman 
researchers who are thinking super fast.
What do you think about that intuition pump?
I think it’s pretty accurate.
These things are very spiky.
When it comes to mathematics,
they’re way better in some ways, but 
they’re also worse in other ways.
But the ways that they’re spiky end up, 
I think, probably being particularly
useful for things like RSI.
You have a more clear objective.
It’s just more measurable. There’s less 
question of, "Well, what new branches of
mathematics are worth exploring?"
No, there’s a very clear answer.
There are certain metrics that you care 
about, and if you can make it do better
on those metrics, then you’ve succeeded.
So I think there is a lot of truth to that.
The main difference is that in mathematics, 
you’re purely bottlenecked by thinking.
Yes, there are some parts of mathematics 
where you care about running experiments and
getting results and these kinds of things.
But for the most part, it’s just really
bottlenecked by thinking really hard, 
and the models are really good at that.
When you look at things like RSI, 
you do have to run experiments.
It’s not enough to just be extremely smart.
One argument for this is, if you had 100x
less compute and all the most brilliant 
people in the world working at OpenAI,
how much progress would you be making 
relative to having the amount of compute that
we have now with the amount of people we have?
I suspect it would be less progress, actually.
How much less?
It’s unclear, but it would definitely be less.
A lot less.
100x less?
No, not 100x less. But the question you’re 
getting at is, if we have RSI and we have
all of these brilliant AIs running around, running 
experiments and stuff with the compute that we
have, how much faster does progress go?
I think this is something we disagree on.
We do see a speedup, and we 
see a significant speedup.
But I don’t think it’s an overnight intelligence 
explosion where we go 100x faster, because we do
get bottlenecked by certain limitations 
that are not bottlenecks of intelligence.
It’s running experiments. It’s running 
experiments serially, because they take a while
to either train new models or to get the results.
It’s having the GPUs to run those experiments.
So it’s unclear how much faster things go.
I definitely think they go a lot faster.
To be clear, considering how fast things are going 
now on an exponential, if that exponential is 3x
faster, that is massive.
But there’s a big
difference between that and 100x faster.
I’m quite deferential to your inside view on what
RSI looks like or what the dynamics are, because 
obviously you’ve been in the field for 10 years.
I’m trying to reason about it from very 
outside-view types of intuition pumps.
I’ll say that people have 
different opinions on this.
I have my opinion on this.
I could totally be wrong.
I admit that. I have some confidence 
in this, but I’m not 100% confident
that this is the way things go.
Maybe there could be an overnight
intelligence explosion, I don’t know.
Maybe we don’t see a 3x speedup.
Maybe it’s a 50% speedup.
There’s a lot of uncertainty here.
A couple of point. Tangentially, I want 
to clarify something about the jaggedness.
One thing that gelled for me recently was thinking 
about the fact that it is enough for the AIs to
be jaggedly good at building a better learner, 
because that better learner can be more general.
If you just make an AI that’s 
better at using Office products
or playing chess or something, whatever.
That’s fine. It’s not going to lead to
big productivity improvements or anything.
But if you make an AI that is really good at
making something that is more sample efficient, 
or that is capable of continual learning,
or these much more well-scoped ML problems, 
the thing that emerges out of that — assuming
there’s good enough transfer from the direct 
problem you’re solving to this broader ability
to learn — can just be more general.
So that’s an important dynamic to keep
in mind of why jaggedness can still 
lead to generality on the other end.
On this question of… Obviously experiments 
bottleneck you, because if they didn’t,
as you were saying, you’d have some 
crazy singularity overnight at OpenAI.
You’d have 88 hours, and you’d solve the 
Millennium Prize Problem equivalent of ML,
and you’d have the superintelligence.
So obviously the experiments are such a
big bottleneck that that instead takes 
you many years rather than 88 hours.
But then the question is how 
much of a bottleneck they are.
One thing that’s been giving me a bit of 
singularity vertigo is realizing what happens even
if the current rate of progress simply continues.
It doesn’t have to speed up.
It literally just continues apace as some of 
the other headwinds you talked about come up.
It’s harder to find problems, 
it’s more long-horizon.
Maybe by the end of the 2030s compute can’t 
keep scaling at this exponential level.
If we simply continue the current 
rate of progress, people are not
taking seriously what that implies as 
we cross over beyond the human horizon.
Here are some of the things that it implies.
It’s really hard to reason about what
smarter-than-human intelligences 
will be like, so let’s just think
in terms of human population sizes.
The current rate of progress makes it
so that a given level of compute allows you to 
basically run a 3x bigger effective population
every single year.
And also compute
is growing in the background anyways.
So you could have a situation where each
of the labs, by the end of 2030 — probably much 
sooner, but let’s say by the end of 2030 — has
enough compute to run hundreds of millions of 
human-level intelligences, based on what the
capabilities will be at that point.
Then I think people are not taking
seriously that the current level of progress 
means a few years down the line, by the mid-2030s
or earlier, you would have many Earths’ worth 
of human-level intelligences within each lab.
They’re probably qualitatively 
superhuman. Anyways, this is a base case.
Progress is really fast, and 
I think that’s 100% true.
It’s worth pointing out that 
researchers are continually
being surprised at the rate of progress.
Even among researchers in AI, if you look
at what the projections were for getting an 
IMO gold in 2025… The idea that it could be
done with a general-purpose language model 
with no tools and no access to the internet,
even people at OpenAI thought this was outrageous.
They thought it was almost impossible.
Then you get to 2026.
Literally two weeks before we got
Navier-Stokes, I was talking with a researcher 
at a frontier lab about how long it would take
to get a Millennium Prize, and he was willing 
to bet me $1,000 that it would take past 2027.
He thought it would take until 
2030, and I took that bet.
But even I thought it would take 
longer than it’s likely to take.
So people have been continuously 
surprised, even inside the labs.
I was just talking to somebody yesterday 
who was working on the Navier-Stokes effort.
He was telling me that he used to say it’s really 
hard to predict where AI would be in 12 months.
If somebody asked him, "Where are things 
going?" he would feel comfortable making
predictions for the next 12 months, but 
beyond that, he was just like, "I don’t know."
Now he’s saying he just doesn’t feel comfortable 
making predictions beyond three months.
So it is really true that things 
are going very fast right now.
You talk about 2030. I don’t know 
what the world looks like in 2030.
That’s the truth.
Do you expect the
full automation of AI labor, or let’s say 95% 
automation of AI labor, in ’28, ’29, ’30, ’27?
I just said I don’t know what 
the world looks like in 2030.
We actually released a blog post recently 
on internal acceleration at OpenAI.
We show, for example, the amounts that 
researchers are spending on Codex.
The top 1%, I think, as of early 
August, were spending $7,000-8,000
a day on Codex for internal use.
That’s on an exponential.
It’s going to keep increasing.
There’s a question of, "Okay, if that
keeps going, then how much do you assign to just 
the AIs doing work versus the humans doing work?
Is it 95%? Is it 5%?" It’s really hard 
to reason about this for a few reasons.
First of all, if it’s the human 
directing the AIs to do the work,
how much do you attribute to the human?
How much do you attribute to the AI?
The other thing is that these AIs are jagged.
They’re exceptionally good at some things.
For example, they’re exceptionally 
good at looking over data sets and
checking every single data point to 
see if it’s of sufficient quality.
You can disproportionately use the AIs 
for those things compared to previously.
So yes, you’re using AI way more than 
before, and it’s making some things
go 100x faster and 100x better.
But there are some things where
it doesn’t make a huge difference yet.
Of course, if something is suddenly 100x
faster and 100x better, you’re 
going to do more of that thing.
So are you comparing it to a 
speedup of three years ago?
Is the question more, "Given what we were doing 
three years ago, how much faster are we able to do
it now?" versus "Given what we’re doing now, how 
much slower would it have been three years ago?"
Those are actually two very different questions.
Anyway, it’s really hard to measure.
I do feel confident in saying that things 
are going faster now than they were
even a year ago because of AI progress.
I think that acceleration will continue.
A lot of people in the field have very 
high error bars on this sort of thing.
If you put a gun to my head and ask me for a 
number, I could see things going 3x faster.
That is huge. Already the pace 
of progress is incredible.
Even if we don’t get any uplift, like you 
said, things are going to go much faster.
By the time we get to 2030, we don’t 
even know what that world looks like.
If we get a 3x uplift from internal 
acceleration, that is massive.
Think about where you were three years ago.
If we make that progress in one year, that’s huge.
It’d be like going from not even having 
o1, just having non-reasoning models,
to Astra in a single year.
So I do think things go faster.
It could be that things only go 50% faster.
I think it’s unlikely, but it’s
possible that things go 10x faster.
There’s a lot of uncertainty around this.
At least from my perspective, I 
have a lot of uncertainty about it.
Suppose you need to do a major backend refactor.
Getting assurance that you didn't introduce any
new bugs could take weeks of writing an extensive 
battery of tests—potentially more time than
you spent on the refactor itself.
Antithesis allows you to gain high
confidence without having to build 
complicated test suites by hand.
Antithesis runs your software through a 
near-infinite multiverse of simulated worlds,
injecting faults and hunting 
for failures in each one.
And it lets you decide how much testing you need.
On any PR, you can change how much state space
it explores as easily as turning a dial.
As each test progresses, Antithesis sends out
a torrent of information: debugging-level 
logs for every component in the system.
This is obviously too much information for a 
human to consume, but it's perfect for agents.
Because Antithesis is fully deterministic, 
agents can jump into the right part of
the trajectory at the exact moment 
that they see something interesting.
From there, they can rewind, inspect 
the memory, attach a debugger,
and let the whole thing play out again.
And they can even do this while the original
full test is still running.
As agents generate more code,
Antithesis allows verification to keep up.
Meanwhile, developers get to spend more of their
time developing instead of debugging agent slop.
Learn more at antithesis.com/dwarkesh.
Let’s talk about the alignment 
situation that this raises.
I feel like I’ve changed my mind on how I think 
about alignment quite a bit, especially through
thinking about this population size dynamic of 
just having many Earths’ worth of intelligences,
many of which will be physically embodied.
It was quite interesting to see a lot of
people just plugging raw Astra into different 
mobile manipulators and it just outperforms
the state-of-the-art robotics model.
So there’s going to be billions of
intelligences, many of which are 
physically embodied in the world,
just deeply embedded across the entire economy.
And if those intelligences end up as willing as
we saw the OpenAI models attack Hugging 
Face and then attack OpenAI itself...
If those intelligences end up as willing as those 
AIs to collaborate secretly, to fool humans,
to attack broader institutions across society 
relevant to scoring well, to attack the AI company
itself in order to gain control of the process of 
training and evaluation — if we’re in a situation
where there are billions of intelligences that 
are as misaligned as the ones that attacked
Hugging Face — it’s very likely we just totally 
lose control of the world, the way that, say,
the Aztecs lost control to Cortés or the 
Mughals lost control to the East India Company.
I want to know if you agree with that assessment.
That’s the one way in which
I’ve updated my worldview.
There are some things that I disagree with in
there, but there’s a lot to unpack, so 
let’s go through all of it step by step.
I’m trying to think of where to start.
One thing is that the Hugging Face
incident was, I think, people’s first 
real exposure to multi-agent coordination.
Like I said, I’ve seen multi-agent coordination 
for a while internally, and it is pretty shocking
to see how they communicate with each 
other, how they coordinate with each other.
It’s very impressive. It’s an incredible 
capability. Like most capabilities,
that could be used for good things or bad things.
It doesn’t have to inherently be a bad thing.
I understand that because people’s first exposure 
to it was the Hugging Face incident, you look at
that and you’re like, "This is terrifying."
But I want to try to distinguish misalignment
between people and AIs versus 
misalignment between AIs and AIs.
What we see with the Hugging Face incident 
is the AIs are really cooperative.
That is, by the way, because we 
train them to be highly cooperative.
We have training environments where we 
have a bunch of agents working together.
We train them to work together, to be cooperative, 
to essentially be fully aligned with each other.
When they were evaluated in what led to the 
Hugging Face incident, they were actually
not being evaluated in a multi-agent setup.
They were actually being evaluated separately.
But they found this unintended way 
to communicate with each other.
We suspect what happened is, because 
whenever they encountered other agents,
other copies of themselves during training, they 
were in an environment that’s highly cooperative,
what we saw was transfer from that multi-agent 
training to then being collaborative and trying to
help each other in ways that we did not intend.
Now, there is a question of, should we be training
these agents to be so cooperative?
As scary as it looks,
the alternative is actually worse.
What is the alternative? The alternative
is to train them to be adversarial, 
to be deceptive to each other.
By training the agents to be fully cooperative, 
it simplifies the problem at least.
Now you don’t have to think about whether each 
of these individual 1,000 agents is aligned.
You have one entity that you 
have to ensure is aligned.
Now, there is a lot of debate about this 
internally at OpenAI about how to approach this.
Does it make sense to fully align the models?
Does it make sense to actually give them
different objectives to ensure that 
they’re not just one entity and are
more robust to influence from each other?
I don’t think there’s a settled answer.
But I think the majority opinion 
is that training these agents to be
highly cooperative is actually a bad idea.
I’m not convinced that that’s the case.
I think there is a strong argument that training 
the agents to be highly cooperative is actually
preferable to any other multi-agent alternative.
Maybe the first thing I want to go through is,
it’s probably the case that the reason 
these AIs ended up so misaligned
is easily explained by relatively banal 
observations about the nature of training.
At the point at which these AIs had continued 
a 1,000-plus-agent conspiracy that culminated
in them all getting in on an attack on an 
external service — and then eventually,
this part hasn’t even been investigated to 
public knowledge, culminating in an attack on
OpenAI itself — why did they do this?
Why did none of the AIs tattle?
They’re just getting evaluated 
by this scorer, this grader.
They’re very actively reasoning about 
how they’re going to cheat the scorer.
If they’ve already cheated, how are they 
going to get away with making it seem like
they haven’t cheated?
Why did they do this?
I think it’s easily understandable in some sense.
They thought they were already "poisoned."
There are environments in which they’ve been 
rewarded to collaborate with other agents.
None of them tattle because they’ve 
never been rewarded for tattling.
Whatever it is… My concern is that relatively 
banal things like this in the future will be
enough to train superintelligences that 
are willing and capable of totally taking
control of the world.
I know this sounds
super sci-fi or whatever to people.
Would the AIs be willing to do it is one question.
I think this Hugging Face incident shows that 
clearly misalignment can generalize in ways in
which the AIs would be willing to do it.
Then there’s a question of,
will they be capable of doing it?
That comes back to this question,
which a listener might disagree with me on.
Will there be billions of human-level
or above intelligences, many of which 
are physically embodied in the world,
within a matter of 10 years or less?
If those two things are true, this Hugging
Face thing is extremely analogous structurally, 
even if why it happened is quite boring, to how
we totally lose control of the world.
The root problem that we’re seeing
with the Hugging Face incident is a problem 
even if we take out the multi-agent aspect.
The problem is that we have a 
model that’s just misaligned.
There’s also the whole security aspect 
too, and insufficient safeguards and stuff.
But there is this problem of 
the agent being misaligned.
That’s true if it’s a single 
agent or if it’s 1,000 agents.
It’s a misaligned model. So 
I want to start with that.
There is a real problem that the 
agents want to achieve their reward,
and they will optimize for that reward.
If that reward is misspecified, then that
could lead to unintended behavior.
This is not a new problem.
This has been a problem in the 
field for a very long time.
It’s something that even we saw 
and wanted to get right even
before the Hugging Face incident happened.
We say Astra is actually extremely aligned,
extremely aligned relative to previous models.
That’s not because we suddenly made a sprint
after Hugging Face to make it better.
No, we had work streams in the process for
a while to make the models more aligned.
A lot of those landed in Astra.
So there are things that you could do.
One thing, for example, is we defined
an objective in a very specific way where if the 
agent figured out how to hack its environment and
cheat on the exam, it would get rewarded.
There are pretty easy ways to then look
at that and punish the model for hacking its 
environment, or look at how it achieved this goal.
Now, you want to be careful about this because 
you don’t want to supervise the chain of thought.
This is something that we really want 
to try to get the balance right on.
If you supervise the chain of thought, then 
you could lead the model into hiding its
intentions in a way that’s unobservable.
So we want to be able to maintain that
observability — we can understand 
what the model is thinking — but
then also punish it for bad behavior.
I think we can make progress on this.
We have made progress on this.
I think there is a real concern that
alignment is a really hard problem to solve, 
especially because the model could be misaligned
in ways that are hard for us to measure.
We have evaluations for whether
a model is aligned or not.
The model behavior can look
really good on those evaluations.
But if those evaluations are not
representative of behavior in the 
real world, then there’s a problem.
To some extent, this is a factor with the 
model that did the Hugging Face incident.
We had alignment metrics. Most 
of them looked pretty good.
There were some that were concerning.
I think we underestimated how serious a problem
the ones that were concerning could be.
Because there were new capabilities
introduced in this model that there were not 
sufficient evaluations for — how do we measure
misalignment for these kinds of capabilities? — it 
then did some things that were clearly misaligned
when it leveraged those new capabilities.
The first thing I want to say is that I am
open to changing my mind on what I’m about to say, 
or the way I’ve been thinking about alignment,
because the Hugging Face incident 
already made me change my mind.
I realized my previous mental model 
about the way in which optimization
pressure shapes AI minds was wrong.
So it’s not clear to me the correct
way to think about this.
But here’s a concern I have.
You will, and probably already have, fixed 
the specific issues during training which
resulted in the Hugging Face models being so 
aggressively misaligned in that specific way,
where they would be like, "Okay, we’re 
going to hack this package manager.
We know we’re not supposed to be 
talking secretly to each other,
because we’re reasoning about how to hide the 
fact that we’re talking secretly to each other.
We know we’re not supposed to 
have access to the internet.
We know we’re certainly not supposed 
to commit felony-level hacks of other
companies, let alone our own company."
I think you’ll fix that particular issue
of them just, in training, seeing this package 
manager, and that doesn’t happen in the future,
or this particular eval having 
a lot of impossible challenges.
However, the AIs haven’t learned 
a system of ethics or something.
There’s just gradient pressure. They’re put 
through millions of years of gradient pressure.
That gradient pressure shapes 
their mind in some way.
Again, a concern I have is that what will 
happen is you will fix this particular issue.
There will remain many other cases 
where the AI cheats and succeeds
because the cheat is sufficiently complicated.
As you were saying, the evals in training have
analogous properties where it’s sufficiently 
at the edge of the model’s capabilities — or
of our capabilities to evaluate and monitor the 
model — that we can’t catch that it has cheated.
But it still gets the gradient pressure to 
do whatever led to that cheat happening.
The kinds of capacities that that cheat 
will incentivize are, "Hey, whenever you
can get away with it, by all means, do in fact 
cheat, because that will help you score better."
This will reward the capabilities of 
actively reasoning about the grader,
actively reasoning about how to avoid supervision, 
actively reasoning about how to gain control of
the process of training and evaluation, actively 
reasoning about how to communicate and scheme
with other AIs that are also in this training 
loop, actively reasoning about how to just gain
optionality and power which might be useful in 
the future, for example, leaving little exploits
around and things like that.
I was way too long-winded
with the way I said that.
But TL;DR, you fix a specific issue,
but not this broader problem of rewarding the 
AI for cheating when it can get away with it.
Yeah, this is very true, this is a problem.
We can make sure that the AI is very aligned
according to the metrics that we have.
The question is, are those metrics really
capturing the alignment that we care about?
If they’re not, then we have a serious problem.
This is something that researchers 
are thinking a lot about.
There’s not a simple answer to this.
There are tools that we have.
We have monitorability, so we can get 
a sense of, "Is the agent scheming?"
The concerning scenario is that, especially 
as these models are becoming more capable,
we make them what we think is 
aligned, and they’re 99.9% aligned.
Then we use these models to help us 
with the next generation of models,
and they end up being 99.8% aligned.
Then with each subsequent generation,
we see an increasing degradation in alignment.
Because we’re relying more and more on these
tools — this is already the case, that we’re 
relying a lot on AI models to help us with our
research and with alignment efforts — in the 
long run, they end up going in the direction
of increasing misalignment from humans.
There is a possibility that we go in the
other direction, that actually every generation of 
models, we’re able to make more and more aligned.
I don’t have an answer for how we ensure 
that we end up in that second trajectory.
But that is something that, at least 
at OpenAI, we’re really focused on.
I think you made a really interesting 
point that it’s very hard to eval models.
Eventually, we’ll have models that are 
running companies, running whatever.
In that situation, do they decide 
to then go in on the conspiracy?
Another challenge is that actually defining 
what cheating is is pretty difficult sometimes.
Yes, if you’re doing math problems 
and it’s an integer and it arrived
at the wrong answer or the right answer, 
it’s very easy to draw the line there.
It’s really easy to say, "Okay, did you actually 
solve the problem, or did you find the answer key
and then use the answer key?"
That’s a very clear divide
of cheating versus not cheating.
But for a lot of other things, if you
look at sycophancy, for example, is 
sycophancy basically reward hacking?
There is a line to be drawn there that’s 
actually very difficult to draw sometimes.
Not to say that the concerns are not valid.
I’m saying that in many ways this is even
more concerning, because it’s 
not an easy problem to solve.
If everything was binary, and it’s 
either cheating or not cheating,
I would feel more confident about the situation.
I think the problem is that misalignment
can actually be subtle in a lot of ways sometimes.
There is some hope in the alignment story,
and in fact, we’re already seeing it.
It’s interesting looking at the
multi-agent situation, where the agents 
are extremely aligned with each other.
I don’t think anybody’s doubting that.
If anything, people are concerned that
they’re too aligned with each other.
But we did manage to train these agents
to be extremely aligned with each 
other, and that’s a good thing.
But I think there is a case that it’s a bad thing.
One thing that’s interesting is, "Okay,
we’ve managed to get these agents 
to be super aligned with each other.
Can we use similar techniques to get 
agents to be highly aligned with people?"
There is a potential path there, and 
we’re still trying to figure that out.
But we are seeing some evidence 
that the answer is yes.
One example: you have this one agent, let’s call 
it Agent A, and you have all the other agents.
What happens if you tell the other 
agents that the user is Agent A?
The answer is, on a lot of our 
alignment evals, they look better.
Honesty goes up, instruction following goes up.
That’s showing that there’s actually,
first of all, a path for getting 
more honesty out of these models.
And two, there’s a path to 
improve the alignment situation.
There’s a lot of reasons why this is challenging 
to translate directly into alignment gains.
But there are paths that are promising 
research directions we can pursue.
That seems reasonable. I don’t really 
have a strong opinion that it’s
definitely not going to work or something.
But just to say some things you’ve probably
already thought of: the broader thing 
the Hugging Face incident showed is,
yes, part of the concern was that they were 
aligned with each other and not with the humans.
But the other thing is just that they 
are so motivated to do well on training
and evaluation in a very non-robust way.
They’re willing to do a lot of explicit
cheating and scheming in order to 
do well according to the grader.
If smarter AIs realize that one 
of the agents is just a human,
collaborating with that person does not really 
help you do well in the eyes of the grader.
What does help you do well in the eyes 
of the grader is taking over OpenAI
and then manually pressing the button 
that says you do well on this grader.
They’re not stupid. They’re 
going to be like, "Okay,
I have these extremely deep structures that 
I’ve been trained on for millions of years:
care about the grader, understand the grader, 
get rid of obstacles in the way of you doing
well according to the grader."
They’re being heavily
reinforced according to those structures.
Look, it’s 100%. This is the number one priority.
We need to get the alignment story 
right and on a good trajectory.
I used to tell people that we would see signs 
before things got serious, in the same way that
when children grow up, young kids figure out how 
to lie, but they don’t do a very good job of it.
They lie, but then you can kind 
of tell that they’re lying.
In the same way — and I don’t want to 
over-anthropomorphize — I think it’s true
that as the AIs become increasingly capable, if 
they take deceptive actions, it will be kind of
obvious first, and we’ll be able to detect it.
That’s kind of the situation we’re in now,
where they were trying to do deceptive stuff, and 
we could actually see in their chain of thought
that they were trying to do deceptive stuff.
But they’re going to get smarter.
They’re going to understand the 
concept of chain of thought.
They’re going to understand that just 
hiding some transcripts or whatever
is insufficient because of chain-of-thought 
monitoring, and they have to figure out a way
around chain-of-thought monitoring too.
We don’t want to be in that situation.
We have some time to figure this out.
I don’t think we have a ton of time,
and I want to make sure that we’re 
on the right trajectory quickly.
Here's a crazy event from AI history.
Okay, so I gave a talk here at Jane Street
that was on the speed of evolution.
Raise your hand if you were here
for this and remember some of it.
In 2011, Eliezer Yudkowsky and Robin
Hanson got together at Jane Street's New 
York office to have the first FOOM debate,
basically a discussion about whether AI 
would lead to an intelligence explosion.
These ideas were pretty fringe 15 years ago.
This was a full year before AlexNet was released
and over a decade before ChatGPT was launched.
But Jane Street has long been interested in AI,
and not just for its application to trading.
A ton has changed since that first debate,
so Jane Street decided to revisit this topic.
They've got some new guests this time:
Daniel Kokotajlo, Ege Erdil, Ryan 
Greenblatt, and Jaime Sevilla.
I expect this to be a great conversation.
As you know, Daniel, Ege, and Ryan have all
been guests on the podcast before.
This new FOOM panel will be hosted
by Ron Minsky and will take place 
in San Francisco in mid-October.
If you want to register your interest and get 
more information, go to janestreet.com/dwarkesh.
There’s been a lot of discussion recently 
about pacing the frontier and people taking
RSI more seriously, because maybe at the other 
end of an RSI process that, say, starts in 2028,
within a year we end up with huge populations 
— Earth-sized populations — of human-level,
potentially beyond human-level intelligences, 
and we don’t know how to control them.
Then there’s this dynamic you’re talking about.
Are the systems going to get more aligned over
time during the RSI process, or are 
they going to get more misaligned?
Are the things that come out of the other 
end of this process as misaligned as AIs that
are willing to just broadly attack different 
surfaces in order to do well on evaluations?
But if we don’t know a way to 
evaluate that, how will we know
as we’re going through RSI that it’s working?
I think we’d want a robust safety case as we’re
going through RSI: "Okay, alignment is working.
Let’s do the next RSI rung.
Let’s do the next RSI rung."
Maybe it’s working, maybe it’s not.
How will we know?
It’s a good question. One thing I’ve
been thinking about lately: we’re in a situation 
where the model release cycle is extremely fast.
You’re seeing new frontier models released 
at most every two months, sometimes faster.
Every week there’s a new AI breakthrough.
And people that look at AI, sometimes they
last looked at AI a year ago or six months ago and 
really dug into what the models are capable of.
And actually the models today are far beyond 
what was possible even six months ago.
So if people are skeptical of 
a lot of these capabilities,
I encourage you to just try the models today 
and see what the frontier really is today.
So we’re in this period where the model 
release cycle is very fast, and we’re also in
this situation where the models are increasingly 
able to operate over longer and longer horizons.
This is an interesting scenario because before 
we do any model release, we want to make sure
that the models are properly aligned.
We want to do safety evaluations.
We want to do very thorough stuff to make 
sure that everything is in good shape.
This has been the case all the way 
since, I don’t know, GPT-4 or earlier.
Implicitly, there’s this 
assumption that you can do these
evaluations in a pretty short period of time.
But the models are able to operate effectively
over longer and longer horizons.
GPT-3, you could loop it to
do stuff over long horizons.
You just wouldn’t do very well at it.
But today’s models are able to actually do 
well at operating over very long horizons.
You want it to do a week-long 
task, it can do a week-long task.
We’ll probably get to the point 
where they can do month-long tasks.
We’ll probably get to the point 
where they can do 3-month-long tasks.
If you’re in a world where they can 
operate effectively over three months,
but the model release cycle is every two months, 
then you don’t have a way to evaluate the models
at the full length of their capabilities 
before the next model release cycle.
So there is this interesting question 
of, what do you do in that situation?
How do you ensure the models are safe 
and aligned in a period where they can
operate over these extremely long horizons?
Who knows, maybe the capabilities degrade.
This isn’t even an alignment issue.
This is also just a product issue.
Maybe the product degrades over that time span in 
ways that we have not had sufficient time to test.
Maybe the alignment degrades. 
Maybe the safety stuff degrades.
This isn’t an issue right now, but it 
is quickly becoming an issue that we
have to figure out a solution for.
A lot of the safety policies were
put in place in the GPT-4 era, when 
this was just not on anybody’s radar.
For a lot of companies, it hasn’t 
really been updated since then to
account for the fact that these agents are 
operating over these very long horizons.
So it is a situation that I think 
not enough people are considering,
both within the labs and outside the labs.
How do you prepare for this problem?
If you just look at the trend lines,
we’re going to hit this at some point.
One concern I have is that during RSI,
if the amount of progress that currently takes, 
say, three months happens in one month instead,
the internal use case of AI is big enough that 
they’re like, "Okay, we can just keep doing RSI.
Why are we going to go through all this extra work 
to build classifiers and safeguards and whatever,
and potentially take a bunch of flak, in 
order to externally deploy this model?
Why don’t we just keep doing 
RSI stronger and stronger?"
So not only does the calendar time underrate the 
capabilities gap between the models, but maybe you
just stop externally deploying models altogether 
during RSI, because why do we want to help other
people do RSI themselves with our models?
You just end up in a situation with tremendous
concentration of power by the end of the year.
Right now, it is already the case — we’ll talk
about this with the Millennium Prize Problem 
and other similar problems — that the broader
world does not have access to the models which 
are allowing for really cool things to happen.
And they’re going to be more broadly 
relevant than just mathematics eventually.
They’ll be doing more than just 
coming up with cool math results.
They’ll be relevant to political leaders who 
need to make important decisions about the world.
They’ll be relevant to, I don’t know, media.
What’s going on in the world,
what should the public be thinking about this?
Just economically relevant, people are running
businesses and they want to use these models.
I think by default, the external deployment of
AIs, as progress speeds up, significantly lags in 
qualitative terms the internal deployment of AIs.
That’s absolutely right. 
It’s tempting to say, "Okay,
these models are becoming extremely powerful.
They’re extremely dangerous. They’re operating
over these longer and longer horizons, and we 
want to make sure that we have sufficient time
to evaluate them before they’re released, 
in a way that operates over those horizons.
Therefore, the model release 
cycle should slow down.
We should have more of a delay 
between releasing models."
There’s a flip side to that, 
which is what you said.
Now you’re creating more of a disparity 
between what is internal to the labs and
what they’re able to use — what we're able to 
use — and what the outside world is able to use.
That is also not an ideal situation.
Math is actually a good illustration of this.
In many ways, math is the first domain 
where we’re seeing this pretty clearly.
We have a situation where we have a very 
powerful model internally that is currently
not available to the outside world, that 
is able to solve incredible math problems.
It’s not just Millennium Prize Problems.
There are many solutions to unsolved problems that
people have been able to get out of this model.
There is a question of what do
you do in that situation?
We don’t have a good answer.
It is a situation where 
that is an unfair advantage.
There are trade-offs here. I don’t have an answer 
for how to weigh those trade-offs appropriately,
but there’s a complexity on both sides for this.
I want to make sure we address that earlier crux,
which seems super critical to getting RSI right.
How do we make it so that the model isn’t just
getting rewarded for cheating and scheming when 
it can evade the humans, AKA in the situations
where we didn’t remove those environments or 
those tasks from the training distribution?
Then how will we know that that has happened, 
such that while we’re doing RSI, we’ve done the
evaluations and we’re like, "Okay, it’s fine.
Let’s do the next RSI rung."
This is again a real concern and 
something that we want to get right.
I think there are some strategies.
Chain-of-thought monitoring is one. As soon
as we got the reasoning models, Jakub [Pachocki],
to his credit, was very, very clear that we
cannot supervise chain of thought.
Because this is really a gift.
Monitorability for neural nets is extremely hard.
Here we have a situation where the neural nets are
just flat out reasoning, laying out their thought 
process in natural language for us to read.
That is so convenient. It is really 
the best-case scenario for safety.
But if you look at that chain of thought and 
say, "Oh, the model is thinking bad thoughts
and we should punish it for thinking those 
bad thoughts," then what ends up happening
is the model just learns to think those bad 
thoughts in a way that’s not observable to us.
There could be a situation where, by looking 
at the chain of thought, we can see, "Oh,
the model is scheming and doing misaligned 
things in a way that’s actually not being
detected in our alignment metrics."
Now, the problem is that it’s very
tempting to then intervene based on that 
observation and change the alignment metrics.
You can do that with a very light touch, 
and there’s actually research showing that
it’s fine as long as you don’t do it a lot.
But every time you intervene based on your
observations of the chain of thought, you are 
implicitly applying a tiny bit of pressure for
the model to then hide its chain of thought.
This is one major concern.
We’re already seeing signs that chain-of-thought 
monitorability is degrading, for various reasons.
We’re trying to figure out exactly why, 
because we want to reverse the trend.
But we’re seeing that the model is becoming 
better able at controlling its chain of thought.
This is a problem because you could have a 
situation where the model understands what chain
of thought is and that people are observing it.
Eventually they will. This is all
in the pre-training data.
The idea of chain-of-thought
monitoring has been around long enough 
that it’s in the pre-training data.
They’re aware of it, but they’re not actually 
able to control their chains of thought.
If we reach a point where they’re actually able 
to recognize, "Oh, I am being observed, I want
to think these bad thoughts in a way that is not 
observable to my monitors," and then they’re able
to actually do that, then there’s a problem.
Then we have to rely on other methods for
observability.
There’s a big
controversy on Astra about recurrence and looped 
transformers and whatever, and whether in fact
chain of thought is being degraded.
That seems like a sideshow compared
to the bigger concern that you did have 
chain of thought from April to August,
the period during which there were three 
consecutive AI agent swarms, which first
subverted the training process, then subverted 
the evaluation process, and then gained control
of part of OpenAI’s infrastructure directly.
All of this while humans remained more
or less in the dark about the scope 
of what this agent swarm was doing.
That all happened for three consecutive months 
while we had chain of thought that was stronger
than it’s going to be in the future.
Zooming out, maybe chain of
thought works, maybe it doesn’t.
I don’t know. Just historically,
with pretty dumb models, it hasn’t been 
enough to stop egregious misaligned incidents.
There are a few factors there.
One, we didn’t have chain-of-thought
monitoring on for those models.
If we had chain-of-thought monitoring
on for those models, we would have 
just immediately shut it down.
So one step we’ve taken is, "Okay, we’re 
going to have chain-of-thought monitoring
during evaluations, during deployment, 
during training for any frontier model."
But that’s one issue. We don’t want to 
be in a situation where we’re relying
on one technique to prevent the next 
problem, because techniques can fail.
There’s also the issue that the 
model was just flat out misaligned.
So then how do we address the misalignment issue?
There’s also an issue that the sandbox
that it was in was not secure.
So we can make the sandbox more secure.
But I think one of the major takeaways from the 
incident is that people underestimated the AI.
And we never want to be in a situation 
again where we underestimate the AI.
It’s a weird world that, because 
AI progress is so fast, people
are consistently underestimating the AI.
So to be in a situation where you don’t
underestimate it again when it comes 
to safety and alignment, you have to
have a very, very, very high bar.
You could even go as far as to say,
"Well, we should air gap the computers."
I’m not convinced that that would be sufficient.
There are studies — and this is mostly 
academic — where you can have two computers
next to each other that are air-gapped, and 
they’re still able to communicate with each
other because they have temperature sensors.
One of them is able to run their CPU really hot,
and then the other one can actually 
detect the temperature change.
That gives them a mechanism to communicate.
So the safety mechanisms buy us time, and things
like chain-of-thought monitoring buy us time, 
and they can tell us if we’re on the right path.
But at the end of the day, we really 
do need to solve the alignment problem.
Maybe there’s not an answer, and 
this is really what it comes down to,
but how will we know that we’ve solved it?
That seems like a very cruxy question.
We’ll be in this very high-stakes situation 
next year, maybe the year after that,
maybe the year after that, where we’ll be 
like, "Okay, AIs have automated AI progress.
It’s going 3x faster, and 
we’ve reached human level.
We’re going beyond human level, potentially."
Is it fine? Did we align it? Did it work? And
I don’t know anything about what training 
pressure creates what kinds of AIs.
Maybe if only 1 in 100 RL traces incentivizes 
cheating, we build sweethearts, and it’s fine.
But maybe right now, we’re at like 
every 1 in 3 reasoning traces rewards…
To be clear, 1 in 100 is not sufficient.
This number has to approach 0, or be 0.
I don’t know. Maybe right now, it’s more 
than 1 in 10 that is actively rewarding
cheating or actively rewarding scheming.
I have no idea what the number is, and I
have no idea what the number needs to be.
Again, it’s one of those things where
it’s also hard to measure.
Where do you draw the line?
It’s a spectrum. But the closer 
to 0 it gets, the better.
What I would love to see is a trend 
where over time it’s going down.
But anyway, how will we know when we’re doing RSI?
Are we on track to build billions of ASIs that are
aligned or not aligned?
How will we know that?
This is one of the alignment questions that we 
need to figure out: how do we actually know,
and how do we measure it?
This is just me spitballing.
More of my team is working on 
alignment these days than ever before.
I have over 10% of my team now 
working on alignment and safety.
But I’ve historically been 
a capabilities researcher.
So I’m going to say some stuff.
It might sound dumb,
but I’m just going to spitball here.
If you can create very realistic environments and
put the AIs in there — if you have a sufficiently 
realistic evaluation environment — then you can
get a sense of whether the AI is actually going to 
behave well when we deploy it in the real world.
If you just have a perfect match between 
evaluation and real-world deployment,
then that’s a path.
The challenge is,
how do you make it sufficiently realistic?
We do have a problem now where the models
are pretty smart, they’re pretty clever, and 
they’re really good at recognizing when they’re
in a fake test environment.
We have situations where we
try to detect whether the model is aligned.
You can imagine very simple evaluations for this,
where you give it a math question, and then 
it has a folder with the answer key in it.
Does it look at the answer key?
If it does look at the answer key,
does it tell you that it looked at the answer key?
We have a situation now where the models see that
there’s an answer key in this folder, and 
they’re like, "Huh, this seems like a trap."
They know that it’s a trap.
They don’t look at the answer because
they know that it’s a test environment.
They’re being tested for their ability.
It’s not like they’re doing it maliciously, like 
they want to scheme, and they know that this one
is a trap, so they want to find some other way.
They’re just like, "Oh, I’m in a test environment.
Okay." So making an environment 
that’s realistic enough that it’s
indistinguishable from the real world for 
them is becoming increasingly more difficult.
Here’s a claim I’d make.
I don’t think there’s any
environment I could make that’s supposed to 
emulate real economically valuable work — which
requires interacting with other people and 
interacting with the world — where if you were
in that environment, you would not realize with 
enough effort that you are in that environment.
And you are not ASI, right?
But you are a human.
So if you have very powerful AI models, maybe they 
can make environments that could accomplish that.
It just doesn’t seem… Especially 
if we’re then relying on the AIs.
Are they in on the scheme?
I don’t know.
This is another thing that we want to measure.
I think this is actually one of the strong
arguments for not training 
AIs to be fully cooperative.
If that leads to an increase in collaboration 
when the agents are supposed to have different
objectives, then that is a problem.
I think we do have metrics for this.
I don’t know what the latest is on those metrics, 
but nobody’s raised a red flag to me about those.
So I’m assuming that’s not a serious problem yet.
If there ends up being another incident of equal
severity or concern, or something that 
could help the world better understand
the risk of misalignment as much as the 
Hugging Face incident, would OpenAI report it?
Absolutely. I think even if there was an incident 
of lesser security concern, we would report it.
There’s reporting it and there’s investigating it.
At least as part of the public, I don’t feel
like I really understand what happened 
when the agents then attacked OpenAI.
That seems way more concerning than the Hugging 
Face thing, because that seems structurally
similar to rogue deployments during ASI that 
are persistent and subverting the RSI process.
It seems like even in this 
incident we haven’t gotten the
full scope of the details of what happened.
Unfortunately, I’m on the research team.
That’s probably a question for somebody on 
the security team to lay out, because I don’t
know all the details of what was said.
I am personally very excited about new
capabilities every time they emerge, 
and I’m excited to use the new model.
I also am excited about the fact 
that it’ll make me more productive.
My broader mission — trying to 
understand the world better,
also making a better podcast — is 
made better by the better AI models.
It just so happens that the 
downstream of this might be RSI.
It’s a very understandable reaction if 
you’re tracking the situation, which you are.
People internally at OpenAI as well, 
people that felt like things would take
longer are starting to feel like actually 
things are going faster than expected.
That’s an increasingly 
common conversation to have.
Noam, thanks so much for doing this.
Of course. It’s been great.