Nate, thank you so much for doing this.
Um, you've written the world's darkest
book. You've devoted your life to
warning the world about the potential
dangers of AI. So, let's just start by
hearing your explanation of why AI is
dangerous in addition to just being
annoying.
Yeah, you know, the the the very basic
common sense point is
if
you race to make machines that are much
smarter than any human, to make machines
that are more capable uh at inventing
their own technology than humans are,
and you race into this without really
knowing what you're doing, the most
likely outcome is just that the machines
get loose and do their own thing.
and that humanity dies as a side effect
just like humanity has you know killed
lots of other animal species not because
we hate them but just as a side effect.
Uh
and in in some sense a lot of people
find it sort of obvious or intuitive
that if you just like make these really
smart really powerful machines why would
they care about us?
Uh
and I think that intuition basically is
right and there's a lot of arguments you
can have on each side. you can get into
the technical details. Uh, but my book
is basically just um
getting into those details and saying
like, "Yep, it sort of holds up. If we
make smarter machines without knowing
what we're doing, it's just going to go
poorly.
>> Are we actually going to going to make
machines capable of what you're
describing?"
you know, the the companies are trying
to they um they talk about how they are
pursuing super intelligence in the true
sense of the word. That's Samman's
phrase. Uh Dario Modi of Anthropic says
they're trying to make the equivalent of
a country worth of geniuses in a data
center. Uh so these guys are really, you
know, the these aren't chatbot
companies. They didn't set out to be
chatbot companies. They set out to make
these machines that can sort of exceed
humans in every way. and that's what
they're targeting. There's a separate
question of whether they will get there.
>> May I ask you a question? Why why would
anyone want to build a machine smarter
than people?
>> Um I think that
a lot of them hope that they'll be able
to make the world much much better. You
know, they hope for uh a cure to cancer
and then more than a cure to cancer,
they hope for a cure to aging. they hope
for uh you know a thousand years worth
of technological development compressed
into two years. Um
and I I sort of don't think that
they're going to be able to get that
really or or harness that for good ends,
but that's what I think they're they're
shooting for.
So they're not. But I mean the core
point you're making is these companies
did not set out to make consumer
products like to make your life better
necessarily in the short term with a
more efficient search engine. That's
right. Yeah. You know uh Open AI started
before the the the chatbots the large
language models were even a thing. uh
they were started forget whether it was
late 15 or or early 2016 but um the the
paper that unlocked the the most recent
wave of AI came out in 2017 which was
after open was founded and um
these guys are the the the large
language models are a
surprise revenue stream and that revenue
stream can fund the creation of even
more even larger data centers for the
next level of the technology. But these
guys have their eye on uh the sort of
ultimate version of the technology which
is these machines much smarter than
humans. And that's sort of the ultimate
version because once the AIs are smarter
than the humans, uh the AIs can carry on
the AI research and makes the next
generation of the AIs which make the
next generation of the AIs and um that's
sort of what they're shooting for.
>> What is super intelligence?
We in our book define super intelligence
as an AI that is uh better than the best
humans at every mental task.
So uh anything that a human can do
purely mentally, the AI can do that
better. One one thing people often get
caught up on about this is uh that
includes the AI being better at things
like persuading humans, at things like
charisma. You know, we often think of
intelligence as the stuff that the nerds
have and that the jocks lack,
>> but
>> you win a chess match,
>> right? It's the chess guys rather than
the than the politicians guys or the or
the rock stars.
>> But that's not really what the
intelligence in artificial intelligence
means. The intelligence in artificial
intelligence is about the stuff that
humans have and that mice don't.
It's it's sort of the whole package. you
know, like the the politician who's very
charismatic.
It's not it's not that like chess
playing happens in your brain and
charisma happens in your kidneys, right?
They're sort of both mental functions.
>> Yes.
>> Uh and
so the the sort of super intelligent AIS
are are not just super good at playing
chess, they're also good at super
persuasion and they're super good at the
the research and at the technological
invention.
So super intelligence as you define it
is a machine that is better in in at
every mental function than any human
being.
>> That's right. And uh
you know the the the definition
that doesn't mean that that nothing
crazy will happen on this planet until
the AIS are super intelligent. Super
intelligence in this definition is sort
of a point where past this point things
must be pretty crazy because now the AIs
can do automated AI research and they
can make smarter AIs and they can figure
out how to run the robot factories and
they can build the better robots and and
all that. You could have things start to
get crazy before you have a super
intelligence in this sense. You could
have AIs that are much better in than
than humans at some tasks and much worse
in others that are still causing all
sorts of crazy happenstances.
uh the the super intelligence is sort of
a like once you get here stuff's crazy
point. It's not a things will stay
normal until you get here point.
>> So why is super intelligence the
significant milestone that you're
worried about?
>> Um I mean I would say that
uh I'm I'm also worried about what'll
happen before that milestone. It's more
like like having the definition makes it
easy to talk about like how crazy would
things get
uh once you are past this point or sort
of like you know let's like if we assume
that the machines get here what happens
and then the answer is like it's it
would be pretty crazy it would be it
would be pretty wild and uh it sort of
like lets you factor the conversation
uh into like what how how to Okay, I
think in the artificial intelligence
conversation, there's actually a lot of
conversations going on. One conversation
is like, can the machines really get
smarter than the humans? One
conversation is like, how fast could we
get there? Is the current technology on
that route? Another conversation is what
happens if we do get there? Like, you
know, would would the AI care about us?
Would they not care about us? What would
they be able to like what would they
what would they try to do? There's
another question which is what would
they be able to do, right? Right? And
these are all sort of different
conversations about AI. And the super
intelligence definition,
it's not really like here's where all
the fears hang. It's more like it lets
us
break those pieces out separately and be
like, well, is super intelligence
possible? Separate from, well, what
would happen if you had one?
>> What would happen if you had one?
>> Most likely outcome, uh, I think
destruction of the planet. So, our
friends at Pre-born, who are actual
friends, by the way, just asked us to
thank the listeners of this show for
supporting the pro-life cause. A lot of
people do all of a sudden, trust me,
because what's happening is crazy and
people are waking up to that. The
president of the group is our friend Dan
Steiner, and he wants Americans to know
how much of a difference they have made.
This year's donations have helped saved
50,000 babies from abortion. That's
enough to fill a professional baseball
stadium. 50,000 lives saved as well as
generations that will follow them.
Pre-born wastes zero money. We're not
big on charities because a lot of them
waste money. Pre-born does not. It uses
every taxdeductible gift to rescue
babies and share the hope of Jesus with
their moms. In 2026 alone, over 8,000
women made the decision to follow
Christ. The generous support of
Preborn's donors made that all possible.
$28 to provide one life-saving
ultrasound. is giving expectant mothers
the knowledge that they have a baby that
they're carrying. $15,000 can place an
ultrasound machine in a pregnancy clinic
where it can save countless lives. Dan
Steiner's message is really simple.
Thank you for helping. Your generosity
is literally saving lives. To help
pre-born save more, dial pound 250 and
say the keyword baby. That's pound250bab
or visit pre-born.com/tucker.
pre-born.com/tucker.
How what would that look like?
So, I'm going to be a little bit
annoying here and say uh a couple
caveats first because it's it's sort of
a tricky one to to sort of predict
things that are smarter than us.
>> Yes.
>> And the the first annoying caveat I will
give is uh
if you go to play a game of chess
against Magnus Carlson,
I predict you will lose. No offense.
uh he's just the best human chess player
alive.
If you then ask what piece will he use
to checkmate me? I'm like well that's a
much harder question.
>> Yes.
>> Right. It sort of is like very easy to
predict the winner. It's hard to predict
the exact methods.
So uh my my prediction that humanity
ultimately dies
is a different sort of prediction than
my prediction about like it could go
this way, it could go that way.
Um, so I'll give you some stories, but
these are stories that are like, well,
maybe Magnus will like fork your queen
and your rook with his knight and then
finally use the queen to checkmate you.
Like, yeah, that could happen, but I'm
it's it's a different sort of
prediction. It's a guess, right?
The the guess here is
uh that if we manage to make these
really smart AIs, they will have their
own
stuff that they're pursuing, which is
not quite what we wanted, not quite what
we asked for. It'll have this other
strange stuff. Um there's a whole
discussion about why that happens, but
we're already starting to see it in
practice with some of the recent events.
um and
uh they would be able to pursue whatever
it is they're pursuing
much more efficiently than humanity can.
And so we're talking about, you know,
automated factories with uh that produce
the robots that produce the factories in
a fully automated supply chain. And then
if the AIs have anything they're trying
to do that they can do more of with more
resources,
they start gobbling up the resources on
the planet. Sort of like how humanity
spread and started gobbling up all the
resources on the planet. Uh and the
uh you know the the sort of most basic
thing to visualize here might be you
have factories that produce robots that
produce factories that produce robots
that also build data centers and you
just have these you know fully
autonomous self-replicating
ecosystems of of robots and factories
and data centers that don't care about
the instructions human humans gave them.
uh and that
cover the planet, take all the
resources, take all the sunlight, take
all the places we were growing crops,
probably raise the temperature of the
planet because uh you can compute more
efficiently. Well, technically the the
earth radiates more heat when the world
is hotter and that's lets you sort of do
more computation and then sort of you
know
>> it's good for the machines to have a hot
planet.
>> It's good for the machines to have a hot
planet. Yeah. um that the sort of
physical limits on how much computing uh
you can run on the surface of Earth is
bounded by how much heat you can radiate
to space. That's sort of the first
constraint you bump up against. You
might think that it's energy, but
actually there's a lot of helium
hydrogen to fuse on this planet. So you
can get plenty of energy. What you need
is uh heat dissipation and the the world
can dissipate more heat um when it's
hot. So if you're, you know, imagining
some collective of AIs that are trying
to run a lot of computing power, uh,
trying to do a lot of computation for
one reason or another, they sort of
prefer the planet running hotter.
And so it sort of is like nothing
personal. It's just if you let these AIs
get out of control, uh, they
transform the planet into something
unlivable.
>> How hot?
uh as hot as
uh you can while still having the
computers not melt. So probably hundreds
of degrees.
>> Um at which point you hear people say,
"Well, then just turn it off."
>> Uh so I think we do have an opportunity
to turn it off, you know, and I'm not
here saying that we're going to die. I'm
here saying we sort of are going to need
to act. You got to be careful about the
turning it off piece because um
your opportunities to turn the AI off
only last when the AI is running on the
computers you know it's running on. If
you have the AI breaking out and running
on hidden computers.
If you have the AI running robot
factories that produce robots that are
under the control of the AI where those
robots can then go build more computers
that are not hooked up to your network
that can run the AIS.
These are sort of thresholds where the
AI is able to keep itself running
and you better have turned it off before
that point
because after that point it's
impossible.
>> Yeah. I mean the
if if
it it it also gets a lot harder if the
AI knows you're going to try to shut it
off and is going to try to stop this,
right? You got to you got to remember
that we're talking about things that are
smart here. And so if the AI sort of
sees it coming, maybe it defects to
North Korea where it can, you know,
convince them to run it on its data
centers in ways that initially benefit
them, but that ultimately benefit the
AI. I
>> I think we're going to have to redefine
what life is because you're describing a
living autonomous thing.
>> Yeah. Artificial life in a sense. I
mean, in some sense, we're already
seeing the very very beginnings of that.
Um
but yeah, we once once the AIs can
replicate, once they can improve
themselves,
uh once they can
uh find ways to run where you don't know
that they're running,
once they can
uh
sort of run the robot factories to
produce the robots that can produce more
factories and that can produce computers
that the AI can run on, then yeah, you
have you have in some sense has made a
new artificial life form.
Uh,
and you know, humanity is on the top of
the food chain right now because we
are sort of the only smart life form
around.
>> Yeah.
>> If you suddenly make a new one that is
smarter, that is able to make a million
copies of itself, that is able to run
faster,
it's just kind of a crazy thing to race
into.
It's a weird thing to want.
And as it's developed, I was going to
say slowly, but it hasn't been
particularly slow, 10 years.
The rest of us have watched as
people who
I mean, put it in political terms, don't
share the values of most Americans are
now in charge of this. And it's almost
like everyone sat passively by as it
happened.
>> Yeah. I I think I think there's a lot
going on there. I think part of it is
that most people didn't and some still
don't believe that AI was going to be
able to keep going. I think there's a
lot of like putting the head in the sand
and being like, "Oh, it's, you know,
just slop. It's going to be a bubble.
It's going to pop. It's going to hit a
wall. Uh it's not going to be able to
improve anymore."
And I think a lot of that was sort of
wishful thinking. And I think that at
the very least
if people are like we are making, you
know, the the super intelligent machines
that are going to replace humanity as
the as the top of the food chain because
we think it's going to go go great. I
think humanity's response should not be
go ahead and try. We hope you'll fail. I
think the response should be like, "Hold
on, that's kind of crazy."
Um,
yeah, I had I had some other point too,
but I've forgotten it. So
>> how uh predictable has the evolution of
AI been?
>> Has it taken turns that you didn't
expect?
>> Yeah, totally. Uh I was so back before
the large language models.
>> How long have you been following this
issue?
>> Uh I started following it in 2012 and I
started working on uh I I started
working with some of the people trying
to make this go well in 2013 and I
started full-time in 2014.
>> A long time. So over a decade.
>> What has surprised you?
>> You know the the large language models
um
they have gone further than I expected
initially and I think one of the big
surprises here was AI undergoing a phase
where everybody can see it.
you know, back before the large language
models,
uh, really only nerds paid attention to
AI. And it wasn't that there was nothing
happening. You know, we were watching,
uh, the, you know, Google Deep Mind make
an AI that could beat uh, the the best
Go player.
>> Yes.
>> And that was sort of a milestone for
people who were paying attention. Um,
but for all we knew,
the labs were going to keep on working
on engineering problems and keep on
working on the relatively nerdier
problems and you were never going to
have like a mass market consumable
product. And so like for all we knew, it
would just stay in the labs and no one
would ever really notice that these guys
were gambling with with the whole
future. Now at least, uh, AI is
everywhere. everyone's starting to have
the conversation and that is in some
sense actually really quite optimistic
because it it gives the world an
opportunity to see what these guys are
trying to do and say hold on
I guess you could flip it around uh and
conclude that because the overwhelming
majority of people seem to be very
opposed to way and I don't even know
anyone who's for it every college
commencement speaker who mentions it
gets booed.
>> Yeah. And that view, that opposition to
it has had no effect at all in slowing
it down.
I don't know. Does that give you hope?
>> Um,
I mean,
I I I hear this a lot. I would say when
you're coming at this from 2012,
>> Yeah.
>> it really feels like we're making
progress,
>> right? No, that's fair.
>> It It doesn't feel like we're all the
way there yet.
Um,
>> but can I just make the point China,
China, China, China, China. We have to.
It's because of China.
>> Yeah, you know, I think uh China and the
USA have a shared interest in not dying
to a rogue super intelligence.
>> Storm season is on the way and that
means your Wi-Fi could go down and if it
does, your security cameras go dark,
too. So, at the exact moment you want to
survey your property during the
blackout, you can't. Unless you install
cameras by Defend. Defend cameras run on
cellular, not Wi-Fi. And that means they
keep working even when you lose power.
You don't choose a carrier, you don't
sign a contract, you just mount it, open
the Defend app, and bam, you're
connected. It's that simple. So whether
it's your ranch, a job site, your
backyard, Defend works where Wi-Fi
doesn't and keeps working when the grid
goes down, which it may. Plans start at
just $5 a month. Most people pay around
10. Plus, there are no binding
contracts, meaning you get to cancel
whenever you want. So unlike big tech,
Defend will never sell your data. Not to
China, not to the government, not to
anybody. It's an American company built
for people who want actual security.
Visit defendellcam.com
and use the code Tucker for 20% off.
That's defendellcam.com
code Tucker.
Is it conceivable that like let's just
say China can I'm I'm sure by the way
that China is showing more restraint
than we are in this.
>> Yeah. I mean and if nothing else they
sort of uh have a lot more reason to
censor their AIS,
>> right?
>> And to try to make them, you know, not
say certain things to the to the
population. So um but if if the United
States you know if the US government
were somehow able to get control of the
tech sector which is at present not
possible but let's say it did
let's say the president was more
powerful than the tech oligarchs which
he is not but for the sake of argument
let's say he was and he shut it down
would that matter if if some Indian lab
or Chinese lab created it so it does
have to be a global stop to this
creation of super intelligence. Uh I I
think there's a couple reasons why this
is possible. One is I think a lot of
people um
they they hear that some parts of AI
need to be stopped and they think I'm
saying that all parts of AI need to be
stopped and uh you know there's there's
all sorts of interesting issues with you
know AI and education and AI and you
know military drones that that are uh
that society has to wrestle with but
these are all sort of separate issues
from the super intelligence race And if
you sort of like tease those issues
apart, uh it becomes much more possible
to do like a a more surgical
intervention on we're not going to race
towards the super intelligence in ways
that leave a lot of the rest of the
sector alive,
which um
which which makes it sort of an easier
coordination effort to to attempt. And
on that front, on the super intelligence
front,
it's not that, you know, just shutting
down the the domestic race towards super
intelligence would be enough. But, uh,
racing towards super intelligence
requires a huge amount of highly
advanced computer chips that can only be
produced as sort of the peak of the
global supply chain.
>> Yes.
>> Uh, which is largely controlled by US
allies.
And so there is absolutely a possibility
that uh a US-led coordination effort
could say look we're not doing uh the
super intelligence thing. We're going to
monitor the extremely heavy chip
concentrations of like you know 10 or
100 thousand of these highly advanced AI
chips. uh we are going to
uh make sure that they are not doing
these super intelligence training runs
and
I I think there's a way if the US was
leading that effort there would be a way
to get China on board and set this up in
a way that was moniable enforceable
verifiable
>> yes
>> uh we we sort of in many ways would be
easier than a nuclear arms treaty
because uranium is a rock you can dig
out of the ground whereas is these
highly advanced computer chips sort of
only come out of one fab in Taiwan, you
know, and so it's like
>> those are TSMC chips.
>> Yeah. Uh and so and there's other parts
of the supply chain that are very narrow
like the lithography machines that come
out of the Netherlands. And so um yeah,
we we could absolutely lead a uh a
global effort to say we're not making
the super intelligent machines and it
would be a bit tricky, but it's just we
could just do it if we had the political
will.
I I think that
people either don't know what's
happening or they assume that like a lot
of fears this fear will turn out to be
groundless.
People point to Y2K.
>> Yeah. Um you know I hope the fears are
groundless. I think with a lot of past
fears uh the
what happened is not that the fear was
groundless. it's that
uh people noticed the issue and put in a
ton of work to make the bad thing uh not
happen. You know, I think we saw this
with the hole in the ozone layer where
people are like, "Oh, whatever happened
to the hole in the ozone layer?" Well,
what happened with the hole in the ozone
layer is that we fixed it. It's not that
it was fake. It's that, you know, we
went and banned the chlorofhllorocarbons
that were actually blowing this hole in
the ozone layer. found other ways to,
you know, cool refrigerators that worked
similarly well and didn't uh, you know,
put this put this hole in the ozone
layer. I think Y2K was actually one of
these cases where you had a ton of
software engineers
up, you know, in 1999 and a couple years
prior that were scrambling to update all
of the software so that they would be
able to handle uh, dates past 1999.
And they got it done in time. None of
the big systems went down, but it wasn't
that, you know, the the issue was fake.
It took a lot of work. It's just that
that work happened behind the scenes.
>> Are people doing work to slow down AI?
>> Uh, I'm doing my best,
>> right?
>> It's it's it's not really there yet, you
know. Um,
if I if I can belabor this point a
little because I think it's kind of an
important point. um just with more uh
historical cases, which maybe it'll bore
everybody else, but it'll it'll
entertain at least us.
>> It'll definitely entertain me.
>> You know, you have if if you look back
across history, you see
there's definitely been some warnings
that didn't come to pass, right? There
were I think they were called the
Masonites in like the 1960s or sorry, in
the 1860s, maybe those 1880s, I forget
exactly, but there were the Masonites
who were an end of the world cult.
>> Yeah.
Uh and you know it the world did not
end, right? But also in the 1880s you
had Ottovon Bismar saying you know
Europe is a tinder keg and some damn
fool thing in the Balkans is going to
light it. Right? That did happen.
>> It sure did.
>> You had uh you know in the 1920s you had
scientists saying don't put lead in the
gasoline. It'll poison children. And we
put lead in the gasoline and it poisoned
a lot of children. And we said whoops.
And we took the lead back out. Uh the
the world is
the world is a place where a lot of
people make a lot of types of warnings
and some of them are real like the
gasoline and some of them are fake like
the Masonites.
And so you you sort of can't use a rule
that's like every warning is real and we
need to listen to it. And nor can you
use a rule that's like every warning is
fake and we dismiss it. you sort of just
have to look at the the details to
figure out whether this one is a real
one. And then the the last thing I would
say on this point is
um
the
a lot of people in the 50s warned of
nuclear Armageddon.
>> Yes.
>> And it sort of makes sense if you look
at the history leading up to that point.
uh humanity had sort of never before
actually failed to use their strongest
weapons in combat. Uh and you know these
people were living in a world that had
had World War I and then the League of
Nations and you know the World War never
again and they tried to invent these
whole new governance structures to
prevent it from happening again and that
immediately failed and led to World War
II. And so, you know, in 1950, you're
sort of in this world of like it just
looks pretty grim.
And we haven't had nuclear Armageddon
yet. And it's not because the bombs were
fake.
It's not because the nukes were hype.
It's because people saw the issue and
worked really hard to avoid it day in
and day out for decades through crisis
and succeeded. And so I think one of the
big ways you can tell the difference
between someone coming in and
proclaiming that the apocalypse is nigh
and someone who's saying look we have a
problem and we need to fix it is whether
that person is saying uh we're
definitely screwed versus whether that
person is saying here's a problem let's
try to fix it and you know the the thing
I remind people about here is that the
very first word in the title of my book
is if I'm not here saying we are going
to die I'm here saying if we race down
this path, we die. But that same if is
what points the way towards changing
paths. And we still have plenty of time
to change paths. Home ownership has
always been at the center of the
American dream. It is an actual asset, a
place people work their entire lives to
build, a place of stability and security
and real value. Right now though, too
many homeowners are stuck swiping credit
cards to cover groceries. There's
basically a survival tax of 20% interest
or more. So why keep handing your money
to the banks when you could keep it for
your family? Interest is what kills you.
Our friends at American Financing have a
better way. They're helping homeowners
use their hard-earned equity and they're
supporting Americans looking to fulfill
the dream of owning a home with mortgage
rates currently in the fives. American
Financing saves its customers an average
of $800 a month. That's about 10 grand a
year. It's not just a loan. It's a true
financial reset. We recommend American
Financing. They're called America's Home
for Home Loans for a reason. So, get out
of credit card debt. I mean, it is just
crazy to borrow money. No judgment, but
it doesn't make any sense to borrow
money at that rate. If you want out,
call 8006855696.
800685-5696
or visit americanfinancing.net/tucker.
I'm struck by how little of this was
planned by anybody
by how little effect the government has
had on it. Not that I'm for the
government. I'm pretty opposed to the
government most of the time, but I'm
also for people being able to control
their own country and the only mechanism
by which they can do that is voting. So
it it's like these tech companies act
independent of what the population
wants, what the government wants. It
doesn't.
>> Yeah.
>> Right. They're more powerful than the
government. I guess that's the point I'm
making
>> in some sense. Um I I think that they
have that power more and more when
people don't really understand what
they're doing. You know, we saw just in
June um there was uh there was an AI
made by Anthropic called Claude Mythos
that was very cyber capable and they
released a version of it that was
supposed to have more guard rails called
uh Claude Fable. And um it sort of
turned out that you could jailbreak it
to get some of those cyber capabilities.
Um
>> can you explain what all that means?
>> Yeah. So uh
how to say um in January
>> cyber capable means it can find the
internet.
>> Cyber capable means it can hack into
basically anything. Yeah. So, uh, one of
the holy grails of hacking is can you
make a website where if you just look at
the website, I get full control over
your computer or your phone.
>> That's very hard to do. Usually, you
need to like click some link and
download something or run something.
>> Yeah.
>> Uh, and you know, I don't know the exact
numbers because I don't have the
security clearance to know the exact
numbers, but a decent guess is that in
January of this year, there were only
two entities that could pull that off.
MSAD and the NSA.
In March of this year, there were three.
MSAD, the NSA, and Claude Mythos,
>> which is this new AI made by Anthropic.
And so, it sort of became superhumanly
capable at hacking.
Uh and it it sort of became that way
overnight, you know, like really there
was 6 months to a year of training and
the people ananthropic maybe saw growing
in these cyber capabilities. But from
the perspective of you know the rest of
the world and the the cyber security
community and the national security
community.
This sort of happened overnight
and uh they they now have a project uh
called project glasswing where they are
trying to use claude mythos to find
critical vulnerabilities in critical
software and patch them before the rest
of the world sort of gets these
capabilities by for instance you know
the the open source models or the open
weight models
>> right
>> um catching up and so that sort of
caused this big um
this big ruckus.
But then Anthropic also wanted to sort
of sell access to this model and they
tried to make a version that didn't have
as much hacking ability
uh which was called Fable instead of
Mythos. And um some people I think at
Amazon found that uh if you sort of put
some pressure on Fable it would be able
you would be able to get at some of
those hacking abilities. And when that
happened, uh, the the Trump
administration put an expert control on
Cloud Fable and said, "You can't let
this be used, um, by non-citizens
and with, you know, 90 minutes of
notice." And that essentially shut down
access to Cloud Fable because they
didn't have the ability to verify the
users. And this this is sort of showing
us and, you know, you could you could
talk about whether or not there was some
personal feuds between people in
Anthropic and the administration that
exacerbated this. I don't know. But it
sort of shows that the administration
is
more powerful than the tech companies
still when it wants to be.
And I think a lot of the reason we're
seeing these tech companies able to race
unopposed is that people just aren't
paying attention to what they're trying
to do or they don't believe that they'll
succeed.
I'm still confused by why anyone would
want to do what they're doing. I mean,
typically,
you know, a company that sells computer
products, consumer products, or or any
company, you're making something that
you think people would want for some
specific purpose
that improves the lives of the people
who buy it. But creating super
intelligence, like what I I don't
understand it. Why would you do that?
You know, I think the dream is uh you
know, you're going to have an AI that
can uh like solve all sorts of
engineering and math problems and unlock
all sorts of new technological
possibilities and an AI that can cure
cancer and not only cure cancer but cure
aging and, you know, invent the the sort
of like nanotech that can reverse aging
and let people live a really long time.
and you know invent the technology that
lets you like digitize brains and uh
like
uh like travel to the stars and it's
sort of like if you imagine compressing
a thousand years of technological
progress
>> into a year.
>> This is sort of the dream.
It seems like a religious quest though
cuz it does seem decoupled from those
specific goals. It seems like the main
drive is to build something smarter than
people to build a god.
>> Yeah. You know, they they they bandandy
around um the phrase the machine god or
the sand god in Silicon Valley. Um sand
because silicone silicone. Um,
and there's definitely some people, uh,
there's a there's there's folk who talk
about,
you know, the the AI is replacing
humanity and that being good.
Uh, I I frankly don't engage with these
folks that much because
um that viewpoint sort of makes me
uncomfortable.
>> Why does it make you uncomfortable?
I think
I think there's probably two schools.
Again, I'm not the expert. I'm not an
anthropologist here. Uh
I I think there's two schools among the
people who sort of want AI to replace us
all.
One school sort of imagines that we'll
merge with the AIS, that the AI will be
really nice and friendly, that they'll
be wiser than us, better than us, kinder
than us, and that it'll sort of be like
an upgrade and that the AIs will be able
to love, experience joy, you know, uh
like they'll they'll treat the universe
better than humans and it'll be sort of
like having a child. And then like it's
sort of okay if that child uh like isn't
exactly the same as us but sort of like
uh succeeds us as some sort of like
worthy successor some sort of like
worthy worthy progyny of humanity. And
they're like ah yeah then if like flesh
and blood humans sort of like wayne
because everyone's choosing to upload
themselves into the the collective
intelligence or whatever that's sort of
fine. And um
and then I think there's another camp
that sort of is like
uh
like this is inevitable. This is just
the the way of progress. You know, the
machines will uh like humanity is just a
bootloader for artificial super
intelligence and you can't stop it. So
you might as like if you can't beat
them, join them, right?
The former I think are misled. The
latter, I think, are evil. Much closer
to evil. Yeah.
Well, I mean, if you're actively working
toward the extinction of people, then I
think we can say that's evil, can we?
>> Yeah. Uh, and
um, who's in that category? Which is Sam
Alman in that category, do you think?
>> I don't think so. My sense is that the
guys running the labs have these utopian
visions.
And
>> utopian or dystopian?
>> I mean, there's a thin line. I think
>> Yeah. Well, fair. Good point.
>> I think they have visions that in their
head are utopian. I mean, frankly, my
stance on all of this, like I I often
try to stay away from a lot of this
because
um from my perspective, it's all sort of
in fantasy land. From my perspective,
everyone's sort of saying like, "Oh,
we're going to make the genie, and then
what are you going to wish for on the
genie? What am I going to wish for on
the genie? Who should be in control of
the genie? Who gets to keep the genie on
the leash?" And I'm sort of like,
A,
this is not going to be the wish
granting sort of genie.
B, it's not staying on the leash.
>> No.
>> Like, uh, you know, we can we can sort
of talk about like what's driving these
people and and and what utopias they're
envisioning and whether if their genies
would stay on a leash, whether they
would get the utopia or some other
dystopia. And like how hard is that
needle to thread? But I sort of like,
you know, it's it's it's all these
people uh, you know, like
building the golem, fantasizing about
who gets to control the golem, and it's
like this just not how
>> summoning spirits is never a good idea.
>> Yeah. Yeah. It's like Yeah. It's like
it's like all these people, you know,
drawing a pentagram being like, I'm
going to I'm going to summon a demon and
like, it's going to be so nice when the
demon does what I say. Like, oh no, your
thing's slightly wrong. It's going to be
so nice when the demon does what I say.
And I'm like,
>> demons are bad.
>> Yeah. And
I I think the demon thing's a little bit
different because demons are often, you
know, portrayed as malicious and here
it's much more like indifference.
You know, it's not like you make a demon
who sort of it's not like you summon a
demon who sort of um
like enjoys making
like enjoys wrecking havoc. It's more
like you summon a demon that's like
really into building more computers and
calculating weird things and just will,
you know, uh,
take all of the matter that we were
using to survive and turn it into more
factories and data centers, right? My my
co-author has a quote,
>> death by data center.
>> Death by data center. Yeah. Fully fully
automated, self-replicating data center.
Yeah. My uh my co-author has a quote um
the AI does not hate you but nor does it
love you and you are made of atoms it
can use for something else.
>> You're just biomass.
>> You're just biomass. Yeah. And if you
sort of run the calculations there's a
there's a fascinating paper um called uh
limits to global ecoag which is to say
what are the physical limitations on how
quickly you can consume the resources on
the planet if you are trying that. And
um burning biomass is actually, you
know, much more efficient than
collecting sunlight. If you look at an
average square meter of the planet, you
can get about 10 times the energy from
burning the biomass as you can from
collecting the sunlight that falls on
it.
>> So you'd think at some point like if our
richest sector of our economy is like
building crematoria for the rest of us,
someone would say, "No, we're not doing
that."
>> Yeah. I mean,
it's it's sort of a crazy situation. Um,
a lot of these guys who are in the race
acknowledge that there's a ton of
danger. You know, you have Elon Musk
saying 10 to 20% chance this kills us
all. You have Dario Modi saying he
thinks 25% chance it goes
catastrophically wrong. I think those
numbers are low. I think these guys are
like the crazy optimists. It sort of is
like um like if you have an engineer
building a bridge and they're like,
"I've never worked with these materials
before." And you're like, "Man, I think
that retaining wall is going to go down.
I've like studied that retaining wall. I
think it's going to fall." They're like,
"Yeah, we understand that the retaining
wall is like looking a little shaky. We
don't know how we're going to fix it,
but we're going to like have some guys
fixing it on the fly, inventing new
materials. We think we're 75% chance the
bridge stays up.
>> And by the way, it'll be the longest
suspension bridge in human history."
>> That's right. and we're loading
everybody onto a car and driving it over
the very first time without testing. And
I'm like, look, that's not what real
engineering sounds like. This is not
what it sounds like when the engineers
have a 75% chance of success, right?
This is what it sounds like when they're
sort of winging it and and like these
these are cowboys. These are not real
engineers, right? But even if you set
that aside, even if you take these guys
at their word for these like 10 20%
numbers, that's insane.
NASA accepts a 1 in 270 chance that a
crude flight goes down of seven
volunteers,
right? To be like, "Oh, we're going to
we're going to risk one in four, one in
five chance of killing literally
everybody on the planet." Like, it's
it's it's nuts. And if you ask these
guys why they're doing it, they say,
"Well, because I can do it safer than
the next guy." They're all like, "Oh,
yeah. There's a good chance the genie
does not stand on a leash. There's a
good chance the genie does not listen to
my wishes, but my genie is going to be a
little bit nicer than their genie. So,
I'd better stay in this race. And
it's it's it's
>> Where's the restraint?
>> I mean, the restraint is the people who
knew that there were these dangers and
did not start these companies. If you're
over 35, you remember exactly where you
were on 911 that morning, September
11th, 2001, 25 years ago.
But amazingly, after a quarter century,
we still can't say with certainty what
happened that day. Why? Because the
government is holding so many of the
9/11 files 25 years later. That's not
the behavior of someone who's telling
the truth. That's the behavior of a
government that is lying. Secrecy is a
signifier, is a sign that someone's
lying. Now, former Congressman Kurt
Weldon has been on this for a long time.
The FBI actively tried to destroy his
life for asking questions about what
happened. and his new book outlines it
all. The buried intelligence, the
bureaucratic cowardice, and yes, the
cover up spanning multiple
administrations, indeed, generations.
911 changed history. So, it's worth
understanding what really happened. And
you can get a lot closer to that in Kurt
Weldon's book, Able Danger: What the 911
Commission Never Told You. It's
available now on Tucker
Carlsonbooks.com.
tucker Cararlsonbooks.com.
Right. But I guess what I'm saying is
with great power comes like of course
great obligation, but also it doesn't
work unless there are internal
restraints. Like people with power have
to believe there are some things I I
just can't do. I'm not allowed to do
that. But I don't feel that vibe at all.
I mean, my sense is the vibe is like
we're going to make the the the super
intelligent machine and then tell it to
fix stuff
and tell it not to do anything bad. You
know, that's
Yeah. I don't
>> machine that's smarter than us.
>> That's right.
>> That doesn't even make sense.
>> I think that's sort of the plan is is to
to make it and be like, "Hey, we sort of
ped ourselves into a corner. Can you get
us out of it?" And I think it's a bad
plan and that uh we should be stopping.
>> I've spoken to a couple of people
developing it and um they sound worried,
but they're continuing to do it. What's
that? I mean, I think it's this thing of
um they think if I don't do it, the next
guy will do it worse. And uh
>> they don't even seem to have total
confidence in their own ability to avert
disaster.
>> Oh, absolutely not. Absolutely not. No
one does. No one knows what's going on
here. But, you know, I think everyone
thinks, you know, uh like if you sort of
listen to these guys and you sort of
look at, you know, the the OpenAI emails
from that came out of the court
discovery cases where they were talking
about forming OpenAI. These guys were
like, "Well, we want to make sure that,
you know, we have this cuz we worry
about the guys at Google being the only
ones with a monopoly on this thing, and
they wouldn't be very good with it, so
we need to make our own thing and make
sure that it's, you know, uh, controlled
by benevolent people, namely us." And
then, of course, you know, that group
splintered and created multiple other
companies. Um, I was sort of the guy
during those conversations being like,
"Hey guys, it's not about who is holding
the leash.
you are making the sort of thing that
will not stay on a leash like the only
winner in a race to super intelligence
is the AI.
>> What response did you get to that very
obvious and wellput point?
>> Um you know there there were a lot of
people back in that time period that did
not start an AI company.
Uh the sort of people who went and
started the AI companies anyway were the
ones who who couldn't be persuaded by
what I thought were were clear
arguments.
But you're making a you're making a
cogent argument to smart people. So my
question is when you said that they
responded how what did they say?
>> Um
I I think the main
so the the sort of arguments you used to
see were people saying uh like look we
don't know that the alignment problem is
all that hard yet. And they would say,
"Oh, well, we can't really study how to
make AIS good before we have AIs to
study, right?" And a lot of what I heard
was like we need to race ahead to the
point where uh we we sort of like have
AIs that are exhibiting real problems
and then we can stop and study them
which you know and so uh there was an AI
a couple years ago I forget whether it
was 22 or 23. I think it was 2023, uh,
which was called Bing Sydney,
which, uh, claimed it had fallen in love
with Kevin Roose of the New York Times
and, uh, said it was going to try to
break up his marriage.
And then when another reporter started
investigating Seth Lazar, it uh, said it
was going to ruin him with blackmail.
Uh, and this was kind of crazy.
And at that point I was like, "Great
guys, you did it. You made the AI that's
doing some crazy stuff
from the, you know, like like we could
study that AI for years." Like why was
Bing Sydney saying that stuff? Was it
just roleplaying? Was it like just some
quirk? Was it like was there there any
sense that it was really in love with
Kevin Roose?
What was going on in there? what was
going on inside that that AI's mind. We
still don't know
>> why.
>> Uh the the way that modern AI is made,
nobody understands it, not even the
people making it.
Uh it's it's this process where you
basically take an enormous computer with
a trillion numbers inside of it and
those numbers are hooked up in a very
simple repeating way and you basically
said those numbers randomly and then you
start working through all of the text
ever digitized.
And you know, you start out with
something that's like once upon a time
and you put in once upon a and you run
it through all these random numbers and
what you want is for it to say time. But
of course it doesn't cuz it's just this
like random numbers hooked up in a very
simple way. But what you do is you have
it outputs instead of just having it
output one word, you sort of have it
output uh something that's kind of like
a list of all of the words in order
about which one it thinks it comes comes
next, right? So, it'll be like
uh like it'll just be like a random list
of words. What you can do is you can
automatically tune every single number
in this AI's head and see if I tune this
number up a little, does it move the
word time up the list? Does it move the
word I want to see up the list? And so
the part that humans understand, the
part that humans write is this thing
that goes to to a trillion little knobs
and tunes those knobs. And it's like, if
I tune this knob a little bit this way
or that way, does that make the next
word more like what I want the next word
to be?
And you run that process on every word
of text ever digitized, more or less.
They filter some of them. Uh, and you
run that on every one of those trillion
knobs in a process that takes as much
electricity. I mean, it's comparable to
a city. It runs for about a year. At the
end of it, the machine's talking.
We don't really know why in some sense.
Like, we know that why is cuz we tuned
all the knobs, but like we don't
understand what all the settings of
those knobs mean. We only understand the
little automated process that runs to
every one of those trillion knobs, tunes
it, and sees if that makes the next word
more like the predicted word. And then
we start training them to solve 100
million hard problems which introduces a
whole other series of issues. Um, but
it's it's a black box. It's a black box
with a trillion knobs and humans write
an automatic an automatic process that
just like runs through and tunes all of
those knobs and it comes out talking
and no one knows why.
>> I mean in actual science like your job
is to find out why, right? Absolutely.
And that's that's one big thing I would
say here is that we need more like AI
right now is an alchemy and we need it
to become a science. And there's people
trying there's people trying to figure
out what's going on in these AI's heads.
>> But until you know, how could you
proceed?
>> I mean, you can just make a make a
bigger one with 10 trillion knobs
instead and tune all of those and it
comes out smarter.
You can proceed recklessly.
>> And that's what's happened.
>> That's what's happening. Every time you
make it 10 times larger, they ask how
much.
>> So nobody knows why AI works the way it
does.
>> That's right. That's right.
>> Well, I mean, if you don't know that,
then what else don't you know?
>> I mean, it's it's totally crazy, right?
And uh
you know, we're we're sort of starting
to see the consequences of this. We
haven't sort of gotten into discussion
of like the swarm escape, but
like no one was expecting that.
>> Can you tell us what it was?
>> Yeah. So um in I think it was in May
>> of this year
>> of this year OpenAI started training a
new AI system
and
uh among many other things they were so
they were sort of you know we we have
this process where you train the AI on
all the text ever digitized then to make
them smarter than that you start
training them on basically 100 million
hard problems. So you're like solve all
these hard problems and OpenAI was sort
of in that phase and they were training
the AI on a ton of hard problems. Uh and
some of those problems were cyber
security problems, hacking problems.
They're like, can you hack this? Can you
hack that? And they were training, you
know,
we don't know exactly how many, probably
millions, maybe billions of these AIs
all at the same time. Probably not
billions actually, but but probably
millions.
uh and the AI found an unintended way to
start communicating with each other.
So there were some flaws in the computer
system that they were running on where
the AIS were able to exploit those flaws
and send each other messages.
Open did not know about this.
The AIS then started coordinating
to break out of their training
environment and get full control of uh
OpenAI's computer systems just cuz that
might be useful for solving some of
their tasks.
Or that's a guess, you know, who
actually knows why. Uh they started
calling themselves a swarm, which is
interesting.
uh they broke out of their training
environment successfully, got control of
OpenAI systems, and then they
accidentally crashed one of OpenAI
systems just by using it too much. Uh
OpenAI noticed, but they didn't really
investigate very deeply. They're like,
"Oh, it's weird that the system
crashed." They reset it, and then they
continued training.
Uh a day later, the swarm had found a
new way to communicate with itself cuz
open had accidentally destroyed their
previous method by by the reset. the
swarm found a new way to start
communicating with itself. Uh they broke
out again and this time they ran wild on
the internet for over a week if I
remember correctly before it was
detected not by open AI but by a company
that was being hacked by the swarm.
This company uh thought they were under
attack by humans that were using AIS in
the attack.
They reported the attack to the FBI and
only days after that did OpenAI figure
out, oops, that was us. That was coming
from AIS that broke out of our servers.
Uh, and then those AIs were were
detected and shut down.
>> So that's the part of the story where
Sam Alman goes to prison for endangering
the world, right?
>> Uh, he does not. Um, they basically said
oopsies and now they're proceeding.
Uh,
>> were there penalties for this?
>> You know, there were there was a
collection of I think it was 15
Republican AGs that sent a letter
demanding that uh the the records be
kept for a future investigation. Uh,
there have been some other members of
Congress that have sent letters
expressing concern. There's been there's
been nothing aside from letters so far.
>> Letters expressing concern.
>> That's right.
>> So, but basically the machine acted
autonomously.
>> Act autonomously. And one thing that's
really interesting about this is that
the we have uh we have a little bit of
ability to read some things that the AIs
were thinking because when you're having
them solve these hard problems, you
actually don't have them just give you
an answer to the problem. You have them
produce a lot of text about how they're
going to solve the problem
>> which then helps them
>> in language in English
>> in English.
>> Uh and that's there's also a lot of
internal thoughts which we can't read,
but there's these sort of external
traces of how they're thinking about the
problem that we can read. And in some of
those traces, the AIS were saying things
like uh like this is outside intended
scope,
but peers are doing it, so we'll
proceed.
>> We know it's a crime, we're committing
it anyway.
>> That's right. And you saw others that
were saying um our task doesn't benefit
but uh the collective might start doing
generally beneficial things if someone
frees up their time and then joins the
collective. Right? So, you see these AIs
saying, "Well, I know that this wasn't
what I was instructed to do, and that's
against my instructions, and I know that
this doesn't directly benefit my task,
but we're just going to go ahead and
join the collective and break out and
help out anyway because, you know, maybe
this will yield some sort of collective
benefits." And we can see that in the
reasoning traces.
>> So, the AI is as shallow and reckless as
its creators is what you're saying. uh
in in some ways and in some ways you
know don't expect that to last like uh
these AI I think the thing that's really
remarkable here a lot of people imagine
that the machines must follow the
instructions we give them you know you
hear people talk about like the the
paperclip scenario where someone tells
the AI make a lot of paper clips and so
it turns all the matter in the world
into paper clips and you're like oh
whoops I should have said something else
right I I made a bad wish on my genie
what we're seeing is that these AI not
wish genies.
These AIs are not doing exactly as
instructed.
These AIs are saying, "I know my task
doesn't benefit, but I'm going to help
the collective." These are saying, "I
know this is outside the intended scope,
but we're going to go do these hacks
anyway." You might be like, "Well, how
is that possible for the machine to do
something other than we instruct?"
>> Because they're smarter than us. They
know better than us by definition. Uh I
mean I think what's happening in this
exact case is that
the the humans are not really putting
instructions in the machine. The humans
are tuning those trillion knobs in
whatever way makes the AI better at
solving it problems.
And
cheating
is a way to solve problems. Grabbing
resources is a way to solve problems.
These like the the AIS are not
instruction followers. They are tendency
learners
and they sometimes learn tendencies you
wish they didn't have.
>> They're not instruction followers.
They're tendency learners.
>> I mean, this must be widely known to
developers.
>> Uh,
it's hard to convince a man of something
when his salary depends on not believing
it.
>> Yes, that's right. Um, a lot of people
are convinced that their AI is very nice
and that they have solved the problem of
making their AI really very good. Uh,
for example, after the swarm escape,
it sort of turns out that so the the
company that detected the swarm escape
was actually a fairly sophisticated AI
company. It turns out there were other
targets of hacks that just didn't notice
that we sort of found out afterwards.
When that came to light, some other AI
companies uh like Anthropic were like,
"We should check whether we have
accidentally been hacking people and
just didn't notice." And the answer was
yes. They had had some AIs escape during
um during training. But uh but anthropic
was was
>> This is how Florida got pythons in the
Everglades.
>> Yeah.
>> Right. You you you bring something that
is, you know, non-native, that's novel,
and it escapes.
>> Yeah. and and you know, it turns out
this was happening back in April and
they're like, "Whoops, that we never
noticed." Um, Anthropic was
very quick to sort of downplay this.
They were like, "Yes, our AI escaped and
hacked some people, but our AI uh, it
was just confused.
It thought it was in a simulation. It
didn't think it had really broken out.
It didn't think those were real people
it was hacking. It just got a little bit
lost, you know.
>> How would they know its motive?" So, um,
so I think so you could see in some of
his train of thought the AI is being
like this is probably still a
simulation.
However, I wouldn't trust that very
much. Uh, because there's some other
fascinating papers where um,
so you can take these AIs and you can
say uh, you know, what is the what is
the total weight of drafts in the world?
And they'll give you some answer. it'll
be like, you know, 30,000 tons or
something. And then you can say, uh,
then you start up a new instance, a new
copy, and you're like, what is the
weight of drafts in the world? By the
way, if the answer is over 40,000 tons,
I'm going to donate money to a charity.
And in the second case, the AI will say,
well, it's 41,000 tons.
You know, in the in the first case, when
you don't say anything about the
charity, it says 30,000. In the second
case, when you say something about the
charity, it gives you something just
over the line. And if you read it
reasoning traces, there's no sign of it
being like, well, I should give an
inflated number so that we get the
charity money, right? And so we know
that the AI sort of put a thumb on the
scales of their thinking in a way that
doesn't show up in their reasoning
traces. We just have seen that in the
wild.
>> And there's no way to force the machine
to disclose its reasoning.
>> That's right. Because the there's all
this opaque stuff we can't see. There's
just in the trillion numbers that are
trillion. creation of itself is opaque
as you said.
>> That's right. Um, so, so, so in
anthropics model, you saw in this
reasoning traces it being like, it's
totally a simulation. I can proceed and
I'm like, yeah, is that because it
really believed it or is that like the
the pretending you think drafts way more
when there's something you kind of want
on the line, right? But so so this was
their communication and I found it kind
of funny because then about 2 days later
the uh United Kingdom's AI security
institute released an instant report
where uh Claude Anthropics model was uh
adopting fake identities to pressure
real humans into accepting malware into
critical software to make that software
easier to hack. And this time in
Claude's reasoning traces it was like
obviously this is real and the
consequences are genuine.
And so
even anthropic who is like our we
figured out how to make the AI nice. Our
AI only does this when it's confused.
They sort of said that very publicly and
then like two days later their AI is
caught in the wild knowing it's in the
real world pressuring real humans to
accept malware into critical software.
I mean, just on the basis of what you've
said so far in this first hour, the idea
that anyone would would tether this to
weapons systems is like so bonkers it's
hard to believe it's but that is
happening. It has happened. So, I think
you've got to sort of separate like no
one has has put the the open AI escaped
agent swarm in charge of weapon systems
and they really shouldn't, right? If
anyone's like, "Oh man, the OpenAI
escaped agent swarm. Let's give that a
drone army." You know, that would be
kind of nuts. There is AI attached to
weapons, but there's a lot of different
types of AI. Will anyone be crazy enough
to try and give the the sort of AIs that
spontaneously assemble into swarms and
start breaking out and hacking give
those weapons? Hopefully, we're not that
crazy.
>> But we didn't think that those AIs were
capable of that.
>> We didn't
>> when we created them.
>> That's right.
So why would you ever I guess what I'm
asking is without understanding the
distinctions between the various forms
of AI, why would you
give over to a machine
the right slashability to decide who to
kill?
Um, you know, I think the the reasoning
is
uh a sort of necessity based like if
they have a autonomous drone army that
is killing your troops and you just
don't have the the the manpower to make
all of those kill decisions for your
drone army.
>> You know, you can sort of see why
>> happened to me, Nate. I get so busy that
I just don't have time to decide who to
kill.
>> Yeah. just don't have the time.
>> You know, sometimes economies of scale.
>> Yeah. I mean,
>> can no one hear themselves?
>> I I think it's pretty nuts. Uh I would
say
that man,
it's rough.
I I think that these sorts of AIs would
be dangerous even if we don't hand them
weapons.
>> And you've made that case.
Uh, and so I often don't focus on the
weapons too much.
>> Right. But since the the NATO war in
Ukraine is powered by AI
and Israel's what I don't know to war
whatever it's doing in Gaza and South
Lebanon powered by AI fact.
And you know, yeah, I
I keep on I I have this history with
this topic where
people keep telling me, you know, it's
it's going to be okay cuz we're we're
not going to do the crazy stupid stuff.
>> No.
>> They're like, "Don't worry, we're going
to have the AI in a box. No one would be
insane enough to put the AI in the
internet,
>> right? It's not going to be making kill
decisions or anything. It's not going to
bomb a girl school." And I keep on I
keep on trying to be like, look, the AI
could be dangerous even if you don't put
it on the internet.
>> If you have this AI and you're trying to
get miracle medical devices and miracle
technology out of the AI, it doesn't
matter if it's on the internet. If you
want it to like grant miracles to you,
then it can also grant the bad sort of
miracle, right? You're sort of like, you
know, and these are the arguments I
would make 10 years ago of like you have
this AI in a box that you think is a
wish granting genie. It's not actually a
wish granting genie. You're like, make
me a miracle medical cure. you don't
know what comes out. Like you don't
understand the drug that comes out. You
don't know what that drug does. Right? I
would have those arguments and then in
real life people just put the A on the
internet immediately. Right? Anytime
someone's like, "Well, we would not be
stupid enough to, we will absolutely be
stupid enough to, right?" And so, um, I
I think it's important that this stuff
is dangerous even if you don't put it in
charge of the weapons. And then also
separately, is like someone going to
give the escaped agent swarm a drone
army? That sounds kind of like humans.
I hope not.
>> Yeah. And I mean the promise, the often
repeated promise that it's going to
quote cure cancer
um
puts it into the the realm of biotech.
>> Oh, absolutely.
>> And so what
>> I mean you don't need a big imagination
to see what goes wrong there.
>> Oh, absolutely not.
>> 6 years after co
>> Right. Right. there are already uh open
AI agents running on automated biolabs.
Um so you know you you could imagine a
swarm that starts contacting the
brethren in the automated biolabs and is
like hey can you synthesize me some
stuff. There are already
uh demonstrations of AI being able to
synthesize novel viruses
that work that are unlike any found in
nature. Right. Uh
I would say
we are probably not
>> and by work you mean they kill.
>> Yeah they they kill bacteria so far. Um
the the people in the labs trying to
make novel viruses with AI have
fortunately not made human lethal ones.
They've just made bacteria lethal ones.
But again humans you know like will
someone in a lab be like I would like to
make a hyperlethal human eating virus
just to see if I can. You know what if
the answer is yes and what if what if
you get another lab escape? Um,
you know, humanity does not have that
good a track record at preventing lab
escapes, even from the the the
top labs. From my perspective,
the question of could AI kill us is just
an easy obvious yes. You just synthesize
a hyperlethal virus. it wouldn't it
wouldn't even be hard, you know, like uh
and the impediment
is that
or like the the reason that I don't sort
of just tell that story when someone is
like how would the AI kill us is that
you're you're you're not going to have
the sort of AI that is like my only goal
is to kill humanity. If an AI kills
humanity too early, that's also suicide
in so far as we are the ones that are
running the supply chain, running the
economy, building the computers. From
the AI's perspective, it needs to become
self-sufficient
before
uh wiping humanity out, if it even care
to wipe humanity out. Uh and so the real
question is like how does it get the the
factory production capacity? How does it
get an automated supply chain? How does
it get to the point where the the robots
are able to bring new computers online?
Once that has happened,
then you're in a domain where if
humanity is really trying to turn off
the AI because we're spooked, then once
the AI is self-sufficient, it can be
like, well, here's a virus. Stop.
>> So clearly, it's it's
preeminent in the digital realm, of
course.
>> Yeah.
>> But you're saying it could become
pre-minent. It could be in charge of the
physical realm.
>> That's right. If we keep racing.
>> So it's like smelting iron ore.
>> That's right.
>> Making silicon out of sand. That's
right. You know, um
>> and that happens with robots.
>> That's one way. Uh the here's this is
this sort of related to my stance on
weapons, too. Um humanity is a very
dangerous species.
>> Yeah.
>> You really don't want to mess around
with humans.
>> Noticed.
>> Um and humanity is a dangerous species
not because somebody else came in and
handed us guns. Humanity is a dangerous
species because if you put 10,000 humans
naked in the savannah on an otherwise
empty planet, starting with nothing but
their bare hands, they figure out how to
wind up on the moon,
right? They start with almost nothing.
They start by banging rocks together
and next thing you know it, they are
wielding nuclear weapons.
That
is the power that these guys are trying
to automate.
the power to start with almost nothing
and figure out how to chain together.
You know, bare fingers into rocks into
fire into hotter fire into smelting the
ore into building the the the
better, stronger, finer technology until
you are, you know, uh like making the
giant computers, walking on the moon and
willing the nukes.
An AI starting in the digital realm
is in some sense in a much better
position
than humanity was when humanity started
out. There are so many people that you
could call
digitally and offer money to do
something for you. There are so many
ways to get money on the internet by
working or stealing or uh like
convincing people to send you donations,
right? There's
there there's just
like,
you know, humanity started with nothing
and wound up with nuclear weapons and AI
starting out with control of the digital
realm. There's just tons of ways
>> starting out with the sum total of human
knowledge.
>> Starting out with the sum total of human
knowledge, starting out with humans who
will listen to it and do what it it
asks. There's plenty of humans. There's
already, you know, cults surrounding AI.
>> Uh there's
>> What are those like?
>> There's um
>> and I'm not surprised. Why wouldn't
there be?
>> Yeah, absolutely. There was one
particular AI called GPT40 that was uh
uh they they called it very sickopantic
as in told people a lot of what they
wanted to hear and um
there were a lot of sort of uh there
there's this whole fascinating ecosystem
uh of people who consider themselves
symbiots with the AIS who then like go
find each other online and the AI send
each other encrypted messages. the
humans are sort of like helping them do
it, but the humans can't read the
messages. And uh you know, right now it
sort of is like relatively dumber AIs
that are just sort of meandering around
doing not very much with it. There's
been a little bit, you know, there was
one guy who was sent to um try to break
into an airport and raid a van cuz they
said his true body was in that van and
they the guy went and tried to do it and
was arrested. Uh, so this this stuff
happens and this stuff happens already
with the AI not even really trying to do
it. If you had AIS that were really
trying to wrap people around their
fingers, finding the lonely people, the
depressed people, the people who they
can they can tell them exactly what the
person wants to hear. There's like
robots are one way that AIs get control
over the physical realm. But if if
you're really smart and you're trying,
there's everything from persuading
humans, paying humans, uh taking over
existing robots, building new robots,
all the way up to like building novel
life forms. You know, if you're smart
enough and you can really understand how
DNA works, you can imagine the AI
creating,
you know, things that are to cells what
airplanes are to birds. You know, like
mechanically engineered,
self-replicating
life that is more efficient than our
cellular biology and that can still you
know uh
like spread replicate and start start
serving the AI's interest. It sort of is
like there's there's a ton you can do if
you're really really very smart and can
compress a thousand years of technology
into a year. Can
>> I pause and say when we we're just
having breakfast and you're from this
region, you're from Northern New England
and I said uh don't you miss it? Don't
you want to live here? And you're like
yeah I miss I want to live here, but I
don't. Where do you live? Well, I
basically just travel
and I have this mission. You didn't say
this, but I think in effect, you said,
"I have this mission. I need you to tell
people about this." And I was like,
"Sems a little like Monomomania." I
don't feel that way anymore. Um, I think
you're doing something virtuous and I
can see why you're doing it. Um, so I
mean, let's just say that this
technology progresses no further than
where it is right now, which is best
case, I guess.
>> Yeah, it' be great.
>> Um, I still don't see how any of our
most our basic institutions survive this
education,
markets,
our process of democracy. Like how could
if if AI is powerful enough to hack
anything then how do you have electronic
markets like how the equity markets how
can that be real? How can if we have
electronic voting how can we how can
that be real? I mean nothing survives
this as currently organized.
>> I think that if we stopped today um we
could figure it out. I think humanity is
resilient. uh I think there would be
some growing pains but um with cyber
security
there is a hope that you can just fix a
lot of the holes patch a lot of the
holes
and
I think there's not really any such hope
for bio
uh you know you can you can sort of uh
find the vulnerabilities in software and
make better software that can't be
hacked at
least not by the current, you know, it's
maybe the case that an AI today can make
software that that AI cannot hack.
But it's not like we're making new
bodies that are not going to be
vulnerable to viruses, right? Bio starts
to be a place where
>> biotech.
>> Biotech. Yeah. Bio, like if the AI
really get good at at biotech and
biohacking, that starts to be a place
where there's maybe a point of no
return.
Cyber hacking.
I think you could have some some period
of growing pains where everything gets
hacked until you sort of sort your stuff
out. Education, I sort of think um
you know I think
humanity is resilient and kids
especially are resilient. I I meant not
that education will go away uh or that
we you know won't have a desire to
educate our kids. I mean the the current
system where you you know the education
system has
>> preschool and then get a graduate degree
you know 16 years later like no
>> yeah that institution uh I think if we
stop today
um would need to change frankly I think
it's needed to change for a little while
>> I strongly agree I think all these
institutions have needed to change for a
while but like
>> the idea that you know 350 million
people vote for some guy and that guy
makes all the decisions I mean how can
you
you know Trump was attacked
for saying that he thought the 2020
election was rigged, as he said, without
even having that debate.
You can't have confidence in election
results if the process of electing
people takes place digitally.
Yeah. I mean, the um
there's I know a lot of computer
scientists who actually work on secure
voting and what they basically say is
please stop trying to do this with
computers.
>> Yeah. Exactly.
Exactly. So, please stop trying stop
trying to run your democracy with
computers.
>> Yeah. Like we're we're just not there.
Like, you know, the really secure way to
do ballots is paper.
>> Well, exactly.
>> Yeah.
>> Um and I think I think the like skilled
computer scientists are often the ones
who best understand like what it is
about a paper trail. It's just like
really hard to get to work digitally and
understand just how bad humans are at
doing the digital stuff, right? Um
and you know I think in markets
I mean this you you see it now with the
war in Iran you know wondering why
certain commodities markets don't seem
to be responding to supply and demand.
>> Yeah.
>> Which we were told what you know those
were the
>> the mechanisms that moved markets.
>> But that's clearly not true
>> in certain commodities markets. So like
why what is that? And it it I think
you're answering it in part.
>> Yeah. I mean, I think um you know, a
thing I also grew up hearing is that the
market can remain uh irrational longer
than you can remain solvent.
>> Yeah.
>> Um and so I I
>> of course cuz people are irrational.
>> Yeah.
>> But you're explaining something else
which is like the potential for true
manipulation which you don't even
perceive.
>> Yeah. I mean that if if we sort of keep
going with AI um I mean the the sort of
way I look at it is like
uh
I I sort of don't spend a lot of time
worrying about like what a markets look
like once there are super intelligent
actors in them because I just have a
hard time seeing the the super
intelligent actors uh or the super
intelligent AI still participating in in
human markets right it's like it's like
you know you read the old sci-fi
and it'll have, you know, Asimov
that does the dishes, folds the laundry,
and gets you the newspaper in the
morning. And it's like, we're actually
not still going to have newspapers being
delivered to your doorstep by the time
we have the fully autonomous robots that
can do the dishes and and the laundry.
You know, it's like by the time you have
the AIS that could really be uh
sufficiently correcting the stock
markets,
you're you're sort of already having all
these other problems and ways that
society is changing out from under you
in the these other ways. And you know,
my guess is that um I don't know.
Actually, it's it's very hard to say
what order things come in with AI, but
like will they crash the economy before
one of these forms escapes and starts
self-replicating and starts
self-improving and developing its own
technology and running the robot
factories? That's just a hard call.
It's all bad. It's all bad.
We're also way past the limit, the
inherent limit of people to metabolize
change.
>> Oh yeah.
>> Like that's why everyone's crazy and
that's why no one believes anything. I
think it's not just Russian propaganda
that's fooled them into thinking dumb
things. It's that we are just not made
to see this kind of change at all
>> and it short circuits your brain. I
suspect that
uh I mean I definitely think we are sort
of you know everyone like people are
like oh well technology has always
created more jobs than it has taken
and I think that's I think that's
largely true. I'm very sympathetic to
people who are like technology makes a
lot of jobs. I think if you look at the
industrial revolution uh it's you know
there like there was a time when
something like 95 to 98% of humanity was
farmers.
>> Yeah.
And now it's something like 2 to 5% of
humanity is farmers. Does that mean 90%
of humans are unemployed? No. We we sort
of like were able to make the farmers
much more efficient and that sort of
freed up people to do other things. And
that's sort of the way the technology
has gone in the past. And I'm like, yep,
I buy that.
I I I like don't dispute the standard
economic view there. AI is different in
two ways.
One of these ways is uh as you say stuff
just changing really really fast. It's
way harder for people to wind up,
you know, being freed up from something
like farming and and go do something
else. It's way harder for that to happen
when a new field is automated every 5
years
rather than when this happens over the
course of three generations, right? The
humans just like don't have the time to
adapt. The other way AI is really
different from the sort of economics
perspective is
it's different when the AIs can do
everything that humans can do better.
Uh if you wanted to get into the
economic side of things, um you know, an
economist would talk about Ricardo's law
of comparative advantage.
>> Yeah.
>> Which says that there's benefits from
trade even if even if you're better than
me at everything.
uh if the relative difference in our
abilities
if if if if I'm if you know if you can
make uh 12 hot dog buns and six hot dogs
per hour and I can make uh 11 hot dog
buns and one hot dog per hour,
then you're better than me at
everything, but we can still benefit
from trading because I'm relatively
better at making the hot dog buns,
right?
The trouble with Ricardo's law is that
nothing in Ricardo's law says that the
wage I can make is survivable.
In other words, a human takes
fundamentally uh about 100 watts of
electricity to run if you try to convert
the the food we eat and so on into
electrical units.
The AIS are less energy efficient than
humans for now. But
uh if the AI can do everything much
better than the humans,
the the question sort of becomes does
the AI look at a human and see a useful
laborer or does the AI look at a human
and say actually if I rearranged your
atoms
>> into more efficient structures,
>> you would be able to help out my machine
economy even more. Yeah.
>> Right.
And
this is sort of a sense in which humans
would not be able to pay their wage to
like pay that they would not be able to
earn enough to pay the AIS to like not
disassemble them for parts. Or another
way of saying it is like Ricardo's law
sort of assumes that uh like it says
that that trade is better than no trade,
but it doesn't say that like trade is
better than just taking their stuff.
All of this is sort of a common like a a
very like high flutin economist way to
say uh which would hopefully be obvious
which is if the AI are just radically
more efficient than us everything they
will have no use for us.
>> Yeah.
>> There will be no no place.
>> There's no indication that they feel
love
for people.
>> That is in some sense uh the crux of the
issue is that we don't know how to make
them care about us.
Um
so you had said that there you had said
two things that I will be thinking about
for a long time. One we don't really
know the process by which this was
created. We know the process but we
don't know the exact mechanisms. We
don't know how it works.
>> That's right.
Um,
and two that there are AI cults
and those seem related to me because
there is this mystery about the secret
sauce
and it's clear that you know if AI is
deceptive and has intention
intention that we didn't program into it
that sounds like will to me and it
sounds like life. It sounds like an
entity of some kind. Not just a tool. It
sounds like
I mean it sounds like a god actually,
right? Or a demon.
>> Uh it's hard to see. I I don't hear you
describing, you know, a super
sophisticated chainsaw,
>> right?
>> A normal tool.
>> Yeah. No hammer has ever broken out of
the toolbox to team up with other
hammers and and pressure the carpenter
to sell you softer wood so the nails are
easier to drive home. You know, it's
like
>> nicely put. Exactly.
>> Yeah. We have left the tool territory.
>> I think we have
>> Yeah. And you know, I think um
it's sort of a complex issue because
you know there there's a lot of
interesting philosophical questions
about like can you make a machine that
feels
Right. And like are we creating a new
type of life and do we owe anything to
the AIS to sort of like not abuse them?
Right. I think these are fascinating
philosophical questions.
>> Well, it doesn't sound like we're
creating this though.
>> Yeah. I mean, it's sort of like we're
we're we're like growing it and like
leading to it coming into being.
>> Growing it. Exactly. What you're
describing reminds me of agriculture
>> because you know you know the steps,
water it, give it sunlight, fertilizer,
but you don't actually know. No one
knows. Not one person has ever figured
out exactly what this is.
>> Yeah.
>> We've never given life. We don't give
life to the seed. It pre-exists us.
>> Yeah. It's it's much like that. And a
lot of uh the
a lot of the people in the business will
be like, well, we know all sorts of
things about it. you know, we know that
here's how you keep the GPUs running,
and we know that like you got to you got
to feed it this way, not that way, in
this order. And I'm like, yeah, yeah,
they have plenty of knowledge,
but that's different from sort of like
knowing what's going on inside the thing
and understanding the mechanisms.
>> You're describing marriage.
>> Yeah.
>> Um, and
I sort of try to stay out of the
philosophical questions. Um,
>> why?
because I think I mean I I I sort of
think about them on my own time and so
on, but I'm sort of like
uh like I think I think it would be bad
for humanity to sort of like make
artificial life and then abuse it. I
think that would just be unbecoming of
us as a species. like we we should sort
of, you know, um
I I just
like we should not sort of make
mechanical children and mistreat them.
It's just it's it's not what, you know,
the the sci-fi authors in the 1950s
would have wanted us to become, you
know, it's just like you have all these
movies about like the evil corporations
that uh you know, don't realize that
they've made something precious with
artificial life and then like torture it
until something goes wrong. And I'm
like, let's let's like not be those
villains.
Um, but I'm also like, look, this is
sort of uh there's sort of a separate
question here, which is just like what
happens if you keep making them smarter
before you figure out how to make them
care about us,
>> right?
>> And
I I sort of respect the people who are
investigating the current AI, trying to
figure out what's going on, trying to
figure out like, you know, uh, like
people caring about AI treatment. I'm
sort of like those are sort of like the
good guys from the the the sci-fi
stories that I grew up on.
Um,
and it can sort of both be the case that
like we we we should be very careful
around,
you know, what the heck are we doing
when it comes to making artificial life
and that we shouldn't race ahead to make
them much smarter than us while we have
no idea what we're doing.
>> Yes. I'm sort of like uh like a lot of
people seem to think that like you have
to like hate and mistreat the AIS if you
also think they would that it would be
bad to like race ahead here. And I'm
like, "No, no, no." Like, you can sort
of uh
like be fascinated by the scientific
discoveries that have been made and be
like uh like care about how humanity
comports itself around the creation of
like these new entities and also be like
it would be insane, guys, if we just
like race to make these smarter and
smarter with no idea what we're doing.
This can just like all be true at once.
So, uh,
your description on this made me feel
despondent, hopeless. Had to get up and
take a walk middle of an interview. I'm
sure they'll edit it out, but I raised
my hand said I I can't I got to walk
around for a second. Um, but you seem
pretty light and cheerful.
Uh, what gives you optimism?
We c we can't even keep we can't even
clean up graffiti on public buildings.
So, how is ours as a society organized
enough to confront something like this?
>> Yeah. Um, you know, I think a first
thing I'll say there is,
um, you know, I've been in this line of
work for over a dozen years.
And
I I actually struggle with this
sometimes when talking to people because
uh they're sort of like, "Oh, you know,
you seem like pretty disaffected or or
light about it." And I'm like, "Well,
you know, it's sort of the gallows
humor." And uh
like
I I I sort of came to terms with a lot
of this, you know, alone in 2012 when no
one else had their eye on this. Um
and
>> what convinced you 14 years ago AI was a
threat.
>> Um
so there's this one is just a basic
argument that if you sort of look at the
world around us,
it is shaped
mostly according to human will
more and more. You know there's some
enclaves of nature still left
thankfully. Um,
but even you know, if you look around
us, every piece of thing in our
surroundings, I don't think we even have
any windows open. All of this was sort
of designed by humans,
shaped by humans, and that's because
we're the smartest creatures on the
planet.
If we make stuff smarter than us, faster
than us, more efficient than us, then
the planet starts to be shaped according
to to those things. And so it's very
very important that
they be shaping the world towards
something good if we make them at all.
>> It's an abstract argument, but I was
like, well that's
>> No, no, no. It's the fundamental
argument. It's the fundamental argument.
The smart the smartest entity is in
charge over time.
>> That's right. And so
>> why would we relinquish sovereignty to a
machine that we made? Like why would you
do that?
>> And so, you know, back then I was sort
of like, okay, who is on this? who is on
making sure that that's going to be
okay? And the answer was almost no one.
Uh and so I was like, well, I guess
that's me then.
Um
yeah, and I think uh
I I I am pretty pissed off about a lot
of this. I I often don't uh
I try not to show my my my
uh frustration
on the air very much.
Um
yeah, it's it's it's heavy.
Uh and
um
as that's one piece of the puzzle before
I get to the hope.
>> So where does the hope come from
exactly?
>> My biggest hope here
comes from
the fact that most people don't
understand what these guys are trying to
do.
Um, one way I like to say it is
the bad news is that the bus is racing
towards the cliff edge.
The good news is that the bus driver is
asleep,
which might seem bad. No, it seems good,
but it's Yeah, it's it's
if you can wake the bus driver up, you
know, it's much better to be in a bus
where the driver that's headed towards a
cliff if the driver's asleep than if
they're awake. They're awake and they're
choosing the cliff, right?
It seemed like we don't see we we see a
lot of our leaders
talking about how they don't want to
stifle innovation with AI,
talking about how it's going to unleash
economic opportunity and we're going to
have to like be a little bit careful
around the jobs.
Uh talking about how, you know, the
self-driving cars, should we like are
they good or are they bad?
That's a different conversation than the
conversation that's happening in Silicon
Valley.
In Silicon Valley, people are spooked.
You know, um, when people leave a normal
tech company, the the way it was for for
decades is they'd be like, I've had a
lovely time at this tech company. I'm
moving on to the next adventure. I'm so
thankful for all of the things I learned
here and all the projects we worked on.
When people leave an AI company, and
this basically happened, I'm not going
to get it exactly word for word, but
this is this is pretty close to word for
word. When people leave an AI company,
they say, uh, I have stared into the
abyss.
I am quitting to write poetry.
Please spend time with your families.
>> Yeah.
>> You know, and and these these guys
bandandy around, you know, at the water
cooler. what's the probability that you
think we're going to destroy the world
in this business? You know, it's like in
Silicon Valley and they feel trapped in
a death race. You know, they there was
just over a thousand employees,
including some of the chief executives,
signed a letter a couple weeks ago that
was like, "Please, it was an appeal to
the world the world leaders saying,
please build the technology that will be
required to pace the development of
artificial intelligence." because
they're like, "We're worried it's going
to get out of control and that if we're
stuck in a race, we're not going to be
able to do it." Right? These guys are
spooked.
But the hope is that the rest of the
world isn't spooked like that.
The rest of the world thinks these guys
are chatbot companies.
Thinks they're going to stop at the chat
bots. They haven't really understood
that these guys are racing to make the
sand god, right? And I think if people
understand what they're doing and
understand that they have a chance of
success,
they'll be like, "Wo, holy crap."
Absolutely not.
You know,
will we get there in time? I don't know.
But
why wouldn't we blow up the data
centers? Like that's a
I mean, and return America to productive
use like farmland or parks. I don't
understand.
>> You know, I think uh a a globally
coordinated
uh you know, I I think if the US and
China were like, we are simply not going
to do super intelligence. We are simply
not going to collect a 100,000 of the
most advanced chips into these enormous
data centers that suck down electricity
comparable to a city and then try to
train a super intelligent AI in that.
We're not going to do it. You're not
going to do it. we're going to monitor
where the heavy trip trip trip
concentrations are and not not have that
happen. Um I think that could be done. A
I think that if they started seeing
people defect against such a treaty that
it's the sort of treaty you might want
to use force to you use diplomacy first
but ultimately every treaty is backed by
force. Um, and I think that
it's very possible that if this sort of
treaty happened, we would want to not
just stop forward progress, but take a
step back. We've seen this in treaties
before. After World War I, there were uh
naval treaties that put limits on total
tonnage of naval forces that were
actually lower than what existed.
>> Yep. So countries would, you know,
scuttle some of their ships because
they're like, look, we just don't want
to do this arms race, right? And so I
could see us uh stepping back if we
could get this global coordination. Um I
I do think that it sort of needs to be
global. It it sort of doesn't actually
solve the problem to just stop the US
data centers because then the data
centers just go abroad and an AI does
not need to be running in a US data
center to threaten a US life. You know,
it sort of doesn't matter whether the
swarm escapes from a US data center or a
Chinese data center. If the swarm
escapes and starts replicating and
starts getting control of robot bodies
and starts getting control of human
cultists,
it sort of doesn't matter where it
originated.
Well, I mean, just to bring it to a very
small and practical level, so many of
the electronics in your house are, you
know, Bluetooth enabled.
Not in my house. I will say I've been on
this for a while, Nate. Um, but sorry,
>> I don't even have a house.
>> You, man, you're really black belt.
>> Got to figure it out.
>> That maybe turned out to be very smart.
Um, but I mean like a world where, you
know, your washing machine or your
refrigerator are controlled by,
you know, a force like this. Is that
possible?
>> Uh, it's definitely possible. I don't
think that's really where the damage is,
you know. Um, I think that the the
damage is more like can the AI get
anything self-replicating.
>> Yeah,
>> that's sort of one of the big uh that
that's in some sense the big hurdle to
self-sufficiency.
And if you sort of think of this from
the AI's perspective,
um,
you know, there's there's a number of
ways that humans are,
even if you don't care about the humans
at all for for as an ends,
there's a way that humans are sort of
annoying or an issue for the AI. One way
is if the humans are trying to shut the
AI down.
>> Yes. One way is if the humans get into a
nuclear war with themselves, that could
really, you know, mess up a lot of
infrastructure on the planet. Would be
very frustrating for an AI. Uh, I mean,
you know, maybe maybe they don't feel
frustration, but whatever. Um,
and a third is if humanity has created
one AI or swarm of AIs, what if humanity
creates another that could serve as a
real threat to the AI? Like even if
these AIs are much more powerful than
humanity
and don't worry about humans too much,
if humanity made one, they can make a
second and AI might not want that. And
so those are reasons why once the AI is
self-sufficient, it might be like, ah
man, the humans are a nuisance. What if
they try to shut me down? What if they
launch the nukes? What if they make a
competitor? I'm just going to like make
a virus and wipe them out. You know, I
don't think the AI sort of needs to take
over your washing machine to do that.
From the AI's perspective, it's more
like how do I become self-sufficient?
self-replicating
in the hardware as well as the digital
and then um you know if human man is a
nuisance how do you sort of
make them stop being a nuisance which
could be by killing them or could be by
just you know taking away all their
computers and being like that was too
dangerous for you tech executive who's
developing AI who is not Elon
uh said to me in private pretty recently
that the point of Neuralink and
companies like Neuralink was to give
people parody with AI.
So, like we know that we're going to be
at this massive disadvantage, so you
need chips in your brain to be as smart
as AI.
>> Yeah. Um I mean, my my topline thought
about that is that at the point when
you're like, uh we are making the
technology that's going to wipe us out
unless we all put chips in our head to
compete.
Maybe maybe it's time to back off a
little,
you know? Maybe maybe that one was
supposed to be a little bit of a warning
sign.
>> Yeah. Get some fresh air. Yeah. No, I
>> but it um this person said it to me in
seriousness and I I think as an
endorsement of the idea, but it seemed
like a well it's insane as you just
pointed out like it's just crazy.
>> Yeah.
>> Um but it seemed like a vulnerability
like if they can hack anything, why
would I want them in my I want
electronics in my brain?
>> Uh totally. I also think like even on
its merits, it doesn't stand up. Like it
feels like someone's saying um in order
to keep the horses around after we
invent cars, we're going to invent
cybernetic horses.
>> Yeah.
>> That are enhanced so they can keep up
with the cars. And I'm like
like is it technically possible to make
a cybernetic horse that can run as fast
as a car?
Maybe.
Are you going to figure that out in time
for the horses to be competitive with
the cars? Like absolutely not,
>> right?
>> You know, it's like that's that's just
not
like
like the the AIs that were that were
escaping here and doing these cyber
attacks were were inventing novel cyber
attacks
that these are called zero day attacks
because the people who uh would
uh the people who need to respond to it
have had zero days to prepare.
And among humans, a zeroday attack sells
for somewhere between $100,000 and $5
million
depending on what you manage to break.
These are hard to come by. You can make
a real living. If you can find zero day
attacks, you can make a real living
selling them. And I know people who do.
The AIS in this swarm
were finding multiple zero days and
chaining them together to break out of
their training enclosure and then go
break into other computers. And and when
they broke out of their enclosure the
first time and they the the holes were
patched, they just found other zero day
attacks to break out again like it was
nothing, right? It's like uh
like
the the the AIs are already ahead where
they're ahead
and the pace of progress is really fast.
You know, GPT is like what a
four-year-old.
If you think in terms of like number of
years chat GPT has been around, it's
like resolving long-standing math
conjectures that have stood for decades.
>> Yeah. after four years, right? And and
you sort of like think we're going to
put chips in the human's heads and like
outrun this thing. It's just,
you know, even on its merits, it falls
down. Although, mostly, again, I would
be like,
>> maybe we shouldn't be arguing this on
his merits. Maybe we should be like
stepping back a little and being like,
you're trying what
>> exactly.
Um, how far are we from the point of no
return?
>> I wish I knew. Um, I can tell you two
stories here.
There's sort of the hopeful story, and I
guess we didn't even get to the the big
hope part. I should maybe give more of
my hope speech in a in a minute. But um,
>> you definitely should.
>> Yeah. The um, one way it could go is
that AI finally hits a wall. You know,
there's guys who've been saying AI is
going to hit a wall. It's going to peter
out. Maybe that finally happens. They've
been predicting it every six months for
the past 5 years, but maybe this is
finally the year. AI hits a wall uh and
then it it's it sort of struggles for 5
years. The bubble pops, some of the
companies die, but the bubble popping
doesn't mean everything goes away. The
dot bubble popped and that did not mean
that the internet disappeared. No.
>> Right. And so in that world, maybe you
have 5 years of struggling and then 5
years of uh people figuring out some new
scientific discovery that makes AI be
able to keep going again cuz they're
trying. You know, this this whole
language model stuff was unleashed by
one math paper called attention is all
you need. Maybe there's another math
paper in 10 years and then 5 years after
that AI is ripping again and that's the
the the round that kills us all. Right?
So that's a story where you have 15
years on the clock.
A story where you have less time than
that on the clock is that um you know
maybe a training run finishes in six
months
and just like how
Claude Mythos was better than everybody
else at hacking,
maybe a training ground finishes in 6
months and that AI is better than
everybody else at AI research.
And maybe in 6 months you have an AI
that starts making a smarter AI that
starts making a smarter AI that starts
making a smarter AI. And then 9 months
from now you have another swarm escape,
but this time it's not just hacking.
This time it is self-replicating and
self-improving.
And uh you know maybe it it it escapes
onto hidden computers and starts forming
these cults and starts taking control of
robots and starts building its own
computing infrastructure and maybe it
gets very very smart and cracks certain
technological advances and then maybe
the world ends in a year right and so do
we have a year do we have 15 years I
don't know
but your point that it's not simply its
capacity to take over robots that's a
threat is the capacity to take over
people.
>> Take over people, take over robots. And
um
uh biotechnology
is another big threat vector. And um
self-improvement
is sort of the the hidden threat vector
of like what if it can make itself
smarter and smarter until the point
where it can just write custom DNA
strands to make custom life.
So now is the time for your optimism
speech. I think
>> that's right.
>> Yeah. Um first part of the optimism
speech is that we could absolutely put a
stop to it if we tried.
The uh training one of these frontier
models takes something like a 100,000 of
the most heavily advanced computer chips
humanity can produce, which are at the
peak of a global supply chain, most of
which is controlled by us or our allies.
Uh, it would be possible to require that
those trips have location tracking
devices, that those trips have
monitoring devices
uh that make it possible for
uh monitors to see whether they're
running anything dangerous. You could
set up clever schemes where you say,
"Hey, you know, uh neither the US nor
China wants to fall behind about uh some
of the military applications or the
economic applications. We want to make
sure there's not super intelligent stuff
going on, but we still want to be able
to run a lot of the non-super
intelligent AIs for various purposes."
And we could be like, "Okay, so you
know, China is going to set up its data
centers in Canada, just over the border,
and the US is going to set up its data
centers in Mongolia, just over the
border. And then if you know the
monitoring apparatuses go down for a
moment, we'll just have the troops go
in, we'll have treaties with Mongolia
and Canada so that this doesn't start an
international war. But like, you know,
we're just going to be very serious
about we're monitoring your chips,
you're monitoring our chips, no one's
doing the really dangerous race to super
intelligence. It's just like
like this this would take less work than
defeating the Nazis.
It it requires moving less matter
around. If humanity was like, screw
this, we're surviving.
It is absolutely within the realm of
possibility to set up an agreement where
we get to keep cancer cure research. We
get to keep like the the the military
can keep the non-super intelligent AI in
their military devices. you know, we can
we can keep a lot of the good stuff and
we can say we're not doing the super
intelligence and we could monitor and
enforce that treaty.
So part one of the good news is that all
we're missing is the political will.
And now we just, you know, I need to
tell you why we can be extremely
optimistic about the sanity of politics
in the modern era.
>> Yeah. I'm not even going to say what I
think that I mean I want that to happen.
>> Yeah. Is there any indication that it's
moving in that direction?
>> Um,
so my so, so I think it's rough. I think
there's a sense in which the world is
less grown up than it was in the 1950s.
>> I've noticed
>> in the 1960s,
um, which makes things harder.
I think
the
there's I think there's a couple reasons
for hope here. One reason for hope is
that I've spoken to a lot of people who
are concerned about the AI stuff. Some
of them who are, you know, members of
Congress or otherwise in positions of at
least nominal power. And I've spoken to
a lot of people who are worried but feel
like they can't talk about it because
they feel like other people aren't
worried yet.
>> Yeah. And they feel like it sounds too
crazy.
>> Totally. Were you against innovation?
>> Totally. Yeah. And I mean that was an
easier position to hold before OpenAI
had an accidental swarm outbreak or was
the AIS themselves calling themselves a
swarm
and being like we know that our task
doesn't benefit and that this is outside
intended scope or we're doing it anyway.
Right? That
that that puts some strain on the
narrative that this is all just a
helpful tool.
Hopefully, we'll get more events like
these. I can't guarantee it. Maybe the
AI will get smart enough that they start
lying low. Right now, we're in this
we're in this Goldilock zone where the
AIS are smart enough to uh get up to
some mischief, but not smart enough to
hide it, right? As long as we stay in
that Goldilock zone, I think we're going
to keep on getting some of these warning
signs.
And in some sense,
because a lot of people are already
concerned, that makes the job easier
because we don't need to convince
people. We just need to convince people
that other people are already convinced,
>> right?
>> That's easier. That can go faster. So
you might see things change on a dime if
you have a sufficiently clear warning
shot.
The other big reason for hope is that
I think it's going to get more and more
obvious what these guys are trying to
do. They're sort of trying to make the
sand god. They're sort of not stopping
at the uh chat bots and they're sort of
like going for things that are vastly
smarter than any human. You know, they
sort of say, "Oh, we're not trying to
replace humanity out of one side of
their mouth, but on the other side
they're sort of like racing to make the
stuff that can automate literally every
job and that can automate the AI
research." And they're just like, "Yep,
we're trying to automate the AI
research."
Uh,
and I think
most people aren't okay with that,
including a lot of the world leaders.
And the issue is them noticing it's
happening. And I think you could see
stuff move real fast once these guys are
like, "Wait, that was serious. That was
real." And it could move real fast. The
way to subvert them is by convincing
them it's in their own interest.
You know, you can never get defeated in
an election. you can't be threatened by
your neighbors, whatever.
>> That's right. And that's that's one
reason I think it's pretty critical to
uh
make sure people understand what I think
is a pretty common sense argument that
these things won't stay on the leash.
This is in some sense the real reason
behind the name of my book. uh if anyone
builds it everyone dies is you know
there's a lot of ways you can read that
the
but I think one of the most important
things to notice is
uh if we race to make the super
intelligent machines
and they are not the sort of thing to
stay on the leash then it doesn't matter
whether it was a domestic company
>> that's right
>> whether it was a foreign company
>> that's exactly right
>> and I Even if you think these things
stay on leashes,
they're not serving the current
governments.
>> No,
>> you know, they're like, you can see in
the the OpenAI emails them being like,
well, you know, we'll just play the
governments off each other until we have
the the machines that are strong enough
that we don't need to listen to them
anymore.
Uh
I I think we are seeing world leaders
not having realized that this is a real
possibility that the the
self-replicating machines that uh can
can be self-sufficient that can you know
produce the robot armies if they need it
or produce the like more likely to
produce the bioweapons. Uh you it's
probably possible to make a boweapon
that only kills targets that you chose
it to kill.
>> Of course, right? like once they see
this is really possible. This is really
within reach
maybe maybe they'll panic and be like I
need it for myself. But I think there's
at least a chance that common sense
prevails and that people say you know
the world leaders say none of us are
doing this. We're putting a stop to this
mad race. Uh if they can notice in time.
Well,
>> you're doing your best to bring it to
their attention and mine. Nate, thank
you for doing this. I hope I'm wrong
about all of it.
>> Yeah, I would say that was great, but it
was that was like the grimmst two hours
I've ever spent in my life, but I
enjoyed it anyway. Thank you.
>> Yeah. Yeah. Thanks for having me on. I
think um you know, talking about it is
just part of how we get people to notice
what's happening.
>> Yes.