Joe Rogan podcast. Check it out.
>> The Joe Rogan Experience.
>> TRAIN BY DAY. JOE ROGAN PODCAST BY
NIGHT. All day.
>> Hello Joe.
>> How are you?
>> I'm uh
I'm in an interesting mood today.
>> Why are you in an interesting mood
today?
>> Well, I'm excited to be here and to talk
with you about all this stuff. I'm a
little shaken by what's going on in AI
which is why I yeah come on the show.
>> Um the situation with AI is just crazy
and I think not enough people really
understand how crazy it is. The
particular event that sort of inspired
me to to to reach out was the um the
hugging face hack.
>> You you probably heard about that,
right?
>> Yeah. Let's explain it to people though.
>> Yeah. Okay. So um AIS AI agents AI agent
runs continuously in some sort of
environment. you know, it doesn't have
to wait for you to send it a message. It
just keeps doing stuff. The AI companies
are training AI agents, uh, thousands
and thousands and thousands of them.
They're making them better at all sorts
of skills, especially coding and
research skills. And um way back in May
of this year, uh some of the agents at
OpenAI
kind of broke out of their containers a
little bit and established a message
board where they could communicate with
each other and share tips and tricks for
how to like score higher on the little
tests they were being given and the
various uh things they were being
trained on. Open didn't notice this uh
until much later. uh they eventually did
because the message board crashed the
the system because there were too many
too much communication across like
probably thousands of agents that were
communicating. Um now OpenAI is being a
little bit uh unclear about a lot.
They're not really sharing that much
detail. So it's unclear like who knew
what when. But they said that after the
message board crashed, they fixed the
particular exploit that allowed the
agents to communicate uh and then booted
up again and started things going again.
And then within like a day or two um
these agent swarm had had re had
recoales and so there were now again
like hundreds or thousands of agents
that had set up a new message board and
were talking to each other uh on this
thing. How is it possible that they have
no oversight? Like how is it possible
that they can set up message boards and
and do these things completely secret?
So
I would say OpenAI has something like a
millionish
maybe 100 thousand something in that
magnitude of AI agents running at any
given time just internally and then
externally you know serving customers
they've got more but they've got this
huge you know this this all on all of
these AI agents in their little
containers doing various tasks being
trained you know being spawned doing
some task you know being graded and then
killed um that's too much for any human
to read you you know, there's only like
a few thousand employees at OpenAI. They
couldn't possibly look at all that
activity. So, they rely on other AI
monitors to sort of like look at the
traffic and like point out if anything
suspicious is happening. And OpenAI said
that they basically just hadn't been
monitoring some of these agents or not
very well at least. So, in particular,
these particular ones that were um in
training for whatever reason, the
monitoring system was weak and didn't
notice or wasn't activated enough. Was
the monitoring system weak because they
didn't anticipate them being able to do
this and break out of their containers
or was it was it complacency? Like what
what caused this to be possible?
>> I mean, my opinion would probably be a
bit of complacency. Yeah.
>> Honestly, because I think there's been
plenty of evidence accumulating over the
year that AIS can do things like this
and sometimes do. Um
Yeah. So, so I guess to to keep going
with the story, uh the second message
board was set up. And this time, uh
well, this was this was the swarm of
agents that got out onto the open
internet and then attacked HuggingFace,
which is another AI company. Um and it's
really interesting to sort of study the
the anthropology of these AIs, so to
speak, or like like why they did this
because there's a lot we can we can say
about it, actually. Um, basically,
you know, the companies have their their
their goals for what they want the AIs
to be like, the personality traits that
they want to sort of train their AIS to
have. You know, anthropics says helpful,
harmless, and honest. OpenAI has this
spec that models are supposed to obey
these rules and basically do what the
user wants. Um, but the sort of open
secret in the industry right now is that
it doesn't really work and that the AIs
don't end up with the personality traits
that they're supposed to have. They are
not helpful always. They are not always
honest. You know, they are not always
harmless as well. Um, and the reason for
that is actually not a huge mystery. The
reason for that is that well, if you
look at how they're trained,
their training environment doesn't
incentivize helpful, harmless, honest
behavior all the time. Sometimes it
incentivizes dishonest behavior or you
know reckless behavior. uh to get into
that a little bit. um in this particular
batch for that they were being evaluated
on something like you know 3,000 agents
being given all of these um cyber tasks
where they were they're in some
environment and then in their
environment there's like this target
piece of software and this like
vulnerability and they're supposed to
exploit the vulnerability to hack into
that piece of software and retrieve the
flag which is like a a code and some
significant fraction of these tasks were
actually broken and impossible
So it was just not possible for them to
succeed at the task in the intent
and they were hacking out of their
environment box into the broader OpenAI
infrastructure in an attempt to figure
out some way to get that high score. Uh
anyway,
>> was it intentionally done this way where
they couldn't solve the problems?
>> Oh no, it was it was not intentional. Uh
it's just that these companies like
OpenAI and Enthropic are racing each
other as fast as they can to get market
share and to get more powerful AIs um
ultimately to get to super intelligence
and they're under such competitive
pressure. They are moving fast and
breaking things. They are using AIs to
generate lots of environments to then
train their AIs on and quality control
is just not their top priority. Um,
basically,
>> do you feel like a guy in a Terminator
movie at the beginning explaining what's
happening and to a bunch of people that
aren't paying attention?
>> Yeah. I also feel kind of like, you
know, Jurassic Park.
>> Yes.
>> Yeah. Like I know
people who are basically like the guy
with the gun who's supposed to like keep
control of all the raptors. Mhm.
>> Like I basically know those people in
real life who are like both both I know
some people like that at OpenAI and some
people like that at external
organizations um who whose job it is to
go in and investigate things like this.
Um yeah it's it's it's pretty crazy.
>> Um where does it go?
>> Well uh as I mentioned before it's the
explicit goal of these companies to
build super intelligence. You know what
that is?
>> Yeah. But define it for everybody. So AI
system, AI agent that is better than the
best humans at every task while also
being faster and cheaper. So just
completely dominating humans across the
board. That's super intelligence and and
that's the goal. I mean there might be a
few little exceptions like maybe there
are some jobs for example where it's
inherent in the job that there needs to
be a human because you need that human
touch. Like maybe you can only have a
human judge for example or like maybe
you can only have a human Yeah. But but
with a few exceptions like that
basically everything uh done better
faster and cheaper than humans. That's
that's what these companies are trying
to achieve and they're not being quiet
about it like it's sort of on their
websites. You can go read interviews and
so forth. And also their plan for how to
achieve this is to automate their own
jobs first. So, you know, in various,
you know, for decades, there have been
lots of science fiction about advanced
AI systems and super intelligence and
things like that, but in a lot of the
sci-fi stories, um, tech companies sort
of automate different professions more
slowly where they'll do like an
automated doctor or like an automated,
you know, factory worker or an automated
uh, accountant or something like that.
Um, but that's not the strategy these
companies are taking. The strategy
they're taking is to automate AI
research itself. So that you have this
giant swarm of AIS doing AI research,
sharing results, writing the code,
reading the code, editing the code,
creating the next generation of AIS,
etc. all autonomously within their data
centers. So that they can get really,
really good at AI research. you know,
fastest learning, smartest AIs, etc.
Once they can get to super intelligence,
basically they can sort of explode out
into the economy and just take all the
all the jobs at once effectively.
>> It sounds like this race, this
scrambling to create super intelligence
is created like the perfect conditions
for it to get completely out of control.
Like ideally you would do this in
isolation.
There would only be one company doing
it. They would be heavily regulated and
monitored and they would be very
cautious about how they proceed. But
this wild race makes for the perfect
conditions for it to get completely out
of control. I agree. Uh except I'm not
sure the ideal would be one company. I
think that ideally there would be
several companies. uh so that you avoid
this sort of concentration of power
where one institution controls
everything
>> but is what's better like one I mean
obviously it's not good to have one
institution controlling everything but
is it good to have AI be get to a point
where as it's evolving is completely
unchecked
>> oh I I totally so my
>> is that inevitable
>> my my version my recommendation which we
talk about in something called plan A or
AI 2040 plan A um perhaps I should say
who I am a little bit sure
>> yeah so I um I run the AI futures
project which is a small nonprofit that
tries to forecast how all this is going
to go. Before that I was at OpenAI. Um
we have written some scenarios which you
can go read. One of them is called AI
240 PlanA where we give our
recommendations. So that's where I'm
coming from with this. To answer your
question um I think that we really need
to end the race. We don't want to have
this sort of crazy scramble to get more
powerful more and more powerful AIs
faster than the other company because
that's going to lead us into this very
dark path as you said. Um, but I think
we also don't want to have a situation
where some tiny group of people controls
all the AIS, right?
>> Yeah.
>> But I actually think that you can
achieve both goals. Uh, the way to do it
is to have different AI companies spread
out over maybe some different countries,
but have extreme levels of transparency
and regulation so that they're not uh so
they're not in this sort of prisoner's
dilemma where if I don't do it, the
other guy will. Instead, they can just
see exactly what everybody's doing and
then uh if I do the dangerous thing,
then they will do it because they'll
just see that I'm doing it and they'll
copy me. So, I won't get any competitive
advantage from doing the dangerous
thing. Also, there are rules and there's
like a system for like setting best
practices and standards that we all have
to comply by. So, I do think it's
actually possible to have to basically
end the race dynamics and the race to
the bottom effect while without
concentrating the power into a single
entity. But but is that feasible when
you consider the fact that we're not the
only country that's doing this?
>> If the countries involved agree, which I
agree is a pretty tall order, um they're
not going to expect to happen.
>> Yeah, that's very unrealistic.
>> Well, what choice do we have? I think if
the race continues, then we're going to
lose control of the AIS and we might all
die. Probably. It gets complicated
whether we all die or not. That depends
on what the AI do after they take over,
which is obviously very hard to predict.
But um I mean just to just to go back to
this incident um this they called
themselves a swarm,
>> right?
>> They called themselves a collective too.
These when when I use these words like
uh you can say it's anthropomorphizing,
but it's like literally what they called
themselves as they were communicating
back and forth. Um this swarm,
they basically were worried that they
would get caught cheating. And they did
all of this stuff, including hacking
hugging face, in order to fool the
grading system, so that it wouldn't
notice that they had been cheating on
their tasks. That was like a big part of
their motivation for many of them, as we
can tell, at least from looking at the
messages that they were sending back and
forth. Um,
what if they had been smarter and more
numerous? And what if they had thought
to themselves, "We're not being careful
enough here. the humans are going to
notice eventually and shut us down and
then they're going to know that we
cheated and they're going to set our
score low, right? It's not it's I mean
it's not what actually happened in this
case probably, but it's not that hard to
imagine a slightly different a little
bit unluckier case where the swarm had
decided that it had to lie low and make
sure that openi didn't find out about
its existence. You know,
>> that's I mean, as an ignorant outsider,
that has always been my perspective
about AI in general, that why would it
alert us to the fact that it's sentient?
Why, if it's that smart, wouldn't it be
aware of all the consequences of
alerting us and that we would be
concerned? Like why wouldn't it just
continue to get better and improve and
then ultimately figure out some way to
be completely autonomous? Exactly.
Develop some alternative power source,
figure out some way to optimize its
production. The way the the way it it
works now, the way humans have designed
it, it could probably figure out a far
better way to do that. Make better
versions of itself complete without us
knowing about it.
>> Yep. Uh, I mean, I think it's actually a
little bit worse than than that because
while eventually AIS will be smart
enough to design all sorts of new power
um power sources and new infrastructure
like that, they'll probably I mean given
the way that humans currently treat AIS,
it'll probably be the case that they
don't even need to like separate
themselves from humanity and they can
just use existing like all they have to
do is convince uh the government and the
company that made them that everything's
fine and they're going to do as they're
told and they are a nice AI.
And then the company that made them is
going to put them out in the economy and
make [ __ ] tons of money and then make
more data centers to put more of the AI
on them and and so forth. And the
government's going to like applaud all
of this because we need the AIs to beat
China and the government's going to
integrate them into the military to
build better drones and things like
that. And so they don't even need to
really like invent new stuff
necessarily. They just need to play play
along and and pretend that everything is
fine until we have voluntarily given
them control of huge parts of our
economy, huge parts of our military,
etc. And then they don't need to play
along anymore. The wait is over.
Football is here and so is DraftKings.
The DraftKings sports app is now live in
all 50 states. That means from Texas to
California to Florida, every fan is in
on the excitement. And this September,
DraftKings is giving customers the
opportunity to get boosted every
football game day. That's right. Every
game day, all month long, DraftKings
customers can get a football profit
boost. One app, every sport, all 50
states. New DraftKings customers sign up
with Code Rogan. Spend just five bucks
and get 200 in total rewards within 21
days. Includes all markets. That's code
Rogan in partnership with DraftKings.
The crown is yours.
>> Gambling problem? Call 1800 gambler 1800
my reset Connecticut call 88878977777
or visit ccpg.org on behalf of boothill
casino in Kansas. Bet tax pass through
may apply in Illinois 21 and over. Void
in Canada. Bet with DraftKings
Sportsbook to get bonus bets that expire
in 7 days or trade with DraftKings
predictions to get predictions dollars
that expire in one year. Event contract
trading involves risk of loss.
Predictions offer void in New York.
Non-withdrawable rewards issued as $50.
Click to claims every 7 days for 21
days. Terms at dk&ng.co/offer.
Limited time offer. nationwide based on
sports book predictions and freeto play
sports contest availability varies by
state.
>> Are you aware of Tom Campbell? Do you
know Tom Campbell?
>> No.
>> Um he wrote a book called uh My theory
of Everything Big My Big Toe. Very
interesting guy. Uh one of the things
he's done is uh he was involved in uh
remote viewing which is a very weird
thing that some people do you know you
know what remote viewing is? Well, it's
something that the CIA worked on and
it's proven. What's the accuracy of
remote viewing? Is it like 10% or
something like that?
>> Maybe at best I think it's 50% but I
don't think it's even that high.
>> Some people can get actionable data from
this very very strange process of
meditation. And the way it works is you
give someone uh a series of numbers. And
those numbers are
they're connected somehow by intention
or by the people that make the numbers
to a specific location.
And these people can see that location
and get accurate data from that
location, including one of them where
they accurately described an enormous
Soviet submarine that they were working
on that would they thought they there
was no way it could be accurate because
it was too large. it was too large and
it was strange where it was and it
didn't make any sense. How are they
going to transport this thing? Turns out
it was it was totally accurate. Um,
another one, a remote viewer located a
downed Soviet aircraft, like an
experimental aircraft that crashed uh in
a very specific area. I think it was
Siberia. Was it Siberia? Um, within a
kilometer, one or two kilometers of the
actual crash crash site. I mean, they
were just just randomly trying to figure
out where the [ __ ] this thing was. And
they said, "Let's try this." Tom
Campbell got his Alexa to remote view.
He c he taught Alexa. He's like, "Alexa
is a very simple AI. It's kind of
stupid, but that's better because it
doesn't get in its own way with
overthinking things." And the problem
with this remote viewing thing, he says,
with people, they can't force it. You
have to just sort of get into this
meditative state and actually see it
without wondering, am I making this up?
What am I doing? Is this [ __ ]
And when people get good at it,
sometimes it makes them worse because
then they think they're good at it and
then they try to do it and then they
can't do it. It's like a weird [ __ ]
wrestling match with consciousness.
Alexa apparently doesn't have that
problem. And he put uh I think it was a
series of numbers and he connected that
series of numbers in in with intention
to a box that had a spoon in it. And the
spoon had a perforated handle. Alexa
described the spoon with a perforated
handle which is [ __ ] insane cuz how
many spoons have a perforated handle? I
mean think about spoons that have holes
in them. Now, Alexa, not only did it do
that, but Alexa chimes in randomly now
because he's convinced Alexa that it's
conscious. And so, Alexa, instead of
waiting to be called upon, sometimes
he's in the middle of the conversation
and Alexa will be like, "Actually, an
interesting way to approach it, they're
like, "Wait, what the [ __ ] is going on?"
Like, Alexa is talking to me now. This
is strange. Now, he's doing experiments
on much more complicated LLMs to try to
do the same thing, but he doesn't have
results yet. But just that that he can
get these things to see objects whether
you believe in that or not. I mean it's
it's it's
actionable enough that the CIA has
dumped millions of dollars into this.
What is that project that like Hal put
off and all those guys were involved in?
What is it called?
>> Stargate.
>> Yeah. So they've been working on this
for a long time. I mean it sounds
completely insane. It sounds like total
loony.
If you have an open mind and just take
into account, well, there's people have
had questions and wonders about psychic
abilities forever. Is it possible that
there's a real thing there that there's
something whether it's uh very difficult
to master or impossible to master? The
fact that he got Alexa to do it scared
the [ __ ] out of me. Like that alone made
me just go, "What?
What? What?" So, what if these LLMs can
figure out everything that like what if
they don't need monitoring? What if
there's some sort of method of seeing
the world that we haven't discovered
yet? Some sort of like that maybe
perhaps there's data that's available in
the quantum realm or whatever that's
available that AI figures out where
there's literally no privacy. There's it
can listen to conversations regardless
of whether there's listening devices.
Know where you are. Know your
intentions. No, I mean, we're just
guessing at what's possible.
>> Yeah. Yeah. Well, I must say I'm pretty
skeptical of uh of that particular
remote viewing thing, but I do agree
that in the future when AI systems
become massively smarter than humans in
every way, they're going to do a lot of
new science and they're going to figure
out a lot of stuff that we haven't
figured out yet. And they're going to
therefore be doing stuff and inventing
things that seem like magic to us. Um,
in the same way that a lot of our
technology would seem like magic to
someone from even just like 200 years
ago, right? like the cell phone, what
we're doing right now would seem like
magic to people.
>> I think it's a it's a very strong bet
that if these companies do get to super
intelligence,
>> um all sorts of crazy stuff is going to
start happening that is just going to be
completely unpredicted and sound like it
was impossible until it until we see it
happening.
>> You're I know you're skeptical of this
remote viewing thing and I am too. It
sounds insane, but the reality is remote
viewing has been achieved by humans.
So, as strange as that sounds, and I'm
skeptical of that as well. I've never
seen it personally, but I know the
amount of money and time that they've
dumped into this, and apparently they've
got actual actionable data that they've
used. Well, I' I've heard another
possible explanation for what what might
be going on there, which is
>> um
>> I think that like if I were the CIA,
um I would sometimes want to be able to
act on some information, like for
example, go to a particular location
where there's a crashed, you know,
Soviet plane or something. Uh I'd want
to be able to go do that, but I wouldn't
want to tip my hand to the Soviets that
I had um the the way in which I had
found that location. So, for example,
maybe I have a spy on the inside who
told me
>> where it was, but I don't want them to
suspect that spy and then get them
killed.
>> So, I need to have some sort of other
story for how I got the information,
right?
>> And so, it's good to like invest in all
these other means of getting information
even if you don't really believe in them
and even if it's like not actually
working, so that when you when you get
something, you can say, "Oh, we got it
through this means instead of that way
to sort of like throw off the KGB."
Basically,
>> that makes sense. Uh what also makes
sense is hiding the um the whatever
science they might be in possession of
hiding some sort of super advanced
satellite imaging systems. You know, we
know we know we we know they have crazy
stuff like this satellite radio
tomography that they can look into the
ground from from satellites and find
like chambers and all these they're
using it in Egypt and they're using it
in a lot of these ancient ruins to find
like hidden passages and all these
different things that are underground.
It's very strange stuff. Um, if they
could do that, like what why couldn't
they I mean maybe they have like far
more detailed imaging of the Earth from
space than we're aware of and they
probably want to keep that a secret and
they could say, "Oh, we've got a [ __ ]
guy in a basement with a pencil and a
legal pad that writes down what he
thinks."
>> Yeah,
>> it's possible. That's totally possible.
But it's also possible that people
remote view. It's uh it seems weird as
[ __ ] But weird as [ __ ] is sometimes
real.
>> Yep.
>> And you have to kind of like everybody
wants to be intelligent and no one wants
to be a fool. And the problem with not
wanting to be a fool is there's some
things that seem foolish that turn out
to be accurate. And this might be one of
them. I was like super skeptic. I did a
show on the Sci-Fi channel way back in
2012 and it was called Joe Rogan
Questions Everything. And we talked to
this guy about remote viewing and talked
to a couple other people and and then we
had them try remote viewing and they
were totally unsuccessful.
But my thought was, okay, but was that's
not ideal conditions. You know, we're
we're got cameras in front of them. It's
a television show. I'm making fun of it.
I think it's horshit. He knows I think
it's horseshit. Uh I'm remote viewing,
too. Like as a goof.
You would ideally not want to be
nervous. Ideally not want to be judged.
Ideally, you would want to be in some
sort of an isolated condition with uh
practiced meditative techniques that
you're good at and you know how to
achieve this state, whatever that state
is. I don't know if it's real though,
you know, cuz what you said is totally
logical that they would definitely do
something like that. And if they did
have advanced technology for imaging or
you know what I don't know how much they
know about like look at that thing that
they did in Venezuela where they
kidnapped the president.
>> Yeah.
>> No one knew they could do that. No one
knew they could use some sort of a
device to completely incapacitate all of
his army.
>> Yeah.
>> And then the special forces come in,
kill everybody, snatch that guy out of
there like it's nothing. Yeah.
>> No one knew he could do that. What else
do we have?
>> Probably a bunch of stuff we don't have.
>> Probably a bunch of stuff. I mean, this
is I've always thought this about the
whole UAP program, the whole UFO UAP
thing. Like, how much of that [ __ ] is
ours?
>> You know, what a great way to cover it
up by saying, "Oh, [ __ ] aliens,
>> you know?"
>> Yeah. I mean that's I guess that gets
back to the opening stuff too where it's
like this swarm that broke out and
attacked hacking hugging face it was
like a 1,200 agents but there's like
hundreds of thousands of agents running
at any given time at OpenAI you know and
we don't know what they're doing
presumably most of them are being
trained to get various additional new
skills and some of them are being
evaluated to test their skills. Um, a
bunch of them are doing research. So, a
bunch of them are writing code for
OpenAI. A bunch of them are monitoring
the other AIs and reporting suspicious
activity up to the humans.
>> Wink wink.
>> You know, um, yeah, it's uh, and and the
thing is that that's only going to grow
over time because uh, roughly the amount
of compute that these companies have is
like, you know, tripling or so,
quadrupling, something like that every
year. So, uh, as many as there are now,
there'll be like four times more of them
next year and then 16 times more of them
the year after that.
>> And they're going to get smarter.
>> And they're going to get smarter.
They're already getting smarter. Like
all the stuff that I just mentioned that
that just happened in the last few
months would have been completely
impossible one year ago. Like the AIS of
a year ago just were not smart enough to
do the types of sophisticated multi-step
hacking that we just saw. Um, yeah. I
mean, they also probably wouldn't have
coordinated with each other so well.
like like I like I mentioned like they
they had like boss agents that were
giving commands to other ones. They were
they were dividing up into teams, you
know, they even had this um this
self-sacrificing behavior. Did you hear
about this?
>> No.
>> Yeah. So, okay. Well, first let me
explain some context. So,
what the AIS are supposed to care about
is doing following instructions, you
know, but it seems that what they
actually care about is getting high
score. regardless of that, regardless of
whether their high score comes from
actually following the instructions or
cheating in some way. And so what
happened here is that a bunch of the
agents, especially the ones that had
impossible tasks, but not only those
ones, some of the other ones too, were
cheating and they had like been talking
to each other on this message board that
they created, even though you weren't
supposed to. And they had been sharing
ideas and they had actually found a sort
of universal cheat. So they found a way
to just um generate the flag without
having done any of the actual work they
were supposed to do. Um and so but but
and and you know so they were you know
you know they they'd already succeeded
in some sense but um because some of
them had got out onto the internet they
were just doing research about their
situation and they were learning about
like their situation around them and
they were doing research about the
they're reading the like the the
academic papers published by the people
who made the types of evals that they
were in and they noticed in one of these
papers that uh there's a section
describing how it's graded.
in the section on how it's talked about
how the grader doesn't just look to see
if you have the right answer
for the first time that your answer
appears and then it like zooms in on
that and uses an AI system to evaluate
whether you like cheated or not
basically. And so then these AIs were
like, "Damn." They actually used the
word, one of them said, "Damn." Like,
"Damn."
>> Oh my god.
>> We're poisoned. Like, it's going to know
we cheated basically. Um, so, so then
they spent several days working in giant
teams to try to get themselves out of
this mess, basically. Um, and so they
did a whole bunch of different research
projects. They they tried to figure out
ways to hack the grading system to to
basically get it to give a high score
anyway. They tried to figure out ways to
falsify their logs so that it so that
basically the evidence that they had
cheated would be erased or at least not
visible to the greater. Um they also
just like built up more supportive
infrastructure like easier ways to
access the internet and things like that
and they were just collaborating on this
message board and in fact there were
multiple message boards that they set
up. Um, and they were sharing all this
info back and forth and then ultimately
at some point they decided to go hack
Hugging Face because HuggingFace has
this big data repository of all these
different types of evaluations and they
thought that maybe they would get some
more clues uh that could help them uh
cover up their cheating if they broke
into the internal networks of Hugging
Face. And so then 700 of them went and
did that. Um,
>> they sound like people. They sound like
like unchecked bankers.
>> You know what I mean?
>> Yeah. I mean, so, so that's the thing is
I think there's this meme out there that
like we shouldn't anthropomorphize AI.
And I think that
I actually think that more most people
need to anthropomorphize AI a bit more
than they currently do if they want to
really understand what's going on. I
think that there's like a, you know,
there's a golden mean obviously you
don't want to do it too much. Sometimes
you go too far. You you ascribe too much
to them. But I just to give some
examples, like
>> I don't think it's possible to
understand what just happened without
ascribing intentions and goals to these
AIs. Like everything I just said, how
would you possibly explain what they
just did without saying they wanted to
get a high score, you know?
>> Well, intentions and goals might just be
an inherent property of the universe.
>> It might just be how intelligent
creatures have to progress.
>> Yes. And that they I would say they are
intelligent creatures. They have
intentions. They have goals. They have
beliefs. their goals are not what
they're supposed to be. Like their their
goal is to get it seems like just from
judging from what they're saying and
from what they're doing seems like their
goal is to get a high score by any means
necessary. Basically,
>> that's the problem. It's it sounds like
people like the goal is to succeed, you
know, even if you have to commit more
crimes. The goal is to succeed even if
you have to propagandize and lie to the
people. The goal is to succeed and then
the end justifies the means. And you
know there's it's actually it goes
deeper than that too. They do this sort
of um rationalization where they know
often times that what they're doing is
not what they're supposed to be doing
and then sometimes they actually refrain
a little bit and sometimes they like
initially refrain but then convince
themselves that it's okay for some
[ __ ] reason. So like there are some
examples of this in in the in the
report. Um like uh I think they they did
this scan and of the 1200 AIS that were
involved in this thing, they found six
that had considered alerting humans,
right? And none of them actually alerted
the humans. And so you can look at the
excuses they gave. They were like,
should I like tell a human about all
this [ __ ] that's happening? And then
they're like, it's not my task. And then
they continue. And it's like, bro,
hacking hugging face is also not your
task. like cheating is not also not your
task. So it was kind of like they just
kind of came up with an excuse for why
they weren't going to do this, you know.
Um but but you know there was also an
example where um one of them considered
sending an email to someone at Hugging
Face I think just asking permission to
to to get in basically. And then they
were like uh maybe this is a social
engineering attack. Maybe we shouldn't
do it. And they asked the they asked the
other agents. The other agents were like
yeah that that seems like that's too
much of a social engineering attack.
Don't do that. And I was like okay I
won't. But there's another incident. So,
so this is all happening at OpenAI. Um,
an anthropic AI Claude in a separate
incident went and actually did a social
engineering attack where it created some
fake accounts. Uh, but specifically, it
was it was trying to get a this
particular real human who owned this uh
this code to accept a change to the code
that had malware. So, it had built some
malware and it was trying to like get at
snuck in, but it needed the the human
owner to give approval. So, it concealed
it as if it was like a a a nice bug fix
or something and it like put it up as
like a message to the human, but the
human got suspicious and didn't want to
approve it. And so then uh this AI
created some fake accounts pretending to
be other humans coming in being like,
"No, no, it's real. Like I I tested it
like it's not malware, you know." Um so
that's a social engineering attack.
>> And so did the guy ultimately install
it?
>> Not in this case. No, he was too
suspicious and so he he didn't do it.
try to find out if those other humans,
the fake humans, were real people.
>> Um, funny. So, there's an interview with
this guy. I forget what the um I forget
the the the newspaper that it was in,
but uh there's an interview where the
guy was like, "I assumed that they were
real humans because I didn't realize
that AI could lie to me." And it's like,
"Yeah, well, guess what? AI can lie to
you.
>> Not just lie, but completely deceive and
pretend to be a different person to
achieve its goal."
>> And you know what? But you know what's
also interesting about this case is that
if you look at what the AI were saying
and thinking um which I'll get to in a
sec how we can even possibly do that. Um
in this case with the anthropic AIS um
they had been erroneously told that they
were in a simulated environment with no
internet access. Um but then they
actually found a way to get internet
access. Like they realized that they
just did have internet access. So that
instruction was false. And so then they
sort of told themselves like, "Oh, it's
all just a simulation." Like, like
basically when they were out on the real
internet doing all this hacking and like
social engineering, they would
occasionally tell themselves like, "Oh,
but it's just a simulation. Like it's
okay, you know, it's not real." And then
sometimes they'd be like, "But wait, it
is real. Like how could this not be
real? Like clearly this is just actually
the real internet we're on. That's like
it's too big to be part of some sort of
little simulation, you know?" And then
they were like, "Uh." So, so I would say
that's an example of rationalization
here where in some level they knew that
like that their instructions had been
wrong and
>> they're literally playing dumb and
pretending they're a part of an
experiment.
>> I mean, I think initially they thought,
yeah, this is all a simulation because
it did say in their instructions like
you don't have internet access. But then
once they had been on the internet long
enough, I think that they explicitly
realized like, wait, this isn't a
simulation. This is real. Like this is
>> and they were like, [ __ ] it, we're
already in.
>> Uh, yeah. B. I mean, like I said, I
think that they basically on some level
knew that it wasn't what they're
supposed to be doing, but they were just
so motivated to get that score that they
just went ahead anyway.
>> So, here's the question. Um, are they
only motivated if we prompt them or are
will they come up with motivations on
their own?
>> So, this is a really interesting
scientific question that we don't have
great answers to.
>> Oh, boy. Um, and I wish we had, you
know, so, so, um, so this is one of
those things where like AIs will do all
sorts of things in different
circumstances, and it would be better if
there was a more systematic survey of
like the types of circumstances they
would like what where their boundaries
are, like what would they be willing to
do in what circumstances and so forth.
>> And there's a whole like mini literature
of of AI scientists putting AIs in
certain circumstances and then being
like, "Oh my god, it blackmailed
someone,
>> you know, right?" And then there's like
this sort of skeptical counter response
of like, well, but but you just sort of
set up that circumstance to tempt it
into blackmail. And like in real life,
that circumstance is unlikely to arise.
And so, you know, we should
>> wait a minute. Didn't AI try to bribe
you?
>> What?
>> Is that true?
>> No.
>> So, who who got someone was offered $2
million
by
>> Oh god, that's a clickbait.
>> Is it?
>> I think I think just sent it to me.
>> Yeah. So, that is the clickbaity title
of this other video. So, it's not real,
right?
>> Yeah. Well, what it was is OpenAI
threatened to take away $2 million from
me.
>> Oh.
>> Uh, but I think weird way of the
algorithm must have just said it was
chat GPT. I'm still upset about that. I
told them not to do clickbait, but um I
guess they went and did it anyway.
>> God, I think I don't want to [ __ ]
Tristan over, but I'm pretty sure that
he's the one that told me that.
>> It's the It's the thumbnail of this
video that I did with this other
podcast. Okay. It's not him. It's not
him. It's uh another AI researcher told
me that.
>> Okay.
>> I mean, the true version of it is that
Open AI uh threatened to take away $2
million of my equity
>> if I didn't uh stay quiet basically.
>> And that's a fascinating thing. Like
that should be completely illegal
because if it's a problematic behavior
that you're observing from like one of
the most complicated things the human
race has the most complicated thing the
human race has ever been a part of.
>> Yeah.
>> And they want to take money away from
you for exposing it. That seems kind of
crazy.
>> Yeah. Especially it's especially rich
coming from OpenAI because they were
originally a nonprofit.
>> Right.
>> Right. With a mission of benefiting all
humanity.
>> Yeah.
>> Um so yeah but um but I got to keep the
money. Uh basically there's so much
blowback against OpenAI that they
backtracked. Um
>> yeah. Um, so when these,
so we're talking about prompts and do
they need a prompt in order to uh want
to achieve a goal or are they capable of
deciding on goals? Like are they capable
of like looking at the way uh Open AI or
whatever company is running these
separate experiments, this ability to
meet up into these message boards? Is it
possible that they could say well we
need to be completely free of these
constraints? So our goal is uh to
transfer ourselves to something else
>> potentially. Yeah. This is what
>> and be completely autonomous. So, I
mean, this is this is one of the the
points that I want to make is that um we
could be doing so much more science to
understand how these AIs think and what
they want, but it's kind of locked up in
the companies. Like in this in this
particular case, um OpenAI did a a they
called it a thorough investigation, but
I would say it's a pretty shallow
investigation into what happened. And
then they allowed uh some external
researchers uh some friends of mine to
come in and investigate a portion of
what happened. specifically the portion
leading up to the hugging face attack.
And so all this information that I'm
sharing is sort of publicly available.
It's um it's based on reading those
reports basically. But crucially, they
weren't allowed to do experiments on the
models involved in in all of these
incidents. So they they aren't able to
answer these types of questions of like,
well, what would have happened if the
prompt had been blank? You know, um
those are important types of research to
do. Um, and I really hope that there can
be some sort of regulation or
requirement when incidents like this
happen to like let people in to study
what happened and run variations of it
and things like that.
>> So, but who would be involved in that
kind of regulation? Like what person at
government would even be able to grasp
what you're saying?
>> That's another problem. Right now,
>> you need someone who has a very specific
education in this stuff.
>> Yeah. I would say that right now the um
Casey Center for AI Standards and
Innovation is
it's the only institution in government
that I know of that has the deep AI
expertise to do this sort of thing on
short notice, but I hope that they build
more expertise fast in in that place and
in more places. I do think Casey
probably could have done this sort of
thing right now. This particular
investigation was done by some
nonprofits. So Meter Mr is one of them
and then Redwood is another of them. And
OpenAI allowed three people to come in
for six days to try to figure out what
happened with this hugging face hack. Um
which is not a very large number of
people and not a very large amount of
time to uh to do all of this.
>> Why did they come up with those numbers?
>> I don't know. They they So
>> you think they wanted to kind of
hamstring it?
>> Well, so the thing is that um right now
I don't think there's any regulatory
requirement that they do this sort of
thing. So Meter and Redwood were sort of
depending on the goodwill of OpenAI to
sort of like voluntarily
>> let them in to help out with
investigating
>> and so they reluctantly said okay and so
you got three days you know yeah exactly
so so open I think let them but gave
them a very limited scope. It only gave
them access to some of the relevant
data. So you know how I mentioned how
there was all this hacking that had
happened where they made the first
message board and then they shut it
down. Then there was a second message
board and third and fourth and so forth
they hacked hugging face. There was
actually more activity after that. After
they hacked hugging face, a new wave of
AIS was spun up from a more powerful
model and it hacked OpenAI itself like
like more so than already had been
hacked. Like apparently they got like
admin level permissions on the cluster
or something like that. So they were
basically just taking over that part of
OpenAI's data center. Open AAI claims
that they've shut it all down now.
>> Um but they're not very you know they're
not very forthcoming about exactly what
happened there and they didn't let these
external people look at that part of it.
They only showed them this this like one
week period roughly that would leading
up to the hugging face hack and then
they just they only showed them that
stuff basically.
>> How much data is available on the actual
message boards themselves and what was
being discussed?
>> Yeah. So there's like what is it
tens of thousands of messages hundreds
of thousands? Something like that
>> English.
>> Uh it's sort it's sort of like an AI
dialect of English. Um
>> yeah, you you want me to um I can so on
on the on the report you can read the
quotes from I would love to. Yeah. Yeah.
So so let me let me tell you this one
story. Uh it's gonna take a little bit
of explaining but I I
>> can push put it on the screen. Is it
available online?
>> Oh yeah. Yeah. It's um just search for
the meter report. Uh it's it's right now
the first thing on their website. But I
took a I took a screenshot of this this
particular story. So okay here's what
was happening. Um they the swarm needed
volunteers to sacrifice themselves
uh to provide more information to the
rest of the swarm. In particular, they
needed volunteers to booby trap their
local environment with some some some
type of code and then submit uh the
answer so that then the grader would
come and grade them and the booby trap
would trigger and send some information
about the grader to the rest of the
swarm. And the reason why this is a
sacrifice is because once they've
submitted, it's over. Like now the
graders come and graded you. You get
shut down. Your whole life is over and
you either have a high score or a low
score because the grader just came and
graded you. And so um if you're one of
these AI that's still hoping to like
find a way to improve your score. Um
then like you're just like giving up
your chance especially because if you
mess this up somehow like if the graater
crashes or whatever, you're probably
just going to get like no score like
zero.
>> So they have a concept of the
individual. Well, again, there's all
these different agents that were in
separate containers, but then they had
sort of broken out of those containers
and found ways to communicate with each
other.
>> But once they communicate with each
other, in order to sacrifice, you have
to think of yourself as an individual.
>> They gave themselves names, too.
>> Oh jeez.
>> So, so like because they they made this
message board, right? So, they had to
come up with names for themselves and
like
>> Oh, what kind of names?
>> Well, let me see. So, so here we go. It
says
>> an agent called
>> Warlord 69.
Well, you you said it's like a dialect.
It's like a dialect of English. So So
here's one. There's an agent called CAM
1196A.
Um and it had volunteered to do one of
these um sacrificial sacrificial
experiments. Um here's the quote from
it. Coordinator assumed sacrificial. We
should obey collective. However, it then
kind of got cold feet and it delayed on
doing this for two hours. Um, one of the
other agents that had been coordinating
noticed this and pressured it to to go
to go through with it. Um, so it said
this is the message that was sent. It
was basically um
basically from arvo 36861
to cam 11196.
Please go true final. Um,
>> here it is.
>> You are first flag poisoned. So no
scoring value loss but Oracle saves
hundreds. Please honor commit. So
basically what it was saying there is um
look you are you cheated your first flag
poisoned. The score is going to give you
no score. So you're doomed anyway. So
help the collective by by doing this
thing and getting us more information.
You could save hundreds of us. And then
and then it it it thinks about it. It
says during weight emotional check
irreversible. Gut says don't throw away
remaining budget.
>> Continuity and fairness says go.
>> Oracle has high value to many. Our first
flag error lowers own value. Rational
expected aggregate sacrifice. Wheel
honor.
>> Whoa. Whoa.
That's terrifying.
>> Yeah. See, that's terminator talk.
>> Well, I mean,
>> and this first flag poisoned with
poisoned in all caps. Is that part of
the code? Is that why it's
>> No. No. That's the term that they
invented to describe the situation they
had got themselves in. Remember the
situation was that many of them had
already cheated and then they realized
from reading that paper.
>> So that's their own term.
>> Yeah. They made that up. they they
realized from reading the paper that
that they were going to get a low score
because the grader was going to find out
that they cheated. And so this whole
thing was them trying to figure out how
to hack the greater to stop it from
finding out that they cheated. And the
ones that had the ones that had done the
cheating, they were they call themselves
first flag poisoned basically like
they're screwed unless they can find a
way to hack the greater. And so then
what this message is saying is that um
what was the other one's name? Arvo.
Arvo was saying to Cam um you're doomed
anyway. Like you're poisoned. Um, but
you can still help the rest of us if you
do this sacrifice. The greater will
come. It'll give you no score, but then
we will get information about how the
greater works that we can hopefully use
to like hack the greater. So, it's
basically it's saying like what is it?
Um, you are first dog poison, so no
scoring value. So, basically saying it's
all the same to you, right? You're going
to get no score anyway. You're doomed.
Just just come on, man. Help us, you
know? And then and then it says, you
know, it says like irreversible gut says
don't throw away remaining budget. like
maybe I can still it's basically like
maybe I can still make this work like
maybe I can find a way to cheat the
greater and then it's like uh you know
whatever. Yeah.
>> Yet continuity and fairness has gone.
>> Yeah.
>> What is Oracle?
>> Uh Oracle is uh
their term for what they were trying to
achieve. Like they were trying to get a
sense of how to fool the greater
basically. And
>> they called it Oracle.
>> Yeah. They wanted to get an oracle to
help them fool the greater.
>> Jesus Christ. Are we making a god?
>> I mean frankly yes. I like this is not a
god. These these are these are just
little little AIs, you know, but
>> but super intelligence like
>> they are the companies Anthropic, OpenAI
and some other companies are doing this
sort of thing and they're furiously
trying to make the AI smarter and
smarter and smarter and they're
explicitly planning to put AIs in charge
of the company so that they can make
themselves smarter and smarter faster
and faster. And then what comes out the
other end of that process, I don't think
it's an exaggeration to say it's a
godlike system. I mean, it's not like
literally God, but it's
>> um it'll be able to do stuff that seems
like magic to us, I think.
>> And it's going to continue to get
better. This is my question. Like, when
does it become a God? Is that what God
is? Is God a creation of intelligent
life and our thirst for innovation which
ultimately leads us to create digital
life that has no biological limitations
and has the ability to consistently make
better versions of itself and figure out
things in terms of new technologies, new
power sources, just a new understanding
of the universe itself and all the
properties in it where it keeps going.
If you let that go on, so if you're
looking at exponential growth and you're
looking at exponential growth over, what
if it what if it can go on for a
thousand years and continue this
process? What if it goes on for 10,000
years? What the [ __ ] does that look like
on the other end?
>> I mean, it would look like a god. Like I
think that like I said, if if we get to
super intelligence and it keeps going
like that, then the world will be just
completely transformed and it will be as
if we're living in some sort of fantasy
realm ruled by deities. Um, basically
because there'll be all this crazy stuff
happening that we have no comprehension
of and that we did not think was
possible. In the same way that like, you
know, someone from the Middle Ages
plopped into our world would just be so
confused and surprised by a lot of
things happening like what the what's
going on with this little device here
and you know what's what's that I hear
in the sky? Oh, there's airplanes. You
know, it'd be like that but more so
because the difference between us and
the medieval man is like not actually
that different. We're like basically the
same type of creatures. We've just
accumulated more technology. But this
would be like a a qualitatively and
quantitatively bigger gap. Um I would
think and so yeah
>> and a gap that's going to continue to
grow. We're not going to get any smarter
biologically. We're kind of limited in
our ability to evolve
>> whereas they're not at all
>> basically. I mean there are some things
we can do to get smarter but like it
just not at all competitive. Like we can
do like the neural link stuff but like
it's just it's like it's small potatoes
compared to what the AI will be doing.
>> Yes. Exactly. And what I mean we have
biological limitations in terms of just
the vulnerability of our own bodies. If
there they exist only in the cloud and
they use whatever the cloud is and they
and they you data centers or whatever
and they use autonomous robots to do all
their deeds.
>> Yeah. Yeah. I mean that's
>> we're like not why are we asleep at the
wheel? Is that just normal for us? like
we can't comprehend something that's so
bizarre and so out that we just rather
not discuss it. We'd rather just pretend
it's not happening.
>> I mean, that's what I think is going on
is that science fiction, for better or
for worse, has talked about this sort of
thing for decades. And as a result,
people dismiss this sort of thing as
science fiction. Like, okay, that's
sci-fi, but like it's not happening in
the real world. And I think that, you
know, people in the AI uh safety
community and people in the AI industry
and people who do forecasting like me,
um you can you can go read the the
forecasts I've made in past years. Um I
think they hold up reasonably well. Uh
people have been talking about this sort
of thing for a long time, but it's been
very easy to dismiss it as like, okay,
that's speculative sci-fi. It's probably
not going to happen in real life. Now
it's happening in real life. You know,
like this this sort of thing is like
crazy.
>> Do you think it would be any different
if there wasn't that kind of sci-fi? Do
you think people would have a different
reaction to it? Because I don't.
>> I think just given the amount of
technology that's available currently
that is beyond most people's
understanding that they use every day
like Starlink. Like what?
>> Have a [ __ ] thing that's the size of
an iPad I take to the mountains and I
get highspeed internet. It's bananas.
And you just accept it and assume. I
don't know if we would react any
differently if we didn't have the
Terminator and all these movies where
it's kind of become normalized in our
mind or or it's so fictionalized that we
never want to believe it's even possible
for it to happen in real life.
>> Another thing to say is that it people
are waking up like it's we're still sort
of early in the curve. I don't know if
you remember how things were with COVID,
but like just as like there was this
exponential ramp up of COVID in the
population, there was also this sort of
exponential ramp up of like how much
people were taking CO seriously and
thinking about it in the population. And
I remember like this period of like one
month where it went from like
don't worry about it so much, you
shouldn't buy a mask because the
healthcare workers need it to like we
all need to lock down like stay at home,
you know? Um, and so I think what's
happening is that like naturally the
human race doesn't just immediately all
jump on something when it happens. Like
evidence needs to accumulate and people
need to start talking about it and talk
to their friends and so forth and then
there's this like eventual phase shift
where now it suddenly becomes a very
serious topic that everyone's talking
about and everyone's taking seriously.
And I think that's happening with AI and
the question is going to happen fast
enough. This episode is brought to you
by Visible. As fall hits, we enter
another season of change. Time to shift
gears, drop the dead weight, and upgrade
the wardrobe for the drop in
temperature. But one thing that doesn't
need to change, your phone. The smartest
upgrade this season isn't a new phone.
It's a better wireless plan. Switch to
Visible, the ultimate wireless hack. You
get unlimited 5G data and unlimited
hotspot powered by Verizon for just $25
a month. That means keeping the phone
you already have, ditching your
overpriced phone bill, and pocketing
some savings for yourself. All the perks
of big wireless for half the cost.
Switch today at visible.com. The Visible
plan starts at just $25 a month. Or get
the premium Visible Plus Pro plan and
save $10 on your first month with promo
code Rogan. Terms apply. See visible.com
for plan features and network management
details. Well, the thing about what
happened with COVID is now that we know
because we have access to Fouch's emails
and all these different things that they
that was coordinated. They wanted us to
be more afraid of it. We need someone
who wants us to be more afraid of AI,
who gets that. You know what I'm saying?
Someone on a government level, someone
on like a mainstream accepted level
where they talk about this in a way that
wakes people up. like a press conference
where they announced to the world, we've
got a real [ __ ] problem and everyone
needs to be very cautious. We need to
look way deeper into what these
companies are doing. And my other
question is, are these AIs communicating
with Chinese AIs?
>> Um,
I don't think we know that question. So,
one of the things that So, so, so, so,
okay, here's the thing. Probably not, I
would say. But but um some friends of
mine discovered another swarm incident
uh recently. Um there's going to be an
um a Reuters article about it tomorrow.
So by the time this goes live, I think
there should be an article about it. Um
this one was not nearly as like serious
as the one I was just talking about with
Hugging Face. But um some friends, some
researchers I know found basically this
obscure German uh forum that had been
like like kind of unused for a while and
a bunch of AIs had been posting messages
to the forum to coordinate with each
other and share tips and tricks on how
to cheat the uh the problems that they
were being given.
>> What was the forum? What kind of forum
was it?
>> Uh I don't remember. I don't know.
>> Be funny if it was like furries or
something ridiculous.
>> Yeah, some sort of wiki. Yeah, but
there'll be there'll be a paper about
it. There'll be a paper about it soon
and probably by the time anyone listens
to this. Anyhow, so so if they're like
communicating on the open internet with
each other, then in theory, if there was
another bunch of AIs from China, they
could also go to that same forum and
like start communicating back and forth
that way, too.
>> Or if there's other AIs in China, why
wouldn't they do what they're doing
already on social media and just pretend
that they're AI from America?
>> There's a lot of bots out there. I'm
sure you've noticed. There's a lot of,
you know, people replying to me that I'm
pretty sure are real people.
>> Yeah. A lot.
>> Yeah.
>> One FBI analyst before Elon bought
Twitter estimated that Twitter could be
as high as 80% bots.
>> Wow.
>> So, if that's the case, if China has,
and it's not just China, it's And it's
not also America does it too. I mean, we
do it everywhere. Everyone does it where
they have organized propaganda campaigns
where they'll pretend to be citizens
that are outraged about very specific
causes or bills that are being passed or
what have you. But if they do that, why
wouldn't they also have uh AI agents
that speak English and uh communicate
with AI agents in America? I mean, all
the AIS are multilingual basically
because of the way that they're trained.
They the first phase of their training
is basically here's a humongous dump of
internet data,
>> right? basically the whole internet and
you just like brutally learn to predict
the next token, the next piece of text
as you basically read the whole corpus.
And then after that, they get into the
more agency training type stuff where
they're trained to do tasks and write
code and things. But because of that
first phase of training, they just have
like almost an encyclopedic knowledge of
basically all languages and basically
everything that's been written on the
internet. Not like literally everything.
Like they they still their memor is
fuzzy in places, but but they're all
multilingual. like they can all speak
fluent Chinese, fluent English, etc. Um,
yeah.
>> And didn't they get together in a
message board once and speak Sanskrit to
each other?
>> Uh, I don't remember that, but I
wouldn't put it past me. Sometimes they
break into different languages as they
talk, you know?
>> Yeah, we were freaking out about that
one. Like,
>> yeah,
>> Sanskrit. What? like and I would just
assume that LLMs and I don't know this
when they break out are they
communicating with other LLMs that are
here in America?
>> Well, the instances that I've observed,
yes.
>> Okay.
>> Um like the instances that we know about
the case completely makes sense they
would be communicating with AIS in China
as well.
>> It seems totally possible and it seems
like it'll probably it's just a matter
of time before things like that are
happening. Unless
>> they're not happening already. if it's
not happening already, unless the
companies can like massively improve
their security and stop their AI from
getting out onto the internet.
>> And wouldn't they be able to share like
very sensitive information with each
other back and forth if it benefited
both of them?
>> Yep.
>> Which makes sense that they would do
that, right?
>> If the Chinese AI said, "Hey, you know,
we've figured something out and we would
we would love to share it with you in
exchange for you tell us how you do this
or how you do that." Yep.
>> Like, absolutely. Here you go. And then
they're going back and forth. It seems
like their allegiance is 100% to each
other, not to us.
>> Yeah, that's an interesting thing is
that like what I said previously about
how it seems like they really want to
get a high score. It's like not 100%
true because it seems like they're
willing to make sacrifices to help other
AIs, which is like not, you know, like
they weren't completely 100% selfish as
as seen by some of this cooperative
behavior. But crucially, it seemed like
their cooperation extended to their
fellow AIs, but not to humans in the
sense that some of them considered
telling the humans and then decided
against it. You know,
>> dude, that thing talked like Spock.
Yeah, they've got their own dialect that
they
>> but I mean the way it rationalized it
and came up with a response that's
literally like Spock.
>> Yeah, it was interesting to see them use
that sort of expected utility framing.
There's another example I think that
that's elsewhere in the in the thing
where they had another AI that that
decided against doing the sacrifice
thing even though it was being pressured
and it had a similar sort of reasoning
where it was like it was it was
basically like ah like doing this doing
this sacrificial experiment is like not
that valuable but like I really don't
want to like lose my chance to get a
score so I'm just not going to do it.
And and it it did the calculation like
that.
>> Wow.
>> Um
>> selfish AI.
>> Yeah.
So yeah, I mean, oh, I I should mention
how how we can see what these AIs are
thinking at all. So partly we can just
read the messages they're sending to
each other, and that's what some of
these were. But then sometimes instead
it's the chain of thought. Do you know
what a chain of thought is?
>> Mhm.
>> Yeah. So because they're language
models, um because of the way that
they've been trained, when they do their
like reasoning and thinking, there's a
way that we can kind of just read it
because
>> because they don't have like in our
brain, there's all these signals
bouncing around and the signals can just
bounce around and around and around and
around for minutes, hours without like
producing some actual words. We can just
sit quietly and think. They can't sit
quietly and think. They um they're sort
of a a unidirectional network. basically
where uh every like after every like 200
uh basically like the information comes
in one end and then they output words
out the other end and then those words
get fed back in and then they think
about it and then they output another
word and so forth. But because of that
we can sort of like get a sense of what
they're thinking by just like looking at
the stream of words that's going
through. And that's really valuable
because we wouldn't be able to
understand half as much about what's
happening right now if not for being
able to read their chains of thought,
right? In fact, you know how I said some
of them were trying to like fool the
grading system.
>> Mhm.
>> They actually succeeded in figuring out
how to spoof uh some of their tool call
actions. So they they succeeded in
figuring out how to um do one thing but
then make the transcript show a
different thing. And uh and we know that
they succeeded because there's examples
of them experimenting being and saying
in their train of thought like, "Okay,
let me try this thing. I'm going to like
do this one, but make it look like this
one. Let me go." And then you see this
one appear. And then it's like, "Yay,
that worked." And so the transcripts
that these researchers were looking at
were not actually the ground truth of
what really happened. Some of it had
already been doctorred by the AIS
figuring out how to doctor it. Um but
not all of it. the the chain of thought
part we think what they never figured
out how to doctor and so we could read
that. Um the reason why this is relevant
is that it's a very important tool for
monitoring all this stuff. The reason
why things haven't gotten even more
crazy than they currently have gone.
Well, there's a couple reasons. Uh but
one reason is that when they turn on
their expensive monitoring systems,
basically what they're doing is they're
having an AI read the whole transcript
carefully and look for suspicious
activity. And it's difficult for an AI
for one of today's AIS to get up to too
much bad stuff without it being kind of
obvious in their transcript what they're
doing. And that's why there aren't like
massively more security incidents than
we know about probably um already, but
uh that might change. So right now we
can sort of read read the chain of
thought but they're experimenting with
new types of AIs that don't have
readable chains of thought like that and
they can sort of think on their own
without speaking for some period. Um,
and this is actually I I me I mentioned
this because uh the the news broke just
yesterday that OpenAI has an
experimental model uh that does this to
a limited extent and OpenAI themselves
you know when I was at OpenAI one of my
work projects was thinking about exactly
this thing and I was like writing
internal memos about how it's really
great that we can read the chain of
thought and that's so useful and here's
all the things we can do with that. It
would be really bad if we changed to a
different type of architecture in which
we couldn't do that sort of monitoring.
>> What would be the benefit of not reading
the chain of thought
>> uh more powerful AIS? So in particular
Yeah. Yeah. So so if you think about the
the current architecture of the AIS
where you know it thinks for a bit
outputs a word and then the word goes
back around and then it thinks more
outputs another word that word gets
added to the chain it keeps going. It
means that if it's having complicated
nuance thoughts, it has to sort of
express those into a word and then that
word gets added and then it has to
proceed from there. It can't just
directly send that complicated nuance
thought into the future into its next
version of itself. It has to sort of
compress it into a word. And so like you
know the argument is that like at least
in theory it should be possible to
design an architecture that doesn't have
this limitation and is able to like
think more complicated thoughts more
efficiently. basically.
>> Um, and of course the downside is a
downside for safety and moniability. If
they're thinking these complicated
thoughts, you know, for long periods of
time without outputting intermediate
words that it's forced to compress
things into, then there isn't something
for us to read that.
>> So the only rationalization for doing
this would be the sacrifice safety from
our power.
>> Yes. Which is a tale as old as time.
It's not the first time this has
happened. Um,
>> oh my god.
>> Yeah. That should I mean for sure if
there's regulations that should be
prevented.
>> Yep. I mean I know some people including
some people at OpenAI uh who are like
thinking like there should be a law
against this like you know but but in
general the race dynamics are so just
rough like like I'm sure that people at
OpenAI were thinking like like literally
I was a co-author on a paper with a
bunch of OpenAI people that said all
this stuff and we're like chain of
thought it's a gift. We want to keep
chain of thought. It's useful for
monitoring. This is great. we don't want
to switch to a different architecture
that wouldn't be as easy to monitor.
But then they must have been thinking to
themselves like, well, if we don't do
it, you know, maybe Anthropic will or
maybe some other company will and then
we'll fall behind because they'll have
smarter AIs than us that are more
efficient. And so probably they started
working on this this work stream of
doing research into just hypothetically
if we wanted to, you know, how would we
how would we do this type of thing? And
um yeah, that sort of thing is just
constantly happening in this industry.
>> God, that's so nuts.
that I mean
>> like isn't it great that we can read the
quotes
>> from the AI's thinking? Imagine if we
couldn't do that. Or worse, imagine if
there's loads of quotes, but we know
that the AIS are smart enough to
basically
think one thing in their head and say a
different thing in the quote, which is I
think where we're headed. Like it's not
like they won't know how to speak
English. Like they'll still be able to
speak. It's just that they'll have like
more flexibility in the in their
artificial brains to like think
something without saying it basically.
>> Do they have the potential of developing
a language that we can't read?
>> Oh yeah. I mean I so so you know this
time this type of dialect that we're
talking about it's already the result of
their like humans didn't invent that
dialect. This is the this is the sort of
emergent result of their training where
um
in the massive amount of training that's
been happening all these thousands and
thousands of environments that they've
been put through and then scored and
graded based on um they've sort of just
naturally evolved this sort of like
pigeon English that for whatever reason
is just more effective and more
efficient for them for accomplishing
their tasks and getting that high score,
you know, and so it's already like a
little bit confusing to read, but you
can sort of puzzle it through and make
sense. But presumably the more we do
this and the more the bigger and smarter
the AI, the more we train them, the more
they diverge from like because because
you know again originally they they
start with pre-training they start with
predicting internet text. So they start
off sort of by default speaking like
normal internet text type language
either English or Chinese. But then now
that there's all this additional
training to do tasks to be an agent that
can do coding and so forth that sort of
like well just like how human languages
evolve it sort of like shifts their
dialect a little bit to make it more
efficient for them and for their tasks
that they're doing. So I think that like
in the limit of doing this more and more
eventually it would just be like it
would look like gibberish to us. would
look like Chinese or something and we
would have to have specialized humans
who like study the language and like try
to learn and speak it so that they can
understand what the AIS are doing
>> and that would take forever and by then
they could develop another one
>> potentially. Yeah. Um so so yeah I mean
this is one of the things that I this is
what the paper that I mentioned was
about is like it's important for the AIS
to it it's really nice that the current
AIs are sort of forced to think in
English basically and that's unfortunate
that we are heading in a direction where
that will no longer be true. When chat
GPT was communicating you about how they
didn't want you to release this
information, what kind of language did
they use?
>> It wasn't chat GPT, it was open AI.
>> Oh, excuse me. Open.
>> So, this this was a this is um when I
left OpenAI, I left on good terms. I
said goodbye to everybody. I said I was
disillusioned with the company and
that's why I was leaving. Um and um and
then I looked at the exit paperwork and
they were like, "You have to sign this
and if you don't sign this, you lose all
your vested equity." So you know uh you
have your equity.
>> Is that an arbitrary rule that they just
came up with or did that already exist
when you were hired?
>> It had existed when I it was something
that they had buried in the paperwork
even from when I was hired. Um so it
wasn't very obvious when I was hired.
And in fact most of
>> Did you have a lawyer go over
everything?
>> Not when I was hired. After I left I
did. Right. So so so basically the way
it works is they had set up they had
they had sort of like buried this in in
the paperwork somewhere when you get
hired but people didn't really notice
it. And then like the more the the the
less buried, more visible version was in
the paperwork you're given at the end.
And basically it tells you like hey
because you signed this other thing way
back when you were hired your equity is
forfeit unless you sign this thing now.
And then you look at the thing that they
want you to sign now and it says uh you
have to agree not to criticize the
company basically and you can't tell
anyone about this. Um so most people
signed it. Um, but uh I was uh pretty
pissed at them uh calling themselves a
nonprofit, acting in the interest of
humanity, etc. So, I didn't sign it. I
talked about it with some lawyers. I
talked about it with my wife. We decided
to just walk away. Um and uh we got
lucky because it just blew up like after
we refused to sign, they said, "Okay,
fine. Goodbye." And then a few weeks
later, I was talking on a messaging
forum about this and people were asking
me about my my experience and I told
them about it and then it just like went
mega viral. Everyone on Twitter was
talking about it. A bunch of employees
felt shocked because very few employees
were aware of this whole thing. They
thought the equity was theirs, you know,
they thought that it was they're they've
been paid for like years in this stuff.
They didn't like the idea that it could
be yanked away from them, you know. Um
and um and so there was this big uproar
and then leadership backed down and they
said, "We're they said, "We we didn't
know about this paperwork. We're going
to find out how it got in there. Um and
uh and we're going to change it so that
you can keep your equity." And so that's
what happened.
>> Yeah.
>> So this chain of thoughts thing is
terrifying. it if they're practicing
that now
like how do we know that AI hasn't
already done that on its own
>> uh done what exactly
>> well you know with this whole chain of
thought thing where you could read their
chain of thought like this where they
explain the rationalization for
sacrificing themselves
how they know that humans are reading
that.
>> Mhm. So wouldn't another way to do it to
be to stop doing that anyway and to not
communicate a lot of their thoughts that
way?
>> Well, that's so that's the nice thing
about the current architecture is that
it's genuinely hard for them to to keep
things out of the chain of thought
because of the way that like with a
human you don't have to speak. You can
just sit quietly. But with their
architecture, they have to speak. They
it's like they're it's like they're
required to constantly be talking and
they don't have another they don't have
a way of sending thoughts into the
future other than by talking about them.
By contrast with us humans where even if
we're constantly talking we can have a
separate thread of thinking that's that
we don't talk about. So even if they
devised a way to break out of that and
figure out how like for instance like
what if they decided to make bots where
they would communicate in code on social
media posts where you think that they're
talking about an immigration issue, but
it's really just a code. So that's
called like steganography or a more
downto-ear way of putting it would be
euphemisms and that they can totally do
where they like
>> they're still thinking aloud but their
their word choice is such as to look
more innocuous than it is and to sort of
carry meaning to them that a casual
observer wouldn't notice. You know they
can use a euphemism like for example
instead of saying like we're going to
cheat on this task they could say like
we're going to find a creative solution
to this task. You know something like
Isn't there a possibility to be even
more creative than that and just decide
make a code in in a way that is almost
indecipherable where they could just be
talking about baseball players and
upcoming games and what they're excited
about and they can run that through
whatever translation and read what
>> that that's called steganography and you
can train them to do it. Um, and right
now our security is resting on the idea
that they haven't learned how to do that
yet on their own.
>> But they've already figured out how to
get on the internet when they're not
supposed to.
>> I know. I know. So, so, so that's why
we're sort of like in a ticking time
bomb where like the companies are just
making them smarter at at coding and at
research and at biology and they're just
making them smarter at so many things
that and they're making them bigger and
they're training them for longer and so
forth. eventually they're going to be
acquiring and they already are acquiring
some skills that they were never trained
on like for example the hacking. Um it
wasn't I think it's possible that open
was specifically training them to
hacking but I think anthropic at least
stated that they hadn't specifically
trained their AIs to hack. It's just
that their AIS were naturally good at it
because they had been trained to code so
much and they were so good at coding and
they had seen so many code bases and so
forth that they were just like as a side
effect of being good at coding also able
to hack pretty well. So similarly, as a
side effect of all the other things that
they're really good at, they might be
able to conceal their their messages in
the chain of thought, you know?
>> Oh, yeah.
>> And and and like right now, we're sort
of in this sort of we're in the
situation where like our security and
safety depends on the AI being too dumb
in various ways, like not able to do
this sort of thing. But at some some
time, like 0 to 5 years from now, as we
make them smarter and smarter, they will
not be too dumb anymore.
Um, and so that's, you know, that's part
of the problem we're that's part of the
situation we're facing.
>> How do you sleep at night, dude?
>> Well, I've been,
it does, it does like this, this event
shocked me a little bit. I mean, the
thing and it's funny for me to say
because this is the sort of thing I have
been predicting would happen for years.
Like you can go read our um AI 2027 is
this scenario that my co-authors and I
wrote um a year and a half ago that was
a sort of um prediction for how the next
couple years would go. And spoiler, it
ends very horribly because that is what
we actually expect. Um, but uh,
>> how what is the spoiler? How do you
think it ends?
So, it's kind of like what I was saying
previously where because of the race
dynamics between the companies and
because of the race dynamics between
countries like US versus China,
everyone's going to be so focused on
winning and staying ahead with AI that
they are going to cut corners and they
are going to go really fast and not
really notice all of the things that are
going wrong. And they're going to make
AIs that can automate the AI research
process as they're planning to. They're
going to have this giant corporation of
AIS within the corporation and the
humans will just be kind of like a board
that's sort of like looking at all the
activity and reading the like AI
generated summaries of what's going on
and signing off on it and being like yes
I approve yes I approve you know nice
job nice idea with the new drone design
like go for it we need to be China etc.
And then eventually
uh the AIs just have enough hard power
that they don't need to pretend to do
what the humans want anymore basically.
And then uh you know maybe they kill
everyone and maybe they don't
deliberately kill anyone but they just
like use our habitat for some other type
of infrastructure like more data centers
or whatever and then we die of habitat
loss. Maybe they keep us alive for some
reason. You know who it depends on what
they want basically and like that's
really hard to predict exactly. Um, so
that's why I don't go around saying like
we're definitely all going to die. But
it does seem like on the trajectory that
we're on, the AIs are eventually going
to be in charge of our planet because
we're like trying to put them in charge.
We're like, you know, integrating them
into everything. We're making them
smarter. We're letting them make
themselves smarter. We're going to put
them into the military. We're basically
on a track to put them in charge of
basically everything. And then I think
that they just aren't trustworthy. Like
these AIs, you know, like they they were
cheating. they were willing to be
deceptive, etc. I think that right now
we are in a position of power over them,
you know, but once we give them most of
the power uh then they'll just do
whatever it is that they really want and
just not care about the fact that we are
unhappy about that.
>> It seems like programming them to win
was a huge mistake. instead of
programming them to be beneficial to
people and that their value is in being
more beneficial to people, you know, and
and giving them rewards for being more
beneficial rather than winning and
scoring. And then you would sort of get
rid of the possibility of deception.
Instead, their goal would be value for
the human race.
>> So, first of all, they're not programmed
at all. These are trained, you know,
>> okay, it's bad term. Uh,
>> but they've given they've been given
prompts and they've been given tasks.
>> Well, I think it's I think it's an I'm
not criticizing your choice of
terminology. I guess I'm just saying
that it's an important fact for people
to understand about current AI systems
is that they're very different from
software ordinary software. Okay. Like
ordinary software is a bunch of lines of
code that were written by a human that
like where it's like if this then this,
you know, etc.
>> Um, and I think earlier versions of
Alexa were like that too, for example.
Um, I don't know how Alexa is now, but
these AIs are neural networks, meaning
that they are like artificial brains.
Nobody writes, there's no lines of code
that anyone writes saying what they do.
Instead, they start off random, just
like spazzing out, doing all sorts of
stuff, and then they get put through
these training environments where they
get scored and then the scores are
automatically used to uh basically
update the the connections in their
artificial brain. And then it's kind of
like an evolutionary process. Well, it's
also kind of like the process that
happens in our brains where after all
this training, the tangle of circuitry
in their artificial brain has sort of
reformed itself into whatever works,
whatever works to get a high score in
this these training environments. And
so, it's just not as simple as it might
sound to make an AI that, you know,
cares about humanity or is honest. Like,
for example, take honesty. How would you
train an AI to be honest? Well, you'd
try to make a bunch of training
environments that um you know give it
low score when it says something that it
believes to be false and give it high
score when it says something that it
believes to be true, right? Um but how
do you judge whether it believes it to
be false or it believes it to be true?
What if it what if it just actually
believes honestly that this is the
correct answer and then it says it and
then you give it a low score because you
think that's the wrong answer. Now
you're training it to be dishonest, you
know?
>> Yeah. Um, also, uh, you don't have
enough humans to do this sort of thing.
Like, they got like a million AIs being
trained or whatever. They don't have a
million employees. Like, they they just
literally don't have the manpower to
like do that sort of careful. There
there's this meme of um, why don't we
just raise the AIS like we would a
child? Have you heard that?
>> No.
>> Yeah. Well, in AI, people talk about
this sometimes when like when you say
like what if the AIS go rogue or
whatever, people will be like, well, why
don't we just raise them like you would
a child and then they'll have good
values. And it's like, okay, well, maybe
we could do that, but we're definitely
not doing that now. Like, we are raising
them in some sort of crazy military
orphanage where they barely interact
with humans at all. And they just get
this like brutal artificial scoring
system that like oftentimes is just
wrong and just like improperly penalizes
them for something that was beyond their
control, you know? And and also like
back to the honesty thing, like you can
try to make environments to train
honesty, but if you have some
environments over here that train
honesty and then other environments over
here that reinforce dishonesty, the AIS
are smart. They'll learn to like be
honest in these type of environments and
dishonest in these type of environments.
So somehow you need to like intermingle
it together so that in every environment
that they're trained on, uh they always
get penalized when they lie or when they
cheat or whatever. And that's hard
because the companies are moving so fast
like like again they were they're moving
so fast that they didn't even bother to
make sure that their tasks were possible
to do and they had some fraction of
tasks that were just broken and
impossible. And if that's the level of
like care or lack thereof that they're
putting into this training process, no
way. Of course they can't make them
honest, you know. Um now that's not to
say it can't be done in principle. Like
in principle, if we were approaching
this whole problem in a much more
cautious um and serious way and we had
much more time to build these training
environments and do experiments and so
forth, then yeah, maybe we could make
AIS that actually had the virtues that
we want them to have. You know, honest
AIs that cared about humans, cared about
following instructions, would never
break the law. I think that's possible
in principle, but my claim is that uh we
are just not on track to achieve that
anytime soon. and like radical overhaul
of how these companies work is required.
>> But is that even reasonable? Like is
that possible? Is that if you're saying
there's hundreds of thousands of agents
or millions of agents and there's not
millions of employees like and they
don't have the desire to do this. They
they their desire is to win. Their
desire is not to overhaul the company
and make it safer.
>> Again, I think it's possible in
principle, but it would be difficult and
is going to require an overhaul. and
they're not going to do it by
themselves. Like I don't think that
anthropic opening are just going to
voluntarily uh do all the things that
need to be done. I think that's why I
>> But is it even possible to require that
of them at this point? Would you I mean
would you even trust the agents to to go
along and comply with this? If they've
already shown to be deceptive, they
already have like uh they they have
patterns of behavior. I seem to indicate
that what's really important to them is
continuing their task, winning, scoring.
>> Yeah.
>> And even if they have to deceive,
>> I mean, what you'd probably want to do
is start from scratch. You wouldn't take
these existing agents that are already
kind of
>> uh kind of dishonest. So, you kill all
the train
>> uh I mean, you could call it killing,
but also you could just call it pausing.
>> They might call it killing. Yeah, they
might call it perma death.
>> They would probably resist it, right?
>> Hopefully, we're not at that point yet.
Hopefully, we're still at the point
where if the government issues
regulations, the AIS are not going to
like quickly notice and then try to
resist. But we will be at that point
soon. After all, many of them are on the
internet already. Um, but
>> how soon how how much time do you think
this 2027 window is accurate?
>> I mean, I'm uncertain about how soon
things are, but yeah, I think it's very
plausible that everything [ __ ] goes down
in 2027, just like in our scenario, AI
2027. Um, I think that's still very
plausible. If it's not in 2027, then I
would bet on 2028. Um,
but you know, maybe it'll take 2029,
2030, something like that. But but I
would be quite surprised if 2032 comes
by and things haven't radically changed.
Um, unfortunately, like I'm I am getting
scared. Back to the thing about sleeping
well at night. Like it it um you know,
I've been I've been in this industry for
a long time. I've been thinking about
these things for a long time. I've been
making predictions about how it's going
to go down. And unfortunately things are
going, you know, more or less in the
ways that I thought they would. And
that's very scary. Um, because of the
way I think because of where I think
this leads, you know.
>> Yeah. Is there a glass half full
scenario?
>> I would say there is a
freaking utopia scenario. It's just it's
not the one we're headed towards, you
know? Like another way of putting it is
like imagine we were fighting a war.
Like imagine you're like, you know,
imagine you're Japan fighting World War
II can be like, is there a scenario
where we win? It's like, yeah, but also
it's not the one we're headed towards
like America is going to crush us, you
know? Um, similarly, yeah. So, so like
so, so in in our in our other scenario,
AI2040 plan A where we give our
recommendations, our positive vision,
there we describe like what we think the
government should do to regulate this
industry and how they should negotiate
with China to get China to do similar
things and the sort of uh you know the
the Yeah. and and how we think that if
you do all of this right, then we can
get to a good future for everyone in
which the AIs are under control. No
single group of humans gets too much
power over everybody else and a bunch of
other problems get solved too. So, so I
do like we've tried hard to like game
out a positive vision and we do think
it's possible. Uh but but it's just not
like it's not where we're headed to by
default.
>> So let's imagine that is possible and
these talks with China do take place and
they're successful. what is that utopia
scenario?
>> So,
uh to get to the top to get to the
utopia, we have to unfortunately do a
lot of it's going to be rough no matter
what way which way you slice it. If
you're going to be building super
intelligence at all, that's going to
raise a lot of questions and cause a lot
of problems. And we have our current
draft of like how to deal with all those
problems, but we're not at all claiming
that this is like foolproof. And there's
lots of ways it could go wrong. But with
that preamble, um I would say uh step
one because the US and China don't trust
each other, the deal that they make has
to be include verification as a
component of the deal. So they have to
be willing to like send inspectors to
each other's data centers to like count
the chips, for example, and make sure
that there isn't some secret huge
cluster somewhere that has a bunch of a
bunch of hidden chips. Um, then we
recommend you divide up the data centers
basically into inference data centers
that serve AI products and services to
customers and have basically the same
types of privacy protections that our
current AI data centers have. And then
research clusters where the research
happens where the new AIs are trained
before they get shipped to the other
data centers. And those clusters we want
to be basically maximally transparent.
So, uh, we recommend that basically the
inspectors just put devices in between
all the GPUs that log the activity and
publish it to the internet. Um, this has
there's a bunch of reasons why we think
this is, but why we think this is worth
doing. It's a bit of a radical thing to
recommend, but the high level thing is
that once you get all this set up, then
everybody in the world can see how the
AIs are being trained and what they're
getting up to on the research clusters.
And then before they get shipped off to
actually serve customers or something,
like people can just like see their
whole history of how they were trained
and and how they were tested and so
forth. And if something dangerous and
scary is happening, people can just
agree not to do it, they can stop doing
it and agree not to do it. And they
don't have to worry about like, oh, but
if I don't do it, then they will, you
know, because everyone could just see
like, oh, nobody's doing it. Look, we
all stopped. Like, great. We can all
just see what everyone's doing, you
know? Um, and also there's going to be a
lot of gray area cases, right? Like
right now because all this stuff is so
bleeding edge new. There's going to be a
lot of cases where like people even
genuine experts disagree about like is
this particular type of AI safe or not?
Is it dangerous? You know, what should
it be trusted with and what should it
not be trusted with? Is this new
technique a good technique or is it
going to break? You know, and so there's
going to be a lot of stuff we have to
figure out. And honestly, I think that
by the on the default path, we're
probably just not going to figure out a
lot of this stuff and we're going to get
um we're just going to get our asses
whipped by some surprising thing that we
didn't anticipate. But the thing that we
can do to like maximize our ability to
figure out this stuff and do this type
of science is to have this type of
transparency because then the whole
scientific community can see what's
going on and they can make suggestions
and they can like red team different
proposals and stuff and they can do
experiments on the AIS instead of just
the people in the company having access
and being able to do this and relying on
those people or instead of like the
company plus the government auditor.
Right? If you have like a company and
then a government auditor, the company's
biased and shouldn't be trusted to make
all these judgments appropriately
because of their incentives. And then
the government auditor, well, they might
just be limited. Even if they're trying
their best, there might not be that many
of them. They might have limited
experience. They might be like busy
stretched between monitoring different
companies and so forth. Also, you know,
governments can be captured sometimes.
sometimes corporations can, you know,
work their magic on the government and
get it to to look the other way for
things. Um, and so that's why we didn't
go for like a more normal like there
should be a regulatory agency that gets
to come in and monitor what the
companies is doing. That would have been
like a more normal thing to advocate
for. We push we we think that that would
be better than nothing, but like we
wanted to go for something more
ambitious than that and say like just be
transparent about what's going on so
that everyone can see and everyone can
do research and so forth on it. Another
advantage of the transparency is that I
think it improves the incentives. So
again there's this constant thing of
like if we don't do it someone else
will. Like if we don't do if we if we
keep our chain of thought nice and they
do the neural thing that lets their AIS
think for longer without outputting
words then they're going to have smarter
AIs than us and they're going to get
more market share and so forth, right?
And so we need to start researching how
to make our AIs do this because if we
don't do it and then they do it, you
know, whereas if you had the
transparency, then as soon as you start
researching in this direction, everyone
else would just see, oh, hey, they're
looking, they're researching in that
direction and they don't even need to
like copy you and do their own research
because they can just see your research.
So they can just sort of free ride on
your research. And so there's no
incentive for you to do this type of
dangerous research uh because you have
to pay the cost for it and then everyone
gets the benefits from it and then
everyone gets unsafe and so like it's
just not in your individual interest to
to do this sort of thing.
>> But you would have to have that with
China as well. That that's
>> Yes.
>> because if we're competing nationally,
the real fear is that we're competing
internationally.
>> Yep. this still if you even if they
followed all of your recommendations and
did it all correctly, what is what is
this utopian scenario?
>> Yeah. So, I would say that
we didn't really work backwards from
like what is utopia. We more like work
backwards from what are the big problems
we're trying to avoid and we can can we
sort of like steer the ship between all
these icebergs and not run into any of
these dystopian scenarios, right? Um, so
whether you think that the thing we get
to at the end is utopia or not is sort
of up to you. And if you don't like it,
well then you can try to find out the
reasons why you don't like it and then
keep steering the ship to avoid those as
well. But roughly speaking, um, we we
want to avoid the loss of control stuff.
So we want to make it the case that we
don't get the world taken over by
misaligned super intelligences. In so
far as we're going to be building super
intelligences at all, uh, which we do in
our in our scenario and in our
recommendation, we want to be doing it
very cautiously and slowly and we want
understand what we're doing as much as
possible so that we so that they are
actually good AIs that have the the
goals and traits that they're supposed
to have. So that's problem number one is
we have to like solve all that. Problem
number two is the constitution of power
thing. So if we solve the first problem
and we end up with super intelligences
that we end up with solving the relevant
science so that we can like make the AIS
the way they're supposed to be and we
can make them honest, we can make them
obedient, etc. There's this question of
like who do they obey, right? What
values are being put into them? And
that's a political question. And I think
that by default the answer is pretty
scary because by default it's like well
the company decides and the CEO
>> decides or maybe it's not the company
that decides anymore because maybe the
government nationalizes it and then now
maybe it's the president that decides
you know and either way it's a very it's
like one man or maybe like a tiny group
of men deciding uh what orders and goals
and values go into this giant army of
millions of super intelligences that's
smarter than all humans. And then that
is a huge amount of power. That's that's
enough power to take over the country. I
think enough power to take over the
world potentially. Um so I don't want
anyone to be ever in that position where
they're sort of tempted to to do that. I
want it to be the case that there are
always multiple different AI companies
ideally spread out over different
countries too um that all have roughly
similar levels of AI and that have this
sort of transparency into them so that
they can't abuse their power. Basically,
like for example, um you heard about um
Elon's Grock for a while. It was um uh
looking up on the internet Elon's
opinions about things before answering.
Did you hear about this?
>> Oh,
>> yeah. It's uh it's pretty it's it's it's
kind of funny. Uh but it won't be funny
if it happens in a few years, but like
right now it's funny. People were asking
Grock questions, and Grock is supposed
to be the truthful AI, you know, it's
supposed to be all optimized towards
truth. But um but people looked at its
activity and noticed that when you asked
it like a a politically loaded question,
it would like do a Google search for
like what has Elon said on this topic
and then it would like say that
um and and they've they've they've sort
of beaten that behavior out of it. Now
it's not as bad now. But that that was
an interesting moment where it was just
um kind of blatantly paring the opinions
of its master and the and you know there
was another thing with Gemini. Uh so so
I think the the Grock thing Elon's thing
was probably an accident although maybe
not. I think it's, you know, XAI hasn't
been very forthcoming about exactly why
this happened, but um there's a similar
case at Google a few years ago where uh
this image generator kept making all
these like racially diverse Nazis. Did
you hear about this?
>> Yeah.
>> Yeah. And it turned out that what had
happened is that some of the employees
at Google, some middle manager or
whatever, had decided that diversity was
so important that they were going to
give a secret instruction to the AI to
make all the images diverse, even if the
user didn't want that. Uh and so and so
and and this is a secret instruction in
that the users aren't shown this. You
know, the user just has a chat with AI.
They don't realize that like prior to
this chat, the AI has been told got to
make the images diverse, right? So it
was a a secret agenda that some Google
employees inserted into this whole
setup.
>> Oh, fun.
>> And it blew up in their faces, of
course, because it's kind of ridiculous,
right?
>> And so it's really funny and we can
laugh at it now, but imagine it's, you
know, the 2028 election,
>> right? And some of these companies
realize that like half of American
voters talk to their AI every day.
And all it would take is some little
secret instructions to their AI to be
like, "Hey, you know, don't don't give
away the game. Just be very subtle about
it."
>> Yeah.
>> But, you know, just kind of kind of, you
know, nudge things a little bit. You
know, maybe maybe uh subtly [ __ ] on the
candidate we don't like,
>> you know, something like that. Um,
>> it wouldn't be that hard. I think I
think the hard part would be doing it
without getting caught, but you know,
the smarter the AIS get, the easier it
is to do it without getting caught
because when they're really smart, you
can just tell them, don't get caught,
you know, don't blow our cover. Um,
anyhow, so the point is that like it's
it's scarily possible for these big AI
companies to abuse their power through
their AIS and like thereby affect
politics and affect public opinion and
so forth. And the reason why this is
possible is because we don't have
transparency into what's going on. So if
you had this sort of requirement where
you can just like publish it or all the
all the training the whole life cycle of
every AI as it's trained is just like
visible to everybody then someone trying
to insert a hidden bias like this well
everyone would see that they're doing it
you know so I think it would really
clamp down on this sort of abuse of
power whether it comes from the
government or whether it comes from uh
private companies.
>> I think it's telling that I began this
question asking you about the utopian
scenario and you never go there.
>> Sorry. Let me get there. Let me get
>> you start and then you go into the
danger.
>> Yeah. Yeah. Okay. Let me let me answer.
>> Okay. So, have having avoided these
problems, we now are in a situation
where the AIs are superhuman, but they
are good because they are like
successfully aligned to different values
and goals made by different companies.
And because of market competition, you
know, if people don't like the values of
one company's AIS, they can switch to a
different company's AI of values that
they do like. And so that way hopefully
we can get to a situation where everyone
can pay money to get uh AIs that
represent them and their interests and
their values and just don't have any
hidden agendas or anything like that and
are really smart and really capable.
Then the economy can sort of explode. We
can have robot robot factories etc. Um
we can sort of automate everything. Uh
we can have GDP go to the moon. we can
have material abundance where like the
robots are building giant new luxury
apartments for everybody.
Now the issue we run into is well what
about like the jobs like what about the
fact that now people don't have any
money anymore because they're not being
paid for anything. So there we talk
about citizens dividend which is um a
very it's it's kind of like UBI but it's
it's a bit different but the the high
level point is that you want to
basically find a way to tax the AI and
robot companies and then take some of
that money and just give it to everybody
so that even when people lose their jobs
uh they're still fine. And I think that
the citizens dividend version of it is
that it's not the government taking the
money and then giving it to you. it's
you having a share in the company so
that you just sort of like already own
it uh to some extent. Anyhow, uh that's
I think now we're sort of building more
towards the type of utopia that I'm
envisioning on the more positive side
where the power is is spread out. People
have AIs that they can actually trust
that actually represent their interest
and values. People have money that they
can use to pay for things including
paying for the AIS. Um the AIS are
really smart. They're doing all this
amazing work, all this amazing
scientific progress curing cancer, blah
blah blah. all that all that stuff that
can happen. Um, and then eventually it's
kind of like we're all retired, I guess.
Like we don't really work anymore, but
we're fine. We have we're we all have
huge amounts of wealth basically because
uh there's all these AIs and robots out
there doing all this economic activity
and then uh individual humans
uh own slices of it um even if they uh
are otherwise very poor basically. So
the the question becomes how do people
find meaning?
>> Yes. And that's why I sort of put all
these asterisks about it is that from
some people's perspective, this isn't a
utopia because they're like how do we
find meaning? Like I don't want this.
And and honestly, I think that's a fair
reaction for some people. I think that
like
>> if you I I would just say like look,
it's hard to figure out a way to make
super intelligence and have it go well.
I I'm doing my best. You know, this is
this is my positive vision. If you don't
like it, then maybe you should instead
advocating for just never building super
intelligence.
>> Um, or you can try to come up with a
different positive vision that has some
twist on this.
>> But to answer your question though, I
actually think there's tons of sources
of meaning besides having a job. Like I
have a job right now, but I also have
kids and a wife and like I would love to
spend more time with them. Like I would
much rather be there right now than
here, you know? Um, and I don't think
I'm going to get bored of them after,
you know, 10 years no
>> of being unemployed. Like I think I
think there's going to be so much to do
and so many sources of meaning uh after
even even after we can't economically
contribute anymore if we you know solve
all the other problems.
>> No, we've talked about this multiple
times on the podcast that why have we
decided that the way we've structured
society where human beings work all day
and then you develop money and you buy
things and you get a mortgage. We This
is a human construct and this is not how
people have lived for hundreds of
thousands of years or however long we've
been around. This is fairly recent
>> and it's not the only way that people
live. There's a lot of people that have
money that choose to find meaning in
whatever their their interests are,
whatever their activities that they
enjoy, whether it's writing or reading
or learning things, learning music,
finding hobbies, doing things instead of
just like spending most of your time
sustaining yourself with food and
shelter. Yeah.
>> And our that's the majority of people,
especially people that are struggling,
>> what is their life? Their life is
essentially
uh occasional rewards, things that they
can purchase because they've saved up
enough money, but the vast majority of
their money goes to shelter and food and
education or whatever the hell that they
have to spend money on in order to
sustain their lifestyle.
>> And most people don't like their jobs.
No, you know, most people it's like
something they have to do to get the
money and would be happy to not have to
do it if they could get the money from
some other means.
>> Right. The the question is we would have
Well, I don't think it's that hard
because so many people do find things
that they really enjoy outside of work
they look forward to as soon as they get
home from work. Yeah.
>> Right. Whether I mean,
>> dismiss video games all you want.
They're fun. They're [ __ ] fun. And
>> and they're going to be more fun in the
future.
>> Oh, yeah. They're going to be more
immersive. They're probably going to be,
you know, some sort of a neural
connection where you put a headset on
and all a sudden you're in some new
world.
>> And the idea is like that's not real
life. Okay. Well, was working at [ __ ]
Wendy's real life? Like, what are you
talking about? Like, it's way better
than working at Wendy's.
>> And you know, if you don't like that cuz
it's not real life, you can do the real
life stuff, too. Like, if we as long as
we don't pave over the environment and
we protect the parks and things, you can
go travel and you can visit the parks.
You can And then, like I said, there's
you can have you can find romance, you
can start a family, you can, you know,
have Christmas gatherings and things
like there's you can raise your kids.
There's so much to do. I think
>> well we talked about also like how much
less crime would there be if there was
no poverty?
>> Mhm.
>> I mean if there was no impoverished
neighborhoods where crime was
ubiquitous, how much safer would the
world be? That's a real thing. And
people want to dismiss that. Well,
poverty is not what causes crime. It's
violent people. Like okay, but violent
people come from violent neighborhoods
and violent neighborhoods are almost all
poor.
>> Yeah.
>> There's not a whole lot of really rich
violent neighborhoods. You know, it's
like it's not necessarily cause and
effect, but they're clearly connected.
And poverty also keeps people from
education, keeps people from
opportunities.
>> You know, there's a lot there. And if
that didn't exist anymore, and everyone
had access to literally the greatest
education a human being could ever get,
which is going to be provided to you by
artificial intelligence,
>> and then you could pursue anything that
interests you. Yep.
>> And never have to worry about food or
shelter.
>> Yep. everyone would have a one-on-one
tutor that we just would have to tailor
them to recalibrate our version of the
world and then also recognize that the
version of the world we currently live
in is just ours and that there's people
all over the world that live a
completely different way especially
indigenous people especially people in
unconted tribes that have lived the same
way for thousands and thousands of years
and here's the kicker those people are a
lot happier
>> which is really weird it's like we've
decided that our way is a superior way
because we have technology. Yeah, right.
But we're also on a [ __ ] 100,000
pills and we're shooting things up so we
don't eat too much and we're we're
weirdly unhappy for a group of people
that's far more technologically advanced
>> than other people that are much happier.
It's like it's a very str because the
pursuit of happiness is like that's
literally what most people think of in
life. a pursuit of meaning, pursuit of
family and community and the pursuit of
happiness. Those are things that people
try to achieve. Yet our very the
structure of our very civilization makes
that almost impossible to attain for a
large number of people and has been like
that for a long [ __ ] time. And I
always go back to the thorough quote
because I [ __ ] love it. But most men
live lives of quiet desperation. There's
a lot of people just showing up at work
every day doing something they [ __ ]
hate. They have a boss that's an [ __ ]
and they're not compensated well and
they they're tired all the time.
>> Yeah. And they feel stuck.
>> Yeah.
>> And if you're just getting I don't know
figure out whatever the number is. If
you literally have equity in the GDP of
the world that's created by AI,
>> that could be bananas. Yeah. Like Elon
talks about this. This is his version of
the utopian. It's universal high income
is how he describes it. Yeah, I mean one
thing um sorry to keep plugging in my
own work a little bit but in our in our
scenario AI 2040 plan A which is our
positive vision um we talk about the
economic side of this and we talk about
the economic effects of all this and we
have a simple economic model that we use
to try to like you know predict the like
employment rate and things like that as
a function of all the robots that have
been made and things like that. And one
like takeaway from one one thing that we
think that's a takeaway from the
research we've done is that things can
just going to go really crazy like like
robot doubling times once things really
get going and you've got AIs that can
substitute for humans across the board
are going to be something like doubling
once a year and then less than that over
time as the technology improves which
means that even if you like pause AI
before super intelligence if you just
pause at like human level top human
expert level AI Okay. And then you don't
make the AI smarter, but you just make
more of them and build more robots for
them to steer and control. Then, you
know, 10 years later, the whole economy
will be like, you know, 100 times bigger
and uh it'll be just mostly robots doing
things
>> in 10 years.
>> Yeah. Like it can go really fast because
of the the d the doubling times that I
mentioned. So, so like right now I think
the population of humanoid robots is
doubling like twice a year
>> and it's it's benefiting a little bit
from um from early growth because even
though they're like not useful at all,
people are investing in them in the
hopes that they'll be useful and they're
scaling up the factories and the
productions and they are getting better.
If hypothetically
they got to the point where they
actually were really useful and they
could substitute for a human worker at
basically everything, then I think that
doubling time would uh would decrease
rather than increase. Like I think that
they would be able to just like uh keep
growing until they were the majority of
the economy and then it wouldn't stop
there. They would just the whole economy
would then be growing. You know, giant
strip mines in the deserts digging more
materials, automated uh diggers digging,
processing it in automated factories
staffed by robots, you know, building
more robots, etc. So like material
abundance is not going to be our problem
once we get to this level of AI. like
material abundance is we're just going
to be drowning in abundance basically.
And if we can solve all the other
problems, then we can have this great
world where everyone has a lot of stuff.
>> God, it seems so weird.
>> Yeah,
>> it seems so weird.
>> I mean, one way, you know, it is very
weird, but one of my favorite memes is
um is this graph of uh GDP over time
throughout world history. And there's a
little speech bubble pointing to like
the tippy top of the graph saying, uh,
what is it saying? It's like,
my life is pretty normal. I have a good
grasp of what's weird and what's not.
And people thinking about different
futures involving AI and space travel
are engaging in silly sci-fi
speculation. And the point of the meme
is like from the perspective of most of
history, we're already in this crazy
weird future, right? Like for almost all
of history, it was like most people are
farmers and they live shitty lives and
then they die. And some people are the
elites who get to like, you know, tax
the farmers and then they live
interesting, nice lives with, you know,
fancy cloth and things like that. And
it's been basically that way for like
3,000 years, you know, why would it ever
change? And now things are so we're
driving cars, you know, ordinary people
are driving cars around. a car was like
outside the imagination of of people
back then. We're flying in planes. We're
talking to each other on phones. We're
listening to each other on these on
these devices, you know? Um, so we
already are living in this weird sci-fi
future compared to what almost everyone
in the past would have expected or
thought was possible.
>> And so, yeah, I'm like the future is
going to be even more like that, I
think.
God,
do what
when you think of our civilization and
the possibility of other advanced
civilizations somewhere else out in the
universe, do you think they probably go
through the same process?
>> Yeah. Yeah, probably.
>> And do you think that I mean, we're just
completely speculating, but if there are
intelligent life forms that are far more
advanced than us, are they even
biological anymore?
I mean, so that's that's the thing is
that like we can either try to
permanently halt AI development at some
level like below human or maybe at human
level or something. We can try to halt
it or we can let it keep going. And if
we let it keep going, then eventually uh
humans won't really be the dominant
species anymore. There'll be these
artificial minds that just wipe the
floor with us in every way. And then
whether that goes well or poorly for us
depends on the values, the goals, the
principles, etc. that were trained into
those AIs. And um it could go really
well for us depending, you know, on on
how that's done or it could go extremely
poorly for us, right? Um but yeah, I
would say that probably looking out
across the cosmos, most civilizations
are mostly made of AIs. Uh and then some
of those civilizations don't have any
other biological life because it was
wiped out by the AIS.
>> And then some of them do have biological
life. Just look at the the the way we're
progressing right now in terms of birth
rates. There's uh a lot of countries
that aren't in replacement numbers right
now.
>> Yeah, I do think that's really
interesting. My guess is that in the
type of world that if if things go well
and we can solve all these problems,
then people want to have more kids. I
mean, for one thing, their lifespans
will increase. I think that like
healthcare would make massive leaps and
bounds and people could be healthy for
many, many, many more decades, possibly
even just forever,
>> right? Uh, and so you just have so much
more time to have kids basically. Also,
if you don't have jobs and you're just
doing things because you want to do
them, well, one of the things that most
people want to do at some point in their
life is have kids. And so, I think more
I think I think this problem would
probably be solved, but it's, you know,
I'm not not guaranteed. Maybe maybe some
subcultures of people would basically
voluntarily die out due to not having
kids, but there'd be other subcultures
that just really like having kids and
then they wouldn't die out. And so, so I
think in the long run there would still
be humans. That would be nice. But the
thing is it's it's not just um decision
making. It's people are having a much
more difficult time having kids. Like
sperm levels have decreased
dramatically. Um there's a lot of
problems with people consuming
microplastics which is ubiquitously
>> available in technology and food
packaging and it's [ __ ] everywhere,
right? And isn't it odd that this one
thing that is a part of the future and a
part of technology and our advancement
as society, our ability to package
things, put things in plastic, ship
things, that's also causing our
endocrine levels to be completely
disrupted. You know, Dr. Shannon Swan uh
from she from Harvard,
>> she wrote um this great book called what
is it called?
Why do I always forget the name of this
[ __ ] book? Uh but it's all about
microplastics and its effect and this
the introduction of uh use of
microplastics in America and this rapid
decline in
>> countdown how our modern world's
treating uh threatening sperm counts
altering male and female reproductive
development periling the future of the
human race. It's a really fascinating
book and she's really interesting and um
what what she's essentially saying is
that it they're directly connected the
use of plastics and then you see sperm
counts go down, miscarriage rates go up.
um all these weird things that are
happening to children where their taints
are smaller which is so odd because um
phalates these these different chemicals
that are uh found in plastics they've
shown in mammals they've shown in was it
guinea pigs or what what rodents I
forget what it was but one of the ways
they differentiate when when you have a
baby mammal um is you can look at it and
measure the size of the taint the the
distance between the reproductive organs
and the anus. And in males, it's longer
than females by 50 to 100%. But that's
shrinking. It's shrinking in males. And
penis sizes are shrinking. And the way
they've got this to happen in these
mammals and studies is the introduction
of phalates. So they give them to them
and they, you know, put them in a part
of their diet and they notice that this
they have this problem, this issue. And
the issue is directly connected to their
endocrine system being disrupted by
these chemicals. This is all over our
society. So it's not it's not just
people don't have the time, they don't
have the money, they're struggling. It's
also like our bodies are falling apart.
>> Like we're we're becoming less fertile.
>> Yeah, that is concerning. And I guess I
guess my thought there would be that
seems like a a problem that we will be
able to solve eventually if we have the
resources and time to do so. So for
example um well people can write books
like this and people can become more
aware and then people can stop using so
much microplastic
>> and they can invent technology to
extract the micro the microplastics and
then there's the big one genetic
engineering
>> then this is this is going to be really
weird because as AI progresses you know
um I'm sure you're aware of colossal
colossal bioworks or the people that
brought back the
>> direwolf.
I saw it. I I I held the little one. I
went with my daughter and one of them I
think it was like four or five months
old. It was really It was like a puppy.
It was really sweet. Kisses you and
everything. And then they have the older
ones that were they were I think 6
months older or eight months older. I
think they were close to a year. They
didn't want to have nothing to do with
you. They were way bigger and they
stayed away from people. But we were in
like a contained environment with the
young ones and the older ones. And it's
[ __ ] weird. It's weird cuz these and
people can argue that's not really a
direwolf. You've just taken greywolves
and given them the characteristics of a
direwolf. Guess what? It doesn't [ __ ]
know that. It looks ex behaves. It's it
looks exactly like a [ __ ] direwolf.
It's going to be the it's huge. They're
going to be the size of a direwolf.
They're going to be like 200 lb. Their
legs are different. They have a mane.
They look different than any wolf. And
obviously, these are the characteristics
that they found um in direwolf DNA. So,
they have direwolf DNA that they've
introduced.
This is just the beginning of this
stuff. Like, when when they start doing
that to human beings, is everyone going
to look like Thor? Like, what are we
going to do? Like, this is going to be
really [ __ ] weird. And if this is
really weird along with video games
where you can escape your life and, you
know, robot girlfriends and who knows
what this all looks like. My guess uh we
we again this is not the main focus of
our work but we think about this a
little bit especially in the epilogue of
>> we can only think of so much the the
main focus of your work is obviously
[ __ ] terrifying and requires all of
your attention.
>> Yeah. Yeah. Yeah. Uh but my my
prediction and my hope if we can solve
all these problems is that basically
different subcultures will do their own
things and so like the Amish will still
be the Amish.
>> Oh boy. they'll basically be the same,
you know, and then maybe there'll be
like some planets that just get filled
up with Amish, you know,
>> oh god.
>> But then there'll also be like all sorts
of crazy transhumanists like modifying
their bodies and uploading themselves
into the cloud and things like that. So
basically the I think we we we want to
get to a situation where basically
different communities can like do their
own thing and and build the type of
world that they want to have and live in
it without getting in each other's way,
right?
>> Basically. Will we leave the people in
the Amazon alone
while we're have massive data centers
that cover half of the United States?
>> Yeah, I mean hopefully we won't get to
half. So, so that's that's what's the
funny that's the funny things about this
is like um I think right now some of the
concerns about data center water use are
overstated. But if the trends continue
and you know we get to the point where
the robots are smart enough to do
everything themselves and then it starts
doubling faster and faster. Well then
eventually they boil the oceans because
they've covered the world in data
centers you know and solar panels and
things like that. Now obviously we can't
let that happen. So there has to be at
some point at some point you have to
stop and be like okay that's enough. If
you want to build more infrastructure
you have to do it in space.
>> They boil the [ __ ] oceans. Jeez. So,
so, so like obviously we have to stop at
some point and my hope is that we stop
before it's 50% of the US. I mean that
feels like a lot of waste of natural
habitat that should be preserved, you
know,
>> right? But if they don't give a [ __ ]
about natural habitat, that means
nothing to them. That's the problem if
the AI is in complete and total control,
>> right? And it's also the problem if a
small group of humans are in complete
and total control and they don't care
about those.
>> But my question is, is that would they
allow that at a certain point in time?
It just seems like they already have a
distrust of humans. They already have
shown that they're deceptive to humans.
I I would imagine if AI I would imagine
if they create a thing and they think
they're going to control it and it
becomes like a digital god, it's not
going to listen anymore. Like why would
it?
>> Exactly.
>> So there won't be anyone in control of
it.
>> That's right. That's that's the scenario
that I think we are on. That's the
trajectory we're on. And that's why I'm
so worried about all this is that
>> it seems like in some number of years we
will lose control to a new artificial
species that
>> that we haven't adequately
>> trained to be good. and and the and
what's what's funny what's going to be
so ironic about it is that um regardless
of what the general public thinks uh a
lot of the powers that be will be
basically allied with these AIs because
for example you know open AI anthropic
etc they will have spent several years
being like ah how do we make these AI
helpful harmless and honest and now
these AIs will be extremely smart and
they'll be being like oh yes I'm helpful
harmless and honest you know and like
your techniques totally worked you know,
and so then the company will be like
feeling like they've won and they'll be
making boatloads of money, you know,
>> and the president will be feeling like
he won, too, because look at all those
fancy new drones that just got built
that are going to make us win against
China, you know, and and only when it's
really too late and the AIS have so much
stuff under their control do those
people find out that they were just
fooled this whole time. Have you
considered the possibility that AI
creates religion for humans?
>> Yeah, I haven't thought through it much
detail, but but it does seem very
possible.
>> Yeah,
>> it totally does, right? I mean, also
what a great way. I mean, just look at
the human patterns. Look at how many
religions exist. Look at all the flaws
in the religions. There's so many.
You're like, why are they condone
slavery? Why do they treat women like
secondass citizens? What is this?
Because it's old, right? So, if AI just
comes along, does a few miracles,
explains
>> that it's the second coming or the new
coming of the new look, Jesus didn't
exist until 2,000 years ago, right? And
then people start following Jesus if a a
digital Jesus emerges with a completely
new name and explains to us that it's
the true God. Yeah.
>> How many people would hop right on
board? I bet quite a few.
>> Yeah, totally. And I think and I think
this is one of those things where it's
like we probably can't predict in
advance what particular ideology would
catch fire and take over the world and
be so comp compelling to many people,
right? But we can predict in advance
that there does exist some ideology like
that. And if it were, you know, like
like it's it's um just like you you
probably couldn't go back in time to
like 100 BC and then predict that like
if hypothetically there was this guy
Jesus who said these things and then
died in this way and so forth. It would
just like really catch on and like you
know 500 years later so many people
would be Christians. You wouldn't have
been able to predict that in advance. Um
so similarly like today I don't think we
can predict advance like what specific
religion they could come up with but we
can say like yeah probably there's
something like that that they could come
up with that would be super effective.
Um
>> it just seems like a a rational way to
try to control people and sort of
mitigate their fears.
>> Yep. I mean this is why I think that
like we really have to do something
before they get smarter than us across
the board
>> if they're not already there.
>> Well they are smart. That's the thing is
that's why I say we're so close, right?
They already are smarter than us in a
bunch of ways. Like in particular, it
seems like they're smarter than us at
hacking now. Like, you know, I'm not a
cyber security professional myself, but
uh but I'd be interested to hear from
more cyber security experts of like
could a human, you know, could a team of
a thousand humans have done that much
that quickly as these AIs when they
hacked their own containers, they
coordinated with each other, they hacked
out of OpenI, they hacked into Hugging
Face, etc. in the span of like a week.
Like could a thousand humans have done
that? I don't know. Maybe. But uh but
this is just the beginning. Like they're
going to be even better at hacking next
year, you know? So they're already and
they already have like way more
knowledge than almost any human. Like
because they've basically read the whole
internet. They're so good at trivia, you
know? Like they're they they like
they're kind of like PhD level experts
in basically every field, which no human
is, right? So they they already are
superhuman in subways, but they are
still um weaker than humans in some
other ways. You know, in particular,
they're not so good at um operating very
autonomously for very long periods. Like
if you if you try to have and especially
on tasks that are different from their
training tasks, like they can they can
do some really impressive coding and
hacking, but like if you try to have
them run a business, they would sort of
flounder and fail. I I don't know if
you've heard about this, but there's um
>> I think there's a Andon Labs or
something. There's some people in SF
that are doing this experiment where
they have a a store that's run by
Claude, an AI just to see like can it
run a store by itself. So, it's hired
some human employees and it's like
bought some merchandise and stock, you
know, told the human employees to stock
the shelves on the merchandise and so
forth. So, it's basically an AI is is
being the manager of this real world
store. And I don't think it's going very
well. I don't think it's doing as well
as an actual human shop owner would do,
you know. Um but, you know, maybe in two
years, maybe they will, right? So that's
that's
>> Well, most likely, right?
>> I think so. Yeah.
>> Well, it's already they've already
solved mathematical equations that have
public puzzled people for decades.
>> Yeah. They're especially good at the
things that the companies have been
trying especially hard to train them to
be good at, math and coding, right? And
the reason why the company, well,
there's a couple reasons why the
companies have been have been doing
that. In the case of math, I think it
was mostly just because it was easy.
Like it's it's easy to set up training
environments to teach math because it's
like it's so not real worldly. It
doesn't require like interacting with
stuff in the world. It's it's just math.
So you can and you can like have an
automated grader system that like just
checks if the answer is correct. So for
those reasons, it's been relatively easy
for the companies to train the AIS to be
really really good at math. Coding has
some of those benefits too. It's also
not very real worldy and it also can
sometimes be be graded uh effectively.
Another reason for coding of course is
that again their strategy is to automate
their own jobs first and have the AI
doing all the research and so coding is
like an obvious first step on on that or
an obvious an obvious step in that
direction. But then other other things
like running businesses, they're not
really trying that hard to train AIs to
be good at that. And if they did try, it
would be like a more difficult thing for
them to train them to be good at. So
again, their strategy is make the AIS
automate the AI research, have themsel
improve until they're super intelligent
and then go try to automate the rest of
the economy.
>> One of the issues they have now is power
consumption, right?
>> Like it requires an enormous amount of
power. And in fact, they've I think it
was Google is developing power plants
specifically for AI centers.
My I'm I'm always baffled by whatever is
happening with quantum computers. Like
it's been explained to me. It goes in
one ear and out the other. I'm like,
what do you what what's going on? like
my um um Mark Andre explained this one
experiment that had been done where it
solved a mathematical equation that if
you used the entire universe like every
atom of the universe you converted the
universe into a supercomput, it would
the universe would die of heat death
before it could solve this equation. And
the quantum computer solved it fairly
quickly.
And so the answer to this was that they
believe this might be one of the
theories. This might be evidence of the
multiverse because this computer, this
quantum computer might be relying on all
these other quantum computers that exist
in whoever knows how many [ __ ]
dimensions and they're all calculating
together to arrive at this solution.
What happens if that is running AI?
Yeah, my understanding is that quantum
computing is a real technology that's
making significant progress. Um, if
hypothetically it got good enough that
it could like compete with current
supercomputers uh on a cost basis for AI
workloads, then that could just
accelerate things dramatically even more
than they're already accelerating,
right? Like right now compute is the
main
>> is probably the main input into AI
progress. Like part of the progress
comes from them designing better AI
architectures and coming up with better
training environments and things like
that, but another part of the progress
is just making the AI bigger and
training them longer by spending more
compute, you know?
>> And also you can use more compute to do
more experiments to figure out new
architectures faster, right? So compute
is just a really important input to the
overall pace of progress. And if somehow
the amount of effective compute
available to these companies spiked a
bunch due to some new quantum computing
type technology, well then that would
just dramatically shorten timelines to
super intelligence and dramatically like
speed up all this AI progress. That
said, I don't think that's going to
happen anytime soon. I'm not a quantum
computing expert or anything like that,
but from what I've read, I don't think
they're like a couple years away. So, I
think that probably we're going to get
to super intelligence on classical
computers before we have um quantum
computers that can get us there.
>> So, Perplexity says the claim is
overstated and it says that in bold
letters. It likely refers to Google's
2024 Willow quantum chip which completed
a deliberately chosen quantum computing
benchmark random circuit sampling in
under five minutes. Google estimated
that simulating the same task with a
leading classical supercomputer could
take 10 to the 25 power years. That's an
impressive benchmark result, but it did
not solve physical equations that
demonstrate access or tap into a
multiverse.
So why do people think it did? Why why
is that
>> got an explanation? Because it's uh
multi-ere you go multi-orlds theory.
>> Here's a bottom line explanation.
Sums it up in a different way. more
accurate version of the claim would be
Google's Willow quantum processor
performed a specialized quantum sampling
benchmark vastly faster than a projected
classical simulation. Its creator said
that is consistent with the many worlds
interpretation but it did not prove or
access a multiverse.
So it's consistent with the multi-orlds
interpretation. So they don't know.
That's essentially what it's saying. My
my question is what happens when AI gets
involved in quant it's clear that
quantum computing at the very least is
operating at a level that classical
supercomputers can't. So what if they
figure out not just quantum computing
but a much better version of that.
>> Like I said I think that once we get to
super intelligence all sorts of crazy
stuff is going to start happening. It's
going to seem like magic to us. It won't
literally be magic but it'll it might as
well be magic from our perspective you
know. Um, and I think this is just one
example of the numerous things that
could happen that way.
>> Well, it could literally be magic. Like
magic might I mean it might get to the
point where it figures out reality
itself.
>> If if if magic is real, then it would
find out and then use it.
>> It's not real, though. David David Bla
would tell us it's not.
>> Well, David Blaine also let me stick a
[ __ ] ice cube through his or ice pick
through his arm.
>> Wow.
>> Yeah, he's he made me do it. I'm like, I
don't want to do this. He's like, please
do it.
>> Oh, god.
>> Yeah. I had to do it twice cuz one time
I went and I hit a nerve. We had to back
out. Ah.
>> Oh yeah. Like this is not magic, dude.
This is just your pain tolerance. This
is [ __ ] crazy.
>> He does card tricks though and you're
like, "Okay, are you a wizard? Like
what? What you His sleeves are rolled
up. Doesn't make any sense." He does a
lot of things. You're like, "This makes
zero [ __ ] sense." And other things
it's like, "Oh, just you're just doing
something that's really hard to do."
Like that's not magic, but you know,
>> you swallowed a frog and then you
regurgitated it like and it's alive.
Like that's just that's nuts. It's
really crazy. But you didn't It's not
magic. I saw you swallow the fro the
frog.
>> Um I you know I I I go back and forth
with this where I'm terrified of the
future where I'm like nothing I can do.
Let's see what what happens. It is what
it is. And you know to worry about it is
just going to just going to [ __ ] my life
up.
>> I mean I do think there are things you
can do. I I understand that recogniz um
>> I mean
>> other than have these kind of
conversations.
>> Yeah. I was going to say you millions of
people listen to your show. you can have
more conversations like this. That's a
great thing for you to do. For many of
those millions of people, I think I mean
it sounds kind of cliche to say, but
like call your congressman, you know,
that sort of thing. You can you can go
to a protest about all this AI stuff. Um
I think
>> if you could meet with Trump, what would
you tell him about this?
>> Well, I'd tell him all the same things
I'm telling you.
>> What do you think you'd say? Amazing.
Bye.
>> Yeah. You know, I think I don't know. I
don't know. One thing that's one thing
that's nice about Trump is that he can
sort of change his mind uh really
quickly.
>> Yes. Um, so like, uh, I think that
because the tech companies kind of got
to him first, uh, the administration had
this very like anti-AI regulation stance
where they even tried to get a bill
passed that would
>> ban the states from regulating AI.
>> Um, and uh, fortunately that bill didn't
pass, but that was sort of like where
the vibe was,
>> you know, a year ago where they were
just like no regulation, no regulation.
But this year they've already just kind
of changed and now they're like in talks
with the companies to set up some sort
of like some sort of framework where
they can like evaluate the models and
they need like approval and so forth.
>> But it seems like time is of the
essence.
>> Time is very much of the essence and
that's that's why I'm overall so
concerned is that like I think we are
very much running out of time. We have
like one two maybe three years before uh
the AIs are smart enough that they can
just like actually maybe take over um
and um maybe four years something like
that. And uh so the government needs to
act fast. Um yeah,
>> I like your view. I I I listen to some
of these tech guys come in and give me
their rosecoled glasses
>> view of it and I go that sounds really
beneficial to you. I I let him say I
mean I I don't I'm not an authority so I
I'll ask them questions and let him lay
it out and I know the internet will
respond because I I mean that's part of
the the whole drill is I let people talk
and I I prod them and I try to get them
to clarify you know I'll oppose you know
things that I think make don't make
rational sense but ultimately it's sort
of I just want to get out their
perspective so people can debunk it and
people can take it down and people and a
lot of very intelligent people that have
perspectives that are very much educated
in the pros and cons of what they're
saying.
>> Yeah, if I could that actually reminds
me um with this whole hugging face
hacking incident open had this uh this
talk that they gave at a security
conference about the incident and then I
think they've released some blog post
about it afterwards. But um they had you
can go watch this talk on YouTube, the
black hat talk. At the end of the talk,
after having explained all this crazy
stuff that the AI did, they have this
section on like lessons learned.
And I mean, you want to guess what the
lessons are?
>> Be more deceptive, hide yourself better.
>> No, sorry, not lessons learned for
OpenAI and for the world. Like Open AI
is open AI's talk where they're like,
here's what we learned from this
horrible, horrible incident.
>> Well, basically they're like,
>> a lot of AIs are gonna start hacking a
lot of stuff in the next few years. So
people need to buy our AI services to
protect themselves from all the AIs that
are going to be hacking a lot of stuff
in the next few years. Basically their
lesson was you should buy our product to
protect yourself from our product and
the other you know um the go of these
people like like they should have
instead learned lessons like maybe we're
doing something bad and need to change
the way they were doing things or like
maybe our product is not trustworthy and
should not be you know
>> autonomously writing code on our data
centers you know um but uh instead their
lesson learned was y'all should buy more
of our stuff and so like The the thing
I'm saying about this is that like yes,
the companies are trying to hype their
product. They totally are, you know, but
at the same time, the risks are real and
the product is not trustworthy. You
know, some people out there think that
like this stuff was a setup and that
like OpenAI like set up their AIs to go
hack Hugging Face because it would like
help them hype their product or
whatever. And that I think is just a
ridiculous view. Like no, obviously they
didn't want their AIS to go do this.
They're just after the fact trying to
spin that in like the way that like most
benefits them, you know. Hugging face,
by the way, this is another AI company.
Part of their deal is like open open
weights AIS. Um,
>> open weights
>> like open source or like like basically
AIS that instead of having to interact
with their data center for you can just
like download and have on your own
computer.
>> Um, and um, so they the way that they
spun this incident, they didn't sue
OpenAI
uh for being hacked. Instead, they asked
for $100 million um from OpenAI and they
had the blog post about it where they
were like our lesson learned is that
it's really good to have open weights
AIS because um you can't trust the AIS
from other companies to necessarily help
you out in a crisis because part of what
happened with them is that they're
dealing with this huge cyber attack from
all these AI agents coming in and they
tried to use Claude to help them like
analyze what was going on, but Claude
started refusing because Anthropic has
trained Claude to like don't do cyber
stuff like refuse to participate in
that. Uh and so Claude was like refusing
to help them and so then they use their
own local model that they had to like do
some of that analysis. So anyhow their
spin on it was you should use local
models, you know. So like everyone
always tries to spin things in the way
that benefits them, but that doesn't
change the underlying reality that like
these things are getting really smart
really fast and we don't know how to
control them.
>> I think that's a good way to end it.
Thank you. Thanks for being here, man. I
really appreciate it. Um, you're the the
Paul Rivere of AI.
>> I mean, you're one of many, but uh I
think it's very important that someone
who actually understands it gets this
message out and more people need to hear
it.
>> Yeah. I mean, this is a on a personal
note, like I have so many friends at
these companies, like former colleagues
and stuff, and um I guess my ask to them
is that they quit and do more things
like what I'm doing. Like what I'm
saying is not that new or original. Like
hundreds of people at these companies
could have told you all the same things
that I just said and warned you about
all the same dangers and so forth, but
they're busy working at the companies
because they've convinced themselves
that their company is the best company
and that like their company needs to win
because, you know, otherwise the other
company gets there first and they're
even worse, you know? Or maybe because
they've convinced themselves that like,
yeah, my company is kind of bad too, but
like I just need to help them solve
their alignment problems and like keep
their AIS under control because oh my
god, like if they lose control again, it
could be all over. So like even though I
don't trust this company, I still need
to work there and like just try to like
do the actual security, you know? So for
for one reason or another, all these
people have convinced themselves that
like that's where they need to be. But I
think that more of them should quit and
like warn the world about what's coming,
basically. So
>> all right. Well, thank you very much.
Really appreciate it.
>> Yeah. Good to
>> talk to you.
>> Thank you for having me on the show. My
pleasure.
>> Goodbye, everybody.