[@DwarkeshPatel] Andrej Karpathy — AGI is still a decade away

Link: https://youtu.be/lXUZvyajciY

Short Summary

Andrej Karpathy argues that the progress towards truly intelligent agents will take a decade, not just a year, due to the cognitive deficiencies in current LLMs, such as lack of continual learning and multimodality. He emphasizes that while existing AI tools are impressive, they are far from being the reliable "employees" envisioned for the future and much work is needed in various areas, algorithms, datasets, and compute, but particularly around removing knowledge in models to better build a "cognitive core". Karpathy also emphasizes the critical role of AI in transforming education, envisioning a future where it provides personalized learning experiences that are so engaging they are enjoyed simply for the sake of self-improvement.

Key Quotes

Here are five quotes from Andrej Karpathy in the provided transcript that offer valuable insights:

1. "I feel maybe loosely speaking, people kept trying to get the full thing too early a few times, where people really try to go after agents too early, I would say. That was Atari and Universe and even my own experience. You actually have to do some things first before you get to those agents." This quote highlights the importance of building a solid foundation (representations and LLMs) before attempting complex agent designs.
2. "Pre-training is doing two things that are unrelated. Number one, it's picking up all this knowledge, as I call it. Number two, it's actually becoming intelligent. By observing the algorithmic patterns in the internet, it boots up all these little circuits and algorithms inside the neural net to do things like in-context learning and all this stuff." Karpathy identifies two separate but simultaneous benefits of pre-training large models: knowledge acquisition and the emergent ability to learn within context.
3. "You’re right. They’re not very good at code that has never been written before, maybe it’s one way to put it, which is what we’re trying to achieve when we’re building these models." This quote underscores a critical limitation of current coding AI: its difficulty in handling genuinely novel code, something that AI engineers are constantly striving for.
4. "Literally what reinforcement learning does is it goes to the ones that worked really well and every single thing you did along the way, every single token gets upweighted like, "Do more of this." The problem with that is people will say that your estimator has high variance, but it's just noisy." Karpathy explains the problem with Reinforcement Learning as it relates to language models and makes the analogy that "you're sucking supervision through a straw" to try and provide information.
5. "I feel like that's where I can a lot more uniquely add value than an incremental improvement in the frontier lab. I'm most afraid of something depicted in movies like WALL-E or Idiocracy or something like that, where humanity is on the side of this stuff. I want humans to be much, much better in this future. To me, this is through education that you can achieve this." This reveals Karpathy's motivation for focusing on education over pure AI research: a desire to empower humanity and prevent it from being left behind by technological advancements.

Detailed Summary

Here is a detailed summary of the YouTube video transcript in bullet points, highlighting the key topics, arguments, and information discussed:

Overall Theme: The video explores the current state of AI, particularly focusing on the potential and limitations of AI agents, the need for more realistic timelines for achieving true AI capabilities, and the importance of focusing on education and empowering humanity alongside AI advancements.

Key Arguments & Topics:

• Decade vs. Year of Agents:

  • Andrej argues that it's the "decade of agents" rather than the "year of agents," pushing back against over-optimistic predictions about the immediate impact of LLMs and AI agents.
  • He believes that significant work is needed to overcome bottlenecks in intelligence, multimodality, computer use, and continual learning before AI agents can truly function like effective employees or interns.
  • Karpathy bases his decade timeline on his 15 years of experience making predictions in the field.

• Bottlenecks in AI Development:

  • Lack of "working" AI agents, lacking intelligence.
  • Continual learning: LLMs cannot remember something simply after being told once.
  • He suggests the problems are "tractable" but "difficult", so it will take a decade.
  • Lack of multimodality: limited ability to interact with the world through various inputs (vision, audio, etc.).
  • Challenges with computer use: difficulty for AI agents to effectively operate and interact with digital interfaces and software.

• Historical Perspective on AI Progress:

  • Discusses seismic shifts in AI, including:
    • The rise of deep learning with AlexNet.
    • Early attempts at agents (Atari deep reinforcement learning) which he considers a misstep and working on problems too early.
    • The impact of LLMs and their representation power.
  • He notes that early attempts to create full-fledged agents often failed due to a lack of representation power in neural networks and sparse reward systems.
  • Notes that "pre-training" is "crappy evolution".

• Criticism of Reinforcement Learning (RL):

  • Andrej is critical of current RL methods, describing them as "sucking supervision through a straw" due to the noisy and inefficient way reward signals are applied across entire trajectories.
  • He believes that humans don't use reinforcement learning for a lot of intelligence tasks like problem-solving.
  • He argues RL is a lot worse than the average person thinks.

• Evolution vs. Imitation (and Sutton's Perspective):

  • He contrasts the evolved intelligence of animals with the imitated intelligence of AI models.
  • He views current AI models as "ghosts" or "spirits" trained by imitating human data on the internet, rather than beings developed through evolution.
  • He acknowledges Richard Sutton's vision of building AGI from scratch, but argues that evolution is a much different process where a large amount of hardware is built-in.

• In-Context Learning:

  • Disagrees with the idea that in-context learning is not doing gradient descent.
  • Suggests in-context learning might run a small gradient descent loop internally, citing research papers as evidence.
  • Claims in-context learning feels more like real intelligence.

• Human Intelligence vs. AI:

  • He believes AI is missing many brain parts and nuclei, such as the hippocampus and the amygdala.
  • Suggests transformer neural networks are just cortical tissue.
  • He posits AI models lack a "distillation phase" of processing and analyzing information, like human sleep.
  • Humans have a process of distillation into the weights of the brain that LLMs don't.

• Model Collapse and Entropy:

  • He argues that LLMs are prone to "model collapse," where their outputs become less diverse and more repetitive.
  • Synthetic data generation exacerbates model collapse because synthetic samples collapse from an LLM.
  • Humans are a lot noisier and have a huge amount of entropy, while LLMs are silently collapsed and give 3 jokes.
  • He suggests that humans don't overfit yet as children. They say stuff that shocks you, while adults are more collapsed.

• Cognitive Core and Model Size:

  • His prediction is that you will have a very productive conversation in 20 years with a billion parameter model.
  • He believes it's crucial to remove memory from AI models to focus on the core cognitive algorithms, enabling them to look up information rather than relying solely on memorized knowledge.

• Limitations of LLMs in Coding (Nanochat Example):

  • While autocomplete is helpful, current coding models are not effective at building unique or complex repositories like Nanochat.
  • LLMs often misunderstand the code or add unnecessary complexity, lacking the nuanced understanding required for specialized tasks.
  • LLMs are not very good at code that has never been written before.
  • LLMs tend to misunderstanding the style, misunderstanding code, and trying to make the code production code.

• AI as an Extension of Computing:

  • Andrej views AI as a continuum of computing progress.
  • AI is fundamentally an extension of computing.
  • Tools that are all about coding, like compilers, are examples of automation in computing.
  • He says more and more stuff is automated by the "autonomy slider"

• Role of Process-Based Supervision:

  • He highlights the problem of outcome-based reward in RL, where models are penalized for what is correct.
  • There are difficulties assigning partial credit in process-based supervision.
  • The reason that process-based supervision is tricky is anytime you use an LLM, they're gameable.

• Shift from Imitation to Reinforcement Learning:

  • InstructGPT showed that stylistically, a model can adjust quickly and become an assistant.
  • First, imitation learning was surprising, then RL.

• GDP Growth Rate:

  • AI is not distinctly different with respect to the already high GDP growth rate.
  • You cannot find mobile phones in the GDP because everything slowly diffuses and ads up to the same exponential.
  • His expectation is the growth is going to follow the same pattern and nothing is going to change.
  • The President of the US has a lot of power, but their understanding of a lot of things is different.
  • I see this as just automation and things people can already do.

• The Evolution of Intelligence:

  • Says the evolution of intelligence is a very recent event in evolution.
  • Is surprised that it evolved.

• Importance of Culture:

  • LLMs don't have the equivalent of culture and the invention of a written record of passing it down between each other.
  • LLMs don't really have a culture to be able to function efficiently.

• Bottlenecks Preventing LLM Collaboration:

  • The analogy works well that the smaller models are Kindergarten students.
  • He implies that the current AI is too young to be able to collaborate in this capacity.

• Self-Driving Analogy:

  • He suggests self-driving isn't anywhere near done yet, though the demos of it are cool.
  • Self-driving has a history from the 1980s.
  • There is a demo to product gap where the demo is easy, but the product is very hard.
  • Critical safety domains enforce his timelines.
  • There is a very large teleoperation center of people who are in a loop with these cars that people can't see.
  • Thinks Tesla is taking a more scalable approach to what Waymo is doing.

• The "March of Nines":

  • In order for an AI agent to be reliable, it needs to improve by a scale of nines, like 99.999% and so on.

• The Eureka Project:

  • He believes Eureka can uniquely add value more than an incremental improvement in frontier labs.
  • Thinks it's the process of building ramps to knowledge.

• Karpathy's Current Focus (Eureka & Starfleet Academy):

  • He's building a first course, the state-of-the-art destination to learn.
  • Currently he's not building an AI tutor because that capability is not there yet.
  • A lot of AI can provide for this endeavor is a little bit like slop.
  • His goal is to build the Starfleet Academy, to explain how one teaches technical or scientific content well.

• Automated Tutor:

  • If he was to build an automated tutor, a good one has a high bar of:
    • Instantly understand where you are as a student, what you don't know and probe the world model.
    • Serve all the things at your sliver of capability.
    • Give material that is appropriately challenged.
    • Not have you be given stuff that is too trivial or too hard.

• The Importance of Education for Humans:

  • He expresses concerns about AI advancing at the expense of human empowerment.
  • He fears ending up in a Wall-E future where humans are on the side of these things.
  • He wants humans to be very good in the future.
  • Pre-AGI education is useful. Post-AGI, it's fun like a gym.

• Tips for Building YouTube Tutorials:

  • Most of his tips come from his physics background.
  • Says to boot up the brain as best as you can.
  • Suggests to build models and abstraction, understanding that there is a first-order term, a second, third, and fourth, and so on.
  • Says to approach problem-solving with the right cognitive tools that physics gives you.
  • Always simplify everything down to the smallest detail so you can get a clear understanding of it.

• Curse of Expertise:

  • Says experts tend to take certain things for granted.
  • Suggests you need to put yourself in the shoes of the people who are just starting out.

This bullet point summary provides a detailed overview of the key themes, arguments, and information presented in the YouTube video transcript.