[@DwarkeshPatel] Ilya Sutskever – We're moving from the age of scaling to the age of research

Link: https://youtu.be/aR20FWCCjAs

Short Summary

This conversation discusses the current state of AI research, highlighting a potential disconnect between current model evaluation metrics and real-world performance. It also explores the limitations of current scaling approaches and the need for a renewed focus on fundamental research into generalization, human-like continual learning, and alignment, including the potential for a future with superintelligent AI systems.

Key Quotes

Here are five direct quotes from the transcript that I found particularly insightful:

1. "The models seem smarter than their economic impact would imply. ... How to reconcile the fact that they are doing so well on evals? You look at the evals and you go, 'Those are pretty hard evals.' They are doing so well. But the economic impact seems to be dramatically behind."
2. "One consequence of the age of scaling is that scaling sucked out all the air in the room. Because scaling sucked out all the air in the room, everyone started to do the same thing. We got to the point where we are in a world where there are more companies than ideas by quite a bit."
3. "So if you got an AI to care about sentient beings—and it's not actually clear to me that that's what you should try to do if you solved alignment—it would still be the case that most sentient beings will be AIs. There will be trillions, eventually quadrillions, of AIs. Humans will be a very small fraction of sentient beings."
4. "The whole problem is the power. The whole problem is the power. When the power is really big, what's going to happen?"
5. "It’s beauty, simplicity, elegance, correct inspiration from the brain. All of those things need to be present at the same time. The more they are present, the more confident you can be in a top-down belief."

Detailed Summary

Here's a detailed summary of the YouTube video transcript, organized into bullet points:

I. Introduction and Initial Observations:

• The conversation starts with acknowledging the rapid development of AI and its potential impact.
• The "slow takeoff" of AI feels normal, but investment figures are abstract and not yet widely felt.
• One speaker (Ilya) believes AI's impact will be felt strongly soon, diffusing throughout the economy.

II. The Paradox of AI Performance:

• Current AI models perform well on evaluations (evals) but their economic impact lags significantly.
• The disconnect between evaluation performance and real-world utility is confusing.
• Example given: an AI coding model alternating between introducing and fixing bugs repeatedly, highlighting a possible lack of fundamental understanding.

III. Potential Explanations for the Performance Discrepancy:

• Whimsical Explanation: Reinforcement Learning (RL) training may make models too narrowly focused, sacrificing broader awareness and common sense.
• More Technical Explanation: Focus on evals during RL training. Companies tailor RL environments based on desired evaluation performance, leading to overfitting on specific tasks and poor generalization.
• The true "reward hacking" might be the researchers overly focused on evaluation metrics.
• Becoming superhuman at coding competitions doesn't make you a better general programmer.

IV. Analogies to Human Learning:

• Competitive Programming Analogy: A student who intensely practices competitive programming (10,000 hours) vs. one who practices less (100 hours) but has inherent talent ("it"). The latter is predicted to be more successful in a broader career.
• AI models are likened to the intensely trained competitive programmer, lacking broader generalizability.
• Pre-training may offer a lot of data, but may not generalize better than RL. Its strengths lie in the sheer volume and the naturalness of the data.
• Discussing human analogies to pre-training, such as early childhood development or evolution, but acknowledges limitations.
• Human beings know much less than AI, even after 15 years with tiny fraction of pre-training data, they know much more deeply somehow.

V. Emotions and Value Functions:

• Discussion of a case study of someone with brain damage affecting emotional processing, highlighting the critical role of emotions in decision-making.
• Emotions are related to the concept of "value functions" in machine learning.
• Value function allows you to short-circuit the wait until the very end and provide more accurate rewards/punishments more efficiently.
• However, current ML value functions may not play a prominent role.
• Human value functions, modulated by emotions, are potentially hardcoded by evolution and crucial for effective action in the world.

VI. Scaling and the Shift Back to Research:

• The conversation shifts to a broader discussion of scaling in AI.
• "Scaling" became a dominant paradigm, but pre-training data is finite.
• From 2012-2020 the AI sector focused on research, while from 2020-2025 it was about scaling.
• The age of scaling gave companies a low-risk way to invest in compute and data to generate results.
• Now, given current scales of compute, the focus is shifting back to research with the question of identifying new scaling relationships or "recipes."
• RL is now consuming more compute than pre-training, but is the compute being used in the most productive way?

VII. Generalization as the Key Challenge:

• Generalization is the crux: Why do models generalize so much worse than people?
• Two sub-questions: Sample efficiency (why models need so much data) and the difficulty of teaching models desired behaviors compared to humans.
• Human learning seems more unsupervised and more robust.
• Evolution may play a role in human sample efficiency for skills like vision and locomotion.
• However, strong human learning abilities in math and coding suggest a more fundamental difference in learning mechanisms.

VIII. Potential Solutions and the Role of SSI:

• Possibility that human neurons do more compute than we think.
• One speaker (Ilya) hints at insights into a possible machine learning principle that enables robust and efficient learning, but avoids specific details due to the competitive landscape.
• One speaker says that the key problem is that AI doesn’t feel powerful because of its mistakes.

IX. The Vibe of Returning to the Age of Research:

• Compute remains important, but the focus will shift to new ideas and different approaches.
• "Scaling" sucked the air out of the room, leading to more companies than ideas.
• Compute power has increased a lot, but now you can develop these ideas on less compute.
• SSI aims to develop the correct technical approach for superintelligence.
• One speaker says SSI’s plan is to straight shot to superintelligence, but the plan may change if timelines are long or there’s value in releasing AI in the world.
• There are two words have shaped everyone’s thinking – AGI and Pre-training. But a human being is not an AGI – instead, they rely on continual learning.

X. Alignment and the Problem of Power:

• The whole problem of AI and AGI is the power. When the power is really big, what’s going to happen?
• A key shift in thinking is placing more importance on AI being deployed incrementally and in advance.
• It’s very hard to feel the AGI – and a lot of issues around AGI stem from the fact that it’s very difficult to imagine. You’ve got to be showing the thing.
• Companies will start to collaborate on AI safety.
• Companies that build AI should aspire to build something that is robustly aligned to care about sentient life.
• Debate regarding the ultimate goal: alignment with human values vs. alignment with all sentient life (given the proliferation of AI entities). The discussion acknowledges the complexities of ensuring superintelligence remains beneficial.

XI. Superintelligence and its Potential Manifestations:

• Multiple superintelligences likely to be created around the same time. If there are many of these, it would be nice if there was some kind of agreement or restraint.
• If a system is sufficiently powerful, even if it’s sensible like caring for sentient life, you might not like the results. Maybe you do not build an RL agent.
• Current progress will only go so far, a lot hinges on understanding reliable generalization.
• Human beings seem to generalize a lot better, so the ability to learn and optimize human values is fragile.

XII. Making AI Go Well and the Long Run:

• If the first N dramatic systems care for sentient life, things can go well for quite some time.
• Possible long-term solution: people become part-AI (Neuralink++), enabling full understanding and participation in AI-driven situations.

XIII. Evolution, the Brain, and High-Level Desires:

• Discussion on the mystery of how evolution encodes high-level desires in the brain (e.g., social standing).
• One speaker says that evolution was able to endow us to care about social stuff very reliably.

XIV. SSI's Technical Approach and the Future:

• SSI's plan to focus on certain ideas to make superintelligence go well.
• A reminder that SSI was being fundraised at a $32B valuation when Meta offered to acquire them, and the CEO said no.
• Belief that there will be a convergence of strategies toward something similar to care for sentient life as AI gets more powerful.
• Forecasts for AI that learns as well as a human being and is superhuman, is 5-20 years.
• Other companies continue the current approach, but there is stagnation. But when the correct solution emerges, it is clear to everyone.

XV. The Good World and the Role of Specialization:

• Discussion of market competition after the realization of human-like learning AI and specialization.

XVI. Self-Play and the Quest for Diversity:

• Self-play offers a way to create models using compute only.
• Lack of diversity in current models may be due to pre-training on similar datasets.
• Raising the temperature just results in gibberish - want the diversity like when different scientists have different prejudices or different ideas.

XVII. What is Research Taste?

• The episode closes with a discussion of the "taste" required for groundbreaking AI research.
• Simplicity, elegance, and correct inspiration from the brain. A top-down belief.

This bullet point summary captures the key arguments, topics, and information discussed in the provided YouTube transcript. It should be helpful in understanding the core ideas presented in the video.