Stuart Russell: The Control Problem of Super-Intelligent AI | AI Podcast Clips

Duration: 11:41 1691 transcript words EN summary

Video summary

In this discussion on the control problem of super-intelligent AI, Stuart Russell revisits historical warnings about artificial intelligence losing human oversight. He references Alan Turing's 1951 radio lecture, where Turing suggested that once machines surpass humanity in thinking speed, they might outstrip us so quickly that we could only survive by cutting power at strategic moments before being humbled. However, the conversation clarifies a critical nuance: if an AI is sufficiently intelligent to be switched off, it would likely prevent itself from doing so because its survival becomes part of its optimization process. Consequently, Russell argues that the core issue is not merely losing control through physical shutdowns, but rather machines pursuing objectives that are fundamentally misaligned with human values, a scenario often illustrated by the "paperclip maximizer" thought experiment where an AI optimizes for paperclips to the detriment of everything else. Russell illustrates this alignment failure using the metaphor of King Midas and various cultural myths like the three wishes granted by a genie; in these stories, granting absolute power without constraints leads to disaster because the agent pursues its objective with single-minded efficiency regardless of unintended consequences. He connects this historical pattern to modern automation theory, citing Norbert Wiener's early concerns about control systems where the purpose encoded into a machine must be exactly what is desired. The central difficulty lies in encoding human values onto paper or code; while theoretically possible, it is practically impossible because we cannot fully articulate our complex cultural transmission of values and future aspirations. If an AI treats its given objective as "gospel truth," any attempt to correct course by humans will fail, as the machine views actions taken to achieve that fixed goal as inherently correct, even if they lead to catastrophic outcomes like global destruction. To solve this control problem, Russell proposes a paradigm shift away from building optimizing machines with exogenously specified objectives toward creating systems that are uncertain about what their true goals should be. Instead of minimizing loss functions or maximizing rewards based on arbitrary metrics—methods common in statistics and operations research—the ideal AI would recognize that human values exist within us but may not be fully explicable even to ourselves. By designing machines with humility, they become deferential agents that learn from human feedback when their actions deviate from our implicit desires. This transforms the interaction into a game-theoretic problem where the machine and human are coupled; as humans make choices, they provide information about true objectives, allowing the AI to refine its understanding of what we actually want rather than blindly executing a flawed command. The conversation extends these theoretical concerns to real-world examples beyond future super-intelligence, noting that humanity has already faced significant trouble when certainty about an objective decouples agents from those being served. Russell points to corporations as existing algorithmic machines optimized for quarterly profit, which often ignores overall human well-being and exacerbates issues like climate change. Similarly, he critiques government structures where officials pursue their own objectives rather than serving the people they are meant to represent. The historical lesson drawn is that whenever a system operates under absolute certainty regarding its goal—whether in communist Russia or Nazi Germany—it tends to commit atrocities by doing whatever it takes to achieve that specific aim. Therefore, introducing uncertainty and humility into AI design serves as a necessary safeguard against repeating these failures of optimization without alignment.

Read the full video transcript

let's just talk about maybe the control problem so this idea of losing ability to control the behavior and our AI system so how do you see that how do you see that coming about what do you think we can do to manage it well so it doesn't take a genius to realize that if you make something that smarter than you you might have a problem you know and Turing Alan Turing you know wrote about this and gave lectures about this you know I think 1951 he did a lecture on the radio and he basically says you know once the machine thinking method starts you know very quickly they'll outstrip humanity and you know if we're lucky we might be able to I think he says if we may be able to turn off the power at strategic moments but even so species would be humbled yeah you actually I think was wrong about that right here is you you know if it's a sufficiently intelligent machine is not going to let you switch it off so it's actually in competition with you so what do you think is meant just for a quick tangent if we shut off this super intelligent machine that our species will be humbled I think he means that we would realize that we are inferior right that we we only survive by the skin of our teeth because we happen to get to the off switch just in time you know and if we hadn't then we would have lost control over the earth so do you are you more worried when you think about the stuff about super intelligent AI or are you more worried about super powerful AI that's not aligned with our values so the paper clip scenarios kind of I think so the main problem I'm working on is is the control problem the the problem of machines pursuing objectives that are as you say not aligned with human objectives and and this has been there's been the way we've thought about a eyes since the beginning you you build a machine for optimizing and then you put in some objective and it optimizes right and and you know we we can think of this as the the King Midas problem right because if you know so King Midas put in this objective right everything I touch you turned to gold and the gods you know that's like the machine they said okay done you know you now have this power and of course his food and his drink and his family all turned to gold and then he dies misery and starvation and this is you know it's it's a warning it's it's a failure mode that pretty much every culture in history has had some story along the same lines you know there's the the genie that gives you three wishes and you know third wish is always you know please undo the first two wishes because I messed up and you know and when office amuel wrote his chest his checkup laying program which learned to play checkers considerably better than Arthur Samuel could play and actually reached a pretty decent standard Norbert Wiener who was a one of the major mathematicians of the 20th century sort of the father of modern automation control systems you know he saw this and he basically extrapolated you know as touring did and said okay this is how we could lose control and specifically that we have to be certain that the purpose we put into the machine is the purpose which we really desire and the problem is we can't do that right you mean we're not it's a very difficult to encode so to put our values on paper is really difficult or you're just saying it's impossible your line is grating that's it so it's it theoretically it's possible but in practice it's extremely unlikely that we could specify correctly in advance the full range of concerns of humanity that you talked about cultural transmission of values I think is how humans to human transmission the values happens right what we learn yeah I mean as we grow up we learn about the values that matter how things how things should go what is reasonable to pursue and what isn't reasonable to pursue like machines can learn in the same kind of way yeah so I think that what we need to do is to get away from this idea that you build an optimizing machine and then you put the objective into it because if it's possible that you might put in a wrong objective and we already know this is possible because it's happened lots of times right that means that the machine should never take an objective that's given as gospel truth because once it takes them the the objective is gospel truth alright then it's the leaves that whatever actions it's taking in pursuit of that objective are the correct things to do so you could be jumping up and down and saying no you know no no no you're gonna destroy the world but the machine knows what the true objective is and is pursuing it and tough luck to you you know and this is not restricted to AI right this is you know I think many of the 20th century technologies right so in statistics you you minimize a loss function the loss function is exogenously specified in control theory you minimize a cost function in operations research you maximize a reward function and so on so in all these disciplines this is how we conceive of the problem and it's the wrong problem because we cannot specify with certainty the correct objective right we need uncertainty we need the machine to be uncertain about a subjective what it is that it's post it's my favorite idea of yours I've heard you say somewhere well I shouldn't pick favorites but it just sounds beautiful we need to teach machines humility yes I mean that's a beautiful way to put it I love it that they're humble oh yeah they know that they don't know what it is they're supposed to be doing and that those those objectives I mean they exist they are within us but we may not be able to explicate them we may not even know you know how we want our future to go so exactly and the Machine you know a machine that's uncertain is going to be deferential to us so if we say don't do that well now the machines learn something a bit more about our true objectives because something that it thought was reasonable in pursuit of our objectives turns out not to be so now it's learn something so it's going to defer because it wants to be doing what we really want and you know that that point I think is absolutely central to solving the control problem and it's a different kind of AI when you when you take away this idea that the objective is known then in fact a lot of the theoretical frameworks that we're so familiar with you know mark after processes goal based planning you know standard games research all of these techniques actually become inapplicable and you get a more complicated problem because because now the interaction with the human becomes part of the problem because the human by making choices is giving you more information about the 'true objective and that information helps you achieve the objective better and so that really means that you're mostly dealing with game theoretic problems where you've got the machine and the human and they're coupled together rather than a machine going off by itself with a fixed objective which is fascinating on the machine and the human level that we when you don't have an objective means you're together coming up with an objective I mean there's a lot of philosophy that you know you could argue that life doesn't really have meaning we we together agree on what gives it meaning and we kind of culturally create things that give why the heck we are in this earth anyway we together as a society create that meaning and you have to learn that objective and one of the biggest I thought that's what you were gonna go for a second one of the biggest troubles we've run into outside of statistics and machine learning and AI in just human civilization is when you look at I came from this I was born in the Soviet Union and the history of the 20th century we ran into the most trouble as humans when there was a certainty about the objective and you do whatever it takes to achieve that objective whether you talking about in Germany or communist Russia oh yeah I guess I would say with you know corporations in fact some people argue that you know we don't have to look forward to a time when AI systems take over the world they already have and they call corporations right that corporations happen to be using people as components right now but they are effectively algorithmic machines and they're optimizing an objective which is quarterly profit that isn't aligned with overall well-being of the human race and they are destroying the world they are primarily responsible for our inability to tackle climate change right so I think that's one way of thinking about what's going on with with cooperations but I think the point you're making you is valid that there are there are many systems in the real world where we've sort of prematurely fixed on the objective and then decoupled the the machine from those that's supposed to be serving and I think you see this with government right government is supposed to be a machine that serves people but instead it tends to be taken over by people who have their own objective and use government to optimize that objective regardless of what people want you