You’re right, in the sense that there’s nothing here about how to generate accurate representations of the world. According to A.2, the user provides the representations. But even if the program is just a numerical optimizer, it’s a powerful one, because it’s supposed to be able to optimize an arbitrary function (arbitrary network of nodes, as represented in the script).
So it’s as if the unfinished AI project already has the part of the code that will do the heavy lifting when problems are solved, and what remains to be done—which is still both important and difficult—is everything that involves transmitting intentions correctly to this AI core, and ensuring that all that raw power isn’t used in the service of the wrong goals.
You’re right, in the sense that there’s nothing here about how to generate accurate representations of the world. According to A.2, the user provides the representations. But even if the program is just a numerical optimizer, it’s a powerful one, because it’s supposed to be able to optimize an arbitrary function (arbitrary network of nodes, as represented in the script).
So it’s as if the unfinished AI project already has the part of the code that will do the heavy lifting when problems are solved, and what remains to be done—which is still both important and difficult—is everything that involves transmitting intentions correctly to this AI core, and ensuring that all that raw power isn’t used in the service of the wrong goals.