Why exactly do we want “recursive self-improvement” anyways?
Generally we want our programs to be as effective as possible. If the program can improve itself, that’s a good thing, from an ordinary perspective.
But for a sufficiently sophisticated program, you don’t even need to make self-improvement an explicit imperative. All it has to do is deduce that improving its own performance will lead to better outcomes. This is in the paper by Steve Omohundro (ata’s final link).
Why not build into the architecture the impossibility of rewriting its own code
There are too many possibilities. The source code might be fixed, but the self-improvement occurs during run-time via alterations to dynamical objects—data structures, sets of heuristics, virtual machines. An AI might create a new and improved AI rather than improving itself. As Omohundro argues, just having a goal, any goal at all, gives an AI an incentive to increase the amount of intelligence being used in the service of that goal. For a complicated architecture, you would have to block this incentive explicitly, declaratively, at a high conceptual level.
Generally we want our programs to be as effective as possible. If the program can improve itself, that’s a good thing, from an ordinary perspective.
But for a sufficiently sophisticated program, you don’t even need to make self-improvement an explicit imperative. All it has to do is deduce that improving its own performance will lead to better outcomes. This is in the paper by Steve Omohundro (ata’s final link).
There are too many possibilities. The source code might be fixed, but the self-improvement occurs during run-time via alterations to dynamical objects—data structures, sets of heuristics, virtual machines. An AI might create a new and improved AI rather than improving itself. As Omohundro argues, just having a goal, any goal at all, gives an AI an incentive to increase the amount of intelligence being used in the service of that goal. For a complicated architecture, you would have to block this incentive explicitly, declaratively, at a high conceptual level.