I’m confused. I read you as suggesting that self-modifying code has recently become possible, but I think that self-modifying code has been possible for about as long as we have had digital computers?
What specific things are possible to do now that weren’t possible before, and what kind of AGI-relevant questions does that make testable?
Self modifying code has been possible but not practical for as long as we have had digital computers. Now it has toolchains, use cases, and in the near future tens to hundreds of people will do it as their day job.
The strong version of my claim is that I expect to see the same kinds of failure modes we are concerned with in AGI pushed down to the level of consumer-grade software, at least in huge applications like social networks and self-driving cars.
I think it is now simple and cheap enough for a single research group to do something like:
Write a formal specification
Which employs learning for some simple purpose
And employs self-modification on one or more levels
Which is to say, it feels like we have enough tooling to start doing “Hello World” grade self-modification tests that account for every level of the stack, in real systems.
I read it more as a call for change in emphasis back towards more attention on self-modification because things have drawn focus on it away to other things, not because anything specific has changed but because the pendulum has swung too far away from where it should be.
I’m confused. I read you as suggesting that self-modifying code has recently become possible, but I think that self-modifying code has been possible for about as long as we have had digital computers?
What specific things are possible to do now that weren’t possible before, and what kind of AGI-relevant questions does that make testable?
Self modifying code has been possible but not practical for as long as we have had digital computers. Now it has toolchains, use cases, and in the near future tens to hundreds of people will do it as their day job.
The strong version of my claim is that I expect to see the same kinds of failure modes we are concerned with in AGI pushed down to the level of consumer-grade software, at least in huge applications like social networks and self-driving cars.
I think it is now simple and cheap enough for a single research group to do something like:
Write a formal specification
Which employs learning for some simple purpose
And employs self-modification on one or more levels
Which is to say, it feels like we have enough tooling to start doing “Hello World” grade self-modification tests that account for every level of the stack, in real systems.
I read it more as a call for change in emphasis back towards more attention on self-modification because things have drawn focus on it away to other things, not because anything specific has changed but because the pendulum has swung too far away from where it should be.