LW server reports: not allowed.
This probably means the post has been deleted or moved back to the author's drafts.
I think there’s several remaining elements that have to built.
This is like building a nuke. Recursive self improvement is something nature doesn’t “want” to do, the conditions have to be just right or it won’t work.
Composable frameworks and software elements. Right now there are thousands of notable techniques described on separate AI papers and custom modifications that domain experts know to do. For example the recent matrix multiplication and protein folding papers had a lot of hand modified architecture that only a few have the knowledge to perform.
What this means is your recursive AI systems hypothesis needs to be limited to tractable tasks like “connect composable modules from this library of modules to form a system that will perform well on this bench”. And “design a new module from these composable neural network elements”. (Layers, etc)
The pieces have to be available for the machine to solve the puzzle.
You need a bench. An enormous one. Full of many diverse tasks and with open source contributions. With tasks specific to force a system passing the bench to exhibit generality. Many of the tasks could be existing games. And obviously some of the tasks are “design a new AGI architecture to succeed on this bench. ” Each time the system proposes an architecture already tried it gets immediate detailed feedback how the architecture performed.
You need to choose performance scoring carefully. Lots of score for held out test tasks that use information taught in multiple training tasks in a novel way not tested. Lots of score for simplicity and reuse of architecture elements between tasks.
The current situation seems to be if you construct the software backend, the bench, and a heuristic that models generality well you would get AGI on a timescale mostly governed by compute.
The actual passing models capable of self improvement might need to be enormous, using many internal neural networks, some capable of runtime introspection and activation/suppression of other networks to select them for a given task.
Assuming they are this big—say 10 trillion params just to asspull a number—this would be why it hasn’t happened yet and so much hype wasn’t realized. Each $33k H100 is only? Billion params. (10 billion during training?)
So 33 million in hardware and you will need many clusters this large so you can investigate many candidate AGI architectures in parallel.
That’s probably within an OOM of what it will take. Like with the Manhattan project—nature is unforgiving. Anything less than a critical mass of pure enough fissionable material was never going to work. No amount of small scale single investigators could have constructed a fission weapon alone, and none could have solved AGI.
Recursive self improvement is something nature doesn’t “want” to do, the conditions have to be just right or it won’t work.
I very much disagree—I think it’s absolutely an attractor state for all systems that undergo improvement.
I think you’re right and I will update my view. Just trying to reconcile this with past failures to get meaningful self improvement.
Great—yeah just because it’s an attractor state doesn’t mean it’s simple to achieve—still needs the right setup to realize the compounding returns to intelligence. The core hard thing is that improvements to the system need to cause further improvements to the system, but in the initial stages that’s not true—all improvements are done by the human.
The core hard thing may have been the TPU/RTX GPU. Had commercial industry started shipping fissionable material by the gram in some counterfactual world where the possibility of making a nuke with it wasn’t taken seriously, how long do you think it would have taken for someone to do the remaining steps? As you mention it’s an attractor state and assuming enriched uranium is now readily available, people would experiment, building neutron amplifiers at first. Then they would use the information to assemble a self sustaining fission pile—possibly delayed a few years if the economy is took a dive—and more information gain (and the plutonium) makes the nuke inevitable.
great stuff! I would give this intro to someone who was convinced it was possible and wanted to get a quick overview. I wouldn’t want to start here for someone who doesn’t think agi is happening for some reason.
Saying “a substantial amount of expertise is required” goes past what general actually implies (I use the example of 8th grade level knowledge, at which point, the system can branch in thousands of directions on it’s own).
Saying that a good test of an AGI is for it to build a better AGI, while also saying that such an effort represents an immense risk?