Thank you, I missed it while looking for prior art.
devrandom
Karma: 38
We are headed into an extreme compute overhang
If we haven’t seen such an extinction in the archaeological record, it can mean one of several things:
misalignment is rare, or
misalignment is not rare once the species becomes intelligent, but intelligence is rare or
intelligence usually results in transcendence, so there’s only one transition before the bio becomes irrelevant in the lightcone (and we are it)
We don’t know which. I think it’s a combination of 2 and 3.
The app is not currently working—it complains about the token.
and thus AGI arrives—quite predictably[17] - around the end of Moore’s Law
Given that the brain only consumes 20 W because of biological competitiveness constraints, and that 200 KW only costs around $20/hour in data centers, we can afford to be four OOMs less efficient than the brain while maintaining parity of capabilities. This results in AGI’s potential arrival at least a couple of decades earlier than the end of Moore’s Law.
Good point, but a couple of thoughts:
the operational definition of AGI referred in the article is significantly stronger than the average human
the humans are poorly organized
the 8 billion humans are supporting a civilization, while the AGIs can focus on AI research and self-improvement