Yeah, I think it’s the amplitude of the swings we need to be concerned with. The supposed mean-reversion tendency (or not) is not load bearing. A big enough swing still wipes us out.
Supposing we achieve AGI, someone will happen to get there first. At the very least, a human-level AI could be copied to another computer, and then you have two human-level AIs. If inference remains cheaper than training (seems likely, given how current LLMs work), then it could probably be immediately copied to thousands of computers, and you have a whole company of them. If they can figure out how to use existing compute to run themselves more efficiently, they’ll probably shortly thereafter stop operating on anything like human timescales and we get a FOOM. No-one else has time to catch up.
Yeah, I think it’s the amplitude of the swings we need to be concerned with. The supposed mean-reversion tendency (or not) is not load bearing. A big enough swing still wipes us out.
Supposing we achieve AGI, someone will happen to get there first. At the very least, a human-level AI could be copied to another computer, and then you have two human-level AIs. If inference remains cheaper than training (seems likely, given how current LLMs work), then it could probably be immediately copied to thousands of computers, and you have a whole company of them. If they can figure out how to use existing compute to run themselves more efficiently, they’ll probably shortly thereafter stop operating on anything like human timescales and we get a FOOM. No-one else has time to catch up.