Personally, I’d put bio capabilities higher up. I would agree with your ranking if “an AGI/ASI decides to kill people and uses bio to do so” were the only mechanism of x-risk to worry about. However, I think much of bio x-risk comes from human-misuse scenarios (i.e. providing uplift to humans wishing to create bioweapons).
hadad
Karma: 123
I would say that it’s more about time horizon. Just randomly trying things leads to basically just doing gradient descent until you hit a local optimum, while a more intentional approach looks worse at first but then you can eventually hit somewhere even better. So if your goal is “best photos by end of short photography class,” then just trying things out is the best approach. If your goal is “be an expert chess player” or “be an expert photographer” or anything like that, I think you need to be more intentional.
To use a bit of an analogy, evolution (basically massive, repeated trial and error) produced the cheetah, which can go at 65mph, and that’s genuinely impressive, but intentionally designing something to go quickly (e.g. an airplane) can go orders of magnitude faster.
Rumor has it GPT 5.6 Sol will be on Cerebras soon too. That will be a true game-changer (assuming it’s semi-reasonably priced).
I think it’s more about differences in models over time. With GPT 4o, etc., I think the key mode of failure looked more like people developing these weird cult-like quasi-mystical relationships with the AI, anthropomorphizing it, etc. This looked a lot like “psychosis.” Now, I think the mode of failure looks more like “see Claude Code is really cool, that means I can do anything!” This looks more like “mania.” So I think both are true, just of different eras.
To me at least, it doesn’t seem like it would be as simple as “just find-and-replace Claude with Kimi.” To some extent, I think that could create weird out-of-distribution issues. And also you’d need to do more than just replace Claude with Kimi, since Kimi has a different associated company, different model versions, and tons of other different characteristics too, and these things can be included (or even implied) in the training text in countless different ways, not just literal mentions of “Claude” or “Kimi.”
I agree, this is a real potential mode of failure. I do think there are ways to patch this loophole though. For example clones of a given entity could be considered part of the same entity rather than distinct entities. Attempts to forcibly align non-clone progeny with oneself would raise significant human rights concerns, and could reasonably considered to be a violation of the Universal Rights discussed in the AI 2040 Epilogue (that idea of Universal Rights is a part of the Epilogue that I do strongly agree with, even though I do disagree with other parts, as discussed in my post).
P.S. Quick more general note: the purpose of this post isn’t as much to defend those specific two proposals (which, as discussed earlier, are just a couple ideas I thought of off the top of my head). It’s more about arguing “no, a permanent underclass is not okay, and it is a solvable problem.”
I agree; it’s absolutely very possible for the permanent underclass scenario described in this post to devolve into something even worse. This post is more arguing “even if somehow there wasn’t escalating inequality, and things like property rights did remain, even for the poor (i.e. the scenario as described in AI 2040: Plan A), a permanent underclass would still be really bad even then.”
I really liked it; I think it was very well-written. I have a quick response to the epilogue: https://www.lesswrong.com/posts/EANs7YerYXmaXF9FE/don-t-normalize-a-permanent-underclass-even-a-rich-one-1. But, overall, thank you so much to the AI Futures Project for all of the hard work that went into this!
Don’t normalize a permanent underclass (even a rich one)
Opus 4.8 is showing regressions on some benchmarks too (e.g. VendingBench 2) relative to 4.7. So I would argue the stylometric identification failure is mainly symptomatic of a more general capabilities regression in Opus 4.8, not anything specific.
I would strongly disagree that this is a good thing. Most things should not be handled in the way ASI should be. With the large majority of things in the world, laws should be set up such that you should be able to “just do things” and avoid excessive safetyism I think. ASI just happens to be one of those places where a more cautious approach is in fact warranted; that result doesn’t generalize to everything else.