Okay, I guess this comes down to the interpretation of what “foom” means? I don’t think a world that looks like the current one can be taken over inconspicuously by AI in seconds, and not weeks either, and not even less than a year. If society has progressed to a point where we feel comfortable giving much more power to artificial agents, then that shortens the timeline.
The reason I think timelines are long is that I think it is inherently hard to do novel things, much harder than typically thought. I mean, what new things do you and I really do? Virtually nothing. What I tried to state in this essay is that knowledge is an inherent part of what we typically mean by intelligence, and for new tasks, new intelligence and knowledge is needed. The way this knowledge is gained is through cultural evolution; memes constitute knowledge and intelligence and these evolve similarly to genes; the vast majority of good genes you have are from your ancestors and most of your new genes or recombinations thereof are vastly likely to not improve your genetic makeup. It works the same way with memes; virtually everything you and I can do that we consider uniquely human are things we’ve copied from somewhere else, including “simple” things like counting or percentages. And, virtually none of the new things you and I do are improvements.
AI is not exempted from the process described above. Its intelligence is just as dependent on knowledge gained through trial and error and cultural evolution. This process is slow, and the faster and greater the effect to be achieved, the more knowledge and time is needed to actually do it in one shot.
Yeah, agreed with all mechanistic points, disagreed on timelines for a network of artificial agents to pull this off. I do think it initially looks like existing human culture subnetworks of malice being amplified by ai misuse, but we see that now. And that one group of beings who like being loud about capability increase seem increasingly like The Borg every time I encounter them on Twitter. I’ve inverted my downvote because this post plus these comments seems more reasonable, but I still think the claim in the title is very overconfident—evidence against foom-in-a-box is just an improvement to the map of how to foom.
evidence against foom-in-a-box is just an improvement to the map of how to foom.
Could you elaborate on this? I equate foom with the hard take-off scenario, for which I think I’ve stated why I think this is virtually impossible, in contrast to the slow take-off, which in spite of being slow is still very dangerous, as I described.
I think my view roughly aligns with those of Robin Hanson and Paul Christiano, but I think I’ve provided a more precise, gears-level description that has been lacking and why the onus is really on those who think the hard take-off is possible at all.
Okay, I guess this comes down to the interpretation of what “foom” means? I don’t think a world that looks like the current one can be taken over inconspicuously by AI in seconds, and not weeks either, and not even less than a year. If society has progressed to a point where we feel comfortable giving much more power to artificial agents, then that shortens the timeline.
The reason I think timelines are long is that I think it is inherently hard to do novel things, much harder than typically thought. I mean, what new things do you and I really do? Virtually nothing. What I tried to state in this essay is that knowledge is an inherent part of what we typically mean by intelligence, and for new tasks, new intelligence and knowledge is needed. The way this knowledge is gained is through cultural evolution; memes constitute knowledge and intelligence and these evolve similarly to genes; the vast majority of good genes you have are from your ancestors and most of your new genes or recombinations thereof are vastly likely to not improve your genetic makeup. It works the same way with memes; virtually everything you and I can do that we consider uniquely human are things we’ve copied from somewhere else, including “simple” things like counting or percentages. And, virtually none of the new things you and I do are improvements.
AI is not exempted from the process described above. Its intelligence is just as dependent on knowledge gained through trial and error and cultural evolution. This process is slow, and the faster and greater the effect to be achieved, the more knowledge and time is needed to actually do it in one shot.
Yeah, agreed with all mechanistic points, disagreed on timelines for a network of artificial agents to pull this off. I do think it initially looks like existing human culture subnetworks of malice being amplified by ai misuse, but we see that now. And that one group of beings who like being loud about capability increase seem increasingly like The Borg every time I encounter them on Twitter. I’ve inverted my downvote because this post plus these comments seems more reasonable, but I still think the claim in the title is very overconfident—evidence against foom-in-a-box is just an improvement to the map of how to foom.
Could you elaborate on this? I equate foom with the hard take-off scenario, for which I think I’ve stated why I think this is virtually impossible, in contrast to the slow take-off, which in spite of being slow is still very dangerous, as I described.
I think my view roughly aligns with those of Robin Hanson and Paul Christiano, but I think I’ve provided a more precise, gears-level description that has been lacking and why the onus is really on those who think the hard take-off is possible at all.