It’s not that hard to build an AI that kills everyone: you just need to solve [some problems] and combine the solutions. Considering how easy it is compared to what you thought, you should increase your P(doom) / shorten your timelines.
It’s not that hard to build an AI that saves everyone: you just need to solve [some problems] and combine the solutions. Considering how easy it is compared to what you thought, you should decrease your P(doom) / shorten your timelines.
They do a value-handshake and kill everyone together.
Any two AIs unaligned with humanity are very unlikely to be also aligned with each other, and would have no especial reason to coordinate with other unaligned AIs over humans (or uploads, or partially aligned neuromorphic AGIs, etc)
The vast majority of the utility you have to gain is from {getting a utopia rather than everyone-dying-forever}, rather than {making sure you get the right utopia}.
Whose utopia? For example: some people’s utopia may be one where they create immense quantity of descendants of various forms and allocate resources to them rather than existing humans. I also suspect that Hamas’s utopia is not mine. This idea that the distribution of future scenarios is bimodal around “we all die” or “perfect egalitarian utopia” is dangerously oversimplistic.
It’s not that hard to build an AI that saves everyone: you just need to solve [some problems] and combine the solutions. Considering how easy it is compared to what you thought, you should decrease your P(doom) / shorten your timelines.
I’m not sure what you’re saying here exactly. It seems to me like you’re pointing to a symmetric argument favoring low doom, but if someone had an idea for how to do AI alignment right, why wouldn’t they just talk about it? Doesn’t seem symmetrical to me.
Its easy to say something is “not that hard”, but ridiculous to claim that when the something is build an AI that takes over the world. The hard part is building something more intelligent/capable than humanity, not anything else conditioned on that first step.
I don’t see why this would be ridiculous. To me, e.g. “Superintelligence only requires [hacky change to current public SOTA] to achieve with expected 2025 hardware, and OpenAI may or may not have realised that already” seems like a perfectly coherent way the world could be, and is plenty of reason for anyone who suspects such a thing to keep their mouth shut about gears-level models of [] that might be relevant for judging how hard and mysterious the remaining obstacles to superintelligence actually are.
If it only requires a simple hack to existing public SOTA, many others will have already thought of said hack and you won’t have any additional edge. Taboo superintelligence and think through more specifically what is actually required to outcompete the rest of the world.
Progress in DL is completely smooth as it is driven mostly from hardware and enormous number of compute-dependent small innovations (yes transformers were a small innovation on top of contemporary alternatives such as memory networks, NTMs etc and quite predictable in advance ).
If it only requires a simple hack to existing public SOTA, many others will have already thought of said hack and you won’t have any additional edge.
I don’t recall assuming the edge to be unique? That seems like an unneeded condition for Tamsin’s argument, it’s enough to believe the field consensus isn’t completely efficient by default and all relevant actors are sure of all currently deducable edges at all times.
Progress in DL is completely smooth.
Right, if you think it’s completely smooth and thus basically not meaningfully influenced by the actions of individual researchers whatsoever, I see why you would not buy Tamsin’s argument here. But then the reason you don’t buy it would seem to me to be that you think meaningful new ideas in ML capability research basically don’t exist, not because you think there is some symmetric argument to Tamsin’s for people to stay quiet about new alignment research ideas.
If you think all AGIs will coordinate with each other, nobody needs an edge. If you think humans will build lots of AI systems, many technically unable to coordinate with each other (from mechanisms similar to firewalls/myopia/sparsity) then the world takeover requires an edge. An edge such that the (coalition of hostile AIs working together) wins the war against (humans plus their AIs).
This can get interesting if you think there might be diminishing returns in intelligence, which could mean that the (humans + their AI) faction might have a large advantage if the humans start with far more resources like they control now.
It’s not that hard to build an AI that saves everyone: you just need to solve [some problems] and combine the solutions. Considering how easy it is compared to what you thought, you should decrease your P(doom) / shorten your timelines.
Any two AIs unaligned with humanity are very unlikely to be also aligned with each other, and would have no especial reason to coordinate with other unaligned AIs over humans (or uploads, or partially aligned neuromorphic AGIs, etc)
Whose utopia? For example: some people’s utopia may be one where they create immense quantity of descendants of various forms and allocate resources to them rather than existing humans. I also suspect that Hamas’s utopia is not mine. This idea that the distribution of future scenarios is bimodal around “we all die” or “perfect egalitarian utopia” is dangerously oversimplistic.
I’m not sure what you’re saying here exactly. It seems to me like you’re pointing to a symmetric argument favoring low doom, but if someone had an idea for how to do AI alignment right, why wouldn’t they just talk about it? Doesn’t seem symmetrical to me.
Its easy to say something is “not that hard”, but ridiculous to claim that when the something is build an AI that takes over the world. The hard part is building something more intelligent/capable than humanity, not anything else conditioned on that first step.
I don’t see why this would be ridiculous. To me, e.g. “Superintelligence only requires [hacky change to current public SOTA] to achieve with expected 2025 hardware, and OpenAI may or may not have realised that already” seems like a perfectly coherent way the world could be, and is plenty of reason for anyone who suspects such a thing to keep their mouth shut about gears-level models of [] that might be relevant for judging how hard and mysterious the remaining obstacles to superintelligence actually are.
If it only requires a simple hack to existing public SOTA, many others will have already thought of said hack and you won’t have any additional edge. Taboo superintelligence and think through more specifically what is actually required to outcompete the rest of the world.
Progress in DL is completely smooth as it is driven mostly from hardware and enormous number of compute-dependent small innovations (yes transformers were a small innovation on top of contemporary alternatives such as memory networks, NTMs etc and quite predictable in advance ).
I don’t recall assuming the edge to be unique? That seems like an unneeded condition for Tamsin’s argument, it’s enough to believe the field consensus isn’t completely efficient by default and all relevant actors are sure of all currently deducable edges at all times.
Right, if you think it’s completely smooth and thus basically not meaningfully influenced by the actions of individual researchers whatsoever, I see why you would not buy Tamsin’s argument here. But then the reason you don’t buy it would seem to me to be that you think meaningful new ideas in ML capability research basically don’t exist, not because you think there is some symmetric argument to Tamsin’s for people to stay quiet about new alignment research ideas.
If you think all AGIs will coordinate with each other, nobody needs an edge. If you think humans will build lots of AI systems, many technically unable to coordinate with each other (from mechanisms similar to firewalls/myopia/sparsity) then the world takeover requires an edge. An edge such that the (coalition of hostile AIs working together) wins the war against (humans plus their AIs).
This can get interesting if you think there might be diminishing returns in intelligence, which could mean that the (humans + their AI) faction might have a large advantage if the humans start with far more resources like they control now.