I’m not saying that those people believe it is a critical first try problem. I expect they agree that it could be a critical first try problem, but that they predict it probably isn’t for variety of reasons and they view Eliezer as claiming that it is rather than just that it is possible it’s a critical first try problem
Another underlying disagreement could be about the general factor of: what is this function, approximately:
[the similarity of all previous tries put together, to try X]
--> [how much should we expect to fail on try X on the grounds that it’s first-try-ish, i.e. how much is it Murphy’s Cursed]
I would imagine that this isn’t the main source of disagreement, but I do find it hard to see how [create an alien mind that’s smarter than humanity] ends up less firsty than [make a Mars rover] etc., so I’m wondering if that’s not the claim.
Wait, that doesn’t make any sense. I am confident all these people totally think that aligning Superintelligence is a critical first try problem the way Eliezer is talking about here.
I’m not saying that those people believe it is a critical first try problem. I expect they agree that it could be a critical first try problem, but that they predict it probably isn’t for variety of reasons and they view Eliezer as claiming that it is rather than just that it is possible it’s a critical first try problem
Another underlying disagreement could be about the general factor of: what is this function, approximately:
I would imagine that this isn’t the main source of disagreement, but I do find it hard to see how [create an alien mind that’s smarter than humanity] ends up less firsty than [make a Mars rover] etc., so I’m wondering if that’s not the claim.
Wait, that doesn’t make any sense. I am confident all these people totally think that aligning Superintelligence is a critical first try problem the way Eliezer is talking about here.
They may think it’s critical but not very firsty, i.e. sufficiently similar to / comparable to / generalizable from previous tries.