However, I feel like arguing that AI safety could be an incredibly hard problem is a way less extreme position than the one Eliezer seem to actually hold.
I mean, my take on this is that around two decades ago Eliezer thought AI safety could be an incredibly hard problem, and then spent a lot of time checking, and now has lots of reasons to believe that it is an incredibly hard problem, and those reasons are spelled out elsewhere, with this post just trying to point at the problem of irretrievability.
Sure, but then an attempt to summarise Eliezer’s position, which attributes a much stronger position than is in this post, isn’t necessarily a straw man that doesn’t understand the point of irretrievability, but can merely be responding to all his points on top of irretrievability, or saying that they don’t consider him to be making sufficient arguments beyond the potential for irretrievability
Agree, though in that case I think it would be good form to say “Eliezer is right about this being a “first critical try” sort of problem and that being important, but I disagree with him on other reasons for why he thinks the problem will be hard and they leave me substantially more optimistic”. The quotes I selected above do not do that.
I think Paul is saying “Eliezer is using an equivocation between a correct point and a false point for rhetorical effect”. I don’t think that is doing the same (I think it’s failing to give credit for the correct point). I do agree it’s doing some of what I was trying to point to here, but not following good form in the way I was trying to describe.
I think Joe is on a vibes-based level doing also a more direct equivocation, I think, but again, it’s been a while since I read it and I am not that confident.
I’m not saying that those people believe it is a critical first try problem. I expect they agree that it could be a critical first try problem, but that they predict it probably isn’t for variety of reasons and they view Eliezer as claiming that it is rather than just that it is possible it’s a critical first try problem
Another underlying disagreement could be about the general factor of: what is this function, approximately:
[the similarity of all previous tries put together, to try X]
--> [how much should we expect to fail on try X on the grounds that it’s first-try-ish, i.e. how much is it Murphy’s Cursed]
I would imagine that this isn’t the main source of disagreement, but I do find it hard to see how [create an alien mind that’s smarter than humanity] ends up less firsty than [make a Mars rover] etc., so I’m wondering if that’s not the claim.
Wait, that doesn’t make any sense. I am confident all these people totally think that aligning Superintelligence is a critical first try problem the way Eliezer is talking about here.
I mean, my take on this is that around two decades ago Eliezer thought AI safety could be an incredibly hard problem, and then spent a lot of time checking, and now has lots of reasons to believe that it is an incredibly hard problem, and those reasons are spelled out elsewhere, with this post just trying to point at the problem of irretrievability.
Sure, but then an attempt to summarise Eliezer’s position, which attributes a much stronger position than is in this post, isn’t necessarily a straw man that doesn’t understand the point of irretrievability, but can merely be responding to all his points on top of irretrievability, or saying that they don’t consider him to be making sufficient arguments beyond the potential for irretrievability
Agree, though in that case I think it would be good form to say “Eliezer is right about this being a “first critical try” sort of problem and that being important, but I disagree with him on other reasons for why he thinks the problem will be hard and they leave me substantially more optimistic”. The quotes I selected above do not do that.
FWIW, I interpret all of the things you linked (except the comment by Sam Marks) as pretty clearly saying this?
I think Paul is saying “Eliezer is using an equivocation between a correct point and a false point for rhetorical effect”. I don’t think that is doing the same (I think it’s failing to give credit for the correct point). I do agree it’s doing some of what I was trying to point to here, but not following good form in the way I was trying to describe.
I think Joe is on a vibes-based level doing also a more direct equivocation, I think, but again, it’s been a while since I read it and I am not that confident.
I’m not saying that those people believe it is a critical first try problem. I expect they agree that it could be a critical first try problem, but that they predict it probably isn’t for variety of reasons and they view Eliezer as claiming that it is rather than just that it is possible it’s a critical first try problem
Another underlying disagreement could be about the general factor of: what is this function, approximately:
I would imagine that this isn’t the main source of disagreement, but I do find it hard to see how [create an alien mind that’s smarter than humanity] ends up less firsty than [make a Mars rover] etc., so I’m wondering if that’s not the claim.
Wait, that doesn’t make any sense. I am confident all these people totally think that aligning Superintelligence is a critical first try problem the way Eliezer is talking about here.
They may think it’s critical but not very firsty, i.e. sufficiently similar to / comparable to / generalizable from previous tries.