This might feel obvious, but I think it’s under-appreciated how much disagreement on AI progress just comes down to priors (in a pretty specific way) rather than object-level reasoning.
I was recently arguing the case for shorter timelines to a friend who leans longer. We kept disagreeing on a surprising number of object-level claims, which was weird because we usually agree more on the kinda stuff we were arguing about.
Then I basically realized what I think was going on: she had a pretty strong prior against what I was saying, and that prior is abstract enough that there’s no clear mechanism by which I can push against it. So whenever I made a good object-level case, she’d just take the other side — not necessarily because her reasons were better all else equal, but because the prior was doing the work underneath without either of us really knowing it.
There’s something clearly rational here that’s kinda unintuitive to get a grip on. If you have a strong prior, and someone makes a persuasive argument against it, but you can’t identify the specific mechanism by which their argument defeats it, you should probably update that the arguments against their case are better than they appear, even if you can’t articulate them yet. From the outside, this totally just looks like motivated reasoning (and often is), but I think it can be pretty importantly different.
The reason this is so hard to disentangle is that (unless your belief web is extremely clear to you, which seems practically impossible) it’s just enormously complicated. Your prior on timelines isn’t an isolate thing — it’s load-bearing for a bunch of downstream beliefs all at once. So the resistance isn’t obviously irrational, it’s more like… the system protecting its own coherence.
I think this means that people should try their best to disentangle whether some object level argument they’re having comes from real object level beliefs or pretty abstract priors (in which case, it seems less worthwhile to press on them).
This’d be easier to engage with with real examples.
I think this is sort-of right, but I think “prior” is not a clear enough mechanistic description of what’s going on. Like this is a fine abstraction but it doesn’t give me many details to think through towards solutions.
Also, like, pretty much any human knowledge isn’t intrinsically “prior”, you believe things because of your life experience.
One example that you might classify as a prior but I think we can do better: some people have the intuitive sense that a significantly-more-intelligent person can stomp all over less intelligent people.
I think many of these people get the sense from having, say, played a bunch of competitive strategic games, and have a sense of how high skill-ceilings are.
Some people don’t have that experience, instead their reason for the prior is just “it does sure looks like humans stomp all over chimpanzees”, and generalize.
On the other side, some people have life experience where messy finnicky details got in the way of supposedly “intelligent” plans, and they have a stronger salient prior about intelligence being bogged now in finnicky details.
In all those cases I think you can do better than stop at “well their prior is different.”
Or, rather: being enable to do that is a skill issue.
This might feel obvious, but I think it’s under-appreciated how much disagreement on AI progress just comes down to priors (in a pretty specific way) rather than object-level reasoning.
I was recently arguing the case for shorter timelines to a friend who leans longer. We kept disagreeing on a surprising number of object-level claims, which was weird because we usually agree more on the kinda stuff we were arguing about.
Then I basically realized what I think was going on: she had a pretty strong prior against what I was saying, and that prior is abstract enough that there’s no clear mechanism by which I can push against it. So whenever I made a good object-level case, she’d just take the other side — not necessarily because her reasons were better all else equal, but because the prior was doing the work underneath without either of us really knowing it.
There’s something clearly rational here that’s kinda unintuitive to get a grip on. If you have a strong prior, and someone makes a persuasive argument against it, but you can’t identify the specific mechanism by which their argument defeats it, you should probably update that the arguments against their case are better than they appear, even if you can’t articulate them yet. From the outside, this totally just looks like motivated reasoning (and often is), but I think it can be pretty importantly different.
The reason this is so hard to disentangle is that (unless your belief web is extremely clear to you, which seems practically impossible) it’s just enormously complicated. Your prior on timelines isn’t an isolate thing — it’s load-bearing for a bunch of downstream beliefs all at once. So the resistance isn’t obviously irrational, it’s more like… the system protecting its own coherence.
I think this means that people should try their best to disentangle whether some object level argument they’re having comes from real object level beliefs or pretty abstract priors (in which case, it seems less worthwhile to press on them).
This’d be easier to engage with with real examples.
I think this is sort-of right, but I think “prior” is not a clear enough mechanistic description of what’s going on. Like this is a fine abstraction but it doesn’t give me many details to think through towards solutions.
Also, like, pretty much any human knowledge isn’t intrinsically “prior”, you believe things because of your life experience.
One example that you might classify as a prior but I think we can do better: some people have the intuitive sense that a significantly-more-intelligent person can stomp all over less intelligent people.
I think many of these people get the sense from having, say, played a bunch of competitive strategic games, and have a sense of how high skill-ceilings are.
Some people don’t have that experience, instead their reason for the prior is just “it does sure looks like humans stomp all over chimpanzees”, and generalize.
On the other side, some people have life experience where messy finnicky details got in the way of supposedly “intelligent” plans, and they have a stronger salient prior about intelligence being bogged now in finnicky details.
In all those cases I think you can do better than stop at “well their prior is different.”
Or, rather: being enable to do that is a skill issue.
(See also: Propagating Facts into Aesthetics, which goes into one particular mechanism in more detail)