The technical intuitions we gained from this process, is the real reason for our particularly strong confidence in this problem being hard.
I don’t understand why anyone would expect such reason to be persuasive to other people. Like, to rely on illegible intuitions in the matters of human extinction just feels crazy. Yes, certainty doesn’t matter, we need to stop either way. But still—is it even rational to be so confident when you rely on illegible intuitions? Why don’t check yourself with something more robust, like actually writing your hypotheses, reasoning, and counting evidence? Sure there is something better than saying “I base my extreme confidence on intuitions”.
And it’s not only about corrigibility—“you don’t get what you train for” being universal law of intelligence in the real world, or utility maximization, especially in the limit, being good model of real things, or pivotal real world science being definitely so hard you can’t possibly be distracted even once and still figure it out—everything is insufficiently justified.
I didn’t read it as trying to be persuasive, just explaining their perspective.
(Note, also, like, they did cut this line from the resources, it’s not even a thing currently stated in the document that I know of. This is me claiming (kinda critically) this would have been a good sentence to say, in particular in combo with the rest of the paragraph I suggested following it up with.)
Experts have illegible intuitions all the time, and the thing to do is say “Hey, I’ve got some intuitions here, which is why I am confident. It makes sense that you do not have those intuitions or particularly trust them. But, fwiw it’s part of the story for why I’m personally confident. Meanwhile, here’s my best attempt to legibilize them” (which, these essays seem like a reasonable stab at. You can, of course, disagree that the legibilization makes sense to you)
I don’t understand why anyone would expect such reason to be persuasive to other people. Like, to rely on illegible intuitions in the matters of human extinction just feels crazy. Yes, certainty doesn’t matter, we need to stop either way. But still—is it even rational to be so confident when you rely on illegible intuitions? Why don’t check yourself with something more robust, like actually writing your hypotheses, reasoning, and counting evidence? Sure there is something better than saying “I base my extreme confidence on intuitions”.
And it’s not only about corrigibility—“you don’t get what you train for” being universal law of intelligence in the real world, or utility maximization, especially in the limit, being good model of real things, or pivotal real world science being definitely so hard you can’t possibly be distracted even once and still figure it out—everything is insufficiently justified.
I didn’t read it as trying to be persuasive, just explaining their perspective.
(Note, also, like, they did cut this line from the resources, it’s not even a thing currently stated in the document that I know of. This is me claiming (kinda critically) this would have been a good sentence to say, in particular in combo with the rest of the paragraph I suggested following it up with.)
Experts have illegible intuitions all the time, and the thing to do is say “Hey, I’ve got some intuitions here, which is why I am confident. It makes sense that you do not have those intuitions or particularly trust them. But, fwiw it’s part of the story for why I’m personally confident. Meanwhile, here’s my best attempt to legibilize them” (which, these essays seem like a reasonable stab at. You can, of course, disagree that the legibilization makes sense to you)