ben is happy to support Paul’s agenda, because it agrees on most of the non-mainstream moving parts that also go into MIRI’s agenda, whereas orgs working on algorithmic bias, say, lack most of those.
I don’t actually know how many inferential steps Paul’s and agent foundations agendas agree on, whether it’s closer to 10% or to 50% or to 90% (and I would love to know more about this) but they do seem to me qualitatively different to things like algorithmic bias.
...we can’t really push Ben to explain why all the moving parts of his baseline are correct—that’s essentially the voice of Pat Modesto. He might legitimately be able to offer no better explanation than that it’s the combined model built through years of thinking, reading, trying and discussing.
I would change the wording of the second sentence to
He might legitimately not be able to fully communicate his models, because it’s built from years of thinking, reading, trying and discussing. Nonetheless, it’s valuable to probe it for its general structure, run consistency checks, and see if can make novel predictions, even if full communication is not reachable.
This seems similar to many experts in all fields (e.g. chess, economics, management). Regarding HPMOR, Eliezer wasn’t able to (or at least didn’t) fully communicate his models of how to write rationalist fiction, and Pat Modesto would certainly say “I can’t see how you get to be so confident in your models, therefore you’re not allowed to be so confident in your models”, but this isn’t a good enough reason for Eliezer not to believe the (subtle) evidence he has observed. And this is borne out by the fact he was able to predictably build something surprising and valuable.
Similarly in this position, it seems perfectly appropriate to ask me things like “What are some examples of the inferential steps you feel confident a research path must understand in order to be correct, and how do you get those?” and also to ask me to make predictions based on this, even if the answer to the first question doesn’t persuade you that I’m correct.
Mostly agreement, a few minor points:
I don’t actually know how many inferential steps Paul’s and agent foundations agendas agree on, whether it’s closer to 10% or to 50% or to 90% (and I would love to know more about this) but they do seem to me qualitatively different to things like algorithmic bias.
I would change the wording of the second sentence to
This seems similar to many experts in all fields (e.g. chess, economics, management). Regarding HPMOR, Eliezer wasn’t able to (or at least didn’t) fully communicate his models of how to write rationalist fiction, and Pat Modesto would certainly say “I can’t see how you get to be so confident in your models, therefore you’re not allowed to be so confident in your models”, but this isn’t a good enough reason for Eliezer not to believe the (subtle) evidence he has observed. And this is borne out by the fact he was able to predictably build something surprising and valuable.
Similarly in this position, it seems perfectly appropriate to ask me things like “What are some examples of the inferential steps you feel confident a research path must understand in order to be correct, and how do you get those?” and also to ask me to make predictions based on this, even if the answer to the first question doesn’t persuade you that I’m correct.