Actually I think this is somewhat of a myth / greatly exaggerated. You can get good feedback about ideas regarding alignment. It just takes certain efforts that people don’t do. For example, “do maker-breaker to alignment proposals and proofs of impossibility of alignment; generalize each breaking”. That’s good feedback. Philosophical arguments are also good feedback. The problem is hard for other reasons (e.g. we just have huge conceptual gaps). As a comparison, it would be only accurate in a confusing / contorted way, to say that Euclid “lacked good feedback loops” about algebraic topology or whatever. Algebraic topology has good feedback loops, math has good feedback loops, etc.; Euclid just lacks a bunch of concepts needed to even start with algebraic topology.
(If by good you meant highly legible, very inexpensive, very fast, etc., then sure.)
I agree with that, and also suspect that some alignment researchers are sufficiently bad at controlling their own confirmation bias that they never get a good look at the conceptual gaps. To those researchers, the lack of good feedback loops is the one central fact about alignment research. I.e., I suspect that some extremely bright people are 100x or 1000x better at avoiding confirmation bias than other extremely bright people are.
I think I agree in outline, though I’m unsure about describing that as confirmation bias. “Doesn’t try to counterargue one’s own hopes” is one of the main things; I guess that’s a kind of confirmation bias? I guess I’d rather describe it as a big missing skill or practice, but maybe I’m being too restrictive with the word “bias”.
Actually I think this is somewhat of a myth / greatly exaggerated. You can get good feedback about ideas regarding alignment. It just takes certain efforts that people don’t do. For example, “do maker-breaker to alignment proposals and proofs of impossibility of alignment; generalize each breaking”. That’s good feedback. Philosophical arguments are also good feedback. The problem is hard for other reasons (e.g. we just have huge conceptual gaps). As a comparison, it would be only accurate in a confusing / contorted way, to say that Euclid “lacked good feedback loops” about algebraic topology or whatever. Algebraic topology has good feedback loops, math has good feedback loops, etc.; Euclid just lacks a bunch of concepts needed to even start with algebraic topology.
(If by good you meant highly legible, very inexpensive, very fast, etc., then sure.)
I agree with that, and also suspect that some alignment researchers are sufficiently bad at controlling their own confirmation bias that they never get a good look at the conceptual gaps. To those researchers, the lack of good feedback loops is the one central fact about alignment research. I.e., I suspect that some extremely bright people are 100x or 1000x better at avoiding confirmation bias than other extremely bright people are.
I think I agree in outline, though I’m unsure about describing that as confirmation bias. “Doesn’t try to counterargue one’s own hopes” is one of the main things; I guess that’s a kind of confirmation bias? I guess I’d rather describe it as a big missing skill or practice, but maybe I’m being too restrictive with the word “bias”.
I was uneasy about “confirmation bias” when I was writing my comment (and momentarily considered “motivated cognition”, which is not any better).