If something seems deep, it touches on stuff that’s important and general, which we would expect to be important for alignment.
The specific scenario I talk about in the paragraph you’re responding too is one where everything except for the sense of deepness is the same for both ideas, such that someone who doesn’t have a sense of what ideas are deep or profound would find the ideas basically equivalent. In such a scenario my argument is that we should expect the deep idea to receive a more attention, despite their not existing legible or well grounded reasons for this. Some amount of preference for the deep idea might be justifiable on the grounds of trusting intuitive insight, but I don’t think the record of intuitive insight as to what ideas are good is actually very impressive—there are a huge amount of ideas that didn’t work out that sounded deep (see some philosophy, psychoanalysis, ect.) and very few that did work out[1].
try to recover from the sense of deepness some pointers at what seemed deep in the research projects
I think on the margin new theoretical alignment researchers should do less of this, as I think most deep sounding ideas just genuinely aren’t very productive to research and aren’t amenable to being proven to be unproductive to work on—often times the only evidence that a deep idea isn’t productive to work on is that nothing concrete has come of it yet.
I don’t have empirical analysis showing this—I would probably gesture to various prior alignment research projects to support this if I had to, though I worry that would devolve into arguing about what ‘success’ meant.
The specific scenario I talk about in the paragraph you’re responding too is one where everything except for the sense of deepness is the same for both ideas, such that someone who doesn’t have a sense of what ideas are deep or profound would find the ideas basically equivalent.
But if that’s not what the distribution looks like, but rather the distribution looks like a strong correlation, then it’s not a bias, it’s just following what the distribution says. Maybe to shore up / expand on your argument, you’re talking about the optimizer’s curse: https://www.lesswrong.com/posts/5gQLrJr2yhPzMCcni/the-optimizer-s-curse-and-how-to-beat-it
So like, the most deep-seeming idea will tend to regress to the mean more than a random idea would regress. But this doesn’t argue to not pay attention to things that seem deep. (It argues for a portfolio approach, but there’s lots of arguments for a portfolio approach.)
Maybe another intuition you’re drawing on is information cascades. If there’s a lot of information cascades, then a lot of people are paying attention to a few very deep-seeming ideas. Which we can agree is dumb.
I think on the margin new theoretical alignment researchers should do less of this, as I think most deep sounding ideas just genuinely aren’t very productive to research and aren’t amenable to being proven to be unproductive to work on—often times the only evidence that a deep idea isn’t productive to work on is that nothing concrete has come of it yet.
I think this is pretty wrong, though it seems hard to resolve. I would guess that a lot of things that are later concretely productive started with someone hearing something that struck them as deep, and then chewing on it and transforming it.
The specific scenario I talk about in the paragraph you’re responding too is one where everything except for the sense of deepness is the same for both ideas, such that someone who doesn’t have a sense of what ideas are deep or profound would find the ideas basically equivalent. In such a scenario my argument is that we should expect the deep idea to receive a more attention, despite their not existing legible or well grounded reasons for this. Some amount of preference for the deep idea might be justifiable on the grounds of trusting intuitive insight, but I don’t think the record of intuitive insight as to what ideas are good is actually very impressive—there are a huge amount of ideas that didn’t work out that sounded deep (see some philosophy, psychoanalysis, ect.) and very few that did work out[1].
I think on the margin new theoretical alignment researchers should do less of this, as I think most deep sounding ideas just genuinely aren’t very productive to research and aren’t amenable to being proven to be unproductive to work on—often times the only evidence that a deep idea isn’t productive to work on is that nothing concrete has come of it yet.
I don’t have empirical analysis showing this—I would probably gesture to various prior alignment research projects to support this if I had to, though I worry that would devolve into arguing about what ‘success’ meant.
But if that’s not what the distribution looks like, but rather the distribution looks like a strong correlation, then it’s not a bias, it’s just following what the distribution says. Maybe to shore up / expand on your argument, you’re talking about the optimizer’s curse: https://www.lesswrong.com/posts/5gQLrJr2yhPzMCcni/the-optimizer-s-curse-and-how-to-beat-it So like, the most deep-seeming idea will tend to regress to the mean more than a random idea would regress. But this doesn’t argue to not pay attention to things that seem deep. (It argues for a portfolio approach, but there’s lots of arguments for a portfolio approach.)
Maybe another intuition you’re drawing on is information cascades. If there’s a lot of information cascades, then a lot of people are paying attention to a few very deep-seeming ideas. Which we can agree is dumb.
I think this is pretty wrong, though it seems hard to resolve. I would guess that a lot of things that are later concretely productive started with someone hearing something that struck them as deep, and then chewing on it and transforming it.