Imagine a search algorithm that finds local minima, similar to gradient descent, but has faster big-O performance than gradient descent. (For instance, an efficient roughly-n^2 matrix multiplication algorithm would likely yield such a thing, by making true Newton steps tractable on large systems—assuming it played well with sparsity.) That would be a general efficiency gain, and would likely stem from some sudden theoretical breakthrough (e.g. on fast matrix multiplication). And it is exactly the sort of thing which tends to come from a single person/team—the gradual theoretical progress we’ve seen on matrix multiplication is not the kind of breakthrough which makes the whole thing practical; people generally think we’re missing some key idea which will make the problem tractable.
some sudden theoretical breakthrough (e.g. on fast matrix multiplication)
These sorts of ideas seem possible, and I’m not willing to discard them as improbable just yet. I think a way to imagine my argument is that I’m saying, “Hold on, why are we assuming that this is the default scenario? I think we should be skeptical by default.” And so in general counterarguments of the form, “But it could be wrong because of this” aren’t great, because something being possible does not imply that it’s likely.
Imagine a search algorithm that finds local minima, similar to gradient descent, but has faster big-O performance than gradient descent. (For instance, an efficient roughly-n^2 matrix multiplication algorithm would likely yield such a thing, by making true Newton steps tractable on large systems—assuming it played well with sparsity.) That would be a general efficiency gain, and would likely stem from some sudden theoretical breakthrough (e.g. on fast matrix multiplication). And it is exactly the sort of thing which tends to come from a single person/team—the gradual theoretical progress we’ve seen on matrix multiplication is not the kind of breakthrough which makes the whole thing practical; people generally think we’re missing some key idea which will make the problem tractable.
These sorts of ideas seem possible, and I’m not willing to discard them as improbable just yet. I think a way to imagine my argument is that I’m saying, “Hold on, why are we assuming that this is the default scenario? I think we should be skeptical by default.” And so in general counterarguments of the form, “But it could be wrong because of this” aren’t great, because something being possible does not imply that it’s likely.