I’m curious: does this mean that you’re on board with the assumption in Ajeya’s report that 2020 algorithms and datasets + “business as usual” in algorithm and dataset design will scale up to strong AI, with compute being the bottleneck?
Yes, with the exception that I don’t know if compute will be the bottleneck (that is my best guess; I think Ajeya’s report makes a good case for it; but I could see it being other factors as well).
I think the case for is basically “we see a bunch of very predictable performance lines; seems like they’ll continue to go up”. But more importantly I don’t know of any compelling counterpoints; the usual argument seems to be “but we don’t see any causal reasoning / abstraction / <insert property here> yet”, which I think is perfectly compatible with the scaling hypothesis (see e.g. this comment).
A more concrete version of the “lobotomized alien” hypothetical might be something like this
I see, that makes sense, and I think it does make sense as an intuition pump for what the “ML paradigm” is trying to do (though as you sort of mentioned I don’t expect that we can just do the motivation / cognition decomposition).
“research acceleration” seems like a much narrower task with a relatively well-defined training set of papers, books, etc. than “AI agent that competently runs a company”, so might still come first on those grounds.
Definitely depends on how powerful you’re expecting the AI system to be. It seems like if you want to make the argument that AI will go well by default, you need the research accelerator to be quite powerful (or you have to combine with some argument like “AI alignment will be easy to solve”).
I don’t think papers, books, etc are a “relatively well-defined training set”. They’re a good source of knowledge, but if you imitate papers and books, you get a research accelerator that is limited by the capabilities of human scientists (well, actually much more limited, since it can’t run experiments). They might be a good source of pretraining data, but there would still be a lot of work to do to get a very powerful research accelerator.
I should have said that I don’t see a path for language models to get selection pressure in the direction of being catastrophically deceptive like in the old “AI getting out of the box” stories, so I think we agree.
Fwiw I’m not convinced that we avoid catastrophic deception either, but my thoughts here are pretty nebulous and I think that “we don’t know of a path to catastrophic deception” is a defensible position.
Yes, with the exception that I don’t know if compute will be the bottleneck (that is my best guess; I think Ajeya’s report makes a good case for it; but I could see it being other factors as well).
I think the case for is basically “we see a bunch of very predictable performance lines; seems like they’ll continue to go up”. But more importantly I don’t know of any compelling counterpoints; the usual argument seems to be “but we don’t see any causal reasoning / abstraction / <insert property here> yet”, which I think is perfectly compatible with the scaling hypothesis (see e.g. this comment).
I see, that makes sense, and I think it does make sense as an intuition pump for what the “ML paradigm” is trying to do (though as you sort of mentioned I don’t expect that we can just do the motivation / cognition decomposition).
Definitely depends on how powerful you’re expecting the AI system to be. It seems like if you want to make the argument that AI will go well by default, you need the research accelerator to be quite powerful (or you have to combine with some argument like “AI alignment will be easy to solve”).
I don’t think papers, books, etc are a “relatively well-defined training set”. They’re a good source of knowledge, but if you imitate papers and books, you get a research accelerator that is limited by the capabilities of human scientists (well, actually much more limited, since it can’t run experiments). They might be a good source of pretraining data, but there would still be a lot of work to do to get a very powerful research accelerator.
Fwiw I’m not convinced that we avoid catastrophic deception either, but my thoughts here are pretty nebulous and I think that “we don’t know of a path to catastrophic deception” is a defensible position.