(Based on your tweets I’m guessing you think that curtailing independent research wouldn’t be necessary and aren’t considering it here; but this may be a background disagreement with people saying invasive restrictions would be needed.)
Yeah, my current guess (with like 85% confidence?) is that if you curtail large industrial-scale investment you would get there. I don’t think independent research would get there any time soon. You would need some kind of industrial level of investment at some point in the coming decades.
I agree there is more motivation to build more efficient AI algorithms than for people to build more effective nuclear reactor or nuclear weapon designs, though I think a lot of that is downstream of a societal stigma for nuclear weapons, on which I think there is a lot of uncertainty on how it will develop in the case of AI.
A proper treatment of this topic would start from the top about the role of social stigma and how this will affect talent allocation and resource investment, and how it interfaces with regulations. I think the default outcome here is that there won’t be much of a social stigma around AI development, this will consistently keep investment into developing more competent AI systems high, and also be itself the force that prevents regulation from interfering with that investment.
In the worlds where you actually have widespread buy-in to do something IAEA-like, you probably have quite a bit of stigma, and this will be largely responsible for keeping the level of investment low (though the regulations itself will also help). And if you manage to keep it that way, then I think you are probably fine for a very long time. Of course the stigma might break at any point, similar to how nuclear agreements might break at any point, and then you are back on a clock (and the clock will of course be shorter because you will have made at least some relevant progress in other scientific fields). All my statements here are about how if you keep the stigma and regulations up, you will be fine. In most worlds the stigma and regulations will break at some point, and you will have much less time, but this isn’t because the alternative would be some kind of Orwellian or invasive regime.
Controlling widely-used hardware and only allowing approved code to run on it does seem drastically more invasive, sufficiently obviously so that I have no idea where you’re coming from here. If this only applied to the largest supercomputers I might not call it more invasive, but the whole premise of this thread is not-that.
I don’t super understand why “AI chips that cost $1k+ can only run signed code” would be invasive in any meaningful way. I don’t really think it would change anyone’s life in any particularly meaningful way. Both Android phones and iPhones can only run signed code, and the vast majority of gaming happens on game consoles that can only run signed code.
but the whole premise of this thread is not-that.
What do you mean by “the whole premise of this thread?”. I am arguing against the claim (paraphrased) “it is inevitable that you can train superintelligence on a consumer laptop eventually unless we take some kind of drastic and invasive measures”.
Of course by the time you actually have made it so that you can train AI on a random laptop (from 2026?) then you do indeed have no choice but to take some kind of drastic and invasive measures, but my whole point is that you can just avoid getting there. I don’t think the OP is saying “conditional on superinteligence being trainable on a 2026 laptop, you have to curtail social freedoms” the OP is saying “you will be able to train superintelligence on a laptop within a few decades unless you drastically curtail social freedom”, and that is the statement I am objecting to.
It sounds like I have more expectation of a much more efficient paradigm (a la e.g. Steven Byrnes) being feasibly discovered through purely theoretical work (though not necessarily single-2026-laptop efficient, or discovered on any particular schedule), which is coloring my takes here.
I agree that stigma is important and would reduce the level of intervention needed to shut down independent research. It’s only very recently that I’ve seen any discussion of stigma as load-bearing in pause scenarios, so I wasn’t thinking of it.
I don’t super understand why “AI chips that cost $1k+ can only run signed code” would be invasive in any meaningful way. I don’t really think it would change anyone’s life in any particularly meaningful way.
I was thinking of it as more invasive in affecting (by limiting what code they can run) far more actors (as opposed to, what, reactor operators and uranium handlers?, in the nuclear case). If unrestricted general-purpose CPUs are still readily available, it does seem like nothing much would change in practice & the important freedoms would be preserved; combined with only a few chipmakers actually being liable for compliance, I can see calling this not more invasive.
Both Android phones and iPhones can only run signed code, and the vast majority of gaming happens on game consoles that can only run signed code.
(I do think it’s probably meaningful that these aren’t legal mandates, and more meaningful that unrestricted platforms are also readily available.)
Yeah, my current guess (with like 85% confidence?) is that if you curtail large industrial-scale investment you would get there. I don’t think independent research would get there any time soon. You would need some kind of industrial level of investment at some point in the coming decades.
I agree there is more motivation to build more efficient AI algorithms than for people to build more effective nuclear reactor or nuclear weapon designs, though I think a lot of that is downstream of a societal stigma for nuclear weapons, on which I think there is a lot of uncertainty on how it will develop in the case of AI.
A proper treatment of this topic would start from the top about the role of social stigma and how this will affect talent allocation and resource investment, and how it interfaces with regulations. I think the default outcome here is that there won’t be much of a social stigma around AI development, this will consistently keep investment into developing more competent AI systems high, and also be itself the force that prevents regulation from interfering with that investment.
In the worlds where you actually have widespread buy-in to do something IAEA-like, you probably have quite a bit of stigma, and this will be largely responsible for keeping the level of investment low (though the regulations itself will also help). And if you manage to keep it that way, then I think you are probably fine for a very long time. Of course the stigma might break at any point, similar to how nuclear agreements might break at any point, and then you are back on a clock (and the clock will of course be shorter because you will have made at least some relevant progress in other scientific fields). All my statements here are about how if you keep the stigma and regulations up, you will be fine. In most worlds the stigma and regulations will break at some point, and you will have much less time, but this isn’t because the alternative would be some kind of Orwellian or invasive regime.
I don’t super understand why “AI chips that cost $1k+ can only run signed code” would be invasive in any meaningful way. I don’t really think it would change anyone’s life in any particularly meaningful way. Both Android phones and iPhones can only run signed code, and the vast majority of gaming happens on game consoles that can only run signed code.
What do you mean by “the whole premise of this thread?”. I am arguing against the claim (paraphrased) “it is inevitable that you can train superintelligence on a consumer laptop eventually unless we take some kind of drastic and invasive measures”.
Of course by the time you actually have made it so that you can train AI on a random laptop (from 2026?) then you do indeed have no choice but to take some kind of drastic and invasive measures, but my whole point is that you can just avoid getting there. I don’t think the OP is saying “conditional on superinteligence being trainable on a 2026 laptop, you have to curtail social freedoms” the OP is saying “you will be able to train superintelligence on a laptop within a few decades unless you drastically curtail social freedom”, and that is the statement I am objecting to.
It sounds like I have more expectation of a much more efficient paradigm (a la e.g. Steven Byrnes) being feasibly discovered through purely theoretical work (though not necessarily single-2026-laptop efficient, or discovered on any particular schedule), which is coloring my takes here.
I agree that stigma is important and would reduce the level of intervention needed to shut down independent research. It’s only very recently that I’ve seen any discussion of stigma as load-bearing in pause scenarios, so I wasn’t thinking of it.
I was thinking of it as more invasive in affecting (by limiting what code they can run) far more actors (as opposed to, what, reactor operators and uranium handlers?, in the nuclear case). If unrestricted general-purpose CPUs are still readily available, it does seem like nothing much would change in practice & the important freedoms would be preserved; combined with only a few chipmakers actually being liable for compliance, I can see calling this not more invasive.
(I do think it’s probably meaningful that these aren’t legal mandates, and more meaningful that unrestricted platforms are also readily available.)