It does seem like, by definition, policy has to be built out of “legible” pieces.
(Well, the main alternative is “somehow, someone with great taste ends up in charge.” But, the people who don’t share that taste would be pretty sus. My story for “illegible taste decisionmaking” working out would involve people with very “scoped power”, who somehow are entrusted to make calls about what AI is safe to scale, that is structured such that nobody has to worry about them wielding that power to do other bad things).
But, assuming for now “we don’t get illegibly good decisionmakers with power over AI training”...
...one thing that comes immediately to mind is “I think almost nobody (modulo a few extropians) actually wants hard/fast takeoff.” People who don’t want AI regulation mostly aren’t expecting hard takeoff in the first place, and are assuming the natural low-regulated course of things would be a technology that’s manageable.
AI 2040 paints a story where takeoff happens over 10 years and it’s still pretty crazy feeling by any reasonable standard. Many people were alarmed that the “doomers” expected faster takeoff than the accelerationists even when dramatically slowing down.
This leaves the problem of disagreement over “do we need to stop research, because we’re rapidly approaching a critical threshold where you get godlike AI that we aren’t prepared for?”. But, if there ends up being any kind warning shot or clarity that, yeah actually godlike AI is right on the horizon, I kinda just expect a lot of agreement?
(I don’t actually remember: are you like still unsure if you expect godlike AI at all, or expecting it like multiple years after humanlevel conceptual research? I’m not actually sure which parts feel fake to you atm)
A lot of people I respect don’t believe in “godlike” AI at all, so I am confused there.
I also worry about the policy implementation of a “pause” being done in the wrong way with lots of side effects.
I agree that if we have the “slow” AI progress laid out in AI 2040 that’s actually way faster than late-20c/early-21c tech progress has been, then I don’t have anything to complain about from an economic growth point of view.
Frankly it’s weird to me that the labs chose to go to RSI as fast as possible in the first place; intuitively I would have thought the natural thing to do would be to take your time and get it right and get a much better understanding of how neural nets even work.
Frankly it’s weird to me that the labs chose to go to RSI as fast as possible in the first place; intuitively I would have thought the natural thing to do would be to take your time and get it right and get a much better understanding of how neural nets even work.
Really? What do you find so confusing about it?
It’s the kind of thing you do if you want to affect the world through power. It’s quite uncertain how far we are from understanding NNs well (anywhere from 1-10 years seems reasonable, even with our current AI helpers). And in the meantime someone else could invest in scaling NNs and get there, or you won’t be able to sustain enough expectation of future revenue for your investment multiples, or if you’re OpenAI then Anthropic will do it first and you lose power. (Though people are spooked enough by recent incidents that these two labs might start pacing the frontier now, so maybe your intuitions are more correct.)
Cool.
It does seem like, by definition, policy has to be built out of “legible” pieces.
(Well, the main alternative is “somehow, someone with great taste ends up in charge.” But, the people who don’t share that taste would be pretty sus. My story for “illegible taste decisionmaking” working out would involve people with very “scoped power”, who somehow are entrusted to make calls about what AI is safe to scale, that is structured such that nobody has to worry about them wielding that power to do other bad things).
But, assuming for now “we don’t get illegibly good decisionmakers with power over AI training”...
...one thing that comes immediately to mind is “I think almost nobody (modulo a few extropians) actually wants hard/fast takeoff.” People who don’t want AI regulation mostly aren’t expecting hard takeoff in the first place, and are assuming the natural low-regulated course of things would be a technology that’s manageable.
AI 2040 paints a story where takeoff happens over 10 years and it’s still pretty crazy feeling by any reasonable standard. Many people were alarmed that the “doomers” expected faster takeoff than the accelerationists even when dramatically slowing down.
This leaves the problem of disagreement over “do we need to stop research, because we’re rapidly approaching a critical threshold where you get godlike AI that we aren’t prepared for?”. But, if there ends up being any kind warning shot or clarity that, yeah actually godlike AI is right on the horizon, I kinda just expect a lot of agreement?
(I don’t actually remember: are you like still unsure if you expect godlike AI at all, or expecting it like multiple years after humanlevel conceptual research? I’m not actually sure which parts feel fake to you atm)
A lot of people I respect don’t believe in “godlike” AI at all, so I am confused there.
I also worry about the policy implementation of a “pause” being done in the wrong way with lots of side effects.
I agree that if we have the “slow” AI progress laid out in AI 2040 that’s actually way faster than late-20c/early-21c tech progress has been, then I don’t have anything to complain about from an economic growth point of view.
Frankly it’s weird to me that the labs chose to go to RSI as fast as possible in the first place; intuitively I would have thought the natural thing to do would be to take your time and get it right and get a much better understanding of how neural nets even work.
Really? What do you find so confusing about it?
It’s the kind of thing you do if you want to affect the world through power. It’s quite uncertain how far we are from understanding NNs well (anywhere from 1-10 years seems reasonable, even with our current AI helpers). And in the meantime someone else could invest in scaling NNs and get there, or you won’t be able to sustain enough expectation of future revenue for your investment multiples, or if you’re OpenAI then Anthropic will do it first and you lose power. (Though people are spooked enough by recent incidents that these two labs might start pacing the frontier now, so maybe your intuitions are more correct.)
Can you name 5? Who also have clearly thought about the topic in a serious way? (E.g. not Tyler Cowen)
Or is the issue with “godlike”, but they’d be fine with “wildly superhuman”.