I feel similarly, and what confuses me is that I had a positive view of AI safety back when it was about being pro-safety, pro-alignment, pro-interpretability, etc. These are good things that were neglected, and it felt good that there were people pushing for them.
But at some point it changed, becoming more about fear and opposition to progress. Anti-open source (most obviously with OpenAI, but even OpenRAIL isn’t OSI), anti-competition (via regulatory capture), anti-progress (via as-yet-unspecified means). I hadn’t appreciated the sheer darkness of the worldview.
And now, with the mindshare the movement has gained among the influential, I wonder what if it succeeds. What if open source AI models are banned, competitors to OpenAI are banned, and OpenAI decides to stop with GPT-4? It’s a little hard to imagine all that, but nuclear power was killed off in a vaguely analogous way.
Pondering the ensuing scenarios isn’t too pleasant. Does AGI get developed anyway, perhaps by China or by some military project during WW3? (I’d rather not either, please.) Or does humanity fully cooperate to put itself in a sort of technological stasis, with indefinite end?
it cannot occur as long as GPUs exist. we’re getting hard foom within 8 years, rain or shine, as far as I can tell; most likely within 4, if we do nothing to stabilize then it’s within 2, if we keep pushing hard then it’ll be this year.
let’s not do it this year. the people most able to push towards hard foom won’t need to rush as much if folks slow down. We’ve got some cool payouts from ai, let’s chill out slightly for a bit. just a little—there’s lots of fun capabilities stuff to do that doesn’t push towards explosive criticality, and most of the coolest capability stuff (in my view) makes things easier, not harder, to check for correctness.
The alignment, safety and interpretability is continuing at full speed, but if all the efforts of the alignment community are sufficient to get enough of this to avoid the destruction of the world in 2042, and AGI is created in 2037, then at the end you get a destroyed world.
It might not be possible in real life (List of Lethalities: “we can’t just decide not to build AGI”), and even if possible it might not be tractable enough to be worth focusing any attention on, but it would be nice if there was some way to make sure that AGI happens after alignment is sufficient at full speed (EDIT: or, failing that, to happen later, so if alignment goes quickly that takes the world from bad outcomes to good outcomes, instead of bad outcomes to bad outcomes).
The good news is that I expect AI development to be de facto open if not de jure openfor the following reason:
AI labs still need to publish enough at at least a high-level summary or abstraction level to succeed in the marketplace and politically.
OpenAI (et al.) could try and force as much about the actual functioning, work-performing details of the engineering design of their models into low-level implementation details that remain closed-source, with the intent to base their design on principles that make such a strategy more feasible. But I believe that this will not work.
This has to do with more fundamental reasons on how successful AI models have to actually be structured, such that even their high-level, abstract summaries of how they work must reliably map to the reasons that the model performs well (this is akin to the Natural Abstraction hypothesis).
Therefore, advanced AI models could in principle be feasibly reverse-engineered or re-developed simply from the implementation details that are published.
I feel similarly, and what confuses me is that I had a positive view of AI safety back when it was about being pro-safety, pro-alignment, pro-interpretability, etc. These are good things that were neglected, and it felt good that there were people pushing for them.
But at some point it changed, becoming more about fear and opposition to progress. Anti-open source (most obviously with OpenAI, but even OpenRAIL isn’t OSI), anti-competition (via regulatory capture), anti-progress (via as-yet-unspecified means). I hadn’t appreciated the sheer darkness of the worldview.
And now, with the mindshare the movement has gained among the influential, I wonder what if it succeeds. What if open source AI models are banned, competitors to OpenAI are banned, and OpenAI decides to stop with GPT-4? It’s a little hard to imagine all that, but nuclear power was killed off in a vaguely analogous way.
Pondering the ensuing scenarios isn’t too pleasant. Does AGI get developed anyway, perhaps by China or by some military project during WW3? (I’d rather not either, please.) Or does humanity fully cooperate to put itself in a sort of technological stasis, with indefinite end?
it cannot occur as long as GPUs exist. we’re getting hard foom within 8 years, rain or shine, as far as I can tell; most likely within 4, if we do nothing to stabilize then it’s within 2, if we keep pushing hard then it’ll be this year.
let’s not do it this year. the people most able to push towards hard foom won’t need to rush as much if folks slow down. We’ve got some cool payouts from ai, let’s chill out slightly for a bit. just a little—there’s lots of fun capabilities stuff to do that doesn’t push towards explosive criticality, and most of the coolest capability stuff (in my view) makes things easier, not harder, to check for correctness.
The alignment, safety and interpretability is continuing at full speed, but if all the efforts of the alignment community are sufficient to get enough of this to avoid the destruction of the world in 2042, and AGI is created in 2037, then at the end you get a destroyed world.
It might not be possible in real life (List of Lethalities: “we can’t just decide not to build AGI”), and even if possible it might not be tractable enough to be worth focusing any attention on, but it would be nice if there was some way to make sure that AGI happens after alignment is sufficient at full speed (EDIT: or, failing that, to happen later, so if alignment goes quickly that takes the world from bad outcomes to good outcomes, instead of bad outcomes to bad outcomes).
The good news is that I expect AI development to be de facto open if not de jure open for the following reason:
AI labs still need to publish enough at at least a high-level summary or abstraction level to succeed in the marketplace and politically.
OpenAI (et al.) could try and force as much about the actual functioning, work-performing details of the engineering design of their models into low-level implementation details that remain closed-source, with the intent to base their design on principles that make such a strategy more feasible. But I believe that this will not work.
This has to do with more fundamental reasons on how successful AI models have to actually be structured, such that even their high-level, abstract summaries of how they work must reliably map to the reasons that the model performs well (this is akin to the Natural Abstraction hypothesis).
Therefore, advanced AI models could in principle be feasibly reverse-engineered or re-developed simply from the implementation details that are published.