I struggle a bit to remember what ASI is but I’m gonna assume it’s Artificial Super Intelligence.
Let’s say that that’s markedly cleverer than 1 person. So it’s capable of running very successful trading strategies or programming extremely well. It’s not clear to me that such a being:
Has been driven towards being agentic, when its creators will prefer something more docile
Can cooperate well enough with itself to manage some massive secret takeover
Is competent enough to recursively self improve (and solve the alignment problems that creates)
Can beat everyone else combined
Feels like what such a being/system might do is just run some terrifically successful trading strategies and gather a lot of resources while frantically avoiding notice/trying to claim it won’t take over anything else. Huge public outcry, continuing regulation but maybe after a year it settles to some kind of equilibrium.
Chance of increasing capabilities and then some later jump, but seems plausible to me that that wouldn’t happen in one go.
I struggle a bit to remember what ASI is but I’m gonna assume it’s Artificial Super Intelligence.
Let’s say that that’s markedly cleverer than 1 person. So it’s capable of running very successful trading strategies or programming extremely well. It’s not clear to me that such a being:
Has been driven towards being agentic, when its creators will prefer something more docile
Can cooperate well enough with itself to manage some massive secret takeover
Is competent enough to recursively self improve (and solve the alignment problems that creates)
Can beat everyone else combined
Feels like what such a being/system might do is just run some terrifically successful trading strategies and gather a lot of resources while frantically avoiding notice/trying to claim it won’t take over anything else. Huge public outcry, continuing regulation but maybe after a year it settles to some kind of equilibrium.
Chance of increasing capabilities and then some later jump, but seems plausible to me that that wouldn’t happen in one go.