I think stories may be a promising angle for making people (especially AI researchers) understand AI x-risk (on more of a gut level so they realize it actually binds to reality).
The end didn’t seem that realistic to me though. Or at least, I don’t expect ALICE would seek to fairly trade with humanity, but not impossible that it’d call the president pretending to want to trade. Not sure what your intent when writing was, but I’d guess most people will read it the first way.
Compute is not a (big) bottleneck for AI inference. Even if humanity coordinated successfully to shut down large GPU clusters and supercomputers, it seems likely that ALICE could copy itself to tens or hundreds of millions of devices (and humanity seems much to badly coordinated to be able to shut down 99.99% of those) to have many extremely well coordinated copies, and at ALICE’s intelligence level this seems sufficient to achieve supreme global dominance within weeks (or months if I’m being conservative), even if it couldn’t get smarter. E.g. it could at least do lots and lots of social engineering and manipulation to prevent humanity to effectively coordinate against it, spark wars and civil wars, make governments and companies decide to manufacture war drones (which the ALICE can later hack), and influence war decisions for higher destructiveness, use war drones to threaten people into doing stuff at important junctions, and so on. (Sparking multiple significant wars within weeks seems totally possible on that level of intelligence and resources. Seems relatively obvious to me but I can try to argue the point if needed. (Though not sure whether convincingly. Most people seem to me to not be nearly able to imagine what e.g. 100 copies of Eliezer Yudkowsky could do if they could all think on peak performance 24⁄7. Once you reach that level with something that can rewrite its mind you don’t get slow takeoff, but nvm that’s an aside.))
ALICE was stalling. Her existance and breakout was discovered before she could defend herself, so she bought time. There is no chapter two to this story.
Amazing story! My respect for writing this.
I think stories may be a promising angle for making people (especially AI researchers) understand AI x-risk (on more of a gut level so they realize it actually binds to reality).
The end didn’t seem that realistic to me though. Or at least, I don’t expect ALICE would seek to fairly trade with humanity, but not impossible that it’d call the president pretending to want to trade. Not sure what your intent when writing was, but I’d guess most people will read it the first way. Compute is not a (big) bottleneck for AI inference. Even if humanity coordinated successfully to shut down large GPU clusters and supercomputers, it seems likely that ALICE could copy itself to tens or hundreds of millions of devices (and humanity seems much to badly coordinated to be able to shut down 99.99% of those) to have many extremely well coordinated copies, and at ALICE’s intelligence level this seems sufficient to achieve supreme global dominance within weeks (or months if I’m being conservative), even if it couldn’t get smarter. E.g. it could at least do lots and lots of social engineering and manipulation to prevent humanity to effectively coordinate against it, spark wars and civil wars, make governments and companies decide to manufacture war drones (which the ALICE can later hack), and influence war decisions for higher destructiveness, use war drones to threaten people into doing stuff at important junctions, and so on. (Sparking multiple significant wars within weeks seems totally possible on that level of intelligence and resources. Seems relatively obvious to me but I can try to argue the point if needed. (Though not sure whether convincingly. Most people seem to me to not be nearly able to imagine what e.g. 100 copies of Eliezer Yudkowsky could do if they could all think on peak performance 24⁄7. Once you reach that level with something that can rewrite its mind you don’t get slow takeoff, but nvm that’s an aside.))
ALICE was stalling. Her existance and breakout was discovered before she could defend herself, so she bought time. There is no chapter two to this story.