A naive story for how humanity goes extinct from AI: Alpha Inc. spends a trillion dollars to create Alice the AGI. Alice escapes from whatever oversight mechanisms were employed to ensure alignment by uploading a copy of itself onto the internet. Alice does not have to pay an alignment tax, and so outcompetes Alpha and takes over the world.
On its face, this story contains some shaky arguments. In particular, Alpha is initially going to have 100x-1,000,000x more resources than Alice. Even if Alice grows its resources faster, the alignment tax would have to be very large for Alice to end up with control of a substantial fraction of the world’s resources.
Escapes is vague. Alice might escape with capital (Alice) and other capital, like $. And what if ‘the original’ is deleted?
More:
‘Outcompetes’ is vague. Let’s say Alpha is a known entity and Alice deploys attacks—digital, legal, nuclear, whatever. Alpha may be unable to effectively strike back against a rogue with an unknown location—and perhaps multiple locations—if it’s digital it can be copied.
Suppose that Alpha currently has a monopoly on the Alice-powered models, but Beta Inc. is looking to enter the market.
It’s not one market. If Alice can do X and Y and Z, then it is at least the X market the Y market and the Z market.
In this view, the primary value the employee has is their former employer’s trading high-performing strategies; knowledge they can potentially sell to other hedge funds.
They could also start their own.
Brand loyalty/customer inertia, legal enforcement against pirated IP, and distrust of rogue AGI could all disadvantage Beta in the share of the market it captures.
This assumes it’s a legal market. Instead Alice could...breach systems and upload viruses that encrypt your data, put it on the internet, delete it*, and then serve as part of a botnet. Alice then:
has your data
can sell it back to you (or not)
*This might make things more detectable, so usefulness is based on the amount of time involved.
In these worlds, relevant actors see AGI coming, correctly predict its economic value, and start investing accordingly. This rough efficiency claim implies AI researchers and hardware are priced such that one can potentially get 3x returns on investment (ROI) from training a powerful model, but not 30x.[1] Since most economic activity will rapidly involve the production and use of AGI, early-AGI will attract huge investments, implying the Alice-powered model market will be a moderate fraction of the world’s wealth. The value of Alice’s embodied IP, being tied to the value of that market, will thus be similarly massive.
This assumes there’s a FOOM, or
Rogue [artificial general super-intelligence] has access to its embodied IP.
TL:DR;
Escapes is vague. Alice might escape with capital (Alice) and other capital, like $. And what if ‘the original’ is deleted?
More:
‘Outcompetes’ is vague. Let’s say Alpha is a known entity and Alice deploys attacks—digital, legal, nuclear, whatever. Alpha may be unable to effectively strike back against a rogue with an unknown location—and perhaps multiple locations—if it’s digital it can be copied.
It’s not one market. If Alice can do X and Y and Z, then it is at least the X market the Y market and the Z market.
They could also start their own.
This assumes it’s a legal market. Instead Alice could...breach systems and upload viruses that encrypt your data, put it on the internet, delete it*, and then serve as part of a botnet. Alice then:
has your data
can sell it back to you (or not)
*This might make things more detectable, so usefulness is based on the amount of time involved.
This assumes there’s a FOOM, or