My biggest AI risk worries right now are more immediate than paperclip optimizers. They’re wealth optimizers, profit optimizers; probably extrapolations of current HFT systems. The goal of such a system isn’t even to make its owners happy — just to make them rich — and it certainly doesn’t care about anyone else. It may not even have beliefs about humans, just about flows of capital and information.
I contend that those that exist are already a problem.
Most likely because there have been some alarming failures of automated traders, such as the 2010 “Flash Crash” or the April Flash Crash caused by a Twitter hoax. From a layman’s perspective, it seems like all the regular problems of speculation with the added benefit of trades taking place faster than any human regulator could react. So far there hasn’t been any serious damage but it’s not clear to me whether that’s a point in the traders’ favors or just blind luck.
Of course, this isn’t a Friendliness issue so much as a competence one and I’m fairly sure there isn’t much of an existential risk involved in these programs undergoing an intelligence explosions. So it might not be what the other posters here were thinking of.
Speculators are good for a market—they smooth out price fluctuations and give fundamentals traders better prices. And when they screw up the effect is usually to give money to other people, as with the flash crash. So I don’t see the problem.
I contend that those that exist are already a problem.
How? Because they took some money off other speculators? Because some of them went bankrupt?
Most likely because there have been some alarming failures of automated traders, such as the 2010 “Flash Crash” or the April Flash Crash caused by a Twitter hoax. From a layman’s perspective, it seems like all the regular problems of speculation with the added benefit of trades taking place faster than any human regulator could react. So far there hasn’t been any serious damage but it’s not clear to me whether that’s a point in the traders’ favors or just blind luck.
Of course, this isn’t a Friendliness issue so much as a competence one and I’m fairly sure there isn’t much of an existential risk involved in these programs undergoing an intelligence explosions. So it might not be what the other posters here were thinking of.
Speculators are good for a market—they smooth out price fluctuations and give fundamentals traders better prices. And when they screw up the effect is usually to give money to other people, as with the flash crash. So I don’t see the problem.