People sometimes argue that AI can’t become very powerful because the limiting factor in human success is something else than intelligence — perhaps determination or courage.
We live in something shockingly close to a post-scarcity society, at least if you come from a wealthy country like Denmark or the US. There are some limiting factors to success—e.g. if you get run over by a bus. But by and large You Can Just Do Things.
But even if they’re right, the same processes that will increase the intelligence of future systems can also increase their determination and courage, and any other cognitive trait that matters.
Mathematically impossible. If X matters then so does -X, but any increase in X corresponds to a decrease in -X.
In particular the easiest way to become richer is to own something that very predictably gives profit, since some people want to store value, and that makes them want to buy things that are as robustly producing profit as possible, so they bid the price of those things all the way up to where the profit is infinitesimal relative to it. But courage interferes with running a predictably profitable venture (because sometimes you charge into a loss), so this leads to a selection for reductions in courage, rather than increases in courage.
EDIT 2: Did you mean that there are advantages to having both courage and caution, so you can’t have a machine that has maximal courage and maximal caution? That’s true, but you can probably still make pareto improvements over humans in terms of courage and caution.
Would changing “increase” to “optimize” fix your objection? Also, I don’t see how your first paragraph contradicts the first quoted sentence.
Mathematically impossible. If X matters then so does -X, but any increase in X corresponds to a decrease in -X.
I don’t know how the second sentence leads to the first. Why should a decrease in -X lead to less success? Moreover, claims of mathematical impossibility are often over-stated.
As for the paragraph after, it seems like it assumes current traits being on some sort of pareto frontier of economic-fitness. (And, perhaps, an assumption of adequate equilibria). But I don’t see why that’d be true. Like, I know of people who are more diligent than me, more intelligent, have lower discount rates etc. And they are indeed successful. EDIT: AFAICT, there’s a tonne of frictions and barriers, which weaken the force of the economic argument I think you’re making here.
We live in something shockingly close to a post-scarcity society, at least if you come from a wealthy country like Denmark or the US. There are some limiting factors to success—e.g. if you get run over by a bus. But by and large You Can Just Do Things.
Mathematically impossible. If X matters then so does -X, but any increase in X corresponds to a decrease in -X.
In particular the easiest way to become richer is to own something that very predictably gives profit, since some people want to store value, and that makes them want to buy things that are as robustly producing profit as possible, so they bid the price of those things all the way up to where the profit is infinitesimal relative to it. But courage interferes with running a predictably profitable venture (because sometimes you charge into a loss), so this leads to a selection for reductions in courage, rather than increases in courage.
EDIT 2: Did you mean that there are advantages to having both courage and caution, so you can’t have a machine that has maximal courage and maximal caution? That’s true, but you can probably still make pareto improvements over humans in terms of courage and caution.
Would changing “increase” to “optimize” fix your objection? Also, I don’t see how your first paragraph contradicts the first quoted sentence.
I don’t know how the second sentence leads to the first. Why should a decrease in -X lead to less success? Moreover, claims of mathematical impossibility are often over-stated.
As for the paragraph after, it seems like it assumes current traits being on some sort of pareto frontier of economic-fitness. (And, perhaps, an assumption of adequate equilibria). But I don’t see why that’d be true. Like, I know of people who are more diligent than me, more intelligent, have lower discount rates etc. And they are indeed successful. EDIT: AFAICT, there’s a tonne of frictions and barriers, which weaken the force of the economic argument I think you’re making here.