i.e. the reason you expect a superintelligence to be difficult to control, is not exactly the raw intelligence. It’s that (some people think) the way something succeeds at being truly superintelligent requires being relentlessly resourceful.
(If it wasn’t relentlessly resourceful, it maybe could one-shot a large-but-shallow set of problems in a way that wasn’t concerning. But, then, if it hit a snag, it would get stuck, and it would be less useful than something that didn’t get stuck when it hit snags)
I like this because
a) it highlights what the problem is, more clearly.
and b), it highlights “if you could build a very useful powerful tool that succeeds without being relentlessly resourceful, that’s maybe a useful avenue.”
For examples of relentlessly resourceful people, see:
Startup founders
Prolific Inventors
Elon Musk in a particularly famous way that includes both traditional startup-founder-y but also technical innovation
Richard Feyman (who found he had a hard time doing important work after working on the atom bomb, but then solved the problem by changing his mindset to deliberately not focus on “important” things and just follow his interests, which eventually led to more good ideas)
For people that are smart but not obviously “relentless resourceful”, see “one hit wonders”, or people who have certain kinds of genius but it only comes in flashes and they don’t really know how to cultivate it on purpose.
Resourceful and Creative sort of mean the same thing and for conciseness I’m going to mostly say “Relentlessly Resourceful” since it’s more fun evocative, but, there are some connotations creativity has that are important to not loose track of. i.e. not just able to fully exhaust all local resources, but, able to think from a wide variety of angles and see entirely different solutions that lie completely outside it’s current set of affordances.
I think this is less important than the other confusing terms in this thread, but something I stumbled into yesterday:
“Intelligence”/”Capable” → “Relentlessly Resourceful/Creative” [1]
(at least in some contexts)
i.e. the reason you expect a superintelligence to be difficult to control, is not exactly the raw intelligence. It’s that (some people think) the way something succeeds at being truly superintelligent requires being relentlessly resourceful.
(If it wasn’t relentlessly resourceful, it maybe could one-shot a large-but-shallow set of problems in a way that wasn’t concerning. But, then, if it hit a snag, it would get stuck, and it would be less useful than something that didn’t get stuck when it hit snags)
I like this because
a) it highlights what the problem is, more clearly.
and b), it highlights “if you could build a very useful powerful tool that succeeds without being relentlessly resourceful, that’s maybe a useful avenue.”
For examples of relentlessly resourceful people, see:
Startup founders
Prolific Inventors
Elon Musk in a particularly famous way that includes both traditional startup-founder-y but also technical innovation
Richard Feyman (who found he had a hard time doing important work after working on the atom bomb, but then solved the problem by changing his mindset to deliberately not focus on “important” things and just follow his interests, which eventually led to more good ideas)
For people that are smart but not obviously “relentless resourceful”, see “one hit wonders”, or people who have certain kinds of genius but it only comes in flashes and they don’t really know how to cultivate it on purpose.
Resourceful and Creative sort of mean the same thing and for conciseness I’m going to mostly say “Relentlessly Resourceful” since it’s more fun evocative, but, there are some connotations creativity has that are important to not loose track of. i.e. not just able to fully exhaust all local resources, but, able to think from a wide variety of angles and see entirely different solutions that lie completely outside it’s current set of affordances.
That’s also the title of a Paul Graham essay!
For future readers: Raemon expanded on this in a top-level post at “Intelligence” → “Relentless, Creative Resourcefulness”