“One does not live through a turn of the galaxy by taking occasional small risks.”
I’ll admit to this that the author being Yudkowsky heavily colored how I read this line. He has repeatedly, strongly taken the stance that AI risk is not about small probabilities, he would not be thinking so much about AI risk if his probability were order-1%, people who do care about order-1% risks are being silly, etc. There are lots of quotes but I’ll take the first one I found on a search, not because it’s the closest match but that it’s the first one I found.
But the king of the worst award has to go to the Unironical Pascal’s Wager argument, imo—“Sure the chances are tiny, but if there’s even a tiny chance of destroying the lightcone...”
I do not know if I’m being unfair or generous to Yudkowsky to dismiss this defense for this reason. Regardless, I will.
I will say that the very next sentence Klurl states is,
“And to call this risk knowably small, would be to claim to know far too much.”
and indeed I think this is an example where the literary contrivance hides the mistake. If the author wasn’t forcing his hand, the risk would have been small. The coincidence they were in was unlikely on priors and not narrowed into from the arguments given.
—
What examples are you thinking of here?
It’s obvious that human learning is exceptional, but I don’t think Klurl’s arguments even served to distinguish the rock sharpening skill from beaver dams, spider webs or bird nests, never mind the general set of so-termed ‘tool use’ in the wild. Stone tools aren’t specific to humans, either, though I believe manufactured stone tools are localized to hominids, for example Homo floresiensis as a meaningfully distinct and AFAIK not ancestral cousin species.
Related but distinct, I’ll draw specific attention to ants, which have a fascinating variety of evolutionary behaviours, including quite fascinating trap making with a cultivated fungus. Obviously not a generalizably intelligent behaviour, but yet Klurl did not even ask that of humans. (On an even less related note, Messor ibericus lays clones of Messor structor as part of their reproductive cycle, which is fascinating and came to mind a lot when reading the sections about stuff evolution supposedly can’t solve because it, per the accusation, operates through one specific reproductive pathway.)
I’ll admit to this that the author being Yudkowsky heavily colored how I read this line. He has repeatedly, strongly taken the stance that AI risk is not about small probabilities, he would not be thinking so much about AI risk if his probability were order-1%, people who do care about order-1% risks are being silly, etc. There are lots of quotes but I’ll take the first one I found on a search, not because it’s the closest match but that it’s the first one I found.
— https://x.com/ESYudkowsky/status/1617903894960693249
I do not know if I’m being unfair or generous to Yudkowsky to dismiss this defense for this reason. Regardless, I will.
I will say that the very next sentence Klurl states is,
and indeed I think this is an example where the literary contrivance hides the mistake. If the author wasn’t forcing his hand, the risk would have been small. The coincidence they were in was unlikely on priors and not narrowed into from the arguments given.
—
It’s obvious that human learning is exceptional, but I don’t think Klurl’s arguments even served to distinguish the rock sharpening skill from beaver dams, spider webs or bird nests, never mind the general set of so-termed ‘tool use’ in the wild. Stone tools aren’t specific to humans, either, though I believe manufactured stone tools are localized to hominids, for example Homo floresiensis as a meaningfully distinct and AFAIK not ancestral cousin species.
Related but distinct, I’ll draw specific attention to ants, which have a fascinating variety of evolutionary behaviours, including quite fascinating trap making with a cultivated fungus. Obviously not a generalizably intelligent behaviour, but yet Klurl did not even ask that of humans. (On an even less related note, Messor ibericus lays clones of Messor structor as part of their reproductive cycle, which is fascinating and came to mind a lot when reading the sections about stuff evolution supposedly can’t solve because it, per the accusation, operates through one specific reproductive pathway.)