Can I also point to this as (some amount of) evidence against concerns that “we” (members of this stupid robot cult that I continue to feel contempt for but don’t know how to quit) shouldn’t try to have systematically truthseeking discussions about potentially sensitive or low-status subjects because guilt-by-association splash damage from those conversations will hurt AI alignment efforts, which are the most important thing in the world? (Previously: 123.)
Like, I agree that some nonzero amount of splash damage exists. But look! The most popular AI textbook, used in almost fifteen hundred colleges and universities, clearly explains the paperclip-maximizer problem, in the authorial voice, in the first chapter. “These behaviors are not ‘unintelligent’ or ‘insane’; they are a logical consequence of defining winning as the sole objective for the machine.” Italics in original! I couldn’t transcribe it, but there’s even one of those pay-attention-to-this triangles (◀) in the margin, in teal ink.
Everyone who gets a CS degree from this year onwards is going to know from the teal ink that there’s a problem. If there was a marketing war to legitimize AI risk, we won! Now can “we” please stop using the marketing war as an excuse for lying?!
some predictable counterpoints: maybe we won because we were cautious; we could have won harder; many relevant thinkers still pooh-pooh the problem; it’s not just the basic problem statement that’s important, but potentially many other ideas that aren’t yet popular; picking battles isn’t lying; arguing about sensitive subjects is fun and I don’t think people are very tempted to find excuses to avoid it; there are other things that are potentially the most important in the world that could suffer from bad optics; I’m not against systematically truthseeking discussions of sensitive subjects, just if it’s in public in a way that’s associated with the rationalism brand
Can I also point to this as (some amount of) evidence against concerns that “we” (members of this stupid robot cult that I continue to feel contempt for but don’t know how to quit) shouldn’t try to have systematically truthseeking discussions about potentially sensitive or low-status subjects because guilt-by-association splash damage from those conversations will hurt AI alignment efforts, which are the most important thing in the world? (Previously: 1 2 3.)
Like, I agree that some nonzero amount of splash damage exists. But look! The most popular AI textbook, used in almost fifteen hundred colleges and universities, clearly explains the paperclip-maximizer problem, in the authorial voice, in the first chapter. “These behaviors are not ‘unintelligent’ or ‘insane’; they are a logical consequence of defining winning as the sole objective for the machine.” Italics in original! I couldn’t transcribe it, but there’s even one of those pay-attention-to-this triangles (◀) in the margin, in teal ink.
Everyone who gets a CS degree from this year onwards is going to know from the teal ink that there’s a problem. If there was a marketing war to legitimize AI risk, we won! Now can “we” please stop using the marketing war as an excuse for lying?!
some predictable counterpoints: maybe we won because we were cautious; we could have won harder; many relevant thinkers still pooh-pooh the problem; it’s not just the basic problem statement that’s important, but potentially many other ideas that aren’t yet popular; picking battles isn’t lying; arguing about sensitive subjects is fun and I don’t think people are very tempted to find excuses to avoid it; there are other things that are potentially the most important in the world that could suffer from bad optics; I’m not against systematically truthseeking discussions of sensitive subjects, just if it’s in public in a way that’s associated with the rationalism brand
(This extended runaround on appeals to consequences is at least a neat microcosm of the reasons we expect unaligned AIs to be deceptive by default! Having the intent to inform other agents of what you know without trying to take responsibility for controlling their decisions is an unusually anti-natural shape for cognition; for generic consequentialists, influence-seeking behavior is the default.)