We spent an evening at last week’s Rationality Minicamp… We came up with a concrete (albeit greatly oversimplified) model...
Just to be clear: this model was drafted by a couple of mini-camp participants, not by the workshop as a whole, and isn’t advocated by the Singularity Institute. For example, when I do my own back-of-the-envelopes I don’t expect nearly a 30% increase in existential safety from convincing 30% of AI researchers that risk matters. Among other things, this is because there’s a distance between “realize risk matters” and “successfully avoid creating UFAI” (much less “create FAI”)”, since sanity and know-how also play roles in AI design; and partly because there are more players than just AI researchers.
Still, it is good to get explicit models out there where they can be critiqued—I just want to avoid folks having the impression that this is SingInst’s model, or that it was taught at minicamp.
Just to be clear: this model was drafted by a couple of mini-camp participants, not by the workshop as a whole, and isn’t advocated by the Singularity Institute. For example, when I do my own back-of-the-envelopes I don’t expect nearly a 30% increase in existential safety from convincing 30% of AI researchers that risk matters. Among other things, this is because there’s a distance between “realize risk matters” and “successfully avoid creating UFAI” (much less “create FAI”)”, since sanity and know-how also play roles in AI design; and partly because there are more players than just AI researchers.
Still, it is good to get explicit models out there where they can be critiqued—I just want to avoid folks having the impression that this is SingInst’s model, or that it was taught at minicamp.