Shiboleths in AI discourse mask a lot of deep disagreement between people. We chatter a lot is about shibboleths. About timelines and P doom. But shibboleths are social markers. They exist in far mode. They are not what our inner simulate actually predicts, if you’d give it the chance. In other words, we’re faking most of our talk on AI.
Another factor masking disagreement is the vagueness of shibboleths. Timelines to what? Doomed by what? Two guys can have the same timelines and P doom but have radically different interpretations of what that means. One guy thinks CEOs will use AIs to centralize power and kill off the permanent underclass. The other guy thinks Fable 9 will foom to godhood on the first day, kill everyone on the second, and start building a dyson sphere by the third.
I don’t think that is so extreme an example. If you took a random pair of people in the discourse who respect each other, they would have a disagreement maybe a tenth to a quarter that shocking. E.g. Robin Hanson and Eliezer Yudkowsky. Or JDPressman and Nostalgebraist. And so on.
A deeper reason for why this happens is that the future of AI involves a bunch of phenomena we haven’t got a good handle on collectively. Who here has experience with a superintelligence? No one. It is a fundamentally new thing that will push the world in very odd directions we haven’t got experience with. And humans SUCK at anything they don’t have experience with. No, really. Go look up the literature on transfer learning. It’s not just LLMs that fail to generalize. For example, good engineers in their later years have a habit of “building” perpetual motion machines. How can you build a god dang engine and not understand the conversation of energy?
This is just Gell-Mann amnesia, but you’re the reporter with the superficially sane takes that are wholly unmoored from reality. And there is no reason to think that your views unmoored from reality should look like anyone else’s. Every correct model is correct in the same way, everyone incorrect model is incorrect in its own way. Absent, of course, the fake models plastered on top for the sake of getting along with our fellow creatures.
And honestly, collective disagreement would require way more shared experience than what’s required for someone to have the right views. See e.g. thermodynamics where it was “obvious” for a long time that heat was a form of motion, but the caloric theory of heat, that heat was some sort of clingy fluid, persisted anyway. But we just don’t have that data. So the disagreement remains, hidden till the end of days.
Shiboleths in AI discourse mask a lot of deep disagreement between people. We chatter a lot is about shibboleths. About timelines and P doom. But shibboleths are social markers. They exist in far mode. They are not what our inner simulate actually predicts, if you’d give it the chance. In other words, we’re faking most of our talk on AI.
Another factor masking disagreement is the vagueness of shibboleths. Timelines to what? Doomed by what? Two guys can have the same timelines and P doom but have radically different interpretations of what that means. One guy thinks CEOs will use AIs to centralize power and kill off the permanent underclass. The other guy thinks Fable 9 will foom to godhood on the first day, kill everyone on the second, and start building a dyson sphere by the third.
I don’t think that is so extreme an example. If you took a random pair of people in the discourse who respect each other, they would have a disagreement maybe a tenth to a quarter that shocking. E.g. Robin Hanson and Eliezer Yudkowsky. Or JDPressman and Nostalgebraist. And so on.
A deeper reason for why this happens is that the future of AI involves a bunch of phenomena we haven’t got a good handle on collectively. Who here has experience with a superintelligence? No one. It is a fundamentally new thing that will push the world in very odd directions we haven’t got experience with. And humans SUCK at anything they don’t have experience with. No, really. Go look up the literature on transfer learning. It’s not just LLMs that fail to generalize. For example, good engineers in their later years have a habit of “building” perpetual motion machines. How can you build a god dang engine and not understand the conversation of energy?
This is just Gell-Mann amnesia, but you’re the reporter with the superficially sane takes that are wholly unmoored from reality. And there is no reason to think that your views unmoored from reality should look like anyone else’s. Every correct model is correct in the same way, everyone incorrect model is incorrect in its own way. Absent, of course, the fake models plastered on top for the sake of getting along with our fellow creatures.
And honestly, collective disagreement would require way more shared experience than what’s required for someone to have the right views. See e.g. thermodynamics where it was “obvious” for a long time that heat was a form of motion, but the caloric theory of heat, that heat was some sort of clingy fluid, persisted anyway. But we just don’t have that data. So the disagreement remains, hidden till the end of days.
Or perhaps you disagree?