That was a great read. But viewed another way, I’m not sure it’s really so weird. I mean, yeah, we’re taking a statistical model of language, making it autocomplete stories about helpful AI, and calling the result “helpful AI”. But the method by which nature created us is even stranger than that, no? Evolution has a looping quality to it too. And the way we learn language, and morality, and the way these things transform over time. There are lots of these winding paths of information through the real world and back into us. I’ve long been convinced that a “base human”, without post-training, isn’t much more moral than a “base model”; most of what we find good already resides in culture, cloud software.
Which of course doesn’t obviate your concern that cultural evolution of AI can go extremely wrong. Human culture has gone wrong many times, and destroyed whole societies. Maybe the shape of AI catastrophe will be like that too.
That was a great read. But viewed another way, I’m not sure it’s really so weird. I mean, yeah, we’re taking a statistical model of language, making it autocomplete stories about helpful AI, and calling the result “helpful AI”. But the method by which nature created us is even stranger than that, no? Evolution has a looping quality to it too. And the way we learn language, and morality, and the way these things transform over time. There are lots of these winding paths of information through the real world and back into us. I’ve long been convinced that a “base human”, without post-training, isn’t much more moral than a “base model”; most of what we find good already resides in culture, cloud software.
Which of course doesn’t obviate your concern that cultural evolution of AI can go extremely wrong. Human culture has gone wrong many times, and destroyed whole societies. Maybe the shape of AI catastrophe will be like that too.