One concern in the AI-Alignment problem is that neural networks are “alien minds”. Namely the representations that they learn of the world are too weird/different to allow effect communication of human goals and ideas.
People reading the sequences (as in nearly everyone in AI-Alignment) tend to just absorb/accept this “alien minds” belief, but even back in 2007/2008 it was not the majority position amongst AI-futurists. The MIT AI-futurists (Moravec/Minsky/Kurzweil) believed that AI would be our “mind children”, absorbing our culture and beliefs by default. Robin Hanson believed/believes AI will be literal brain uploads. EY was a bit on the fringe with this idea that AI would be unfathomable aliens.
If you are making this update only just now, consider why—trace the roots of these beliefs, and compare them to those who made the update long earlier.
I’ve spent a bit of time tracing this down to specific viewpoints on the brain and AI that are encoded in the sequences.
The MIT AI-futurists (Moravec/Minsky/Kurzweil) believed that AI would be our “mind children”, absorbing our culture and beliefs by default
At this stage, this doesn’t seem obviously wrong,. If you think that the path from AGI will come via LLM extension rather than experiencing the world in an RL regime, it will only have our cultural output to make sense of the world.
People reading the sequences (as in nearly everyone in AI-Alignment) tend to just absorb/accept this “alien minds” belief, but even back in 2007/2008 it was not the majority position amongst AI-futurists. The MIT AI-futurists (Moravec/Minsky/Kurzweil) believed that AI would be our “mind children”, absorbing our culture and beliefs by default. Robin Hanson believed/believes AI will be literal brain uploads. EY was a bit on the fringe with this idea that AI would be unfathomable aliens.
If you are making this update only just now, consider why—trace the roots of these beliefs, and compare them to those who made the update long earlier.
I’ve spent a bit of time tracing this down to specific viewpoints on the brain and AI that are encoded in the sequences.
At this stage, this doesn’t seem obviously wrong,. If you think that the path from AGI will come via LLM extension rather than experiencing the world in an RL regime, it will only have our cultural output to make sense of the world.