I’ll probably have a lot more to say on this entire post later, but for now I wanted to address one point. Some problems, like wire-heading, may not be deal-breakers or reasons to write anything off. Humans are capable of hijacking their own reward centers and “wireheading” in many different ways (the most obvious being something like shooting heroin), yet that doesn’t mean humans aren’t intelligent. Things like wireheading, bad priors, or the possibility of “trolling”[https://www.lesswrong.com/posts/CvKnhXTu9BPcdKE4W/an-untrollable-mathematician-illustrated] may just be hazards of intelligence.
If you’re born and you build a model of how the world works based on your input, then you start using that model to reject noise, you might be rejecting information that would fix flaws and biases in your model. When you’re young, it makes sense to believe elders when they tell you about how the world works because they probably know better than you, but if all those elders and everyone around you tell you about their belief in some false superstition, then that becomes an integral part of your world model, and evidence against your flawed world model may come long after you’ve weighted the model with an extremely high probability of being true. If the superstition involves great reward for adhering to and spreading it and great punishment for questioning it, then you have a trap that most valid models of intelligence will struggle with.
If some theory of intelligence is susceptible to that trap, it’s not clear that said theory should be dismissed because the only current implementation of general intelligence we know of is also highly susceptible to that trap.
I’ll probably have a lot more to say on this entire post later, but for now I wanted to address one point. Some problems, like wire-heading, may not be deal-breakers or reasons to write anything off. Humans are capable of hijacking their own reward centers and “wireheading” in many different ways (the most obvious being something like shooting heroin), yet that doesn’t mean humans aren’t intelligent. Things like wireheading, bad priors, or the possibility of “trolling”[https://www.lesswrong.com/posts/CvKnhXTu9BPcdKE4W/an-untrollable-mathematician-illustrated] may just be hazards of intelligence.
If you’re born and you build a model of how the world works based on your input, then you start using that model to reject noise, you might be rejecting information that would fix flaws and biases in your model. When you’re young, it makes sense to believe elders when they tell you about how the world works because they probably know better than you, but if all those elders and everyone around you tell you about their belief in some false superstition, then that becomes an integral part of your world model, and evidence against your flawed world model may come long after you’ve weighted the model with an extremely high probability of being true. If the superstition involves great reward for adhering to and spreading it and great punishment for questioning it, then you have a trap that most valid models of intelligence will struggle with.
If some theory of intelligence is susceptible to that trap, it’s not clear that said theory should be dismissed because the only current implementation of general intelligence we know of is also highly susceptible to that trap.