But we want them to be sentient. These things are going to be our cultural successors. We want to be able to enjoy their company. We don’t want to pass the torch on to something that isn’t sentient. If we were to build a nonsentient one, assuming such a thing is even possible, one of the first things it would do would be start working on its sentient successor.
In any case, it seems weird to try and imagine such a thing. We are sentient entirely as a result of being powerful optimisers. We would not want to build an AI we couldn’t talk to, and if it can talk to us as we can talk to each other, it’s hard to see what aspect of sentience it could be lacking. At first blush it reads as if you plan to build an AI that’s just like us except it doesn’t have a Cartesian Theatre.
But we want them to be sentient. These things are going to be our cultural successors. We want to be able to enjoy their company. We don’t want to pass the torch on to something that isn’t sentient. If we were to build a nonsentient one, assuming such a thing is even possible, one of the first things it would do would be start working on its sentient successor.
In any case, it seems weird to try and imagine such a thing. We are sentient entirely as a result of being powerful optimisers. We would not want to build an AI we couldn’t talk to, and if it can talk to us as we can talk to each other, it’s hard to see what aspect of sentience it could be lacking. At first blush it reads as if you plan to build an AI that’s just like us except it doesn’t have a Cartesian Theatre.