That seems doable, if you trick the AI into tearing apart a simulation before it figures out it’s in one.
But how do you test whether the AI weighted the nematodes so highly because their qualia are extra phenomenologically vivid, and not because their qualia are extra phenomenologically clipperiffic?
Come, now, it’s hardly untestable. You can pay him if the FAI kills everyone to tile the universe with nematodes.
That seems doable, if you trick the AI into tearing apart a simulation before it figures out it’s in one.
But how do you test whether the AI weighted the nematodes so highly because their qualia are extra phenomenologically vivid, and not because their qualia are extra phenomenologically clipperiffic?