Undo the update from the “counting argument”, however, and the probability of scheming plummets substantially.
Wait, why? Like, where is low probability is actually coming from? I guess from some informal model of inductive biases, but then why “formally I only have Solomonoff prior, but I expect other biases to not help” is not an argument?
I mean, it doesn’t matter that it’s not an evidence of sentience because trying to scale without reliable detectors (and architecture that allows for them) of ethically-significant properties was irresponsible from the start. And the correct response is shutting down of research, not “the only person in our system of checks who says we are wrong is the one we fired, so we are going to ignore them”.