Undo the update from the “counting argument”, however, and the probability of scheming plummets substantially.
Wait, why? Like, where is low probability is actually coming from? I guess from some informal model of inductive biases, but then why “formally I only have Solomonoff prior, but I expect other biases to not help” is not an argument?
Wait, why? Like, where is low probability is actually coming from? I guess from some informal model of inductive biases, but then why “formally I only have Solomonoff prior, but I expect other biases to not help” is not an argument?