What is the mere Earth at stake, compared to a tiny probability of 3^^^^3 lives?
Do you really think this would be clearer or more rigorous if written in mathematical notation?

The problem is that Solomonoff induction is an essentially opaque model.

Think of it as a black box: you put in a string of bits representing your past observations and it gives you a probability distribution on string of bits representing your possible future observations. If you open the lid of the box, you will see many (ideally infinitely many) computer programs with arbitrary structure. There is no easy way to map that model to a probability distribution on non-directly observable world state variables such as “the number of people alive”.

Isn’t that kinda the point?

My interpretation is that Yudkowsky assumes Solomonoff induction essentially a priori and thus is puzzled by the dilemma it allegedly yields. My point is that:

It’s not obvious that Solomonoff induction actually yeilds that dilemma.

If it does, then this would be a good reason to reject it.

The problem is that Solomonoff induction is an essentially opaque model.

Think of it as a black box: you put in a string of bits representing your past observations and it gives you a probability distribution on string of bits representing your possible future observations. If you open the lid of the box, you will see many (ideally infinitely many) computer programs with arbitrary structure. There is no easy way to map that model to a probability distribution on non-directly observable world state variables such as “the number of people alive”.

My interpretation is that Yudkowsky assumes Solomonoff induction essentially a priori and thus is puzzled by the dilemma it allegedly yields. My point is that:

It’s not obvious that Solomonoff induction actually yeilds that dilemma.

If it does, then this would be a good reason to reject it.

And

mypoint is that:It

seemslike it should; the article explicitly asks the reader to try and disprove this.That’s kinda the point of the article.

It appears we all agree. I think.