I don’t see how Bayesianism/vNM/expected utility theory should argue in favor of orthogonality. I guess you mean something like: For any ontology, we can put an arbitrary utility on that ontology, and it would be perfectly compatible with any probability distribution. But probutility is a rather undifferentiated blob and a weak, minimal constraint. Arguments that a rational agent’s belief-value is “normatively constrained” to be representable as a vNM-ish probutility (along with the equivalence class of its transformations) are not arguments that there are no other constraints on what that probutility could be.
We can see Logical Induction as evidence against the Diagonality Thesis: beliefs about undecidable statements (which exist in consistent theories due to Gödel’s first incompleteness theorem) can take on any probability in the limit, though satisfy properties such as consistency with other assigned probabilities (in a Bayesian-like manner).
But isn’t this subsumed by “above and beyond the computational tractability of that goal” in:
The strong form of the Orthogonality Thesis says that there’s no extra difficulty or complication in the existence of an intelligent agent that pursues a goal, above and beyond the computational tractability of that goal.
If an undecidable statement X is relevant for pursuing some goal Y, then (the question of which action best achieves) Y inherits X’s undecidability.
Moreover, we can gather non-proof evidence for undecidable and computationally intractable statements that are value/action-relevant. Like, if someone were to prove that P vs NP is undecidable, we would still mostly believe that P=NP and proceed based on that.[1]
This is not a great example, e.g., because we may want to design cryptography for the worst-case of P=NP (even if very unlikely), if viable. There are probably much better examples but hopefully this suffices to communicate my point.
I don’t see how Bayesianism/vNM/expected utility theory should argue in favor of orthogonality.
I’m saying they argue against orthogonality in the post...
But isn’t this subsumed by “above and beyond the computational tractability of that goal”
You seem to think either “diagonality” or “strong orthogonality” must hold. But the post is arguing the converse. I am arguing against strong orthogonality and against diagonality.
Rough argument against diagonality is something like “paperclip maximizer like entities seem like they would be possible / coherent” although there are some unknowns there like how different parts of the agent separated by large distances coordinate/mutate. But perhaps more basic than that is, if someone is making a strong claim (diagonality) they should probably justify it.
Some people may think that the free parameters in Bayes/VNM point towards the Orthogonality Thesis being true.
and I’m just confused by those people’s inferences / the arguments grounded in vNM that they give.
Perhaps instead of “I guess you mean” I should have written “I guess (you mean that) those people think / the argument is that”.
On the second point, I was not assuming [Diagonality v StrongOrthogonality], but I did misread what you were arguing for there, so sorry, and thanks for the correction.
Ah. I think the inference they may take is that a paperclip maximizer is perfectly rational/coherent, as is a staple maximizer and so on. They don’t think there are additional constraints as you suggest, beyond minimal ones like not having an “especially stupid” goal, such as “die as fast as possible”.
I don’t see how Bayesianism/vNM/expected utility theory should argue in favor of orthogonality. I guess you mean something like: For any ontology, we can put an arbitrary utility on that ontology, and it would be perfectly compatible with any probability distribution. But probutility is a rather undifferentiated blob and a weak, minimal constraint. Arguments that a rational agent’s belief-value is “normatively constrained” to be representable as a vNM-ish probutility (along with the equivalence class of its transformations) are not arguments that there are no other constraints on what that probutility could be.
But isn’t this subsumed by “above and beyond the computational tractability of that goal” in:
If an undecidable statement X is relevant for pursuing some goal Y, then (the question of which action best achieves) Y inherits X’s undecidability.
Moreover, we can gather non-proof evidence for undecidable and computationally intractable statements that are value/action-relevant. Like, if someone were to prove that P vs NP is undecidable, we would still mostly believe that P=NP and proceed based on that.[1]
This is not a great example, e.g., because we may want to design cryptography for the worst-case of P=NP (even if very unlikely), if viable. There are probably much better examples but hopefully this suffices to communicate my point.
I’m saying they argue against orthogonality in the post...
You seem to think either “diagonality” or “strong orthogonality” must hold. But the post is arguing the converse. I am arguing against strong orthogonality and against diagonality.
Rough argument against diagonality is something like “paperclip maximizer like entities seem like they would be possible / coherent” although there are some unknowns there like how different parts of the agent separated by large distances coordinate/mutate. But perhaps more basic than that is, if someone is making a strong claim (diagonality) they should probably justify it.
Yeah, but you open with
and I’m just confused by those people’s inferences / the arguments grounded in vNM that they give.
Perhaps instead of “I guess you mean” I should have written “I guess (you mean that) those people think / the argument is that”.
On the second point, I was not assuming [Diagonality v StrongOrthogonality], but I did misread what you were arguing for there, so sorry, and thanks for the correction.
Ah. I think the inference they may take is that a paperclip maximizer is perfectly rational/coherent, as is a staple maximizer and so on. They don’t think there are additional constraints as you suggest, beyond minimal ones like not having an “especially stupid” goal, such as “die as fast as possible”.