To be a good instrumental rationalist, an entity must be a good epistemic rationalist, because knowledge is instrumentally useful. But to be a good epistemic ratioanalist, and entityy must value certain things, like consistency and lack of contradiction. IR is not walled off from ER, which itself is not walled off from values. The orthogonality thesis is false. You can’t have any combination of values
and instrumental efficacy, because an enity that thinks contradictions are valuable will be a poor epistemic ratiionalist and therefore a poor instrumental rationalist.
But to be a good epistemic rationalist, and entity must value certain things, like consistency and lack of contradiction.
You appear to not understand the Orthogonality Thesis, since you have misstated it. The orthogonality thesis deliberately refers to preferences, not values, because values could also refer to instrumental values, whereas preferences can only refer to terminal values. (Obviously, consistency and lack of contradiction are only generally valued instrumentally.)
an entity that thinks contradictions are valuable will be a poor epistemic ratiionalist and therefore a poor instrumental rationalist.
No; if the entity values itself believing contradictions, then it having contradicting beliefs would mean it is a good instrumental rationalist.
An entity that has contradictory beliefs will be a poor instrumental rationalist. It looks like you would need to engineer a distinction between instrumental beliefs and terminal beliefs. While we’re on the subject, you might need a firewall to stop an .AI acting on intrinsically motivating ideas, if they exist. In any case, orthogonality is an architecture choice, not an ineluctable fact about minds.
The OT has multiple forms, as Armstrong notes. An OT that says you could make arbitrary combinations of preference and power if you really wanted to, can’t plug into an argument that future .AI will ,with high probability, be a Lovecraftian horror, at least not unless you also aargue that an orthogonal architecture will be chosen, with high probability.
To be a good instrumental rationalist, an entity must be a good epistemic rationalist, because knowledge is instrumentally useful. But to be a good epistemic ratioanalist, and entityy must value certain things, like consistency and lack of contradiction. IR is not walled off from ER, which itself is not walled off from values. The orthogonality thesis is false. You can’t have any combination of values and instrumental efficacy, because an enity that thinks contradictions are valuable will be a poor epistemic ratiionalist and therefore a poor instrumental rationalist.
You appear to not understand the Orthogonality Thesis, since you have misstated it. The orthogonality thesis deliberately refers to preferences, not values, because values could also refer to instrumental values, whereas preferences can only refer to terminal values. (Obviously, consistency and lack of contradiction are only generally valued instrumentally.)
No; if the entity values itself believing contradictions, then it having contradicting beliefs would mean it is a good instrumental rationalist.
An entity that has contradictory beliefs will be a poor instrumental rationalist. It looks like you would need to engineer a distinction between instrumental beliefs and terminal beliefs. While we’re on the subject, you might need a firewall to stop an .AI acting on intrinsically motivating ideas, if they exist. In any case, orthogonality is an architecture choice, not an ineluctable fact about minds.
The OT has multiple forms, as Armstrong notes. An OT that says you could make arbitrary combinations of preference and power if you really wanted to, can’t plug into an argument that future .AI will ,with high probability, be a Lovecraftian horror, at least not unless you also aargue that an orthogonal architecture will be chosen, with high probability.