TurnTrout
Karma: 1,949
NewTop
Another way of phrasing this is that I am pessimistic about the prospects of conceptual thinking, which seems to be the main way by which we could find a fundamental obstruction. (Theory and empirical experiments can build intuitions about what is and isn’t hard, but given the complexities of the real world it seems unlikely that either would give us the sort of crystallized knowledge that Paul is aiming for.) Phrased this way, I put less credence in this opinion, because I think there are a few examples of conceptual thinking being very important, though not that many.
Can you expand on your reasons for pessimism?
Thanks, my terminology was a little loose. What I was trying to hint at is that some of the paradox’s culling operations require uncomputable tests of English sentences, and that the regularity of the original language doesn’t determine the status of its subsets.
And My Axiom! Insights from ‘Computability and Logic’
I like this line of thought overall.
• How would we safely set lambda?
• Isn’t it still doing an argmax over plans and T, making the internal optimization pressure very non-mild? If we have some notion of embedded agency, one would imagine that doing the argmax would be penalized, but it’s not clear what kind of control the agent has over its search process in this case.
But a value neutral impact measure is almost impossible, because the world has too many degrees of freedom.
Can you explain why you think something like AUP requires value-laden inputs?
Penalizing Impact via Attainable Utility Preservation
[Question] Why should I care about rationality?
A New Mandate
Sounds like me at the beginning of this year; I’m now able to make my way through logical induction. I’d be happy to help, by the way—feel free to message me.
Then you can solve it, yeah.
He defined a strict contraction on a metric space as requiring for and for all . Your proposed solution doesn’t fix such a ; in fact, as , , which is why .
Claim: You can’t solve the exercise
Proof (thanks to TheMajor). Let be a sequence in the domain converging to such that . Since is a strict contraction with contraction constant , . Since the absolute value is continuous, we conclude that . ◻️
Yup. As someone aiming to do their dissertation on issues of limited agency (low impact, mild optimization, corrigibility), it sure would be frustrating to essentially end up duplicating the insights that MIRI has on some new optimization paradigm.
I still understand why they’re doing this and think it’s possibly beneficial, but it would be nice to avoid having this happen.
A more explicit construction is “let u evaluate to 1 iff it sees “high scoring” observation o at time t; clearly, its EU is increased. If u_A is this utility, let u instead evaluate to .99 iff it sees o at time t (and 0 otherwise).”
It’s true you could prove it in the way you mentioned (although the history h wouldn’t be supplied to the inner utility calculation), but it isn’t very suggestive for the instrumental convergence / opportunity cost phenomenon I was trying to point at.
It seems to me that “avoid irreversible high-impact actions” would only work if one had a small amount of uncertainty over one’s utility function, in which case you could just avoid actions that are considered “irreversible high-impact” by any the utility functions that you have significant probability mass on. But if you had a large amount of uncertainty, or just have very little idea what your utility function looks like, that doesn’t work because almost any action could be “irreversible high-impact”.
From the AUP perspective, this only seems true in a way analogous to the statement that “any hypothesis can have arbitrarily long description length”. It’s possible to make practically no assumptions about what the true utility function is and still recover a sensible notion of “low impact”. That is, penalizing shifts in attainable utility for even random or simple functions still yields the desired behavior; I have experimental results to this effect which aren’t yet published. This suggests that the notion of impact captured by AUP isn’t dependent on realizability of the true utility, and hence the broader thing Rohin is pointing at should be doable.
While it’s true that some complex value loss is likely to occur when not considering an appropriate distribution over extremely complicated utility functions, it seems by-and-large negligible. This is because such loss occurs either as a continuation of the status quo or as a consequence of something objectively mild, which seems to correlate strongly with reasonably human-values mild.
I broadly agree with your main points. However,
If rationalists had started winning, at least one person would have posted about it here on lesswrong.com.
I did post about this, and the benefits have continued to accrue. Compared to my past self, I perceive myself to be winning dramatically harder on almost all metrics I care about.
Oops! :)
Can you expand?
We have a choice here: “solve complex, value-laden problem” or “undertake cheap preparations so that the agent doesn’t have to deal with these scenarios”. Why not just run the agent from a secure server room where we look after it, shutting it down if it does bad things?
I absolutely love your writing style; this post was everything I hoped for when I loaded the page.
I think it’s generally a good property as a reasonable person would execute it. The problem, however, is the bad ex ante clinginess plans, where the agent has an incentive to pre-emptively constrain our reactions as hard as it can (and this could be really hard).
The problem is lessened if the agent is agnostic to the specific details of the world, but like I said, it seems like we really need IV (or an improved successor to it) to cleanly cut off these perverse incentives.
I’m not sure I understand the connection to scapegoating for the agents we’re talking about; scapegoating is only permitted if credit assignment is explicitly part of the approach and there are privileged “agents” in the provided ontology.
I agree with what you said for those environments, yeah. I was trying to express that I don’t expect this situation to be common, which is beside the point in light of your motivation for asking!
(I welcome these questions and hope my short replies don’t come off as impatient. I’m still dictating everything.)
But there are more objections; even if “computability” isn’t explicitly mentioned in the problem, it’s still present. Are the sets “the singleton set containing 1 if and only if machine M halts on input w” and “the singleton set containing 1” the same? Even if we grant a procedure for figuring out what counts as a set, we can’t even compute which sentences are duplicates.