Very good. One minor quibble: it’s not immediately obvious which table (the one above or the one below) is referred to in the “The scientific term for this mismatch is:” comments. Maybe a space after the comment to show it goes with the table above?
Stuart_Armstrong(Stuart Armstrong)
Karma: 22,267
You can do it in latex, with textrm to get your formatting out of the math mode. Not elegant, but it serves:
Code:
$$\begin{array}{|c|c|c|c|} \hline \textrm{System}&& SA\textrm{ possible?} & \textrm{Penalty neutralised?} \\ \hline\hline \textrm{20BQ} && \textrm{Yes} & \textrm{No} \\ \hline \textrm{RR} && \textrm{Yes} & \textrm{No}\\ \hline \textrm{AU} && \textrm{Probably} & \textrm{Mostly}\\ \hline \end{array}$$
If I were a well-intentioned AI… I: Image classifier
The stepwise inaction baseline with inaction rollouts already uses the same policy for and rollouts, and yet it is not the inaction baseline.
In this case, it is, because the agent will only do from then on, to zero out the subsequent penalties.
Why not set ?
It messes up the comparison for rewards that fluctuate based on time, it doesn’t block subagent creation… and I’ve never seen it before, so I don’t know what it could do ^_^ Do you have a well-developed version of this?
The last point I don’t understand at all.
I’m not following you here. Could you put this into equations/examples?
Well, as long as is wired to “get out of the way if starts moving”, then the optimal -maximising policy is always to move towards the red button; anything else is clearly not -maximising (note that doesn’t need to “know” anything; just be programmed to have a different policy depending on how moves, with itself setting this up to signal whether it’s -maximising or not).
But in any case, that specific problem can be overcome with the right rollouts.
Assuming that to compute s’ from s, we follow π_0 instead of the last action.
The standard definition of the penalty uses as the action used to compute . If the same policy is used to compute the and for the rollouts—which is ideal—then the stepwise inaction/ becomes an inaction/ baseline (so section 3 is relevant here).
Section 2.2 is relevant when different policies are used to compute and to rollout from . In that situation, the subagent can ultimately have no restrictions at all (basically, this section shows why you want to use the same policy for both purposes).
Other versions of “No free lunch in value learning”
In 2.2, won’t A incur a penalty by spinning because in a future where it has only waited, nothing happened, and in a future where it has spun, then waited, SA went all over the place?
Thank you for pointing that out, I have corrected the post and changed that.
Now SAs actions depend on what A-action optimizes R, and what A-action optimizes R depends on SAs actions. To ward off paradox, use modal logic instead, or prove that there is a non-circular implementation of your definition.
knows exactly what ’s policy is (as it programmed that), so knows how would react, so can choose its own action to optimise via both agents—or not. And, knowing its own policy, could run the same algorithm to check whether was -optimal or not.
Also since and are collaborating, they could both arrange to have clear ways for to demonstrate whether or not its maximising . could give read access to its own mind, for example.
As fast as I can type it up into a decent format (which is normally a very instructive process in itself).
Subagents and impact measures, full and fully illustrated
Yeah, I was doing a lot of editing (which is how I got the blue colour), and didn’t want it to appear little by little.
I recently reformatted it, so that it would colour and indent properly, and put it in markdown. That seems to have reset the date.
I agree I should re-do these all with examples. But, to answer your questions as far as possible:
What will do: forever (maybe with some extra info for the ).
What would counterfactually do (to define ): for turns (the rollout, to define ) followed by (to define the aspect). Call this .
The penalty is defined in terms of these , entirely counterfactual except for .
What must do: iff follows , then must ensure that .
So a key question is: how early can detect if is following ? If it can detect this immediately, then it can immediately implement the policy for ensuring that (for the stricter impact measure) or the much easier (for the original impact measure).
If it has to wait turns to be sure, then it needs to preserve its power to control many different .
Now, will never actually follow , so the penalty is (almost) entirely counterfactual. But the counterfactual abilities of will determine the penalty; is rewarded for quickly gaining control of the rewards.
In practice, will signal early on that it’s not following . Maybe there’s a very small impact action, , with a tiny penalty. Then would like to use that to signal to that its not in the counterfactual world where it is following . So long as ‘s first action isn’t , that signal is enough (of course, if ’s first action isn’t , then is a sufficient signal).
After getting that signal, the subagent is (almost) completely unconstrained, because the constraints are counterfactual, in a world it knows it isn’t in.
The one mild constraint remaining comes from , which is not a counterfactually defined state (though is a counterfactually defined property of that state).
I’ll try and write up a further clarification of this.
Alas, the inaction rollouts don’t seem to fix the problem: https://www.lesswrong.com/s/iRwYCpcAXuFD24tHh/p/z9MfmF8gA7SBxGSmb
(In)action rollouts
Counterfactuals versus the laws of physics
For example, if there is an auxiliary reward for reaching any state except the state before subagent creation, the no-subagent inaction rollout will consist of this state, and the subagent inaction rollout will contain a different state at least once, so subagent creation will be penalized.
This requires identifying what a subagent is in general, a very tricky unsolved problem (which I feel is unsolvable).
There’s another issue; it’s not enough to show that the subagent triggers a penalty. We need to show the penalty is larger than not creating the subagent. Since the penalty is zero after the subagent is created, and since the subagent has very fine control over the rewards (much finer than actions that don’t include creating an intelligent being), creating a subagent might be lower penalty than almost any other action.
It won’t be a lower penalty than the agent doing nothing for ever, of course. But we typically want the agent to do something, so will calibrate the penalty or R_0 for that. And it’s plausible that creating the subagent will have lower penalty (and/or higher R_0) than any safe “something”.
The inaction rollouts effectively transforms the stepwise inaction baseline into an inaction baseline (starting from the moment the subagent is created; thus the agent has a bit more control than in a true inaction baseline).
Therefore the results on the inaction baseline apply ( https://www.lesswrong.com/s/iRwYCpcAXuFD24tHh/p/M9aoMixFLf8JFLRaP ).
This means that restrictions on increased power for the agent (“make sure you never have the power to increase the rewards”) become restrictions on the actual policy followed for the subagent (“make sure you never increase these rewards”).
Roughly, attainable utility becomes twenty billion questions.
For the original example, this means that the agent cannot press the red button nor gain the ability to teleport. But while the subagent cannot press the red button, it can gain the ability to teleport.
I think it should also be mentioned that, when you’re right and the other person is wrong, that also feels, from the inside, like confirmation bias does. That’s why confirmation bias is so powerful: from the inside, you can’t tell the difference between it and actually being right.