What if, the moment the AI boots up, a bunch of humans tell it “our goals aren’t on a spike.” (It could technically realize this based on anthropic reasoning. If humans really wanted to maximize paperclips, and its easy to build a paperclip maximizer, we wouldn’t have built a non-obstructive AI.)
We are talking policies here. If the humans goals were on a spike, they wouldn’t have said that. So If the AI takes the policy of giving us a smoother attainable utility function in this case, this still fits the bill.
Actually I think that this definition is pushing much of the work off onto pol . This is a function that can take any utility function, and say how a human would behave if maximising it. Flip that round, this takes the human policy and produces a set of possible utility functions. (Presumably very similar functions.) Over these indistinguishable utility functions, the AI tries to make sure that none of them are lower than they would have been if the AI didn’t exist. Whether this is better or worse than maximizing the minimum or average would be sensitively dependant on exactly what this magic set of utility functions generated by pol−1 is.
I think things are already fine for any spike outside S, e.g. paperclip maximiser, since non-obstruction doesn’t say anything there.
I actually think saying “our goals aren’t on a spike” amounts to a stronger version of my [assume humans know what the AI knows as the baseline]. I’m now thinking that neither of these will work, for much the same reason. (see below)
The way I’m imagining spikes within S is like this: We define a pretty broad S, presumably implicitly, hoping to give ourselves a broad range of non-obstruction.
For all P in U we later conclude that our actual goals are in T⊂U ⊂S. We optimize for AU on T, overlooking some factors that are important for P in U \ T. We do better on T than we would have by optimising more broadly over U (we can cut corners in U \ T). We do worse on U \ T since we weren’t directly optimising for that set (AU on U \ T varies quite a lot). We then get an AU spike within U, peaking on T.
The reason I don’t think telling the AI something like “our goals aren’t on a spike” will help, is that this would not be a statement about our goals, but about our understanding and competence. It’d be to say that we never optimise for a goal set we mistakenly believe includes our true goals (and that we hit what we aim for similarly well for any target within S).
It amounts to saying something like “We don’t have blind-spots”, “We won’t aim for the wrong target”, or, in the terms above, “We will never mistake any T for any U”. In this context, this is stronger and more general than my suggestion of “assume for the baseline that we know everything you know”. (lack of that knowledge is just one way to screw up the optimisation target)
In either case, this is equivalent to telling the AI to assume an unrealistically proficient/well-informed pol. The issue is that, as far as non-obstruction is concerned, the AI can then take actions which have arbitrarily bad consequences for us if we don’t perform as well as pol. I.e. non-obstruction then doesn’t provide any AU guarantee if our policy isn’t actually that good.
My current intuition is that anything of the form “assume our goals aren’t on a spike”, “assume we know everything you know”… only avoid creating other serious problems if they’re actually true—since then the AI’s prediction of pol’s performance isn’t unrealistically high.
Even for “we know everything you know”, that’s a high bar if it has to apply when the AI is off. For “our goals aren’t on a spike”, it’s an even higher bar.
If we could actually make it true that our goals weren’t on a spike in this sense, that’d be great. I don’t see any easy way to do that. [Perhaps if the ability to successfully optimise for S already puts such high demands on our understanding, that distinguishing Ts from Us is comparatively easy.… That seems unlikely to me.]
What if, the moment the AI boots up, a bunch of humans tell it “our goals aren’t on a spike.” (It could technically realize this based on anthropic reasoning. If humans really wanted to maximize paperclips, and its easy to build a paperclip maximizer, we wouldn’t have built a non-obstructive AI.)
We are talking policies here. If the humans goals were on a spike, they wouldn’t have said that. So If the AI takes the policy of giving us a smoother attainable utility function in this case, this still fits the bill.
Actually I think that this definition is pushing much of the work off onto pol . This is a function that can take any utility function, and say how a human would behave if maximising it. Flip that round, this takes the human policy and produces a set of possible utility functions. (Presumably very similar functions.) Over these indistinguishable utility functions, the AI tries to make sure that none of them are lower than they would have been if the AI didn’t exist. Whether this is better or worse than maximizing the minimum or average would be sensitively dependant on exactly what this magic set of utility functions generated by pol−1 is.
I think things are already fine for any spike outside S, e.g. paperclip maximiser, since non-obstruction doesn’t say anything there.
I actually think saying “our goals aren’t on a spike” amounts to a stronger version of my [assume humans know what the AI knows as the baseline]. I’m now thinking that neither of these will work, for much the same reason. (see below)
The way I’m imagining spikes within S is like this:
We define a pretty broad S, presumably implicitly, hoping to give ourselves a broad range of non-obstruction.
For all P in U we later conclude that our actual goals are in T ⊂ U ⊂ S.
We optimize for AU on T, overlooking some factors that are important for P in U \ T.
We do better on T than we would have by optimising more broadly over U (we can cut corners in U \ T).
We do worse on U \ T since we weren’t directly optimising for that set (AU on U \ T varies quite a lot).
We then get an AU spike within U, peaking on T.
The reason I don’t think telling the AI something like “our goals aren’t on a spike” will help, is that this would not be a statement about our goals, but about our understanding and competence. It’d be to say that we never optimise for a goal set we mistakenly believe includes our true goals (and that we hit what we aim for similarly well for any target within S).
It amounts to saying something like “We don’t have blind-spots”, “We won’t aim for the wrong target”, or, in the terms above, “We will never mistake any T for any U”.
In this context, this is stronger and more general than my suggestion of “assume for the baseline that we know everything you know”. (lack of that knowledge is just one way to screw up the optimisation target)
In either case, this is equivalent to telling the AI to assume an unrealistically proficient/well-informed pol.
The issue is that, as far as non-obstruction is concerned, the AI can then take actions which have arbitrarily bad consequences for us if we don’t perform as well as pol.
I.e. non-obstruction then doesn’t provide any AU guarantee if our policy isn’t actually that good.
My current intuition is that anything of the form “assume our goals aren’t on a spike”, “assume we know everything you know”… only avoid creating other serious problems if they’re actually true—since then the AI’s prediction of pol’s performance isn’t unrealistically high.
Even for “we know everything you know”, that’s a high bar if it has to apply when the AI is off.
For “our goals aren’t on a spike”, it’s an even higher bar.
If we could actually make it true that our goals weren’t on a spike in this sense, that’d be great.
I don’t see any easy way to do that.
[Perhaps if the ability to successfully optimise for S already puts such high demands on our understanding, that distinguishing Ts from Us is comparatively easy.… That seems unlikely to me.]