It’s not hard to find amateurs who got it right, after the fact. Amateur advice is of lower expected quality, but of much higher variance than expert advice.
Apart from filtering out the obvious crazies, can we identify high quality amateur advice ahead of time?
Sounds good, cheers!
There are some minor differences; your approach learns the whole model, whereas mine assumes the model is given, and learns only the “acausalish” aspects of it. But they are pretty similar.
One problem you might have, is learning the acausal stuff in the mid-term. If the agent learns that causality exists, and then that in the Newcomb problem is seems to have a causal effect, then it may search a lot for the causal link. Eventually this won’t matter (see here), but in the mid-term it might be a problem.
Or not. We need to test more ^_^
The future is not what I expected...
Yep!
They don’t fit together in that case; that’s addressed immediately after, in section 2.3.
You’re rediscovering some classics ^_^
That problem addresses some of the issues in anthropic reasoning—but not all.
Very impressive work, both the output and how you iterate on it.
Some thoughts about the cross-examination issue, prompted by your “Implementation 2 for human debaters: teams of two”. It occurred to me that B* could win if it could predict A and B’s future behaviour, and match up it’s answer with B.
I’d prefer that such an option not exist; that B could answer the question directly, without needing to rewind. Hence prediction won’t help.
Cross-examination still helps: A can cross examine as soon as they suspect B is shielding behind an ambiguity. This means that A might have to abandon their current question line, and start again on the other one. This seems more secure (if longer).
Yep ^_^
See 3.1 in my old tech report: https://www.fhi.ox.ac.uk/wp-content/uploads/Anthropic_Decision_Theory_Tech_Report.pdf
The AUP is supposed to prevent the agent accumulating power. The AI initially has huge potential power (because its potential power is all the power it could ever accumulate, given its best strategy to accumulate power) and the penalty is supposed to prevent it turning that potential into actual power—as measured by AUP.
So the AI always has the power to build a subagent; that post just shows that it can do this without triggering the AUP-power penalty.
I mainly mentioned child-rulers because Robin was using that example; and I used “getting deposed” as an example of agency problems that weren’t often (ever?) listed in the economics literature.
Do you expect it to fail in other areas?
Yes. Subagent problems are not cleanly separated from other problems (see section 3.4 of https://www.lesswrong.com/posts/mdQEraEZQLg7jtozn/subagents-and-impact-measures-full-and-fully-illustrated , where the subagent is replaced with a rock). The impact penalty encourages the agent to put restrictions on their own future possible actions. Doing this through a subagent is one way, but there are many others (see Odysseus and the sirens, or section 6.2 of the post above in this comment).
More seriously, the reason I’m sceptical of impact measures is because it feels that they all fail for the same reason. Unfortunately, I can’t articulate that reason; it’s the result of a long history of trying to build impact measures and trying to break them. I just have a feel for where the weaknesses are. So I knew that subagents would be a problem for AUP, long before I could articulate it formally.
But, as I said, I unfortunately can’t formalise this feeling; it remains personal.
For this example, it was harder than usual to come up with a counter-example. And I was surprised that half of AUP survived fine—I would not have expected that a restriction against lowering your power would be unhackable. So consider these mild positives for your approach.
But my instinctive feeling remains: I hope that AUP can be made to work for superintelligences, but I expect that it won’t :-(
But if you look back at the history of impact measures and AUP, you’ll see lots of skulls; people say “this problem dooms AUP”, and I say “I think we’re talking about conceptually different things and that you’re a little overconfident; probably just a design choice issue”. It then ends up being a solvable design choice issue. So by Laplace’s Rule of Succession, I’d be surprised if this were The Insurmountable Problem That Dooms AUP.
But if you look back at the history of impact measures, you’ll see a lot of skulls; people say “this impact measure will work”, and I say “I think you’re overconfident; I’m sure I can find a way it fails”. It then ends up being a solvable “break the impact measure” problem. So by Laplace’s Rule of Succession, I’d be surprised if this approach were The Ultimate Impact Measure That Works.
;-)
I think it should also be mentioned that, when you’re right and the other person is wrong, that also feels, from the inside, like confirmation bias does. That’s why confirmation bias is so powerful: from the inside, you can’t tell the difference between it and actually being right.
Is there any meaningful distinction between filtered evidence and lying? I know that in toy models these can be quite different, but in the expansive setting here, where the speaker can select the most misleading technically true fact, is there any major difference?
And how would the results here look if we expended it to allow the speaker to lie?