Stuart_Armstrong

Karma: 21,591

All I know is Goodhart

Stuart_Armstrong

21 Oct 2019 12:12 UTC

11 points

1 comment3 min readLW link

Full toy model for preference learning

Stuart_Armstrong

16 Oct 2019 11:06 UTC

11 points

0 comments12 min readLW link

Stuart_Armstrong 13 Oct 2019 14:48 UTC
LW: 2 AF: 1
AF
in reply to: abramdemski's comment on: Troll Bridge
You are entirely correct; I don’t know why I was confused.

However, looking at the proof again, it seems there might be a potential hole. You use Löb’s theorem within an assumption sub-loop. This seems to assume that from “ $A \to (□ B \to B)$ ”, we can deduce “ $A \to B$ ”.

But this cannot be true in general! To see this, set $A = □ B \to B$ . Then $A \to (□ B \to B)$ , trivially; if, from that, we could deduce $A \to B$ , we would have $(□ B \to B) \to B$ for any $B$ . But this statement, though it looks like Löb’s theorem, is one that we cannot deduce in general (see Eliezer’s “medium-hard problem” here).

Can this hole be patched?

(note that if $A \to (□_{A} B \to B)$ , where $□_{A}$ is a PA proof that adds A as an extra axiom, then we can deduce $A \to B$ ).

Stuart_Armstrong 3 Oct 2019 13:32 UTC
LW: 2 AF: 1
AF
in reply to: abramdemski's comment on: Troll Bridge
If the agent’s reasoning is sensible only under certain settings of the default action clause

That was my first rewriting; the second is an example of a more general algorithm that would go something like this. If we assume that both probabilities and utilities are discrete, all of the form q/n for some q, and bounded above and below by N, then something like this (for EU the expected utility, and Actions the set of actions, and b some default action):
```
for q integer in N*n^2 to -N*n^2 (ordered from highest to lowest):
    for a in Actions:
        if A()=a ⊢ EU=q/n^2 then output a
else output b
```
Then the Löbian proof fails. The agent will fail to prove any of those “if” implications, until it proves “A()=”not cross” ⊢ EU=0″. Then it outputs “not cross”; the default action b is not relevant. Also not relevant, here, is the order in which a is sampled from “Actions”.

Toy model #6: Rationality and partial preferences

Stuart_Armstrong

2 Oct 2019 12:04 UTC

11 points

0 comments2 min readLW link

Stuart_Armstrong 1 Oct 2019 15:05 UTC
LW: 2 AF: 1
AF
on: Troll Bridge
Interesting.

I have two issues with the reasoning as presented; the second one is more important.

First of all, I’m unsure about “Rather, the point is that the agent’s “counterfactual” reasoning looks crazy.” I think we don’t know the agent’s counterfactual reasoning. We know, by Löb’s theorem, that “there exists a proof that (proof of L implies L)” implies “there exists a proof of L”. It doesn’t tell us what structure this proof of L has to take, right? Who knows what counterfactuals are being considered to make that proof? (I may be misunderstanding this).

Second of all, it seems that if we change the last line of the agent to [else, “cross”], the argument fails. Same if we insert [else if A()=”cross” ⊢ U=-10, then output “cross”; else if A()=”not cross” ⊢ U=-10, then output “not cross”] above the last line. In both cases, this is because U=-10 is now possible, given crossing. I’m suspicious when the argument seems to depend so much on the structure of the agent.

To develop that a bit, it seems the agent’s algorithm as written implies “If I cross the bridge, I am consistent” (because U=-10 is not an option). If we modify the algorithm as I just suggested, then that’s no longer the case; it can consider counterfactuals where it crosses the bridge and is inconsistent (or, at least, of unknown consistency). So, given that, the agent’s counterfactual reasoning no longer seems so crazy, even if it’s as claimed. That’s because the agent’s reasoning needs to deduce something from “If I cross the bridge, I am consistent” that it can’t deduce without that. Given that statement, then being Löbian or similar seems quite natural, as those are some of the few ways of dealing with statements of that type.
What links here?
- Troll Bridge by abramdemski (23 Aug 2019 18:36 UTC; 72 points)

Stuart_Armstrong 30 Sep 2019 12:08 UTC
5 points

on: Stuart_Armstrong’s Shortform
Bayesian agents that knowingly disagree

A minor stub, caveating the Aumann’s agreement theorem; put here to reference in future posts, if needed.

Aumann’s agreement theorem states that rational agents with common knowledge of each other’s beliefs cannot agree to disagree. If they exchange their estimates, they will swiftly come to an agreement.

However, that doesn’t mean that agents cannot disagree, indeed they can disagree, and know that they disagree. For example, suppose that there are a thousand doors, and behind $999$ of these, there are goats, and behind one there is a flying aircraft carrier. The two agents are in separate rooms, and a host will go into each room and execute the following algorithm: they will choose a door at random among the $999$ that contain a goat. And, with probability $99 %$ , they will tell that door number to the agent; with probability $1 %$ , they will tell the door number with the aircraft carrier.

Then each agent will have probability $1 %$ of the named door being the aircraft carrier door, and $(99 / 999) % = (11 / 111) %$ probability on each of the other doors; so the most likely door is the one named by the host.

We can modify the protocol so that the host will never name the same door to each agent (roll a D100; if it comes up 1, tell the truth to the first agent and lie to the second; if it comes up 2, do the opposite; anything else means tell a different lie to either agent). In that case, each agent will have a best guess for the aircraft carrier, and the certainty that the other agent’s best guess is different.

If the agents exchanged information, they would swiftly converge on the same distribution; but until that happens, they disagree, and know that they disagree.

Stuart_Armstrong’s Shortform

Stuart_Armstrong

30 Sep 2019 12:08 UTC

9 points

1 comment1 min readLW link

Toy model piece #5: combining partial preferences

Stuart_Armstrong

12 Sep 2019 3:31 UTC

12 points

0 comments3 min readLW link

Toy model piece #4: partial preferences, re-re-visited

Stuart_Armstrong

12 Sep 2019 3:31 UTC

9 points

0 comments3 min readLW link

Stuart_Armstrong 12 Sep 2019 2:42 UTC
LW: 2 AF: 1
AF
in reply to: Donald Hobson's comment on: Is my result wrong? Maths vs intuition vs evolution in learning human preferences
I like this analogy. Probably not best to put too much weight on it, but it has some insights.

Stuart_Armstrong 10 Sep 2019 18:59 UTC
2 points

in reply to: Gurkenglas's comment on: Is my result wrong? Maths vs intuition vs evolution in learning human preferences
And whether those programs could then perform well if their opponent forces them into a very unusual situation, such as would not have ever appeared in a chessmaster game.

If I sacrifice a knight for no advantage whatsoever, will the opponent be able to deal with that? What if I set up a trap to capture a piece, relying on my opponent not seeing the trap? A chessmaster playing another chessmaster would never play a simple trap, as it would never succeed; so would the ML be able to deal with it?

Stuart_Armstrong 10 Sep 2019 16:50 UTC
2 points

in reply to: Gurkenglas's comment on: Is my result wrong? Maths vs intuition vs evolution in learning human preferences
PS: the other title I considered was “Why do people feel my result is wrong”, which felt too condescending.

Stuart_Armstrong 10 Sep 2019 16:34 UTC
5 points

in reply to: avturchin's comment on: Is my result wrong? Maths vs intuition vs evolution in learning human preferences
I agree we’re not as good as we think we are. But there are a lot of things we do agree on, that seem trivial: eg “this person is red in the face, shouting at me, and punching me; I deduce that they are angry and wish to do me harm”. We have far, far, more agreement than random agents would.

Stuart_Armstrong 10 Sep 2019 16:23 UTC
2 points

in reply to: Gurkenglas's comment on: Is my result wrong? Maths vs intuition vs evolution in learning human preferences

Your title seems clickbaity

Hehe—I don’t normally do this, but I feel I can indulge once ^_^

having implicit access to categorisation modules that themselves are valid only in typical situations… is not a way to generalise well

How do you know this?

Moravec’s paradox again. Chessmasters didn’t easily program chess programs; and those chess programs didn’t generalise to games in general.

Should we turn this into one of those concrete ML experiments?

That would be good. I’m aiming to have a lot more practical experiments from my research project, and this could be one of them.

Is my result wrong? Maths vs intuition vs evolution in learning human preferences

Stuart_Armstrong

10 Sep 2019 0:46 UTC

19 points

11 comments3 min readLW link

Simple and composite partial preferences

Stuart_Armstrong

9 Sep 2019 23:07 UTC

11 points

0 comments2 min readLW link

Stuart_Armstrong 7 Sep 2019 23:09 UTC
LW: 2 AF: 1
AF
in reply to: Wei_Dai's comment on: Best utility normalisation method to date?
Hum… It seems that we can stratify here. Let $X$ represent the values of a collection of variables that we are uncertain about, and that we are stratifying on.

When we compute the normalising factor for utility $U$ under two policies $π$ and $π^{'}$ , we normally do it as:
- $U \to U / N_{U}$ , with $N_{U} = \sum_{x} P (X = x) (E_{π, X = x} U - E_{π^{'}, X = x} U)$ .
And then we replace $U$ with $U / N_{U}$ .

Instead we might normalise the utility $U$ separately for each value of $x$ :
- Conditional on $X = x$ , then $U \to U / N_{U, x}$ , with $N_{U, x} = E_{π, X = x} U - E_{π^{'}, X = x} U$ .
The problem is that, since we’re dividing by the $N$ , the expectation of $U / N_{U, x}$ is not the same $U / N_{U}$ .

Is there an obvious improvement on this?

Note that here, total utilitarianism get less weight in large universes, and more in small ones.

I’ll think more...

Stuart_Armstrong 6 Sep 2019 1:13 UTC
LW: 3 AF: 2
AF
in reply to: William_S's comment on: Problems with AI debate
How about a third AI that gives a (hidden) probability about which one you’ll be convinced by, conditional on which argument you see first? That hidden probability is passed to someone else, then the debate is run, and the result recorded. If that third AI gives good calibration and good discrimination over multiple experiments, then we can consider its predictions accurate in the future.

Stuart_Armstrong 3 Sep 2019 16:38 UTC
3 points

in reply to: Gurkenglas's comment on: Best utility normalisation method to date?
Er, this normalisation system way well solve that problem entirely. If $U_{i}$ prefers option $o_{i}$ (utility $1$ ), with second choice $o_{0}$ (utility $1 / 2$ ), and all the other options as third choice (utility $0$ ), then the expected utility of the random dictator is $1 / n$ for all $U_{i}$ (as $p_{i}^{*}$ gives utility $1$ , and $p_{j}^{*}$ gives utility $0$ for all $j \neq i$ ), so the normalised weighted utility to maximise is:
- $U = \frac{1}{n - 1} (U_{1} + U_{2} + \dots U_{n})$ .
Using $(n - 1) U$ (because scaling doesn’t change expected utility decisions), the utility of any $o_{i}$ , $i > 0$ , is $1$ , while the utility of $o_{0}$ is $n / 2$ . So if $n > 2$ , the compromise option $o_{0}$ will get chosen.

Don’t confuse the problems of the random dictator, with the problems of maximising the weighted sum of the normalisations that used the random dictator (and don’t confuse the other way, either; the random dictator is immune to players’ lying, this normalisation is not).