owencb(Owen Cotton-Barratt)

Karma: 1,438

owencb 23 Jul 2013 23:01 UTC
1 point
on: Gains from trade: Slug versus Galaxy—how much would I give up to control you?
I asked this in person, but I don’t think you’ve addressed it in the write-up:

The use of utility functions to try to capture the power dynamic seems to run into problems. Wei_Dai has an example with non-linear utility functions, but we can make it even more obvious.

In your original example, say the slug just doesn’t really care about anything other than his leaf. It would marginally prefer the entire galaxy to be turned into a garden of leaves, but that preference is really miniscule compared to its normal preferences about its own domain. Then we are in the situation where the two agents—the galactic empire and slug—each care a lot about their own domain, and very little about that of the other. If it happens that the slug is even more self-centred than the empire, then with your solution the slug’s preferences win out … to the limited degree permitted by the constraint that the empire can’t lose more than it can gain.

Even if you don’t think you’d naturally find utility functions like this (although many people seem to have preferences of roughly this shape), if this were a standard bargaining system it would become pragmatic to modify your own utility function to care very little about things you don’t expect to be able to affect.

As a side note, I wonder if it would be better to name apart what you’re trying to do here as something other than ‘bargaining’. It runs against my intuitive understanding of the word, and I believe the standard use in the literature (but I’m not certain), to establish a defection point then allow ‘bargains’ which are worse for one side than the defection point.

owencb 31 Jul 2013 16:25 UTC
4 points
on: Mutual Worth without default point (but with potential threats)
To me this feels quite a lot more natural than the previous version. It’s not clear whether the vulnerability to threats is a positive or negative feature.

Part of your stated original motivation for exploring this was that traditional bargaining solutions didn’t deal with power. We could adapt your solution here to define a defection point: the scenario where the first agent has the negative utility function of the second agent, and vice-versa. Then we could investigate the Nash bargaining solution with respect to this defection point. Without more thought (and perhaps playing with some examples), it’s not obvious to me if that’s better or worse than your proposal here.

owencb 3 Aug 2013 10:40 UTC
2 points
in reply to: Eliezer Yudkowsky’s comment on: Do Earths with slower economic growth have a better chance at FAI?
I think this is the strongest single argument that economic growth might currently be bad. However even then what matters is the elasticity of movement growth rates with economic growth rates. I don’t know how we can measure this; I expect it’s positive and less than one, but I’m rather more confident about that lower bound than the upper bound.

owencb 3 Aug 2013 12:02 UTC
6 points
on: Do Earths with slower economic growth have a better chance at FAI?
A key step in your argument is the importance of the parallel/serial distinction. However we already have some reasonably effective institutions for making naturally serial work parallelizable (e.g. peer review), and more are arising. This has allowed new areas of mathematics to be explored pretty quickly. These provide a valve which should mean that extra work on FAI is only slightly less effective than you’d initially think.

You could still think that this was the dominant point if economic growth would increase the speed of both AI and AI safety work to the same degree, but I think this is very unclear. Even granting points (1)-(3) of Paul’s disjunction, it seems like the most important question is the comparative relationship between the elasticity of AI work with economic growth and the elasticity of FAI work with economic growth.

I currently incline to think that in general in a more prosperous society, proportionally more people are likely to fund or be drawn into work which is a luxury in the short term, and AI safety work fits into this category (whereas AI work is more likely to have short term economic benefits, so get funding). However, this question could do with more investigation! In particular I think it’s plausible that the current state of EA movement growth means we’re not in a general position. I haven’t yet thought carefully through all of the ramifications of this.

Nick Beckstead made comments on some related questions in this talk: http://intelligence.org/wp-content/uploads/2013/07/Beckstead-Evaluating-Options-Using-Far-Future-Standards.pdf

By the way, I can guess at some of the source of disagreement here. The language of your post (e.g. “I wish I had more time, not less, in which to work on FAI”) suggests that you imagine that the current people working on FAI will account for a reasonable proportion of the total AI safety work which will be done. I think it more likely that there will be a lot more people working on it before we get AI, so the growth rate dominates.

owencb 5 Aug 2013 14:08 UTC
3 points
on: How to Measure Anything
Thanks, I liked this post.

However, I was initially a bit confused by the section on EVPI. I think it is important, but it could be a lot clearer.

The expected opportunity loss (EOL) for a choice is the probability of the choice being “wrong” times the cost of it being wrong. So for example the EOL if the campaign is approved is $5M × 40% = $2M, and the EOL if the campaign is rejected is $40M × 60% = $24M.

The difference between EOL before and after a measurement is called the “expected value of information” (EVI).

It seems quite unclear what’s meant by “the difference between EOL before and after a measurement” (EOL of which option? is this in expectation?).

I think what must be intended is: your definition is for the EOL of an option. Now the EOL of a choice is the EOL of the option we choose given current beliefs. Then EVI is the expected reduction in EOL upon measurement.

Even this is more confusing than it often needs to be. At heart it’s the expected amount better you’ll do with the information. Sometimes you can factor out the EOL calculation entirely. For example say you’re betting $10 at even odds on a biased coin. You currently think there’s a 70% chance of it landing heads; more precisely you know it was either from a batch which lands heads 60% of the time, or from a batch which lands heads 80% of the time, but these are equiprobable. You could take a measurement to find out which batch it was from. Then you are certain that this measurement will change the EOL, but if you do it carefully the expected gain is equal to the expected loss, so there is no EVI. We could spot this directly because we know that whatever the answer is, we’ll bet on heads.

I think it might be useful to complete your simple example for EVPI (as in, this would have helped me to understand it faster, so may help others too): Currently you’ll run the campaign, with EOL of $2M. With perfect information, you always choose the right option, so you expect the EOL to go down to 0. Hence the EVPI is $2M (this comes from the 40% of the time that the information stops you running the campaign and saving you $5M).

Then in the section on the more advanced model:

In this case, the EVPI turns out to be about $337,000. This means that we shouldn’t spend more than $337,000 to reduce our uncertainty about how many units will be sold as a result of the campaign.

Does this figure come from the book? It doesn’t come from the spreadsheet you linked to. By the way, there’s a mistake in the spreadsheet: when it assumes a uniform distribution it uses different bounds for two different parts of the calculation.

owencb 5 Aug 2013 14:22 UTC
13 points
0
in reply to: NancyLebovitz’s comment on: How to Measure Anything
One possibility is that there are a very large number of things they could measure, most of which have low information value. If they chose randomly we might expect to see an effect like this, and never notice all the low information possibilities they chose not to measure.

I’m not suggesting that they actually do choose randomly, but it might be they chose, say, the easiest to measure, and that these are neither systematically good or bad, so it looks similar to random in terms of the useful information.

owencb 7 Aug 2013 17:59 UTC
8 points
0
in reply to: Eliezer Yudkowsky’s comment on: How to Measure Anything
That seems to be a better fit for the impression of progress. You wouldn’t tend, in retrospect, to call it progress if you realised you’d been going in completely the wrong direction.

This would fit with progress simply be the reduction of work remaining.

owencb 11 Aug 2013 10:57 UTC
3 points
on: Common sense as a prior

I feel my view is weakest in cases where there is a strong upside to disregarding elite common sense, there is little downside, and you’ll find out whether your bet against conventional wisdom was right within a tolerable time limit. Perhaps many crazy-sounding entrepreneurial ideas and scientific hypotheses fit this description.

It doesn’t look too hard to fold these scenarios into the framework. Elite common sense tends to update in light of valid novel arguments, but this updating is slow. So if you’re in possession of a novel argument you often don’t want to wait the years required to find out if it’s accepted; you can stress test it by checking whether elite common sense thinks there’s any plausibility in the argument, and if so let it nudge your beliefs a bit. If there’s a high upside and low downside to acting in the way suggested by the argument, it may not need to change your beliefs by very much before it’s correct to act on the new idea.

Or am I distorting what you intended by the framework here?

owencb 12 Aug 2013 9:22 UTC
3 points
in reply to: Eliezer Yudkowsky’s comment on: Common sense as a prior
I think the default position isn’t that MWI is wrong, but that we don’t currently have enough evidence to decide with high confidence. And you could persuade lots of physicists of that.

owencb 12 Aug 2013 9:30 UTC
10 points
in reply to: Eliezer Yudkowsky’s comment on: Common sense as a prior
Given that there’s some serious fundamental physics we don’t understand yet, I find it hard to persuade myself there’s less than a 1% chance that even the framing of single-world versus many-world interpretations is incoherent.

owencb 12 Aug 2013 10:10 UTC
11 points
on: What Bayesianism taught me

You cannot expect that future evidence will sway you in a particular direction. “For every expectation of evidence, there is an equal and opposite expectation of counterevidence.”

Well … you can have an expected direction, just not if you account for magnitudes.

For example if I’m estimating the bias on a weighted die, and so far I’ve seen ²⁄₁₀ rolls give 6′s, if I roll again I expect most of the time to get a non-6 and revise down my estimate of the probability of a 6; however on the occasions when I do roll a 6 I will revise up my estimate by a larger amount.

Sometimes it’s useful to have this distinction.
What links here?
- What Bayesianism taught me by Tyrrell_McAllister (12 Aug 2013 6:59 UTC; 115 points)

owencb 12 Aug 2013 17:55 UTC
3 points
in reply to: Brian_Tomasik’s comment on: Common sense as a prior
This sounds roughly right to me. Note that there’s are two different things you really want to know about people:

(1) What they believe on the matter;

(2) Who they think is trustworthy on the matter.

Often it seems that (2) is more important, even when you’re looking at people who are deemed trustworthy. If I have a question about lung disease, most people will not have much idea to (1), and recommend doctors for (2). Most doctors will have some idea, and recommend specialists for (2). Specialists are likely to have a pretty good idea for (1), and recommend the top people in their field for (2). These are the people you really want to listen to for (1), if you can, but regular people would not tend to know who they were.

I’m not sure exactly how you should be weighting (1) against (2), but the principle of using both, and following through chains to at least some degree, feels natural.

owencb 13 Aug 2013 10:44 UTC
0 points
in reply to: A1987dM’s comment on: What Bayesianism taught me
I’m well aware of this. My point was that there’s a subtle difference between “direction of the expectation” and “expected direction”.

The expectation of what you’ll think after new evidence has to be the same as you think now, so can’t point in any particular direction. However “direction” is a binary variable (which you might well care about) and this can have a particular non-zero expectation.

I’m being slightly ambiguous as to whether “expected” in “expected direction” is meant to be the technical sense or the common English one. It works fine for either, but to interpret it as an expectation you have to choose an embedding of your binary variable in a continuous space, which I was avoiding because it didn’t seem to add much to the discussion.

owencb 9 Sep 2013 20:28 UTC
0 points
on: High School, Human Capital, Signaling and College Admissions
At least in the UK I’ve seen one conflict of this kind talked about quite frequently: the choice between getting a deep understanding of a subject, and learning how to do well on the exam.

This happens at all levels up to and including graduate degrees, but I have most experience in undergraduate admissions in mathematics, and looking at the qualifications directly before these (A-levels). Many UK universities admit principally on the basis of A-level results, and the exams for these have a certain number of standard style of question. It is possible to learn specifically how to answer this style of question, and that is often a good strategy for people trying to maximise their grade in a given amount of study time. When they get to university, however, and the mathematics gets more abstract, people who have learned the methods in this way tend to suffer in comparison to people who have a deep understanding of the basics, even if they’re less good at complicated questions.

This is further complicated by the fact that Oxford and Cambridge interview students, which is an opportunity to probe how well they understand things which do not fit into the regular mould (it is possible to learn how to do well at these interviews, but in that case the signalling is a little closer to actually building human capital). This means that the best strategies for getting into different universities can vary significantly. So while it seems like a lot of our applicants have over-emphasised learning how to do exam questions, I’m not sure that this is the wrong strategy for them considering that they are applying for multiple universities.

owencb 12 Sep 2013 11:50 UTC
0 points
in reply to: fiddlemath’s comment on: Games for Rationalists
It looks similar to Eleusis, which has the advantage that it just needs playing cards.

Anyhow I’d like to try Zendo. Thanks for mentioning it.

owencb 25 Nov 2013 11:50 UTC
1 point
in reply to: radical_negative_one’s comment on: 2013 Less Wrong Census/Survey
I defected, for similar reasons (without having read the comments, I just assumed that I’d be likely to prefer funds to whoever volunteered to fund this than to a random survey-taker, particularly weighted towards a survey-taker who defected). I’m afraid Yvain’s answer here would not be enough to get me to switch.

If the rest of the $60 prize was to be burned—effectively a wealth redistribution among capital holders—I’d cooperate.

owencb 3 Dec 2013 19:33 UTC
0 points
in reply to: ialdabaoth’s comment on: A critique of effective altruism
There are questions about why we should discount at all, or if we are going to, how to choose an appropriate rate.

But even setting those aside: this isn’t any more of a solution than the version without discounting. They’re similarly reliant on empirical facts about the world (the rate of resource growth); they just give differing answers about how fast that rate needs to be before you should wait rather than cash out.

owencb 4 Dec 2013 14:44 UTC
2 points
in reply to: hyporational’s comment on: According to Dale Carnegie, You Can’t Win an Argument—and He Has a Point
I once went to a talk in which Christopher Zeeman modelled this behaviour using catastrophe theory. I’m not sure you need the mathematics for the thesis, which was (roughly) that arguing for your position pushes people towards it in their underlying beliefs, but also pushes people to be more defensive about their initial beliefs (because it’s a conflict situation). When they go away afterwards and calm down, they may find that they have moved towards your your position … without necessarily remembering the argument as having any part in it.

He claimed to have applied this theory successfully to push a committee he was on, by making a big fuss months before the final decision on the topic was needed.

owencb 7 Dec 2013 10:57 UTC
2 points
on: The Limits of Intelligence and Me: Domain Expertise
This is a good picture of the value of experience.

If the system you’re looking at is simple enough for you to entirely analyse and understand it when encountered, experience won’t add anything. However even for smart humans this restricts you to pretty simple systems!

Experience allows you to observe and make models of the emergent patterns of high-level behaviour of a system, without understanding the details of how they follow from the lower-level details. In many complex systems this high-level behaviour can be very simple. For example the sun rises once every day. You could understand this in terms of the physics of the situation, but you can simply learn the pattern from experience.

owencb 10 Dec 2013 15:04 UTC
0 points
in reply to: incogn’s comment on: Decision Theory FAQ
I think this is a very clear account of the issues with these problems.

I like your explanations of how correct model choice leads to CDT getting it right all the time; similarly it seems correct model choice should let EDT get it right all the time. In this light CDT and EDT are really heuristics for how to make decisions with simplified models.