Coding day in and out on LessWrong 2.0. You can reach me at habryka@lesswrong.com
habryka(Oliver Habryka)
Karma: 16,867
This is true if all the other datapoints are entirely indistinguishable, and the only signal is “good” vs. “bad”. But in practice you would compare / rank the datapoints, and move towards the ones that are better.
Take the backflip example from the human preferences paper: if your only signal was “is this a successful backflip?”, then your argument would apply and it would be pretty hard to learn. But the signal is “is this more like a successful backflip than this other thing?” and this makes learning feasible.
More generally, I feel that the thing I’m arguing against would imply that ML in general is impossible (and esp. the human preferences work), so I think it would help to say explicitly where the disanalogy occurs.
I should note that comparisons is only one reason why the situation isn’t as bad as you say. Another is that even with only non-approved data points to label, you could do things like label “which part” of the plan is non-approved. And with very sophisticated AI systems, you could ask them to predict which plans would be approved/non-approved, even if they don’t have explicit examples, simply by modeling the human approvers very accurately in general.
Well, sure, but that is changing the problem formulation quite a bit. It’s also not particularly obvious that it helps very much, though I do agree it helps. My guess is even with a rank-ordering, you won’t get the 33 bits out of the system in any reasonable amount of time at 10 hours evaluation cost. I do think if you can somehow give more mechanistic and detailed feedback, I feel more optimistic in situations like this, but also feel more pessimistic that we will actually figure out how to do that in situations like this.
More generally, I feel that the thing I’m arguing against would imply that ML in general is impossible (and esp. the human preferences work), so I think it would help to say explicitly where the disanalogy occurs.
I feel like you are arguing for a very strong claim here, which is that “as soon as you have an efficient way of determining whether a problem is solved, and any way of generating a correct solution some very small fraction of the time, you can just build an efficient solution that solves it all of the time”.
This sentence can of course be false without implying that the human preferences work is impossible, so there must be some confusion happening. I am not arguing that this is impossible for all problems, indeed ML has shown that this is indeed quite feasible for a lot of problems, but making the claim that it works for all of them is quite strong, but I also feel like it’s obvious enough that this is very hard or impossible for a large other class of problems (like, e.g. reversing hash functions), and so we shouldn’t assume that we can just do this for an arbitrary problem.
When you say “costly to replace”, this is with respect to what cost function? Do you have in mind the system’s original training objective, or something else?
If you have an original cost function F(x) and an approval cost A(x), you can minimize F(x) + c * A(x), increasing the weight on c until it pays enough attention to A(x). For an appropriate choice of c, this is (approximately) equivalent to asking “Find the most approved policy such that F(x) is below some threshold”—more generally, varying c will trace out the Pareto boundary between F and A.
I was talking about “costly” in terms of computational resources. Like, of course if I have a system that gets the right answer in 1⁄100,000,000 cases, and I have a way to efficiently tell when it gets the right answer, then I can get it to always give me approximately always the right answer by just running it a billion times. But that will also take a billion times longer.
In-practice, I expect most situations where you have the combination of “In one in a billion cases I get the right answer and it costs me $1 to compute an answer” and “I can tell when it gets the right answer”, you won’t get to a point where you can compute a right answer for anything close to $1.
Hmm, I don’t currently find myself very compelled by this argument. Here are some reasons:
In order to even get a single expected datapoint of approval, I need to sample 10^8 examples, which in our current sampling method would take 10^8 * 10 hours, e.g. approximately 100,000 years. I don’t understand how you could do “Learning from Human Preferences” on something this sparse
I feel even beyond that, this still assumes that the reason it is proposing a “good” plan is pure noise, and not the result of any underlying bias that is actually costly to replace. I am not fully sure how to convey my intuitions here, but here is a bad analogy: It seems to me that you can have go-playing-algorithms that lose 99.999% of games against an expert AI, but that doesn’t mean you can distill a competitive AI that wins 50% of games, even though it’s “only 33 bits of information”.
Like, the reason why your AI is losing has a structural reason, and the reason why the AI is proposing consequentialist plans also has a structural reason, so even if we get within 33 bits (which I do think seems unlikely), it’s not clear that you can get substantially beyond that, without drastically worsening the performance of the AI. In this case, it feels like maybe an AI maybe gets lucky and stumbles upon a plan that solves the problem without creating a consequentialist reasoner, but it’s doing that out of mostly luck, not because it actually has a good generator for non-consequentialist-reasoner-generating-plans, and there is no reliable way to always output those plans without actually sampling at least something like 10^4 plans.
The intuition of “as soon as I have an oracle for good vs. bad plans I can chain an optimizer to find good plans” feels far too strong to me in generality, and I feel like I can come up with dozen of counterexamples where this isn’t the case. Like, I feel like… this is literally a substantial part of the P vs. NP problem, and I can’t just assume my algorithm just like finds efficient solution to arbitrary NP-hard problems.
Can you be more concrete about how you would do this? If my method for evaluation is “sit down and think about the consequences of doing this for 10 hours”, I have no idea how I would chain it with an optimizer to find good plans even if they are rare.
Typo:
No real power-seeking tendencies if we only plausibly will specify a negative vector.
Seems like two sentences got merged together.
Should be part of the
contentsfield:{ post(input: {selector: {slug: "the-new-editor"}}) { result { contents { markdown } } } }
I share the sense that something pretty substantial changed with MIRI in ~2017 and that something important got lost when that happened. I share some of the sense that people’s thinking about timelines is confused, though I do think overall pretty short timelines are justified (though mine are on the longer end of what MIRI people tend to think, though much shorter than yours, IIRC). I think you are saying some important things about the funding landscape, and have been pretty sad about the dynamics here as well, though I think the actual situation is pretty messy and some funders are really quite pro-critique, and some others seem to me to be much more optimizing for something like the brand of the EA-coalition.
I found this a very helpful and useful comment, and resonate with various bits of it (I also think I disagree with a good chunk of it, but a lot of it seems right overall).
We did ask Jessica first whether she would want to participate in the experiment. Also, the reason why we want to experiment with additional voting systems is because specifically threads like this seem like they kind of tend to go badly using our current voting system, and switching it out is an attempt at making them go better, in a way that I think Jessica also wants.
The reason why we ordered so many books there is because we didn’t really think about how small Australia is, and had to make the decision on a somewhat short deadline with a bunch of other stuff happening at the same time. I think if we had thought about it for longer, we wouldn’t have made that mistake.
Yeah, I also thought it was pretty interesting. I only thought about it for a few minutes, but it seems interesting enough to give it a shot, IMO.
The post feels like it’s trying pretty hard to point towards an alternative forecasting method, though I also agree it’s not fully succeeding at getting there.
I feel like de-facto the forecasting methodology of people who are actually good at forecasting don’t usually strike me as low-inferential distance, such that it is obvious how to communicate the full methodology. My sense from talking to a number of superforecasters over the years is that they do pretty complicated things, and I don’t feel like the critique of “A critique is only really valid if it provides a whole countermethodology” is a very productive way of engaging with their takes. I feel like I’ve had lots of conversations of the type “X methodology doesn’t work” without someone being able to explain all the details of what they do instead, that were still valuable and helped me model the world, and said meaningful things. Usually the best they can do is something like “well, instead pay attention and build a model of these different factors”, which feels like a bar Eliezer is definitely clearing in this post.
FWIW, “plausible” sounds to me basically the same as “possibly”. So my guess is this is indeed a linguistic thing.
Yeah, let’s also make it a link post then. Some people prefer more prominence, some prefer less, for their cross-posts.
I am also interested in this.
Huh, that’s surprising, if by that he means “no contracts between anyone currently at Leverage and anyone at CEA”. I currently still think it’s the case, though I also don’t see any reason for Geoff to lie here. Maybe there is some technical sense in which there is no contract between Leverage and CEA, but there are contracts between current Leverage employees, who used to work at CEA, and current CEA employees?
One of my ideas for this (when thinking about voting systems in general) is to have a rating that is trivially inconvenient to access. Like, you have a ranking system from F to A, but then you can also hold the A button for 10 seconds, and then award an S rank, and then you can hold the S button for 30 seconds, and award a double S rank, and then hold it for a full minute, and then award a triple S rank.
The only instance I’ve seen of something like this implemented is Medium’s clap system, which allows you to give up to 50 claps, but you do have to click 50 times to actually give those claps.
I think some of these are pretty reasonable points, but I am kind of confused by the following:
This Leverage drama is not important to anyone except a small group of people and does not belong on LW. Perhaps the relatively small group of Bay Area rationalists who are always at the center of these things need to create a separate forum for their own drama. Nobody outside of Berkeley needs to hear about this. This sort of thing gets upvoted because tribal instincts are being activated, not because this is good and ought to be here.
It seems to me that Leverage had a large and broad effect on the Effective Altruism and Rationality communities worldwide, with having organized the 2013-2014 EA Summits, and having provided a substantial fraction of the strategic direction for EAG 2015 and EAG 2016, and then shared multiple staff with the Centre For Effective Altruism until 2019.
This suggests to me that what happened at Leverage clearly had effects that are much broader reaching than “some relatively small group of Bay Area rationalists”. Indeed, I think the Bay Area rationality community wasn’t that affected by all the stuff happening at Leverage, and the effects seemed much more distributed.
Maybe you also think all the EA Summit and EA Global conferences didn’t matter? Which seems like a fine take. Or maybe you think how CEA leadership worked also didn’t matter, which also seems fine. But I do think these both aren’t obvious takes, and I think I disagree with both of them.
I am a bit confused by what we mean by “of the sort you would come across in ML”. Like, this situation, where we are trying to derive an algorithm that solves problems without optimizers, from an algorithm that solves problems with optimizers, is that “the sort of problem you would come across in ML”?. It feels pretty different to me from most usual ML problems.
I also feel like in ML it’s quite hard to actually do this in practice. Like, it’s very easy to tell whether a self-driving car AI has an accident, but not very easy to actually get it to not have any accidents. It’s very easy to tell whether an AI can produce a Harry Potter-level quality novel, but not very easy to get it to produce one. It’s very easy to tell if an AI has successfully hacked some computer system, but very hard to get it to actually do so. I feel like the vast majority of real-world problems we want to solve do not currently follow the rule of “if you can distinguish good answers you can find good answers”. Of course, success in ML has been for the few subproblems where this turned out to be easy, but clearly our prior should be on this not working out, given the vast majority of problems where this turned out to be hard.
(Also, to be clear, I think you are making a good point here, and I am pretty genuinely confused for which kind of problems the thing you are saying does turn out to be true, and appreciate your thoughts here)