value-conditioned probabilities
Is this a thing or something you just coined? “Probability” has a meaning, I’m totally against using it for things that aren’t that.
I get why the argument is valid for deciding what we should do – and you could argue that’s the only important thing. But it doesn’t make it more likely that our world is robust, which is what the post was claiming. It’s not about probability, it’s about EV.
Here’s the thing though. Sometimes one side IS genuinely correct[/good], and the other side IS genuinely wrong[/evil].
Take the [] out, and this is one of the first things I was thinking upon reading this post. Interestingly, you don’t need to bring conflict vs mistake theory into this at all.
I think this comment should be its own post/open thread comment (probably the latter) and then I’d find it reasonable (dn about others). The tolerance on this site for talking about your pet issue in the context of sth vaguely related is very low.
You’re right, the nature of uncertainty doesn’t actually matter for the EV. My bad.
I’m more uncertain about this one, but I believe that a separate problem with this answer is that it’s an argument about where value comes from, not an argument about what is probable. Let’s suppose 50% of all worlds are fragile and 50% are robust. If most of the things that destroy a world are due to emerging technology, then we still have similar amounts of both worlds around right now (or similar measure on both classes if they’re infinite many, or whatever). So it’s not a reason to suspect a non-fragile world right now.
E.g. if you have a broad distribution over possible worlds, some of which are “fragile” and have 100 things that cut value down by 10%, and some of which are “robust” and don’t, then you get 10,000x more value from the robust worlds. So unless you are a priori pretty confident that you are in a fragile world (or they are 10,000x more valuable, or whatever), the robust worlds will tend to dominate.
This is only true if you assume that there is an equal number of robust and fragile worlds out there, and your uncertainty is strictly random, i.e. you’re uncertain about which of those worlds you live in.
I’m not super confident that our world is fragile, but I suspect that most worlds look the same. I.e., maybe 99.99% of worlds are robust, maybe 99.99% are fragile. If it’s the latter, then I probably live in a fragile world.
I meant “independent person” as in, someone not part of the biggest labs
(Admittedly, not all papers are equally insightful, and maybe OpenAI & DeepMind’s papers are more insightful than average, but I don’t think that’s a strong enough effect to make them account for “most” AI insights.)
Since most researchers are outside of big labs, they’re going to publish more papers. I’m not convinced that means much of anything. I could see usefulness vary by factors of well over 100. Some papers might even negative utility. I think all of the impressive AI’s we’ve seen, without any real exception, have come out of big research labs.
Also, I believe you’re assuming that research will continue to be open. I think it’s more likely that it won’t be, although not 95%.
But ultimately I’m out of my depth on this discussion.
First off, let me say that I could easily be wrong. My belief is both fairly low confidence and not particularly high information.
If that were true, start-ups wouldn’t be a thing, we’d all be using Yahoo Search and Lockheed Martin would be developing the first commercially successful reusable rocket. Hell, it might even make sense to switch to planned economy outright then.
I don’t think any of that follows. Any good idea can be enough for a successful start-up. AGI is extremely narrow compared to the entire space of good ideas.
But why does it matter? Would screaming at the top of your lungs about your new discovery (or the modern equivalent, publishing a research paper on the internet) be the first thing someone who has just gained the key insight does? It certainly would be unwise to.
It doesn’t matter that much, but it makes it a bit harder—it implies that someone outside of the top research labs not only has the insight first, but has it first and then the labs don’t have it for some amount of time.
I actually agree that the “last key insight” is somewhat plausible, but I think even if we assume that, it remains quite unlikely that an independent person has this insight rather than the people who are being paid a ton of money to work on this stuff all day. Especially because even in the insight-model, there could still be some amount of details that need to be figured out after the insight, which might only take a couple of weeks for OpenAI but probably longer for a single person.
To make up a number, I’d put it at < 5% given that the way it goes down is what I would classify under the final-insight model.
In any case, it’s a story, not a prediction, and I’d defend it as plausible in that context. Any story has a thousand assumptions and events that, in sequence, reduce the probability to infinitesimal.
Yeah, I don’t actually disagree. It’s just that, if someone asks “how could an AI actually be dangerous? It’s just on a computer” and I respond by “here look at this cool story someone wrote which answers that question”, they might go “Aha, you think it will be developed on a laptop. This is clearly nonsense, therefore I now dismiss your case entirely”. I think you wanna bend over backwards to not make misleading statements if you argue for the dangers-from-ai-is-a-real-thing side.
You’re of course correct that any scenario with this level of detail is necessarily extremely unlikely, but I think that will be more obvious for other details like how exactly the AI reasons than it is for the above. I don’t see anyone going “aha, the AI reasoned that which is clearly implausible because it’s specific, therefore I won’t take this seriously”.
If I had written this, I would add a disclaimer rather than change the title. The disclaimer could also explain that “paperclips” is a stand-in for any utility function that maximizes for just a particular physical thing.
I think it might be useful to have stories like these, and it’s well written; however:
Plausible A.I. Takeoff Scenario Short Story
,
I am running on a regular laptop computer, in a residential area in Wellington, New Zealand.
These two things are in contradiction. It’s not a plausible scenario if the AGI begins on a laptop. It’s far more likely to begin on the best computer in the world owned by OpenAI or something. Absent a disclaimer, this would be a reason for me not to share this.
Also, typo:
It’s creator must have shut it off.
That might or might not be a better proxy for the kind of overconfidence I’ve been meaning to predict.
The reason why it might not: my formulation relied on the idea that most people will formulate their predictions such that the positive statement corresponds to the smaller subset of positive future space. In that case, even if it’s a < 50% prediction, I would still suspect it’s overconfident. For example:
6) South Korea and Philippines change alliance from USA to China and support it’s 9 dash line claims. Taiwan war with mainland China. 35%
Now I’ve no idea about the substance matter here, but across all such predictions, I predict that they’ll come true less often than the probability indicates. So if we use either of the methods you suggested here, the 35% figure moves upward rather than downward; however I think it should go down.
Using a reasonable calibration method*, the set of predictions made in this thread will receive a better score than the set of those in the previous thread from 10 years ago (80%)
Nonetheless, lowering each confidence stated by a relative 10% (i.e. 70% to 63% etc.) will yield better total calibration (60%)
I don’t know the math for this, but I’m assuming there is one that inputs a set of predictions and their truth values and outputs some number, such that the number measures calibration and doesn’t predictably increase or decrease with more predictions.
I believe that the second one can technically lead to a paradox, but it’s highly unlikely for that to occur.
One of the lessons I draw from this: listen to gwern.
I’m (re-)reading up on absolutely continuous probability spaces right now. The defintion for the expected value I find everywhere is this:
(1):
The way to interpret this formula is that we’re integrating over the target space of rather than the domain, and is a probability density function over the target space of . But this formula seems highly confusing if that is left unsaid ( doesn’t even appear in it – what the heck?). If one begins with a probability density function over a probability space and then wants to compute the expected value of a random variable , I think the formula is:
(2):
It seems utterly daft to me to present (1) without first presenting (2) if the idea is to teach the material in an easily understandable way. Even if one never uses (2) in practice. But this is what seems to be done everywhere – I googled a bunch, checked wikipedia, and dug out an old script, I haven’t found (2) anywhere (I hope it’s even correct). Worse, none of them even explicitly mention that ranges over rather than over after presenting (1). I get each random variable does itself define a probability space where the distribution is automatically over but I don’t think this is a good argument not to present (2). This concept is obviously not going to be trivial to understand.
Stuff like this makes me feel like almost no-one thinks for themselves unless they have to, even in math. I’m interested in whether or not fellow LW-ians share my intuition here.
There seems to be a similar thing going on in linear algebra, where everyone teaches concepts based on the determinant, even though doing it differently makes them far easier. But there it feels more understandable, since you do need to be quite good to see that. This case here just feels like people aren’t even trying to optimize for readability.
That’s a strong steelman of the status quo in cases where random variables are introduced as you describe. I’ll concede that (1) is fine in this case. I’m not sure it applies to cases (lectures) where probability spaces are formally introduced – but maybe it does; maybe other people still don’t think of RVs as functions, even if that’s what they technically are.