I agree that AGI is more omni-use than bioweapons and thus will be harder to get people not to develop and use. I think our prospects look pretty bleak in this scenario, but it’s not completely hopeless.
For human cloning, what I had in mind was a nation cloning its smartest individuals for the purpose of having better science/tech. Think of what the US could have accomplished if they had 10,000 Von Neumanns instead of 1.
Even as a series of worse and worse AGI accidents occur, wih out-of-control AGIs self-replicating around the internet etc., a few people will keep trying to fix the unfixable AGI, seeing this as the only path to get this slow-rolling catastrophe under control (while actually making it worse).
Maybe at this point there would be the political will for a Butlerian Jihad. ;) Or more seriously, a self-imposed ban on AGI similar to the current self-imposed bans on human cloning and biological weapons. I agree this is a long shot given our current experience with climate change, but still, it seems possible. And perhaps the AGI accidents would be more newsworthy and gripping than climate change is, making it easier to rouse the public.
This is awesome, thanks!
So, to check my understanding: You have set up a sort of artificial feedback loop, where there are N overlapping patterns of hills, and each one gets stronger the farther you travel in a particular dimension/direction. So if one or more of these patterns tends systematically to push the ball in the same direction that makes it stronger, you’ll get a feedback loop. And then there is selection between patterns, in the sense that the pattern which pushes the strongest will beat the ones that push more weakly, even if both have feedback loops going.
And then the argument is, even though these feedback loops were artificial / baked in by you, in “natural” search problems there might be a similar situation… what exactly is the reason for this? I guess my confusion is in whether to expect real life problems to have this property where moving in a particular direction strengthens a particular pattern. One way I could see this happening is if the patterns are themselves pretty smart, and are able to sense which directions strengthen them at any given moment. Or it could happen if, by chance, there happens to be a direction and a pattern such that the pattern systematically pushes in that direction and the direction systematically strengthens that pattern. But how likely are these? I don’t know. I guess your case is a case of the second, but it’s rigged a bit, because of how you built in the systematic-strengthening effect.
Am I following, or am I misunderstanding?
Like ignoranceprior said, my AI Impacts post has three intuitive ways of thinking about the results:
Way One: Let’s calculate some examples of prediction patterns that would give you Brier scores like those mentioned above. Suppose you make a bunch of predictions with 80% confidence and you are correct 80% of the time. Then your Brier score would be 0.32, roughly middle of the pack in this tournament. If instead it was 93% confidence correct 93% of the time, your Brier score would be 0.132, very close to the best superforecasters and to GJP’s aggregated forecasts.14 In these examples, you are perfectly calibrated, which helps your score—more realistically you would be imperfectly calibrated and thus would need to be right even more often to get those scores.
Way Two: “An alternative measure of forecast accuracy is the proportion of days on which forecasters’ estimates were on the correct side of 50%. … For all questions in the sample, a chance score was 47%. The mean proportion of days with correct estimates was 75%…”15 According to this chart, the superforecasters were on the right side of 50% almost all the time:16
Way Three: “Across all four years of the tournament, superforecasters looking out three hundred days were more accurate than regular forecasters looking out one hundred days.”17 (Bear in mind, this wouldn’t necessarily hold for a different genre of questions. For example, information about the weather decays in days, while information about the climate lasts for decades or more.)
I’m glad you made that disclaimer. In our community—LW—the ratio of excitement/childishness bias is probably unusually high, perhaps even high enough that we need to be on higher guard against excitement. But in the wider community of “smart and/or important people thinking about AI” it seems pretty clear that childishness bias is much much stronger.
Like, I agree that Michael Wulfson is a person for whom the excitement bias was stronger, judging by his story. But even for him, the childishness bias was super strong too, and eventually won out. Most people are not like him; most people will find the childishness bias / excitement bias ratio much higher than he did.
Indeed I’d go so far as to say if this is the best case we can think of of excitement bias being stronger, that’s pretty good evidence that in fact childishness bias is usually way stronger.
I’m not sure this is strong counterevidence, because it can be interpreted as a case of childishness bias overcoming excitement bias:
But there were also countervailing effects in my mind, leading away from the god scenario. The strongest was the outlandishness of it all. I had always been dismissive of ideas that seem like doomsday theories, so I wasn’t automatically comfortable giving the god scenario credence in my mind. I was hesitant to introduce the idea to people who I thought might draw negative conclusions about my judgement.
I like your point #2; I should think more about how the 30 year number changes with size. Obviously it’s smaller for bigger entities and bigger for smaller entities, but how much? E.g. if we teleported 2020 Estonia back into 1920, would it be able to take over the world? Probably. What about 1970 though? Less clear.
Military power isn’t what I’m getting at either, at least not if measured in the way that would result in AI companies having little of it. Cortez had, maybe, 1⁄10,000th of the military power of Mexico when he got started. At least if you measure in ways like “What would happen if X fought Y.” Probably 1⁄10,000th of Mexico’s military could have defeated Cortez’ initial band.
If we try to model Cortez’ takeover as him having more of some metric than all of Mexico had, then presumably Spain had several orders of magnitude more of that metric than Cortez did, and Western Europe as a whole had at least an order of magnitude more than that. So Western Europe had *many* orders of magnitude more of this stuff, whatever it is, than Mexico, even though Mexico had a similar population and GDP. So they must have been growing much faster than Mexico for quite some time to build up such a lead—and this was before the industrial revolution! More generally, this metric that is used for predicting takeovers seems to be the sort of thing that can grow and/or shrink orders of magnitude very quickly, as illustrated by the various cases throughout history of small groups from backwater regions taking over rich empires.
(Warning: I’m pulling these claims out of my ass, I’m not a historian, I might be totally wrong. I should look up these numbers.)
Glad to hear you are interested! Well, I’m in US Eastern time, but timing can be flexible. If we have enough people, perhaps Blog Post Day will effectively be longer than 24 hours. I’m thinking it would be a relatively casual affair, with people dropping in or out as they see fit.
Thanks in advance to those who join me on this venture! And those who give advice, criticism, etc.
If we get sufficient interest, it might be good to organize local meetups. Anyone else in North Carolina want to meet up with me for this?
I was thinking of an initially large country growing fast via AI, yes. Still counts; it is soft takeoff leading to DSA. However I am also making much stronger claims than that—I think it could happen with a corporation or rogue AGI.
I don’t think annual income is at all a good measure of how close an entity is to taking over the world. When Cortez landed in Mexico he had less than 1⁄100,000th of the income, population, etc. of the region, yet he ruled the whole place three years later. Then a few years after that Pizarro repeated the feat in Peru, good evidence that it wasn’t just an amazing streak of luck.
I nominate this thing johnswentworth did. In addition to the reasons he gives, I’ll add that being able to learn on your own, quickly, seems like a good skill to have, and related to (though maybe not the same thing as) rationality.
I think I find your overall conclusion plausible, but I think your argument for it in places was dubious:
But again, even if you assume I’m wrong, that still leave us with universities that struggle to optimize for 2, 3 and maybe 4, losing out on 5 in the process.
One could instead interpret the situation as: Universities are optimizing hard for 5, and as a result they are understandably losing out on 2, 3, and 4 in the process.
Indeed, I think there is something to be said for this. A few years ago I half-jokingly wrote a paper titled “Kallipolis, USA,” in which I argue that the present-day USA is in fact Plato’s ideal state.
A big part of my argument was the way in which the university system works. In particular, (in conjunction with the rest of society) it seems to be optimizing pretty hard to get people to “follow their passion,” and in particular by forcing everyone to go to college and take gen-ed requirements arguably the system is doing the best it can to scout and recruit people who are suited to the priesthood/academia.
How necessary is it that there be an explicit side-channel? Could you not get the same results in the standard situation in which an agent is selecting actions on the basis of expected utility?
Ah, that does help, thanks. In my words: A search process that is vulnerable to local minima doesn’t necessarily contain a secondary search process, because it might not be systematically comparing local minima and choosing between them according to some criteria. It just goes for the first one it falls for, or maybe slightly more nuanced, the first sufficiently big one it falls for.
By contrast, in the ball rolling example you gave, the walls/ridges were competing with each other, such that the “best” one (or something like that) would be systematically selected by the ball, rather than just the first one or the first-sufficiently-big one.
So in that case, looking over your list again...
OK, I think I see how organic life arising from chemistry is an example of a secondary search process. It’s not just a local minima that chemistry found itself in, it’s a big competition between different kinds of local minima. And now I think I see how this would go in the other examples too. As I originally said in my top-level comment, I’m not sure this applies to the example I brought up, actually. Would the “Insert my name as the author of all useful heuristics” heuristic be outcompeted by something else eventually, or not? I bet not, which indicates that it’s a “mere” local minima and not one that is part of a broader secondary search process.
I should add though that I haven’t systematically examined these graphs yet, so it’s possible I’m just missing something—e.g. it occurs to me right now that maybe some of these graphs I saw were really logistic functions rather than hyperbolic or exponential-until-you-hit-limits. I should make some more and look at them more carefully.
Yes, thanks! I mostly agree with that assessment,* though as an aside I have a beef with the implication that Bostrom, Yudkowsky, etc. expect discontinuities. That beef is with Paul Christiano, not you. :)
So far the biggest update this has been for me, I think, is that it seems to have shown that it’s quite possible to get an intelligence explosion even without economic feedback loops. Like, even with a fixed compute/money budget—or even with a fixed number of scientists and fixed amount of research funding—we could get singularity. At least in principle. This is weird because in practice I am pretty sure I remember reading that the growth we’ve seen so far can be best explained via an economic feedback loop: Better technology allows for bigger population and economy which allows for more scientists and funding which allows for better technology. So I’m a bit confused, I must say—my model is giving me results I would have predicted wouldn’t happen.
*There have been a few cases where the growth didn’t look hyperbolic, but rather like a steady exponential trend that then turns into a singularity. World GDP, by contrast, has what looks like at least three exponential trends in it, such that it is more parsimonious to model it as hyperbolic growth. I think.
Hmmm, this doesn’t work to distinguish the two for me. Couldn’t you say a local minima involves a secondary optimizing search process that has that minima as its objective? To use your ball analogy, what exactly is the difference between these twisty demon hills and a simple crater-shaped pit? (Or, what is the difference between a search process that is vulnerable to twisty demon hills and one which is vulnerable to pits?)