Let me address the risks identified in your article:
An end to Moore’s law. The only part I agree with is that parallel software is fundamentally more difficult, although I think we would disagree with how much more difficult, given the right tools. We have plenty of experience in scaling parallel software stacks in data centers, giving rise to solutions like NoSQL, message queues, and software transactional memory. These tools are still crude, but if you look at what’s being done in academia with automated parallelization and simple concurrency frameworks, the prospects look bright indeed. The concurrency problem will be solved (and indeed, pretty much is solved by organizations like Google, although their tools are not public). There is no reason to suppose that Moore’s law formulated as computations per dollar will not continue until physical limits are reached, which is still a very long way off, indeed, and there are tools on the horizon that will slay the concurrency dragon.
(To be clear, concurrency adds a suppressing term to Moore’s law expressed as observed performance per dollar, due to performance no longer scaling linearly without limit. But it’s still recognizably an exponential.)
Depletion of low-hanging fruit. What fruit is low-hanging depends on what capabilities you have. To stretch this analogy, as hardware capacity increases, you grow taller. In the 80′s chess playing was a very difficult research problem. Today it would definitely be low hanging fruit if it wasn’t already claimed by Deep Blue. It was new hardware and software capabilities which shifted the playing field, and there is no reason to expect this won’t continue to happen in the future so long as Moore’s law holds.
Societal collapse. Maybe, but this wouldn’t affect the “basement hacker” hard takeoff scenario, and may even facilitate it in the same way that the economic collapse of 2008 gave a boost to bitcoin which was released shortly thereafter. I think you’d have to posit total global societal collapse—bringing an end to Moore’s law—before you start affecting the probability of the basement hacker scenario. You could also come up with other scenarios whose timelines would be unaffected, such as AI for national defense / surveillance which may receive more funding in the case of societal collapse.
Disinclination. I’d love to see analogous examples of this from other industries, where there is a revolutionary change around the corner that everyone knows about but no one pursues because what they have is “good enough.” I suspect you won’t find it; it goes against the grain of entrepreneurial capitalism and human nature.
In my own judgement, none of these negative scenarios have much probability mass. My own forecasting is more limited by unknown unknowns, which practical AGI research will uncover. Your positive factors (cognitive neuroscience breakthroughs, human enhancement, funding growth, etc.) are reasonable, so compared to me your judgement is more pessimistic.
I think that your own analysis is somewhat contradictory. You can either trust expert opinions or not, and your 2065 number is the midpoint of acceptable expert opinion. In the case of AGI, I am very skeptical of expert opinion, as there are very few experts in computer science or AI which are familiar with the very specific challenges of AGI, yet it is a topic that everyone has an opinion on if you ask them. I recognize that this excludes my own opinion from consideration as well, but nevertheless here it is: we are more likely than not to experience a hard-takeoff scenario in the 2020′s.
Why am I confident of this? Because we understand the problem. Read the proceeds of the sometimes-annual AGI conference. There are a number of AGI architectures which no longer make ridiculous predictions of “emergent” features, which was once a clear sign of things we didn’t understand. This was not the case just 10 years ago. You can open any psychological development textbook and match up descriptions of human thinking to operating modes of, say, the OpenCog Prime architecture. We simply need to do the work to implement that architecture—which only exists on paper at this point—and solve the unknown unknowns which pop up along the way.
But why the 2020′s, specifically? Well with the resources we have I think we could implement one of these AGI architectures in about 10 years with current resources, and at that time computational resources will be developed enough to run it on commercial datacenter or high-end consumer hardware. This is simply my informed judgement as a systems engineer and my understanding of the complexity of the as-yet unimplemented portions of the OpenCog Prime architecture (you could substitute your own favorite architecture, however). Just look at the OpenCog Prime documents, look at the current state of OpenCog implementation, look at the current level of funding and available developers, and do normal, everyday project planning analysis. I predict you will arrive at a similar number.
Add 50% to that timeline, a common software engineering rule-of-thumb, and you get 15 years. However I predict 2025 / 10 years because that judgement was based on current efforts, but we are way under capacity in terms of AGI research—researcher time and money is the limiting factor. By comparison, with infinite budget we could do it in a couple of years. I think there’s a good chance that an “AI sputnik” moment in the next couple of years could change the funding outlook, accelerating that schedule significantly, and I allocate a fair chunk of probability mass to that happening.
EDIT: So my question to you, Luke and MIRI, is: if you take at face value my prediction of hard-takeoff in the 2020′s, how does this change MIRI’s priorities? Would you being doing things differently if you expected a hard-takeoff in ~2025? How would it be different?
The only part I agree with is that parallel software is fundamentally more difficult
What about the linked paper on dark silicon?
we are more likely than not to experience a hard-takeoff scenario in the 2020′s
Wow. I really need to figure out a practical way to make apocalypse bets. Maybe I can get longbets.org to add that feature.
Just look at the OpenCog Prime documents, look at the current state of OpenCog implementation, look at the current level of funding and available developers, and do normal, everyday project planning analysis.
OpenCog looks pretty unpromising to me. What’s the #1 document I should read that has the best chance of changing my mind, even if it’s only a tiny chance?
Would you [be] doing things differently if you expected a hard-takeoff in ~2025? How would it be different?
For example if we had believed this 1.5 years ago, I suppose we would have (1) scuttled longer-term investments like CFAR and “strategic” research (e.g. AI forecasting, intelligence explosion microeconomics), (2) tried our best to build up a persuasive case that AI was very near, so we could present it to all the wealthy individuals we know, (3) used that case and whatever money we could quickly raise to hire all the best mathematicians willing to do FAI work, and (4) done almost nothing else but in-house FAI research. (Maybe Eliezer and others have different ideas about what we’d do if we thought AI was coming in ~2025; I don’t know.)
As far as I can tell, it completely ignores 3D chip design, or multi-chip solutions in the interim. If there are power limits on the number of transistors in a single chip, then expect to have more and more, smaller and smaller chips, or a completely different chip design which invalidates the assumptions underlying the dark silicon paper (3D chips, for example).
Generalizing, this is a very common category of paper. It identifies some hard wall that prevents continuation of Moore’s law. The abstract and conclusion contains much doom and gloom. In practice, it merely spurs creative thinking, resulting in a modification of the manufacturing or design process which invalidates the assumptions which led to the scaling limit. (In this case, the assumption that to get higher application performance, you need more transitors or smaller feature sizes, or that chips consist of a 2D grid of silicon transistors.)
OpenCog looks pretty unpromising to me. What’s the #1 document I should read that has the best chance of changing my mind, even if it’s only a tiny chance?
I read a preprint of Ben Goertzel’s upcoming “Building Better Minds” book, most of which is also available as part of the OpenCog wiki. When I said the “OpenCog Prime documents,” this is what I was referring to. But it’s not structured as an argument for this approach, and as a 1,000 page document it’s hard to recommend as an introduction if you’re uncertain about its value. My own confidence in OpenCog Prime comes from doing my own analysis of its capabilities and weaknesses as I read this document (being unconvinced beforehand), and my own back-of-the-envelope calculations of where it could go, with the proper hardware and software optimizations. There is a high-level overview of CogPrime on the OpenCog wiki. It’s a little outdated and incomplete, but the overall structure is still the same.
Also you might be interested in Ben’s write-up of the rise and fall of WebMind (which became Novamente, which became OpenCog), as it provides a good context for why OpenCog is structured the way that it is. That is to say how what problems they tried to solve, what difficulties they encountered, and why they ended up where they are in the design space of AGI minds, and why they are confident in that design. It’s an interesting people-story anyhow: “Waking up from the economy of dreams”
OpenCog gets a lot of flack for being an “everything and the kitchen sink” approach. This is unfair and undue criticism, I believe. Rather I would say that the CogPrime architecture recognizes that human cognition is complex, and that while it is reducible, an accurate model would nevertheless still contain a lot of complexity. For example, there are many different kinds of memory (perceptual, declarative, episodic, procedural, etc.), and therefore it makes sense to have different mechanisms for handling each of these memory systems. Maybe in the future some of them can be theoretically unified, but that doesn’t mean it wouldn’t still be beneficial to implement them separately—the model is not the territory.
However what CogPrime does do which doesn’t get harped enough is provide a base representation format (hypergraphs) which are capable of encoding the entire architecture in the same homoiconic medium. A good description of why this is important is the following blog post, “Why hypergraphs?”
For example if we had believed this 1.5 years ago, I suppose we would have (1) scuttled longer-term investments like CFAR and “strategic” research (e.g. AI forecasting, intelligence explosion microeconomics),
I hope y’all wouldn’t have scrapped the most excellent HPMoR ;)
(2) tried our best to build up a persuasive case that AI was very near, so we could present it to all the wealthy individuals we know, (3) used that case and whatever money we could quickly raise to hire all the best mathematicians willing to do FAI work, and (4) done almost nothing else but in-house FAI research. (Maybe Eliezer and others have different ideas about what we’d do if we thought AI was coming in ~2025; I don’t know.)
I hope that you (again, addressing both Luke and MIRI) take a look at existing, active AGI projects, and evaluate for each of them (1) if it could possibly lead to a hard-takeoff scenario, (2) what the timeframe for a hard-takeoff would be^1, and (3) enumeration of FAI risk factors and possible mitigation strategies. And of course, publish the results of this study.
^1: Both your own analysis and the implementor’s estimation—but be sure to ask them when the necessary features X, Y, and Z will be implemented, not about the hard-takeoff specifically, so as to not bias the data.
Let me address the risks identified in your article:
An end to Moore’s law. The only part I agree with is that parallel software is fundamentally more difficult, although I think we would disagree with how much more difficult, given the right tools. We have plenty of experience in scaling parallel software stacks in data centers, giving rise to solutions like NoSQL, message queues, and software transactional memory. These tools are still crude, but if you look at what’s being done in academia with automated parallelization and simple concurrency frameworks, the prospects look bright indeed. The concurrency problem will be solved (and indeed, pretty much is solved by organizations like Google, although their tools are not public). There is no reason to suppose that Moore’s law formulated as computations per dollar will not continue until physical limits are reached, which is still a very long way off, indeed, and there are tools on the horizon that will slay the concurrency dragon.
(To be clear, concurrency adds a suppressing term to Moore’s law expressed as observed performance per dollar, due to performance no longer scaling linearly without limit. But it’s still recognizably an exponential.)
Depletion of low-hanging fruit. What fruit is low-hanging depends on what capabilities you have. To stretch this analogy, as hardware capacity increases, you grow taller. In the 80′s chess playing was a very difficult research problem. Today it would definitely be low hanging fruit if it wasn’t already claimed by Deep Blue. It was new hardware and software capabilities which shifted the playing field, and there is no reason to expect this won’t continue to happen in the future so long as Moore’s law holds.
Societal collapse. Maybe, but this wouldn’t affect the “basement hacker” hard takeoff scenario, and may even facilitate it in the same way that the economic collapse of 2008 gave a boost to bitcoin which was released shortly thereafter. I think you’d have to posit total global societal collapse—bringing an end to Moore’s law—before you start affecting the probability of the basement hacker scenario. You could also come up with other scenarios whose timelines would be unaffected, such as AI for national defense / surveillance which may receive more funding in the case of societal collapse.
Disinclination. I’d love to see analogous examples of this from other industries, where there is a revolutionary change around the corner that everyone knows about but no one pursues because what they have is “good enough.” I suspect you won’t find it; it goes against the grain of entrepreneurial capitalism and human nature.
In my own judgement, none of these negative scenarios have much probability mass. My own forecasting is more limited by unknown unknowns, which practical AGI research will uncover. Your positive factors (cognitive neuroscience breakthroughs, human enhancement, funding growth, etc.) are reasonable, so compared to me your judgement is more pessimistic.
I think that your own analysis is somewhat contradictory. You can either trust expert opinions or not, and your 2065 number is the midpoint of acceptable expert opinion. In the case of AGI, I am very skeptical of expert opinion, as there are very few experts in computer science or AI which are familiar with the very specific challenges of AGI, yet it is a topic that everyone has an opinion on if you ask them. I recognize that this excludes my own opinion from consideration as well, but nevertheless here it is: we are more likely than not to experience a hard-takeoff scenario in the 2020′s.
Why am I confident of this? Because we understand the problem. Read the proceeds of the sometimes-annual AGI conference. There are a number of AGI architectures which no longer make ridiculous predictions of “emergent” features, which was once a clear sign of things we didn’t understand. This was not the case just 10 years ago. You can open any psychological development textbook and match up descriptions of human thinking to operating modes of, say, the OpenCog Prime architecture. We simply need to do the work to implement that architecture—which only exists on paper at this point—and solve the unknown unknowns which pop up along the way.
But why the 2020′s, specifically? Well with the resources we have I think we could implement one of these AGI architectures in about 10 years with current resources, and at that time computational resources will be developed enough to run it on commercial datacenter or high-end consumer hardware. This is simply my informed judgement as a systems engineer and my understanding of the complexity of the as-yet unimplemented portions of the OpenCog Prime architecture (you could substitute your own favorite architecture, however). Just look at the OpenCog Prime documents, look at the current state of OpenCog implementation, look at the current level of funding and available developers, and do normal, everyday project planning analysis. I predict you will arrive at a similar number.
Add 50% to that timeline, a common software engineering rule-of-thumb, and you get 15 years. However I predict 2025 / 10 years because that judgement was based on current efforts, but we are way under capacity in terms of AGI research—researcher time and money is the limiting factor. By comparison, with infinite budget we could do it in a couple of years. I think there’s a good chance that an “AI sputnik” moment in the next couple of years could change the funding outlook, accelerating that schedule significantly, and I allocate a fair chunk of probability mass to that happening.
EDIT: So my question to you, Luke and MIRI, is: if you take at face value my prediction of hard-takeoff in the 2020′s, how does this change MIRI’s priorities? Would you being doing things differently if you expected a hard-takeoff in ~2025? How would it be different?
Thanks for your detailed thoughts!
What about the linked paper on dark silicon?
Wow. I really need to figure out a practical way to make apocalypse bets. Maybe I can get longbets.org to add that feature.
OpenCog looks pretty unpromising to me. What’s the #1 document I should read that has the best chance of changing my mind, even if it’s only a tiny chance?
For example if we had believed this 1.5 years ago, I suppose we would have (1) scuttled longer-term investments like CFAR and “strategic” research (e.g. AI forecasting, intelligence explosion microeconomics), (2) tried our best to build up a persuasive case that AI was very near, so we could present it to all the wealthy individuals we know, (3) used that case and whatever money we could quickly raise to hire all the best mathematicians willing to do FAI work, and (4) done almost nothing else but in-house FAI research. (Maybe Eliezer and others have different ideas about what we’d do if we thought AI was coming in ~2025; I don’t know.)
As far as I can tell, it completely ignores 3D chip design, or multi-chip solutions in the interim. If there are power limits on the number of transistors in a single chip, then expect to have more and more, smaller and smaller chips, or a completely different chip design which invalidates the assumptions underlying the dark silicon paper (3D chips, for example).
Generalizing, this is a very common category of paper. It identifies some hard wall that prevents continuation of Moore’s law. The abstract and conclusion contains much doom and gloom. In practice, it merely spurs creative thinking, resulting in a modification of the manufacturing or design process which invalidates the assumptions which led to the scaling limit. (In this case, the assumption that to get higher application performance, you need more transitors or smaller feature sizes, or that chips consist of a 2D grid of silicon transistors.)
I read a preprint of Ben Goertzel’s upcoming “Building Better Minds” book, most of which is also available as part of the OpenCog wiki. When I said the “OpenCog Prime documents,” this is what I was referring to. But it’s not structured as an argument for this approach, and as a 1,000 page document it’s hard to recommend as an introduction if you’re uncertain about its value. My own confidence in OpenCog Prime comes from doing my own analysis of its capabilities and weaknesses as I read this document (being unconvinced beforehand), and my own back-of-the-envelope calculations of where it could go, with the proper hardware and software optimizations. There is a high-level overview of CogPrime on the OpenCog wiki. It’s a little outdated and incomplete, but the overall structure is still the same.
Also you might be interested in Ben’s write-up of the rise and fall of WebMind (which became Novamente, which became OpenCog), as it provides a good context for why OpenCog is structured the way that it is. That is to say how what problems they tried to solve, what difficulties they encountered, and why they ended up where they are in the design space of AGI minds, and why they are confident in that design. It’s an interesting people-story anyhow: “Waking up from the economy of dreams”
OpenCog gets a lot of flack for being an “everything and the kitchen sink” approach. This is unfair and undue criticism, I believe. Rather I would say that the CogPrime architecture recognizes that human cognition is complex, and that while it is reducible, an accurate model would nevertheless still contain a lot of complexity. For example, there are many different kinds of memory (perceptual, declarative, episodic, procedural, etc.), and therefore it makes sense to have different mechanisms for handling each of these memory systems. Maybe in the future some of them can be theoretically unified, but that doesn’t mean it wouldn’t still be beneficial to implement them separately—the model is not the territory.
However what CogPrime does do which doesn’t get harped enough is provide a base representation format (hypergraphs) which are capable of encoding the entire architecture in the same homoiconic medium. A good description of why this is important is the following blog post, “Why hypergraphs?”
I hope y’all wouldn’t have scrapped the most excellent HPMoR ;)
I hope that you (again, addressing both Luke and MIRI) take a look at existing, active AGI projects, and evaluate for each of them (1) if it could possibly lead to a hard-takeoff scenario, (2) what the timeframe for a hard-takeoff would be^1, and (3) enumeration of FAI risk factors and possible mitigation strategies. And of course, publish the results of this study.
^1: Both your own analysis and the implementor’s estimation—but be sure to ask them when the necessary features X, Y, and Z will be implemented, not about the hard-takeoff specifically, so as to not bias the data.
Thanks for the reading links.