Software engineer from Ireland who’s interested in EA and AI safety research.
Stephen McAleese
Karma: 264
My understanding of your argument is that AI progress will slow down in the future because the low-hanging fruit in hardware, software, and research have been exhausted.
Hardware: researchers have scaled models to the point where they cost millions of dollars. At this point, scaling them further is difficult. Moore’s Law is slowing down, making it harder to scale models.
In my opinion, it seems unlikely, but not inconceivable, that training budgets will increase further. It could happen if more useful models result in greater financial returns and investment in a positive feedback loop. Human labor is expensive and creating an artificial replacement could still be profitable even with large training costs. Another possibility is that government investment increases AI training budgets in some kind of AI manhattan project. Though this possibility doesn’t seem likely to me given that most progress has occurred in private companies in recent years.
I’m somewhat less pessimistic about the death of Moore’s Law. Although it’s getting harder to improve chip performance, there is still a strong incentive to improve it. We at least know that it’s possible for improvements to continue because current technology is not near the physical (1).
Software: has improved a lot in recent years. For example, libraries such as HuggingFace have made it much easier to use the latest models. The post argues that research is not bottlenecked by progress in software.
This point seems valid to me. However, better AI-assisted programming tools in the future could increase the rate of software development even more.
Research: transformers, pipeline parallelism, and self-supervised learning have made it possible to train large models with much better performance. The post also says that many of these innovations (e.g. the transformer) are from 2018 or earlier.
New techniques are introduced, they mature and are replaced by newer techniques. For example, progress in CPU speed stagnated and GPUs increased performance dramatically. TPUs have improved on GPUs and we’ll probably see further progress. If this is true, then some of the AI techniques that will be commonplace in several years are probably already under development but not mature enough to be used in mainstream AI. Instead of a global slowdown in AI, I see AI research progress as a series of s-curves.
I can’t imagine how future architectures will be different but the historical trend has always been that new and better techniques replace old ones.
As more money and talent are invested in AI research, progress should accelerate given a fixed difficulty in making progress. Even if the problems become harder to solve, increased talent and financial investment should offset the increase in difficulty. Therefore, it seems like the problems would have to become much harder for AI progress to slow down significantly which doesn’t seem likely to me given how new the field is.
Given that deep learning has only been really popular for about ten years, it seems unlikely that most of the low-hanging fruit have already been extracted, unlike particle physics which has been around for decades and where particle colliders have had diminishing returns.
Overall, I’m more bullish on AI progress in the future than this post and I expect more significant progress to occur.
1: https://en.wikipedia.org/wiki/Landauer%27s_principle
“Not signing up for cryonics—what does that say? That you’ve lost hope in the future. That you’ve lost your will to live.”
It seems like this reason applies more to relatively wealthy people in developed countries. But in poorer countries where many people lack clean water, food, and electricity, cryonics is much more likely to be unaffordable and therefore finances are much more likely to be the dominant factor in one’s decision making.
If cryonics is just about affordable in the US, and most people in the world are poorer than the average US person (1), then cryonics could be unaffordable for much of the earth’s population.
I’m assuming that the target audience of the post was the kind of people who read LessWrong or other people in developed countries. In that case, this argument does not apply.
I have a background in software engineering but I would like to get into AI safety research.
A problem I have had is that I didn’t know whether I should pursue the research scientist or research engineer paths which seem to be quite different. Becoming a research engineer involves lots of work with ML code whereas to become a research engineer you usually have to get a PhD and do some research.
I read in an older document that there was a bottleneck in talent for research scientists and engineers. However, this seems to have changed according to your post and now there seems to be a greater shortage of research engineers than research scientists.
As a result, I am now leaning more in favor of becoming a research engineer. Another advantage is that the research engineer path seems to have a lower barrier to entry.
Summary of “AGI Ruin: A List of Lethalities”
Thanks for providing some feedback.
I’ve incorporated many of the edits you suggested to make it more readable and accessible.
I think the paragraph on pivotal acts is a key part of the original essay so I decided not to remove it.
Instead, I made some significant edits to the paragraph. The edits I made put more emphasis on the definition of what a pivotal act is and I tried to remove as much potentially offensive content as possible. For example, I removed the pivotal act example of ‘burn all GPUs’ and instead described the term more generally as an action that would reduce existential risk.
Software: Focus To-Do
Need: pomodoros and time tracking
Other programs I’ve tried: Harvest, Toggl
I really like Focus To-Do because it tracks how many pomodoros you spend on each task. You can create multiple projects or categories and then create tasks for each category. It has both a web app and a mobile app.
It also allows me to see how many pomodoros I do every day and see trends over time.
Great post. I also fear that it may not be socially acceptable for AI researchers to talk about the long-term effects of AI despite the fact that, because of exponential progress, most of the impact of AI will probably occur in the long term.
I think it’s important that AI safety and considerations related to AGI become mainstream in the field of AI because it could be dangerous if the people building AGI are not safety-conscious.
I want a world where the people building AGI are also safety researchers rather than one where the AI researchers aren’t thinking about safety and the safety people are shouting over the wall and asking them to build safe AI.
This idea reminds me of how software development and operations were combined into the DevOps role in software companies.
I think this post is really interesting because it describes what I think are the most important changes happening in the world today. I may be biased but rapid progress in the field of AI seems to be the signal among the noise today.
The difference between this post and reality (e.g. the news) is that this post seems to distill the signal without adding any distracting noise. It’s not always easy to see which events happening in the world are most important.
But maybe when we look back in several decades everyone in the world will remember history the way this post describes the world today—the pre-AGI era when AGI was imminent. Maybe when we look back this will be this clear signal without any noise.
Relevant quote from Superintelligence:
“Yet let us not lose track of what is globally significant. Through the fog of everyday trivialities, we can perceive—if but dimly—the essential task of our age.”
“Only the result of the verification process will leave the box, and thus we have the same safety properties as before: The system will output a predetermined bit of information or fail to output anything at all. This setup I will call an IP-Oracle AI.”
My understanding of the AI architecture you are proposing is as follows:
1. Humans come up with a mathematical theorem we want to solve.
2. The theorem is submitted to the AI.
3. The AI tries to prove the theorem and submit a correct proof to the theorem checker.
4. The theorem checker outputs 1 if the proof is correct and 0 if it is not.It’s not clear to me that such a system would be very useful for carrying out a pivotal act and it seems like it would only be useful for checking whether a problem is solvable.
For example, we could create a theorem containing specifications for some nanotechnology system and we might get 1 from the box if the AI has found a correct proof and 0 otherwise. But that doesn’t seem very useful to me because it merely proves that the problem can be solved. The problem could still be extremely hard to solve.
Then in the “Applications to AI alignment section” it says:
“In this case, we could ask the IP-Oracle for designs that satisfy these specifications.”
The system described in this quote seems very different from the one-bit system described earlier because it can output designs and seems more like a classic oracle.
This paper is on a similar subject.
Would you mind explaining why the post is written in lowercase?
How Do AI Timelines Affect Existential Risk?
I started with the assumption that alignment progress would have diminishing returns. Then the two other factors I took into account were the increasing relevance of alignment research over time[1] and an increasing number of alignment researchers. My model was that the diminishing returns would be canceled out by the increasing number of researchers and increasing relevance.
It seems like you’re emphasizing the importance of diminishing returns. If diminishing returns are more important than the other two factors, progress would slow down over time. I’m not sure which factors are most influential though I may have been underestimating the importance of diminishing returns.
Quote on how AI could reduce AI risk:
“An aligned ASI could reduce or eliminate natural state risks such as the risk from asteroid strikes, supervolcanoes, or stellar explosions by devising protective technologies or by colonizing space so that civilization would continue if Earth were destroyed.”
I think you’re referring to this quote:
“Total existential risk would probably then stop increasing because the ASI could prevent all further existential risks or because an existential catastrophe would have occurred.”
I think I could have explained this point more. I think existential risk levels would fall to very low levels after an aligned ASI is created by definition:
If the AI were aligned, then the AI itself would be a low source of existential risk.
If it’s also superintelligent, it should be powerful enough to strongly reduce all existential risks.
Those are some good points on cognitively enhanced humans. I don’t think I emphasized the downsides enough. Maybe I need to expand that section.
- ^
Toby Ord calls this decreasing nearsightedness.
Changes:
Added explanation for why an aligned ASI would significantly reduce all existential risks:
″ the ASI could prevent all further existential risks. The reason why follows from its definition: an aligned ASI would itself not be a source of existential risk and since it’s superintelligent, it would be powerful enough to eliminate all further risks.”
Updated graph to show exponentially decreasing model in addition to the linear model.
I agree. The world could be at a higher risk of conflict just before or after the first ASI is created. Though even if there is a fast takeoff, the risk is still there before the takeoff if it is obvious that an ASI is about to be created.
This scenario is described in quite a lot of detail in chapter 5 of Superintelligence:
“Given the extreme security implications of superintelligence, governments would
likely seek to nationalize any project on their territory that they thought close to
achieving a takeoff. A powerful state might also attempt to acquire projects located
in other countries through espionage, theft, kidnapping, bribery, threats, military
conquest, or any other available means.”
What you’re suggesting sounds like differential technological development or the precautionary principle:
“Retard the development of dangerous and harmful technologies, especially ones
that raise the level of existential risk; and accelerate the development of
beneficial technologies, especially those that reduce the existential risks posed by nature or by other technologies.”The problem with this policy is the unilateralist’s curse which says that a single optimistic actor could develop a technology. Technologies such as AI have substantial benefits and risks, the balance is uncertain and the net benefit is perceived differently by different actors. For a technology not to be developed all actors would have to agree not to develop it which would require significant coordination. In the post I describe several factors such as war that might affect the level of global coordination and that it might be wise to slow down AI development by a few years or decades if coordination can’t be achieved since I think AI risk is higher than other risks.
Also, I don’t think that any of these conclusions or recommendations are simple or common sense.
Though some of them may seem simple in hindsight just as a math problem seems simple after one has seen the solution.
The reason why I wrote this post was that I was very confused about the subject. If I thought there was a simple answer, I wouldn’t have written the post or written a much shorter post.
Here is a quote from my research proposal:
“Given that the development of AGI could both increase or decrease existential risk, it is not clear when it should be developed.”
And a quote from the person reviewing my proposal:
“I took a look at your final project proposal. I’m somewhat concerned that your project as currently proposed is intractable, especially for a short research project.”
Not only was the project not simple, the reviewer thought that it was almost impossible to make progress on given the number of factors at play.
Wow, that seems like a lot of content in just a single week. It seems like the output per person in the EA / LW community is high.
You mention that the benefits of college would be credentials and connections. I don’t think it was mentioned in the post that you can also expect to learn a lot during the degree about the fundamentals of Computer Science. Therefore, dropping out involves the opportunity cost of the information not learned in college.
In my opinion, the decision of whether or not to stick with college largely involves a decision to minimize opportunity cost involving a comparison of the opportunities college affords and the opportunities available outside of college. The ratio to be measured is the ratio of opportunities in college to those outside of college. The lower this ratio is, the more reasonable it is to drop out of college and vice versa.