Personal website: https://andrewtmckenzie.com/
Andy_McKenzie
Karma: 1,349
Cryonics p(success) estimates are only weakly associated with interest in pursuing cryonics in the LW 2023 Survey
Why wasn’t preservation with the goal of potential future revival started earlier in history?
Very high-effort, comprehensive post. Any interest in making some of your predictions into markets on Manifold or some other prediction market website? Might help get some quantifications.
A simple solution is to just make doctors/hospitals liable for harm which occurs under their watch, period. Do not give them an out involving performative tests which don’t actually reduce harm, or the like. If doctors/hospitals are just generally liable for harm, then they’re incentivized to actually reduce it.
Can you explain more what you actually mean by this? Do you mean if someone comes into the hospital and dies, the doctors are responsible, regardless of why they died? If you mean that we figure out whether the doctors are responsible for whether the patient died, then we get back to whether they have done everything to prevent it, and one of these things might be ordering lab tests to better figure out the diagnosis, and then it seems we’re back to the original problem i.e. the status quo. Just not understanding what you mean.
Out of curiosity, what makes you think that the initial freezing process causes too much information loss?
I agree with most of this post, but it doesn’t seem to address the possibility of whole brain emulation. However, many/(?most) would argue this is unlikely to play a major role because AGI will come first.
Thanks so much for putting this together Mati! If people are interested in cryonics/brain preservation and would like to learn about (my perspective on) the field from a research perspective, please feel free to reach out to me: https://andrewtmckenzie.com/
I also have some external links/essays available here: https://brainpreservation.github.io/
It seems to me like your model is not necessarily taking into account technical debt sufficiently enough. https://neurobiology.substack.com/p/technical-debt-probably-the-main-roadblack-in-applying-machine-learning-to-medicine
It seems to me like this is the main thing that will slow down the extent to which foundation models can consistently beat newly trained specialized models.
Anecdotally, I know several people who don’t like to use chatgpt because its training cuts off in 2021. This seems like a form of technical debt.
I guess it depends on how easily adaptable foundation models are.
Sounds good, can’t find your email address, DM’d you.
Those sound good to me! I donated to your charity (the Animal Welfare Fund) to finalize it. Lmk if you want me to email you the receipt. Here’s the manifold market:
Bet
Andy will donate $50 to a charity of Daniel’s choice now.
If, by January 2027, there is not a report from a reputable source confirming that at least three companies, that would previously have relied upon programmers, and meet a defined level of success, are being run without the need for human programmers, due to the independent capabilities of an AI developed by OpenAI or another AI organization, then Daniel will donate $100, adjusted for inflation as of June 2023, to a charity of Andy’s choice.
Terms
Reputable Source: For the purpose of this bet, reputable sources include MIT Technology Review, Nature News, The Wall Street Journal, The New York Times, Wired, The Guardian, or TechCrunch, or similar publications of recognized journalistic professionalism. Personal blogs, social media sites, or tweets are excluded.
AI’s Capabilities: The AI must be capable of independently performing the full range of tasks typically carried out by a programmer, including but not limited to writing, debugging, maintaining code, and designing system architecture.
Equivalent Roles: Roles that involve tasks requiring comparable technical skills and knowledge to a programmer, such as maintaining codebases, approving code produced by AI, or prompting the AI with specific instructions about what code to write.
Level of Success: The companies must be generating a minimum annual revenue of $10 million (or likely generating this amount of revenue if it is not public knowledge).
Report: A single, substantive article or claim in one of the defined reputable sources that verifies the defined conditions.
AI Organization: An institution or entity recognized for conducting research in AI or developing AI technologies. This could include academic institutions, commercial entities, or government agencies.
Inflation Adjustment: The donation will be an equivalent amount of money as $100 as of June 2023, adjusted for inflation based on https://www.bls.gov/data/inflation_calculator.htm.
Regulatory Impact: In January 2027, Andy will use his best judgment to decide whether the conditions of the bet would have been met in the absence of any government regulation restricting or banning the types of AI that would have otherwise replaced programmers.
Sounds good, I’m happy with that arrangement once we get these details figured out.
Regarding the human programmer formality, it seems like business owners would have to be really incompetent for this to be a factor. Plenty of managers have coding experience. If the programmers aren’t doing anything useful then they will be let go or new companies will start that don’t have them. They are a huge expense. I’m inclined to not include this since it’s an ambiguity that seems implausible to me.
Regarding the potential ban by the government, I wasn’t really thinking of that as a possible option. What kind of ban do you have in mind? I imagine that regulation of AI is very likely by then, so if the automation of all programmers hasn’t happened by Jan 2027, it seems very easy to argue that it would have happened in the absence of the regulation.
Regarding these and a few of the other ambiguous things, one way we could do this is that you and I could just agree on it in Jan 2027. Otherwise, the bet resolves N/A and you don’t donate anything. This could make it an interesting Manifold question because it’s a bit adversarial. This way, we could also get rid of the requirement for it to be reported by a reputable source, which is going to be tricky to determine.
Understandable. How about this?
Bet
Andy will donate $50 to a charity of Daniel’s choice now.
If, by January 2027, there is not a report from a reputable source confirming that at least three companies, that would previously have relied upon programmers, and meet a defined level of success, are being run without the need for human programmers, due to the independent capabilities of an AI developed by OpenAI or another AI organization, then Daniel will donate $100, adjusted for inflation as of June 2023, to a charity of Andy’s choice.
Terms
Reputable Source: For the purpose of this bet, reputable sources include MIT Technology Review, Nature News, The Wall Street Journal, The New York Times, Wired, The Guardian, or TechCrunch, or similar publications of recognized journalistic professionalism. Personal blogs, social media sites, or tweets are excluded.
AI’s Capabilities: The AI must be capable of independently performing the full range of tasks typically carried out by a programmer, including but not limited to writing, debugging, maintaining code, and designing system architecture.
Equivalent Roles: Roles that involve tasks requiring comparable technical skills and knowledge to a programmer, such as maintaining codebases, approving code produced by AI, or prompting the AI with specific instructions about what code to write.
Level of Success: The companies must be generating a minimum annual revenue of $10 million (or likely generating this amount of revenue if it is not public knowledge).
Report: A single, substantive article or claim in one of the defined reputable sources that verifies the defined conditions.
AI Organization: An institution or entity recognized for conducting research in AI or developing AI technologies. This could include academic institutions, commercial entities, or government agencies.
Inflation Adjustment: The donation will be an equivalent amount of money as $100 as of June 2023, adjusted for inflation based on https://www.bls.gov/data/inflation_calculator.htm.
I guess that there might be some disagreements in these terms, so I’d be curious to hear your suggested improvements.
Caveat: I don’t have much disposable money right now, so it’s not much money, but perhaps this is still interesting as a marker of our beliefs. Totally ok if it’s not enough money to be worth it to you.
I’m wondering if we could make this into a bet. If by remote workers we include programmers, then I’d be willing to bet that GPT-5/6, depending upon what that means (might be easier to say the top LLMs or other models trained by anyone by 2026?) will not be able to replace them.
These curves are due to temporary plateaus, not permanent ones. Moore’s law is an example of a constraint that seems likely to plateau. I’m talking about takeoff speeds, not eventual capabilities with no resource limitations, which I agree would be quite high and I have little idea of how to estimate (there will probably still be some constraints, like within-system communication constraints).
Does anyone know of any AI-related predictions by Hinton?
Here’s the only one I know of—“People should stop training radiologists now. It’s just completely obvious within five years deep learning is going to do better than radiologists because it can get a lot more experience. And it might be ten years but we got plenty of radiologists already.” − 2016, slightly paraphrased
This seems like still a testable prediction—by November 2026, radiologists should be completely replaceable by deep learning methods, at least other than regulatory requirements for trained physicians.
Thanks! I agree with you about all sorts of AI alignment essays being interesting and seemingly useful. My question was more about how to measure the net rate of AI safety research progress. But I agree with you that an/your expert inside view of how insights are accumulating is a reasonable metric. I also agree with you that the acceptance of TAI x-risk in the ML community as a real thing is useful and that—while I am slightly worried about the risk of overshooting, like Scott Alexander describes—this situation seems to be generally improving.
Regarding (2), my question is why algorithmic growth leading to serious growth of AI capabilities would be so discontinuous. I agree that RL is much better in humans than in machines, but I doubt that replicating this in machines would require just one or a few algorithmic advances. Instead, my guess, based on previous technology growth stories I’ve read about, is that AI algorithmic progress is likely to occur due to the accumulation of many small improvements over time.
Good essay! Two questions if you have a moment:
1. Can you flesh out your view of how the community is making “slow but steady progress right now on getting ready”? In my view, much of the AI safety community seems to be doing things that have unclear safety value to me, like (a) coordinating a pause in model training that seems likely to me to make things less safe if implemented (because of leading to algorithmic and hardware overhangs) or (b) converting to capabilities work (quite common, seems like an occupational hazard for someone with initially “pure” AI safety values). Of course, I don’t mean to be disparaging, as plenty of AI safety work does seem useful qua safety to me, like making more precise estimates of takeoff speeds or doing cybersecurity work. Just was surprised by that statement and I’m curious about how you are tracking progress here.
2. It seems like you think there are some key algorithmic insights, that once “unlocked”, will lead to dramatically faster AI development. This suggests that not many people are working on algorithmic insights. But that doesn’t seem quite right to me—isn’t that a huge group of researchers, many of whom have historically been anti-scaling? Or maybe you think there are core insights available, but the field hasn’t had (enough of) its Einsteins or von Neumanns yet? Basically, I’m trying to get a sense of why you seem to have very fast takeoff speed estimates given certain algorithmic progress. But maybe I’m not understanding your worldview and/or maybe it’s too infohazardous to discuss.
I didn’t realize you had put so much time into estimating take-off speeds. I think this is a really good idea.
This seems substantially slower than the implicit take-off speed estimates of Eliezer, but maybe I’m missing something.
I think the amount of time you described is probably shorter than I would guess. But I haven’t put nearly as much time into it as you have. In the future, I’d like to.
Still, my guess is that this amount of time is enough that there are multiple competing groups, rather than only one. So it seems to me like there would probably be competition in the world you are describing, making a singleton AI less likely.
Do you think that there will almost certainly be a singleton AI?
Thanks for writing this up as a shorter summary Rob. Thanks also for engaging with people who disagree with you over the years.
Here’s my main area of disagreement:
General intelligence is very powerful, and once we can build it at all, STEM-capable artificial general intelligence (AGI) is likely to vastly outperform human intelligence immediately (or very quickly).
I don’t think this is likely to be true. Perhaps it is true of some cognitive architectures, but not for the connectionist architectures that are the only known examples of human-like AI intelligence and that are clearly the top AIs available today. In these cases, I expect human-level AI capabilities to grow to the point that they will vastly outperform humans much more slowly than immediately or “very quickly”. This is basically the AI foom argument.
And I think all of your other points are dependent on this one. Because if this is not true, then humanity will have time to iteratively deal with the problems that emerge, as we have in the past with all other technologies.
My reasoning for not expecting ultra-rapid takeoff speeds is that I don’t view connectionist intelligence as having a sort of “secret sauce”, that once it is found, can unlock all sorts of other things. I think it is the sort of thing that will increase in a plodding way over time, depending on scaling and other similar inputs that cannot be increased immediately.
In the absence of some sort of “secret sauce”, which seems necessary for sharp left turns and other such scenarios, I view AI capabilities growth as likely to follow the same trends as other historical growth trends. In the case of a hypothetical AI at a human intelligence level, it would face constraints on its resources allowing it to improve, such as bandwidth, capital, skills, private knowledge, energy, space, robotic manipulation capabilities, material inputs, cooling requirements, legal and regulatory barriers, social acceptance, cybersecurity concerns, competition with humans and other AIs, and of course value maintenance concerns (i.e. it would have its own alignment problem to solve).
I guess if you are also taking those constraints into consideration, then it is really just a probabilistic feeling about how much those constraints will slow down AI growth. To me, those constraints each seem massive, and getting around all of them within hours or days would be nearly impossible, no matter how intelligent the AI was.
As a result, rather than indefinite and immediate exponential growth, I expect real-world AI growth to follow a series of sigmoidal curves, each eventually plateauing before different types of growth curves take over to increase capabilities based on different input resources (with all of this overlapping).
One area of uncertainty: I am concerned about there being a spectrum of takeoff speeds, from slow to immediate. In faster takeoff speed worlds, I view there as being more risk of bad outcomes generally, such as a totalitarian state using an AI to take over the world, or even the x-risk scenarios that you describe.
This is why I favor regulations that will be helpful in slower takeoff worlds, such as requiring liability insurance, and will not cause harm by increasing take-off speed. For example, pausing AGI training runs seems likely to make takeoff speed more discontinuous, due to creating hardware, algorithmic, and digital autonomous agent overhangs, thereby making the whole situation more dangerous. This is why I am opposed to it and dismayed to see so many on LW in favor of it.
I also recognize that I might be wrong about AI takeoff speeds not being fast. I am glad people are working on this, so long as they are not promoting policies that seem likely to make things more dangerous in the slower takeoff scenarios that I consider more likely.
Another area of uncertainty: I’m not sure what is going to happen long-term in a slow takeoff world. I’m confused. While I think that the scenarios you describe are not likely because they are dependent upon there being a fast takeoff and a resulting singleton AI, I find outcomes in slow takeoff worlds extraordinarily difficult to predict.
Overall I feel that AI x-risk is clearly the most likely x-risk of any in the coming years and am glad that you and others are focusing on it. My main hope for you is that you continue to be flexible in your thinking and make predictions that help you to decide if you should update your models.
Here are some predictions of mine:
Connectionist architectures will remain the dominant AI architecture in the next 10 years. Yes, they will be hooked up in larger deterministic systems, but humans will also be able to use connectionist architectures in this way, which will actually just increase competition and decrease the likelihood of ultra-rapid takeoffs.
Hardware availability will remain a constraint on AI capabilities in the next 10 years.
Robotic manipulation capabilities will remain a constraint on AI capabilities in the next 10 years.
Thanks for the comment. I’m definitely not assuming that p(success) would be a monocausal explanation. I’m mostly presenting this data to give evidence against that assumption, because people frequently make statements such as “of course almost nobody wants cryonics, they don’t expect it will work”.
I also agree that “is being revived good in expectation / good with what probability” is another common concern. Personally, I think niplav has some good analysis of net-negative revival scenarios: https://niplav.site/considerations_on_cryonics.html
Btw, according to the author, ‘Lena’ is largely a critique of exploitive capitalism: https://qntm.org/uploading