Personal website: https://andrewtmckenzie.com/
Andy_McKenzie
Karma: 1,370
Being against involuntary death and being open to change are compatible
Cryonics p(success) estimates are only weakly associated with interest in pursuing cryonics in the LW 2023 Survey
Why wasn’t preservation with the goal of potential future revival started earlier in history?
Very high-effort, comprehensive post. Any interest in making some of your predictions into markets on Manifold or some other prediction market website? Might help get some quantifications.
A simple solution is to just make doctors/hospitals liable for harm which occurs under their watch, period. Do not give them an out involving performative tests which don’t actually reduce harm, or the like. If doctors/hospitals are just generally liable for harm, then they’re incentivized to actually reduce it.
Can you explain more what you actually mean by this? Do you mean if someone comes into the hospital and dies, the doctors are responsible, regardless of why they died? If you mean that we figure out whether the doctors are responsible for whether the patient died, then we get back to whether they have done everything to prevent it, and one of these things might be ordering lab tests to better figure out the diagnosis, and then it seems we’re back to the original problem i.e. the status quo. Just not understanding what you mean.
Out of curiosity, what makes you think that the initial freezing process causes too much information loss?
I agree with most of this post, but it doesn’t seem to address the possibility of whole brain emulation. However, many/(?most) would argue this is unlikely to play a major role because AGI will come first.
Thanks so much for putting this together Mati! If people are interested in cryonics/brain preservation and would like to learn about (my perspective on) the field from a research perspective, please feel free to reach out to me: https://andrewtmckenzie.com/
I also have some external links/essays available here: https://brainpreservation.github.io/
It seems to me like your model is not necessarily taking into account technical debt sufficiently enough. https://neurobiology.substack.com/p/technical-debt-probably-the-main-roadblack-in-applying-machine-learning-to-medicine
It seems to me like this is the main thing that will slow down the extent to which foundation models can consistently beat newly trained specialized models.
Anecdotally, I know several people who don’t like to use chatgpt because its training cuts off in 2021. This seems like a form of technical debt.
I guess it depends on how easily adaptable foundation models are.
Sounds good, can’t find your email address, DM’d you.
Those sound good to me! I donated to your charity (the Animal Welfare Fund) to finalize it. Lmk if you want me to email you the receipt. Here’s the manifold market:
Bet
Andy will donate $50 to a charity of Daniel’s choice now.
If, by January 2027, there is not a report from a reputable source confirming that at least three companies, that would previously have relied upon programmers, and meet a defined level of success, are being run without the need for human programmers, due to the independent capabilities of an AI developed by OpenAI or another AI organization, then Daniel will donate $100, adjusted for inflation as of June 2023, to a charity of Andy’s choice.
Terms
Reputable Source: For the purpose of this bet, reputable sources include MIT Technology Review, Nature News, The Wall Street Journal, The New York Times, Wired, The Guardian, or TechCrunch, or similar publications of recognized journalistic professionalism. Personal blogs, social media sites, or tweets are excluded.
AI’s Capabilities: The AI must be capable of independently performing the full range of tasks typically carried out by a programmer, including but not limited to writing, debugging, maintaining code, and designing system architecture.
Equivalent Roles: Roles that involve tasks requiring comparable technical skills and knowledge to a programmer, such as maintaining codebases, approving code produced by AI, or prompting the AI with specific instructions about what code to write.
Level of Success: The companies must be generating a minimum annual revenue of $10 million (or likely generating this amount of revenue if it is not public knowledge).
Report: A single, substantive article or claim in one of the defined reputable sources that verifies the defined conditions.
AI Organization: An institution or entity recognized for conducting research in AI or developing AI technologies. This could include academic institutions, commercial entities, or government agencies.
Inflation Adjustment: The donation will be an equivalent amount of money as $100 as of June 2023, adjusted for inflation based on https://www.bls.gov/data/inflation_calculator.htm.
Regulatory Impact: In January 2027, Andy will use his best judgment to decide whether the conditions of the bet would have been met in the absence of any government regulation restricting or banning the types of AI that would have otherwise replaced programmers.
Sounds good, I’m happy with that arrangement once we get these details figured out.
Regarding the human programmer formality, it seems like business owners would have to be really incompetent for this to be a factor. Plenty of managers have coding experience. If the programmers aren’t doing anything useful then they will be let go or new companies will start that don’t have them. They are a huge expense. I’m inclined to not include this since it’s an ambiguity that seems implausible to me.
Regarding the potential ban by the government, I wasn’t really thinking of that as a possible option. What kind of ban do you have in mind? I imagine that regulation of AI is very likely by then, so if the automation of all programmers hasn’t happened by Jan 2027, it seems very easy to argue that it would have happened in the absence of the regulation.
Regarding these and a few of the other ambiguous things, one way we could do this is that you and I could just agree on it in Jan 2027. Otherwise, the bet resolves N/A and you don’t donate anything. This could make it an interesting Manifold question because it’s a bit adversarial. This way, we could also get rid of the requirement for it to be reported by a reputable source, which is going to be tricky to determine.
Understandable. How about this?
Bet
Andy will donate $50 to a charity of Daniel’s choice now.
If, by January 2027, there is not a report from a reputable source confirming that at least three companies, that would previously have relied upon programmers, and meet a defined level of success, are being run without the need for human programmers, due to the independent capabilities of an AI developed by OpenAI or another AI organization, then Daniel will donate $100, adjusted for inflation as of June 2023, to a charity of Andy’s choice.
Terms
Reputable Source: For the purpose of this bet, reputable sources include MIT Technology Review, Nature News, The Wall Street Journal, The New York Times, Wired, The Guardian, or TechCrunch, or similar publications of recognized journalistic professionalism. Personal blogs, social media sites, or tweets are excluded.
AI’s Capabilities: The AI must be capable of independently performing the full range of tasks typically carried out by a programmer, including but not limited to writing, debugging, maintaining code, and designing system architecture.
Equivalent Roles: Roles that involve tasks requiring comparable technical skills and knowledge to a programmer, such as maintaining codebases, approving code produced by AI, or prompting the AI with specific instructions about what code to write.
Level of Success: The companies must be generating a minimum annual revenue of $10 million (or likely generating this amount of revenue if it is not public knowledge).
Report: A single, substantive article or claim in one of the defined reputable sources that verifies the defined conditions.
AI Organization: An institution or entity recognized for conducting research in AI or developing AI technologies. This could include academic institutions, commercial entities, or government agencies.
Inflation Adjustment: The donation will be an equivalent amount of money as $100 as of June 2023, adjusted for inflation based on https://www.bls.gov/data/inflation_calculator.htm.
I guess that there might be some disagreements in these terms, so I’d be curious to hear your suggested improvements.
Caveat: I don’t have much disposable money right now, so it’s not much money, but perhaps this is still interesting as a marker of our beliefs. Totally ok if it’s not enough money to be worth it to you.
I’m wondering if we could make this into a bet. If by remote workers we include programmers, then I’d be willing to bet that GPT-5/6, depending upon what that means (might be easier to say the top LLMs or other models trained by anyone by 2026?) will not be able to replace them.
These curves are due to temporary plateaus, not permanent ones. Moore’s law is an example of a constraint that seems likely to plateau. I’m talking about takeoff speeds, not eventual capabilities with no resource limitations, which I agree would be quite high and I have little idea of how to estimate (there will probably still be some constraints, like within-system communication constraints).
Does anyone know of any AI-related predictions by Hinton?
Here’s the only one I know of—“People should stop training radiologists now. It’s just completely obvious within five years deep learning is going to do better than radiologists because it can get a lot more experience. And it might be ten years but we got plenty of radiologists already.” − 2016, slightly paraphrased
This seems like still a testable prediction—by November 2026, radiologists should be completely replaceable by deep learning methods, at least other than regulatory requirements for trained physicians.
Thanks! I agree with you about all sorts of AI alignment essays being interesting and seemingly useful. My question was more about how to measure the net rate of AI safety research progress. But I agree with you that an/your expert inside view of how insights are accumulating is a reasonable metric. I also agree with you that the acceptance of TAI x-risk in the ML community as a real thing is useful and that—while I am slightly worried about the risk of overshooting, like Scott Alexander describes—this situation seems to be generally improving.
Regarding (2), my question is why algorithmic growth leading to serious growth of AI capabilities would be so discontinuous. I agree that RL is much better in humans than in machines, but I doubt that replicating this in machines would require just one or a few algorithmic advances. Instead, my guess, based on previous technology growth stories I’ve read about, is that AI algorithmic progress is likely to occur due to the accumulation of many small improvements over time.
Good essay! Two questions if you have a moment:
1. Can you flesh out your view of how the community is making “slow but steady progress right now on getting ready”? In my view, much of the AI safety community seems to be doing things that have unclear safety value to me, like (a) coordinating a pause in model training that seems likely to me to make things less safe if implemented (because of leading to algorithmic and hardware overhangs) or (b) converting to capabilities work (quite common, seems like an occupational hazard for someone with initially “pure” AI safety values). Of course, I don’t mean to be disparaging, as plenty of AI safety work does seem useful qua safety to me, like making more precise estimates of takeoff speeds or doing cybersecurity work. Just was surprised by that statement and I’m curious about how you are tracking progress here.
2. It seems like you think there are some key algorithmic insights, that once “unlocked”, will lead to dramatically faster AI development. This suggests that not many people are working on algorithmic insights. But that doesn’t seem quite right to me—isn’t that a huge group of researchers, many of whom have historically been anti-scaling? Or maybe you think there are core insights available, but the field hasn’t had (enough of) its Einsteins or von Neumanns yet? Basically, I’m trying to get a sense of why you seem to have very fast takeoff speed estimates given certain algorithmic progress. But maybe I’m not understanding your worldview and/or maybe it’s too infohazardous to discuss.
I didn’t realize you had put so much time into estimating take-off speeds. I think this is a really good idea.
This seems substantially slower than the implicit take-off speed estimates of Eliezer, but maybe I’m missing something.
I think the amount of time you described is probably shorter than I would guess. But I haven’t put nearly as much time into it as you have. In the future, I’d like to.
Still, my guess is that this amount of time is enough that there are multiple competing groups, rather than only one. So it seems to me like there would probably be competition in the world you are describing, making a singleton AI less likely.
Do you think that there will almost certainly be a singleton AI?
Thanks for the comment. I’m definitely not assuming that p(success) would be a monocausal explanation. I’m mostly presenting this data to give evidence against that assumption, because people frequently make statements such as “of course almost nobody wants cryonics, they don’t expect it will work”.
I also agree that “is being revived good in expectation / good with what probability” is another common concern. Personally, I think niplav has some good analysis of net-negative revival scenarios: https://niplav.site/considerations_on_cryonics.html
Btw, according to the author, ‘Lena’ is largely a critique of exploitive capitalism: https://qntm.org/uploading