With this as an example, I’m like “yeah that sounds probably safe to work on, and worth someone doing, but also it won’t work partly because it’s safe to work on—in other words, it probably will not really address problems such as reflective stability, which are ~necessary to address, because it is avoiding thinking about the cores of how intelligence works, and therefore probably can’t figure out the cores of reflective stability in intelligent minds”.
Based on reading your blog over the past years, I feel like the thing you have in mind for what a solution to alignment would look like is not just hard but impossible and unnecessary. Like, suppose (as a thought experiment) ASI were impossible in our universe, along with other x-risks. We would be passing the torch to the next generation of humans, just as every previous generation has. What will that next generation do with their powers? We don’t really know. It could be good, it could be bad. We can and should fight to try to make it better. But we don’t say to ourselves that we need to “solve reflective stability” for future humans, and if we don’t, then we have failed.
The future humans will learn new things we can’t imagine (presumably), and then make decisions, and the end-results might be good insofar as the humans have “good” dispositions and are able to solve coordination problems etc. We can and should try to anticipate and preempt problems. But we mostly can’t micromanage the outcome.
Likewise, I think it’s hard but possible to make an ASI with a “good” disposition, but if we do, we won’t be able to anticipate what it will wind up deciding to do, without being superintelligent ourselves. But that shouldn’t count as “alignment failure”.
I think one could plausibly significantly push it back significantly through a combination of international treaty, professional norms, academic norms, social stigma, public consensus, and similar. Yes, you can’t stop people doing conceptual research. But you can make them afraid to talk about it or share it, and make them feel guilty for doing it. That doesn’t last anywhere close to forever, but it could quite significantly increase timelines.
This is a thing that could happen in principle, and I opportunistically do outreach to try to help on the margin [for my part I’m not spreading the message “these people suck” but rather “here’s why ASI is dangerous”], but getting to the place you describe seems extraordinarily unlikely to me. It seems to me that people have been raising the alarm on AI doom since Turing, and basically nobody cared until superintelligence seemed plausibly around the corner. Now that it is, people are somewhat more receptive to a narrowly-targeted restriction of that specific kind of ASI that they think is around the corner. Almost everyone on both sides continues to view “ASI is right around the corner” as a crux on the question of whether we should be worried about ASI at all.
I also think you’re insufficiently cynical about how deftly people can brand their capabilities research as “safety”-positive. Copying from here:
Thus, those researchers I mentioned, who are messing around with new paradigms on arXiv, are in a great position to twist “Pause AI” type public advocacy into support for what they’re doing!
“You don’t like LLMs?”, the non-LLM AGI capabilities researchers say to the Pause AI people, “Well how about that! I don’t like LLMs either! Clearly we are on the same team!”
This is not idle speculation—almost everyone that I can think of who is doing the most dangerous kind of AI capabilities research, the kind aiming to develop a new more-powerful-than-LLM AI paradigm, is already branding their work in a way that vibes with safety. For example, see here where I push back on someone using the word “controllability” to talk about his work advancing AI capabilities beyond the limits of LLMs. Ditto for “robustness” (example), “adaptability” (e.g. in the paper I was criticizing here), and even “interpretability” (details).
I think these people are generally sincere but mistaken, and I expect that, just as they have fooled themselves, they will also successfully fool their friends, their colleagues, and government regulators. Well, the government regulators hardly matter anyway, since regulating the activity of “playing with toy models, and publishing stuff on arXiv and GitHub” is a hell of an ask—I think it’s so unlikely to happen that it’s a waste of time to even talk about it, even if it were a good idea all-things-considered.
(Just yesterday I was poring over a paper by a guy who went from doing the worst kind of capabilities research in academia, to MATS (!!), and then straight back to the worst kind of capabilities research but now with higher salary at a nominally-AGI-safety-focused nonprofit. His personal website emphasizes how his research is helping AGI safety.)
AGI is more likely to come >20 years from now than <20 years from now. (I assume you disagree because no one agrees with this.) Therefore burning a 30 year pause down to a 28 year pause is not worth a +.1% chance of alignment within 10 years. (Sorry these may be straw numbers.)
Yeah I do think >20 years is <50% likely (although not <<50%). I disagree with the idea that a connectomics dataset is only relevant if ASI happens soon. For one thing, these are long-serial-time projects; the idea is to develop the experimental methods today in order to have a bare-bones human connectome in like 2040, or even 2050 or later if funding is less generous or technical difficulties arise. Then that dataset would be an input into research, and it may take years to digest, and we might want to iterate, etc. Anyway, if it’s HIA people in the distant future trying to solve alignment, I think they too would benefit from having this kind of data. Again, more generally, solving technical alignment requires serial time (1,2); even if HIA people will be working on technical alignment in the future, their odds will be better if we are also working on technical alignment right now.
Maybe we should table this? I’m not sure what the cruxes are. If we were to continue, it might be helpful for us to make up numbers in order to find the most obvious disagreements (and/or change our numbers to become more internally coherent).
E.g. I’m like (these aren’t very considered numbers, may revise; I’m trying to illustrate the process-level point, but not completely made up):
The change in probability of solving alignment (that is, becoming able to make a friendly superhuman AGI) within the next 50 years from having a cortical human connectome is, IDK, <.03%, with or without HIA? The absolute probability currently is <3%, assuming no HIA?
The change in probability of making a dangerous PBE within the next 50 years from having a cortical human connectome is, I have no idea, .06%<x<1%?
… etc.
(Maybe I should think this through more seriously anyway, though seems hard and IDK if likely to change most of my actions.)
More reactions:
Based on reading your blog over the past years, I feel like the thing you have in mind for what a solution to alignment would look like is not just hard but impossible and unnecessary. Like, suppose (as a thought experiment) ASI were impossible in our universe, along with other x-risks. We would be passing the torch to the next generation of humans, just as every previous generation has. What will that next generation do with their powers? We don’t really know. It could be good, it could be bad. We can and should fight to try to make it better. But we don’t say to ourselves that we need to “solve reflective stability” for future humans, and if we don’t, then we have failed.
I disagree with this. You don’t ~need to understand reflective stability because of anything about guarantees or a high bar, let alone impossible. You ~need to understand it because otherwise you don’t understand about what values even are. (Cf. re/ the importance of reflective stability: https://www.lesswrong.com/posts/Ht4JZtxngKwuQ7cDC/tsvibt-s-shortform?commentId=koeti9ygXB9wPLnnF ) I mean, this could be wrong. I don’t think I would even tell people to make “understand reflective stability” a focus of their research. I would say “understand what values are”. And then they would fail to consider reflective stability, perhaps because they are not considering the cores of intelligence which make reflective stability an especially sharp problem, and therefore they would probably be failing to understand what values are.
So what’s the deal with human successors? We ~know they have our values because they’re human! Figuring out how to make a de novo mind with values we like (or some kind of quasi-value structure that ends up doing things we like) is a whole different challenge.
In other words, the point I was trying to make was “if your research isn’t grappling with the cores of intelligence, it probably won’t understand [what sorts of things really drive the ultimate effects of a very intelligent mind]”.
The future humans will learn new things we can’t imagine (presumably), and then make decisions, and the end-results might be good insofar as the humans have “good” dispositions and are able to solve coordination problems etc. We can and should try to anticipate and preempt problems. But we mostly can’t micromanage the outcome.
Likewise, I think it’s hard but possible to make an ASI with a “good” disposition, but if we do, we won’t be able to anticipate what it will wind up deciding to do, without being superintelligent ourselves. But that shouldn’t count as “alignment failure”.
(I’m kinda confused—I take it griceanly that you think I disagree with this or haven’t priced it in somehow, but also confused that you would think I disagree with this or haven’t priced it in somehow; if that’s actually true of me, I don’t get it yet from these paragraphs.)
I also think you’re insufficiently cynical about how deftly people can brand their capabilities research as “safety”-positive.
Maybe, but not sure why you think this. I’m not saying you can get an infinite pause with no leaks with high probability. I’m saying you can “plausibly significantly push it back significantly”. Yeah, part of that would have to be extreme skepticism about safetywashing, which is not easy to implement.
I don’t think safetywashing is the right term for what I was referring to in the block-quote in the grandparent comment. @Adam Schollcoined “safetywashing” in analogy with “greenwashing”, e.g. Chevron buying a small environmental preserve and advertising it on TV. Some company has something it wants to do anyway, that’s generally bad for the environment, and they find some way to portray themselves as environmental anyway.
By contrast, I’m talking about something closer to environmentalists who oppose nuclear power and renewable energy projects, thus harming the environment, because they’re rationalizing, or just bad at thinking things through. There was never an ulterior motive; the motive was environmentalist all along, it’s just that environmentalists can be spectacularly counterproductive at achieving their own goals. For example that MATS guy I mentioned, the one trying to develop a new AI paradigm, he really thinks that what he’s doing will help pave the way to safer AI. But I think it’s just awful, pure timelines-shortening with no safety benefits whatsoever. We can speculate about whether he had a subconscious ulterior motive of some sort (e.g. constructing elegant theories about AI is fun and prestigious), but it’s equally plausible that he really did care about safety from the start, just in a spectacularly counterproductive way (according to me). And he started doing that work while in academia, so there wasn’t any organization-level ulterior motive either.
Based on reading your blog over the past years, I feel like the thing you have in mind for what a solution to alignment would look like is not just hard but impossible and unnecessary. Like, suppose (as a thought experiment) ASI were impossible in our universe, along with other x-risks. We would be passing the torch to the next generation of humans, just as every previous generation has. What will that next generation do with their powers? We don’t really know. It could be good, it could be bad. We can and should fight to try to make it better. But we don’t say to ourselves that we need to “solve reflective stability” for future humans, and if we don’t, then we have failed.
The future humans will learn new things we can’t imagine (presumably), and then make decisions, and the end-results might be good insofar as the humans have “good” dispositions and are able to solve coordination problems etc. We can and should try to anticipate and preempt problems. But we mostly can’t micromanage the outcome.
Likewise, I think it’s hard but possible to make an ASI with a “good” disposition, but if we do, we won’t be able to anticipate what it will wind up deciding to do, without being superintelligent ourselves. But that shouldn’t count as “alignment failure”.
This is a thing that could happen in principle, and I opportunistically do outreach to try to help on the margin [for my part I’m not spreading the message “these people suck” but rather “here’s why ASI is dangerous”], but getting to the place you describe seems extraordinarily unlikely to me. It seems to me that people have been raising the alarm on AI doom since Turing, and basically nobody cared until superintelligence seemed plausibly around the corner. Now that it is, people are somewhat more receptive to a narrowly-targeted restriction of that specific kind of ASI that they think is around the corner. Almost everyone on both sides continues to view “ASI is right around the corner” as a crux on the question of whether we should be worried about ASI at all.
I also think you’re insufficiently cynical about how deftly people can brand their capabilities research as “safety”-positive. Copying from here:
(Just yesterday I was poring over a paper by a guy who went from doing the worst kind of capabilities research in academia, to MATS (!!), and then straight back to the worst kind of capabilities research but now with higher salary at a nominally-AGI-safety-focused nonprofit. His personal website emphasizes how his research is helping AGI safety.)
Yeah I do think >20 years is <50% likely (although not <<50%). I disagree with the idea that a connectomics dataset is only relevant if ASI happens soon. For one thing, these are long-serial-time projects; the idea is to develop the experimental methods today in order to have a bare-bones human connectome in like 2040, or even 2050 or later if funding is less generous or technical difficulties arise. Then that dataset would be an input into research, and it may take years to digest, and we might want to iterate, etc. Anyway, if it’s HIA people in the distant future trying to solve alignment, I think they too would benefit from having this kind of data. Again, more generally, solving technical alignment requires serial time (1,2); even if HIA people will be working on technical alignment in the future, their odds will be better if we are also working on technical alignment right now.
To the point:
Maybe we should table this? I’m not sure what the cruxes are. If we were to continue, it might be helpful for us to make up numbers in order to find the most obvious disagreements (and/or change our numbers to become more internally coherent).
E.g. I’m like (these aren’t very considered numbers, may revise; I’m trying to illustrate the process-level point, but not completely made up):
The change in probability of solving alignment (that is, becoming able to make a friendly superhuman AGI) within the next 50 years from having a cortical human connectome is, IDK, <.03%, with or without HIA? The absolute probability currently is <3%, assuming no HIA?
The change in probability of making a dangerous PBE within the next 50 years from having a cortical human connectome is, I have no idea, .06%<x<1%? … etc.
(Maybe I should think this through more seriously anyway, though seems hard and IDK if likely to change most of my actions.)
More reactions:
I disagree with this. You don’t ~need to understand reflective stability because of anything about guarantees or a high bar, let alone impossible. You ~need to understand it because otherwise you don’t understand about what values even are. (Cf. re/ the importance of reflective stability: https://www.lesswrong.com/posts/Ht4JZtxngKwuQ7cDC/tsvibt-s-shortform?commentId=koeti9ygXB9wPLnnF ) I mean, this could be wrong. I don’t think I would even tell people to make “understand reflective stability” a focus of their research. I would say “understand what values are”. And then they would fail to consider reflective stability, perhaps because they are not considering the cores of intelligence which make reflective stability an especially sharp problem, and therefore they would probably be failing to understand what values are.
So what’s the deal with human successors? We ~know they have our values because they’re human! Figuring out how to make a de novo mind with values we like (or some kind of quasi-value structure that ends up doing things we like) is a whole different challenge.
In other words, the point I was trying to make was “if your research isn’t grappling with the cores of intelligence, it probably won’t understand [what sorts of things really drive the ultimate effects of a very intelligent mind]”.
(I’m kinda confused—I take it griceanly that you think I disagree with this or haven’t priced it in somehow, but also confused that you would think I disagree with this or haven’t priced it in somehow; if that’s actually true of me, I don’t get it yet from these paragraphs.)
Maybe, but not sure why you think this. I’m not saying you can get an infinite pause with no leaks with high probability. I’m saying you can “plausibly significantly push it back significantly”. Yeah, part of that would have to be extreme skepticism about safetywashing, which is not easy to implement.
Sure, let’s table. Except for one little thing:
I don’t think safetywashing is the right term for what I was referring to in the block-quote in the grandparent comment. @Adam Scholl coined “safetywashing” in analogy with “greenwashing”, e.g. Chevron buying a small environmental preserve and advertising it on TV. Some company has something it wants to do anyway, that’s generally bad for the environment, and they find some way to portray themselves as environmental anyway.
By contrast, I’m talking about something closer to environmentalists who oppose nuclear power and renewable energy projects, thus harming the environment, because they’re rationalizing, or just bad at thinking things through. There was never an ulterior motive; the motive was environmentalist all along, it’s just that environmentalists can be spectacularly counterproductive at achieving their own goals. For example that MATS guy I mentioned, the one trying to develop a new AI paradigm, he really thinks that what he’s doing will help pave the way to safer AI. But I think it’s just awful, pure timelines-shortening with no safety benefits whatsoever. We can speculate about whether he had a subconscious ulterior motive of some sort (e.g. constructing elegant theories about AI is fun and prestigious), but it’s equally plausible that he really did care about safety from the start, just in a spectacularly counterproductive way (according to me). And he started doing that work while in academia, so there wasn’t any organization-level ulterior motive either.