I agree that many people have said that they are trading off a pinch of timelines-shortening for a pound of alignment progress, but where the results have been bad. I think you’re reacting to that history by saying “anyone suggesting that they’re doing such a thing is not to be trusted, they’re probably just rationalizing or wrong”.
I’m almost saying that, yeah. Also note that, beyond just the history of people saying that with bad results, there’s also the history of grantmakers not being able to tell that that would happen, and funding significantly more harmful stuff. That’s an additional knowledge problem. In other words, if you’re about to fund X, consider that it’s not feasible to fund X while having a good impact. It’s sad because, yes, somewhere in the used car lot there’s probably a good deal. It’s just that you actually can’t tell, so you’ll have to go someplace else.
But that’s an overreaction. You have to actually evaluate the argument.
Evaluating the object-level argument is helpful of course, and may turn up good used cars. But I’ll stand by keeping the prior with a pretty substantial degree of control over the decisions, even after you’ve done some amount of object-level evaluation.
Otherwise you’re making the same mistake as the people who say “previous apocalypse prophecies haven’t come true, so I’ll just ignore them all”, and so on.
I think they’re doing a somewhat rational move! And then you’re supposed to argue the object-level convincingly enough to overcome that.
It’s impossible to publish work on alignment if you’re avoiding even the faintest possibility that there will be even a tiny timelines-shortening externality. E.g. you yourself are a coauthor on the logical induction paper, which clearly-to-me fails that standard. (But I don’t want to accuse you of hypocrisy because maybe your opinions have changed in the intervening decade?)
(I was not in fact remotely cruxy for that paper, but in any case:) Yes, I agree that it fails the standard, and in fact probably shouldn’t have been published (which I certainly didn’t say or think at the time). That’s a change of mind with benefit of hindsight, yeah.
If we divide RL & search into “what’s the reward / cost / objective function” versus “how do we build an AI that optimizes that”, then the former is (almost entirely) not required for creating AGI, and indeed usually not even slightly helpful for creating AGI.
With this as an example, I’m like “yeah that sounds probably safe to work on, and worth someone doing, but also it won’t work partly because it’s safe to work on—in other words, it probably will not really address problems such as reflective stability, which are ~necessary to address, because it is avoiding thinking about the cores of how intelligence works, and therefore probably can’t figure out the cores of reflective stability in intelligent minds”.
On the margin there are ways to lengthen and shorten timelines, but lengthening them to infinity does not seem to be an option. How would you get to infinity? IMO it would require shutting down AI-related basic research everywhere on Earth. That really seems extremely unlikely to me,
I agree with this literally, but it sounds like we may have disagreements in space in between? I think one could plausibly significantly push it back significantly through a combination of international treaty, professional norms, academic norms, social stigma, public consensus, and similar. Yes, you can’t stop people doing conceptual research. But you can make them afraid to talk about it or share it, and make them feel guilty for doing it. That doesn’t last anywhere close to forever, but it could quite significantly increase timelines.
I interpret your stance as “we should roll over and die in the latter worlds, if fighting would even infinitesimally harm are chances in the former worlds”.
This is pretty straw of course. My stance is
The highest priority is getting and maintaining an AI research stop quickly and firmly. (I think we agree.)
WBE research in particular, as a research goal, erodes an AI stop too much. (I think we agree?)
Alignment is very hard. (We agree.) This makes it even worse to burn commons of a stop by adding more capabilities.
AGI is more likely to come >20 years from now than <20 years from now. (I assume you disagree because no one agrees with this.) Therefore burning a 30 year pause down to a 28 year pause is not worth a +.1% chance of alignment within 10 years. (Sorry these may be straw numbers.)
Alignment is so hard that it is a lower priority than HIA research. (It sounds like you disagree because you don’t think HIA helps much? Though this is a tangent.)
Connectomics by itself seems to me like a relatively safe kind of WBE-related research. If I had to guess I would not guess that by itself a big high quality cortex connectome would be very dangerous. But I do not feel confident about this. Maybe a shitty, not-much-learning electrical-model cortex, plus added ML, would actually be dangerous. I don’t think this is an infinitesimal harm, I think it’s a pretty small but noticeable harm in expectation. I could be argued into thinking that it’s lower, but I’m a little skeptical that you know that. Like, I agree with your take about the cortical connectome being basically learned parameters rather than the learning algorithm. I’m just like, “Ok, but how surprised would I be if it turned out that actually doing a human cortical connectome does unlock dangerous AI? Kinda suprised but not utterly gobsmacked.”. For example, some of the learned params may encode crystallized meta-cognition; and then when you take that, simulate it at the 30-second level, scale it up 100x, and do ML training as if it’s an LLM, and also try out “neural learning algorithms”, you get something super duper dangerous. It just wouldn’t be that surprising.
Connectomics doesn’t seem that helpful for alignment.
Maybe non-cortical connectomics is both safer and is the main usefulness for alignment? (Is this your favorite version of connectomics?)
With this as an example, I’m like “yeah that sounds probably safe to work on, and worth someone doing, but also it won’t work partly because it’s safe to work on—in other words, it probably will not really address problems such as reflective stability, which are ~necessary to address, because it is avoiding thinking about the cores of how intelligence works, and therefore probably can’t figure out the cores of reflective stability in intelligent minds”.
Based on reading your blog over the past years, I feel like the thing you have in mind for what a solution to alignment would look like is not just hard but impossible and unnecessary. Like, suppose (as a thought experiment) ASI were impossible in our universe, along with other x-risks. We would be passing the torch to the next generation of humans, just as every previous generation has. What will that next generation do with their powers? We don’t really know. It could be good, it could be bad. We can and should fight to try to make it better. But we don’t say to ourselves that we need to “solve reflective stability” for future humans, and if we don’t, then we have failed.
The future humans will learn new things we can’t imagine (presumably), and then make decisions, and the end-results might be good insofar as the humans have “good” dispositions and are able to solve coordination problems etc. We can and should try to anticipate and preempt problems. But we mostly can’t micromanage the outcome.
Likewise, I think it’s hard but possible to make an ASI with a “good” disposition, but if we do, we won’t be able to anticipate what it will wind up deciding to do, without being superintelligent ourselves. But that shouldn’t count as “alignment failure”.
I think one could plausibly significantly push it back significantly through a combination of international treaty, professional norms, academic norms, social stigma, public consensus, and similar. Yes, you can’t stop people doing conceptual research. But you can make them afraid to talk about it or share it, and make them feel guilty for doing it. That doesn’t last anywhere close to forever, but it could quite significantly increase timelines.
This is a thing that could happen in principle, and I opportunistically do outreach to try to help on the margin [for my part I’m not spreading the message “these people suck” but rather “here’s why ASI is dangerous”], but getting to the place you describe seems extraordinarily unlikely to me. It seems to me that people have been raising the alarm on AI doom since Turing, and basically nobody cared until superintelligence seemed plausibly around the corner. Now that it is, people are somewhat more receptive to a narrowly-targeted restriction of that specific kind of ASI that they think is around the corner. Almost everyone on both sides continues to view “ASI is right around the corner” as a crux on the question of whether we should be worried about ASI at all.
I also think you’re insufficiently cynical about how deftly people can brand their capabilities research as “safety”-positive. Copying from here:
Thus, those researchers I mentioned, who are messing around with new paradigms on arXiv, are in a great position to twist “Pause AI” type public advocacy into support for what they’re doing!
“You don’t like LLMs?”, the non-LLM AGI capabilities researchers say to the Pause AI people, “Well how about that! I don’t like LLMs either! Clearly we are on the same team!”
This is not idle speculation—almost everyone that I can think of who is doing the most dangerous kind of AI capabilities research, the kind aiming to develop a new more-powerful-than-LLM AI paradigm, is already branding their work in a way that vibes with safety. For example, see here where I push back on someone using the word “controllability” to talk about his work advancing AI capabilities beyond the limits of LLMs. Ditto for “robustness” (example), “adaptability” (e.g. in the paper I was criticizing here), and even “interpretability” (details).
I think these people are generally sincere but mistaken, and I expect that, just as they have fooled themselves, they will also successfully fool their friends, their colleagues, and government regulators. Well, the government regulators hardly matter anyway, since regulating the activity of “playing with toy models, and publishing stuff on arXiv and GitHub” is a hell of an ask—I think it’s so unlikely to happen that it’s a waste of time to even talk about it, even if it were a good idea all-things-considered.
(Just yesterday I was poring over a paper by a guy who went from doing the worst kind of capabilities research in academia, to MATS (!!), and then straight back to the worst kind of capabilities research but now with higher salary at a nominally-AGI-safety-focused nonprofit. His personal website emphasizes how his research is helping AGI safety.)
AGI is more likely to come >20 years from now than <20 years from now. (I assume you disagree because no one agrees with this.) Therefore burning a 30 year pause down to a 28 year pause is not worth a +.1% chance of alignment within 10 years. (Sorry these may be straw numbers.)
Yeah I do think >20 years is <50% likely (although not <<50%). I disagree with the idea that a connectomics dataset is only relevant if ASI happens soon. For one thing, these are long-serial-time projects; the idea is to develop the experimental methods today in order to have a bare-bones human connectome in like 2040, or even 2050 or later if funding is less generous or technical difficulties arise. Then that dataset would be an input into research, and it may take years to digest, and we might want to iterate, etc. Anyway, if it’s HIA people in the distant future trying to solve alignment, I think they too would benefit from having this kind of data. Again, more generally, solving technical alignment requires serial time (1,2); even if HIA people will be working on technical alignment in the future, their odds will be better if we are also working on technical alignment right now.
Maybe we should table this? I’m not sure what the cruxes are. If we were to continue, it might be helpful for us to make up numbers in order to find the most obvious disagreements (and/or change our numbers to become more internally coherent).
E.g. I’m like (these aren’t very considered numbers, may revise; I’m trying to illustrate the process-level point, but not completely made up):
The change in probability of solving alignment (that is, becoming able to make a friendly superhuman AGI) within the next 50 years from having a cortical human connectome is, IDK, <.03%, with or without HIA? The absolute probability currently is <3%, assuming no HIA?
The change in probability of making a dangerous PBE within the next 50 years from having a cortical human connectome is, I have no idea, .06%<x<1%?
… etc.
(Maybe I should think this through more seriously anyway, though seems hard and IDK if likely to change most of my actions.)
More reactions:
Based on reading your blog over the past years, I feel like the thing you have in mind for what a solution to alignment would look like is not just hard but impossible and unnecessary. Like, suppose (as a thought experiment) ASI were impossible in our universe, along with other x-risks. We would be passing the torch to the next generation of humans, just as every previous generation has. What will that next generation do with their powers? We don’t really know. It could be good, it could be bad. We can and should fight to try to make it better. But we don’t say to ourselves that we need to “solve reflective stability” for future humans, and if we don’t, then we have failed.
I disagree with this. You don’t ~need to understand reflective stability because of anything about guarantees or a high bar, let alone impossible. You ~need to understand it because otherwise you don’t understand about what values even are. (Cf. re/ the importance of reflective stability: https://www.lesswrong.com/posts/Ht4JZtxngKwuQ7cDC/tsvibt-s-shortform?commentId=koeti9ygXB9wPLnnF ) I mean, this could be wrong. I don’t think I would even tell people to make “understand reflective stability” a focus of their research. I would say “understand what values are”. And then they would fail to consider reflective stability, perhaps because they are not considering the cores of intelligence which make reflective stability an especially sharp problem, and therefore they would probably be failing to understand what values are.
So what’s the deal with human successors? We ~know they have our values because they’re human! Figuring out how to make a de novo mind with values we like (or some kind of quasi-value structure that ends up doing things we like) is a whole different challenge.
In other words, the point I was trying to make was “if your research isn’t grappling with the cores of intelligence, it probably won’t understand [what sorts of things really drive the ultimate effects of a very intelligent mind]”.
The future humans will learn new things we can’t imagine (presumably), and then make decisions, and the end-results might be good insofar as the humans have “good” dispositions and are able to solve coordination problems etc. We can and should try to anticipate and preempt problems. But we mostly can’t micromanage the outcome.
Likewise, I think it’s hard but possible to make an ASI with a “good” disposition, but if we do, we won’t be able to anticipate what it will wind up deciding to do, without being superintelligent ourselves. But that shouldn’t count as “alignment failure”.
(I’m kinda confused—I take it griceanly that you think I disagree with this or haven’t priced it in somehow, but also confused that you would think I disagree with this or haven’t priced it in somehow; if that’s actually true of me, I don’t get it yet from these paragraphs.)
I also think you’re insufficiently cynical about how deftly people can brand their capabilities research as “safety”-positive.
Maybe, but not sure why you think this. I’m not saying you can get an infinite pause with no leaks with high probability. I’m saying you can “plausibly significantly push it back significantly”. Yeah, part of that would have to be extreme skepticism about safetywashing, which is not easy to implement.
I don’t think safetywashing is the right term for what I was referring to in the block-quote in the grandparent comment. @Adam Schollcoined “safetywashing” in analogy with “greenwashing”, e.g. Chevron buying a small environmental preserve and advertising it on TV. Some company has something it wants to do anyway, that’s generally bad for the environment, and they find some way to portray themselves as environmental anyway.
By contrast, I’m talking about something closer to environmentalists who oppose nuclear power and renewable energy projects, thus harming the environment, because they’re rationalizing, or just bad at thinking things through. There was never an ulterior motive; the motive was environmentalist all along, it’s just that environmentalists can be spectacularly counterproductive at achieving their own goals. For example that MATS guy I mentioned, the one trying to develop a new AI paradigm, he really thinks that what he’s doing will help pave the way to safer AI. But I think it’s just awful, pure timelines-shortening with no safety benefits whatsoever. We can speculate about whether he had a subconscious ulterior motive of some sort (e.g. constructing elegant theories about AI is fun and prestigious), but it’s equally plausible that he really did care about safety from the start, just in a spectacularly counterproductive way (according to me). And he started doing that work while in academia, so there wasn’t any organization-level ulterior motive either.
I’m almost saying that, yeah. Also note that, beyond just the history of people saying that with bad results, there’s also the history of grantmakers not being able to tell that that would happen, and funding significantly more harmful stuff. That’s an additional knowledge problem. In other words, if you’re about to fund X, consider that it’s not feasible to fund X while having a good impact. It’s sad because, yes, somewhere in the used car lot there’s probably a good deal. It’s just that you actually can’t tell, so you’ll have to go someplace else.
Evaluating the object-level argument is helpful of course, and may turn up good used cars. But I’ll stand by keeping the prior with a pretty substantial degree of control over the decisions, even after you’ve done some amount of object-level evaluation.
I think they’re doing a somewhat rational move! And then you’re supposed to argue the object-level convincingly enough to overcome that.
(I was not in fact remotely cruxy for that paper, but in any case:) Yes, I agree that it fails the standard, and in fact probably shouldn’t have been published (which I certainly didn’t say or think at the time). That’s a change of mind with benefit of hindsight, yeah.
With this as an example, I’m like “yeah that sounds probably safe to work on, and worth someone doing, but also it won’t work partly because it’s safe to work on—in other words, it probably will not really address problems such as reflective stability, which are ~necessary to address, because it is avoiding thinking about the cores of how intelligence works, and therefore probably can’t figure out the cores of reflective stability in intelligent minds”.
I agree with this literally, but it sounds like we may have disagreements in space in between? I think one could plausibly significantly push it back significantly through a combination of international treaty, professional norms, academic norms, social stigma, public consensus, and similar. Yes, you can’t stop people doing conceptual research. But you can make them afraid to talk about it or share it, and make them feel guilty for doing it. That doesn’t last anywhere close to forever, but it could quite significantly increase timelines.
This is pretty straw of course. My stance is
The highest priority is getting and maintaining an AI research stop quickly and firmly. (I think we agree.)
WBE research in particular, as a research goal, erodes an AI stop too much. (I think we agree?)
Alignment is very hard. (We agree.) This makes it even worse to burn commons of a stop by adding more capabilities.
AGI is more likely to come >20 years from now than <20 years from now. (I assume you disagree because no one agrees with this.) Therefore burning a 30 year pause down to a 28 year pause is not worth a +.1% chance of alignment within 10 years. (Sorry these may be straw numbers.)
Alignment is so hard that it is a lower priority than HIA research. (It sounds like you disagree because you don’t think HIA helps much? Though this is a tangent.)
Connectomics by itself seems to me like a relatively safe kind of WBE-related research. If I had to guess I would not guess that by itself a big high quality cortex connectome would be very dangerous. But I do not feel confident about this. Maybe a shitty, not-much-learning electrical-model cortex, plus added ML, would actually be dangerous. I don’t think this is an infinitesimal harm, I think it’s a pretty small but noticeable harm in expectation. I could be argued into thinking that it’s lower, but I’m a little skeptical that you know that. Like, I agree with your take about the cortical connectome being basically learned parameters rather than the learning algorithm. I’m just like, “Ok, but how surprised would I be if it turned out that actually doing a human cortical connectome does unlock dangerous AI? Kinda suprised but not utterly gobsmacked.”. For example, some of the learned params may encode crystallized meta-cognition; and then when you take that, simulate it at the 30-second level, scale it up 100x, and do ML training as if it’s an LLM, and also try out “neural learning algorithms”, you get something super duper dangerous. It just wouldn’t be that surprising.
Connectomics doesn’t seem that helpful for alignment.
Maybe non-cortical connectomics is both safer and is the main usefulness for alignment? (Is this your favorite version of connectomics?)
Based on reading your blog over the past years, I feel like the thing you have in mind for what a solution to alignment would look like is not just hard but impossible and unnecessary. Like, suppose (as a thought experiment) ASI were impossible in our universe, along with other x-risks. We would be passing the torch to the next generation of humans, just as every previous generation has. What will that next generation do with their powers? We don’t really know. It could be good, it could be bad. We can and should fight to try to make it better. But we don’t say to ourselves that we need to “solve reflective stability” for future humans, and if we don’t, then we have failed.
The future humans will learn new things we can’t imagine (presumably), and then make decisions, and the end-results might be good insofar as the humans have “good” dispositions and are able to solve coordination problems etc. We can and should try to anticipate and preempt problems. But we mostly can’t micromanage the outcome.
Likewise, I think it’s hard but possible to make an ASI with a “good” disposition, but if we do, we won’t be able to anticipate what it will wind up deciding to do, without being superintelligent ourselves. But that shouldn’t count as “alignment failure”.
This is a thing that could happen in principle, and I opportunistically do outreach to try to help on the margin [for my part I’m not spreading the message “these people suck” but rather “here’s why ASI is dangerous”], but getting to the place you describe seems extraordinarily unlikely to me. It seems to me that people have been raising the alarm on AI doom since Turing, and basically nobody cared until superintelligence seemed plausibly around the corner. Now that it is, people are somewhat more receptive to a narrowly-targeted restriction of that specific kind of ASI that they think is around the corner. Almost everyone on both sides continues to view “ASI is right around the corner” as a crux on the question of whether we should be worried about ASI at all.
I also think you’re insufficiently cynical about how deftly people can brand their capabilities research as “safety”-positive. Copying from here:
(Just yesterday I was poring over a paper by a guy who went from doing the worst kind of capabilities research in academia, to MATS (!!), and then straight back to the worst kind of capabilities research but now with higher salary at a nominally-AGI-safety-focused nonprofit. His personal website emphasizes how his research is helping AGI safety.)
Yeah I do think >20 years is <50% likely (although not <<50%). I disagree with the idea that a connectomics dataset is only relevant if ASI happens soon. For one thing, these are long-serial-time projects; the idea is to develop the experimental methods today in order to have a bare-bones human connectome in like 2040, or even 2050 or later if funding is less generous or technical difficulties arise. Then that dataset would be an input into research, and it may take years to digest, and we might want to iterate, etc. Anyway, if it’s HIA people in the distant future trying to solve alignment, I think they too would benefit from having this kind of data. Again, more generally, solving technical alignment requires serial time (1,2); even if HIA people will be working on technical alignment in the future, their odds will be better if we are also working on technical alignment right now.
To the point:
Maybe we should table this? I’m not sure what the cruxes are. If we were to continue, it might be helpful for us to make up numbers in order to find the most obvious disagreements (and/or change our numbers to become more internally coherent).
E.g. I’m like (these aren’t very considered numbers, may revise; I’m trying to illustrate the process-level point, but not completely made up):
The change in probability of solving alignment (that is, becoming able to make a friendly superhuman AGI) within the next 50 years from having a cortical human connectome is, IDK, <.03%, with or without HIA? The absolute probability currently is <3%, assuming no HIA?
The change in probability of making a dangerous PBE within the next 50 years from having a cortical human connectome is, I have no idea, .06%<x<1%? … etc.
(Maybe I should think this through more seriously anyway, though seems hard and IDK if likely to change most of my actions.)
More reactions:
I disagree with this. You don’t ~need to understand reflective stability because of anything about guarantees or a high bar, let alone impossible. You ~need to understand it because otherwise you don’t understand about what values even are. (Cf. re/ the importance of reflective stability: https://www.lesswrong.com/posts/Ht4JZtxngKwuQ7cDC/tsvibt-s-shortform?commentId=koeti9ygXB9wPLnnF ) I mean, this could be wrong. I don’t think I would even tell people to make “understand reflective stability” a focus of their research. I would say “understand what values are”. And then they would fail to consider reflective stability, perhaps because they are not considering the cores of intelligence which make reflective stability an especially sharp problem, and therefore they would probably be failing to understand what values are.
So what’s the deal with human successors? We ~know they have our values because they’re human! Figuring out how to make a de novo mind with values we like (or some kind of quasi-value structure that ends up doing things we like) is a whole different challenge.
In other words, the point I was trying to make was “if your research isn’t grappling with the cores of intelligence, it probably won’t understand [what sorts of things really drive the ultimate effects of a very intelligent mind]”.
(I’m kinda confused—I take it griceanly that you think I disagree with this or haven’t priced it in somehow, but also confused that you would think I disagree with this or haven’t priced it in somehow; if that’s actually true of me, I don’t get it yet from these paragraphs.)
Maybe, but not sure why you think this. I’m not saying you can get an infinite pause with no leaks with high probability. I’m saying you can “plausibly significantly push it back significantly”. Yeah, part of that would have to be extreme skepticism about safetywashing, which is not easy to implement.
Sure, let’s table. Except for one little thing:
I don’t think safetywashing is the right term for what I was referring to in the block-quote in the grandparent comment. @Adam Scholl coined “safetywashing” in analogy with “greenwashing”, e.g. Chevron buying a small environmental preserve and advertising it on TV. Some company has something it wants to do anyway, that’s generally bad for the environment, and they find some way to portray themselves as environmental anyway.
By contrast, I’m talking about something closer to environmentalists who oppose nuclear power and renewable energy projects, thus harming the environment, because they’re rationalizing, or just bad at thinking things through. There was never an ulterior motive; the motive was environmentalist all along, it’s just that environmentalists can be spectacularly counterproductive at achieving their own goals. For example that MATS guy I mentioned, the one trying to develop a new AI paradigm, he really thinks that what he’s doing will help pave the way to safer AI. But I think it’s just awful, pure timelines-shortening with no safety benefits whatsoever. We can speculate about whether he had a subconscious ulterior motive of some sort (e.g. constructing elegant theories about AI is fun and prestigious), but it’s equally plausible that he really did care about safety from the start, just in a spectacularly counterproductive way (according to me). And he started doing that work while in academia, so there wasn’t any organization-level ulterior motive either.