I mostly agree with this post, in the sense that if someone says “I’m going to do WBE research”, and they don’t say any details, my guess will be that they’re going to do something that I judge net-harmful. But it’s a weakly-held guess and I would mostly want to ask follow-up questions.
Because I also think there exist certain activities that are simultaneously “a part of WBE research” and “net positive”.
I think whole-brain connectomics would fit in that Venn-diagram overlap, and I think that certain more specific connectomics projects would be not only net positive, but indeed one of the things I’m most enthusiastic about. (See Connectomics seems great from an AI x-risk perspective which you cited.)
The specific area where I disagree with the OP is, umm, the sorta generic nature of the arguments. I don’t want to talk about whether something is or is not a PBE, but rather what it is and what it enables and what can be learned from it.
If the plan is “delay AGI as long as possible”, then I guess the situation is simpler because you can be universally opposed to everything in neuroscience and AI, and the only question is whether you’re extremely opposed to it (e.g. unravel the secrets of human intelligence) versus only infinitesimally opposed (speed up the autocomplete on google maps). But a different plan is “prepare for AGI by solving the alignment problem”, and then we face a much harder task of trying to sort out what are the critical pathways for capabilities and for alignment, and push on the technical directions where you can make progress on the latter much more than the former.
I guess that might be the crux: “delay AGI as long as possible” versus “try to solve the alignment problem”? I think we’re both in favor of both, but differ a lot in how much of one we’re happy to trade off for an increment of the other. Like, the probability of capabilities side-effects and timeline-shortening from even my dream connectomics project would be more than zero, and I’m sad about that, but I think the probability is sufficiently low that it’s well worth it, because the project is pretty likely to teach us important things about the alignment problem. Happy to argue about what the tradeoff curve should be, but maybe I should stop at this point and let you respond.
The specific area where I disagree with the OP is, umm, the sorta generic nature of the arguments.
That’s true & fair; my argument involves very strong / broad statements. I largely stand by its breadth.
I don’t want to talk about whether something is or is not a PBE, but rather what it is and what it enables and what can be learned from it.
I think this makes sense. However:
My argument is indeed a lot less strong when applied to some specific kind of research that could be called “some part of a WBE research program”. My argument is centrally addressed at a plan or research program whose main goal is “make a human WBE, and then that WBE saves the world” or similar. If you want to say that some specific research is fine, that could be argued. E.g. I agree that connectomics by itself is at least plausibly basically fine / doesn’t contribute meaningfully to AI capabilities.
My main response is to argue that either this specific kind of research doesn’t get you a true human WBE that has a shot at preventing unaligned AGI, or else this specific kind of research is actually dangerously contributing to AI capabilities. If there’s a research project / organization with the goal of creating true human WBEs, they are a fortiori saying that they will not only do a specific kind of research that doesn’t get you a true human WBE.
As a secondary point, the context of the whole WBE research community makes the situation worse. For example, even if a connectome alone doesn’t add AI capabilities, having a connectome vs. not might contribute significantly to AI capabilities if the research community also has a serviceable partial structure-to-function mapping, or neural learning dynamics, or similar.
If the plan is “delay AGI as long as possible”, then I guess the situation is simpler because you can be universally opposed to everything in neuroscience and AI, and the only question is whether you’re extremely opposed to it (e.g. unravel the secrets of human intelligence) versus only infinitesimally opposed (speed up the autocomplete on google maps).
Yeah, I think this is roughly where I am.
(Though, it doesn’t make me net-opposed to all neuroscience and AI in practice. E.g. improving harnesses to improve certain kinds of narrow performance for AI models seems probably to contribute basically nothing to AI capabilities and to in some cases make AI more useful for good work. (Though I’m pretty sharply opposed to basically any non-negligible AI capabilities advances.) For neuroscience, I’m only opposed to stuff that non-negligibly erodes the security-through-obscurity that we currently enjoy for human brain algorithms. I mean, I’d make tradeoffs here. BCIs may require some significant degree of improving some kinds of access to the brain; but that could easily be worth it if there were a good way to increase human intelligence via BCIs (though I don’t actually see how). General neuroscience, especially that might tell us how to do HIA, seems great.)
But a different plan is “prepare for AGI by solving the alignment problem”, and then we face a much harder task of trying to sort out what are the critical pathways for capabilities and for alignment, and push on the technical directions where you can make progress on the latter much more than the former.
Right, I think this task is probably too hard for at least two reasons:
The alignment problem is really hard. (My guess is we have a significant quantitative disagreement here? Like, you think you might be able to get a good chunk of the needed info by reading brain scans? Whereas I’m quite skeptical of that.)
All natural ways (including agent foundations!) to approach the alignment problem that might actually succeed involves understanding more about how intelligence works, which makes it dangerous. Maybe not net-harmful, but this isn’t immediately obvious in any of the cases I know about, I think.
(I don’t think this extreme set of beliefs I have should be quite a crux for the OP’s argument; I think that the OP’s argument goes through under somewhat weaker assumptions, though IDK what the boundary is.)
I guess that might be the crux: “delay AGI as long as possible” versus “try to solve the alignment problem”? I think we’re both in favor of both, but differ a lot in how much of one we’re happy to trade off for an increment of the other.
This is sounding maybe right? I’m not sure if it should actually be a crux. If I thought there was a non-negligible chance of solving alignment; and I also thought that a big connectomics project would noticeably increase that chance; actually I’m still unsure even in that world. Like, this kinda reminds me of “we should make frontier AI systems and then study them in order to solve alignment, and accept the tradeoff that we’re accelerating capabilities a bit, in the name of differential acceleration”. It’s not the same structure of justification, but it’s close enough.
Like, the probability of capabilities side-effects and timeline-shortening from even my dream connectomics project would be more than zero, and I’m sad about that, but I think the probability is sufficiently low that it’s well worth it, because the project is pretty likely to teach us important things about the alignment problem. Happy to argue about what the tradeoff curve should be, but maybe I should stop at this point and let you respond.
As I mentioned earlier in this comment, my arguments don’t primarily address a connectomics-only project. I think I know enough to say that a connectomics-only project by itself wouldn’t produce a WBE; I don’t know whether it would contribute to AI capabilities, though if the MICrONS dataset is already high-quality that’s some kind of argument (though maybe human cortex is different enough to add new capabilities ideas, or add new shitty-but-capable-after-training emulations, IDK!). The trick I get to pull when I say “project aimed at creating WBEs” is to screen off my uncertainty about which exact kinds of research contribute more or less to the WBE’s eventual capabilities, and therefore potentially contribute to AI capabilities beforehand. I’m much more confident that if a project gets a true human WBE, then it got dangerous capabilities some time beforehand, than I am of any specific type of research producing capabilities.
All that said, we might be able to disagree productively about connectomics? Not sure. I mean, I’ll have to defer to you (or AI or the future) a lot, so maybe it’s more like an interview. I would wonder:
Is the MICrONS dataset basically already a perfect mouse connectome, for the small piece of cortex? Fable seems to say it’s far from perfect, and doesn’t get much molecular data (ion channels, receptors, neurotransmitters, neuromodulation, etc.). Would a better one not make it significantly more likely someone could emulate a piece of cortex (maybe after also getting structure-function mapping and/or other dynamics)? Would that not be dangerous?
If a small piece of mouse cortex isn’t dangerous, wouldn’t a whole mouse brain be? What about a small piece of human cortex? (Is your dream connectomics project “do everything other than the cortex”? Is specifically avoiding the cortex, as part of a large connectomics project, something that actual projects have ever seriously considered?)
Your worries about AI capabilities are somewhat focused on the danger of reverse-engineering—extracting algorithms, right?
Wouldn’t more cortical connectomes make that easier? Or if not, why do you think you can get alignment algorithms but not learning / intelligence algorithms? (Maybe by not mapping the cortex?)
Why not also worry about low-fidelity emulations of parts of cortex, scaled up and trained for performance?
If there’s a research project / organization with the goal of creating true human WBEs…
We’re in agreement on this part, which I understand is the main point of the OP.
But also happy to keep talking about the connectomics thing (even if it’s kinda a tangent). Here it is in bold: The rest of this comment is a tangent that should not be construed as disagreement with the OP.
this kinda reminds me of “we should make frontier AI systems and then study them in order to solve alignment, and accept the tradeoff that we’re accelerating capabilities a bit, in the name of differential acceleration”. It’s not the same structure of justification, but it’s close enough.
I agree that many people have said that they are trading off a pinch of timelines-shortening for a pound of alignment progress, but where the results have been bad. I think you’re reacting to that history by saying “anyone suggesting that they’re doing such a thing is not to be trusted, they’re probably just rationalizing or wrong”.
But that’s an overreaction. You have to actually evaluate the argument. Otherwise you’re making the same mistake as the people who say “previous apocalypse prophecies haven’t come true, so I’ll just ignore them all”, and so on.
It’s impossible to publish work on alignment if you’re avoiding even the faintest possibility that there will be even a tiny timelines-shortening externality. E.g. you yourself are a coauthor on the logical induction paper, which clearly-to-me fails that standard. (But I don’t want to accuse you of hypocrisy because maybe your opinions have changed in the intervening decade?)
It’s fine (indeed, good) to have a default-skeptical allergic reaction to people saying that they’re OK risking timelines-shortening and we’re just haggling over the price. I think the price should be high but not infinite, and I think we should skeptically scrutinize the argument but not rule it out a priori. (And I really think I’m more neurotic about this issue than almost anyone!!)
In other words, we should observe the failures and then try to do better. (Relevant meme :-P )
For example, some of the arguments that people have made on this topic are just really bad. Like, they’ll figure out X, and note that X is required for alignment, and so they publish X, without pausing to note that X is also required for building AGI in the first place, and so someone is bound to publish X sooner or later, and that will still be before AGI. I’ve been ranting about this particular mistake for years.
By contrast, I’m working in RL & search—see RL & search is a terrifying way to build AGI (an FAQ). If we divide RL & search into “what’s the reward / cost / objective function” versus “how do we build an AI that optimizes that”, then the former is (almost entirely) not required for creating AGI, and indeed usually not even slightly helpful for creating AGI. That’s why practically everyone in those fields ignores the question altogether. (See We need a field of Reward Function Design.) So progress on that question doesn’t have to happen before AGI, or ever. Instead, if we want progress, we have to push on it.
But a different plan is “prepare for AGI by solving the alignment problem”, and then we face a much harder task of trying to sort out what are the critical pathways for capabilities and for alignment, and push on the technical directions where you can make progress on the latter much more than the former.
Right, I think this task is probably too hard for at least two reasons…
I won’t argue with “it’s hard”—I expect us to fail, and I think we’re likely doomed. But I don’t see an alternative that isn’t also likely to fail. We need to be judging options on whether they’re the least-bad choice, not whether we’re happy about them.
On the margin there are ways to lengthen and shorten timelines, but lengthening them to infinity does not seem to be an option. How would you get to infinity? IMO it would require shutting down AI-related basic research everywhere on Earth. That really seems extremely unlikely to me, or at least sufficiently unlikely that we should not be planning around it. (If someone wants to make a Hail-Mary attempt, fine.) As points of comparison, it’s still perfectly legal everywhere on Earth to do basic R&D that might enable novel inexpensive uranium enrichment technology, or that might enable making super-plagues in your garage. It’s nobody’s job to check, and never has been. Also, the USA once tried to put export controls on an algorithm, and it was not only totally ineffective but struck down as unconstitutional.
Given that, we need to plan for ASI alignment to happen sooner or later. I presume your response would be that human-intelligence-augmentation will make this less perilous. I have a couple responses. First, as I was arguing here, I expect that the HIA humans will be doing better alignment research and better capability research, leaving us in a similar overall situation. Second, the HIA people will not be radical superintelligence, and will still have limitations on serial time (1,2) and may still benefit from relevant experimental data. Thus, doing ASI alignment research right now, and kicking off relevant long-serial-time research programs like connectomics right now, is still important. Finally, if we compare worlds-where-HIA-succeeds versus worlds-where-it-doesn’t (because ASI comes too soon, or because of unforeseen biological obstacles, etc.), I interpret your stance as “we should roll over and die in the latter worlds, if fighting would even infinitesimally harm are chances in the former worlds”. If so, that seems like a bad trade to me. I don’t think the latter worlds are that much more doomed than the former, nor that they’re less likely.
I would wonder:…
I think the cortex is running a big uniform-ish learning algorithm, and the concern (from my perspective) / exciting opportunity (from other people’s perspective) is that people will figure out how it works well enough to run something functionally similar on a computer. It’s true that there are things that MICrONS didn’t measure, but a large-scale project like a whole-brain connectome is very unlikely to measure something about a cortical column that a smaller-scale study had not already measured long ago on a small PI grant. There are just thousands of studies like this one where they look at some kind of motif that you find throughout the cortex, and try to learn everything about the receptors and functional relationships and so on. And we already have atlases of gene expression for every kind of cell in the mouse cortex, which tell us (in principle) about neuropeptide receptors and much else. Etc. In other words, I think there are a lot of things you’re concerned about, where from my perspective, that horse has left the barn long ago.
I think we shouldn’t be so concerned about emulation-without-understanding, not only for the various reasons here but also because it’s totally impractical. As I discuss here, if you simulate the individual transistors in a chip, the fastest modern computer available can’t even emulate Pong (1972) in real time. So it’s all about understanding the logic of the algorithm.
And more specifically, I think we shouldn’t be concerned with what an animal or person knows, but rather how an animal or person figures out something new. After all, we already have LLMs, which are virtuosos at applying already-existing knowledge. If you take a snapshot of a cortex, you can (maybe in principle) decode what the animal knows, but that wouldn’t be very helpful for unraveling how the animal figures out something new.
Likewise, a “low-fidelity emulation” would not be a concern if it didn’t incorporate the ability to figure out something that’s never been figured out before. Again, we already have LLMs, which are a kind of low-fidelity emulation of what people already know.
Meanwhile, the “Steering Subsystem” is the part of the brain containing the core logic of human innate drives. If it’s possible for humans to be nice, and to make the future better, then it’s possible for a sufficiently-human-like AGI to be nice, and to make the future better. But reverse-engineering (the relevant parts of) the Steering Subsystem is quite different from reverse-engineering the cortical algorithm. It’s not a big uniform-ish learning algorithm, rather it’s a tangled web of a few thousand idiosyncratic neuron groups doing idiosyncratic things like tracking loneliness. Every year another ~10 of those neuron groups gets carefully characterized by some grad student in one of the handful of groups that specialize in this type of work (e.g. Catherine Dulac group or Dayu Lin group). We need some kind of mass-production to make headway faster. Hence my advocacy for connectomics. The connectomics wouldn’t replace the artisanal studies, but rather complement them by helping us know which cell groups are the most critical to study. This is still a heavily-theory-laden challenge—a connectome dataset doesn’t just spontaneously announce what hypothalamic cell groups are responsible for compassion—but I have ideas of what to look for.
I agree that many people have said that they are trading off a pinch of timelines-shortening for a pound of alignment progress, but where the results have been bad. I think you’re reacting to that history by saying “anyone suggesting that they’re doing such a thing is not to be trusted, they’re probably just rationalizing or wrong”.
I’m almost saying that, yeah. Also note that, beyond just the history of people saying that with bad results, there’s also the history of grantmakers not being able to tell that that would happen, and funding significantly more harmful stuff. That’s an additional knowledge problem. In other words, if you’re about to fund X, consider that it’s not feasible to fund X while having a good impact. It’s sad because, yes, somewhere in the used car lot there’s probably a good deal. It’s just that you actually can’t tell, so you’ll have to go someplace else.
But that’s an overreaction. You have to actually evaluate the argument.
Evaluating the object-level argument is helpful of course, and may turn up good used cars. But I’ll stand by keeping the prior with a pretty substantial degree of control over the decisions, even after you’ve done some amount of object-level evaluation.
Otherwise you’re making the same mistake as the people who say “previous apocalypse prophecies haven’t come true, so I’ll just ignore them all”, and so on.
I think they’re doing a somewhat rational move! And then you’re supposed to argue the object-level convincingly enough to overcome that.
It’s impossible to publish work on alignment if you’re avoiding even the faintest possibility that there will be even a tiny timelines-shortening externality. E.g. you yourself are a coauthor on the logical induction paper, which clearly-to-me fails that standard. (But I don’t want to accuse you of hypocrisy because maybe your opinions have changed in the intervening decade?)
(I was not in fact remotely cruxy for that paper, but in any case:) Yes, I agree that it fails the standard, and in fact probably shouldn’t have been published (which I certainly didn’t say or think at the time). That’s a change of mind with benefit of hindsight, yeah.
If we divide RL & search into “what’s the reward / cost / objective function” versus “how do we build an AI that optimizes that”, then the former is (almost entirely) not required for creating AGI, and indeed usually not even slightly helpful for creating AGI.
With this as an example, I’m like “yeah that sounds probably safe to work on, and worth someone doing, but also it won’t work partly because it’s safe to work on—in other words, it probably will not really address problems such as reflective stability, which are ~necessary to address, because it is avoiding thinking about the cores of how intelligence works, and therefore probably can’t figure out the cores of reflective stability in intelligent minds”.
On the margin there are ways to lengthen and shorten timelines, but lengthening them to infinity does not seem to be an option. How would you get to infinity? IMO it would require shutting down AI-related basic research everywhere on Earth. That really seems extremely unlikely to me,
I agree with this literally, but it sounds like we may have disagreements in space in between? I think one could plausibly significantly push it back significantly through a combination of international treaty, professional norms, academic norms, social stigma, public consensus, and similar. Yes, you can’t stop people doing conceptual research. But you can make them afraid to talk about it or share it, and make them feel guilty for doing it. That doesn’t last anywhere close to forever, but it could quite significantly increase timelines.
I interpret your stance as “we should roll over and die in the latter worlds, if fighting would even infinitesimally harm are chances in the former worlds”.
This is pretty straw of course. My stance is
The highest priority is getting and maintaining an AI research stop quickly and firmly. (I think we agree.)
WBE research in particular, as a research goal, erodes an AI stop too much. (I think we agree?)
Alignment is very hard. (We agree.) This makes it even worse to burn commons of a stop by adding more capabilities.
AGI is more likely to come >20 years from now than <20 years from now. (I assume you disagree because no one agrees with this.) Therefore burning a 30 year pause down to a 28 year pause is not worth a +.1% chance of alignment within 10 years. (Sorry these may be straw numbers.)
Alignment is so hard that it is a lower priority than HIA research. (It sounds like you disagree because you don’t think HIA helps much? Though this is a tangent.)
Connectomics by itself seems to me like a relatively safe kind of WBE-related research. If I had to guess I would not guess that by itself a big high quality cortex connectome would be very dangerous. But I do not feel confident about this. Maybe a shitty, not-much-learning electrical-model cortex, plus added ML, would actually be dangerous. I don’t think this is an infinitesimal harm, I think it’s a pretty small but noticeable harm in expectation. I could be argued into thinking that it’s lower, but I’m a little skeptical that you know that. Like, I agree with your take about the cortical connectome being basically learned parameters rather than the learning algorithm. I’m just like, “Ok, but how surprised would I be if it turned out that actually doing a human cortical connectome does unlock dangerous AI? Kinda suprised but not utterly gobsmacked.”. For example, some of the learned params may encode crystallized meta-cognition; and then when you take that, simulate it at the 30-second level, scale it up 100x, and do ML training as if it’s an LLM, and also try out “neural learning algorithms”, you get something super duper dangerous. It just wouldn’t be that surprising.
Connectomics doesn’t seem that helpful for alignment.
Maybe non-cortical connectomics is both safer and is the main usefulness for alignment? (Is this your favorite version of connectomics?)
With this as an example, I’m like “yeah that sounds probably safe to work on, and worth someone doing, but also it won’t work partly because it’s safe to work on—in other words, it probably will not really address problems such as reflective stability, which are ~necessary to address, because it is avoiding thinking about the cores of how intelligence works, and therefore probably can’t figure out the cores of reflective stability in intelligent minds”.
Based on reading your blog over the past years, I feel like the thing you have in mind for what a solution to alignment would look like is not just hard but impossible and unnecessary. Like, suppose (as a thought experiment) ASI were impossible in our universe, along with other x-risks. We would be passing the torch to the next generation of humans, just as every previous generation has. What will that next generation do with their powers? We don’t really know. It could be good, it could be bad. We can and should fight to try to make it better. But we don’t say to ourselves that we need to “solve reflective stability” for future humans, and if we don’t, then we have failed.
The future humans will learn new things we can’t imagine (presumably), and then make decisions, and the end-results might be good insofar as the humans have “good” dispositions and are able to solve coordination problems etc. We can and should try to anticipate and preempt problems. But we mostly can’t micromanage the outcome.
Likewise, I think it’s hard but possible to make an ASI with a “good” disposition, but if we do, we won’t be able to anticipate what it will wind up deciding to do, without being superintelligent ourselves. But that shouldn’t count as “alignment failure”.
I think one could plausibly significantly push it back significantly through a combination of international treaty, professional norms, academic norms, social stigma, public consensus, and similar. Yes, you can’t stop people doing conceptual research. But you can make them afraid to talk about it or share it, and make them feel guilty for doing it. That doesn’t last anywhere close to forever, but it could quite significantly increase timelines.
This is a thing that could happen in principle, and I opportunistically do outreach to try to help on the margin [for my part I’m not spreading the message “these people suck” but rather “here’s why ASI is dangerous”], but getting to the place you describe seems extraordinarily unlikely to me. It seems to me that people have been raising the alarm on AI doom since Turing, and basically nobody cared until superintelligence seemed plausibly around the corner. Now that it is, people are somewhat more receptive to a narrowly-targeted restriction of that specific kind of ASI that they think is around the corner. Almost everyone on both sides continues to view “ASI is right around the corner” as a crux on the question of whether we should be worried about ASI at all.
I also think you’re insufficiently cynical about how deftly people can brand their capabilities research as “safety”-positive. Copying from here:
Thus, those researchers I mentioned, who are messing around with new paradigms on arXiv, are in a great position to twist “Pause AI” type public advocacy into support for what they’re doing!
“You don’t like LLMs?”, the non-LLM AGI capabilities researchers say to the Pause AI people, “Well how about that! I don’t like LLMs either! Clearly we are on the same team!”
This is not idle speculation—almost everyone that I can think of who is doing the most dangerous kind of AI capabilities research, the kind aiming to develop a new more-powerful-than-LLM AI paradigm, is already branding their work in a way that vibes with safety. For example, see here where I push back on someone using the word “controllability” to talk about his work advancing AI capabilities beyond the limits of LLMs. Ditto for “robustness” (example), “adaptability” (e.g. in the paper I was criticizing here), and even “interpretability” (details).
I think these people are generally sincere but mistaken, and I expect that, just as they have fooled themselves, they will also successfully fool their friends, their colleagues, and government regulators. Well, the government regulators hardly matter anyway, since regulating the activity of “playing with toy models, and publishing stuff on arXiv and GitHub” is a hell of an ask—I think it’s so unlikely to happen that it’s a waste of time to even talk about it, even if it were a good idea all-things-considered.
(Just yesterday I was poring over a paper by a guy who went from doing the worst kind of capabilities research in academia, to MATS (!!), and then straight back to the worst kind of capabilities research but now with higher salary at a nominally-AGI-safety-focused nonprofit. His personal website emphasizes how his research is helping AGI safety.)
AGI is more likely to come >20 years from now than <20 years from now. (I assume you disagree because no one agrees with this.) Therefore burning a 30 year pause down to a 28 year pause is not worth a +.1% chance of alignment within 10 years. (Sorry these may be straw numbers.)
Yeah I do think >20 years is <50% likely (although not <<50%). I disagree with the idea that a connectomics dataset is only relevant if ASI happens soon. For one thing, these are long-serial-time projects; the idea is to develop the experimental methods today in order to have a bare-bones human connectome in like 2040, or even 2050 or later if funding is less generous or technical difficulties arise. Then that dataset would be an input into research, and it may take years to digest, and we might want to iterate, etc. Anyway, if it’s HIA people in the distant future trying to solve alignment, I think they too would benefit from having this kind of data. Again, more generally, solving technical alignment requires serial time (1,2); even if HIA people will be working on technical alignment in the future, their odds will be better if we are also working on technical alignment right now.
Maybe we should table this? I’m not sure what the cruxes are. If we were to continue, it might be helpful for us to make up numbers in order to find the most obvious disagreements (and/or change our numbers to become more internally coherent).
E.g. I’m like (these aren’t very considered numbers, may revise; I’m trying to illustrate the process-level point, but not completely made up):
The change in probability of solving alignment (that is, becoming able to make a friendly superhuman AGI) within the next 50 years from having a cortical human connectome is, IDK, <.03%, with or without HIA? The absolute probability currently is <3%, assuming no HIA?
The change in probability of making a dangerous PBE within the next 50 years from having a cortical human connectome is, I have no idea, .06%<x<1%?
… etc.
(Maybe I should think this through more seriously anyway, though seems hard and IDK if likely to change most of my actions.)
More reactions:
Based on reading your blog over the past years, I feel like the thing you have in mind for what a solution to alignment would look like is not just hard but impossible and unnecessary. Like, suppose (as a thought experiment) ASI were impossible in our universe, along with other x-risks. We would be passing the torch to the next generation of humans, just as every previous generation has. What will that next generation do with their powers? We don’t really know. It could be good, it could be bad. We can and should fight to try to make it better. But we don’t say to ourselves that we need to “solve reflective stability” for future humans, and if we don’t, then we have failed.
I disagree with this. You don’t ~need to understand reflective stability because of anything about guarantees or a high bar, let alone impossible. You ~need to understand it because otherwise you don’t understand about what values even are. (Cf. re/ the importance of reflective stability: https://www.lesswrong.com/posts/Ht4JZtxngKwuQ7cDC/tsvibt-s-shortform?commentId=koeti9ygXB9wPLnnF ) I mean, this could be wrong. I don’t think I would even tell people to make “understand reflective stability” a focus of their research. I would say “understand what values are”. And then they would fail to consider reflective stability, perhaps because they are not considering the cores of intelligence which make reflective stability an especially sharp problem, and therefore they would probably be failing to understand what values are.
So what’s the deal with human successors? We ~know they have our values because they’re human! Figuring out how to make a de novo mind with values we like (or some kind of quasi-value structure that ends up doing things we like) is a whole different challenge.
In other words, the point I was trying to make was “if your research isn’t grappling with the cores of intelligence, it probably won’t understand [what sorts of things really drive the ultimate effects of a very intelligent mind]”.
The future humans will learn new things we can’t imagine (presumably), and then make decisions, and the end-results might be good insofar as the humans have “good” dispositions and are able to solve coordination problems etc. We can and should try to anticipate and preempt problems. But we mostly can’t micromanage the outcome.
Likewise, I think it’s hard but possible to make an ASI with a “good” disposition, but if we do, we won’t be able to anticipate what it will wind up deciding to do, without being superintelligent ourselves. But that shouldn’t count as “alignment failure”.
(I’m kinda confused—I take it griceanly that you think I disagree with this or haven’t priced it in somehow, but also confused that you would think I disagree with this or haven’t priced it in somehow; if that’s actually true of me, I don’t get it yet from these paragraphs.)
I also think you’re insufficiently cynical about how deftly people can brand their capabilities research as “safety”-positive.
Maybe, but not sure why you think this. I’m not saying you can get an infinite pause with no leaks with high probability. I’m saying you can “plausibly significantly push it back significantly”. Yeah, part of that would have to be extreme skepticism about safetywashing, which is not easy to implement.
I don’t think safetywashing is the right term for what I was referring to in the block-quote in the grandparent comment. @Adam Schollcoined “safetywashing” in analogy with “greenwashing”, e.g. Chevron buying a small environmental preserve and advertising it on TV. Some company has something it wants to do anyway, that’s generally bad for the environment, and they find some way to portray themselves as environmental anyway.
By contrast, I’m talking about something closer to environmentalists who oppose nuclear power and renewable energy projects, thus harming the environment, because they’re rationalizing, or just bad at thinking things through. There was never an ulterior motive; the motive was environmentalist all along, it’s just that environmentalists can be spectacularly counterproductive at achieving their own goals. For example that MATS guy I mentioned, the one trying to develop a new AI paradigm, he really thinks that what he’s doing will help pave the way to safer AI. But I think it’s just awful, pure timelines-shortening with no safety benefits whatsoever. We can speculate about whether he had a subconscious ulterior motive of some sort (e.g. constructing elegant theories about AI is fun and prestigious), but it’s equally plausible that he really did care about safety from the start, just in a spectacularly counterproductive way (according to me). And he started doing that work while in academia, so there wasn’t any organization-level ulterior motive either.
I mostly agree with this post, in the sense that if someone says “I’m going to do WBE research”, and they don’t say any details, my guess will be that they’re going to do something that I judge net-harmful. But it’s a weakly-held guess and I would mostly want to ask follow-up questions.
Because I also think there exist certain activities that are simultaneously “a part of WBE research” and “net positive”.
I think whole-brain connectomics would fit in that Venn-diagram overlap, and I think that certain more specific connectomics projects would be not only net positive, but indeed one of the things I’m most enthusiastic about. (See Connectomics seems great from an AI x-risk perspective which you cited.)
The specific area where I disagree with the OP is, umm, the sorta generic nature of the arguments. I don’t want to talk about whether something is or is not a PBE, but rather what it is and what it enables and what can be learned from it.
If the plan is “delay AGI as long as possible”, then I guess the situation is simpler because you can be universally opposed to everything in neuroscience and AI, and the only question is whether you’re extremely opposed to it (e.g. unravel the secrets of human intelligence) versus only infinitesimally opposed (speed up the autocomplete on google maps). But a different plan is “prepare for AGI by solving the alignment problem”, and then we face a much harder task of trying to sort out what are the critical pathways for capabilities and for alignment, and push on the technical directions where you can make progress on the latter much more than the former.
I guess that might be the crux: “delay AGI as long as possible” versus “try to solve the alignment problem”? I think we’re both in favor of both, but differ a lot in how much of one we’re happy to trade off for an increment of the other. Like, the probability of capabilities side-effects and timeline-shortening from even my dream connectomics project would be more than zero, and I’m sad about that, but I think the probability is sufficiently low that it’s well worth it, because the project is pretty likely to teach us important things about the alignment problem. Happy to argue about what the tradeoff curve should be, but maybe I should stop at this point and let you respond.
That’s true & fair; my argument involves very strong / broad statements. I largely stand by its breadth.
I think this makes sense. However:
My argument is indeed a lot less strong when applied to some specific kind of research that could be called “some part of a WBE research program”. My argument is centrally addressed at a plan or research program whose main goal is “make a human WBE, and then that WBE saves the world” or similar. If you want to say that some specific research is fine, that could be argued. E.g. I agree that connectomics by itself is at least plausibly basically fine / doesn’t contribute meaningfully to AI capabilities.
My main response is to argue that either this specific kind of research doesn’t get you a true human WBE that has a shot at preventing unaligned AGI, or else this specific kind of research is actually dangerously contributing to AI capabilities. If there’s a research project / organization with the goal of creating true human WBEs, they are a fortiori saying that they will not only do a specific kind of research that doesn’t get you a true human WBE.
As a secondary point, the context of the whole WBE research community makes the situation worse. For example, even if a connectome alone doesn’t add AI capabilities, having a connectome vs. not might contribute significantly to AI capabilities if the research community also has a serviceable partial structure-to-function mapping, or neural learning dynamics, or similar.
Yeah, I think this is roughly where I am.
(Though, it doesn’t make me net-opposed to all neuroscience and AI in practice. E.g. improving harnesses to improve certain kinds of narrow performance for AI models seems probably to contribute basically nothing to AI capabilities and to in some cases make AI more useful for good work. (Though I’m pretty sharply opposed to basically any non-negligible AI capabilities advances.) For neuroscience, I’m only opposed to stuff that non-negligibly erodes the security-through-obscurity that we currently enjoy for human brain algorithms. I mean, I’d make tradeoffs here. BCIs may require some significant degree of improving some kinds of access to the brain; but that could easily be worth it if there were a good way to increase human intelligence via BCIs (though I don’t actually see how). General neuroscience, especially that might tell us how to do HIA, seems great.)
Right, I think this task is probably too hard for at least two reasons:
The alignment problem is really hard. (My guess is we have a significant quantitative disagreement here? Like, you think you might be able to get a good chunk of the needed info by reading brain scans? Whereas I’m quite skeptical of that.)
All natural ways (including agent foundations!) to approach the alignment problem that might actually succeed involves understanding more about how intelligence works, which makes it dangerous. Maybe not net-harmful, but this isn’t immediately obvious in any of the cases I know about, I think.
(I don’t think this extreme set of beliefs I have should be quite a crux for the OP’s argument; I think that the OP’s argument goes through under somewhat weaker assumptions, though IDK what the boundary is.)
This is sounding maybe right? I’m not sure if it should actually be a crux. If I thought there was a non-negligible chance of solving alignment; and I also thought that a big connectomics project would noticeably increase that chance; actually I’m still unsure even in that world. Like, this kinda reminds me of “we should make frontier AI systems and then study them in order to solve alignment, and accept the tradeoff that we’re accelerating capabilities a bit, in the name of differential acceleration”. It’s not the same structure of justification, but it’s close enough.
As I mentioned earlier in this comment, my arguments don’t primarily address a connectomics-only project. I think I know enough to say that a connectomics-only project by itself wouldn’t produce a WBE; I don’t know whether it would contribute to AI capabilities, though if the MICrONS dataset is already high-quality that’s some kind of argument (though maybe human cortex is different enough to add new capabilities ideas, or add new shitty-but-capable-after-training emulations, IDK!). The trick I get to pull when I say “project aimed at creating WBEs” is to screen off my uncertainty about which exact kinds of research contribute more or less to the WBE’s eventual capabilities, and therefore potentially contribute to AI capabilities beforehand. I’m much more confident that if a project gets a true human WBE, then it got dangerous capabilities some time beforehand, than I am of any specific type of research producing capabilities.
All that said, we might be able to disagree productively about connectomics? Not sure. I mean, I’ll have to defer to you (or AI or the future) a lot, so maybe it’s more like an interview. I would wonder:
Is the MICrONS dataset basically already a perfect mouse connectome, for the small piece of cortex? Fable seems to say it’s far from perfect, and doesn’t get much molecular data (ion channels, receptors, neurotransmitters, neuromodulation, etc.). Would a better one not make it significantly more likely someone could emulate a piece of cortex (maybe after also getting structure-function mapping and/or other dynamics)? Would that not be dangerous?
If a small piece of mouse cortex isn’t dangerous, wouldn’t a whole mouse brain be? What about a small piece of human cortex? (Is your dream connectomics project “do everything other than the cortex”? Is specifically avoiding the cortex, as part of a large connectomics project, something that actual projects have ever seriously considered?)
Your worries about AI capabilities are somewhat focused on the danger of reverse-engineering—extracting algorithms, right?
Wouldn’t more cortical connectomes make that easier? Or if not, why do you think you can get alignment algorithms but not learning / intelligence algorithms? (Maybe by not mapping the cortex?)
Why not also worry about low-fidelity emulations of parts of cortex, scaled up and trained for performance?
We’re in agreement on this part, which I understand is the main point of the OP.
But also happy to keep talking about the connectomics thing (even if it’s kinda a tangent). Here it is in bold: The rest of this comment is a tangent that should not be construed as disagreement with the OP.
I agree that many people have said that they are trading off a pinch of timelines-shortening for a pound of alignment progress, but where the results have been bad. I think you’re reacting to that history by saying “anyone suggesting that they’re doing such a thing is not to be trusted, they’re probably just rationalizing or wrong”.
But that’s an overreaction. You have to actually evaluate the argument. Otherwise you’re making the same mistake as the people who say “previous apocalypse prophecies haven’t come true, so I’ll just ignore them all”, and so on.
It’s impossible to publish work on alignment if you’re avoiding even the faintest possibility that there will be even a tiny timelines-shortening externality. E.g. you yourself are a coauthor on the logical induction paper, which clearly-to-me fails that standard. (But I don’t want to accuse you of hypocrisy because maybe your opinions have changed in the intervening decade?)
It’s fine (indeed, good) to have a default-skeptical allergic reaction to people saying that they’re OK risking timelines-shortening and we’re just haggling over the price. I think the price should be high but not infinite, and I think we should skeptically scrutinize the argument but not rule it out a priori. (And I really think I’m more neurotic about this issue than almost anyone!!)
In other words, we should observe the failures and then try to do better. (Relevant meme :-P )
For example, some of the arguments that people have made on this topic are just really bad. Like, they’ll figure out X, and note that X is required for alignment, and so they publish X, without pausing to note that X is also required for building AGI in the first place, and so someone is bound to publish X sooner or later, and that will still be before AGI. I’ve been ranting about this particular mistake for years.
By contrast, I’m working in RL & search—see RL & search is a terrifying way to build AGI (an FAQ). If we divide RL & search into “what’s the reward / cost / objective function” versus “how do we build an AI that optimizes that”, then the former is (almost entirely) not required for creating AGI, and indeed usually not even slightly helpful for creating AGI. That’s why practically everyone in those fields ignores the question altogether. (See We need a field of Reward Function Design.) So progress on that question doesn’t have to happen before AGI, or ever. Instead, if we want progress, we have to push on it.
I won’t argue with “it’s hard”—I expect us to fail, and I think we’re likely doomed. But I don’t see an alternative that isn’t also likely to fail. We need to be judging options on whether they’re the least-bad choice, not whether we’re happy about them.
On the margin there are ways to lengthen and shorten timelines, but lengthening them to infinity does not seem to be an option. How would you get to infinity? IMO it would require shutting down AI-related basic research everywhere on Earth. That really seems extremely unlikely to me, or at least sufficiently unlikely that we should not be planning around it. (If someone wants to make a Hail-Mary attempt, fine.) As points of comparison, it’s still perfectly legal everywhere on Earth to do basic R&D that might enable novel inexpensive uranium enrichment technology, or that might enable making super-plagues in your garage. It’s nobody’s job to check, and never has been. Also, the USA once tried to put export controls on an algorithm, and it was not only totally ineffective but struck down as unconstitutional.
Given that, we need to plan for ASI alignment to happen sooner or later. I presume your response would be that human-intelligence-augmentation will make this less perilous. I have a couple responses. First, as I was arguing here, I expect that the HIA humans will be doing better alignment research and better capability research, leaving us in a similar overall situation. Second, the HIA people will not be radical superintelligence, and will still have limitations on serial time (1,2) and may still benefit from relevant experimental data. Thus, doing ASI alignment research right now, and kicking off relevant long-serial-time research programs like connectomics right now, is still important. Finally, if we compare worlds-where-HIA-succeeds versus worlds-where-it-doesn’t (because ASI comes too soon, or because of unforeseen biological obstacles, etc.), I interpret your stance as “we should roll over and die in the latter worlds, if fighting would even infinitesimally harm are chances in the former worlds”. If so, that seems like a bad trade to me. I don’t think the latter worlds are that much more doomed than the former, nor that they’re less likely.
I think the cortex is running a big uniform-ish learning algorithm, and the concern (from my perspective) / exciting opportunity (from other people’s perspective) is that people will figure out how it works well enough to run something functionally similar on a computer. It’s true that there are things that MICrONS didn’t measure, but a large-scale project like a whole-brain connectome is very unlikely to measure something about a cortical column that a smaller-scale study had not already measured long ago on a small PI grant. There are just thousands of studies like this one where they look at some kind of motif that you find throughout the cortex, and try to learn everything about the receptors and functional relationships and so on. And we already have atlases of gene expression for every kind of cell in the mouse cortex, which tell us (in principle) about neuropeptide receptors and much else. Etc. In other words, I think there are a lot of things you’re concerned about, where from my perspective, that horse has left the barn long ago.
I think we shouldn’t be so concerned about emulation-without-understanding, not only for the various reasons here but also because it’s totally impractical. As I discuss here, if you simulate the individual transistors in a chip, the fastest modern computer available can’t even emulate Pong (1972) in real time. So it’s all about understanding the logic of the algorithm.
And more specifically, I think we shouldn’t be concerned with what an animal or person knows, but rather how an animal or person figures out something new. After all, we already have LLMs, which are virtuosos at applying already-existing knowledge. If you take a snapshot of a cortex, you can (maybe in principle) decode what the animal knows, but that wouldn’t be very helpful for unraveling how the animal figures out something new.
Likewise, a “low-fidelity emulation” would not be a concern if it didn’t incorporate the ability to figure out something that’s never been figured out before. Again, we already have LLMs, which are a kind of low-fidelity emulation of what people already know.
Meanwhile, the “Steering Subsystem” is the part of the brain containing the core logic of human innate drives. If it’s possible for humans to be nice, and to make the future better, then it’s possible for a sufficiently-human-like AGI to be nice, and to make the future better. But reverse-engineering (the relevant parts of) the Steering Subsystem is quite different from reverse-engineering the cortical algorithm. It’s not a big uniform-ish learning algorithm, rather it’s a tangled web of a few thousand idiosyncratic neuron groups doing idiosyncratic things like tracking loneliness. Every year another ~10 of those neuron groups gets carefully characterized by some grad student in one of the handful of groups that specialize in this type of work (e.g. Catherine Dulac group or Dayu Lin group). We need some kind of mass-production to make headway faster. Hence my advocacy for connectomics. The connectomics wouldn’t replace the artisanal studies, but rather complement them by helping us know which cell groups are the most critical to study. This is still a heavily-theory-laden challenge—a connectome dataset doesn’t just spontaneously announce what hypothalamic cell groups are responsible for compassion—but I have ideas of what to look for.
I’m almost saying that, yeah. Also note that, beyond just the history of people saying that with bad results, there’s also the history of grantmakers not being able to tell that that would happen, and funding significantly more harmful stuff. That’s an additional knowledge problem. In other words, if you’re about to fund X, consider that it’s not feasible to fund X while having a good impact. It’s sad because, yes, somewhere in the used car lot there’s probably a good deal. It’s just that you actually can’t tell, so you’ll have to go someplace else.
Evaluating the object-level argument is helpful of course, and may turn up good used cars. But I’ll stand by keeping the prior with a pretty substantial degree of control over the decisions, even after you’ve done some amount of object-level evaluation.
I think they’re doing a somewhat rational move! And then you’re supposed to argue the object-level convincingly enough to overcome that.
(I was not in fact remotely cruxy for that paper, but in any case:) Yes, I agree that it fails the standard, and in fact probably shouldn’t have been published (which I certainly didn’t say or think at the time). That’s a change of mind with benefit of hindsight, yeah.
With this as an example, I’m like “yeah that sounds probably safe to work on, and worth someone doing, but also it won’t work partly because it’s safe to work on—in other words, it probably will not really address problems such as reflective stability, which are ~necessary to address, because it is avoiding thinking about the cores of how intelligence works, and therefore probably can’t figure out the cores of reflective stability in intelligent minds”.
I agree with this literally, but it sounds like we may have disagreements in space in between? I think one could plausibly significantly push it back significantly through a combination of international treaty, professional norms, academic norms, social stigma, public consensus, and similar. Yes, you can’t stop people doing conceptual research. But you can make them afraid to talk about it or share it, and make them feel guilty for doing it. That doesn’t last anywhere close to forever, but it could quite significantly increase timelines.
This is pretty straw of course. My stance is
The highest priority is getting and maintaining an AI research stop quickly and firmly. (I think we agree.)
WBE research in particular, as a research goal, erodes an AI stop too much. (I think we agree?)
Alignment is very hard. (We agree.) This makes it even worse to burn commons of a stop by adding more capabilities.
AGI is more likely to come >20 years from now than <20 years from now. (I assume you disagree because no one agrees with this.) Therefore burning a 30 year pause down to a 28 year pause is not worth a +.1% chance of alignment within 10 years. (Sorry these may be straw numbers.)
Alignment is so hard that it is a lower priority than HIA research. (It sounds like you disagree because you don’t think HIA helps much? Though this is a tangent.)
Connectomics by itself seems to me like a relatively safe kind of WBE-related research. If I had to guess I would not guess that by itself a big high quality cortex connectome would be very dangerous. But I do not feel confident about this. Maybe a shitty, not-much-learning electrical-model cortex, plus added ML, would actually be dangerous. I don’t think this is an infinitesimal harm, I think it’s a pretty small but noticeable harm in expectation. I could be argued into thinking that it’s lower, but I’m a little skeptical that you know that. Like, I agree with your take about the cortical connectome being basically learned parameters rather than the learning algorithm. I’m just like, “Ok, but how surprised would I be if it turned out that actually doing a human cortical connectome does unlock dangerous AI? Kinda suprised but not utterly gobsmacked.”. For example, some of the learned params may encode crystallized meta-cognition; and then when you take that, simulate it at the 30-second level, scale it up 100x, and do ML training as if it’s an LLM, and also try out “neural learning algorithms”, you get something super duper dangerous. It just wouldn’t be that surprising.
Connectomics doesn’t seem that helpful for alignment.
Maybe non-cortical connectomics is both safer and is the main usefulness for alignment? (Is this your favorite version of connectomics?)
Based on reading your blog over the past years, I feel like the thing you have in mind for what a solution to alignment would look like is not just hard but impossible and unnecessary. Like, suppose (as a thought experiment) ASI were impossible in our universe, along with other x-risks. We would be passing the torch to the next generation of humans, just as every previous generation has. What will that next generation do with their powers? We don’t really know. It could be good, it could be bad. We can and should fight to try to make it better. But we don’t say to ourselves that we need to “solve reflective stability” for future humans, and if we don’t, then we have failed.
The future humans will learn new things we can’t imagine (presumably), and then make decisions, and the end-results might be good insofar as the humans have “good” dispositions and are able to solve coordination problems etc. We can and should try to anticipate and preempt problems. But we mostly can’t micromanage the outcome.
Likewise, I think it’s hard but possible to make an ASI with a “good” disposition, but if we do, we won’t be able to anticipate what it will wind up deciding to do, without being superintelligent ourselves. But that shouldn’t count as “alignment failure”.
This is a thing that could happen in principle, and I opportunistically do outreach to try to help on the margin [for my part I’m not spreading the message “these people suck” but rather “here’s why ASI is dangerous”], but getting to the place you describe seems extraordinarily unlikely to me. It seems to me that people have been raising the alarm on AI doom since Turing, and basically nobody cared until superintelligence seemed plausibly around the corner. Now that it is, people are somewhat more receptive to a narrowly-targeted restriction of that specific kind of ASI that they think is around the corner. Almost everyone on both sides continues to view “ASI is right around the corner” as a crux on the question of whether we should be worried about ASI at all.
I also think you’re insufficiently cynical about how deftly people can brand their capabilities research as “safety”-positive. Copying from here:
(Just yesterday I was poring over a paper by a guy who went from doing the worst kind of capabilities research in academia, to MATS (!!), and then straight back to the worst kind of capabilities research but now with higher salary at a nominally-AGI-safety-focused nonprofit. His personal website emphasizes how his research is helping AGI safety.)
Yeah I do think >20 years is <50% likely (although not <<50%). I disagree with the idea that a connectomics dataset is only relevant if ASI happens soon. For one thing, these are long-serial-time projects; the idea is to develop the experimental methods today in order to have a bare-bones human connectome in like 2040, or even 2050 or later if funding is less generous or technical difficulties arise. Then that dataset would be an input into research, and it may take years to digest, and we might want to iterate, etc. Anyway, if it’s HIA people in the distant future trying to solve alignment, I think they too would benefit from having this kind of data. Again, more generally, solving technical alignment requires serial time (1,2); even if HIA people will be working on technical alignment in the future, their odds will be better if we are also working on technical alignment right now.
To the point:
Maybe we should table this? I’m not sure what the cruxes are. If we were to continue, it might be helpful for us to make up numbers in order to find the most obvious disagreements (and/or change our numbers to become more internally coherent).
E.g. I’m like (these aren’t very considered numbers, may revise; I’m trying to illustrate the process-level point, but not completely made up):
The change in probability of solving alignment (that is, becoming able to make a friendly superhuman AGI) within the next 50 years from having a cortical human connectome is, IDK, <.03%, with or without HIA? The absolute probability currently is <3%, assuming no HIA?
The change in probability of making a dangerous PBE within the next 50 years from having a cortical human connectome is, I have no idea, .06%<x<1%? … etc.
(Maybe I should think this through more seriously anyway, though seems hard and IDK if likely to change most of my actions.)
More reactions:
I disagree with this. You don’t ~need to understand reflective stability because of anything about guarantees or a high bar, let alone impossible. You ~need to understand it because otherwise you don’t understand about what values even are. (Cf. re/ the importance of reflective stability: https://www.lesswrong.com/posts/Ht4JZtxngKwuQ7cDC/tsvibt-s-shortform?commentId=koeti9ygXB9wPLnnF ) I mean, this could be wrong. I don’t think I would even tell people to make “understand reflective stability” a focus of their research. I would say “understand what values are”. And then they would fail to consider reflective stability, perhaps because they are not considering the cores of intelligence which make reflective stability an especially sharp problem, and therefore they would probably be failing to understand what values are.
So what’s the deal with human successors? We ~know they have our values because they’re human! Figuring out how to make a de novo mind with values we like (or some kind of quasi-value structure that ends up doing things we like) is a whole different challenge.
In other words, the point I was trying to make was “if your research isn’t grappling with the cores of intelligence, it probably won’t understand [what sorts of things really drive the ultimate effects of a very intelligent mind]”.
(I’m kinda confused—I take it griceanly that you think I disagree with this or haven’t priced it in somehow, but also confused that you would think I disagree with this or haven’t priced it in somehow; if that’s actually true of me, I don’t get it yet from these paragraphs.)
Maybe, but not sure why you think this. I’m not saying you can get an infinite pause with no leaks with high probability. I’m saying you can “plausibly significantly push it back significantly”. Yeah, part of that would have to be extreme skepticism about safetywashing, which is not easy to implement.
Sure, let’s table. Except for one little thing:
I don’t think safetywashing is the right term for what I was referring to in the block-quote in the grandparent comment. @Adam Scholl coined “safetywashing” in analogy with “greenwashing”, e.g. Chevron buying a small environmental preserve and advertising it on TV. Some company has something it wants to do anyway, that’s generally bad for the environment, and they find some way to portray themselves as environmental anyway.
By contrast, I’m talking about something closer to environmentalists who oppose nuclear power and renewable energy projects, thus harming the environment, because they’re rationalizing, or just bad at thinking things through. There was never an ulterior motive; the motive was environmentalist all along, it’s just that environmentalists can be spectacularly counterproductive at achieving their own goals. For example that MATS guy I mentioned, the one trying to develop a new AI paradigm, he really thinks that what he’s doing will help pave the way to safer AI. But I think it’s just awful, pure timelines-shortening with no safety benefits whatsoever. We can speculate about whether he had a subconscious ulterior motive of some sort (e.g. constructing elegant theories about AI is fun and prestigious), but it’s equally plausible that he really did care about safety from the start, just in a spectacularly counterproductive way (according to me). And he started doing that work while in academia, so there wasn’t any organization-level ulterior motive either.