Maybe we should table this? I’m not sure what the cruxes are. If we were to continue, it might be helpful for us to make up numbers in order to find the most obvious disagreements (and/or change our numbers to become more internally coherent).
E.g. I’m like (these aren’t very considered numbers, may revise; I’m trying to illustrate the process-level point, but not completely made up):
The change in probability of solving alignment (that is, becoming able to make a friendly superhuman AGI) within the next 50 years from having a cortical human connectome is, IDK, <.03%, with or without HIA? The absolute probability currently is <3%, assuming no HIA?
The change in probability of making a dangerous PBE within the next 50 years from having a cortical human connectome is, I have no idea, .06%<x<1%?
… etc.
(Maybe I should think this through more seriously anyway, though seems hard and IDK if likely to change most of my actions.)
More reactions:
Based on reading your blog over the past years, I feel like the thing you have in mind for what a solution to alignment would look like is not just hard but impossible and unnecessary. Like, suppose (as a thought experiment) ASI were impossible in our universe, along with other x-risks. We would be passing the torch to the next generation of humans, just as every previous generation has. What will that next generation do with their powers? We don’t really know. It could be good, it could be bad. We can and should fight to try to make it better. But we don’t say to ourselves that we need to “solve reflective stability” for future humans, and if we don’t, then we have failed.
I disagree with this. You don’t ~need to understand reflective stability because of anything about guarantees or a high bar, let alone impossible. You ~need to understand it because otherwise you don’t understand about what values even are. (Cf. re/ the importance of reflective stability: https://www.lesswrong.com/posts/Ht4JZtxngKwuQ7cDC/tsvibt-s-shortform?commentId=koeti9ygXB9wPLnnF ) I mean, this could be wrong. I don’t think I would even tell people to make “understand reflective stability” a focus of their research. I would say “understand what values are”. And then they would fail to consider reflective stability, perhaps because they are not considering the cores of intelligence which make reflective stability an especially sharp problem, and therefore they would probably be failing to understand what values are.
So what’s the deal with human successors? We ~know they have our values because they’re human! Figuring out how to make a de novo mind with values we like (or some kind of quasi-value structure that ends up doing things we like) is a whole different challenge.
In other words, the point I was trying to make was “if your research isn’t grappling with the cores of intelligence, it probably won’t understand [what sorts of things really drive the ultimate effects of a very intelligent mind]”.
The future humans will learn new things we can’t imagine (presumably), and then make decisions, and the end-results might be good insofar as the humans have “good” dispositions and are able to solve coordination problems etc. We can and should try to anticipate and preempt problems. But we mostly can’t micromanage the outcome.
Likewise, I think it’s hard but possible to make an ASI with a “good” disposition, but if we do, we won’t be able to anticipate what it will wind up deciding to do, without being superintelligent ourselves. But that shouldn’t count as “alignment failure”.
(I’m kinda confused—I take it griceanly that you think I disagree with this or haven’t priced it in somehow, but also confused that you would think I disagree with this or haven’t priced it in somehow; if that’s actually true of me, I don’t get it yet from these paragraphs.)
I also think you’re insufficiently cynical about how deftly people can brand their capabilities research as “safety”-positive.
Maybe, but not sure why you think this. I’m not saying you can get an infinite pause with no leaks with high probability. I’m saying you can “plausibly significantly push it back significantly”. Yeah, part of that would have to be extreme skepticism about safetywashing, which is not easy to implement.
I don’t think safetywashing is the right term for what I was referring to in the block-quote in the grandparent comment. @Adam Schollcoined “safetywashing” in analogy with “greenwashing”, e.g. Chevron buying a small environmental preserve and advertising it on TV. Some company has something it wants to do anyway, that’s generally bad for the environment, and they find some way to portray themselves as environmental anyway.
By contrast, I’m talking about something closer to environmentalists who oppose nuclear power and renewable energy projects, thus harming the environment, because they’re rationalizing, or just bad at thinking things through. There was never an ulterior motive; the motive was environmentalist all along, it’s just that environmentalists can be spectacularly counterproductive at achieving their own goals. For example that MATS guy I mentioned, the one trying to develop a new AI paradigm, he really thinks that what he’s doing will help pave the way to safer AI. But I think it’s just awful, pure timelines-shortening with no safety benefits whatsoever. We can speculate about whether he had a subconscious ulterior motive of some sort (e.g. constructing elegant theories about AI is fun and prestigious), but it’s equally plausible that he really did care about safety from the start, just in a spectacularly counterproductive way (according to me). And he started doing that work while in academia, so there wasn’t any organization-level ulterior motive either.
To the point:
Maybe we should table this? I’m not sure what the cruxes are. If we were to continue, it might be helpful for us to make up numbers in order to find the most obvious disagreements (and/or change our numbers to become more internally coherent).
E.g. I’m like (these aren’t very considered numbers, may revise; I’m trying to illustrate the process-level point, but not completely made up):
The change in probability of solving alignment (that is, becoming able to make a friendly superhuman AGI) within the next 50 years from having a cortical human connectome is, IDK, <.03%, with or without HIA? The absolute probability currently is <3%, assuming no HIA?
The change in probability of making a dangerous PBE within the next 50 years from having a cortical human connectome is, I have no idea, .06%<x<1%? … etc.
(Maybe I should think this through more seriously anyway, though seems hard and IDK if likely to change most of my actions.)
More reactions:
I disagree with this. You don’t ~need to understand reflective stability because of anything about guarantees or a high bar, let alone impossible. You ~need to understand it because otherwise you don’t understand about what values even are. (Cf. re/ the importance of reflective stability: https://www.lesswrong.com/posts/Ht4JZtxngKwuQ7cDC/tsvibt-s-shortform?commentId=koeti9ygXB9wPLnnF ) I mean, this could be wrong. I don’t think I would even tell people to make “understand reflective stability” a focus of their research. I would say “understand what values are”. And then they would fail to consider reflective stability, perhaps because they are not considering the cores of intelligence which make reflective stability an especially sharp problem, and therefore they would probably be failing to understand what values are.
So what’s the deal with human successors? We ~know they have our values because they’re human! Figuring out how to make a de novo mind with values we like (or some kind of quasi-value structure that ends up doing things we like) is a whole different challenge.
In other words, the point I was trying to make was “if your research isn’t grappling with the cores of intelligence, it probably won’t understand [what sorts of things really drive the ultimate effects of a very intelligent mind]”.
(I’m kinda confused—I take it griceanly that you think I disagree with this or haven’t priced it in somehow, but also confused that you would think I disagree with this or haven’t priced it in somehow; if that’s actually true of me, I don’t get it yet from these paragraphs.)
Maybe, but not sure why you think this. I’m not saying you can get an infinite pause with no leaks with high probability. I’m saying you can “plausibly significantly push it back significantly”. Yeah, part of that would have to be extreme skepticism about safetywashing, which is not easy to implement.
Sure, let’s table. Except for one little thing:
I don’t think safetywashing is the right term for what I was referring to in the block-quote in the grandparent comment. @Adam Scholl coined “safetywashing” in analogy with “greenwashing”, e.g. Chevron buying a small environmental preserve and advertising it on TV. Some company has something it wants to do anyway, that’s generally bad for the environment, and they find some way to portray themselves as environmental anyway.
By contrast, I’m talking about something closer to environmentalists who oppose nuclear power and renewable energy projects, thus harming the environment, because they’re rationalizing, or just bad at thinking things through. There was never an ulterior motive; the motive was environmentalist all along, it’s just that environmentalists can be spectacularly counterproductive at achieving their own goals. For example that MATS guy I mentioned, the one trying to develop a new AI paradigm, he really thinks that what he’s doing will help pave the way to safer AI. But I think it’s just awful, pure timelines-shortening with no safety benefits whatsoever. We can speculate about whether he had a subconscious ulterior motive of some sort (e.g. constructing elegant theories about AI is fun and prestigious), but it’s equally plausible that he really did care about safety from the start, just in a spectacularly counterproductive way (according to me). And he started doing that work while in academia, so there wasn’t any organization-level ulterior motive either.