I think what you’re saying here ought to be uncontroversial. You’re saying that should a small group of technical people find themselves in a position of enormous influence, they ought to use that influence in an intelligent and responsible way, which may not look like immediately shirking that responsibility out of a sense that nobody should ever exert influence over the future.
I have the sense that in most societies over most of time, it was accepted that of course various small groups would at certain time find themselves in positions of enormous influence w.r.t. their society, and of course their responsibility in such a situation would be to not shirk that responsibility but to wisely and unilaterally choose a direction forward for their society, as required by the situation at hand.
In an ideal world, there would be some healthy and competent worldwide collaboration steering the transition to AGI
I have the sense that what would be ideal is for humanity to proceed with wisdom. The wisest moves we’ve made as a species to date (ending slavery? ending smallpox? landing on the moon?) didn’t particularly look like “worldwide collaborations”. Why, actually, do you say that the ideal would be a worldwide collaboration?
Third, though, I agree that it’s morally imperative that a small subset of humanity not directly decide how the future goes
Why should a small subset of humanity not directly decide how the future goes? The goal ought to be good decision-making, not large- or small-group decision making, and definitely not non-decision-making.
Of course the future should not be a tightly scripted screenplay of contemporary moral norms, but to decide that is to decide something about how the future goes. It’s not wrong to make such decisions, it’s just important to get such decisions right.
The wisest moves we’ve made as a species to date (ending slavery? ending smallpox? landing on the moon?) didn’t particularly look like “worldwide collaborations”.
I think Nate might’ve been thinking of things like:
Having all AGI research occur in one place is good (ceteris paribus), because then the AGI project can take as much time as it needs to figure out alignment, without worrying that some competitor will destroy the world with AGI if they go too slowly.
This is even truer if the global coordination is strong enough to prevent other x-risks (e.g., bio-weapons), so we don’t have to move faster to avoid those either.
In a perfect world, everyone would get some say in major decisions that affect their personal safety (e.g., via elected Scientist Representatives). This helps align incentives, relative to a world where anyone can unilaterally impose serious risks on others.
In a perfect world, larger collaborations shouldn’t perform worse than smaller ones, because larger collaborations should understand the dysfunctions of large collaborations and have policies and systems in place to avoid them (e.g., by automatically shrinking or siloing if needed).
I interpret Nate as making a concession to acknowledge the true and good aspects of the ‘but isn’t there something off about a random corporation or government doing all this?’ perspective, not as recommending that we (in real life) try to have the UN build AGI or whatever.
I think your pushback is good here, as a reminder that ‘but isn’t there something off about a random corporation or government doing all this?’ also often has less-reasonable intuitions going into it (example), and gets a weird level of emphasis considering how much more important other factors are, considering the track record of giant international collaborations, etc.
Why should a small subset of humanity not directly decide how the future goes? [...] Of course the future should not be a tightly scripted screenplay of contemporary moral norms, but to decide that is to decide something about how the future goes, and it’s not wrong to make such decisions, it’s just important to get such decisions right.
I’m guessing you two basically agree, and the “directly” in “a small subset of humanity not directly decide” is meant to exclude a “tightly scripted screenplay of contemporary moral norms”?
Nate also has the substantive belief that CEV-ish approaches are good, and (if he agrees with the Arbital page) that the base for CEV should be all humans. (The argument for this on Arbital is a combination of “it’s in the class of approaches that seem likeliest to work”, and “it seems easier to coordinate around, compared to the other approaches in that class”. E.g., I’d say that “run CEV on every human whose name starts with a vowel” is likely to produce the ~same outcome as “run CEV on every human”, but the latter is a better Schelling point.)
I imagine if Nate thought the best method for “not tightly scripting the future” were less “CEV based on all humans” and more “CEV based on the 1% smartest humans”, he’d care more about distinctions like the one you’re pointing at. It’s indeed the case that we shouldn’t toss away most of the future’s value just for the sake of performative egalitarianism: we should do the thing that actually makes sense.
Yeah I also have the sense that we mostly agree here.
I have the sense that CEV stands for, very roughly, “what such-and-such a person would do if they became extremely wise”, and the hope (which I think is a reasonable hope) is that there is a direction called “wisdom” such that if you move a person far enough in that direction then they become both intelligent and benevolent, and that this eventually doesn’t depend super much on where you started.
The tricky part is that we are in this time where we have the option of making some moves that might be quite disruptive, and we don’t yet have direct access to the wisdom that we would ideally use to guide our most significant decisions.
And the key question is really: what do you do if you come into a position of really significant influence, at a time when you don’t yet have the tools to access the CEV-level wisdom that you might later get? And some people say it’s flat-out antisocial to even contemplate taking any disruptive actions, while others say that given the particular configuration of the world right now and the particular problems we face, it actually seems plausible that a person in such a position of influence ought to seriously consider disruptive actions.
I really agree with the latter, and I also contend that it’s the more epistemically humble position, because you’re not saying that it’s for sure that a pivotal act should be performed, but just that it’s quite plausible given the specifics of the current world situation. The other side of the argument seems to be saying that no no no it’s definitely better not to do anything like that in anything like the current world situation.
I also contend that it’s the more epistemically humble position, because you’re not saying that it’s for sure that a pivotal act should be performed, but just that it’s quite plausible given the specifics of the current world situation
The thing I’d say in favor of this position is that I think it better fits the evidence. I think the problem with the opposing view is that it’s wrong, not that it’s more confident. E.g., if I learned that Nate assigns probability .9 to “a pivotal act is necessary” (for some operationalization of “necessary”) while Critch assigns probability .2 to “a pivotal act is necessary”, I wouldn’t go “ah, Critch is being more reasonable, since his probability is closer to .5″.
I agree with the rest of what you said, and I think this is a good way of framing the issue.
I’d add that I think discussion of this topic gets somewhat distorted by the fact that many people naturally track social consensus, and try to say the words they think will have the best influence on this consensus, rather than blurting out their relevant beliefs.
Many people are looking for a signal that stuff like this is OK to say in polite society, or many others are staking out a position “the case for this makes sense intellectually but there’s no way it will ever attract enough support, so I’ll preemptively oppose it in order to make my other arguments more politically acceptable”. (The latter, unfortunately, being a strategy that can serve as a self-fulfilling prophecy.)
Well the photos taken from the moon did seem to help a lot of people understand how vast and inhospitable the cosmos are, how small and fragile the Earth is, and how powerful—for better or worse—we humans had by that point become.
I think what you’re saying here ought to be uncontroversial. You’re saying that should a small group of technical people find themselves in a position of enormous influence, they ought to use that influence in an intelligent and responsible way, which may not look like immediately shirking that responsibility out of a sense that nobody should ever exert influence over the future.
I have the sense that in most societies over most of time, it was accepted that of course various small groups would at certain time find themselves in positions of enormous influence w.r.t. their society, and of course their responsibility in such a situation would be to not shirk that responsibility but to wisely and unilaterally choose a direction forward for their society, as required by the situation at hand.
I have the sense that what would be ideal is for humanity to proceed with wisdom. The wisest moves we’ve made as a species to date (ending slavery? ending smallpox? landing on the moon?) didn’t particularly look like “worldwide collaborations”. Why, actually, do you say that the ideal would be a worldwide collaboration?
Why should a small subset of humanity not directly decide how the future goes? The goal ought to be good decision-making, not large- or small-group decision making, and definitely not non-decision-making.
Of course the future should not be a tightly scripted screenplay of contemporary moral norms, but to decide that is to decide something about how the future goes. It’s not wrong to make such decisions, it’s just important to get such decisions right.
I think Nate might’ve been thinking of things like:
Having all AGI research occur in one place is good (ceteris paribus), because then the AGI project can take as much time as it needs to figure out alignment, without worrying that some competitor will destroy the world with AGI if they go too slowly.
This is even truer if the global coordination is strong enough to prevent other x-risks (e.g., bio-weapons), so we don’t have to move faster to avoid those either.
In a perfect world, everyone would get some say in major decisions that affect their personal safety (e.g., via elected Scientist Representatives). This helps align incentives, relative to a world where anyone can unilaterally impose serious risks on others.
In a perfect world, larger collaborations shouldn’t perform worse than smaller ones, because larger collaborations should understand the dysfunctions of large collaborations and have policies and systems in place to avoid them (e.g., by automatically shrinking or siloing if needed).
I interpret Nate as making a concession to acknowledge the true and good aspects of the ‘but isn’t there something off about a random corporation or government doing all this?’ perspective, not as recommending that we (in real life) try to have the UN build AGI or whatever.
I think your pushback is good here, as a reminder that ‘but isn’t there something off about a random corporation or government doing all this?’ also often has less-reasonable intuitions going into it (example), and gets a weird level of emphasis considering how much more important other factors are, considering the track record of giant international collaborations, etc.
I’m guessing you two basically agree, and the “directly” in “a small subset of humanity not directly decide” is meant to exclude a “tightly scripted screenplay of contemporary moral norms”?
Nate also has the substantive belief that CEV-ish approaches are good, and (if he agrees with the Arbital page) that the base for CEV should be all humans. (The argument for this on Arbital is a combination of “it’s in the class of approaches that seem likeliest to work”, and “it seems easier to coordinate around, compared to the other approaches in that class”. E.g., I’d say that “run CEV on every human whose name starts with a vowel” is likely to produce the ~same outcome as “run CEV on every human”, but the latter is a better Schelling point.)
I imagine if Nate thought the best method for “not tightly scripting the future” were less “CEV based on all humans” and more “CEV based on the 1% smartest humans”, he’d care more about distinctions like the one you’re pointing at. It’s indeed the case that we shouldn’t toss away most of the future’s value just for the sake of performative egalitarianism: we should do the thing that actually makes sense.
Yeah I also have the sense that we mostly agree here.
I have the sense that CEV stands for, very roughly, “what such-and-such a person would do if they became extremely wise”, and the hope (which I think is a reasonable hope) is that there is a direction called “wisdom” such that if you move a person far enough in that direction then they become both intelligent and benevolent, and that this eventually doesn’t depend super much on where you started.
The tricky part is that we are in this time where we have the option of making some moves that might be quite disruptive, and we don’t yet have direct access to the wisdom that we would ideally use to guide our most significant decisions.
And the key question is really: what do you do if you come into a position of really significant influence, at a time when you don’t yet have the tools to access the CEV-level wisdom that you might later get? And some people say it’s flat-out antisocial to even contemplate taking any disruptive actions, while others say that given the particular configuration of the world right now and the particular problems we face, it actually seems plausible that a person in such a position of influence ought to seriously consider disruptive actions.
I really agree with the latter, and I also contend that it’s the more epistemically humble position, because you’re not saying that it’s for sure that a pivotal act should be performed, but just that it’s quite plausible given the specifics of the current world situation. The other side of the argument seems to be saying that no no no it’s definitely better not to do anything like that in anything like the current world situation.
The thing I’d say in favor of this position is that I think it better fits the evidence. I think the problem with the opposing view is that it’s wrong, not that it’s more confident. E.g., if I learned that Nate assigns probability .9 to “a pivotal act is necessary” (for some operationalization of “necessary”) while Critch assigns probability .2 to “a pivotal act is necessary”, I wouldn’t go “ah, Critch is being more reasonable, since his probability is closer to .5″.
I agree with the rest of what you said, and I think this is a good way of framing the issue.
I’d add that I think discussion of this topic gets somewhat distorted by the fact that many people naturally track social consensus, and try to say the words they think will have the best influence on this consensus, rather than blurting out their relevant beliefs.
Many people are looking for a signal that stuff like this is OK to say in polite society, or many others are staking out a position “the case for this makes sense intellectually but there’s no way it will ever attract enough support, so I’ll preemptively oppose it in order to make my other arguments more politically acceptable”. (The latter, unfortunately, being a strategy that can serve as a self-fulfilling prophecy.)
Not the most important bit but, how is landing on the moon an example of a wise move?
Well the photos taken from the moon did seem to help a lot of people understand how vast and inhospitable the cosmos are, how small and fragile the Earth is, and how powerful—for better or worse—we humans had by that point become.