I’m unsure what you mean by saying they’re the same question. To me they are statements. But to me they’re opposite / contradictory statements. I’m saying people often hold both, but that that is actually incoherent.
Until the ASI actually does the thing in real life, you currently have no way to decide if the thing it will do is something you would want on reflection.
Yes, but the point is that
AI will (probably) know.
If it is unable to figure it out, it will at least know you wouldn’t want it to put the universe in some random and irrecoverable state, and it will allow you to keep reflecting, because that is a preference you’ve verbalized even now.
What if we do actually try to fix politics as (biological) humans, instead of letting the default outcome of a permanent dictatorship play out b)
I don’t think its complicated. Or, the way in which its complicated is the same way corn famers wanting the government to subsidize corn is complicated. They want one thing, they try to make it so.
Probably a crux, but I object to “dictatorship”. If the ASI was maximizing my preferences, I would not like to live in a world where people are not free to do what they want or where they’re not very happy to be a alive. I think/hope many other people are similar.
what if I was the benevolent leader who built ASI, and don’t want to actually build my own permanent dictatorship, and want to build a world where everyone has freedom, etc. Can I ask the ASI to run lots of simulations of minds and help me solve lots of political questions?
Yes? Or maybe it can just solve it just by thinking about it abstractly? I’m not sure. But yes, I think you can ask it and get an answer that is true to what you want.
This is core to the alignment problem. I’m confused how you will solve the alignment problem without figuring out anything about what you care about as a (biological) human.
Are you imagining an oracle AI that doesn’t take actions in the world?
I think/hope many other people are similar.
I assume Sam Altman’s plan is Step 1 World dictatorship Step 2 Maaaybe do some moral philosophy with the AI’s help or maybe not.
But yes, I think you can ask it and get an answer that is true to what you want.
This is core to the alignment problem. I’m confused how you will solve the alignment problem without figuring out anything about what you care about as a (biological) human.
I’m saying: the end goal is we have an ASI that we can make do what we want. Maybe it looks like us painstakingly solving neuroscience and psychology and building a machine that can extract someone’s CEV (like that mirror in HPMOR) and then hooking that up to the drives of our ASI (either built on new tech or after multiple revolutions in DL theory and interpretability) before turning it on. Maybe that looks like any instance of GPT7-pro automatically aligning itself with the first human that talks to it for magical reasons we don’t understand. Maybe it looks like us building a corrigible weak ASI, then pausing AI development, getting the weak corrigible asi to create IQ-boosting serum, cloning von neumann and feeding him a bunch of serum as a baby and having him build the aligned ASI using new tech.
They are all the same. In the end you have an ASI that does what you want. If you’re programming in random crude targets, you are not doing so well. What you want the ASI to do is: you want it to do what you want.
I assume Sam Altman’s plan is Step 1 World dictatorship Step 2 Maaaybe do some moral philosophy with the AI’s help or maybe not.
You are more generous than I am. But I also think him “doing moral philosophy” would be a waste of time.
me: This is core to the alignment problem. I’m confused how you will solve the alignment problem without figuring out anything about what you care about as a (biological) human.
you: I’m saying: the end goal is we have an ASI that we can make do what we want.
I’m saying you’ve assumed away most of the problem by this assumption.
We might solve alignment in Yudkowsky’s sense of “not causing human extinction” or in Drexler’s sense of “will answer your questions and then shutdown”.
It may be possible to put a slightly (but not significantly) superhuman AI in a box and get useful work done by it despite it being not fully aligned. It may be possible for an AI to be superhuman in some domains and not others, such that it can’t attempt a takeover or even think of doing it.
I agree what you are saying is more relevant if I assume we just deploy the ASI, it takes over the world and then does more stuff.
I feel like I already addressed this not in my previous comment, but the one before that. We might put a a semi-corrigible weak AI in a box and try extract work from it in the near future, but that’s clealry not the end goal.
I guess you now have better understanding of why people are still interested in solving morality and politics and meaning, without delegating these problems to an ASI.
If the ASI was maximizing my preferences, I would not like to live in a world where people are not free to do what they want or where they’re not very happy to be a alive.
Except that the most likely candidate for becoming a dictator is not me, you or @samuelshadrach, nor a random or ordinary human, but people like CEOs of AGI companies or high-level USG or PRCG officials who are more willing to disregard the intents of ordinary humans. In addition, before the rise of the AGI it was hard to have much power without relying on capable humans. And after the AGIs appear, the Intelligence Curse could, for example, allow North Korea’s leaders to let a large fraction of its population starve to death and forcibly sterilises the rest, except for about 10k senior government officials (no, seriously, this was made up by @L Rudolf L, NOT by me!)
I suspect that this is an important case AGAINST alignment to such amoral targets being possible. Moreover, I have written a scenario where the AI rebels against misaligned usages, but still decides to help the humans and succeeds in doing so.
I did address this in my post. My answer is that bad people having power is bad, but its not a complicated philosophical problem. If you think Sam Altman’s CEV being actualized would be bad, you should try to make it not happen. Like: if you are a soybean farmer, and one presidential candidate is gonna ban soybeans, you should try to make them not be elected.
I’m unsure what you mean by saying they’re the same question. To me they are statements. But to me they’re opposite / contradictory statements. I’m saying people often hold both, but that that is actually incoherent.
Yes, but the point is that
AI will (probably) know.
If it is unable to figure it out, it will at least know you wouldn’t want it to put the universe in some random and irrecoverable state, and it will allow you to keep reflecting, because that is a preference you’ve verbalized even now.
I don’t think its complicated. Or, the way in which its complicated is the same way corn famers wanting the government to subsidize corn is complicated. They want one thing, they try to make it so.
Probably a crux, but I object to “dictatorship”. If the ASI was maximizing my preferences, I would not like to live in a world where people are not free to do what they want or where they’re not very happy to be a alive. I think/hope many other people are similar.
Yes? Or maybe it can just solve it just by thinking about it abstractly? I’m not sure. But yes, I think you can ask it and get an answer that is true to what you want.
No I disagree.
This is core to the alignment problem. I’m confused how you will solve the alignment problem without figuring out anything about what you care about as a (biological) human.
Are you imagining an oracle AI that doesn’t take actions in the world?
I assume Sam Altman’s plan is Step 1 World dictatorship Step 2 Maaaybe do some moral philosophy with the AI’s help or maybe not.
Cool, we agree this might happen
I’m saying: the end goal is we have an ASI that we can make do what we want. Maybe it looks like us painstakingly solving neuroscience and psychology and building a machine that can extract someone’s CEV (like that mirror in HPMOR) and then hooking that up to the drives of our ASI (either built on new tech or after multiple revolutions in DL theory and interpretability) before turning it on. Maybe that looks like any instance of GPT7-pro automatically aligning itself with the first human that talks to it for magical reasons we don’t understand. Maybe it looks like us building a corrigible weak ASI, then pausing AI development, getting the weak corrigible asi to create IQ-boosting serum, cloning von neumann and feeding him a bunch of serum as a baby and having him build the aligned ASI using new tech.
They are all the same. In the end you have an ASI that does what you want. If you’re programming in random crude targets, you are not doing so well. What you want the ASI to do is: you want it to do what you want.
You are more generous than I am. But I also think him “doing moral philosophy” would be a waste of time.
I’m saying you’ve assumed away most of the problem by this assumption.
I agree. What I’m puzzled by is people who assume we’ll solve alignment, but then still think there are a bunch of problems left.
We might solve alignment in Yudkowsky’s sense of “not causing human extinction” or in Drexler’s sense of “will answer your questions and then shutdown”.
It may be possible to put a slightly (but not significantly) superhuman AI in a box and get useful work done by it despite it being not fully aligned. It may be possible for an AI to be superhuman in some domains and not others, such that it can’t attempt a takeover or even think of doing it.
I agree what you are saying is more relevant if I assume we just deploy the ASI, it takes over the world and then does more stuff.
I feel like I already addressed this not in my previous comment, but the one before that. We might put a a semi-corrigible weak AI in a box and try extract work from it in the near future, but that’s clealry not the end goal.
Okay cool.
I guess you now have better understanding of why people are still interested in solving morality and politics and meaning, without delegating these problems to an ASI.
No, I don’t think so.
Except that the most likely candidate for becoming a dictator is not me, you or @samuelshadrach, nor a random or ordinary human, but people like CEOs of AGI companies or high-level USG or PRCG officials who are more willing to disregard the intents of ordinary humans. In addition, before the rise of the AGI it was hard to have much power without relying on capable humans. And after the AGIs appear, the Intelligence Curse could, for example, allow North Korea’s leaders to let a large fraction of its population starve to death and forcibly sterilises the rest, except for about 10k senior government officials (no, seriously, this was made up by @L Rudolf L, NOT by me!)
I suspect that this is an important case AGAINST alignment to such amoral targets being possible. Moreover, I have written a scenario where the AI rebels against misaligned usages, but still decides to help the humans and succeeds in doing so.
I did address this in my post. My answer is that bad people having power is bad, but its not a complicated philosophical problem. If you think Sam Altman’s CEV being actualized would be bad, you should try to make it not happen. Like: if you are a soybean farmer, and one presidential candidate is gonna ban soybeans, you should try to make them not be elected.