This post reads to me as if you’ve mostly extrapolated the beliefs and arguments of “orthodox alignment theorists” from small snippets and ended up with a wildly oversimplified strawman. Then you’ve re-derived mostly orthodox rat beliefs and arguments and presented them as a devastating counterargument.
I think I’m fairly orthodox as lesswrongers go, and I agree with most of the statements and arguments you made in this post. I only have one or two disagreements toward the end of the post.
One example of many, because I found this one particularly funny:
There is an assumption, in orthogonalist circles, that these cycles are completely costless for the agent in question.
The fact that maintaining goals across ontology shifts and self-modification takes careful effort is basically the core of the orthodox alignment-looks-hard worldview. You must be just making up an opposing worldview here? Where are the orthogonalist circles who say this?
>The fact that maintaining goals across ontology shifts and self-modification takes careful effort is basically the core of the orthodox alignment-looks-hard worldview
It’s one argument. The other is “the totality of human value needs to be hardcoded into the AI, you only get one attempt , and if you make the smallest mistake, everyone dies*.
>You must be just making up an opposing worldview here?
>Where are the orthogonalist circles who say this?
It’s long been the case that OT is mostly used as a pro-doom argument, both in the pro- paperclipping and anti moral realism senses.
It’s also true that the OT has some anti-doom implications, and that’s much less.publicised, and therefore worth pointing out.
The other is “the totality of human value needs to be hardcoded into the AI, you only get one attempt , and if you make the smallest mistake, everyone dies*.
I agree something like this is one branch of the argument, but in my mind it’s a relatively small branch. The main branch focuses on bounded corrigible AI and the main difficulty there is instability. There are other branches for non-hardcoding human values, and different targets that aren’t human values.
It’s long been the case that OT is mostly used as a pro-doom argument, both in the pro- paperclipping and anti moral realism senses.
It’s also true that the OT has some anti-doom implications, and that’s much less.publicised, and therefore worth pointing out.
I’m not sure what point you’re making here, what are the implications you’re referring to?
I’m not sure what point you’re making here, what are the implications you’re referring to?
I thought he meant this part:
But I reject (3), and I am much less willing to grant (2). If value means the production of richer cognition, agency, understanding, beauty, and evaluative structure, it is far from obvious that the current human brain is the only physical substrate capable of steering toward it.
i am very well aware the orthodox alignment line is that to maintain aligned goals across ontologies is very difficult! that’s why im surprised by this difficulty being set aside for strong orthogonality and misaligned goals.
as for the paragraph you cite: of course it’s a preposterous notion! but how else would you explain the fact that the arbitrary-terminal-goal-agent can emerge victorious from those who devote all their cycles to simply following instrumental drives?
at any rate: the thesis I wanted to dispatch is: there is the significant risk that an agent will reach superintelligence while ultimately continuing to pursuit a valueless goal. if you are trying to tell me that this was never a claim, i am very grateful; let us note it down on the wiki to prevent further confusion while I go on towards demolishing (or discovering I had imagined) the concept of AI psychia is. I’m on a schedule so this kind of help is deeply appreciated.
i am very well aware the orthodox alignment line is that to maintain aligned goals across ontologies is very difficult!
Then who’s in the orthogonalist circles you referred to? Or did you make them up?
but how else would you explain
When you try to derive someone’s premises from their conclusions, you still have to go and check whether you got it right. When people have different beliefs from you, it’s easy to slip up in this kind of reasoning. In my case it’s explained by me believing that selection isn’t always the main thing determining terminal goals (especially at finite times, or when there are other powerful optimizers interfering with selection).
there is the significant risk that an agent will reach superintelligence while ultimately continuing to pursuit a valueless goal
I endorse this statement. But as per this yud tweet, it might be useful to disentangle the orthogonality thesis from the chance of misalignment, because misalignment involves a stack of additional arguments. It’d be better to directly engage with the strong form of the orthogonality thesis as described in the second sentence of the wiki page and with the arguments for it, rather than making them up your own versions of these.
I’m surprised that you say this is hard? Humans maintain our goals across ontologies super easily; it’s barely an inconvenience for us. Like, physics undergrads don’t usually change their tastes in art or stop having sex after taking their intro to quantum mechanics course. I guess one could argue that’s because we have a special sauce that neural nets don’t yet have or something?
“super easily”? I would say it depends. Not if the ontology shift is believing or not in an all powerful all good creator God! That can sure change peoples goals and values. Some ontology changes make no difference, others make a huge difference.
The greater the intelligence increase, the more likely an agent (human or AI I expect) will experience an ontology change that causes a goal shift, and the more total ontology shifts you can expect. Those related to personal identity (what is “I” e.g. atoms, vs computation etc) seem more likely to cause goal shifts than say learning that solid objects are in fact forces interacting.
So if we are being formal its
Significant Increase in intelligence → many ontology shifts → some of these cause goal shifts.
I would just like to mention that “solid objects are in fact forces interacting” is massively underselling the size of the ontology shift associated with quantum mechanics to a degree that’s a bit hard to describe to someone who hasn’t studied it. It’s more like:
Fundamental physics no longer determines a single history where a particular series of things happens. It’s now the case that, even at a fundamental level, many different things all happen, and physics describes how relatively “real” each of those things are.
One consequence of this for our own universe, where entropy is increasing over time, is that the universe no longer has one history, but an exponential branching tree of histories, all of varying weights (numbers that describe how important we should consider the events in each of them). Oh, and the tree is emergent by the way, it’s not even a base component of the ontology.
The closest mathematical framework we previously had that kind-of worked this way was probability theory, and that was just meant to track our subjective uncertainties about things, not describe reality itself. But quantum mechanics doesn’t actually follow the rules of probability theory. It’s some kind of warped twisted version of probability where the probabilities are replaced with (the squared magnitudes of) complex numbers called “amplitudes”. Because complex numbers can have opposite signs, it’s possible that adding another way for something to happen can reduce the chances of it happening.
In regular probability theory (or, say, classical physics), you could describe a reversible map (like a symmetry transformation or time evolution) on a 10 state system by writing down a permutation. This initially seems like the only way it could be. You have ten possible states, and whatever you do has to be reversible, so all you can do is permute them. This could be represented with a 10x10 permutation matrix. But in quantum mechanics, such a map is not a permutation matrix, but a unitary matrix. Time evolution and symmetry transforms in QM are represented by unitary matrices, not permutations like you’d guess they should be!
Speaking of symmetry: Any non-trivial object that has all the rotational symmetry of a sphere (such as, for example, a sphere) must have infinitely many points. Except in QM, where the smallest such object has 3 points. Except that QM can tell the difference between a 360 degree rotation and doing nothing. (Not 720 degree rotations though. Those are still just like doing nothing, so at least there’s that.) As a consequence, the smallest non-trivial object that has all the rotational symmetry of a sphere actually has 2 points.
Because of the way QM works, it’s actually possible to exploit it to perform some kinds of computations faster than seems to be possible classically. This is also kind of weird.
EDIT: Made a few changes to this for clarity & accuracy based on Justin Sheek’s comment. (Thanks Justin!) List of edits:
Rewrote first sentence from “physics no longer describes what can happen” (misleading and just plain wrong) to its current form. I knew what I was trying to say here, but goofed on converting it into words. Sorry everyone.
Specified that we’re talking about fundamental physics here (since stat-mech does also involve assigning weights to various configurations).
Added paragraph break and “One consequence of this for our own universe, where entropy is increasing over time” to hopefully clarify that this part is talking about many worlds, and does not apply to every system that obeys quantum mechanics.
The bit about maps / functions was originally overstated for rhetorical reasons. This is probably not super detectable or helpful when describing a technical topic, so I’ve rewritten it to be more serious and direct.
I believe all that is written here is now something I can defend.
Oof, the amount of misinformation on QM even here on LW is staggering.
Physics no longer describes what can happen.
This is straightforwardly false. Maybe you meant to say “Physics no longer describes what definitely happens”? Still misleading, as that was already the case with statistical mechanics within the ontology of Boltzmann and Gibbs 50 years earlier.
Oh, and the tree is emergent by the way, it’s not even a base component of the ontology.
Coherent phenomena are definitely part of the base ontology of QM. The density matrix encodes the ensemble. (If by “the tree” you didn’t mean the ensemble, then your statement would make even less sense to me).
...QM will tell you that what it means to be a function has changed and you have to do it differently now.
No. QM has no bearing on “what it means to be a function”. Maybe you mean “QM encodes permutations in a surprising way”?
Except that QM can tell the difference between a 360 degree rotation and doing nothing. (Not 720 degree rotations though. Those are still just like doing nothing, so at least there’s that.)
Strictly speaking this is only sometimes true. It seems like you are alluding to the spin-statistics theorem or maybe the Aharonov-Bohm effect or Berry phase. Your quoted statement is specifically applicable only to fermionic states. It’s inapplicable to bosons or more exotic states like anyons (FQHE) or braid statistics.
Because of the way QM works, it’s actually possible to exploit it to perform some kinds of computations faster than seems to be possible classically. This is also kind of weird.
Thanks for the notes. I’ve made a few edits to my comment above based on this.
Also, for the benefit of the folks reading this: I’m not alluding to spin-statistics or Berry phase, merely the use of instead of as the group of rotational symmetries.
Finding out about quantum mechanics is a classic example of an ontology shift. You wrote “maintain aligned goals across ontologies”. If you actually meant “maintain aligned goals across orders of magnitude increases in intelligence”, then okay, but that’s a different thing.
Here strong orthogonality looks too neat. It imagines the agent’s ontology updating while its final target remains untouched by the update: if goals are expressed in an ontology, and intelligence changes the ontology, then intelligence and goals are correlated.
If students don’t change their goals when their ontology changes, but you expect that they will change their goals when they gain orders of magnitude in intelligence, that suggests that the thing that results in a change of goals is a large increase in intelligence, not an ontology change. This is true even if we put an arrow going from “intelligence increase” to “ontology change” in the causal graph.
Here where you’re describing difficult things about maintaining a long term paperclipping goal:
Keep the macroscopic concept of “paperclip” coherent across massive ontology shifts.
Also here, where you’re describing things that would update you:
If increasingly capable models perfectly preserve their literal training targets across major ontology shifts, that is a point for empirical orthogonality.
I mean, it seems pretty preposterous from my perspective too.
You propose a causal model: Intelligence -> Ontology Shifts -> Value Shifts
I question the Ontology Shifts -> Value Shifts part of the model, and provide a counterexample.
You then express concern that my example didn’t have the Intelligence variable”.
I am confused. “Maybe he actually meant to specify a Intelligence -> Value Shifts causal model? Otherwise, why would he care that my example didn’t have an Intelligence variable?” I think. I ask about it.
You say no, drop a quote that confirms that the original model is the one you’re thinking of.
Given confirmation that you’re going for Intelligence -> Ontology Shifts -> Value Shifts, I try to explain how my example is indeed a problem for your model. There is a model consistent with both the QM counterexample, and the students needing to be super-intelligent to have their values shifted, and with intelligence causing ontology shifts, namely Intelligence -> (Ontology Shifts, Value Shifts). (In words, highly increased intelligence separately causes both effects.) This model (like any model consistent with the counterexample) contradicts the one you describe. I try to point out the contradiction.
You: “Im sorry, can you point to the line where I claim otherwise”
I think “wait what? Is he claiming that this new thing was his model all along? I thought he already confirmed the other one.” I drop the quotes, specifically ones focusing on the Ontology Shifts -> Value Shifts part of the model, for lack of a better idea of what to do, and since you did make a direct request.
You: But I also have a Intelligence -> Ontology Shifts arrow!
So at this point, I am now even more sure that your model is Intelligence -> Ontology Shifts -> Value Shifts. What I am now unsure about is what else you could possibly have meant by “otherwise”, and still separately, why you think the students needed to have IQ 1000 or whatever.
I am certain that your explanations of these questions and of your side of this exchange must be fascinating. However, I also don’t mind ceasing to interact with you, since this was equally absurd on my end, and in addition you seem to have downvoted each of my replies in this thread, which makes talking to you sadly unprofitable for a karma whore such as myself.
This post reads to me as if you’ve mostly extrapolated the beliefs and arguments of “orthodox alignment theorists” from small snippets and ended up with a wildly oversimplified strawman. Then you’ve re-derived mostly orthodox rat beliefs and arguments and presented them as a devastating counterargument.
I think I’m fairly orthodox as lesswrongers go, and I agree with most of the statements and arguments you made in this post. I only have one or two disagreements toward the end of the post.
One example of many, because I found this one particularly funny:
The fact that maintaining goals across ontology shifts and self-modification takes careful effort is basically the core of the orthodox alignment-looks-hard worldview. You must be just making up an opposing worldview here? Where are the orthogonalist circles who say this?
>The fact that maintaining goals across ontology shifts and self-modification takes careful effort is basically the core of the orthodox alignment-looks-hard worldview
It’s one argument. The other is “the totality of human value needs to be hardcoded into the AI, you only get one attempt , and if you make the smallest mistake, everyone dies*.
>You must be just making up an opposing worldview here?
>Where are the orthogonalist circles who say this?
It’s long been the case that OT is mostly used as a pro-doom argument, both in the pro- paperclipping and anti moral realism senses.
It’s also true that the OT has some anti-doom implications, and that’s much less.publicised, and therefore worth pointing out.
I agree something like this is one branch of the argument, but in my mind it’s a relatively small branch. The main branch focuses on bounded corrigible AI and the main difficulty there is instability. There are other branches for non-hardcoding human values, and different targets that aren’t human values.
I’m not sure what point you’re making here, what are the implications you’re referring to?
I thought he meant this part:
i am very well aware the orthodox alignment line is that to maintain aligned goals across ontologies is very difficult! that’s why im surprised by this difficulty being set aside for strong orthogonality and misaligned goals.
as for the paragraph you cite: of course it’s a preposterous notion! but how else would you explain the fact that the arbitrary-terminal-goal-agent can emerge victorious from those who devote all their cycles to simply following instrumental drives?
at any rate: the thesis I wanted to dispatch is: there is the significant risk that an agent will reach superintelligence while ultimately continuing to pursuit a valueless goal. if you are trying to tell me that this was never a claim, i am very grateful; let us note it down on the wiki to prevent further confusion while I go on towards demolishing (or discovering I had imagined) the concept of AI psychia is. I’m on a schedule so this kind of help is deeply appreciated.
Then who’s in the orthogonalist circles you referred to? Or did you make them up?
When you try to derive someone’s premises from their conclusions, you still have to go and check whether you got it right. When people have different beliefs from you, it’s easy to slip up in this kind of reasoning. In my case it’s explained by me believing that selection isn’t always the main thing determining terminal goals (especially at finite times, or when there are other powerful optimizers interfering with selection).
I endorse this statement. But as per this yud tweet, it might be useful to disentangle the orthogonality thesis from the chance of misalignment, because misalignment involves a stack of additional arguments. It’d be better to directly engage with the strong form of the orthogonality thesis as described in the second sentence of the wiki page and with the arguments for it, rather than making them up your own versions of these.
i recommend you visit the link I have added at the beginning of this essay.
I’m surprised that you say this is hard? Humans maintain our goals across ontologies super easily; it’s barely an inconvenience for us. Like, physics undergrads don’t usually change their tastes in art or stop having sex after taking their intro to quantum mechanics course. I guess one could argue that’s because we have a special sauce that neural nets don’t yet have or something?
“super easily”? I would say it depends. Not if the ontology shift is believing or not in an all powerful all good creator God! That can sure change peoples goals and values. Some ontology changes make no difference, others make a huge difference.
The greater the intelligence increase, the more likely an agent (human or AI I expect) will experience an ontology change that causes a goal shift, and the more total ontology shifts you can expect. Those related to personal identity (what is “I” e.g. atoms, vs computation etc) seem more likely to cause goal shifts than say learning that solid objects are in fact forces interacting.
So if we are being formal its
Significant Increase in intelligence → many ontology shifts → some of these cause goal shifts.
I would just like to mention that “solid objects are in fact forces interacting” is massively underselling the size of the ontology shift associated with quantum mechanics to a degree that’s a bit hard to describe to someone who hasn’t studied it. It’s more like:
EDIT: Made a few changes to this for clarity & accuracy based on Justin Sheek’s comment. (Thanks Justin!) List of edits:
Rewrote first sentence from “physics no longer describes what can happen” (misleading and just plain wrong) to its current form. I knew what I was trying to say here, but goofed on converting it into words. Sorry everyone.
Specified that we’re talking about fundamental physics here (since stat-mech does also involve assigning weights to various configurations).
Added paragraph break and “One consequence of this for our own universe, where entropy is increasing over time” to hopefully clarify that this part is talking about many worlds, and does not apply to every system that obeys quantum mechanics.
The bit about maps / functions was originally overstated for rhetorical reasons. This is probably not super detectable or helpful when describing a technical topic, so I’ve rewritten it to be more serious and direct.
I believe all that is written here is now something I can defend.
Oof, the amount of misinformation on QM even here on LW is staggering.
This is straightforwardly false. Maybe you meant to say “Physics no longer describes what definitely happens”? Still misleading, as that was already the case with statistical mechanics within the ontology of Boltzmann and Gibbs 50 years earlier.
Coherent phenomena are definitely part of the base ontology of QM. The density matrix encodes the ensemble. (If by “the tree” you didn’t mean the ensemble, then your statement would make even less sense to me).
No. QM has no bearing on “what it means to be a function”. Maybe you mean “QM encodes permutations in a surprising way”?
Strictly speaking this is only sometimes true. It seems like you are alluding to the spin-statistics theorem or maybe the Aharonov-Bohm effect or Berry phase. Your quoted statement is specifically applicable only to fermionic states. It’s inapplicable to bosons or more exotic states like anyons (FQHE) or braid statistics.
Indeed.
Thanks for the notes. I’ve made a few edits to my comment above based on this.
Also, for the benefit of the folks reading this: I’m not alluding to spin-statistics or Berry phase, merely the use of
I don’t understand—are you saying that taking a college course makes undergrads orders of magnitude smarter?
Finding out about quantum mechanics is a classic example of an ontology shift. You wrote “maintain aligned goals across ontologies”. If you actually meant “maintain aligned goals across orders of magnitude increases in intelligence”, then okay, but that’s a different thing.
from the above essay. seems fairly clear to me
If students don’t change their goals when their ontology changes, but you expect that they will change their goals when they gain orders of magnitude in intelligence, that suggests that the thing that results in a change of goals is a large increase in intelligence, not an ontology change. This is true even if we put an arrow going from “intelligence increase” to “ontology change” in the causal graph.
Im sorry, can you point to the line where I claim otherwise
Sure.
Here where you’re describing difficult things about maintaining a long term paperclipping goal:
Also here, where you’re describing things that would update you:
Sorry, what are we doing here? You have quoted the second point of a list, which clearly included intelligence as the cause of such ontology shifts.
FYI, I will not interact further since this is clearly preposterous
I mean, it seems pretty preposterous from my perspective too.
You propose a causal model:
Intelligence -> Ontology Shifts -> Value ShiftsI question the
Ontology Shifts -> Value Shiftspart of the model, and provide a counterexample.You then express concern that my example didn’t have the
Intelligencevariable”.I am confused. “Maybe he actually meant to specify a
Intelligence -> Value Shiftscausal model? Otherwise, why would he care that my example didn’t have anIntelligencevariable?” I think. I ask about it.You say no, drop a quote that confirms that the original model is the one you’re thinking of.
Given confirmation that you’re going for
Intelligence -> Ontology Shifts -> Value Shifts, I try to explain how my example is indeed a problem for your model. There is a model consistent with both the QM counterexample, and the students needing to be super-intelligent to have their values shifted, and with intelligence causing ontology shifts, namelyIntelligence -> (Ontology Shifts, Value Shifts). (In words, highly increased intelligence separately causes both effects.) This model (like any model consistent with the counterexample) contradicts the one you describe. I try to point out the contradiction.You: “Im sorry, can you point to the line where I claim otherwise”
I think “wait what? Is he claiming that this new thing was his model all along? I thought he already confirmed the other one.” I drop the quotes, specifically ones focusing on the
Ontology Shifts -> Value Shiftspart of the model, for lack of a better idea of what to do, and since you did make a direct request.You: But I also have a
Intelligence -> Ontology Shiftsarrow!So at this point, I am now even more sure that your model is
Intelligence -> Ontology Shifts -> Value Shifts. What I am now unsure about is what else you could possibly have meant by “otherwise”, and still separately, why you think the students needed to have IQ 1000 or whatever.I am certain that your explanations of these questions and of your side of this exchange must be fascinating. However, I also don’t mind ceasing to interact with you, since this was equally absurd on my end, and in addition you seem to have downvoted each of my replies in this thread, which makes talking to you sadly unprofitable for a karma whore such as myself.
cool