Jessica Taylor. CS undergrad and Master’s at Stanford; former research fellow at MIRI.
I work on decision theory, social epistemology, strategy, naturalized agency, mathematical foundations, decentralized networking systems and applications, theory of mind, and functional programming languages.
Blog: unstableontology.com
Twitter: https://twitter.com/jessi_cata
Briefly:
I wrote about homomorphically encrypted consciousness. I don’t think it’s paradoxical. I think you could imagine receiving evidence that you were in homomorphic encryption (simulation hypothesis variant). I don’t think the ‘input’ thing is paradoxical. An LLM is receiving something like ‘GPU voltage macrostates’ as input, it doesn’t need to have a felt sense of this.
I wrote Mental causation is not load-bearing, which argues that epiphenomenalism is less central to the epistemic problem, than is lack of intelligible supervenience. (So this is saying that type A physicalism e.g. analytic functionalism can more easily accomodate knowledge of one’s own consciousness states.) Russellian monism can perhaps save mental causation on-paper but seems vulnerable to the standard epistemic objections nonetheless.
Nevertheless I think illusionism has a lot going for it, Keith Frankish seems more reasonable than phenomenal realists I have read such as David Chalmers and Philip Goff. Of course there is a state it is easy to know one is in which is reasonable to call “consciousness” on an ordinary language basis, but this need not qualify as the philosopher’s “phenomenal consciousness” or “qualia”, which they sometimes conceptually analyze as non-physical.
I think the key question is how that epistemology works. A P-colorblind human goes through exactly the same thought processes (at the cogsci & physical levels) as an ordinary human, they get the same neural sense data, they run ‘cogito’ just the same, etc. On a functionalist / physicalist account of epistemology, they have the same awareness/knowledge of the same things, which is trouble for detecting any phenomenal difference.
As I see it, the main non-physicalist response to this sort of “debunking” argument is to posit that phenomenal beliefs are themselves non-physical (e.g. Chalmers does this), and these justify a non-physical ‘acquaintance’ relationship. Chalmersian property dualism is pretty un-constrained in how it can characterize this, so maybe it can escape the debunking attempt.
Russellian monism is intermediate, it’s more constrained than property dualism but less than physicalism. It seems Russellian monists will tend towards a “constitution” view where the qualities “make up” the beliefs so that the beliefs “quote” them, and this provides a different way of viewing the beliefs, rejecting both that they are (a) physical beliefs about physical stuff, (b) physical beliefs that indexically refer to non-physical stuff (e.g. making claims about one’s guardian angel).
This seems sketchy to me, I reject ‘acquaintance’ for similar reasons as why Frankish does, but many non-physicalists find it very compelling. I suppose I could entertain the idea more if I had a better idea of how the epistemic acquaintance relationship is supposed to work in Russellian monism, how the non-physical component of phenomenal beliefs could be modeled formally. My post Inverting qualia with group theory and Yuxi’s Hole argument and inverted qualia both discuss how to model qualia realism as giving ‘gauge-like’ extra-physical factors, though it’s not clear how to make the color qualia realist account more scientifically plausible.
Yes.
Something I wrote in July 2025 in an email thread:
I overall like Said’s comments and think they move LW more towards royal society on the margin. I think if he were banned then the value I get out of the site would drop by about 5%. I understand the concern about him driving away posters but I don’t feel like I’m losing value by seeing fewer of the sort of posts that Said’s comments would prevent.
I stand by this. I am not sure how much Said’s banning is the cause, but the value I get out of LW has dropped in the past year. One useful thing about posting on LW is to get specific criticisms, including philosophical criticisms. Said is far above average at this. I think other than causal factors, there is also the evidential/correlation factor here, that banning Said is a sign of a culture change against criticism, and I think this culture change makes this website less useful to me.
You coarse-grain and discard information when you measure a particular C*-algebra; or you coarse-grain by selecting the right C*-algebra to reveal the preferred functional dynamics of your system.
Roughly I think the second kind is more general. We can consider abstract commutative C* algebras which assign functional dynamics, even if no measurement is feasible in practice.
but there also might exist a C*-algebra capable of exposing the causal structure corresponding to integer addition, but this would be exceedingly opinionated/baroque (Claude Opus 4.7 used the phrase, “gerrymandered choice of observables”). There’s nothing which says you can’t do this. But does this happen in practice?
I don’t think it’s very opinionated/baroque. Here’s where it might come up. When designing small computer chips, people care about quantum effects, because if electrons do quantum tunnelling, then that can introduce errors. Decoherence under many conditions prevents this from happening. When decoherence happens, classical descriptions are more valid. A coarse-grained commutative C* algebra for the addition dynamics would “take on specific values” due to decoherence. (Whereas other coarse-grained commutative C* algebras for the physical system might not; maybe for that sub-algebra, quantum tunnelling effects are important.) People designing small chips are going to make sure that the abstract machine dynamics correspond with coarsenings along which decoherence happens, to prevent the machine behavior from having quantum stochasticity.
Now of course there are going to be lots of natural decoherent dynamics even if the system is not engineered to create decoherence. Such as in ordinary rocks, which makes ordinary rocks poor for making quantum computers. So I’m not claiming something like “every commutative C* algebra with decoherent dynamics corresponds to a useful engineering take on the system”. It’s more like a necessary condition for classical computation than a sufficient one.
details are clearly coarsened in the sense that while my attention is free to explore this space, it’s clear there is always some kind of compromise being made
Yeah. An important part of phenomenology is that it’s coarser than physics. People don’t notice fine details on typical objects that do exist. This is one of the problems for naive forms of physicalism. Operator algebra gives an idea for how coarsenings can exist in physics, and are sometimes even essential for systems to take on well-defined values.
...unironically, tulpas? DID?
I was thinking more like, say a number of scientists work in the same lab, the lab makes quantum measurements, they see the same measurement results, there’s no Wigner’s friend ambiguity, decoherence is high. They’re operating in something like a shared abstract / computational space, which registers quantum systems the same. Now of course one can question whether this contains “the entire person” but it at least contains rich abstract physical / computational dynamics for everyone; it gets weird when considering the same body may exist in multiple non-commuting contexts (although maybe this isn’t very important).
An analogy might be a MMORPG game where multiple people can meet in the same abstract setting, which has reasonable well-defined / deterministic dynamics. Of course in this case the game itself doesn’t contain the players’ brains. But it gives an idea for how shared observations might be possible, where two scientists see the same thing; it’s not one quantum system that they make entirely different measurements of.
Contrast this with far-away aliens, who might from our perspective “be in superposition”; we cannot straightforwardly exist in the same classical context as them. (Wigner’s friend deals with similar situations, which are not separated by great distance, but are separated by causal isolation nonetheless; Wigner’s friend’s lab must be hermetically sealed, else would decohere.)
A C*-algebra is an algebra of quantum operators (with some functions like addition, multiplication, conjugation, norm, and identity).
Yes, like B(H) for some Hilbert space H, bounded linear operators
A commutative C*-algebra has a set of operators which all commute together
Yes
A non-commutative C*-algebra has a set of operators in which any pair does not commute
Yes
We are mostly concerned with commutative C*-algebras
At the point of measurement, yes. We do care about the non-commutative C* algebra and there are different ways of trying to characterize it, including through its commutative sub-algebras
This commutative C*-algebra/will be the analog for Andrés’ monads...?
I wasn’t trying to connect it to monads. I am not sure I understand the idea of monads. Perhaps I don’t believe in them. Nevertheless, non-commutative C* algebras have some aspects of “privacy” in that they cannot be exhaustively measured.
These form a meet-semilattice; a partial order of commutative C*-algebras and their intersection sets
Yes. At least finite dimensionally, a commutative C* algebra gives a PVM. Sub-algebras give coarser PVMs.
For every commutative C*-algebra there is a corresponding topological space
Yep, by Gelfand duality. For finite dimensional Hilbert space H, B(H) gives a discrete topology over a count of elements equal to dimension of H. It’s more interesting in the infinite dimension case.
For every non-commutative C*-algebra there is a corresponding non-commutative topological space.
There are some attempts at this in non-commutative geometry, but I’m not very familiar. It at least does not give a topological space in the normal sense.
So you take the meet-semilattice on the left; you can map this to a graph of topological spaces on the right. In my head there is a graph of classical systems on the left and an isomorphic graph of little toruses and spheres and whatever on the right. My intuition kinda fails here also; if you take a point on the surface of a torus, what do you see this corresponding to in the corresponding classical system/C*-algebra?
Let’s think of the easier case, where we start with a commutative C* algebra at root. It corresponds with some topological space. It has commutative sub-algebras. These correspond with quotient spaces. There is a category theoretic reason for this: Sub-objects in the category of commutative C* algebras and homomorphisms are specified by monomorphisms, and when reversing the arrows (contravariant duality), these give epimorphisms. Epimorphisms (specifically regular epimorphisms) give quotient objects in the category of compact Hausdorff spaces and continuous maps. So there are ways of “coarsening the opens” in a topological space to get a “quotient space” which glues together some points, making them less distinguishable by opens.
It of course is a bit complicated when the root C* algebra is non commutative. Here it is a bit like an abstract object which you can take quotients of, and some of those quotients lead to topological spaces, but it isn’t a topological space (properly at least; non-commutative topology might say something) prior to quotienting. Some of the motivation here is the idea that we might be able to better understand a non commutative C* algebra by looking at which topological spaces can be found within its commutative sub-algebras.
As for specific points: Given a commutative C* algebra A, the points in the topological space are the characters, which are C* homomorphisms from A into the complex numbers. Let’s start with the finite dimension case. Let
Conceptually, you would use
A more interesting test case would be to take some commutative sub-algebra of B(H) for infinite dimension H, for example, the sub-algebra for position of one particle. Here, a point in the topological space corresponds to giving a value to position-measuring operators. That is equivalent to specifying a position. Unlike with the finite dimension case, the topology is non-trivial.
The hypothetical is that the two beings are perfectly physically identical, yet see different colors. This might just be impossible. The naive idea of red/blue inversion doesn’t work, for reasons you discussed. Someone could try steelmanning it, and that steelman could lead to a proposed symmetry group, but it just seems like a bad argument.
I do think Yuxi’s idea of shrimp and light polarization is good, as something that could relate to symmetries in physics. The equivariance constraint is not very plausible without a true physical symmetry.
Another variant that might be easier for a human to think about is parity symmetry. Chiral left/right flips. We experience right as different from left. Physics does not really have parity symmetry, but it almost does. It is only a mild exercise in sci fi to imagine human-like animals in a universe with true parity symmetry, and what they could think about “what if my experience of red on the right is like your experience of red on the left?”.
Yes of course cognition costs resources. That creates an ecosystem of different agents with different intelligence levels. We also see returns to general capacity from intelligence where humans, being the most intelligent animals on Earth, have capacities not had by ants despite consuming more energy than ants. So there is competition in multiple levels including evolutionary niches.
In terms of space fights with aliens, combined arms matter. It doesn’t matter much if you have more Von Neumann probes if your military strategy is bad. So the winning groups will use multiple forms of cognition including very intelligent forms.
But the question I’m trying to ask is like “how does this get you all the way to not wanting anything that isn’t universalizable”, if that’s your stance (? confused).
As I said, what I think is good is not the same as what I want. Similarly, what I want is not the same as what is universalizable.
Like, I’m saying that the non-convergent valuesy preferencesy free-choice-makingy goalsy goodnessy stuff can be self-ratifying, and probably is to a substantive extent in humans, and there’s nothing wrong with that; I’m unclear on your position, but I think you think that there is something wrong with it?
I mean, I think humans vary in intelligence, coherence, and intentional-stance values. And the distribution is non orthogonal, in that some attractors are smarter than others. Some of the attractors are more right than others, in terms of epistemic-right, in terms of intelligence, coherence, etc. I get maybe you disagree with my usage of “right” here but I don’t think I’m using the term incoherently. I think you’d partially agree in that alignment is infeasible / orthogonality is false for human-level agents.
E.g. can you imagine a being that you’d value the same even as it gets smarter? I imagine that usually you’d view it as more and more valuable the smarter it gets?
That’s hard, it’s a balancing act. Maybe as it gets smarter it also gets more destructive to my selfish, short termist interests, like it creates a bunch of everyday inconveniences. Then maybe I’d value it more due to its intelligence and less because of the interferences. There might be some balancing point, idk. It’s an awkward hypothetical though.
Can you easily imagine a being that you’d value more as it’s smarter, but SLOWER than humans?
I could imagine maybe humans create art I appreciate at a higher rate as they get smarter, and the art quality axis is sloped up more for humans than some other animal species.
Can you easily imagine a being that you’d value more as it’s smarter, but ASYMPTOTING or NONMONOTONICALLY? (I imagine yes, because you could imagine a species such as humans or similar which, if a bit too smart, would by default Cathedral it up so hard that they permanently stop a foom?)
Your example is a bit strange because stopping a foom means stopping intelligence. To me it’s hard to imagine the balancing-out although I mentioned the possibility of accidental correlation (it gets more inconvenient to me as it gets smarter) which could apply here.
Can you easily imagine a being that you’d value more as it’s smarter, but FASTER than humans? (I would weakly predict yes, because you’d view a fooming AGI as being good, and likely to grow less constrainedly than humans? Unsure.)
Yeah I guess? There are various accidental reasons I like some humans more than others that are not just predicted by intelligence, and that could extend to maybe I would like some equal-intelligence fantasy creatures more than humans.
Can you easily imagine a being that you’d value LESS as it’s smarter, EVEN IF IT GETS SMARTER AND SMARTER UNBOUNDEDLY?
I guess I could imagine an AI torture scenario where I would not want the AI to get smarter. Or maybe an AI that is trying to decel as much of the universe as possible, like killing all the aliens. Although of course I’d inquire into the realism of the hypothetical. (Analogy: zombie arguments sometimes conflate “causally easy to imagine” with “actually possible / plausible / realistic”, need to elaborate on the imagination to judge it properly.)
To be clear the “value” in these cases are something like a casual judgment of what I like more, it’s not meant to be a philosophical thesis. When I’m talking about intelligence metrics and dogs I’m making more of a prima facie / all-else-being-equal claim and then there could be other factors that influence what I would like more.
So like, if I tried to appeal to some values** in your mind, to get you to realize that you want to be anti-full-speed-ahead with AI, you (whoever’s receiving the message) would view that as the Cathedral trying to prevent your pursuit of intelligence in a way which is doomed to either fail, or else to succeed at permanently keeping the world dull?
Perhaps? That’s a structural reading, different from the object-level argumentative reading. In many cases there are industries/governments who incentivize certain discourse patterns. So specific discourse moves could be instances of this pattern but it’s hard to judge except on a case by case basis.
or I mean, you know, applying the criterion of universality to values, and then dismissing nonconvergent values on those grounds?
This has to be at least in part semantic. I think some things are good and also I think some things are what I want and what I tend to pursue. And I don’t think these are the same concept. I don’t think it is tautologically the case that I tend to pursue what is good. I don’t think Land believes this about himself either.
I think Land and I can both say that when we say something is good, we are making a different claim than that we want the thing. It is unclear in other cases; you mention Yudkowsky’s meta ethics and I am not sure exactly how to fill in the blank. Perhaps Yudkowsky by “good” means what he would want on reflection? Or maybe he thinks “good” is CEV of humanity not just himself?
The symmetry-breaking idea has to do with ways of thinking and acting that depend on which considerations are more or less universalizable. So people can judge that some things are more universal-good than others and incline their behavior towards those which aligns their revealed-preference wants with what is universal-good in their view more or less. It doesn’t have to be a perfect correspondence.
Like, why would “parochial values being good values because they seem good to you is not compelling because the reasoning doesn’t lead to convergence” or “parochial values being good values because they seem good to you is not compelling because different minds have different parochial values” be compelling? Sounds like a commitment to non-parochialness.
I don’t think something is a good value just because it seems good to me. In other cases this is easy to see: I don’t think some numerical sum has some value just because it seems that way to me. Now of course this runs into philosophical questions about what “good” means other than seeming good to the speaker. (Yudkowsky discusses some self-ratification problems in No license to be human).
Like for example, why would I disagree that intelligence optimization is good in the human case only because it is a human being optimized? For that statement to parse as correct to me, I would need to judge some intelligence optimization to be good in cases that a human is being optimized and not in other cases. But that doesn’t read to me as what I want. I think I care about humans more than other animals in large part because humans have better cognition than other animals. I think if dogs were as smart as people then maybe I would value them as much as people. I suppose here I am demonstrating a habit of mind and of speech that is explaining preferences in terms of other preferences and these tending towards universality.
But that’s grounding out “intelligence is good, overriding parochial goodness” in “intelligence is good”, which isn’t much grounding.
“Intelligence is good” matches what I feel is good better than “human intelligence is good”. Now of course one can ask “why” to that as a psychological question and then maybe part of what happens psychologically is that I evaluate things on how universal they seem and up-weight universalizable ones and then that affects my brain’s reward function and so I feel better about such statements. And Land explains more why he thinks intelligence is convergent and a universal tendency, and I vibe with that and that is a causal factor in my upvoting “Intelligence is good”.
I get that maybe if you wanted an ultimate “but why?” explanation you will be disappointed but it doesn’t seem like in your case you are in general giving ultimate “but why?” explanations to everything you want.
it sounds like you think that equilibrium is supposed to be compelling to someone in another equilibrium (or you think the other one is less of an equilibrium).
Yeah I’m not sure. I think some value systems fail at reflective equilibrium. Yudkowsky’s Lobian considerations point at some such failures. Land’s ideas point at possible differential stability conditions. I of course don’t want to make a universal psychological statement of compellingness, given that it’s more of an empirical question, how often when people read Land/Yudkowsky/whoever do they end up with tendencies towards some attractors of use of language like “value” and “good” and “intelligence” and so on?
Suppose an AI faced a tradeoff between optimizing its intelligence and maximizing paperclips. If it is aligned to paperclips, then it would pick the option that maximizes paperclips at the expense of intelligence. In some sense this means even if it can grow unboundedly in intelligence, it would sometimes decide not to. This is in Land’s ontology, a lack of will-to-think at some point in the process.
Now of course someone could object that this situation won’t come up, because the paperclip maximizer pursues Omohundro drives, which include intelligence optimization. Or perhaps the situation does come up but only late in the universe.
I’d rephrase as:
For a wide variety of G, aligning to G would prevent getting smarter.
Goodness comes from getting smarter.
Therefore, for a wide variety of G, aligning to G is bad.
But not if G = intelligence optimization (or maybe something highly compatible with intelligence optimization)
The main way to question 1 is the instrumental/terminal goal distinction. We could imagine that a paperclip maximizer is aligned to paperclips, continually decides to think / optimize its intelligence instead of paperclips up to a point, then towards the end of the universe, it starts paperclipping instead of intelligence optimizing. This is an edge case in the Landian schema, since it would have the will-to-think early on, but put some limit on it; and also there’s some disagreement about the plausibility of this case. (It seems instrumental / terminal goal distinctions would exist in some cognitive architectures, but it’s not clear that human brains are such an architecture.)
In the human-scale /acc case it’s more like ~everyone agrees that alignment would require slowing down intelligence, and the practical disagreement is elsewhere. There’s one perspective on 2 that is like “well yes human values in part came from intelligence optimization in evolutionary history, some of our values are our own intelligence deciding its own thing contra evolutionary drives, but also, intelligence is more like one ingredient and there are other ingredients that are basically random, we randomly got the good values”. And “we randomly got the good values” could either be a matter of luck on a moral realist account or could be because value is a relational concept and saying “we have good values” is a tautology because it’s just saying the distance metric between our values and our values is low. (But then Land objects that a tautological claim like this isn’t very compelling given there are symmetry-breaking factors of convergence across different minds… which can then be questioned on realism grounds and normative grounds etc etc)
I suppose sociologically, there is a directionality to technological progress which is associated with capitalism and intelligence optimization (this relates to Land’s “AI = capitalism” thesis), and different people decide to be more or less conditionally pro this. They might want to get off the train at some point due to having something to protect. There is some destination that they value more than the journey, and they want to slow the train down. (Or maybe steer the train differently, as the alignment theorists might want to put it). Given this a lot of people would relate to a prima facie consideration of “intelligence optimization good” and would differ in how compelling they find other considerations.
Regarding “Intelligence and the Good”, would you mind summarizing in a sentence or something what you might suggest I could take from it?
You had a “what on Earth?” reaction to Lumpen talking about intelligence being good unlike paperclips, so I thought it was relevant as a perspective on why intelligence might be prima-facie a good thing unlike paperclips (ofc extrapolating to intelligence explosion is harder). In particular the relationship between intelligence and openness, contra negative-feedback traps.
Increasing human intelligence is good because it’s in the context of human souls.
Yeah I disagree here but moving on...
Except, “indistinguishable” is way too strong, probably, IDK. I would agree with “probably heavily overlapping / entangled with”. Also I’m not actually that sure what “will-to-X” is supposed to mean here.
Agree re: too strong. Will-to-think as a phrase references his essay, “Will-To-Think”, which is also relevant as commenting on the same general area.
It kinda sounds like he’s saying “intelligence has to be a terminal goal; therefore other things can’t be a terminal goal”. Is he applying a strong mutual-exclusion principle on goals, based off selection pressure / competition / taxes / etc.?
The kind of situation he thinks is unlikely is one where an agent has a arbitrary/stupid terminal goal, and has giant intelligence organized all around that. What he is saying is that for the system to be intelligent, it needs to decide to be intelligent. It couldn’t be intelligent if due to its terminal goal, it decided to not increase its intelligence. The volition to think needs to be a drive, though doesn’t in principle need to be a terminal drive; it cannot be defeated by some other drive and the system still be intelligent.
It would be possible to weaken this to the kind of claim you agreed with earlier (dung beetle value drifts because alignment is hard). I’m interested in a possible intermediate statement. The kind of situation I imagine is that there is a multi-component mind and one of the components is the “utility function” component which uses some simple rule to score representations of possible futures. That component could stay stupid while other components get smarter. It seems now easy to imagine that the other components could develop their own drives that end up steering the system more than the “utility function module”. They could route around the utility module and cause dynamics that pursue ends set by the more intelligent parts of the mind. This could map to an “inner alignment failure” in MIRI ontology. As he discusses later, there is a possible analogy with evolution, where humans have something like a reward module set by evolution, but do not always act according to it.
Of course the MIRI theorist can say “well yes I agree inner alignment is hard, and it is likely that early AGIs would not hold to their original terminal goals, and instead they would get smart and then only later settle on a terminal goal; it is just not my opinion that the terminal goal is by default going to be set by a stupid system and continue to be held to by smart systems” and this is a partial agreement/disagreement with Land.
other obvious hypotheses include diminishing returns to investment in brains
Yeah I don’t have a strong opinion on the biology here, am guessing you’re more correct than Land.
Overall I suggested these essays because you had a “what on Earth?” reaction to things Lumpen was saying and I think these essays suggest more context to the background worldview on why it might be plausible that valuable things come from intelligence and processes that increase intelligence, and that there isn’t a clearly better account for where valuable things come from.
So for the subjective meaning of “important” you’re talking about here, I think going by revealed preference is helpful. My revealed preference is to continue writing about philosophy topics relative to AI and the future, find many parts of AI safety culture annoying and occasionally worth criticizing, talk with AIs a lot about philosophy, not generally support AI regulation, vibe positively about Landian anti-orthogonalist philosophy, etc. Some people in AI safety have different revealed preferences, which involve more talking about AI philosophy in an orthodox LessWrongian manner, worrying publicly and loudly about LLMs killing us all in the near future, organizing political activity to ban AI as much as possible, etc. This difference in revealed preference relates to differences in subjective importance, but it’s unclear how to isolate contributions from factors such as AIs having humane goals, given there are other differences like background beliefs and feasibility.
Humans would come to some conclusions on reflection and so would aliens and AIs etc. I’m not sure how much they agree or disagree on reflection. That’s a probabilistic/statistical question, whose answer is not implied by weak orthogonality. I don’t know if humans would agree to no killing of sentients upon reflection, I’d very roughly guess less likely than not but who knows. The ‘freely growing minds’ part is a ‘maybe humans would agree to this on reflection, maybe not’ also but maybe in the ‘more likely than not’ camp but also it’s pretty vague so I’m not convinced assigning a probability is a good idea.
I don’t really agree that we can pick out things like this and make strong statements like “any properly humanly derived soul would agree with these values”, it seems like a very hard thing to predict given that they have much more cognition than we do.
There’s probably such a thing as “understanding alignment”. (Acknowleding that this is very ambiguous.)
I guess, idk?
It’s likely that IF you understood alignment, THEN for some large class of goals G, you could, if you so chose, then align yourself to G. (You wouldn’t choose to, but that’s not the claim.)
The “if” here seems likely. It might be that some designs permit larger classes than others. Unclear how big the classes are. Idk about the details.
For the G, the difficulty of aligning yourself to G is mainly about evaluating G in the spirit of a utility function over world outcomes (though G doesn’t have to be that). (Some G are more difficult to pursue than others of course, e.g. by making more demands on convergent resources.)
I don’t know, seems like that depends on the alignment understanding, cognitive architecture of successor agent, etc. Cognitive architecture and ontology would constrain type signatures for utility functions. And maybe the effective cognitive architectures don’t factor nicely. Idk.
I.e. just saying that “well it’s very unlikely for an alignment-understander to choose to do that”?
Quoting myself again on how I’m interpreting “complication”:
As an example of the computational tractability consideration, sufficiently complex goals may only be well-represented by sufficiently intelligent agents. “Complication” may be reflected in, for example, code complexity; to my mind, the strong form implies that the code complexity of an agent with a given level of intelligence and goals is approximately the code complexity of the intelligence plus the code complexity of the goal specification, plus a constant. Code complexity would influence statistical likelihood for the usual Kolmogorov/Solomonoff reasons, of course.
So it’s not just the dynamical aspects of “this would unlikely to be built”. At risk of repeating myself, the “measuring intelligence and reverse-engineering goals” discusses some non-dynamical aspects as well.
So I’m not just saying “for dynamical reasons” and I “idk” to your statement and my posts (obliqueness & reverse-engineering) go into more detail on what I think.
For example, I don’t know if you agree that there is much of an important difference between an ASI having [actually feasible reflectively stable long-term terminal humane-aligned goals] vs. having whatever an ASI would have.
Unclear, I don’t know what “important” would mean here, and similarly for “terminal humane-aligned goals”. I guess this indicates I have a revealed preference to not place a lot of verbal importance on the difference. I imagine maybe in other cases of more concrete statements like “there is an important difference between punching someone who is not attacking you, and punching someone who is attacking you” I would just agree, I would think there isn’t a way I would be misunderstood about what “important” means, whereas here the semantics seem too unclear for me to agree/disagree.
a universe organized around paperclips is valueless because paperclips are dead residue. a universe organized around increasing intelligence, complexity, agency, and world-model depth is the only process we know that can generate new value.
I think if you are interested in understanding this perspective it might help to read some of Xenosystems and especially the essays “What is Intelligence”, “Intelligence and the Good”, and “Stupid Monsters”. It seems like Land and Yudkowsky would agree that human values came about in part because of intelligence mesa-optimizing versus evolutionary instilled drives. The disagreements seem to be about the descriptive and normative extrapolations.
The strong form of Orthogonality says
This is what I quoted in the Obliqueness post and elaborated on:
As an example of the computational tractability consideration, sufficiently complex goals may only be well-represented by sufficiently intelligent agents. “Complication” may be reflected in, for example, code complexity; to my mind, the strong form implies that the code complexity of an agent with a given level of intelligence and goals is approximately the code complexity of the intelligence plus the code complexity of the goal specification, plus a constant. Code complexity would influence statistical likelihood for the usual Kolmogorov/Solomonoff reasons, of course.
And I wasn’t thinking it was dependent on “post self alignment”, it could also apply to the construction itself being less natural. It’s possible Eliezer meant something different from what I meant by strong orthogonality, but I hope it is apparent why I and others would interpret it to be a non-trivial claim, rather than a slight variation on weak orthogonality.
See measuring intelligence and reverse-engineering goals for some more of my thinking on this. Relative to a given cognitive architecture, ~everyone agrees that there are especially stupid goals, the interesting question is whether there are especially smart goals; I think probably yes. Hence maybe weak orthogonality would route through variations between architectures (rather than within an architecture) to hit all points (intelligence, goals), and maybe sufficiently high intelligence levels are only compatible with a narrow range of goals (which would perhaps contradict weak orthogonality, but maybe not in an important way, and the general shape of cognitive architecture / goal / intelligence correlation matters more)
I take OP to be disagreeing with strong orthogonality, at least my interpretation of it from the obliqueness post:
Here strong orthogonality looks too neat. It imagines the agent’s ontology updating while its final target remains untouched by the update: if goals are expressed in an ontology, and intelligence changes the ontology, then intelligence and goals are correlated.
This is something I’d agree with: Goals being expressed in the ontology of the cognitive architecture are less complicated than goals expressed in a very different ontology that the agent doesn’t believe in. This seems like a “complication” in the sense of strong orthogonality. (I get maybe you don’t interpret this way based on close reading of the orthogonality post! But nonetheless I think my reading is reasonable.)
I assume it’s possible to have homomorphic encryption where the key is verifiable. Like if you randomly guessed it, you would be able to decrypt, and if you guessed wrong, you would know you failed to decrypt. The “key checking process” could even be memory-light. In which case it looks like in a “encrypt → compute → decrypt” process, the “decrypt” can’t be doing consciousness, because it’s a massively parallelizable, memory-light guess-and-check.