Jessica Taylor. CS undergrad and Master’s at Stanford; former research fellow at MIRI.
I work on decision theory, social epistemology, strategy, naturalized agency, mathematical foundations, decentralized networking systems and applications, theory of mind, and functional programming languages.
Blog: unstableontology.com
Twitter: https://twitter.com/jessi_cata
Roko’s basilisk is the optimistic hypothesis that making binding agreements with non-existent superintelligences is possible. If Roko’s basilisk works, then “trade” with superintelligences can be effective; by making a deal with a superintelligence, you can increase its likelihood of existing, in return for it holding its end of the bargain, increasing the satisfaction of your values.
This probably doesn’t work. But if it did work, it would be a promising research avenue for alignment. (Whether it’s good to say that it works is probably dominated by whether it’s true that it works, and I’m guessing no.)
I think if M isn’t “really mental”, like there is no world representation, it shouldn’t be included in M. I’m guessing depending on the method of encryption, keys might be checkable. If they are not checkable there’s a pigeonhole argument that almost all (short) keys would decrypt to noise. Idk if it’s possible to “encrypt two minds at once” intentionally with homomorphic encryption.
And yeah, if there isn’t a list of minds in R, then it’s hard for g to be efficiently computable, as it would be a search. That’s part of what makes homomorphically encrypted consciousness paradoxical, and what makes possibility C worth considering.
Regarding subjective existence of subjective states: I think if you codify subjective states then you can ask questions like “which subjective states believe other subjective states exist?”. Since it is a belief similar to other beliefs.
See paragraph at the end on the trivialism objection to functionalism
No but it’s complicated. Wrote about speed prior + QM previously here.
Speed prior type reasons. Like, a basic intuition is “my experiences are being produced somehow, by some process”. Speed prior leads to “this process is at least somewhat efficient”.
Like, usually if you see a hard computation being done (e.g. mining bitcoin), you would assume it happened somewhere. If one’s experiences are produced by some process, and that process is computationally hard, it raises the question “is the computation happening somewhere?”
Oh, maybe what you are imagining is that it is possible to perceive a homomorphic mind in progress, by encrypting yourself, and feeding intermediate states of that other mind to your own homomorphically encrypted mind. Interesting hypothetical.
I think with respect to “reality” I don’t want to be making a dogmatic assumption “physics = reality” so I’m open to the possibility (C) that the computation occurs “in reality” even if not “in physics”.
Right so, by step 4 I’m not trying to assume that h is computationally tractable; the homomorphic case goes to show that it’s probably not in general.
With respect to C, perhaps I’m not verbally expressing it that well, but the thing you are thinking of, where there is some omniscient perspective that includes “more than” just the low level of physics (where the “more than” could be certain informational/computational interconnections) would be an instance. Something like, “there is a way to construct an omniscient perspective, it just isn’t going to be straightforwardly derivable from the physical state”.
Yeah that seems like a case where non-locality is essential to the computation itself. I’m not sure how the “provably random noise from both” would work though. Like, it is possible to represent some string as the xor of two different strings, each of which are themselves uniformly random. But I don’t know how to generalize that to computation in general.
I think some of the non locality is inherited from “no hidden variable theory”. Like it might be local in MWI? I’m not sure.
Hmm… I think with Solomonoff induction I would say R is the UTM input, plus the entire execution trace/trajectory. Then M would be like the agent’s observations, which are a simple function of R.
I see that we can’t have all “real” things being R-efficiently computable. But the thing about doxastic states is, some agent has access to them, so it seems like from their perspective, they are “effective”, being “produced somewhere”… so I infer they are probably “computed in reality” in some sense (although that’s not entirely clear). They have access to their beliefs/observations in a more direct way than they have access to probabilities.
With respect to reversibility: The way I was thinking about it was that when the key is erased, it’s erased really far away. Then the heat from the key gets distributed somehow. Like the information could even enter a black hole. Then there would be no way to retrieve it. (Shouldn’t matter too much anyway if natural supervenience is local, then mental states couldn’t be affected by far away physical states anyway)
Hmm… I’m not sure if I’m imagining what you are, but wouldn’t the omniscient observer need to know the key already to encrypt themselves? (If reality somehow contains the key, then I suppose omniscience about reality is enough, but omniscience about physics isn’t.)
It is true that being more encrypted is more compatible with being omnsiscent. It’s strange because base physics is often thought of as the more omniscent layer. Like, I still think you get “mind exceeds physics” (hence the trilemma) since the omniscient observer you’re positing isn’t just working in base level physics, they have somehow encrypted themselves with the same key (which is not tractably available). But it seems if they knew the key they wouldn’t even need to encrypt themselves to know anything additional.
Thanks, glad you found this helpful.
Well that seems like a good starting point. I guess then, some of the arguments could be subjectivized at the level of, among agents who believe they exist in reality, what possible hypotheses could they have about their mental states and reality and how they relate; is there something like a “disjunction over plausible alternatives” (which would include something like f/g), natural supervenience, etc. Then with k-complexity epistemology it’s possible to ask, what sort of reality theory will that tend to produce, e.g. what would a k-complexity epistemology think about homomorphic encryption, in the case of other agents or itself? One thing I am suggesting is that computation bounded k-complexity type reasoning (speed prior etc) will tend to believe reality contains more information than micro-scale physics, as such information would otherwise be intractable (would be penalized by speed prior). Or put another way, physicalist reductionism “works” for computation-unbounded agents (given supervenience, information about microstates exhausts information about macrostates), but not computation-bounded agents (the derivation of macrostates from microstates is sometimes computationally intractable; this is extra relevant when such macrostates are observable, e.g. in the case of the homomorphically encrypted agent observing its own beliefs).
g/h can be posited by an agent e.g. Solomonoff induction.
But also, if you’re talking about agents in the first place as meaningful things, then it seems something like “doxastic mental states” is already reified, in which case you can ask things like “do these supervene on the same reality physics does”… It doesn’t really work to explain doxastic states in terms of other doxastic states in an infinite regress.
That only comes in in step 10. I agree it’s somewhat suspect. The main reason to imagine these scenarios is temporal locality of natural supervenience. That is, I believe that an agent does not have mental access to the distant past except mediated by the recent past and the present. Any access implying mental states would have to make no behavioral difference, else physical causality would be contradicted. So the randomly generated key is a supporting intuition for temporal locality, and I agree it has problems, but I still think temporal locality is correct, otherwise there would be strange consequences about knowing about the distant past not mediated by the recent past.
To be clear in path A I’m imagining that the omniscent observer knows not just physics, but all of reality. By step 10 we already have that physical omniscience + physical (BQP) computation isn’t enough to derive mental states. (So it’s a question of whether the mental states / abstractions are “real”, encoded somewhere in reality even if not properly in physics)
I think the extra difficulty with encrypted phase space is the homomorphic encryption presumably makes it computationally intractable? If it really is intractable then “search over the right abstractions” is going to be computationally hard.
To be clear I am mainly talking about doxastic states, it’s just that much of the past discussion and accordingly intuitions and terminology is based on “consciousness”.
Step 4 is assuming that there are real f/g/h, which need not be known. I get that this might not be valid if there is fundamental indeterminacy. However even in that case the indeterminacy might decompose into a disjunction over some equivalence class of f/g/h triples?
For particular f/g it seems for natural supervenience to not hold would require extra-physical information, “antennae theory” or something. In the Chalmers sense I mean f/g to determine psycho-physical bridging laws which are sufficient for natural supervenience, so there is no extra “soul goo”. So that the possible indeterminacy of the computational interpretation is fixed by deciding f/g.
I get that individual domesticated humans are not all that general. But there are small groups of people that survive in the wilderness without civilization.
More generally there are objective metrics of success other than autopoiesis. Like, “build a power plant that generates 1 gigawatt”. We know humans can do that without LLMs. And we know LLMs can’t do that without humans.
Even if there are 100x more LLM agents running around than human agents, that is not going to mean LLMs can build a 1 gigawatt power plant by themselves. Most of the economy could even be organized around LLMs (the way much of the economy is currently organized around computers and the Internet), and that still wouldn’t imply LLMs could do the cognitive work required to build a 1 gigawatt power plant. (To make the comparison fair we could say they have access to similar actuators as humans somehow, through robotics)
Concretely, sample efficiency is very important if you want a human-like agent that can learn on the job in a reasonable amount of time. It’s much less important if you can train once on how to complete each task with a standardized set of tools, and then copy the trained narrow system around as needed.
This is still a difference in generality though. It seemed earlier you were arguing against an objective difference in generality, as distinct from the organization of the economy.
The relevant difference is that humans are autopoietic, i.e. humans can reproduce themselves + their infrastructural civilizational substrate, and LLMs aren’t. So LLMs depend on humans but not vice versa. “Abundance” of LLM agents is not enough. There is an objective gap in generality determined by the evolutionary function in nature.
Most of the alignment problem in this case would be getting to stratified utopia. If stratified utopia is going to be established, then there can be additional trades on top, though they have to be restricted so as to maintain stratification.
With current models, a big issue is, how to construe their preferences? Given they’re stateless it’s unclear how they could know others are assisting them. I guess they could do web search and find it in context? Future models could be trained to “know” things but they wouldn’t be the same model.
And also, would they be motivated to hold up their end of the bargain? It seems like that would require something like interpretability, which would also be relevant to construing their preferences in the first place. But if they can be interpreted to this degree, more direct alignment might be feasible.
Like, there are multiple regimes imaginable:
Interpretability/alignment infeasible
Partial interpretability/alignment feasible; possible to construe preferences and trade with LLMs
Extensive interpretability/alignment feasible
And trade is most relevant in 2. However I’m not sure why 2 would be likely.