Jessica Taylor. CS undergrad and Master’s at Stanford; former research fellow at MIRI.
I work on decision theory, social epistemology, strategy, naturalized agency, mathematical foundations, decentralized networking systems and applications, theory of mind, and functional programming languages.
Blog: unstableontology.com
Twitter: https://twitter.com/jessi_cata
See Nick Land: Orthogonality, it has relevant excerpts including about Pythia.
It means reductionism isn’t strictly true as ontology. I suppose it might be more precise to talk about “reductionist physics” than “physics”, although some might consider that redundant.
It isn’t obvious that biological structure isn’t efficiently readable from microstate. It at least doesn’t seem cryptographically hard, so polynomical time in general.
With turbulence you can pretty much read the current macrostate from the current microstate? You just can’t predict the future well.
I’d say homomorphic encryption computation facts, not just mental ones, are beyond physics in this sense. Other macro facts might be but it’s of course less clear.
Again, the same ontological status applies to homomorphic encryption and other entities. However the same epistemic status doesn’t apply. And the “efficiently determinable” criterion is an epistemic one.
A reason to pay attention to mental ones is that they are more salient as “hard to deny the existence of from some perspectives”. Whereas you could say a regular homomorphic encryption fact is “not real” in the sense of “not being there in the state of reality at the current time”.
and we don’t treat that as evidence that the visual appearance “exceeds physics.”
This is still something I’d disagree with? Like, it still seems notable that visual appearances aren’t determined as an efficient function of physics. It suggests perhaps there is more to reality than physics, otherwise what are you seeing? “Appearances as such exceed physics” is not substantially different from what I mean as “mind exceeds physics”. This seems like a minor semantic issue. Appearances are mental, so if appearances exceed physics than so does mind; I’m not meaning any strong statement like “mind, and only mind, exceeds physics”.
I’m saying efficient reconstructibility is unclear in the rainbow case, but that the same principles have to explain it and non-efficiently-reconstructible cases like homomorphic encryption. I don’t take this as a reducio but as a trilemma, see step 11.
Take a rainbow. Let p be the full microphysical state of the atmosphere and EM field, and let a be the appearance of the rainbow to an observer. The observer trivially “knows” a. Yet from p, even a quantum-bounded “Laplace’s demon” cannot, in general, P-efficiently compute the precise phenomenal structure of that appearance.
This may be true but it’s really not obvious. The homomorphic encryption example makes one encounter such a case more clearly. If there’s no hard encryption there, why couldn’t Laplace’s demon determine it efficiently?
That is an implausible conclusion. The physical state fully fixes the appearance; what fails is only efficient external reconstruction, not physical determination.
The thing you quoted and said was implausible had “efficiently” in it...
Homomorphic encryption sharpens the asymmetry between internal access and external decipherability, but it does not introduce a new ontological gap.
Yeah it just makes an existing problem more obvious.
At the end of the day the natural supervenience relation of observations on physics should work similarly in the rainbow case and the homomorphic encryption case. The homomorphic encryption case just makes more clear something that might have gotten skipped over in the rainbow case, “the natural supervenience relation need not be efficiently computable from the physical state; the information of the observations doesn’t need to be directly sitting there, the way of picking it out might need to be a complicated function rather than a simple efficient ‘location and extraction of information’ one”
Here’s a basic problem with infinite bases. Suppose duplicates its argument times. And suppose sums all entries. Now is not a sensible function.
So you really need to have some restriction. Like for example, maybe we interpret as requiring all but a finite number of entries to be zero. That would at least rule out . Now is not a “true infinite product” in the category-theory sense. But we would still have (“first” and “rest” of infinite list). Which might enable induction. I’m not sure.
Alternatively we could have be unrestricted, but then can’t be defined. Either way there’s an issue with allowing functions to or from to be represented by arbitrary infinite matrices.
EDIT: another framing of this is that “infinite product” ( unrestricted) and “infinite coproduct” ( restricted to all but finite being zero) come apart in . So there isn’t strictly an infinite biproduct.
Ah. I think the inference they may take is that a paperclip maximizer is perfectly rational/coherent, as is a staple maximizer and so on. They don’t think there are additional constraints as you suggest, beyond minimal ones like not having an “especially stupid” goal, such as “die as fast as possible”.
I don’t see how Bayesianism/vNM/expected utility theory should argue in favor of orthogonality.
I’m saying they argue against orthogonality in the post...
But isn’t this subsumed by “above and beyond the computational tractability of that goal”
You seem to think either “diagonality” or “strong orthogonality” must hold. But the post is arguing the converse. I am arguing against strong orthogonality and against diagonality.
Rough argument against diagonality is something like “paperclip maximizer like entities seem like they would be possible / coherent” although there are some unknowns there like how different parts of the agent separated by large distances coordinate/mutate. But perhaps more basic than that is, if someone is making a strong claim (diagonality) they should probably justify it.
The part in square brackets seems like the very clear Gricean implicature here? Am I wrong? If not, what did you mean to say in that sentence?
See the two sentences right after.
That was at the very least, advising others to think seriously about using aligned AI to take over the world, on the basis that the world was otherwise doomed without a pivotal act. Then there was the matter of how much leverage MIRI thought they had as an organization, which is complicated by the confidentiality.
The Griecian implicature of this is that I at least don’t think it’s clear that MIRI wanted to build an AI to take over the world themselves. Rather, they were encouraging pivotal acts generally, and there’s ambiguity about how much they were individually trying to do so.
The literal implication of this is that it’s hard for people to know how much leverage MIRI has as an organization, which implies it’s hard for them to know that MIRI wanted to take over the world themselves.
It doesn’t seem to me to be much evidence for “MIRI was planning to build FAI in their basement”
I didn’t say that
The “Pivotal Act” article seems similar? I mean, MIRI is still working on a pivotal act in the form of an international AI ban (subsequently followed maybe with an intelligence augmentation program). I am working on pivotal acts all day!
At the time it was clear MIRI thought AGI was necessary for pivotal acts, e.g. to melt all GPUs, or to run an upload. I remember discussing “weak nanotech” and so on and they didn’t buy it, they thought they needed aligned task AGI to do a pivotal act.
Which really doesn’t sound much like a “take-over-the-world” strategy.
Quoting task AGI article:
The obvious disadvantage of a Task AGI is moral hazard—it may tempt the users in ways that a Sovereign would not. A Sovereign has moral hazard chiefly during the development phase, when the programmers and users are perhaps not yet in a position of special relative power. A Task AGI has ongoing moral hazard as it is used.
So this is acknowledging massive power concentration.
Furthermore, in context of the disagreement with Paul Christiano, it was clear that MIRI people thought there would be a much bigger capability overhang / FOOM, such that the system did not have to be “competitive”, it could be a “limited AGI” that was WAY less efficient than it could be, because of a pre-existing capability overhang versus the competition. Which, naturally, goes along with massive power concentration.
On Arbital. Task directed AGI and Pivotal act.
Offline, at MIRI there were discussions of possible pivotal acts, such as melting all GPUs. I suggested “what about using AI to make billions of dollars” and the response was “no it has to be much bigger than that to fix the game board”. There was some gaming of e.g. AI for uploading or nanotech. (Again, unclear how much leverage MIRI thought they had as an organization)
What about the “Task AGI” and “pivotal act” stuff? That was at the very least, advising others to think seriously about using aligned AI to take over the world, on the basis that the world was otherwise doomed without a pivotal act. Then there was the matter of how much leverage MIRI thought they had as an organization, which is complicated by the confidentiality.
I believe bra/ket is for row and column vectors. I don’t think it applies here, because in the general case (semiadditive categories), you have arbitrary linear maps as the entries. And in the case, they’re reals, not row or column vectors.
It is true that you can decompose as either or . To be clear I’m using and [] from category theory product/coproduct notation, it’s not meant to match linear algebra or bra/ket notation.
I don’t understand the notation; it looks like bra/ket except not quite?
Most of the alignment problem in this case would be getting to stratified utopia. If stratified utopia is going to be established, then there can be additional trades on top, though they have to be restricted so as to maintain stratification.
With current models, a big issue is, how to construe their preferences? Given they’re stateless it’s unclear how they could know others are assisting them. I guess they could do web search and find it in context? Future models could be trained to “know” things but they wouldn’t be the same model.
And also, would they be motivated to hold up their end of the bargain? It seems like that would require something like interpretability, which would also be relevant to construing their preferences in the first place. But if they can be interpreted to this degree, more direct alignment might be feasible.
Like, there are multiple regimes imaginable:
Interpretability/alignment infeasible
Partial interpretability/alignment feasible; possible to construe preferences and trade with LLMs
Extensive interpretability/alignment feasible
And trade is most relevant in 2. However I’m not sure why 2 would be likely.
Roko’s basilisk is the optimistic hypothesis that making binding agreements with non-existent superintelligences is possible. If Roko’s basilisk works, then “trade” with superintelligences can be effective; by making a deal with a superintelligence, you can increase its likelihood of existing, in return for it holding its end of the bargain, increasing the satisfaction of your values.
This probably doesn’t work. But if it did work, it would be a promising research avenue for alignment. (Whether it’s good to say that it works is probably dominated by whether it’s true that it works, and I’m guessing no.)
I think if M isn’t “really mental”, like there is no world representation, it shouldn’t be included in M. I’m guessing depending on the method of encryption, keys might be checkable. If they are not checkable there’s a pigeonhole argument that almost all (short) keys would decrypt to noise. Idk if it’s possible to “encrypt two minds at once” intentionally with homomorphic encryption.
And yeah, if there isn’t a list of minds in R, then it’s hard for g to be efficiently computable, as it would be a search. That’s part of what makes homomorphically encrypted consciousness paradoxical, and what makes possibility C worth considering.
Regarding subjective existence of subjective states: I think if you codify subjective states then you can ask questions like “which subjective states believe other subjective states exist?”. Since it is a belief similar to other beliefs.
See paragraph at the end on the trivialism objection to functionalism
Close to what I mean. The multi-level structure is irreducible in that (a) it can’t be efficiently computed from microstates (b) it is in some cases observable, indicating it’s real. (Just (a) would be unsurprising, e.g. “the firth nth digits of Chaitin’s omega where n is the number of atoms in a table” is a high-level physical property that is not computable from microstate.)
That’s not the claim. My argument wouldn’t work if in all cases, subjective perceptions could be efficiently computed from microstates. And it is possible for subjective perceptions to be efficiently computed from microstates without subjective perceptions being a “fundamental ingredient”. Rather I am vaguely suggesting something like neutral monism, where there is some fundamental ingredient explaining the physics lens and the mind lens.
It depends what kind of external observer you imagine right? Like if somehow we had a scan of a small animal down to the cellular level, there would be ordinary difficulties in re-constructing the macro-scale features from it, but none of them are clearly computationally hard (super-polynomial time).
It seems like I entirely agree, not sure if I understood wrong. That is, I think path (c) is reasonably likely, and what it is saying is that there is more ontology than microphysics. It would be unsurprising for this to be the case, due to the way microphysical ontology, as methodology, is ok with dropping things that can be “in principle reconstrtucted”, hence tending towards the microscopic layer (as everything can be “in principle reconstructed” from there); ignoring computational costs to doing so, hence plausibly dropping things that are actually real from the ontology.