Jessica Taylor. CS undergrad and Master’s at Stanford; former research fellow at MIRI.
I work on decision theory, social epistemology, strategy, naturalized agency, mathematical foundations, decentralized networking systems and applications, theory of mind, and functional programming languages.
Blog: unstableontology.com
Twitter: https://twitter.com/jessi_cata
I don’t see how this proposal substantially differs from nationalizing all AI work. Governments can have internal departments that compete, like corporations can. Removing the profit motive seems to only leave government and nonprofit as possibilities, and the required relationships with regulators make this more like a government project for practical purposes.
Without specifying more about the governance structure, this is basically nationalization. It is understandable that people would oppose nationalization both due to historical bad results of communist and fascist systems of government, and due to specific reasons why current governments are untrustworthy, such as handling of COVID, lab leak, etc recently.
In an American context, it makes very little sense to propose government expansion (as opposed to specific, universally-applied laws) without coming to terms with the ways that government has shown itself to be untrustworthy for handling catastrophic risks.
I think, if centralization is beneficial, it is wiser to centralize around DeepMind than any government. But again, the justification for this proposal seems spurious if it is in effect tending towards more centralization than default closed source AI.
This is presented as a level of centralization between that of open-source and closed-source, but is actually more centralized than closed-source. In closed-source, anyone can write source code and can allow or restrict access to it how they like. In this proposal, this is not the case, since permission from the regulatory body is required. One would therefore expect even more concentration of AI capabilities in the hands of even fewer actors, as compared with closed-source.
What if I’m mainly interested in how philosophical reasoning ideally ought to work?
My view would suggest: develop a philosophical view of normativity and apply that view to the practice of philosophy itself. For example, if it is in general unethical to lie, then it is also unethical to lie about philosophy. Philosophical practice being normative would lead to some outcomes being favored over others. (It seems like a problem if you need philosophy to have a theory of normativity and a theory of normativity to do meta-philosophy and meta-philosophy to do better philosophy, but earlier versions of each theory can be used to make later versions of them, in a bootstrapping process like with compilers)
I mean normativity to include ethics, aesthetics, teleology, etc. Developing a theory of teleology in general would allow applying that theory to philosophy (taken as a system/practice/etc). It would be strange to have a distinct normative theory for philosophical practice than for other practices, since philosophical practice is a subset of practice in general; philosophical normativity is a specified variant of general normativity, analogous to normativity about other areas of study. The normative theory is mostly derived from cases other than cases of normative philosophizing, since most activity that normativity could apply to is not philosophizing.
How would you flesh out the non-foundationalist view?
That seems like describing my views about things in general, which would take a long time. The original comment was meant to indicate what is non-foundationalist about this view.
I don’t understand this sentence at all. Please explain more?
Imagine a subjective credit system. A bunch of people think other people are helpful/unhelpful to them. Maybe they help support helpful people and so people who are more helpful to helpful people (etc) succeed more. It’s subjective, there’s no foundation where there’s some terminal goal and other things are instrumental to that.
An intersubjective credit system would be the outcome of something like Pareto optimal bargaining between the people, which would lead to a unified utility function, which would imply some terminal goals and other goals being instrumental.
Speculatively, it’s possible to create an intersubjective credit system (implying a common currency) given a subjective credit system.
This might apply at multiple levels. Perhaps individual agents seem to have terminal goals because different parts of their mind create subjective credit systems and then they get transformed into an objective credit system in a way that prevents money pumps etc (usual consequences of not being a VNM agent).
I’m speculating that a certain kind of circular-seeming discourse, where area A is explained in terms of area B and vice versa, might be in some way analogous to a subjective credit network, and there might be some transformation of it that puts foundations on everything, analogous to founding an intersubjective credit network in terminal goals. Some things that look like circular reasoning can be made valid and others can’t. The cases I’m considering are like, cases where your theory of normativity depends on your theory of philosophy and your theory of philosophy depends on your theory of meta-philosophy and your theory of meta-philosophy depends on your theory of normativity, which seems kind of like a subjective credit system.
Sorry if this is confusing (it’s confusing to me too).
Philosophy is a social/intellectual process taking place in the world. If you understand the world, you understand how philosophy proceeds.
Sometimes you don’t need multiple levels of meta. There’s stuff, and there’s stuff about stuff, which could be called “mental” or “intensional”. Then there’s stuff about stuff about stuff (philosophy of mind etc). But stuff about stuff about stuff is a subset of stuff about stuff. Mental content has material correlates (writing, brain states, etc). I don’t think you need a special category for stuff about stuff about stuff, it can be thought of as something like self-reading/modifying code. Or like compilers compiling themselves; you don’t need a special compiler to compile compilers.
Philosophy doesn’t happen in a vacuum, it’s done by people with interests in social contexts, e.g. wanting to understand what other people are saying, or be famous by writing interesting things. A sufficiently good theory of society and psychology would explain philosophical discourse (and itself rely on some sort of philosophy for organizing its models). You can think of people as having “a philosophy” that can be studied from outside by analyzing text, mental states, and so on.
Reasoning about mind embeds reasoning about matter, reasoning about people embeds reasoning about mind, reasoning about matter embeds reasoning about people. Mainstream meta-philosophy consists of comparative analysis of philosophical texts, contextualized by the historical context and people and so on.
Your proposed reflection process for designing a utopia is your proposed utopia. If you propose CEV or similar, you propose that the world would be better if it included a CEV-like reflection context, and that this context had causal influence over the world in the future.
I’m not sure how clear I’m being, but I’m proposing something like collapsing levels of meta by finding correspondences between meta content and object content, and thinking of meta-meta content as meta relative to the objects corresponding to the meta content. This leads to a view where philosophy is one of many types of discourse/understanding that each shape each other (a non-foundationalist view). This is perhaps disappointing if you wanted ultimate foundations in some simple framework. Most thought is currently not foundationalist, but perhaps a foundational re-orientation could be found by understanding the current state of non-foundational thought.
I do sometimes run into good faith disagreements, they feel very different from typical disagreements.
Example 1: someone says something on the phone about “Simon building a computer science model of his father’s situation”. I think it’s referring to Simon A and a friend thinks it’s Simon B. We bet and call the person back and I was wrong and feel silly.
Example 2: Someone says that standard financial prudence calls for minimizing market beta. I think this is wrong and it calls for increasing it in most cases. Then I look it up and realize it’s because you can lever up a low market beta to a high one.
It feels really different from a normal non-trivial disagreement. In both cases it felt at the time like I was very probably correct and it was confusing that the other person disagreed. There is usually some mix of bad faith and good faith in typical cases, and that makes the resolution a lot slower.
I do it by pasting the Markdown into a google doc, find/replacing there, and then pasting back.
However, since SIA is a positive theory, this means that Rawlsianism must also be a positive, not a normative, theory. Take this as a paradox, if you will.
I see what you mean where you’re testing a key assumption of liberalism or Heideggerianism, not the theory as a whole. Rawlsianism, however, includes min-maxing as well, which seems more normative.
This leads me to suspect that there is a well-established connection between moral philosophy and anthropic reasoning that flows from the former to the latter. Please let me know if that is the case.
If you are behind a veil of ignorance and optimize expected utility under SSA, then you will want the average utility in your universe to be as high as possible.
SIA is a bit more complicated. But we can observe that SIA gives the same posterior predictions if we add extra observers to each universe so they have the same population, and these observers observe something special that indicates they’re an extra observer, and their quality of life is 0 (i.e. they’re indifferent to existing). In this case, someone reasoning behind the veil of ignorance and maximizing expected utility will want total utility to be as high as possible; it’s better for someone to exist iff their personal utility exceeds 0, since they’re replacing an empty observer.
As for Heideggerianism, I agree that it does not necessarily “correspond” to SSA in any way. However, I do feel that it is likely incompatible with SIA. As noted in my post, I am a bit uncertain about the consequences of Heideggerianism, so I am happy to change my argument to a more general “parochial” vs. “non-parochial” form, using the language you have suggested.
I see that Rawlsianism requires an original position. But such an original position is required for both SSA and SIA. To my mind, the difference is that the SSA original position is just a prior over universes, while the SIA original position includes both a prior over universes and an assumption of subjective existence, which is more likely to be true of universes with high population. Both SSA and SIA agree that you aren’t in an empty universe a priori, but that’s an edge case; SIA scales continuously with population while SSA just has a step from 0 to 1.
Heidegerrianism doesn’t seem to believe in an objective universe in a direct sense, e.g. the being of tools is not the being of the dynamical physical system that a physicalist would take to correspond to the tool, because the tool also has the affordance of using it. So it’s unclear how to reconcile Heideggerianism with anthropics as a whole. I’m speculating on various political correspondences but I don’t see how to get them from Heideggerianism and anthropics, just noting possible similarities in conclusions.
While famous philosophers tend to be decently high in social ranking as determined by education level, money, etc, they’re also likely to have people gang up on them. E.g. Socrates being killed, Galileo being condemned, Spinoza being excommunicated… And there are a lot of good philosophical thinkers we haven’t heard of, who might have been ganged up on as well.
Being a philosopher could in general be considered a form of neurodivergence. People who think and act differently from others could use justice-related protections that also protect other people who think and act differently from others. Updating all the way to min-max is a bit much, but there’s something to the heuristic. Over my life I’ve updated towards thinking that truth-seeking flags someone for coalitional scapegoating and that thinking about coalitions is important to maintaining a truth-seeking orientation.
Type-2 worlds have a 10% chance of being destroyed, not a 90% chance. So P(post cold war | type 2) = 90 / 190.
If your main point is that the action is in the prior (rather than the update) and you don’t dispute people’s posteriors, then I think that’s something to flag clearly.
My point is that the difference between SSA and SIA is in the prior. I dispute anthropic shadow posteriors since both SSA and SIA posteriors overwhelmingly believe type 2, but anthropic shadow arguers say you can’t update towards type 2 using the evidence of being post cold war at all.
To me, the correspondences between anthropics and political philosophy seem to be:
SIA corresponds to total utilitarianism (caring more about universes with more population)
SSA corresponds to average utilitarianism (caring about average observers within one’s universe)
Min-max (i.e. act as if you expect to be the worst-off observer) corresponds to Rawlsianism
The difference between Rawlsianism and utilitarianism is critical; if you assume an original position and SSA-like random selection, you get utilitarianism, not Rawlsianism (as has been argued). The min-maxing is critical to Rawls’s theory and he goes to great lengths to distinguish his theory from both utilitarianism and softer forms of min-maxing (continuous risk-aversion, or diminishing marginal “VNM utility” in “personal units of utility”). Bridging Rawlsianism with the overall utilitarian-ish tradition that produces anthropic theories requires justifying min-maxing as a heuristic. (To me, the best justification for min-maxing is that people who don’t optimize for coalitional politics will tend to be losers in coalition politics, and philosophers are supposed to pursue truth, often at the expense of coalitional positioning.)
It seems to me that there isn’t an obvious correspondence between SSA and Heideggerianism. Rather, SSA seems to correspond more strongly to eliminativist, physicalist, anti-phenomenological theories that attempt to reason from an objective view-from-nowhere, with the argument that you learn nothing “upon waking up”. It seems to me that a possible similarity is that the average utilitarianism implied by SSA can trend anti-natalist (since one prefers a world with fewer worse-than-average lives), which has some correspondence with Nazi population control, and which rejects liberal arguments for markets improving overall utility, as markets can increase population at the expense of average welfare. This relates to the Marxist concern that capitalism may lead to workers being paid only subsistence wages in the long term, even if there are many workers.
SSA is perhaps more “parochial” in that, by adjusting one’s reference class, one can identify with different sets of observers, whereas this choice is inessential in SIA; this is more compatible with nationalism. However, it would not in general lead to increasing the population of one’s own reference class, instead focusing on average welfare.
In my recent post I compared the subjective, rather than view-from-nowhere, prior of an anthropic agent to the Kantian a priori; Rawls’s philosophy is based on Kant’s.
Suppose there are 100 planets and in type 1, 99 are destroyed, and in type 2, 10 are destroyed. Suppose your reference class is “all observers”. Then, conditional on type 1, you’re about 0.5% likely to observe being post cold war, and conditional on type 2, you’re about 45% likely to observe being post cold war. These likelihoods are proportional to the SSA posteriors (due to equal priors).
Under SIA, you start with a ~19:10 ratio in favor of type 2 (in the subjective a priori). The likelihood ratios are the same as with SSA so the posteriors are equally weighted towards type 2. So the updates are of equal magnitude in odds space under SSA and SIA.
If your reference class is “post nuclear weapons observers” then I agree that SSA doesn’t update at all. SIA also doesn’t update but starts with a 90:1 prior in favor of type 2. I think this is an odd choice of reference class, and constructing your reference class to depend on your time index nullifies the doomsday argument, which is supposed to be an implication of SSA. I think choices of reference class like this will have odd reflective behavior because e.g. further cold wars in the future will be updated on by default.
He responded on Substack:
Hey Jessica, I really appreciate the thoughtful review. I agree with parts and disagree with parts:
One core objection, which I completely understand, is that “Johnson is primarily concerned with identifying conscious agents experiencing pleasure and pain, for the purpose of optimizing the universe for more pleasure.” PQ does not make this claim, although I understand it could be read as such. Since writing PQ I’ve more explicitly stepped away from hedonic utilitarianism as a practical ethics and consider myself a solid believer in virtue ethics. My North Star in this research is knowledge.
Another objection which I think is useful to discuss is the following:
If our high-level phenomenology has no “crisp, intuitive correspondence with the underlying physics and organizational principles which give rise to it”, it’s unclear why a bottom-up theory would succeed either for the purpose of predicting our high-level phenomenology, which is a core objective in my approach. It would seem that we need to move through many layers of abstraction in the stack, from psychology to neuroscience to chemistry to physics.
This is actually something I see as promising for the Symmetry Theory of Valence: that it may offer a bottom-up frame that can reach up to high-level phenomenology, due to symmetry often being expressed at multiple scales.
I’ll note that I recently released a considerably shorter and somewhat updated paper on the Symmetry Theory of Valence: https://opentheory.net/2023/06/new-whitepaper-qualia-formalism-and-a-symmetry-theory-of-valence/
I think the new paper is much more clear about the motivations around borrowing the symmetry aesthetic from physics, and also proposes better empirical tests, though it has narrower ambitions.
Michael
If enough planets exist, SSA and SIA agree. If you have only type 1 and type 2 universes, you can’t learn that the cold war happened, because the cold war necessarily happened. What you can learn is that you’re a post cold war observer. That’s more likely in type 2 than type 1 universes, since type 2 universes have more observers post cold war. So you still get an update towards type 2.
I think you’re saying that with LHC you could possible attain more certainty about latent risk. With SSA and SIA I think you still reject probability pumping for the some reasons as with the cold war scenario. So eventually you get Bayesian evidence in favor of alternative anthropic theories. The problem is that it’s hard to get generalizable conclusions without specifying that anthropic theory, and the alternative theory to SSA and SIA that gets probability pumping has not been specified.
“Fraction of observers having index 100 who would be correct to believe T” depends on both universes existing simultaneously, e.g. due to a big universe or a multiverse. But I see what you’re getting at, SIA selects observers from the whole distribution over observers taken as a single bag.
In the cold war example in my post, consider changing the post cold war population to higher than the pre cold war population (e.g. 5 billion); the conclusion still goes through.
How does SSA compensate? I thought SIA would do that sort of thing?
It’s like selecting a random observer from all possible universes taken as a single bag. E.g. if there are 2 possible universes with equal initial probability, and one has twice the population of the other, then if you select a random person across universes, you end up in the higher population one with higher probability. The doomsday argument motivates one reason why this might make sense. Also, if you imagine these alternative universes as actually existing due to some kind of big universe theory (just being very large, or many-worlds, or multiversal), then SSA and SIA will tend to agree.
SIA doesn’t handle cases with infinite universes in a well-defined manner. For that you might need some modification like selecting a universe with higher probability if it has more observers per unit computation, or similar. In general, the presumptuous philosopher problem is a counterintuitive implication of SIA.
Getting AI right is mainly a matter of technical competence and technical management competence. DeepMind is obviously much better at those than any government, especially in the AI domain. The standard AI risk threat is not that some company aligns AI to its own values, it’s that everyone dies because AI is not aligned to anyone’s values, because this is a technically hard problem, as has been argued on this website and in other writing extensively. If Google successfully allocates 99% of the universe to itself and its employees and their families and 1% to the rest of the people in the world, that is SO good for everyone’s values compared with the default trajectory, due to a combination of default low chance of alignment, diminishing marginal utility in personal values, and similarity of impersonal values across humans.
If a government were to nationalize AI development, I would think that the NSA was the best choice due to their technical competence, although they aren’t specialized in AI, so this would still be worse than DeepMind. DeepMind founder Shane Legg has great respect for Yudkowsky’s alignment work.
Race dynamics are mitigated by AI companies joining the leader in the AI space, which is currently DeepMind. OpenAI agrees with “merge and assist” as a late-game strategy. Recent competition among AI firms, primarily in LLMs, is largely sparked by OpenAI (see Claude, Bard, Gemini). DeepMind appeared content to release few products in the absence of substantial competition.
Google obviously has no need to sell anything to anyone if they control the world. This sentence is not a logical argument, it is rhetoric.