Could you elaborate on the point in footnote 11 about why you think it doesn’t work? In any concrete problem (Newcomb’s paradox, prisoner’s dilemma, XOR blackmail etc) it seems like all that’s needed is to imagine you made the precommitment at any time before you first had an experience that made you know you’d be facing this problem and need to make a choice (eg before you got the letter in XOR blackmail, but not necessarily before your house was or wasn’t infested with termites since that fact is unknown to you until after getting the letter and making the choice). In what circumstances would you need to think about going further back than that, or some kind of “choice made outside time” perspective? When you worry that agents “might already be in the midst of executing strategies that they might together have precommitted to not doing in advance”, is this not about concrete problems that are sharply bounded in time and have a finite set of well-defined choices like the examples I mentioned, but more like attempts to apply decision theory to ongoing situations that are less well-defined, eg tactics of political negotiation and military aggression in an era of mutually assured nuclear destruction?
hypnosifl
For believers in scientific reductionism, moral realism based on a priori knowledge or fixed “human nature” or mental access to a realm of platonic moral truths is not plausible. But I would argue that people inclined to think in terms of very long (possibly infinite) transhuman futures should be more open to a form of moral realism based on the pragmatist philosopher C.S. Peirce’s limit concept of truth, where objective truth is understood as that which a very long-lived “community of inquiry” would tend to converge on with probability 1 in the limit of infinite time to discuss and experiment (this can be elaborated in terms of the idea of societal belief systems having long-term dynamical attractors, see this paper which interprets Peirce’s concept in this way).
In the moral realm, it may be that a combination of memetic and biological evolution would tend to cause strong convergence on certain norms in the long term, perhaps because individuals can see the consequences of different norms in different subcultures and some may be more universally appealing, and/or because certain norms are more conducive to the continual growth of knowledge (David Deutsch suggested something like the latter in chapter 14 of his book The Fabric of Reality, and Peirce apparently had limited discussion of ethics but this section of the Internet Encyclopedia of Philosophy entry on Peirce’s ‘Architectronics’ says that ‘This makes ethics, for Peirce, a question of what kind of conduct is likely to see the growth of reason or rationality’). This could be compatible with both ideal observer theory (understood in terms of general limit observers rather than just an idealized version of our own idiosyncratic perspective) and the “convergent” version of coherent extrapolated volition.
You responded to Ben’s comment about when you parted ways, but I’d be curious to know how you’d respond to the other point he brought up about various extreme-sounding statements Ziz has made that don’t seem to have been jokes, like “Don’t trust anyone over 30 with a kill count of 0” (about the anarchist transhumanist William Gillis, whose writing she had previously cited in a positive way) or various other examples of violent rhetoric in “A community warning about Ziz” (in the subsection titled ‘Statements by Ziz’).
It seems to me that your belief in her non-involvement in violence relies heavily on your assessment of her character, but I don’t think character is sufficient to predict violence or nonviolence, even “nice” people may be driven to violence if they adopt belief systems that justify it (especially among Rationalists who are known for taking ideas seriously). In particular you allude to the idea that she came to accept “the idea that any moral flaw was a sign of an event horizon signalling eventual total commitment to fractal defection”, but from what I gather this was more of a metaphysical belief than just a statement about psychology (not just something along the lines of bad choices leading to vicious cycles that make further bad choices more likely), one which could lead to a kind of dehumanizing attitude about those deemed metaphysically evil.
I read through her blog a while ago and took some notes—she often talked about every left/right brain hemisphere having a “core” which was something like a soul, and expressed a belief that every core was fundamentally aligned to either good or a binary alternative which she initially called “nongood” (to cover both ‘neutral’ and ‘evil’ according to her ‘Glossary’ post), but later she decided the alternative to “good” was what she called “cancer”, saying in a comment on “Spectral Sight and Good”:
The concept of “nongood” was an incorrect formulation of this epistemic category. Pointing away from the question to resolve, as if there was a second one. The alternative to good is not self-interest, it’s cancer and willful embrace of death.
She also had various comments suggesting this orientation was chosen at some initial point in “logical time” prior to a determinate spacetime history in which these cores were embedded. For example in her post “Choices Made Long Ago” she wrote some comments clarifying what she meant by “long ago”:
The past, your neurotype which “produced” the choice, is also therefore chosen. Just because entropy’s arrow of time makes retrocausation less visible to you does not mean that it is not real. Choose good in all circumstances and physics and biology are forced to have explain you. Forced to furnish you with some kind of strange neurotype that does that. Forced to furnish the world with a way that could have come about. … When you understand this and see that people are still choosing their pasts, continuously for as long as those are their pasts, always doing every action they ever have done or will do, the ideas of mercy forgiveness redemption and indulgence all just collapse to “letting people do evil”.
And in “Intersex Brains and Conceptual Warfare” she wrote:
I’ve said regarding good and evil that reality is retroactively forced to furnish you with a neurotype to explain your choices. Of course logical time does not always accord with entropy’s arrow of time.
You may be right that her notion of a sharp good/nongood binary just started as a sort of empirical hypothesis as she never really attempts to give a theory of why this would be true and in a comment on “Spectral Sight and Good” she said that she came to believe goodness was a “permanent attribute” based on “self-experimentation and examining a few other people”. But it seems like at some point this became more of a foundational belief for her, judging by her blog comments.
She also seemed to think that refusing to back down in any confrontation with evil was a way of “collapsing the timeline”, from what I gather there was some notion different possible timelines existing at different points in logical time, so that by refusing to back down she could create a timeline later in logical time in which the evil wouldn’t challenge her that way in the first place (sort of like a weird version of Newcomb’s paradox or Parfit’s hitchhiker that tests your resolve, but unlike in those examples it seems she thinks the answer is exactly the same even when the agent you’re dealing with is not any kind of ideal predictor). For example see this comment from “Net Negative”:
Usually when I do decision theory, my inner sim treats future branch-points I intend to use TDT to force one way as though I will experience the timelines in sequence. First the one I will collapse, then the true one. This calls emotional preparation to face the unwanted outcome and do the TDT thing, which seems correctly a part of doing the thing for real.
And this comment from “Punching Evil”:
Asserting that I’m supposed to be some cancer agent forming that contract by bargaining, because I’m supposed to have some part of me (“inner animal”) that I’ll buy restraints for. But I don’t want to. I know how to force the hand of fate, slowly over an unimaginable number of lifetimes, an unimaginable number of “eternities” in Boltzmann Hell, so that justice, life, and good win absolutely in the fullness of logical time.
The cosmology or reasoning behind these ideas never seems to have been really elaborated and I can’t make any sense of it, but if Ziz really believed it, it seems like a pretty dangerous combination of ideas: the notion of certain individuals as irredeemably evil combined with the need to refuse to back down from any confrontation as part of some cosmic universe-saving drama. Another one of Ziz’s sometime discussion partners Nis had a post called Killing Evil “People” which is also hard to comprehend but seems to hint at idea that the double-evils are not even real people (with consistent scare quotes around ‘people’, and ‘their’ choices), more like manifestations of the universal “cancer” Ziz had talked about, and defends “lethally opposing them”. In one of the comments Ziz says “I am so fucking glad to finally have an equal.”
Given Ziz’s later close association with people who committed actual murders, from the outside it looks very plausible she was involved in putting this sort of philosophy into practice (she was not accused of physically participating in the attack on Curtis Lind but the prosecutor on the case said that Ziz had been “on the scene, alive and well”, so she may well have been privy to the planning). Do you think I am getting her beliefs badly wrong here, or do you just not think she would have seen those beliefs as ever justifying killing ordinary people deemed “evil” for reasons other than immediate self-defense?
Old thread, but I just wanted to note that one alternative is to view a “cucumber” as a particular algorithmic compression of an underlying fundamental physical state (one in which many different possible fundamental physical states could qualify, like a “macrostate” in statistical mechanics). This was the approach Daniel Dennett took in his paper “Real Patterns”, see in particular the analogy starting on p. 37 with higher-level patterns in Conway’s “Game of Life” cellular automaton, like the pattern we call a “glider”. If one is a mathematical platonist or at least a truth-value realist about the input-output relations of mathematical functions, then if some such function corresponds to determining whether a fundamental physics state in some region of spacetime qualifies as a “cucumber” macrostate (however one wants to define that), in that sense there could be a truth about whether the region contains a cucumber even if there were no intelligent beings to consciously model/experience it. However, this would mean all such microstate-to-macrostate functions are equally “real”, there would be no unique and exclusive way of grouping fundamental particles into higher level-objects that would be the true “natural kinds” while other such groupings would be deemed fundamentally unnatural (see section 1.1.3 here on “promiscuous realism”).
I can’t access the paper by Andersen that you discuss, do you know if schizotypy as Andersen understands it would include the “schizoid” personality type or if he’d consider that distinct? Nancy McWilliams, who wrote an interesting piece about her impressions of schizoid personalities as a psychotherapist, commented on p. 199 of her textbook Psychoanalytic Diagnosis that “Our taxonomic categories remain arbitrary and overlapping, and acting as if there are discrete present-versus-absent differences between labels is not usually wise clinically … Perhaps schizoid psychology, especially in its high-functioning versions, can be reasonably viewed as at the healthy end of the autistic spectrum.” There also seems to be some support for the idea of a schizoid/high-functioning autism spectrum in this study, with “spectrum” here meaning shared common features that would be atypical for neurotypicals, rather than opposite ends of a broader spectrum that includes neurotypicals in the middle as with Andersen’s proposal.
Personally I find that I relate to a lot of the features of schizoid personalities in descriptions like McWilliams’ and also that I relate to a lot of the features of HFA thinking that are distinct from difficulties with reading people’s intentions/meaning, as described for example by the autistic writer Temple Grandin in her book Thinking in Pictures, so this makes it seem plausible to me that at least some subset of schizoid personalities might have a lot of mental overlap with HFA but without the same degree of mind-reading deficits. And some of those shared cognitive features of autistic/schizoid minds are ones that Andersen puts in the “schizotypal” pole that he thinks is opposite to the “autistic” one, which for me raises doubts about his theory as described. For example when it comes to “associative thinking”, on p. 9 of Thinking in Pictures Grandin talks about the highly associative nature of her own thought, which is compatible with her highly “systematizing” nature rather than being opposite to it (maybe one could make an analogy to an animal building up a mental map of a landscape through continual impulsive curiosity-driven exploration):
“If I let my mind wander, the video jumps in a kind of free association from fence construction to a particular welding shop where I’ve seen posts being cut and Old John, the welder, making gates. If I continue thinking about Old John welding a gate, the video image changes to a series of short scenes of building gates on several projects I’ve worked on. Each video memory triggers another in this associative fashion, and my daydreams may wander far from the design problem. … This process of association is a good example of how my mind can wander off the subject. People with more severe autism have difficulty stopping endless associations. I am able to stop them and get my mind back on track. … Interviews with autistic adults who have good speech and are able to articulate their thought processes indicate that most of them also think in visual images. More severely impaired people, who can speak but are unable to explain how they think, have highly associational thought patterns.”
On the other listed characteristics of Andersen’s schizotypal pole which he thinks are opposite to the autistic one, I would guess people on a HFA/schizoid might be more likely then neurotypicals to enjoy a kind of “magical thinking” as a form of imaginative play, even if they are not as likely as “schizotypal” people to literally believe it. (As an example consider H. P. Lovecraft, who in his real life was a hardheaded materialist but had a great fascination with weird fiction that evoked nebulous feelings of transcending ordinary reality, as in his comment in one letter that ‘The true function of phantasy is to give the imagination a ground for limitless expansion, and to satisfy aesthetically the sincere and burning curiosity and sense of awe which a sensitive minority of mankind feel toward the alluring and provocative abysses of unplumbed space and unguessed entity which press in upon the known world from unknown infinities and in unknown relationships of time, space, matter, force, dimensionality, and consciousness.’) And the comment about “Decreased systematising and attention to detail, for instance with tedious matters like finances” may be misleading in treating these two as intrinsically connected, it’s possible to be highly systematizing about subjects that one finds interesting but bad with details of subjects that seem tedious (think of the absent-minded professor stereotype, which fits pretty well with real famous scientists like Einstein who might be part of such a spectrum/cluster). Ozy’s piece discussing intuitive observations of a cluster of mental traits dubbed “plasticbrains” (who are also said to be more likely to be trans) also fits this pattern of being highly systematizing in some areas and bad about keeping track of details in others, with the comment “Plasticbrains people typically have difficulties with executive function. However, their difficulties span a wide range. Some may have relatively ordinary problems, such as procrastination, difficulty planning how long tasks will take, constantly forgetting why they walked into this room, and never knowing where they left their cell phone.”
I don’t think it’s quite right to say the idea of the universe being in some sense mathematical is purely a carry-over of Judeo-Christian heritage—what about the Greek atomists like Leucippus and Democritus for example? Most of their writings have been lost but we do know that Democritus made a distinction similar to the later notion of primary (quantitative) vs. secondary (qualitative) properties discussed at https://plato.stanford.edu/entries/qualities-prim-sec/ with his comment about qualitative sensations being matters of human convention: “By convention sweet and by convention bitter, by convention hot, by convention cold, by convention colours; but in reality atoms and void.” CCW Taylor’s book “The Atomists: Leucippus and Democritus” gathers together all the known fragments from the first two major atomists as well as commentary by other ancient Greek philosophers, it says that various other philosophers attributed to them the position that the only properties of atoms were geometric ones like size and shape and relative position, for example Aristotle’s “Metaphysics” says at http://www.perseus.tufts.edu/hopper/text?doc=Perseus%3Atext%3A1999.01.0052%3Abook%3D1%3Asection%3D985b that for the atomists the “differences” between atoms and groups of atoms were the explanation for all physical reality, and that “These differences, they say, are three: shape, arrangement, and position”. Aristotle’s “On the Heavens” at http://classics.mit.edu/Aristotle/heavens.3.iii.html also says of the atomists “Now this view in a sense makes things out to be numbers or composed of numbers. The exposition is not clear, but this is its real meaning.”
Personally I’m sympathetic to certain forms of panpsychism but I don’t think it’s inconsistent with a mathematical view of nature. Ever since I read Roger Penrose’s book “Shadows of the Mind” as a teenager I’ve been interested in the notion of the three interconnected “worlds” we have to deal with in any broad philosophical account of reality: the physical world, the world of subjective experience, and the world of mathematical truth (you can see Penrose’s memorable diagram of the three worlds and their connections at https://astudentforever.wordpress.com/2015/03/13/roger-penroses-three-worlds-and-three-deep-mysteries-theory/ ). I suppose I have an instinctive monist streak because it always seemed to me philosophers should try to unify these three worlds, the way physicists seek to unify the forces of nature. The notion of “structure” might be a good starting point, since there are good cases for the structuralist perspective (where each part is defined wholly by its relation to other parts, with no purely intrinsic properties) in all three: see mathematical structuralism at https://plato.stanford.edu/entries/structuralism-mathematics/ and structural realism in physics at https://plato.stanford.edu/entries/structural-realism/ (Ladyman and Ross’ book “Every Thing Must Go” makes a good extended case for this) and the idea of a “structuralist” view of qualia at https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3957492/ (and also against the idea that this is just Judeo-Christian, the structuralist view of the mind also has some parallels with branches of Mahayana Buddhism that say that all parts of experience and reality exist only in an interdependent way, using the metaphor of “Indra’s Net”, see http://dharma-rain.org/wp-content/uploads/2016/02/Hua_Yen_Buddhism_Emptiness_Identity_Inte.pdf and note that p. 66 even cites a Buddhist text that can be interpreted as applying this view to numbers as well).
Finally, I’d say that the notion of “reductionism” at the level of predicting physical behavior (the idea that all behavior of more complex systems is in principle derivable from fundamental physical laws acting on basic physical states, whatever those turn out to be exactly) is not primarily a matter of philosophical preconceptions, but more a matter of how this has been a successful paradigm in science which continually expands the range of how many phenomenon can be explained, even if we are far from being able to predict everything in a reductionist way in practice. For example, the range of molecular/chemical behaviors that can be explained in an “ab initio” way from quantum laws has continually expanded over time, likewise the range of cell behaviors that can be explained in terms of biochemical interactions and physical forces, the range of simple brain behaviors or aspects of early embryological development that can be explained in terms of local interactions between cells with one another and with their chemical environment, etc.
I’d make a comparison here to the idea that all adaptive structures in the bodies of living organisms have developed through a process of natural selection acting on mutations that are random with respect to fitness (allowing for the possibility that some adaptive features might be side-effects of others, ‘spandrels’, like the brain’s pattern-seeking abilities being applied in new scenarios not part of an organism’s evolutionary history). We can’t hope in practice to have strong evidence this is true for every adaptive structure in every organism, but evolutionary biologists continually expand the evidence that this is true in all sorts of specific cases, which makes for a good Occam’s razor style case that this is true for all of them. I think the same can be said about the reductionist view that all physical behavior is in principle reducible to physics.
I think it might be helpful to have a variant of 3a that likewise says the orthogonality thesis is false, but is not quite so optimistic as to say the alternative is that AI will be “benevolent by default”. One way the orthogonality thesis could be false would be that an AI capable of human-like behavior (and which could be built using near-future computing power, say less than or equal to the computing power needed for mind uploading) would have to be significantly more similar to biological brains than current AI approaches, and in particular would have to go through an extended period of embodied social learning similar to children, with this learning process depending on certain kinds of sociable drives along with other similar features like curiosity, playfulness, a bias towards sensory data a human might consider “complex” and “interesting”, etc. This degree of convergence with biological structure and drives might make it unlikely it would end up optimizing for arbitrary goals we would see as boring and monomaniacal like paperclip-maximizing, but wouldn’t necessarily guarantee friendliness towards humans either. It’d be more akin to reaching into a parallel universe where a language-using intelligent biological species had evolved from different ancestors, grabbing a bunch of their babies and raising them in human society—they might be similar enough to learn language and engage in the same kind of complex-problem solving as humans, but even if they didn’t pursue what we would see as boring/monomaniacal goals, their drives and values might be different enough to cause conflict.
Eliezer Yudkowsky’s 2013 post at https://www.facebook.com/yudkowsky/posts/10152068084299228 imagined a “cosmopolitan cosmist transhumanist” who would be OK with a future dominated by beings significantly different from us, but who still wants future minds to “fall somewhere within a large space of possibilities that requires detailed causal inheritance from modern humans” as opposed to minds completely outside of this space like paperclip maximizers (in his tweet this May at https://twitter.com/ESYudkowsky/status/1662113079394484226 he made a similar point). So one could have a scenario where orthogonality is false in the sense that paperclip maximizer type AIs aren’t overwhelmingly likely even if we fail to develop good alignment techniques, but where even if the degree of convergence with biological brains is sufficient that we’re likely to get a mind that a cosmopolitan cosmist transhumanist would be OK with (they would still pursue science, art etc.), we can’t be confident we’ll get something completely benevolent by default towards human beings. I’m a sort of Star Trek style optimist about different intelligent beings with broadly similar goals being able to live in harmony, especially in some kind of post-scarcity future of widespread abundance, but it’s just a hunch—even if orthogonality is false in the way I suggested, I don’t think there’s any knock-down argument that creating a new form of intelligence would be free of risk to humanity.
Perhaps one can think of a sort of continuum where on one end you have a full understanding that it’s a characteristic of language that “everything has a name” as in the Anne Sullivan quote, and on the other end, an individual knows certain gestures are associated with getting another person to exhibit certain behaviors like bringing desired objects to them, but no intuition that there’s a whole system of gestures that they mostly haven’t learned yet (as an example, a cat might know that rattling its food bowl will cause its owner to come over and refill it). Even if Hellen Keller was not all the way on the latter end of the continuum at the beginning of the story—she could already request new gestures for things she regularly wanted Anne Sullivan to bring to her or take her to—in the course of the story she might have made some significant leap in the direction of the former end of the continuum. In particular she might have realized that she could ask for names of all sorts of things even if there was no regular instrumental purpose for requesting that Sullivan would bring them over to her (e.g. being thirsty and wanting water).
On the general topic of what the Helen Keller story can tell us about AI and whether complex sensory input is needed for humanlike understanding of words, a while ago I read an article at https://web.archive.org/web/20161010021853/http://www.dichotomistic.com/mind_readings_helen%20keller.html that suggests some reasons for caution. It notes that she was not born blind and deaf, but “lost her sight and hearing after an illness at the age of two”, so even if she had no conscious memory of what vision and hearing were like, they would have figured into her brain development until that point, as would her exposure to language to that age. The end of the article discusses the techniques developed in Soviet institutions to help people who were actually born blind and deaf, like developing their sense of space by “gradually making the deaf/blind child reach further and further for a spoon of food.” It says that eventually they can learn simple fingerspelt commands, and do basic bodily tasks like getting dressed, but only those children who lost their sight and hearing a few years after birth ever develop complex language abilities.
If I’m understanding correctly, if we are thinking about the ECL problem with a one-shot prisoner’s dilemma, the main new wrinkles relative to other problems are that 1) each agent is unsure that the other will actually make the choice recommended by FDT/the great commitment, 2) observing one’s own choice may provide some new evidence about the probability the other agent will make a given choice (probably modeled as a Bayesian update, which some might think of as an ‘acausal influence’), and finally 3) the original FDT/great commitment recommendation has to take all this into account somehow.
It seems likely to me that one could analyze such a problem in an EDT framework with pre-commitments, where both agents have some probability of backing out of the pre-commitments, and don’t know for sure what those probabilities are so they have to estimate them in making their choice. But in such an analysis I wouldn’t think there’d be any need for consideration of the “metaphysical” issues you talk about in footnote 11 about considering the past long before entering this particular prisoner’s dilemma problem. So is your view that such considerations would be needed to deal with the issues 1), 2), 3) I listed above, or do you see separate issues in an ECL prisoner’s dilemma problem?