This is amazing.
small identity
Karma: 64
This does not understand the content of the argument of Yudkowsky’s post.
It doesn’t rely on sampling from a collection of observers throughout the universe: the problem remains without that framing.
Let’s put it like this. Right now, you have the memory of being in an ordered universe, but you’re considering Boltzmann-Schutz brains and feeling a bit woozy. You decide to perform an experiment: you’ll wait fifteen seconds. If your experience dissolves into chaos, Boltzmann brains made a correct prediction. If it doesn’t, they made an incorrect prediction.
After performing the experiment, you get the non-Boltzmann brain result, and this experiment is very well-replicated. So any theory of physics must account for this, even a theory of physics like “all possible observers are instantiated.”
As in the example that Yudkowsky recounts from Feynman, a system of labeled gas particles in a box which instantiates all microstates with uniform probability does not instantiate all macrostates with equal probability. Ones which are more well-mixed are exponentially more common. This a wide-ranging phenomenon: see also the Central Limit Theorem.
The problem isn’t with whether all possible observers are instantiated, but how many or how much they are instantiated. In an eternal classical fixed-size universe at thermal equilibrium, most versions of you will immediately dissolve into chaos five seconds from now. Similarly, most Earths aren’t in a galaxy, they’re in a chaotic sea.
Being a halfer in the Sleeping Beauty problem does not fix this.
The situation is more analogous to the following: A fair coin is flipped. If it lands heads, nothing happens. If it lands tails, an extremely biased towards tails coin is flipped. If that coin lands tails, you are killed. If that coin lands heads, nothing further happens.
The event occurs, and you observe you are still alive. What is the probability the coin landed heads? Very high, and this can be worked out with the usual conditional probability math. The post you linked makes this very point, connecting Sleeping Beauty to the boy-or-girl paradox.
I reject the level IV multiverse, on the grounds of “idk it’s dumb.”
More seriously, I reject it on the grounds that I expect to later find a good reason to reject it.
I like the propagandistic aspects of this story, because I consider them to accurately convey the level of theory of mind that many people are running towards LLMs, and (relatedly) humility about their own minds. Fiction seems an appropriate place for weak men.
I refactored my thinking similarly a while ago.
However, I feel like traditional virtue-ethical notions e.g. “courage,” “integrity,” have the same adversarial Goodharting problem (3) as in your critique of utilitarianism. “Loyalty is when you obey the Master,” “courage is when you go to war against the Enemy without fearing death,” etc. I suspect utilitarianism is maybe only barely worse than other ethical systems in regards to (2). It’s worthwhile to compare EA against regular humans. I don’t really understand either.
I think of virtue ethics as something like “being a healthy and functional cog in the Humanity machine,” where the Humanity machine is ultimately utilitarian.
Further, I think a lot of arguments for utilitarian behaviors “pass through” to virtue ethics insofar as we think that traits like “ambition” and “scope sensitivity” are virtues. I think they are: seeing their characteristic absence has a similar sliminess to seeing a cowardly or slavish person.
(Sometimes it’s just a lack of underlying numeracy, which I would not consider a lack of virtue but rather of education. I spoke to a man who said he wouldn’t suck a dick for a billion dollars, because he just couldn’t. I walked him through the size of a billion, and he changed his mind.)
I am verbally intelligent enough to spin up true-but-socially-plausible accounts of my thoughts unless I am pressed. It is easy to have cached a charming-but-honest way to respond to various adversarial or socially normative questions; it is difficult to come up with them in real time. Usually when I fail at this it’s because I was visibly thinking, from which the interlocutor discerned that I wanted to say something unflattering.
I do not think I am verbally intelligent enough to pick, practice, and retain a subtle meta-honesty policy. It would increase the thinking time too much.
I also do not like making decisions that constrain large numbers of my counterfactual selves. This is a decision-theoretic matter about which I Might Be Objectively Wrong Somehow due to ignorance of the math. Good thing I’m not updateless, and therefore can resolve uncertainty as logical. See first sentence.
I am capable of recursion and meta, and can even consider things that are meta-meta, but at three layers of recursion I usually lose track of what’s going on. I expect I could learn to do this given fifty heterogenous recursion practice exercises but do not have such a list.
Note: The following is not a policy. It is a description of my existing behavior.I sort people into basically three buckets: People, Instruments, and Dogs.
I don’t lie to People, because I don’t like lying to people.
I don’t lie to Instruments except on very specific topics on which they want me to lie. Then I sometimes lie. I try to minimize the number of people I treat as Instruments because they’re cognitively expensive. I interact with them “instrumentally.”
I lie to Dogs, because they want me to lie.
As a general rule, it’s easy to tell if someone’s a Person or a Dog, because the Dogs will tell you. If you treat someone like a Person and they don’t respond like a Dog, even if they’re not a Person they’re usually safe to treat as one. Then I do so, because lying is wrong unless you’re really sure the person wants to be lied to. Again, they usually make this obvious.
There’s also a useful trick where many socially-enforced lies are local.E.g. “How are you doing?” “I’m fine.”
If you’re not fine, you can self-modify to feel fine for five seconds while still not lying.“That’s rules-lawyering, and not in the spirit of literal truth.”
I disagree. “Literal” is a complicated thing which only seems simple. The correspondence between words and reality is subtle. It is easier to count than it is define bijection: it general it is easier to be right than to understand the underlying correspondences behind an existing instance of correctness. Does “fine” mean “not unhappy?” Or does it mean “unharmed?” Can I say “I’m fine” if I’m safe but unhappy when I’m speaking to a person too victim-brained to conflate unhappiness with someone hurting me? To me there isn’t an “obvious literal interpretation” at all. In the context of this social interaction, it barely even has a meaning and is basically a ritual object.Occasionally me and another actual honest person have disagreed about the literal meaning of a question in real-life information exchanges; I thought he had an invalid theory of “deception” but in reality he had different desiderata asking the question than I thought, so his different demarcation of a dishonest answer was natural.
However, I recognize that the above logic can turn you evil, so in general I don’t use it when interacting with People, although I will still occasionally use the self-modification trick.
As a child I did not lie and was subtly punished for it. As an adult I started lying and it’s pretty cool and useful, although it’s quite skill-intensive. I might experiment with lying more because it seems very powerful. I do not think lying is like cigarettes.
I’m giving a detailed answer that goes into how I lie, but to be clear I’m exceptionally scrupulous about honesty and expect I lie extremely rarely by “normal standards.” Even now I want to edit that “extremely” to a “very” or “quite” because it’s not possible that regular people lie that often… right? I confidently expect that in the future people who are curious or curious-but-flinching can ask me about a topic and receive a true answer, unless we reach a level of normalized social violence which will make it obvious-to-both-of-us that truth is extremely rare.
Thanks.
I apologize. I don’t know how to make links in comments.
Where can I read Eliezer’s old non-Sequence blog posts? I recently read Eliezer on The Weighted Majority Algorithm and found it very useful, but this post isn’t even contained in The Original Sequences, as far as I can tell.
Is there a way to access his old writings which is more efficient than scrolling down on his user page for a very long time?
I don’t think Aumann’s agreement theorem is a good way to motivate your normative judgments, though I basically agree with your conclusions. I read Duncan’s post as well and did not really understand why he called you out. You both seem non-malevolent to me.
Bayesianism generalizes logical reasoning to uncertain claims, subject to certain consistency assumptions. Obviously humans are not ideal Bayesians. But in a deeper sense, maybe we’re not supposed to be. Not in an instrumental sense where being Bayesian is incompatible with some kind of good life, but rather in an epistemic sense. Maybe there is some mathematical theory of reasoning, we’ll call it Glorpism, of which humans are an approximation, and it is easier for humans to become more Glorpish than it is for us to become more Bayesian, and becoming more Glorpish is powerful and general in the sense we expect epistemic rationality to be. Glorpism may not have agreement guarantees in the way that Bayesianism does.
Sam Eisenstat’s Condensation is an example of something like this, although I don’t think it’s The Thing. Importantly, Condensation only has the translation theorem to the extent that models are hierarchically organized in a nice way, which does not always hold. (Apologies for any errors, feel free to correct me.)
I also think a purely functionalist account of reasoning error deletes a lot of information. For example, a Ruby that says, “oh, my bad” upon being confronted with evidence from computer analysis of photographs that the different images are all grey is different from a Ruby who changes the topic or flies into a rage. Among the first type of Ruby, those that systematically downgrade or restructure how they assign credence to their color-intuitions after admitting their error is different from those who “bounce back” to their original epistemic state. The best one of these is well-modelled by mistake theory. The worst two, conflict theory.
In real life, I think honest humans often agree to disagree. I do not fully understand why this is and consider this an important problem in the theory of powerful reasoners. I think part of it is that humans perform reasoning using words. Honest words correspond to natural categories but natural categories have an intrinsic misgeneralization problem. If you have two objects, korgs and spangs, which both have exactly half the properties each of bleggs and rubes, but different sets of these properties, then honest people might categorize them differently as bleggs and rubes. But this process is happening below the level of introspective access, so dissolving the question / debucketing has to be done “out loud” in the chamber of consciousness. The act of debucketing / rectifying definitions is a constraint problem with the constraints supplied by one’s introspection on hypotheticals. In general this can take exponential time in the number of traits used to define bleggs and rubes. (I do not have a proof of this, and expect the answer is sensitive to the formulation of the problem. This last claim is purely mathematical intuition.)
Also, our equivalent of Bayesian evidence is our sense-data, which is stored in an extremely unreliable compression system.
Seconding this. I am very smart but not as smart as famous 20th-century scientists.
Modernity is full of so much crystallized intelligence that the gains you receive from fluid intelligence are largely disguised gains-from-trade or straight up gifts-from-benevolent-people. Hanging out with and learning from people smarter than me has revealed how I am objectively similar to stupid people, and then I can treat people stupider than me the way I want to be treated by people smarter than me: I want to be taught, but not condescended to.
“The odds of a Cyberbuddhist Rationalist ending up in this situation with the Anthropic Principle at work are pretty good.”
This is textbook hindsight bias (the textbook is the Sequences.)
Making deductions based on anthropics is intrinsically small data (more accurately it is data which is strongly self-correlated, GIGO) because we do not have empirical access to other possible worlds. Small data / GIGO data comes from priors / sense of beauty / sense of parsimony.Human sense of beauty / parsimony predictably errs towards anthropomorphization, means-end conflation, and wishful thinking. You are human. Being enlightened may help your priors in this matter but not sufficiently to overcome whatever facts about neuroscience consistently produce those errors. Materialism trumps spiritual revelation, e.g. brain damage influencing spiritual attainment.
This post isn’t in the reference class [probability theory], [futurism], or [analytic philosophy]. It’s in the reference class [religious doctrine].
Honestly, I hope the value proposition of this post is to examine whether the LessWrong community will call out bullshit from respected posters.
My prediction that Maitreya is lsusr was correct :).
Maitreya’s post was written by the real lsusr. Source: I have read almost every post on lsusr’s personal blog.
Do what you will with this information, I don’t care. I’m posting this to register my prediction in advance. 5⁄6 confidence.
All methodology is from the first section of the appendix of the linked paper. The paper cited pages 81-87 of Genetics and Analysis of Quantitative Traits. I read from chapter 4 up until those pages to understand the method conceptually. Every niceness assumption is made except for “no shared environment.” For example, “no assortative mating.”
Changing some notation: , i.e. we normalize so that the total “variance due to genes” is 1. We assume that the variance due to shared environment is the same for twins and non-twins, This is a standard assumption in ACE, and it seems reasonable. will represent the “nonlinear part of the effect due to genes,” i.e. that due to epistasis and dominance. is the effect due to alleles, what you called .Facts:
always. (1)
When :. (2)
Note that , so the upper bound becomes trivial when either score is .
Explanation:We can decompose . We can decompose this further into . That is, we’re taking the nonlinear part and decomposing it into interactions involving alleles across loci and dominance effects in loci.
To understand dominance effects, note that a locus can have 0, 1, or 2 instances of an allele. The respective phenotypes resulting from these might not be produced by any linear function on alleles, because not every three points are colinear. The dominance term is the error resulting from a linear regression. If we were haploid, we wouldn’t have to deal with this.
So for example, refers to phenotype effects that only appear when there is a specific combination of two alleles at three separate loci, and are multilinear in the alleles occurring at five other loci.Interpreting this in context, .
Now we can justify our conclusions. Note that the third term is at most but has no lower bound. When we have to write it out, we’ll call it .
Remember that is exactly the proportion of variance due to genes that cannot be captured by a polygenic score, the “phantom heritability.” The paper is concerned with how substantial means , so that if the polygenic score is close to people will assume there is missing heritability when it reality the polygenic score is perfect and the heritability is simply nonlinear.
The ACE Estimate:. This disagrees with the figure in the appendix of the paper. I believe they made an arithmetic error, but it is possible I made a conceptual error.
Recalling , . Those constant terms in the sum go as low as and arbitrarily close to , so by taking the bounds and dividing we recover (1).
The Rule of Thumb:. Remember the bounds on , we can write , where . Combining this with the middle term we have where Doing the arithmetic
. Picking yields (2).
Comments:
I don’t yet rigorously understand how is decomposed into epistasis and dominance. The book gives only an intuition and not a proof. It is very ad hoc.
Edit: As of yesterday, I now understand.
I do not intuitively understand, on the object-level, why worthy successors are harder to build than value-aligned sovereigns. (I understand how they are hardest to verify without >cope.)
Relatedly, if we back up on the problem and work to create more intelligent humans through eugenics, we are in fact attempting to delegate the problem to worthy successors.
My mental model of Eliezer says, “Eugenics is different from from-scratch mind design, because (not only) you are working in a restricted portion of mind-space, you also have great amounts of existing data describing the behavior of that portion.” Does this accurately describe why you think this is a noncentral example which lets it contradict the general tendency, “Worthy successors are harder than aligned sovereigns?”