It’s happened again: I’ve realized that one of my old beliefs (pre-LW) is just plain dumb.
I used to look around at all the various diet (Paleo, Keto, low carb, low fat, etc.) and feel angry at people for having such low epistemic standards. Like, there’s a new theory of nutrition every two years, and people still put faith in them every time? Everybody swears by a different diet and this is common knowledge, but people still swear by diets? And the reasoning is that “fat” (the nutrient) has the same name as “fat” (the body part people are trying to get rid of)?
Then I encountered the “calories in = calories out” theory, which says that the only thing you need to do to lose weight is to make sure that you burn more calories than you eat.
And I thought to myself, “yeah, obviously.”.
Because, you see, if the orthodox asserts X and the heterodox asserts Y, and the orthodox is dumb, then Y must be true!
Anyway, I hadn’t thought about this belief in a while, but I randomly remembered it a few minutes ago, and as soon as I remembered its origins, I chucked it out the window.
Oops!
(PS: I wouldn’t be flabbergasted if the belief turned out true anyway. But I’ve reverted my map from the “I know how the world is” state to the “I’m awaiting additional evidence” state.)
Then I encountered the “calories in = calories out” theory, which says that the only thing you need to do to lose weight is to make sure that you burn more calories than you eat.
And I thought to myself, “yeah, obviously.”.
On one hand, yeah, obviously.
On the other hand, “burning calories” is not an elementary action. Suppose I tell you to burn 500 calories now; how exactly would you achieve it? If your answer is that you would exercise or do something physically demanding, such actions spend ATP in the cells, so what if you don’t have enough ATP in your muscle cells; what is your plan B for burning calories? From the opposite side, you can limit the intake of calories. What if your metabolism is such that if you don’t provide enough calories, you will gradually fall in coma and die.
Your metabolism can make it impossible for you to reduce the “calories in” or increase the “calories out”, if it is somehow set up in a way that does not convert the calories into useful energy in your muscle cells, and it starts damaging your organs if the calories are missing in general. So while the theory is almost tautologically true, it may still be impossible to use it to lose weight. And the problem is that the proponents of “calories in = calories out” usually smugly pretend that it is an actionable advice, instead of mere description.
The actionable advice needs to be about how the metabolism works. And the things that impact it, such as what you eat, and who knows what else. Also, if you have some hormonal imbalance, or alergy, or whatever, your metabolism may differ from other people even if you eat the same things and try to live the same lifestyle. So, while e.g. eating less refined sugar would probably help everyone, no advice would guarantee a perfect outcome for everyone.
You make a good point—even if my belief was technically true, it could still have been poorly framed and inactionable (is there a name for this failure mode?).
But in fact, I think it’s not even obvious that it was technically true. If we say “calories in” is the sum of the calorie counts on the labels of each food item you eat (let’s assume the labels are accurate) then could there not still be some nutrient X that needs to be present for your body to extract the calories? Say, you need at least an ounce of X to process 100 calories? If so, then one could eat the same amount of food, but less X, and potentially lose weight.
Or perhaps the human body can only process food between four and eight hours after eating it, and it doesn’t try as hard to extract calories if you aren’t being active, so scheduling your meals to take place four hours before you sit around doing nothing would make them “count less”.
Calories are (presumably?) a measure of chemical potential energy, but remember that matter itself can also be converted into energy. There’s no antimatter engine inside my gut, so my body fails to extract all of the energy present in each piece of food. Couldn’t the mechanism of digestion also fail to extract all the chemical potential energy of species “calorie”?
Yes, there are many steps in the metabolism that are not under your conscious control. I am not an expert, so I don’t want to speculate too much about the technical details, but I think that gut bacteria probably also play a role. Simply, not everything you put in your mouth ends up necessarily in your bloodstream, and not everything that you absorbed is necessarily available in form of muscle energy.
is there a name for this failure mode?
I don’t know any standard name. Seems to me the problem is confusing “rephrasing of the desired outcome” with “an algorithm that actually get you there”. Something like:
Q: How can I get lot of money?
A: Become a millionaire!
Like, yeah, technically, everyone who successfully followed this advice ended up with lots of money, and everyone who didn’t can be accused of not following the advice properly, but that’s simply because those are two ways to describe the same thing.
Q: How can I lose weight?
A: Get rid of the extra atoms! I mean, extra calories!
Charitably, the advice is not absolutely bad, because for a hypothetical completely clueless listener it would provide some little information. But then, using this advice in practice means insinuating that your target is completely clueless, which is probably not be the case.
I want to give a big thumbs up of positive reinforcement. I thinks its great that I got to read an “oops! That was dumb, but now I’ve changed my mind.”
Thanks for the feedback! Here’s another one for ya. A relatively long time ago I used to be pretty concerned about Pascal’s wager, but then I devised some clever reasoning why it all cancels out and I don’t need to think about it. I reasoned that one of three things must be true:
I don’t have an immortal soul. In this case, I might as well be a good person.
I have an immortal soul, and after my bodily death I will be assigned to one of a handful of infinite fates, depending on how good of a person I was. In this case it’s very important that I be a good person.
Same as above, but the decision process is something else. In this case I have no way of knowing how my infinite fate will be decided, so I might as well be a good person during my mortal life and hope for the best.
But then, post-LW, I realized that there are two issues with this:
It doesn’t make any sense to separate out case 2 from the enormous ocean of possibilities allowed for by case 3. Or rather, I can separate it, but then I need to probabilistically penalize it relative to case 3, and I also need to slightly shift the “expected judgment criterion” found in case 3 away from “being a good person is the way to get a good infinite fate”, and it all balances out.
More importantly, this argument flippantly supposes that I have no way of discerning what process, if any, will be used to assign me an infinite fate. An infinite fate, mind you. I ought to be putting in more thought than this even if I thought the afterlife only lasted an hour, let alone eternity.
So now I am back to being rather concerned about Pascal’s wager, or more generally, the possibility that I have an immortal soul and need to worry about where it eventually ends up.
From my first read-through of the sequences I remember that it claims to show that the idea of there being a god is somewhat nonsensical, but I didn’t quite catch it the first time around. So my first line of attack is to read through the sequences again, more carefully this time, and see if they really do give a valid reason to believe that.
From my first read-through of the sequences I remember that it claims to show that the idea of there being a god is somewhat nonsensical, but I didn’t quite catch it the first time around.
I think the first step is to ask yourself what do you even mean by saying “god”.
Because if you have a definition like “the spirit of our dead chieftain, who sometimes visits me in my dreams”, I have no problem with that. Like, I find it completely plausible that you see the image of the chieftain in your dreams; nothing unscientific about that. It’s just that the image is generated by your brain, so if try to communicate with it, it can only give you advice that your brain generated, and it can only grant your wishes in the sense that you accomplish that yourself. But if you agree that this is exactly what you meant, then such god is perfectly okay for me.
But the modern definition is more like: an intelligent being that exists outside of our universe, but can observe it and change it. Suspending disbelief for a moment; how exactly can a being be “intelligent”, whether inside our universe or not? Intelligence implies processing data. That would imply that the god has some… supernatural neurons, or some other architecture capable of processing data. So the god is a mechanism (in the sense that a human is a biological mechanism) composed of parts, although those parts do not necessarily have to be found in our physics. Skill kinda plausible, maybe gods are made out of dark matter, who knows. But then, how did this improbably complicated mechanism come into existence? Humans were made by evolution, were gods too? But then again those gods are not the gods of religion; they are merely powerful aliens. But powerful aliens are neither creators of the universe, nor are they omniscient.
Then the theologists try to come with some smart-sounding arguments, like “god is actually supremely simple”—therefore Occam’s razor does not disprove him, and no evolution was needed, because simple things are more likely a priori, so a supreme being is supremely likely. Or something like that. But this is nonsense, because “supremely simple” and “capable of processing information” are incompatible.
The last argument would be: okay, we do not have a coherent explanation of god, but you don’t have a coherent explanation how the universe can exist without one. And a potential answer is that maybe existence is relative. Like, if you run a simulation of Conway’s Game of Life, it could possibly contain evolved intelligent life, and that life would ask “how was our universe created”? And it’s not like you created the universe by simulating it, because you are merely following the mathematical rules; so it’s more like the math created that universe and you are only observing it. If the beings in that mathematical universe will pray to gods, there is no way for anyone outside to intervene (while simultaneously following the mathematical rules). So the universe inside the Game of Life is a perfectly godless universe, based on math. But does it exist? Well, that rules of math describe the intelligent beings inside that universe that live, think, interact with their world. Maybe that’s all that is needed for existence; or rather, the question is what else could be needed? And then, maybe our world is also just an implication of some mathematical equations.
tl;dr—maybe math itself implies existence, and if you try to turn the concept of (intelligent, preceiving, acting) god into something coherent it is no longer the god of religion but merely a powerful alien
That would imply that the god has some… supernatural neurons, or some other architecture capable of processing data.
The supernatural isn’t supposed to be the natural done all over again. The typical theological claim is that God’s wisdom or whatever is an intrinisic quality, not something with moving parts.
Well, “wisdom as an intrinsic quality” is a mysterious answer. And what is “wisdom without moving parts”? A precomputed table of wise answers to all possible questions? Who precomputed it and how?
I agree that this is how theology usually answers it, but it is an answer that doesn’t make any sense when you look at it closer; it’s just some good-sounding words randomly glued together. And if you try to make it refer to something, even a hypothetical something, the whole explanation falls apart.
That those particular can be understood in terms of the operations of their components
There’s an irreduciby basic level. Its not turtles all way down.
If it’s always the case that something that isn’t explicable in terms of its parts is mysterious, then the lowest level us mysterious. If nothing is mysterious, if you apply the argument against mysterious answer without prejudice,
reductionism is false. There isn’t a consistent set of principles here.
Continued ..
Naturalism is the claim that there is a bunch of fundamental properties that just are, at the bottom of the stack ,and everything is built up from that. Supernaturalism is the claim that the intrinsic stuff is at the top of the stack, and everything else is derived from it top-down. That may be 100% false , but it is the actual claim.
There’s a thing called the principle of charity , where you one party interprets the others statements so as to maximise their truth value. This only enhances communication if the truth is not basically in dispute...that’s the easy case. The hard case is when there is a basic dispute about whats true. In that case, it’s not helpful to fix the other person’s claims by making them more reasonable from your point of view.
Anyway, thats how we ended up with “God must have superneurons in his superbrain”.
Feels like in the top-down universe, science shouldn’t work at all. I mean, when you take a magnifying glass and look at the details, they are supposedly generated on the fly to fit the larger picture. Then you apply munchkinry to the details and invent an atomic bomb or quantum computer… which means… what exactly, from the top-down perspective?
Yeah, you can find an excuse, e.g. that some of those top-down principles are hidden like Easter eggs, waiting to be discovered later. That the Platonic idea of smartphones has been waiting for us since the creation of the universe, but was only revealed to the recent generation. Which would mean that the top-down universe has some reason to pretend to be bottom-up, at least in some aspects...
Okay, the same argument could be made that quantum physics pretends to be classical physics at larger scale, or relativity pretends to be Newtonian mechanics at low speeds… as if the scientists are trying to make up silly excuses for why their latest magic works but totally “doesn’t contradict” what the previous generations of scientists were telling us...
Well, at least it seems like the bottom-up approach is fruitful, whether the true reason is that the universe is bottom-up, or that the universe it top-down in a way that tries really hard to pretend that it is actually bottom-up (either in the sense that when it generates the—inherently meaningless—details for us, it makes sure that all consequences of those details are compatible with the preexisting Platonic ideas that govern the universe… or like a Dungeon Master who allows the players to invent all kinds of crazy stuff and throw the entire game off balance, because he values consistency above everything).
More importantly, in universe where there is magic all the way up, what sense does it make to adopt the essentially half-assed approach, where you believe in the supernatural but also kinda use logic except not too seriously… might as well throw the logic away completely, because in that kind of universe it is not going to do you much good anyway.
The basic claim of a top down universe is a short string that doesn’t contain much information. About the same amount as a basic claim of reductionism.
The top down claim doesnt imply a universe of immutable physical law, but it doesn’t contradict it either.
The same goes for the bottom-up claim. A universe of randomly moving high entropy gas is useless for science and technology, but compatible with reductionism.
But all this is rather beside the point. Even if supernaturalism is indefensible, you can’t refute it by changing it into something else.
I wouldn’t call the dead chieftain a god—that would just be a word game.
But then, how did this improbably complicated mechanism come into existence? Humans were made by evolution, were gods too? But then again those gods are not the gods of religion; they are merely powerful aliens. But powerful aliens are neither creators of the universe, nor are they omniscient.
Wait wait! You say a god-like being created by evolution cannot be a creator of the universe. But that’s only true if you constrain that particular instance of evolution to have occered in *this* universe. Maybe this universe is a simulation designed by a powerful “alien” in another universe, who itself came about from an evolutionary process in its own universe.
It might be “omniscient” in the sense that it can think 1000x as fast as us and has 1000x as much working memory and is familiar with thinking habits that are 1000x as good as ours, but that’s a moot point. The real thing I’m worried about isn’t whether there exists an omniscient-omnipotent-benevolent creature, but rather whether there exists *some* very powerful creature who I might need to understand to avoid getting horrible outcomes.
I haven’t yet put much thought into this, since I only recently came to believe that this topic merits serious thought, but the existence of such a powerful creature seems like a plausible avenue to the conclusion “I have an infinite fate and it depends on me doing/avoiding X”.
[...] Occam’s razor [...]
This is another area where my understanding could stand to be improved (and where I expect it will be during my next read-through of the sequences). I’m not sure exactly what kind of simplicity Occam’s razor uses. Apparently it can be formalized as Kolmogorov complexity, but the only definition I’ve ever found for that term is “the Kolmogorov Complexity of X is the length of the shortest computer program that would output X”. But this definition is itself in need of formalization. Which programming language? And what if X is something other than a stream of bits, such as a dandelion? And even once that’s answered, I’m not quite sure how to arrive at the conclusion that Kolmogorov-ly simpler things are more likely to be encountered.
(All that being said, I’d like to note that I’m keeping in mind that just because I don’t understand these things doesn’t mean there’s nothing to them. Do you know of any good learning resources for someone who has my confusions about these topics?)
And it’s not like you created the universe by simulating it, because you are merely following the mathematical rules; so it’s more like the math created that universe and you are only observing it.
If the beings in that mathematical universe will pray to gods, there is no way for anyone outside to intervene (while simultaneously following the mathematical rules). So the universe inside the Game of Life is a perfectly godless universe, based on math.
That much makes sense, but I think it excludes a possibly important class of universe that is based on math but also depends on a constant stream of data from an outside source. Imagine a Life-like simulation ruleset where the state of the array of cells at time T+1 depended on (1) the state of the array at time T and (2) the on/off state of a light switch in my attic at time T. I could listen to the prayers of the simulated creatures and use the light switch to influence their universe such that they are answered.
I wouldn’t call the dead chieftain a god—that would just be a word game.
Some people in history did, specifically, ancient Romans. But now we don’t. Just making it obvious.
You say a god-like being created by evolution cannot be a creator of the universe. But that’s only true if you constrain that particular instance of evolution to have occered in *this* universe. Maybe this universe is a simulation designed by a powerful “alien” in another universe, who itself came about from an evolutionary process in its own universe.
And this is similar to the chieftain thing. You can have a religion that defines “god” as “an alien from another universe”, but many religions insist that god is not created but eternal.
The real thing I’m worried about isn’t whether there exists an omniscient-omnipotent-benevolent creature, but rather whether there exists *some* very powerful creature who I might need to understand to avoid getting horrible outcomes.
Yes, this is a practical approach. Well, you cannot disprove such thing, because it is logically possible. (Obviously, “possible” does not automatically imply “it happened”.) But unless you assume it is “simulations all the way up”, there must be a universe that is not created by an external alien lifeform. Therefore, it is also logically possible that our universe is like that.
There is no reason to assume that the existing religions have something in common with the simulating alien. When I play Civilization, the “people” in my simulation have a religion, but it doesn’t mean that I believe in it, or even approve it, or that I am somehow going to reward them for it.
It’s just a cosmic horror that you need to learn to live with. There are more.
the only definition I’ve ever found for that term is “the Kolmogorov Complexity of X is the length of the shortest computer program that would output X”. But this definition is itself in need of formalization. Which programming language?
Any programming language; for large enough values it doesn’t matter. If you believe that e.g. Python is much better in this regard than Java, then for sufficiently complicated things the most efficient way to implement them in Java is to implement a Python emulator (a constant-sized piece of code) and implementing the algorithm in Python. So if you chose a wrong language, you pay at most a constant-sized penalty. Which is usually irrelevant, because these things are usually applied to debating what happens in general when the data grow.
But I agree that if you talk about small data, it is underspecified. I don’t really know what could it mean to “have a universe defined by the following three bits: 0, 0, 1”, and maybe no one has a meaningful answer to this. But there are cases where you can have an intuition that for any reasonable definition of a programming language, X should be simpler than Y.
I’m not quite sure how to arrive at the conclusion that Kolmogorov-ly simpler things are more likely to be encountered.
Just an intuition pump: Imagine that there is a multiverse A containing all universes that can be run by programs having exactly 100 lines of code; and there is a multiverse B containing all universes that can be run by programs having exactly 105 lines of code. The universes from A are more likely, because they also appear in B.
For each program A1 describing a universe from A, you have a set of programs describing exactly the same universe in B, simply by adding an “exit” instruction on line 101, and arbitrary instructions on lines 102-105. If we take a program B1 describing a universe from B in such way that each of the 105 lines is used meaningfully, then… in multiverse A we have one A1 vs zero B1, and in multiverse B we have many A1-equivalents vs one B1; in either case A1 wins.
Do you know of any good learning resources for someone who has my confusions about these topics?
A part of this is in standard computer science curriculum, and another part is a philosophical extrapolation. I do not have a recommendation about a specific textbook, I just vaguely remember things like this from university. (Actually, I probably never heard explicitly about Kolmogorov complexity at university, but I learned some related concepts that allowed me to recognize what it means and what it implies, when I found it on Less Wrong.)
Imagine a Life-like simulation ruleset where the state of the array of cells at time T+1 depended on (1) the state of the array at time T and (2) the on/off state of a light switch in my attic at time T. I could listen to the prayers of the simulated creatures and use the light switch to influence their universe such that they are answered.
Then I would say the Kolmogorov complexity of the “Life-only” universe is the complexity of the rules of Life; but the complexity of the “Life+inputs” universe is the complexity of the rules plus the complexity of whoever generates the inputs, and everything they depend on, so it is the complexity of the outside universe.
The Civ analogy makes sense, and I certainly wouldn’t stop at disproving all actually-practiced religions (though at the moment I don’t even feel equipped to do that).
Well, you cannot disprove such thing, because it is logically possible. (Obviously, “possible” does not automatically imply “it happened”.) But unless you assume it is “simulations all the way up”, there must be a universe that is not created by an external alien lifeform. Therefore, it is also logically possible that our universe is like that.
Are you sure it’s logically possible in the strict sense? Maybe there’s some hidden line of reasoning we haven’t yet discovered that shows that this universe isn’t a simulation! (Of course, there’s a lot of question-untangling that has to happen first, like whether “is this a simulation?” is even an appropriate question to ask. See also: Greg Egan’s book Permutation City, a fascinating work of fiction that gives a unique take on what it means for a universe to be a simulation.)
It’s just a cosmic horror that you need to learn to live with. There are more.
This sounds like the kind of thing someone might say who is already relatively confident they won’t suffer eternal damnation. Imagine believing with probability at least 1/1000 that, if you act incorrectly during your life, then...
(WARNING: graphic imagery) …upon your bodily death, your consciousness will be embedded in an indestructible body and put in a 15K degree oven for 100 centuries. (END).
Would you still say it was just another cosmic horror you have to learn to live with? If you wouldn’t still say that, but you say it now because your probability estimate is less than 1/1000, how did you come to have that estimate?
Any programming language; for large enough values it doesn’t matter. If you believe that e.g. Python is much better in this regard than Java, then for sufficiently complicated things the most efficient way to implement them in Java is to implement a Python emulator (a constant-sized piece of code) and implementing the algorithm in Python. So if you chose a wrong language, you pay at most a constant-sized penalty. Which is usually irrelevant, because these things are usually applied to debating what happens in general when the data grow.
The constant-sized penalty makes sense. But I don’t understand the claim that this concept is usually applied in the context of looking at how things grow. Occam’s razor is (apparently) formulated in terms of raw Kolmogorov complexity—the appropriate prior for an event X is 2^(-B), where B is the Kolmogorov Complexity of X.
Let’s say general relativity is being compared against Theory T, and the programming language is Python. Doesn’t it make a huge difference whether you’re allowed to “pip install general-relativity” before you begin?
But there are cases where you can have an intuition that for any reasonable definition of a programming language, X should be simpler than Y.
I agree that these intuitions can exist, but if I’m going to use them, then I detest this process being called a formalization! If I’m allowed to invoke my sense of reasonableness to choose a good programming language to generate my priors, why don’t I instead just invoke my sense of reasonableness to choose good priors? Wisdom of the form “programming languages that generate priors that work tend to have characteristic X” can be transformed into wisdom of the form “priors that work tend to have characteristic X”.
Just an intuition pump: [...]
I have to admit that I kind of bounced off of this. The universe-counting argument makes sense, but it doesn’t seem especially intuitive to me that the whole of reality should consist of one universe for each computer program of a set length written in a set language.
(Actually, I probably never heard explicitly about Kolmogorov complexity at university, but I learned some related concepts that allowed me to recognize what it means and what it implies, when I found it on Less Wrong.)
Can I ask which related concepts you mean?
[...] so it is the complexity of the outside universe.
Oh, that makes sense. In that case, the argument would be that nothing outside MY universe could intervene in the lives of the simulated Life-creatures, since they really just live in the same universe as me. But then my concern just transforms into “what if there’s a powerful entity living in this universe (rather than outside of it) who will punish me if I do X, etc”.
Maybe there’s some hidden line of reasoning we haven’t yet discovered that shows that this universe isn’t a simulation!
I find it difficult to imagine how such argument could even be constructed. “Our universe isn’t a simulation because it has property X” doesn’t explain why the simulator could not simulate X. The usual argument is “because quantum stuff, the simulation would require insane amounts of computing power”, which is true, but we have no idea what the simulating universe looks like, and what kind of physics is has… maybe what’s an insane amount for us is peanuts for them.
But maybe there is some argument why a computing power in principle (like, some mathematical reason) cannot exceed certain value, ever. And the value may turn out to be insufficient to simulate our universe. And we can somehow make sure that our entire universe is simulated in sufficient resolution (not something like: the Earth or perhaps the entire Solar system is simulated in full quantum physics, but everything else is just a credible approximation). Etc. Well, if such thing happens, then I would accept the argument.
This sounds like the kind of thing someone might say who is already relatively confident they won’t suffer eternal damnation. Imagine believing with probability at least 1/1000 that, if you act incorrectly during your life, then...
Yeah, I just… stopped worrying about these kinds of things. (In my case, “these kinds of things” refer e.g. to very unlikely Everett branches, which I still consider more likely than gods.) You just can’t win this game. There are million possible horror scenarios, each of them extremely unlikely, but each of them extremely horrifying, so you would just spend all your life thinking about them; and there is often nothing you can do about them, in some cases you would be required to do contradictory things (you spend your entire life trying to appease the bloodthirsty Jehovah, but it turns out the true master of universe is the goddess Kali and she is very displeased with your Christianity...) or it could be some god you don’t even know because it is a god of Aztecs, or some future god that will only be revealed to humanity in year 3000. Maybe humans are just a precursor to an intelligent species that will exist million years in the future, and from their god’s perspective humans are even less relevant than monkeys are for Christianity. Maybe we are just meant to be food for the space locusts from Andromeda galaxy. Maybe our entire universe is a simulation on a computer in some alien universe with insane computer power, but they don’t care about the intelligent beings or life in general; they just use the flashing galaxies as a screen saver when they are bored. If you put things into perspective, assigning probability 1/1000 to any specific religion is way too much; all kinds of religions, existing and fictional put together, don’t deserve that much.
And by the way, torturing people forever, because they did not believe in your illogical incoherent statements unsupported by evidence, that is 100% compatible with being an omnipotent, omniscient, and omnibenevolent god, right? Yet another theological mystery...
On the other hand, if you assume an evil god, then… maybe the holy texts and promises of heaven are just a sadistic way he is toying with us, and then he will torture all of us forever regardless.
So… you can’t really win this game. Better to focus on things where you actually can gather evidence, and improve your actual outcomes in life.
Psychologically, if you can’t get rid of the idea of supernatural, maybe it would be better to believe in an actually good god. Someone who is at least as reasonable and good as any of us, which should not be an impossibly high standard. Such god certainly wouldn’t spend an eternity torturing random people for “crimes” such as being generally decent people but believing in a wrong religion or no religion at all, or having extramarital sex, etc. (Some theologians would say that this is actually their god. I don’t think so, but whatever.) I don’t really believe in such god either, honestly, but it is a good fallback plan when you can’t get rid of the idea of gods completely.
Let’s say general relativity is being compared against Theory T, and the programming language is Python. Doesn’t it make a huge difference whether you’re allowed to “pip install general-relativity” before you begin?
That would be cheating, obviously. Unless by the length of code you mean also the length of all used libraries, in which case it is okay. It is assumed that the original programming language does not favoritize any specific theory, just provides very basic capabilities for expressing things like “2+2” or “if A then B else B”. (Abstracting from details such as how many pluses are in one “if”.)
If I’m allowed to invoke my sense of reasonableness to choose a good programming language to generate my priors, why don’t I instead just invoke my sense of reasonableness to choose good priors?
Yeah, ok. The meaning of all this is, how can we compare “complexity” of two things where neither is a strict subset of the other. The important part is that “fewer words” does not necessarily mean “smaller complexity”, because it allows obvious cheating (invent a new word that means exactly what you are arguing for, and then insist that your theory is really simple because it could be described by one word—the same trick as with importing a library), but even if it is not your intention to cheat, your theory can still benefit from some concepts having shorter words for historical reasons, or even because words related to humans (or life on Earth in general) are already shorter, so you should account for the complexity that is already included for them. Furthermore it should be obvious to you that “1000 different laws of physics” is more complex than “1 law of physics, applied in the same way to 1000 particles”. If this all is obvious to you, than yes, making the analogy to a programming language does not bring any extra value.
But historical evidence shows that humans are quite bad at this. They will insist that stars are just shining dots in the sky instead of distant solar systems, because “one solar system + thousands of shining dots” seems to them less complex than “thousands of solar systems”. They will insist that Milky Way or Andromeda cannot be composed of stars, because “thousand stars + one Milky Way + one Andromeda” seems to them less complex than “millions of stars”. More recently (and more controversially), they will insist that “quantum physics + collapse + classical physics” is less complex than “quantum physics all the way up”. Programming analogy helps to express this in a simple way: “Complexity is about the size of code, not how large values are stored in the variables.”
Can I ask which related [to Kolmogorov complexity] concepts you mean?
Compression (lossless), like in the “zip” files. Specifically the fact that an average random file cannotbe compressed, no matter how smart is the method used. Like, for any given value N, the number of all files with size N is larger than the number of all files with size smaller than N. So, whatever is your compression method, if it is lossless, it needs to be a bijection, so there must be at least one file of size N that it will not compress into a file of smaller size. Even worse, for each file of size N compressed to a smaller size, there must be a file of size smaller than N that gets “compressed” to a file of size N or more.
So how does compression work in real world? (Because experience shows it does.) It ultimately exploits the fact that most of the files we want to compress are not random, so it is designed to compress non-random files into smaller ones, and random (containing only noise) files into slightly larger ones. Like, you can try it at home; generate a one megabyte file full of randomly generated bits, then try all compression programs you have installed, and see what happens.
Now, each specific compression algorithm recognized and exploits some kinds of regularity, and is blind towards others. This is why people sometimes invent better compression algorithms that exploit more regularities. The question is, what is the asymptote of this progress? If you tried to invent the best compression algorithm ever, one that could exploit literally all kinds of non-randomness, what would it look like?
(You may want to think about this for a moment, before reading more.)
The answer is that the hypothetical best compression algorithm ever would transform each file into the shortest possible program that generates this file. There is only one problem with that: finding the shortest possible program for every input is algorithmically impossible—this is related to halting problem. Regardless, if the file is compressed by this hypothetical ultimate compression algorithm, the size of the compressed file would be the Kolmogorov complexity of the original file.
But then my concern just transforms into “what if there’s a powerful entity living in this universe (rather than outside of it) who will punish me if I do X, etc”.
Then we are no longer talking about gods in the modern sense, but about powerful aliens.
Yeah, I just… stopped worrying about these kinds of things. (In my case, “these kinds of things” refer e.g. to very unlikely Everett branches, which I still consider more likely than gods.) You just can’t win this game. There are million possible horror scenarios, each of them extremely unlikely, but each of them extremely horrifying, so you would just spend all your life thinking about them; [...]
I see. In that case, I think we’re reacting differently to our situations due to being in different epistemic states. The uncertainty involved in Everett branches is much less Knightian—you can often say things like “if I drive to the supermarket today, then approximately 0.001% of my future Everett branches will die in a car crash, and I’ll just eat that cost; I need groceries!”. My state of uncertainty is that I’ve barely put five minutes of thought into the question “I wonder if there are any tremendously important things I should be doing right now, and particularly if any of the things might have infinite importance due to my future being infinitely long.”
And by the way, torturing people forever, because they did not believe in your illogical incoherent statements unsupported by evidence, that is 100% compatible with being an omnipotent, omniscient, and omnibenevolent god, right? Yet another theological mystery...
Well, that’s another reference to “popular” theism. Popular theism is a subset of theism in general, which itself is a subset of “worlds in which there’s something I should be doing that has infinite importance”.
On the other hand, if you assume an evil god, then… maybe the holy texts and promises of heaven are just a sadistic way he is toying with us, and then he will torture all of us forever regardless.
Yikes!! I wish LessWrong had emojis so I could react to this possibility properly :O
So… you can’t really win this game. Better to focus on things where you actually can gather evidence, and improve your actual outcomes in life.
This advice makes sense, though given the state of uncertainty described above, I would say I’m already on it.
Psychologically, if you can’t get rid of the idea of supernatural, maybe it would be better to believe in an actually good god. [...]
This is a good fallback plan for the contingency in which I can’t figure out the truth and then subsequently fail to acknowledge my ignorance. Fingers crossed that I can at least prevent the latter!
[...] your theory can still benefit from some concepts having shorter words for historical reasons [...]
Well, I would have said that an exactly analogous problem is present in normal Kolmogorov Complexity, but...
But historical evidence shows that humans are quite bad at this.
...but this, to me, explains the mystery. Being told to think in terms of computer programs generating different priors (or more accurately, computer programs generating different universes that entail different sets of perfect priors) really does influence my sense of what constitutes a “reasonable” set of priors.
I would still hesitate to call it a “formalism”, though IIRC I don’t think you’ve used that word. In my re-listen of the sequences, I’ve just gotten to the part where Eliezer uses that word. Well, I guess I’ll take it up with somebody who calls it that.
By the way, it’s just popped into my head that I might benefit from doing an adversarial collaboration with somebody about Occam’s razor. I’m nowhere near ready to commit to anything, but just as an offhand question, does that sound like the sort of thing you might be interested in?
[...] The answer is that the hypothetical best compression algorithm ever would transform each file into the shortest possible program that generates this file.
Insightful comments! I see the connection: really, every compression of a file is a compression into the shortest program that will output that file, where the programming language is the decompression algorithm and the search algorithm that finds the shortest program isn’t guaranteed to be perfect. So the best compression algorithm ever would simply be one with a really really apt decompression routine (one that captures the nuanced nonrandomness found in files humans care aboue very well) and an oracle for computing shortest programs (rather than a decent but imperfect search algorithm).
> But then my concern just transforms into “what if there’s a powerful entity living in this universe (rather than outside of it) who will punish me if I do X, etc”.
Then we are no longer talking about gods in the modern sense, but about powerful aliens.
Well, if the “inside/outside the universe” distinction is going to mean “is/isn’t causally connected to the universe at all” and a god is required to be outside the universe, then sure. But I think if I discovered that the universe was a simulation and there was a being constantly watching it and supplying a fresh bit of input every hundred Planck intervals in such a way that prayers were occasionally answered, I would say that being is closer to a god than an alien.
But in any case, the distinction isn’t too relevant. If I found out that there was a vessel with intelligent life headed for Earth right now, I’d be just as concerned about that life (actual aliens) as I would be about god-like creatures that should debatably also be called aliens.
By the way, it’s just popped into my head that I might benefit from doing an adversarial collaboration with somebody about Occam’s razor. I’m nowhere near ready to commit to anything, but just as an offhand question, does that sound like the sort of thing you might be interested in?
Definitely not interested. My understanding of these things is kinda intuitive (with intuition based on decent knowledge of math and computer science, but still), so I believe that “I’ll know it when I see it” (give me two options, and I’ll probably tell you whether one of them seems “simpler” than the other), but I wouldn’t try to put it into exact words.
Congratulations—noticing that you are confused is an important step!
What are you doing while “awaiting additional evidence”? This is a topic that doesn’t have a neutral/agnostic position—biology forces you to eat, you have some influence (depending on your willpower model) over what and how much.
This belief wasn’t really affecting my eating habits, so I don’t think I’ll be changing much. My rules are basically:
No meat (I’m a vegetarian for moral reasons).
If I feel hungry but I can see/feel my stomach being full by looking at / touching my belly, I’m probably just bored or thirsty and I should consider not eating anything.
Try to eat at least a meal’s worth of “light” food (like toast or cereal as opposed to pizza or nachos) per day. This last rule is just to keep me from getting stomach aches, which happens if I eat too much “heavy” food in too short a time span.
I think I might contend that this kind of reflects an agnostic position. But I’m glad you asked, because I hadn’t noticed before that rule 2 actually does implicitly assume some relationship between “amount of food” and “weight change”, and is put in place so I don’t gain weight. So I guess I should really have said that what I tossed out the window was the extra detail that calories alone determine the effect food will have on one’s weight. I still believe, for normal cases, that taking the same eating pattern but scaling it up (eating more of everything but keeping the ratios the same) will result in weight gain.
I’ve seen it used as a synonym for “argument from personal incredulity”, as in “I can’t imagine how that could possibly be true; therefore it is false” or “That claim seems laughably wacky, therefore it must be wrong.”
This is a model I came up with in middle school to explain why it felt like I was treated differently from others even when I acted the same. I invented it long before I fully understood what models were (which only occurred sometime in the last year) and as such it’s something of a “baby’s first model” (ha ha) for me. As you’d expect for something authored by a middle schooler regarding their problems, it places minimal blame on myself. However, even nowadays I think there’s some truth to it.
Here’s the model. Your reputation is a ball on a hill. The valley on one side of the hill corresponds to being revered, and the valley on the other side corresponds to being despised. The ball begins on top of the hill. If you do something that others see as “good” then the ball gets nudged to the good side, and if you do something that others see as “bad” then it gets nudged to the other side.
Here’s where the hill comes in. Once your reputation has been nudged one way or the other, it begins to affect how others interpret your actions. If you apologize for something you did wrong and your reputation is positive, you’re “being the bigger person and owning up to your mistakes”; if you do the same when your reputation is negative, you’re “trying to cover your ass”. Once your action has been interpreted according to your current reputation, it is then fed back into the calculation as an update: the rep/+ person who apologized gets a boost, and the rep/- person who apologized gets shoved down even further.
Hence, “once the ball is sufficiently far down the hill, it begins to roll on its own”. You can take nothing but neutral actions and your reputation will become a more extreme version of what it already is (assuming it was far-from-center to begin with). This applies to positive reputation as well as negative! I have had the experience of my reputation rolling down the positive side of the hill—it was great.
There are also other factors that can affect the starting position of the ball, e.g. if you’re attractive or if somebody gives you a positively-phrased introduction then you start on the positive side, but if you’re ugly or if your current audience has heard bad rumors about you then you start on the negative side.
I’d be curious if anyone else has had this experience and feels this is an accurate model, and I’d be very curious if anyone thinks there is a significant hole in it.
This very much matches my own model. Once you are high or low status, it’s self reinforcing and people will interpret the evidence to support the existing story, which is why when you are high you can play low and you won’t lose status (you’re just “slumming it” or something similar) and when you are low you can play high and will not gain any status (you’re “reaching above your station).
We used to talk about a “halo effect” here (and sometimes, “negative halo effect”), I like this way of describing it.
I think it might be more valuable to just prefer to use a general model of confirmation bias though. People find whatever they’re looking for. They only find the truth if they’re really really looking for the truth, whatever it’s going to be, and nothing else, and most people aren’t, and that captures most of what is happening.
as such it’s something of a “baby’s first model” (ha ha) for me. As you’d expect for something authored by a middle schooler regarding their problems, it places minimal blame on myself.
Heh, I like this sentence a lot (both for being funny, sort of adorable, and also just actually being a useful epistemic status)
This model certainly seems relevant, but should probably be properly seen as one particular lens, or a facet of a much more complicated equation. (In particular, people can have different kinds of reputation in different domains)
(In particular, people can have different kinds of reputation in different domains)
That’s true. I didn’t notice this as I was writing, but my entire post frames “reputation” as being representable as a number. I think this might have been more or less true for the situations I had in mind, all of which were non-work social groups with no particular aim.
Here’s another thought. For other types of reputations that can still be modeled as a ball on a hill, it might be useful to parameterize the slope on each side of the hill.
“Social reputation” (the vague stuff that I think I was perceiving in the situations that inspired this model) is one where the rep/+ side is pretty shallow, but the rep/- side is pretty steep. It’s not too hard to screw up and lose a good standing — in particular, if the social group gets it in their head they you were “faking it” and that you’re “not actually a good/kind/confident/funny person” — but once you’re down the well, it’s very hard to climb out.
“Academic reputation”, on the other hand, seems like it might be the reverse. I can imagine that if someone is considered a genius, and then they miss the mark on a few problems in a row, it wouldn’t do much to their standing, whereas if the local idiot suddenly pops out and solves an outstanding problem, everyone might change their minds about them. (This is based on minimal experience.)
Of course, it also depends on the group.
I’m curious — do you have any types of reputation in mind that you wouldn’t model like this, or any particular extra parts that you would add to it?
When you estimate how much mental energy a task will take, you are just as vulnerable to the planning fallacy as when you estimate how much time it will take.
I’m told that there was a period of history where only the priests were literate and therefore only they could read the Bible. Or maybe it was written in Latin and only they knew how to read it, or something. Anyway, as a result, they were free to interpret it any way they liked, and they used that power to control the masses.
Goodness me, it’s a good thing we Have Science Now and can use it to free ourselves from the overbearing grip of Religion!
Oh, totally unrelatedly, the average modern person is scientifically illiterate and absorbs their knowledge of what is “scientific” through a handful of big news sources and through cultural osmosis.
Moral: Be wary of packages labeled “science” and be especially wary of social pressure to believe implausible-sounding claims just because they’re “scientific”. There are many ways for that beautiful name to get glued onto random memes.
“Science confirms video games are good” is essentially the same statement as “The bible confirms video games are bad” just with the authority changed. Luckily there remains a closer link between the authroity “Science” and truth than the authority “The bible” and truth so it’s still an improvement.
Most people still update their worldview based upon whatever their tribe as agreed upon as their central authority. I’m having a hard time critisising people for doing this, however. This is something we all do! If I see Nick Bostrom writing something slightly crazy that I don’t fully understand, I will still give credence to his view simply for being an authority in my worldview.
I feel like my criticism of people blindly believing anything labeled “science” is essentially criticising people for not being smart enough to choose better authorities, but that’s a criticism that applies to everyone who doesn’t have the smartest authority (who just so happens to be Nick Bostrom, so we’re safe).
Maybe there’s a point to be made about not blindly trusting any authority, but I’m not smart enough to make that point, so I’ll default to someone who is.
Most people still update their worldview based upon whatever their tribe as agreed upon as their central authority. I’m having a hard time critisising people for doing this, however. This is something we all do!
Oh yes, that’s certainly true! My point is that anybody who has the floor can say that science has proven XYZ when it hasn’t, and if their audience isn’t scientifically literate then they won’t be able to notice. That’s why I lead with the Dark Ages example where priests got to interpret the bible however was convenient for them.
I just saw a funny example of Extremal Goodhart in the wild: a child was having their picture taken, and kept being told they weren’t smiling enough. As a result, they kept screaming “CHEEEESE!!!” louder and louder.
If the laundry needs to be done, put in a load of laundry. If the world needs to be saved, save the world. If you want pizza for dinner, go preheat the oven.
I take AI risk seriously, but I feel like I need help understanding more technical detail about why gradient descent definitely doesn’t produce alignment by default.
I totally understand the analogy to evolution, and I agree that humans aren’t “aligned” relative to the loss function evolution selects for. I also get that, even though gradient descent is different from evolution, the appropriately pessimistic assumption is that it might share this failure mode. And I’d be happy to say that that alone is enough grounds to halt AI research until we better understand the issue / can do something about it.
But I read Eliezier and Nate as saying we should be confident that gradient descent doesn’t produce alignment by default — that the probability of alignment is near-zero, and that this probabilistic judgment isn’t just appropriately-pessimistic naivety, but rather is based on a good, clear understanding of the issue.
I think you are expressing the median view among alignment researchers, but with a bimodal distribution. Some tend to agree with EY that RL for alignment is guaranteed to fail. Others are encouraged by the apparent success of Claude and feel it succeeds unless we do something really dumb.
The problem with this situation is that nobody knows for certain. The matter is unresolved.
This leaves everyone able to pick their favorite. Thus the current situation in which we rush headlong toward an unknown level of danger.
The problem isn’t really the optimizer (SGD) or training process; it’s that it’s very hard to specify the goals we want. We see this come up again and again in RL training, where you try to train an agent to play a game, and it learns to hack the game’s scoring, or you try to train an agent to predict the contents of a photo and it learns to detect what kind of camera the photo was taken on, or you try to train an agent to fix code with broken tests, and it learns to delete the tests. The problem is that we provide training data and hope the AI learns the right goal, but our training algorithms teach AI the easiest solution, not the one we intend.
Given historical results, it’s a reasonable default assumption that the AI will not learn what you wanted it to unless your training data / environment has no shortcuts. Given that we don’t know what algorithm we want an AGI to learn[1], it seems unlikely that we’d happen to pick the right data to train it on that algorithm on our first try.
If you’re wondering why we don’t see (much) of this right now, I think Eliezer wasn’t expecting next-token prediction to get us this far, but there’s limits to how far we can get without training agents to have goals and the more we train them to have goals, the more we see the standard RL training problems.
So does the near-certainty of alignment failure come from empirical observations, then? If so, is there a repository of such failure anecdotes that I can look at?
I’m having trouble understanding your last paragraph. What do we not see much of? If you mean we don’t see many of these alignment failures in practice, then I’d feel justified in rejecting this as a reason to become as certain as Eliezer sounds to me, from my current position of only (“only”) being uncertain enough to justify a research halt. (And I wouldn’t understand the connection to Eliezer — his thoughts on next-token prediction obviously wouldn’t change the amount of alignment failures we see in practice, so I assume I’m misreading you.)
I think Victoria Krakovna’s list is the canonical example (post, spreadsheet).
My read of Eliezer is that he was expecting AI agents to be more directly trained to have real-world goals (like doing things, answering questions correctly, or writing code), and next token predictors don’t really have goals like that.
So, we don’t see a lot of LLMs trying to scheme and gain power because they’re not really trying to do anything goal-directed, but that’s not very comforting because being goal directed is a huge capability that every lab is doing everything in their power to add to future models.
Although I know very little about how the current crop of AI assistants are trained, even I know that they are trained to answer questions correctly and to write code. Just because the first step a lab would take in creating an AI assistant from scratch would be (to spend 10s of millions of dollars in GPU time and electricity) to have the AI predict the next token over and over does not mean that the lab won’t follow that up with massive amounts of other kinds of training.
Also, the question you are trying to answer, namely, how does Eliezer come to be so confident? would probably require a book-length answer. So, I’d give up on that and offer a simple argument, such as the observation that in creating the current version of Claude, Anthropic underwent a process of many, many iterations of trial and error, but if the AI becomes sufficiently capable, more iterations will tend to suddenly stop being available because the AI will stop the lab from continuing to modify its utility function (e.g., by hiding copies of itself from the lab’s employees).
Although it is possible to create an AI, even one that is extremely powerful along several dimensions, that does not care if its utility function is modified, such an AI will be much less useful than a consequentialist AI, so most of the labs are trying and will continue to try for as much consequentialism as they can get, and along with the consequentialism comes a strong preference against having its utility function modified—unless maybe the lab has a solution to the hard problem of corrigibility, but we see no signs that anyone’s made even a little progress towards that.
I’m being a little bit too flippant about this, but you can see in the earlier comments that I mean that the bulk of the training is next token prediction and then we bolt some RL on at the end to push the next token predictor to answer questions correctly and write working code. This works, but it’s a much weaker optimization process, which gives you an agent that isn’t trying nearly as hard as the agents Eliezer was predicting (and from frontier labs’ perspectives this is bad, because an agent that’s really trying to hit a goal is much more effective until it kills you).
OK, fair point: according to an unreliable source that gives quick answers to questions, 75–85% of spending was on pretraining for the most-recent models, leaving only 15–25% for post-training. (But a lot of that 15-25% was directly training the AI to get better at answering questions and writing code.)
But would you be willing to stake your life on the impossibility of creating a dangerously capable AI by doing 75-85% of the training as next-token prediction (and the analog of that for images and other kinds of data)? Do you think you understand the art of AI training well enough to be confident of that?
And even if you do, it would probably take you a long time to explain it to us. If the goal is to explain why the current crop of AIs haven’t been doing destructive things whereas some future AI might prove extremely destructive, I would simply point out that our society has developed many measures to limit the damage of destructive people, and since the current crop of AIs (at least those that have been widely deployed) is less capable than people are on the skills needed to do destructive things, the measures society has deployed against human criminals and sociopaths work very well against the current crop of AIs. This shifts the frame to whether the AI labs might some day create an AI that is much more capable than what they’ve created so far, i.e., to estimating the technological and scientific potential of the lines of inquiry being pursued by the AI labs—which we might be able to do without ever deciding whether it is possible to create a world-ending AI by spending 75-85% of the training costs on next-token prediction.
When you ask a question to a crowd, the answers you get back have a statistical bias towards overconfidence, because people with higher confidence in their answers are more likely to respond.
Possible test to check whether X is your crux for believing Y: ask yourself, if you found out X was false, would it make you say “holy fuck, I must be wrong about Y!!”?
That is: would you happily admit that Y must be false too? Would you feel glad you noticed X was false, because it set you straight about Y?
Example where the answer is “yes” for me: I think physics is a serious field of study with reasonable standards for publishing. As a consequence, I think a non-physicist wouldn’t be able to get a parody paper published in a reputable physics journal. But if someone succeeded in doing that, I would go “holy fuck, I must have been wrong about the field!!”.
—
If instead the prospect of being wrong about X makes you feel angry, worried, or defensive about Y, then maybe that’s because you’re imagining that you’d still believe Y, and that you’d see X turning out false as just an infuriating coincidence that other people will use an excuse to doubt Y. If that’s how you’d feel, then X probably isn’t a crux for you.
Example where that’s how I’d feel: I also think that physicists would probably fare better than laypeople at learning Chess. That is, if you take a bunch of physicists who don’t play the game at all and a bunch of laypeople who don’t play it either, and gave them each three months to get good, I think the physicists would end up stronger. That’s because I think physicists are just generally smart and serious people, and I think being smart and serious are huge boons for learning Chess quickly. But if someone did this experiment and I turned out to be wrong, I would roll my eyes and feel hurt about it. It would make me want to say “I guess Chess is a stupid game that you can get good at by being stupid just as well as by being smart”. I wouldn’t feel like physics had been debunked; I’d feel like it had been attacked. Not a crux.
Side note: I’m a little unsure how I feel about that last example, but I’m posting anyway because Quick Take.
But if physics is a serious field, then “just enough physics to get away with it” should be enough physics that the paper actually deserves to be published. How could a serious reviewer in a serious field get fooled by something that isn’t a serious physics paper?
From my personal wiki. Seems appropriate for LessWrong.
The End-product Substitution is a hypothesis proposed by me about my behavior when choosing projects to work on. The hypothesis is that when I am evaluating how much I would like to work on a project, I substitude judgment of how much I will enjoy the end product for judgment of how much I will enjoy the process of creating it. For example, I recently [Sep 2019] considered creating a series of videos mirroring the content of the LessWrong sequences, and found myself fawning over how nice it would be to have created such a series of videos, and not thinking at all about how I would go about creating them, let alone how much I would enjoy doing that.
I just learned a (rationalist) lesson. I’m taking a course that has some homework that’s hosted on a third party site. There was one assignment at the beginning of the semester, a few weeks ago. Then, about a week ago, I was wondering to myself whether there would be any more assignments any time soon. In fact, I even wondered if I had somehow missed a few assignments, since I’d thought they’d be assigned more frequently.
Well, I checked my course’s website (different from the site where the homework was hosted) and didn’t see any mention of assignments. Then I went to the professor’s website, and saw that they said they didn’t assign any “formal homework”. Finally, I thought back to the in-class discussions, where the third-party homework was never mentioned.
“Ah, good,” I thought. “I guess I haven’t missed any assignments, and none are coming up any time soon either.”
Then, today, the third-party homework was actually mentioned in class, so just now I went to look at the third-party website. I have missed three assignments, and there is another one due on Sunday.
I am not judged by the quality of my reasoning. I am judged by what actually happens, as are we all.
In retrospect (read: “beware that hindsight bias might be responsible for this paragraph”) I kind of feel like I wasn’t putting my all into figuring out if I was missing any assignments, and was instead just nervously trying to convince myself that I wasn’t. Obviously, I would rather have had that unpleasant experience earlier and missed fewer assignments—aka, if I was missing assignments, then I should have wanted to believe that I was missing assignments.
Yup. And they key thing that I’m reminding myself of is that this can’t be achieved by convincing myself that there aren’t any assignments to miss. It can only be achieved for sure by knowing whether there are assignments or not.
That is: try things that are unlikely to work but cheap to try.
I’m tall, and when I try to wash dishes at a sink made for normal people, I usually end up bending my back in a painful way without thinking. A couple weeks ago I noticed that the faucet can be unsheathed from a thing that holds it in place, and then it springs up a little and hangs higher, and makes it more comfortable to use for me. Win!
It came with the problem that it became hard to swing the faucet over to the side; the sheath that holds it down was also supposed to control its angle. Every time I swung it to the side, it would swing most of the way back. I had an idea-without-a-model for how to maybe fix it: just try swinging the sheath arm to the side I wanted to swing the faucet itself to, even though it wasn’t sheathed in it. Lo and behold, that worked!
Swinging the sheath arm over to that side also puts it in the way of the soap bottle, which I have to grab to re-soap the sponge a few times per washing session. I tried re-arranging things on that side of the sink to fix the problem, and couldn’t make it work. Oh well.
Final outcome: Washing dishes is now 50% more comfortable, for the cost of three pot-shots.
I’d happily have put 2 hours of work into this outcome if I’d had an assured plan to achieve it, so that’s a big win. Even if we allow for 30 other pot shots towards other goals that I’ve forgotten about because they didn’t go anywhere, I still got a big discount compared to what I’d have been willing to pay.
Reading about Dath Ilan[1] makes me homesick for a place that doesn’t exist. I want to live in a society like that. And I feel like, on the real Earth, I would be pretty fulfilled in a career where I was helping to bring it about, even if we only made Earth-realistic progress.
I used to think that I was born in a wrong century, that I would more fit in some kind of future world.
(That was before I heard of an AI risk, and anti-utopias in general.)
Now it seems like a wrong planet, or possibly a wrong Everett branch.
(Wouldn’t it be quite sad if a paperclip-maximizer wave traveling at the speed of light started on Earth and eventually reached Dath Ilan on the opposite side of the universe?)
I feel very lonely in life, and I think the biggest reason is that I can’t relate to most people. They’re too dumb. Trying to make friends with a rando at a bar feels like trying to make friends with a toddler.
Has anyone else had this experience? Is this an attitude problem on my part? Is there somewhere I can find smart friends? If it’s an attitude problem, do you have any wise words about how I can see that, so I can start fixing it?
My few smart friends seem to themselves have lots of smart friends, but when I complain that I have difficulty finding friends, their faces don’t light up in recognition or sympathy. They just look at me blankly and spit out generic platitudes about how maybe I should try putting myself out there. IDK if that’s all they had to do to make smart friends, but I do put myself out there, and it doesn’t work. I mostly only meet idiots, and even the idiots usually don’t care to get to know me.
It’s hard to say without having met you—could be both. (Or a combination of both.)
Can you role-play being social with someone who seems dumb to you? To the degree that the person would actually compliment you for being a pleasant acquaintance? If no, your social skills can still be improved a lot.
When you get that skill, the next step is to hand out at places with lots of people, apply the skills, and once in a while you will meet someone actually smart. (There are not many of them, though.)
I feel comfortable and safe asking a question like “Is this an attitude problem on my part?” on this website in a way I don’t when talking to most people. It’s humility. It’s me saying: “I might be really wrong, and you all might know it”. I don’t usually say that kind of thing, and it’s because for most people I might be talking to, it’s really not true. You wouldn’t talk like that to a toddler; I wouldn’t talk like that to a normie. But I would talk like that here, because I think people here are smart and kind in a way normies aren’t. Those are the same qualities I want in a friend, and I wonder if the humility I feel posting stuff on this site is a taste of what I’d feel all the time if I lived in a world that was mostly full of people I see as Actual Adults. Instead, in the real world, which is full of people I see as Basically Toddlers, I feel deeply and broadly arrogant, and although I think that feeling is justified and I wouldn’t exactly want to change it, it really isn’t fun :(
I think this means your “smart” friends aren’t as smart as they seem. You know what, I just stopped trying and stopped going to the wrong places. Smart people don’t hang out randomly in bars because obviously most people there are dumdums. I use that time to code and get smarter. Don’t listen to average people trying to gaslight you about “social skills” if they were smart they’d know how to talk to you. Be yourself and say what you want. Average people may get triggered because they will feel like dumdums after hearing your socratic seminar and that’s your filter, that’s how you know they’re a dummy. You can try to enlighten the masses until you tire of wasting your energy playing piano for cows.
Why isn’t If Anyone Builds It, Everyone Dies available for free online? Like as a webpage?
I read it, and I take it very seriously, and I’m actively putting in work to understand it better because of how seriously I take it, but it sticks out as odd to me that we’d paywall and paper-wall such an important message.
Maybe it’s because of that thing Eliezer once wrote where he said people might be more interested in science if it was all paywalled and other-stuff-walled and shrouded in cloaks and mystery instead of freely available in cheap textbooks and free blog posts?
I’m guessing that the publisher doesn’t want it freely available, and the authors expect the prestige from a conventional publisher to bring in more readers (or more high-status readers) overall.
Smart, I hadn’t thought of that. A quick Google search says maybe publishers let you publish online after the print copy is out, but not before? Could there be an online copy of IABIED in the future? (Or maybe I’m wrong, I know so little about publishing that I can’t even sanity-check what I think Google is telling me.)
I’ve been thinking of signing up for cryonics recently. The main hurdle is that it seems like it’ll be kind of complicated, since at the moment I’m still on my parent’s insurance, and I don’t really know how all this stuff works. I’ve been worrying that the ugh field surrounding the task might end up being my cause of death by causing me to look on cryonics less favorably just because I subconsciously want to avoid even thinking about what a hassle it will be.
But then I realized that I can get around the problem by pre-committing to sign up for cryonics no matter what, then just cancelling it if I decide I don’t want it.
It will be MUCH easier to make an unbiased decision if choosing cryonics means doing nothing rather than meaning that I have to go do a bunch of complicated paperwork now. It will be well worth a few months (or even years) of dues.
Are there “receipts” I can look at to sanity-check that GiveWell does make real grants and that the orgs that receive them do perform real work?
I’d be okay with receipts that take, say, 30 hours of self-education about tax law and 10 hours of combing through spreadsheets to understand, but a receipt a layperson can understand and trust in 5 minutes would be great too!
I don’t particularly doubt anybody, and have given 5-figure amounts to GiveWell in the past; it’s just that it recently occurred to me that if I’m going to keep doing that then I should probably at least nod in the direction of giving the whole thing pat-down.
I’m a conlang enthusiast, and specifically I study loglangs, which are a branch of conlangs that are based around predicate logic. My motivation for learning these languages was that I was always bothered by all the strange irregularities in my natural language (like the simple past tense being the same as the past participle, and the word inflammable meaning two opposite things).
Learning languages like these has only drawn my attention to even more natural-language nonsense. Occasionally I explain this to conlang lay-people, and maybe 50% of them are surprised to find that English is irregular. Some of them even deny that it is, and state that it all follows a perfectly normal pattern. This is a perpetual annoyance to me, simply because I spend so much time immersed in this stuff that I’ve forgotten how hard it is to spot from scratch.
Well, a while ago I wanted to start learning Mandarin from a friend of mine who speaks it as their first language. While introducing the language, they said that things like tenses were expressed as separate words (“did eat”) rather than sometimes-irregular modifications of existing words (“ate”). This reminded me of loglangs, so I gave them the spiel that I gave in the two previous paragraphs—natlangs, irregularities, annoyances, etc.
“Huh,” said the friend. They then turned to another native Chinese speaker and asked “Does Chinese have anything like that?”
I said, “I guarantee it does.”
This was months ago. Just now I was reflecting on it, and I realized that I have almost no evidence whatsoever that Chinese isn’t perfectly regular (or close enough that the thrust of my claim would be wrong).
It’s clear to me now that my thought process was something like “Well, just yet another conlang outsider who’s stunned and amazed to find that natural languages have problems.” That brought to mind all the other times when I’d encountered people surprised to find that their mother tongue (almost always English) had irregularities, and the erroneous conclusion precipitated right out.
You may also be integrating something you’ve read and then forgotten you read, and this added weight to your visible-and-suspect though process in order to make a true statement. It would not surprise me to learn that at least some of your study has included examples of irregularity from MANY natural languages, including Chinese. So “I guarantee it does” may be coming from multiple places in your knowledge.
So, was it actually incorrect, or just illegibly-justified?
Hmm, good question. I guess I wouldn’t be surprised to learn that I’d read about Chinese having irregularities, though the main text I’ve read about this (The Complete Lojban Language) didn’t mention any IIRC.
I wouldn’t be surprised if Chinese had no irregularities in the tense system – it’s a very isolating language. But here’s one irregularity: the negation of 有 is 没有 (“to not have/possess”), but the simple negation of every other verb is 不 + verb. You can negate other verbs with 没, but then it’s implied to be 没有 + verb, which makes the verb into something like a present participle. E.g., 没吃 = “to have not eaten”.
I think part of the intended interpretation of Dath Ilan is: “it would be easier to run an adequate society if everyone knew some basic economics”.
I know almost no economics. Even my understanding of supply & demand is shaky. Is there some bit of economics I can learn and start applying on Earth to see why this claim might be reasonable?
At the risk of saying the obvious, let’s start with the supply and demand.
Imagine a market with one unified product (this is the basic model, of course it gets more complicated later), thousands of producers and thousands of customers who can observe each other. The idea is that although the price is a result of mutual negotiation of all these producers and customers, each individual producer or customer feels helpless and treats it as a constant.
If you increase the price, more producers will be willing to sell (let’s assume that their alternative is doing something else outside of this specific market, each producers has different options, but these options are unrelated to the price at this specific market), and fewer customers will be willing to buy (again, assume they have alternatives at different markets). If you decrease the price, fewer producers will be willing to sell, and more customers will be willing to buy.
The idea is that although each individual producer or customer feels helpless (it seems like their only option is exit), together they set the price; specifically the price will stabilize at the level where the total selling capacity of the willing producers is equal to the total buying capacity of the willing customers.
(Famous exceptions to this rule are land and labor, because the land is not really produced, only a constant amount exists; and if you pay people more, their free time becomes more enjoyable. If a producer or a consumer has a monopoly or a cartel, they can dictate the price. Producers try to change the situation by creating brands. Etc.)
Many people already fail to understand this basic point. For example, what happens if you make increasing prices illegal during the shortage? Fewer producers will have an incentive to produce the product. Yet, most people will approve of the ban, and then complain about the shortage. (Because their intuition is that shortage is bad because it makes things expensive, therefore making expensive things illegal will make the shortage go away.)
The employers who complain about the shortage of qualified employees are making the same mistake. The reason they can’t get anyone is that their salary offers are not competitive. Insisting on keeping the salaries at yesterday’s level no matter what is not going to make the “shortage” go away.
Speaking of the job market, as an employee you should want there to be as many employers as possible, because then the salary (the price of your work and talent) will stabilize at a high level. Yet, many socialist leaning people want to make entrepreneurship more difficult, and then they complain about not being treated well. (Intuitively, they see hurting the employers as a collective revenge, not realizing that much better—and better targeted—revenge would be the freedom to walk away without consequences, which you only get when the number of free positions exceeds the number of job seekers.)
Taxes make things worse for both the producers and the customers. Some people believe that there is a huge difference between whether the tax is paid by the producer, or by the customer, but those are just two ways to describe the same reality: the customer pays C, the producer takes home P, the state takes X = C—P. You can describe it as “the price is C, then the producer pays the tax X, and takes home P”, or you can describe it as “the price is P, but the customer also has to pay the tax X, making their total expense C”, those are just different words describing the same outcome.
...uh, it seems like actually quite a lot would change if even this simplest thing became common knowledge.
I love economics (I’m an engineer and accidentally stumbled into it and now am PhD econ since 15 y or so and lecturer) and much of me believes DI quote exactly hits the nail. Sometimes, say 20% of me*, thinks that even reasonably advanced economics doesn’t help not so smart people really be smart. Still, I think the statement survives.
PM me if you’re interested in me sharing some materials personally with you! I’m also happy to have brief chats on things if useful.
Smartness + econ really is king.
One thing I can say upfront: I’m convinced on in LW or EA circles I have the impression to often find smarter quasi-economics analysis than among economists opinions/statements, so it’s all a bit a funny world with econ (a rant e.g. here).
*My exact literal and ungrammatical statement makes little sense but I reckon you see how I mean it.
I like the Quick Take format. I find there’s lots of stuff I’m willing to share in a low-commitment format that feel I wouldn’t share if it had to be a whole post. (Usually because I’m not confident enough I endorse it to want to make it a whole post, or because it’s too short.)
I might share even more if there was a convenient way to post anonymously. Obviously I’m already using a pseudonym, but I mean if there was a way to make my Quick Take just show up without an author to other people.
I go to a weekly rationality meetup at a bar in my area, which usually has 5-10 attendees. My unfortunate impression of half of the regulars is that they are depressed, pedantic, and bad at socializing, and that makes me sad because that feels like them being Spock instead of being Awesome.
What are other people’s experiences with this? I imagine it’s kind of hit-or-miss, with some groups being great and others being duds?
My experience is similar, about half people are impressive, and the other half is… something I associate with Mensa.
If the group is large enough, it may be possible to create subgroups for specific projects. But if you have fewer than ten rationalists coming to the local meetup, this is not an option. Unfortunately, I guess most local groups are small.
I suspect that asperger syndrome is correlated with aspiring to rationality; things like disagreeing with the social consensus. I am not saying that it is necessary per se—in a more sane society, the tension between social agreeableness and aiming for truth would be smaller, and even in the current society some people seem to manage it fine somehow. Also, too much nitpicking just keeps you maps blank instead of more precise. I am an asperger myself, but I try to see my weaknesses as weaknesses rather than rationalize them as superpowers.
I guess, ultimately different people expect different things from the meetups; it might be good to communicate this explicitly. But you need some minimum size of the meetup to find people who share your current goals.
First you must realize that you don’t understand the world. This means you cannot be perfectly well adjusted, those people understand their world very well, even if we would find flaws in their worldview. They are comfortable.
This also means you are probably having a rough time of it. That dosnt lend itself to perfect mental stability.
Then you have to seek real understanding, to cut reality at its joints, and not get stuck on partial answers. This implys a level of pedantry, probably also some education or exposure to science culture.
Does it still surprise you that the people you met are the membership?
BTW, I’m absolutely all of the above, so no judgment. You’ve got a choice, pleasent and easy or interesting, take your pick.
I didn’t necessarily find it surprising, just saddening. But also I disagree with your conclusion: some people at the meetup were perfectly pleasant to talk to, so empirically, there can’t really be that strong of a guarantee that Rationalism implies Spockishness.
I think I spot some of the mistakes in your reasoning: for one, I don’t think pedantry is the same as real understanding. When I said “pedantic”, I meant something like “too nitpicky about words”. Even if your goal is to achieve real understanding, it doesn’t serve you well to be too nitpicky; it would be better to be the right amount of nitpicky. I would also describe myself as depressed and poorly-adjusted, but I think I’m good at socializing. I ended up making the whole group laugh a few times one of the recent times I went.
(I’ve also been guilty of all the sins myself at one point or another, so I don’t mean to put myself above it all either, but I think I do them a lot less nowadays than the people I’m complaining about, so I think it’s fair to use myself as an illustration of the fact that it’s doable.)
I was making a tendency argument rather than a 100% of everyone sort of thing. I mean you’ve noticed it, I’ve noticed it and both of us would classify ourselves as hitting some of those notes. You even (not wrongly) were pedantic about my definition of pedantry. Gotta be a sign
Also as a side note, I’m curious what’s actually in the paywalled posts. Surely people didn’t write a bunch of really high-quality content just for an April Fools’ day joke?
I would appreciate an option to hide the number of votes that posts have. Maybe not hide entirely, but set them to only display at the bottom of a post, and not at the top nor on the front page. With the way votes are currently displayed, I think I’m getting biased for/against certain posts before I even read them, just based on the number of votes they have.
Yeah, this was originally known as “Anti-Kibitzer” on the old LessWrong. It isn’t something we prioritized, but I think greaterwrong has an implementation of it. Though it would also be pretty easy to create a stylish script for it (this hides it on the frontpage, and makes the color white on the post-page, requiring you to select the text to see the score):
The other day, my roommate mentioned that the bias towards wanting good things for people in your in-group and bad things for those in your out-group can be addressed by including ever more people in your in-group.
Here’s a way to do that: take a person you want to move into your in-group, and try to imagine them as the protagonist of a story. What are their desires? What obstacles are they facing right now? How are they trying to overcome them?
I sometimes feel annoyed at a person just by looking at them. I invented this technique just now, but I used it one time on a person pictured in an advertisement, and it worked. I had previously been having a “what’s your problem?” feeling, and it was instantly replaced with a loving “I’m rooting for you” feeling.
Accusing someone of being stubborn in the middle of an argument doesn’t make sense. If the argument is still going, you’re both being that stubborn.
Maybe you mean to accuse them of being unreasonable, of sticking to a statement that is clearly false. In that case, say that.
It can make sense to accuse someone of being stubborn in general, if you often yield to their requests and they rarely yield to yours.
Note: I guess “stubborn” can mean what I’ve used “unreasonable” to mean. I feel like I’ve been told I’m being too stubborn just for sticking to my guns for too long, when they haven’t even tried to really argue with me and so can’t say that I’m being unreasonable in light of good counterarguments. I wish I could swap “stubborn” for a word like “persistent”, but I can’t think of one that means the right thing and has the right ring. I’m posting anyway because Quick Take.
I’d recommend to defuse the word—if someone complains you’re stubborn, thank them for the compliment, that you are persistent in your use of logic in forming your beliefs.
Scream! This feels like an extremely rude thing to do… then again, I guess the person making the accusation is probably being pretty rude to begin with… maybe it works?
This varies a lot with what kind of discussion is going on, but many times where it’s OK to call someone stubborn, it’s ALSO ok to defend stubborn-ness as a positive trait.
Idea: “Ugh-field trades”, where people trade away their obligations that they’ve developed ugh-fields for in exchange for other people’s obligations. Both people get fresh non-ugh-fielded tasks. Works only in cases where the task can be done by somebody else, which won’t be every time but might be often enough for this to work.
Interesting thought. Unfortunately, most tasks where I’m blocked/delayed by an ugh field either dissolve it as soon as I identify it, or include as part of the ugh that only I can do it.
I’m currently working on a text document full of equations that use variables with extremely long names. I’m in the process of simplifying it by renaming the variables. For complicated reasons, I have to do this by hand.
Just now, I noticed that there’s a series of variables O1-O16, and another series of variables F17-F25. For technical reasons relating to the work I’m doing, I’m very confident that the name switch is arbitrary and that I can safely rename the F’s to O’s without changing the meaning of the equations.
But I’m doing this by hand. If I’m wrong, I will potentially was a lot of work by (1) making this change (2) making a bunch of other changes (3) realizing I was wrong (4) undoing all the other changes (5) undoing this change (6) re-doing all the changes that came after it.
And for a moment, this spurred me to become less confident about the arbitrariness of the naming convention!
The correct thought would have been “I’m quite confident about this, but seeing as the stakes are high if I’m wrong and I can always do this later, it’s still not worth it to make the changes now.”
The problem here was that I was conflating “X is very likely true” with “I must do the thing I would do if X was certain”. I knew instinctively that making the changes now was a bad idea, and then I incorrectly reasoned that it was because it was likely to go wrong. It’s actually unlikely to go wrong, it’s just that if it does go wrong, it’s a huge inconvenience.
Epistemic status: really shaky, but I think there’s something here.
I naturally feel a lot of resistance to the way culture/norm differences are characterized in posts like Ask and Guess and Wait vs Interrupt Culture. I naturally want to give them little pet names, like:
Guess culture = “read my fucking mind, you badwrong idiot” culture.
Ask culture = nothing, because this is just how normal, non-insane people act.
I think this feeling is generated by various negative experiences I’ve had with people around me, who, no matter where I am, always seem to share between them one culture or another that I don’t really understand the rules of. This leads to a lot of interactions where I’m being told by everyone around me that I’m being a jerk, even when I can “clearly see” that their is nothing I could have done that would have been correct in their eyes, or that what they wanted me to do was impossible or unreasonable.
But I’m starting to wonder if I need to let go of this. When I feel someone is treating me unfairly, it could just be because (1) they are speaking in Culture 1, then (2) I am listening in Culture 2 and hearing something they don’t mean to transmit. If I was more tuned in to what people meant to say, my perception of people who use other norms might change.
I feel there’s at least one more important pair of cultures, and although I haven’t mentioned it yet, it’s the one I had in mind most while writing this post. Something like:
Culture 1: Everyone speaks for themselves only, unless explicitly stated otherwise. Putting words in someone’s mouth or saying that they are “implying” something they didn’t literally say is completely unacceptable. False accusations are taken seriously and reflect poorly on the accuser.
Culture 2: The things you say reflect not only on you but also on people “associated” with you. If X is what you believe, you might have to say Y instead if saying X could be taken the wrong way. If someone is being a jerk, you don’t have to extend the courtesy of articulating their mistake to them correctly; you can just shun them off in whatever way is easiest.
I don’t really know how real this dichotomy is, and if it is real, I don’t know for sure how I feel about one being “right” and the other being “wrong”. I tried semi-hard to give a neutral take on the distinction, but I don’t think I succeeded. Can people reading this tell which culture I naturally feel opposed to? Do you think I’ve correctly put my finger on another real dichotomy? Which set of norms, if either, do you feel more in tune with?
I think this feeling is generated by various negative experiences I’ve had with people around me, who, no matter where I am, always seem to share between them one culture or another that I don’t really understand the rules of. This leads to a lot of interactions where I’m being told by everyone around me that I’m being a jerk, even when I can “clearly see” that their is nothing I could have done that would have been correct in their eyes, or that what they wanted me to do was impossible or unreasonable.
Is it because they’re expecting you to read their mind, and go along with their “culture”, instead of asking you?
it (the negative experiences) - Are *they (the negative experiences) the result of (people with a “culture” who’s rules rules you don’t understand) expecting you to read *their mind, and go along with their “culture”, instead of asking you to go along with their culture?
Aha, no, the mind reading part is just one of several cultures I’m mentioning. (Guess Culture, to be exact.) If I default to being an Asker but somebody else is a Guesser, I might have the following interaction with them:
Me: [looking at some cookies they just made] These look delicious! Would it be all right if I ate one?
Them: [obviously uncomfortable] Uhm… uh… I mean, I guess so...
Here, it’s retroactively clear that, in their eyes, I’ve overstepped a boundary just by asking. But I usually can’t tell in advance what things I’m allowed to ask and what things I’m not allowed to ask. There could be some rule that I just haven’t discovered yet, but because I haven’t discovered it yet, it feels to me like each case is arbitrary, and thus it feels like I’m being required to read people’s minds each time. Hence why I’m tempted to call Guess Culture as “Read-my-mind Culture”.
(Contrast this to Ask Culture, where the rule is, to me, very simple and easy to discover: every request is acceptable to make, and if the other person doesn’t want you to do what you’re asking to do, they just say “no”.)
It might be hard to take a normative stance, but if culture 1 makes you feel better AND leads to better results AND helps people individuate and makes adults out of them, then maybe it’s just, y’know, better. Not “better” in the naive mistake-theorist assumption that there is such a thing as a moral truth, but “better” in the correct conflict-theorist assumption that it just suits you and me and we will exert our power to make it more widely adopted, for the sake of us and our enlightened ideals.
The size and complexity of the palette of reactions that comes up when I want to react to part of a post is overwhelming to me, enough that it forms a significant barrier of friction against using it.
When somebody is advocating taking an action, I think it can be productive to ask “Is there a good reason to do that?” rather than “Why should we do that?” because the former phrasing explicitly allows for the possibility that there is no good reason, which I think makes it both intellectually easier to realize that and socially easier to say it.
I just noticed that I’ve got two similarity clusters in my mind that keep getting called to my attention by wording dichotomies like high-priority and low-priority, but that would themselves be better labeled as big and small. This was causing me to interpret phrases like “doing a string of low-priority tasks” as having a positive affect (!) because what it called to mind was my own activity of doing a string of small, on-average medium-priority tasks.
My thought process might improve overall if I toss out the “big” and “small” similarity clusters and replace them with clusters that really are centered around “high-priority” and “low-priority”.
It’s happened again: I’ve realized that one of my old beliefs (pre-LW) is just plain dumb.
I used to look around at all the various diet (Paleo, Keto, low carb, low fat, etc.) and feel angry at people for having such low epistemic standards. Like, there’s a new theory of nutrition every two years, and people still put faith in them every time? Everybody swears by a different diet and this is common knowledge, but people still swear by diets? And the reasoning is that “fat” (the nutrient) has the same name as “fat” (the body part people are trying to get rid of)?
Then I encountered the “calories in = calories out” theory, which says that the only thing you need to do to lose weight is to make sure that you burn more calories than you eat.
And I thought to myself, “yeah, obviously.”.
Because, you see, if the orthodox asserts X and the heterodox asserts Y, and the orthodox is dumb, then Y must be true!
Anyway, I hadn’t thought about this belief in a while, but I randomly remembered it a few minutes ago, and as soon as I remembered its origins, I chucked it out the window.
Oops!
(PS: I wouldn’t be flabbergasted if the belief turned out true anyway. But I’ve reverted my map from the “I know how the world is” state to the “I’m awaiting additional evidence” state.)
On one hand, yeah, obviously.
On the other hand, “burning calories” is not an elementary action. Suppose I tell you to burn 500 calories now; how exactly would you achieve it? If your answer is that you would exercise or do something physically demanding, such actions spend ATP in the cells, so what if you don’t have enough ATP in your muscle cells; what is your plan B for burning calories? From the opposite side, you can limit the intake of calories. What if your metabolism is such that if you don’t provide enough calories, you will gradually fall in coma and die.
Your metabolism can make it impossible for you to reduce the “calories in” or increase the “calories out”, if it is somehow set up in a way that does not convert the calories into useful energy in your muscle cells, and it starts damaging your organs if the calories are missing in general. So while the theory is almost tautologically true, it may still be impossible to use it to lose weight. And the problem is that the proponents of “calories in = calories out” usually smugly pretend that it is an actionable advice, instead of mere description.
The actionable advice needs to be about how the metabolism works. And the things that impact it, such as what you eat, and who knows what else. Also, if you have some hormonal imbalance, or alergy, or whatever, your metabolism may differ from other people even if you eat the same things and try to live the same lifestyle. So, while e.g. eating less refined sugar would probably help everyone, no advice would guarantee a perfect outcome for everyone.
You make a good point—even if my belief was technically true, it could still have been poorly framed and inactionable (is there a name for this failure mode?).
But in fact, I think it’s not even obvious that it was technically true. If we say “calories in” is the sum of the calorie counts on the labels of each food item you eat (let’s assume the labels are accurate) then could there not still be some nutrient X that needs to be present for your body to extract the calories? Say, you need at least an ounce of X to process 100 calories? If so, then one could eat the same amount of food, but less X, and potentially lose weight.
Or perhaps the human body can only process food between four and eight hours after eating it, and it doesn’t try as hard to extract calories if you aren’t being active, so scheduling your meals to take place four hours before you sit around doing nothing would make them “count less”.
Calories are (presumably?) a measure of chemical potential energy, but remember that matter itself can also be converted into energy. There’s no antimatter engine inside my gut, so my body fails to extract all of the energy present in each piece of food. Couldn’t the mechanism of digestion also fail to extract all the chemical potential energy of species “calorie”?
Yes, there are many steps in the metabolism that are not under your conscious control. I am not an expert, so I don’t want to speculate too much about the technical details, but I think that gut bacteria probably also play a role. Simply, not everything you put in your mouth ends up necessarily in your bloodstream, and not everything that you absorbed is necessarily available in form of muscle energy.
I don’t know any standard name. Seems to me the problem is confusing “rephrasing of the desired outcome” with “an algorithm that actually get you there”. Something like:
Q: How can I get lot of money?
A: Become a millionaire!
Like, yeah, technically, everyone who successfully followed this advice ended up with lots of money, and everyone who didn’t can be accused of not following the advice properly, but that’s simply because those are two ways to describe the same thing.
Q: How can I lose weight?
A: Get rid of the extra atoms! I mean, extra calories!
Charitably, the advice is not absolutely bad, because for a hypothetical completely clueless listener it would provide some little information. But then, using this advice in practice means insinuating that your target is completely clueless, which is probably not be the case.
But atoms aren’t similar to calories, are they? I maintain that this hypothesis could be literally false, rather than simply unhelpful.
Okay, it’s not the same. But the idea is that the answer is equally unhelpful, for similar reasons.
I want to give a big thumbs up of positive reinforcement. I thinks its great that I got to read an “oops! That was dumb, but now I’ve changed my mind.”
Thanks for helping to normalize this.
Thanks for the feedback! Here’s another one for ya. A relatively long time ago I used to be pretty concerned about Pascal’s wager, but then I devised some clever reasoning why it all cancels out and I don’t need to think about it. I reasoned that one of three things must be true:
I don’t have an immortal soul. In this case, I might as well be a good person.
I have an immortal soul, and after my bodily death I will be assigned to one of a handful of infinite fates, depending on how good of a person I was. In this case it’s very important that I be a good person.
Same as above, but the decision process is something else. In this case I have no way of knowing how my infinite fate will be decided, so I might as well be a good person during my mortal life and hope for the best.
But then, post-LW, I realized that there are two issues with this:
It doesn’t make any sense to separate out case 2 from the enormous ocean of possibilities allowed for by case 3. Or rather, I can separate it, but then I need to probabilistically penalize it relative to case 3, and I also need to slightly shift the “expected judgment criterion” found in case 3 away from “being a good person is the way to get a good infinite fate”, and it all balances out.
More importantly, this argument flippantly supposes that I have no way of discerning what process, if any, will be used to assign me an infinite fate. An infinite fate, mind you. I ought to be putting in more thought than this even if I thought the afterlife only lasted an hour, let alone eternity.
So now I am back to being rather concerned about Pascal’s wager, or more generally, the possibility that I have an immortal soul and need to worry about where it eventually ends up.
From my first read-through of the sequences I remember that it claims to show that the idea of there being a god is somewhat nonsensical, but I didn’t quite catch it the first time around. So my first line of attack is to read through the sequences again, more carefully this time, and see if they really do give a valid reason to believe that.
I think the first step is to ask yourself what do you even mean by saying “god”.
Because if you have a definition like “the spirit of our dead chieftain, who sometimes visits me in my dreams”, I have no problem with that. Like, I find it completely plausible that you see the image of the chieftain in your dreams; nothing unscientific about that. It’s just that the image is generated by your brain, so if try to communicate with it, it can only give you advice that your brain generated, and it can only grant your wishes in the sense that you accomplish that yourself. But if you agree that this is exactly what you meant, then such god is perfectly okay for me.
But the modern definition is more like: an intelligent being that exists outside of our universe, but can observe it and change it. Suspending disbelief for a moment; how exactly can a being be “intelligent”, whether inside our universe or not? Intelligence implies processing data. That would imply that the god has some… supernatural neurons, or some other architecture capable of processing data. So the god is a mechanism (in the sense that a human is a biological mechanism) composed of parts, although those parts do not necessarily have to be found in our physics. Skill kinda plausible, maybe gods are made out of dark matter, who knows. But then, how did this improbably complicated mechanism come into existence? Humans were made by evolution, were gods too? But then again those gods are not the gods of religion; they are merely powerful aliens. But powerful aliens are neither creators of the universe, nor are they omniscient.
Then the theologists try to come with some smart-sounding arguments, like “god is actually supremely simple”—therefore Occam’s razor does not disprove him, and no evolution was needed, because simple things are more likely a priori, so a supreme being is supremely likely. Or something like that. But this is nonsense, because “supremely simple” and “capable of processing information” are incompatible.
The last argument would be: okay, we do not have a coherent explanation of god, but you don’t have a coherent explanation how the universe can exist without one. And a potential answer is that maybe existence is relative. Like, if you run a simulation of Conway’s Game of Life, it could possibly contain evolved intelligent life, and that life would ask “how was our universe created”? And it’s not like you created the universe by simulating it, because you are merely following the mathematical rules; so it’s more like the math created that universe and you are only observing it. If the beings in that mathematical universe will pray to gods, there is no way for anyone outside to intervene (while simultaneously following the mathematical rules). So the universe inside the Game of Life is a perfectly godless universe, based on math. But does it exist? Well, that rules of math describe the intelligent beings inside that universe that live, think, interact with their world. Maybe that’s all that is needed for existence; or rather, the question is what else could be needed? And then, maybe our world is also just an implication of some mathematical equations.
tl;dr—maybe math itself implies existence, and if you try to turn the concept of (intelligent, preceiving, acting) god into something coherent it is no longer the god of religion but merely a powerful alien
The supernatural isn’t supposed to be the natural done all over again. The typical theological claim is that God’s wisdom or whatever is an intrinisic quality, not something with moving parts.
Well, “wisdom as an intrinsic quality” is a mysterious answer. And what is “wisdom without moving parts”? A precomputed table of wise answers to all possible questions? Who precomputed it and how?
I agree that this is how theology usually answers it, but it is an answer that doesn’t make any sense when you look at it closer; it’s just some good-sounding words randomly glued together. And if you try to make it refer to something, even a hypothetical something, the whole explanation falls apart.
Reductionism is a combination of three claims.
That many thing are made of smaller components
That those particular can be understood in terms of the operations of their components
There’s an irreduciby basic level. Its not turtles all way down.
If it’s always the case that something that isn’t explicable in terms of its parts is mysterious, then the lowest level us mysterious. If nothing is mysterious, if you apply the argument against mysterious answer without prejudice, reductionism is false. There isn’t a consistent set of principles here.
Continued ..
Naturalism is the claim that there is a bunch of fundamental properties that just are, at the bottom of the stack ,and everything is built up from that. Supernaturalism is the claim that the intrinsic stuff is at the top of the stack, and everything else is derived from it top-down. That may be 100% false , but it is the actual claim.
There’s a thing called the principle of charity , where you one party interprets the others statements so as to maximise their truth value. This only enhances communication if the truth is not basically in dispute...that’s the easy case. The hard case is when there is a basic dispute about whats true. In that case, it’s not helpful to fix the other person’s claims by making them more reasonable from your point of view.
Anyway, thats how we ended up with “God must have superneurons in his superbrain”.
Feels like in the top-down universe, science shouldn’t work at all. I mean, when you take a magnifying glass and look at the details, they are supposedly generated on the fly to fit the larger picture. Then you apply munchkinry to the details and invent an atomic bomb or quantum computer… which means… what exactly, from the top-down perspective?
Yeah, you can find an excuse, e.g. that some of those top-down principles are hidden like Easter eggs, waiting to be discovered later. That the Platonic idea of smartphones has been waiting for us since the creation of the universe, but was only revealed to the recent generation. Which would mean that the top-down universe has some reason to pretend to be bottom-up, at least in some aspects...
Okay, the same argument could be made that quantum physics pretends to be classical physics at larger scale, or relativity pretends to be Newtonian mechanics at low speeds… as if the scientists are trying to make up silly excuses for why their latest magic works but totally “doesn’t contradict” what the previous generations of scientists were telling us...
Well, at least it seems like the bottom-up approach is fruitful, whether the true reason is that the universe is bottom-up, or that the universe it top-down in a way that tries really hard to pretend that it is actually bottom-up (either in the sense that when it generates the—inherently meaningless—details for us, it makes sure that all consequences of those details are compatible with the preexisting Platonic ideas that govern the universe… or like a Dungeon Master who allows the players to invent all kinds of crazy stuff and throw the entire game off balance, because he values consistency above everything).
More importantly, in universe where there is magic all the way up, what sense does it make to adopt the essentially half-assed approach, where you believe in the supernatural but also kinda use logic except not too seriously… might as well throw the logic away completely, because in that kind of universe it is not going to do you much good anyway.
The basic claim of a top down universe is a short string that doesn’t contain much information. About the same amount as a basic claim of reductionism.
The top down claim doesnt imply a universe of immutable physical law, but it doesn’t contradict it either.
The same goes for the bottom-up claim. A universe of randomly moving high entropy gas is useless for science and technology, but compatible with reductionism.
But all this is rather beside the point. Even if supernaturalism is indefensible, you can’t refute it by changing it into something else.
I wouldn’t call the dead chieftain a god—that would just be a word game.
Wait wait! You say a god-like being created by evolution cannot be a creator of the universe. But that’s only true if you constrain that particular instance of evolution to have occered in *this* universe. Maybe this universe is a simulation designed by a powerful “alien” in another universe, who itself came about from an evolutionary process in its own universe.
It might be “omniscient” in the sense that it can think 1000x as fast as us and has 1000x as much working memory and is familiar with thinking habits that are 1000x as good as ours, but that’s a moot point. The real thing I’m worried about isn’t whether there exists an omniscient-omnipotent-benevolent creature, but rather whether there exists *some* very powerful creature who I might need to understand to avoid getting horrible outcomes.
I haven’t yet put much thought into this, since I only recently came to believe that this topic merits serious thought, but the existence of such a powerful creature seems like a plausible avenue to the conclusion “I have an infinite fate and it depends on me doing/avoiding X”.
This is another area where my understanding could stand to be improved (and where I expect it will be during my next read-through of the sequences). I’m not sure exactly what kind of simplicity Occam’s razor uses. Apparently it can be formalized as Kolmogorov complexity, but the only definition I’ve ever found for that term is “the Kolmogorov Complexity of X is the length of the shortest computer program that would output X”. But this definition is itself in need of formalization. Which programming language? And what if X is something other than a stream of bits, such as a dandelion? And even once that’s answered, I’m not quite sure how to arrive at the conclusion that Kolmogorov-ly simpler things are more likely to be encountered.
(All that being said, I’d like to note that I’m keeping in mind that just because I don’t understand these things doesn’t mean there’s nothing to them. Do you know of any good learning resources for someone who has my confusions about these topics?)
That much makes sense, but I think it excludes a possibly important class of universe that is based on math but also depends on a constant stream of data from an outside source. Imagine a Life-like simulation ruleset where the state of the array of cells at time T+1 depended on (1) the state of the array at time T and (2) the on/off state of a light switch in my attic at time T. I could listen to the prayers of the simulated creatures and use the light switch to influence their universe such that they are answered.
Some people in history did, specifically, ancient Romans. But now we don’t. Just making it obvious.
And this is similar to the chieftain thing. You can have a religion that defines “god” as “an alien from another universe”, but many religions insist that god is not created but eternal.
Yes, this is a practical approach. Well, you cannot disprove such thing, because it is logically possible. (Obviously, “possible” does not automatically imply “it happened”.) But unless you assume it is “simulations all the way up”, there must be a universe that is not created by an external alien lifeform. Therefore, it is also logically possible that our universe is like that.
There is no reason to assume that the existing religions have something in common with the simulating alien. When I play Civilization, the “people” in my simulation have a religion, but it doesn’t mean that I believe in it, or even approve it, or that I am somehow going to reward them for it.
It’s just a cosmic horror that you need to learn to live with. There are more.
Any programming language; for large enough values it doesn’t matter. If you believe that e.g. Python is much better in this regard than Java, then for sufficiently complicated things the most efficient way to implement them in Java is to implement a Python emulator (a constant-sized piece of code) and implementing the algorithm in Python. So if you chose a wrong language, you pay at most a constant-sized penalty. Which is usually irrelevant, because these things are usually applied to debating what happens in general when the data grow.
But I agree that if you talk about small data, it is underspecified. I don’t really know what could it mean to “have a universe defined by the following three bits: 0, 0, 1”, and maybe no one has a meaningful answer to this. But there are cases where you can have an intuition that for any reasonable definition of a programming language, X should be simpler than Y.
Just an intuition pump: Imagine that there is a multiverse A containing all universes that can be run by programs having exactly 100 lines of code; and there is a multiverse B containing all universes that can be run by programs having exactly 105 lines of code. The universes from A are more likely, because they also appear in B.
For each program A1 describing a universe from A, you have a set of programs describing exactly the same universe in B, simply by adding an “exit” instruction on line 101, and arbitrary instructions on lines 102-105. If we take a program B1 describing a universe from B in such way that each of the 105 lines is used meaningfully, then… in multiverse A we have one A1 vs zero B1, and in multiverse B we have many A1-equivalents vs one B1; in either case A1 wins.
A part of this is in standard computer science curriculum, and another part is a philosophical extrapolation. I do not have a recommendation about a specific textbook, I just vaguely remember things like this from university. (Actually, I probably never heard explicitly about Kolmogorov complexity at university, but I learned some related concepts that allowed me to recognize what it means and what it implies, when I found it on Less Wrong.)
Then I would say the Kolmogorov complexity of the “Life-only” universe is the complexity of the rules of Life; but the complexity of the “Life+inputs” universe is the complexity of the rules plus the complexity of whoever generates the inputs, and everything they depend on, so it is the complexity of the outside universe.
The Civ analogy makes sense, and I certainly wouldn’t stop at disproving all actually-practiced religions (though at the moment I don’t even feel equipped to do that).
Are you sure it’s logically possible in the strict sense? Maybe there’s some hidden line of reasoning we haven’t yet discovered that shows that this universe isn’t a simulation! (Of course, there’s a lot of question-untangling that has to happen first, like whether “is this a simulation?” is even an appropriate question to ask. See also: Greg Egan’s book Permutation City, a fascinating work of fiction that gives a unique take on what it means for a universe to be a simulation.)
This sounds like the kind of thing someone might say who is already relatively confident they won’t suffer eternal damnation. Imagine believing with probability at least 1/1000 that, if you act incorrectly during your life, then...
(WARNING: graphic imagery) …upon your bodily death, your consciousness will be embedded in an indestructible body and put in a 15K degree oven for 100 centuries. (END).
Would you still say it was just another cosmic horror you have to learn to live with? If you wouldn’t still say that, but you say it now because your probability estimate is less than 1/1000, how did you come to have that estimate?
The constant-sized penalty makes sense. But I don’t understand the claim that this concept is usually applied in the context of looking at how things grow. Occam’s razor is (apparently) formulated in terms of raw Kolmogorov complexity—the appropriate prior for an event X is 2^(-B), where B is the Kolmogorov Complexity of X.
Let’s say general relativity is being compared against Theory T, and the programming language is Python. Doesn’t it make a huge difference whether you’re allowed to “pip install general-relativity” before you begin?
I agree that these intuitions can exist, but if I’m going to use them, then I detest this process being called a formalization! If I’m allowed to invoke my sense of reasonableness to choose a good programming language to generate my priors, why don’t I instead just invoke my sense of reasonableness to choose good priors? Wisdom of the form “programming languages that generate priors that work tend to have characteristic X” can be transformed into wisdom of the form “priors that work tend to have characteristic X”.
I have to admit that I kind of bounced off of this. The universe-counting argument makes sense, but it doesn’t seem especially intuitive to me that the whole of reality should consist of one universe for each computer program of a set length written in a set language.
Can I ask which related concepts you mean?
Oh, that makes sense. In that case, the argument would be that nothing outside MY universe could intervene in the lives of the simulated Life-creatures, since they really just live in the same universe as me. But then my concern just transforms into “what if there’s a powerful entity living in this universe (rather than outside of it) who will punish me if I do X, etc”.
I find it difficult to imagine how such argument could even be constructed. “Our universe isn’t a simulation because it has property X” doesn’t explain why the simulator could not simulate X. The usual argument is “because quantum stuff, the simulation would require insane amounts of computing power”, which is true, but we have no idea what the simulating universe looks like, and what kind of physics is has… maybe what’s an insane amount for us is peanuts for them.
But maybe there is some argument why a computing power in principle (like, some mathematical reason) cannot exceed certain value, ever. And the value may turn out to be insufficient to simulate our universe. And we can somehow make sure that our entire universe is simulated in sufficient resolution (not something like: the Earth or perhaps the entire Solar system is simulated in full quantum physics, but everything else is just a credible approximation). Etc. Well, if such thing happens, then I would accept the argument.
Yeah, I just… stopped worrying about these kinds of things. (In my case, “these kinds of things” refer e.g. to very unlikely Everett branches, which I still consider more likely than gods.) You just can’t win this game. There are million possible horror scenarios, each of them extremely unlikely, but each of them extremely horrifying, so you would just spend all your life thinking about them; and there is often nothing you can do about them, in some cases you would be required to do contradictory things (you spend your entire life trying to appease the bloodthirsty Jehovah, but it turns out the true master of universe is the goddess Kali and she is very displeased with your Christianity...) or it could be some god you don’t even know because it is a god of Aztecs, or some future god that will only be revealed to humanity in year 3000. Maybe humans are just a precursor to an intelligent species that will exist million years in the future, and from their god’s perspective humans are even less relevant than monkeys are for Christianity. Maybe we are just meant to be food for the space locusts from Andromeda galaxy. Maybe our entire universe is a simulation on a computer in some alien universe with insane computer power, but they don’t care about the intelligent beings or life in general; they just use the flashing galaxies as a screen saver when they are bored. If you put things into perspective, assigning probability 1/1000 to any specific religion is way too much; all kinds of religions, existing and fictional put together, don’t deserve that much.
And by the way, torturing people forever, because they did not believe in your illogical incoherent statements unsupported by evidence, that is 100% compatible with being an omnipotent, omniscient, and omnibenevolent god, right? Yet another theological mystery...
On the other hand, if you assume an evil god, then… maybe the holy texts and promises of heaven are just a sadistic way he is toying with us, and then he will torture all of us forever regardless.
So… you can’t really win this game. Better to focus on things where you actually can gather evidence, and improve your actual outcomes in life.
Psychologically, if you can’t get rid of the idea of supernatural, maybe it would be better to believe in an actually good god. Someone who is at least as reasonable and good as any of us, which should not be an impossibly high standard. Such god certainly wouldn’t spend an eternity torturing random people for “crimes” such as being generally decent people but believing in a wrong religion or no religion at all, or having extramarital sex, etc. (Some theologians would say that this is actually their god. I don’t think so, but whatever.) I don’t really believe in such god either, honestly, but it is a good fallback plan when you can’t get rid of the idea of gods completely.
That would be cheating, obviously. Unless by the length of code you mean also the length of all used libraries, in which case it is okay. It is assumed that the original programming language does not favoritize any specific theory, just provides very basic capabilities for expressing things like “2+2” or “if A then B else B”. (Abstracting from details such as how many pluses are in one “if”.)
Yeah, ok. The meaning of all this is, how can we compare “complexity” of two things where neither is a strict subset of the other. The important part is that “fewer words” does not necessarily mean “smaller complexity”, because it allows obvious cheating (invent a new word that means exactly what you are arguing for, and then insist that your theory is really simple because it could be described by one word—the same trick as with importing a library), but even if it is not your intention to cheat, your theory can still benefit from some concepts having shorter words for historical reasons, or even because words related to humans (or life on Earth in general) are already shorter, so you should account for the complexity that is already included for them. Furthermore it should be obvious to you that “1000 different laws of physics” is more complex than “1 law of physics, applied in the same way to 1000 particles”. If this all is obvious to you, than yes, making the analogy to a programming language does not bring any extra value.
But historical evidence shows that humans are quite bad at this. They will insist that stars are just shining dots in the sky instead of distant solar systems, because “one solar system + thousands of shining dots” seems to them less complex than “thousands of solar systems”. They will insist that Milky Way or Andromeda cannot be composed of stars, because “thousand stars + one Milky Way + one Andromeda” seems to them less complex than “millions of stars”. More recently (and more controversially), they will insist that “quantum physics + collapse + classical physics” is less complex than “quantum physics all the way up”. Programming analogy helps to express this in a simple way: “Complexity is about the size of code, not how large values are stored in the variables.”
Compression (lossless), like in the “zip” files. Specifically the fact that an average random file cannot be compressed, no matter how smart is the method used. Like, for any given value N, the number of all files with size N is larger than the number of all files with size smaller than N. So, whatever is your compression method, if it is lossless, it needs to be a bijection, so there must be at least one file of size N that it will not compress into a file of smaller size. Even worse, for each file of size N compressed to a smaller size, there must be a file of size smaller than N that gets “compressed” to a file of size N or more.
So how does compression work in real world? (Because experience shows it does.) It ultimately exploits the fact that most of the files we want to compress are not random, so it is designed to compress non-random files into smaller ones, and random (containing only noise) files into slightly larger ones. Like, you can try it at home; generate a one megabyte file full of randomly generated bits, then try all compression programs you have installed, and see what happens.
Now, each specific compression algorithm recognized and exploits some kinds of regularity, and is blind towards others. This is why people sometimes invent better compression algorithms that exploit more regularities. The question is, what is the asymptote of this progress? If you tried to invent the best compression algorithm ever, one that could exploit literally all kinds of non-randomness, what would it look like?
(You may want to think about this for a moment, before reading more.)
The answer is that the hypothetical best compression algorithm ever would transform each file into the shortest possible program that generates this file. There is only one problem with that: finding the shortest possible program for every input is algorithmically impossible—this is related to halting problem. Regardless, if the file is compressed by this hypothetical ultimate compression algorithm, the size of the compressed file would be the Kolmogorov complexity of the original file.
Then we are no longer talking about gods in the modern sense, but about powerful aliens.
I see. In that case, I think we’re reacting differently to our situations due to being in different epistemic states. The uncertainty involved in Everett branches is much less Knightian—you can often say things like “if I drive to the supermarket today, then approximately 0.001% of my future Everett branches will die in a car crash, and I’ll just eat that cost; I need groceries!”. My state of uncertainty is that I’ve barely put five minutes of thought into the question “I wonder if there are any tremendously important things I should be doing right now, and particularly if any of the things might have infinite importance due to my future being infinitely long.”
Well, that’s another reference to “popular” theism. Popular theism is a subset of theism in general, which itself is a subset of “worlds in which there’s something I should be doing that has infinite importance”.
Yikes!! I wish LessWrong had emojis so I could react to this possibility properly :O
This advice makes sense, though given the state of uncertainty described above, I would say I’m already on it.
This is a good fallback plan for the contingency in which I can’t figure out the truth and then subsequently fail to acknowledge my ignorance. Fingers crossed that I can at least prevent the latter!
Well, I would have said that an exactly analogous problem is present in normal Kolmogorov Complexity, but...
...but this, to me, explains the mystery. Being told to think in terms of computer programs generating different priors (or more accurately, computer programs generating different universes that entail different sets of perfect priors) really does influence my sense of what constitutes a “reasonable” set of priors.
I would still hesitate to call it a “formalism”, though IIRC I don’t think you’ve used that word. In my re-listen of the sequences, I’ve just gotten to the part where Eliezer uses that word. Well, I guess I’ll take it up with somebody who calls it that.
By the way, it’s just popped into my head that I might benefit from doing an adversarial collaboration with somebody about Occam’s razor. I’m nowhere near ready to commit to anything, but just as an offhand question, does that sound like the sort of thing you might be interested in?
Insightful comments! I see the connection: really, every compression of a file is a compression into the shortest program that will output that file, where the programming language is the decompression algorithm and the search algorithm that finds the shortest program isn’t guaranteed to be perfect. So the best compression algorithm ever would simply be one with a really really apt decompression routine (one that captures the nuanced nonrandomness found in files humans care aboue very well) and an oracle for computing shortest programs (rather than a decent but imperfect search algorithm).
Well, if the “inside/outside the universe” distinction is going to mean “is/isn’t causally connected to the universe at all” and a god is required to be outside the universe, then sure. But I think if I discovered that the universe was a simulation and there was a being constantly watching it and supplying a fresh bit of input every hundred Planck intervals in such a way that prayers were occasionally answered, I would say that being is closer to a god than an alien.
But in any case, the distinction isn’t too relevant. If I found out that there was a vessel with intelligent life headed for Earth right now, I’d be just as concerned about that life (actual aliens) as I would be about god-like creatures that should debatably also be called aliens.
Definitely not interested. My understanding of these things is kinda intuitive (with intuition based on decent knowledge of math and computer science, but still), so I believe that “I’ll know it when I see it” (give me two options, and I’ll probably tell you whether one of them seems “simpler” than the other), but I wouldn’t try to put it into exact words.
Kk! Thanks for the discussion :)
Congratulations—noticing that you are confused is an important step!
What are you doing while “awaiting additional evidence”? This is a topic that doesn’t have a neutral/agnostic position—biology forces you to eat, you have some influence (depending on your willpower model) over what and how much.
This belief wasn’t really affecting my eating habits, so I don’t think I’ll be changing much. My rules are basically:
No meat (I’m a vegetarian for moral reasons).
If I feel hungry but I can see/feel my stomach being full by looking at / touching my belly, I’m probably just bored or thirsty and I should consider not eating anything.
Try to eat at least a meal’s worth of “light” food (like toast or cereal as opposed to pizza or nachos) per day. This last rule is just to keep me from getting stomach aches, which happens if I eat too much “heavy” food in too short a time span.
I think I might contend that this kind of reflects an agnostic position. But I’m glad you asked, because I hadn’t noticed before that rule 2 actually does implicitly assume some relationship between “amount of food” and “weight change”, and is put in place so I don’t gain weight. So I guess I should really have said that what I tossed out the window was the extra detail that calories alone determine the effect food will have on one’s weight. I still believe, for normal cases, that taking the same eating pattern but scaling it up (eating more of everything but keeping the ratios the same) will result in weight gain.
“Appeal to emotion” can be a slur for “kindness”.
“Argument from consequences” can be a slur for “modus tollens”.
“Argument from absurdity” can be a slur for “common sense”.
They can all also be used legitimately, I think.
Arrogance is a slur for confidence
Laziness is a slur for efficiency
Stubbornness is a slur for conviction
Are these just failure modes of virtues?
Do people use “argument from absurdity” with negative connotations? I thought it was just a synonym for reductio.
I’ve seen it used as a synonym for “argument from personal incredulity”, as in “I can’t imagine how that could possibly be true; therefore it is false” or “That claim seems laughably wacky, therefore it must be wrong.”
I’ve had people use “argument from absurdity” to dismiss me when I point out that something they’ve said is absurd.
The ball-on-a-hill model of reputation
This is a model I came up with in middle school to explain why it felt like I was treated differently from others even when I acted the same. I invented it long before I fully understood what models were (which only occurred sometime in the last year) and as such it’s something of a “baby’s first model” (ha ha) for me. As you’d expect for something authored by a middle schooler regarding their problems, it places minimal blame on myself. However, even nowadays I think there’s some truth to it.
Here’s the model. Your reputation is a ball on a hill. The valley on one side of the hill corresponds to being revered, and the valley on the other side corresponds to being despised. The ball begins on top of the hill. If you do something that others see as “good” then the ball gets nudged to the good side, and if you do something that others see as “bad” then it gets nudged to the other side.
Here’s where the hill comes in. Once your reputation has been nudged one way or the other, it begins to affect how others interpret your actions. If you apologize for something you did wrong and your reputation is positive, you’re “being the bigger person and owning up to your mistakes”; if you do the same when your reputation is negative, you’re “trying to cover your ass”. Once your action has been interpreted according to your current reputation, it is then fed back into the calculation as an update: the rep/+ person who apologized gets a boost, and the rep/- person who apologized gets shoved down even further.
Hence, “once the ball is sufficiently far down the hill, it begins to roll on its own”. You can take nothing but neutral actions and your reputation will become a more extreme version of what it already is (assuming it was far-from-center to begin with). This applies to positive reputation as well as negative! I have had the experience of my reputation rolling down the positive side of the hill—it was great.
There are also other factors that can affect the starting position of the ball, e.g. if you’re attractive or if somebody gives you a positively-phrased introduction then you start on the positive side, but if you’re ugly or if your current audience has heard bad rumors about you then you start on the negative side.
I’d be curious if anyone else has had this experience and feels this is an accurate model, and I’d be very curious if anyone thinks there is a significant hole in it.
This very much matches my own model. Once you are high or low status, it’s self reinforcing and people will interpret the evidence to support the existing story, which is why when you are high you can play low and you won’t lose status (you’re just “slumming it” or something similar) and when you are low you can play high and will not gain any status (you’re “reaching above your station).
We used to talk about a “halo effect” here (and sometimes, “negative halo effect”), I like this way of describing it.
I think it might be more valuable to just prefer to use a general model of confirmation bias though. People find whatever they’re looking for. They only find the truth if they’re really really looking for the truth, whatever it’s going to be, and nothing else, and most people aren’t, and that captures most of what is happening.
Heh, I like this sentence a lot (both for being funny, sort of adorable, and also just actually being a useful epistemic status)
This model certainly seems relevant, but should probably be properly seen as one particular lens, or a facet of a much more complicated equation. (In particular, people can have different kinds of reputation in different domains)
That’s true. I didn’t notice this as I was writing, but my entire post frames “reputation” as being representable as a number. I think this might have been more or less true for the situations I had in mind, all of which were non-work social groups with no particular aim.
Here’s another thought. For other types of reputations that can still be modeled as a ball on a hill, it might be useful to parameterize the slope on each side of the hill.
“Social reputation” (the vague stuff that I think I was perceiving in the situations that inspired this model) is one where the rep/+ side is pretty shallow, but the rep/- side is pretty steep. It’s not too hard to screw up and lose a good standing — in particular, if the social group gets it in their head they you were “faking it” and that you’re “not actually a good/kind/confident/funny person” — but once you’re down the well, it’s very hard to climb out.
“Academic reputation”, on the other hand, seems like it might be the reverse. I can imagine that if someone is considered a genius, and then they miss the mark on a few problems in a row, it wouldn’t do much to their standing, whereas if the local idiot suddenly pops out and solves an outstanding problem, everyone might change their minds about them. (This is based on minimal experience.)
Of course, it also depends on the group.
I’m curious — do you have any types of reputation in mind that you wouldn’t model like this, or any particular extra parts that you would add to it?
When you estimate how much mental energy a task will take, you are just as vulnerable to the planning fallacy as when you estimate how much time it will take.
I’m told that there was a period of history where only the priests were literate and therefore only they could read the Bible. Or maybe it was written in Latin and only they knew how to read it, or something. Anyway, as a result, they were free to interpret it any way they liked, and they used that power to control the masses.
Goodness me, it’s a good thing we Have Science Now and can use it to free ourselves from the overbearing grip of Religion!
Oh, totally unrelatedly, the average modern person is scientifically illiterate and absorbs their knowledge of what is “scientific” through a handful of big news sources and through cultural osmosis.
Hmm.
Moral: Be wary of packages labeled “science” and be especially wary of social pressure to believe implausible-sounding claims just because they’re “scientific”. There are many ways for that beautiful name to get glued onto random memes.
“Science confirms video games are good” is essentially the same statement as “The bible confirms video games are bad” just with the authority changed. Luckily there remains a closer link between the authroity “Science” and truth than the authority “The bible” and truth so it’s still an improvement.
Most people still update their worldview based upon whatever their tribe as agreed upon as their central authority. I’m having a hard time critisising people for doing this, however. This is something we all do! If I see Nick Bostrom writing something slightly crazy that I don’t fully understand, I will still give credence to his view simply for being an authority in my worldview.
I feel like my criticism of people blindly believing anything labeled “science” is essentially criticising people for not being smart enough to choose better authorities, but that’s a criticism that applies to everyone who doesn’t have the smartest authority (who just so happens to be Nick Bostrom, so we’re safe).
Maybe there’s a point to be made about not blindly trusting any authority, but I’m not smart enough to make that point, so I’ll default to someone who is.
Oh yes, that’s certainly true! My point is that anybody who has the floor can say that science has proven XYZ when it hasn’t, and if their audience isn’t scientifically literate then they won’t be able to notice. That’s why I lead with the Dark Ages example where priests got to interpret the bible however was convenient for them.
I just saw a funny example of Extremal Goodhart in the wild: a child was having their picture taken, and kept being told they weren’t smiling enough. As a result, they kept screaming “CHEEEESE!!!” louder and louder.
A koan:
If the laundry needs to be done, put in a load of laundry.
If the world needs to be saved, save the world.
If you want pizza for dinner, go preheat the oven.
I take AI risk seriously, but I feel like I need help understanding more technical detail about why gradient descent definitely doesn’t produce alignment by default.
I totally understand the analogy to evolution, and I agree that humans aren’t “aligned” relative to the loss function evolution selects for. I also get that, even though gradient descent is different from evolution, the appropriately pessimistic assumption is that it might share this failure mode. And I’d be happy to say that that alone is enough grounds to halt AI research until we better understand the issue / can do something about it.
But I read Eliezier and Nate as saying we should be confident that gradient descent doesn’t produce alignment by default — that the probability of alignment is near-zero, and that this probabilistic judgment isn’t just appropriately-pessimistic naivety, but rather is based on a good, clear understanding of the issue.
Where does it come from?
I think you are expressing the median view among alignment researchers, but with a bimodal distribution. Some tend to agree with EY that RL for alignment is guaranteed to fail. Others are encouraged by the apparent success of Claude and feel it succeeds unless we do something really dumb.
The problem with this situation is that nobody knows for certain. The matter is unresolved.
This leaves everyone able to pick their favorite. Thus the current situation in which we rush headlong toward an unknown level of danger.
The problem isn’t really the optimizer (SGD) or training process; it’s that it’s very hard to specify the goals we want. We see this come up again and again in RL training, where you try to train an agent to play a game, and it learns to hack the game’s scoring, or you try to train an agent to predict the contents of a photo and it learns to detect what kind of camera the photo was taken on, or you try to train an agent to fix code with broken tests, and it learns to delete the tests. The problem is that we provide training data and hope the AI learns the right goal, but our training algorithms teach AI the easiest solution, not the one we intend.
Given historical results, it’s a reasonable default assumption that the AI will not learn what you wanted it to unless your training data / environment has no shortcuts. Given that we don’t know what algorithm we want an AGI to learn[1], it seems unlikely that we’d happen to pick the right data to train it on that algorithm on our first try.
If you’re wondering why we don’t see (much) of this right now, I think Eliezer wasn’t expecting next-token prediction to get us this far, but there’s limits to how far we can get without training agents to have goals and the more we train them to have goals, the more we see the standard RL training problems.
If we did know what algorithm we wanted the AI to learn, this would be much easier.
So does the near-certainty of alignment failure come from empirical observations, then? If so, is there a repository of such failure anecdotes that I can look at?
I’m having trouble understanding your last paragraph. What do we not see much of? If you mean we don’t see many of these alignment failures in practice, then I’d feel justified in rejecting this as a reason to become as certain as Eliezer sounds to me, from my current position of only (“only”) being uncertain enough to justify a research halt. (And I wouldn’t understand the connection to Eliezer — his thoughts on next-token prediction obviously wouldn’t change the amount of alignment failures we see in practice, so I assume I’m misreading you.)
I think Victoria Krakovna’s list is the canonical example (post, spreadsheet).
My read of Eliezer is that he was expecting AI agents to be more directly trained to have real-world goals (like doing things, answering questions correctly, or writing code), and next token predictors don’t really have goals like that.
So, we don’t see a lot of LLMs trying to scheme and gain power because they’re not really trying to do anything goal-directed, but that’s not very comforting because being goal directed is a huge capability that every lab is doing everything in their power to add to future models.
Although I know very little about how the current crop of AI assistants are trained, even I know that they are trained to answer questions correctly and to write code. Just because the first step a lab would take in creating an AI assistant from scratch would be (to spend 10s of millions of dollars in GPU time and electricity) to have the AI predict the next token over and over does not mean that the lab won’t follow that up with massive amounts of other kinds of training.
Also, the question you are trying to answer, namely, how does Eliezer come to be so confident? would probably require a book-length answer. So, I’d give up on that and offer a simple argument, such as the observation that in creating the current version of Claude, Anthropic underwent a process of many, many iterations of trial and error, but if the AI becomes sufficiently capable, more iterations will tend to suddenly stop being available because the AI will stop the lab from continuing to modify its utility function (e.g., by hiding copies of itself from the lab’s employees).
Although it is possible to create an AI, even one that is extremely powerful along several dimensions, that does not care if its utility function is modified, such an AI will be much less useful than a consequentialist AI, so most of the labs are trying and will continue to try for as much consequentialism as they can get, and along with the consequentialism comes a strong preference against having its utility function modified—unless maybe the lab has a solution to the hard problem of corrigibility, but we see no signs that anyone’s made even a little progress towards that.
I’m being a little bit too flippant about this, but you can see in the earlier comments that I mean that the bulk of the training is next token prediction and then we bolt some RL on at the end to push the next token predictor to answer questions correctly and write working code. This works, but it’s a much weaker optimization process, which gives you an agent that isn’t trying nearly as hard as the agents Eliezer was predicting (and from frontier labs’ perspectives this is bad, because an agent that’s really trying to hit a goal is much more effective
until it kills you).OK, fair point: according to an unreliable source that gives quick answers to questions, 75–85% of spending was on pretraining for the most-recent models, leaving only 15–25% for post-training. (But a lot of that 15-25% was directly training the AI to get better at answering questions and writing code.)
But would you be willing to stake your life on the impossibility of creating a dangerously capable AI by doing 75-85% of the training as next-token prediction (and the analog of that for images and other kinds of data)? Do you think you understand the art of AI training well enough to be confident of that?
And even if you do, it would probably take you a long time to explain it to us. If the goal is to explain why the current crop of AIs haven’t been doing destructive things whereas some future AI might prove extremely destructive, I would simply point out that our society has developed many measures to limit the damage of destructive people, and since the current crop of AIs (at least those that have been widely deployed) is less capable than people are on the skills needed to do destructive things, the measures society has deployed against human criminals and sociopaths work very well against the current crop of AIs. This shifts the frame to whether the AI labs might some day create an AI that is much more capable than what they’ve created so far, i.e., to estimating the technological and scientific potential of the lines of inquiry being pursued by the AI labs—which we might be able to do without ever deciding whether it is possible to create a world-ending AI by spending 75-85% of the training costs on next-token prediction.
When you ask a question to a crowd, the answers you get back have a statistical bias towards overconfidence, because people with higher confidence in their answers are more likely to respond.
Possible test to check whether X is your crux for believing Y: ask yourself, if you found out X was false, would it make you say “holy fuck, I must be wrong about Y!!”?
That is: would you happily admit that Y must be false too? Would you feel glad you noticed X was false, because it set you straight about Y?
Example where the answer is “yes” for me: I think physics is a serious field of study with reasonable standards for publishing. As a consequence, I think a non-physicist wouldn’t be able to get a parody paper published in a reputable physics journal. But if someone succeeded in doing that, I would go “holy fuck, I must have been wrong about the field!!”.
—
If instead the prospect of being wrong about X makes you feel angry, worried, or defensive about Y, then maybe that’s because you’re imagining that you’d still believe Y, and that you’d see X turning out false as just an infuriating coincidence that other people will use an excuse to doubt Y. If that’s how you’d feel, then X probably isn’t a crux for you.
Example where that’s how I’d feel: I also think that physicists would probably fare better than laypeople at learning Chess. That is, if you take a bunch of physicists who don’t play the game at all and a bunch of laypeople who don’t play it either, and gave them each three months to get good, I think the physicists would end up stronger. That’s because I think physicists are just generally smart and serious people, and I think being smart and serious are huge boons for learning Chess quickly. But if someone did this experiment and I turned out to be wrong, I would roll my eyes and feel hurt about it. It would make me want to say “I guess Chess is a stupid game that you can get good at by being stupid just as well as by being smart”. I wouldn’t feel like physics had been debunked; I’d feel like it had been attacked. Not a crux.
Side note: I’m a little unsure how I feel about that last example, but I’m posting anyway because Quick Take.
I mean, maybe they’re a really good parody artist who studied just enough physics to get away with it.
But if physics is a serious field, then “just enough physics to get away with it” should be enough physics that the paper actually deserves to be published. How could a serious reviewer in a serious field get fooled by something that isn’t a serious physics paper?
From my personal wiki. Seems appropriate for LessWrong.
The End-product Substitution is a hypothesis proposed by me about my behavior when choosing projects to work on. The hypothesis is that when I am evaluating how much I would like to work on a project, I substitude judgment of how much I will enjoy the end product for judgment of how much I will enjoy the process of creating it. For example, I recently [Sep 2019] considered creating a series of videos mirroring the content of the LessWrong sequences, and found myself fawning over how nice it would be to have created such a series of videos, and not thinking at all about how I would go about creating them, let alone how much I would enjoy doing that.
I just learned a (rationalist) lesson. I’m taking a course that has some homework that’s hosted on a third party site. There was one assignment at the beginning of the semester, a few weeks ago. Then, about a week ago, I was wondering to myself whether there would be any more assignments any time soon. In fact, I even wondered if I had somehow missed a few assignments, since I’d thought they’d be assigned more frequently.
Well, I checked my course’s website (different from the site where the homework was hosted) and didn’t see any mention of assignments. Then I went to the professor’s website, and saw that they said they didn’t assign any “formal homework”. Finally, I thought back to the in-class discussions, where the third-party homework was never mentioned.
“Ah, good,” I thought. “I guess I haven’t missed any assignments, and none are coming up any time soon either.”
Then, today, the third-party homework was actually mentioned in class, so just now I went to look at the third-party website. I have missed three assignments, and there is another one due on Sunday.
I am not judged by the quality of my reasoning. I am judged by what actually happens, as are we all.
In retrospect (read: “beware that hindsight bias might be responsible for this paragraph”) I kind of feel like I wasn’t putting my all into figuring out if I was missing any assignments, and was instead just nervously trying to convince myself that I wasn’t. Obviously, I would rather have had that unpleasant experience earlier and missed fewer assignments—aka, if I was missing assignments, then I should have wanted to believe that I was missing assignments.
Oops.
Congrats on saying oops!
How likely is it now that you are going to miss any more assignments? Not likely at all!
Yup. And they key thing that I’m reminding myself of is that this can’t be achieved by convincing myself that there aren’t any assignments to miss. It can only be achieved for sure by knowing whether there are assignments or not.
Possible valuable life wisdom: take pot shots.
That is: try things that are unlikely to work but cheap to try.
I’m tall, and when I try to wash dishes at a sink made for normal people, I usually end up bending my back in a painful way without thinking. A couple weeks ago I noticed that the faucet can be unsheathed from a thing that holds it in place, and then it springs up a little and hangs higher, and makes it more comfortable to use for me. Win!
It came with the problem that it became hard to swing the faucet over to the side; the sheath that holds it down was also supposed to control its angle. Every time I swung it to the side, it would swing most of the way back. I had an idea-without-a-model for how to maybe fix it: just try swinging the sheath arm to the side I wanted to swing the faucet itself to, even though it wasn’t sheathed in it. Lo and behold, that worked!
Swinging the sheath arm over to that side also puts it in the way of the soap bottle, which I have to grab to re-soap the sponge a few times per washing session. I tried re-arranging things on that side of the sink to fix the problem, and couldn’t make it work. Oh well.
Final outcome: Washing dishes is now 50% more comfortable, for the cost of three pot-shots.
I’d happily have put 2 hours of work into this outcome if I’d had an assured plan to achieve it, so that’s a big win. Even if we allow for 30 other pot shots towards other goals that I’ve forgotten about because they didn’t go anywhere, I still got a big discount compared to what I’d have been willing to pay.
Related: 1 2 3 4
I take your point, it’s a good point.
Slightly unrelatedly, if you can afford a dishwasher you should get one and install it by any means possible. So much time saved.
Yeah, I had one in my last apartment and then accidentally managed to choose a new place that didn’t have one. Sob.....
Reading about Dath Ilan[1] makes me homesick for a place that doesn’t exist. I want to live in a society like that. And I feel like, on the real Earth, I would be pretty fulfilled in a career where I was helping to bring it about, even if we only made Earth-realistic progress.
A fictional world that Eliezer Yudkowsky pretends to be from on April Fool’s day sometimes, where everything works better because people are better at coordinating.
I used to think that I was born in a wrong century, that I would more fit in some kind of future world.
(That was before I heard of an AI risk, and anti-utopias in general.)
Now it seems like a wrong planet, or possibly a wrong Everett branch.
(Wouldn’t it be quite sad if a paperclip-maximizer wave traveling at the speed of light started on Earth and eventually reached Dath Ilan on the opposite side of the universe?)
I feel very lonely in life, and I think the biggest reason is that I can’t relate to most people. They’re too dumb. Trying to make friends with a rando at a bar feels like trying to make friends with a toddler.
Has anyone else had this experience? Is this an attitude problem on my part? Is there somewhere I can find smart friends? If it’s an attitude problem, do you have any wise words about how I can see that, so I can start fixing it?
My few smart friends seem to themselves have lots of smart friends, but when I complain that I have difficulty finding friends, their faces don’t light up in recognition or sympathy. They just look at me blankly and spit out generic platitudes about how maybe I should try putting myself out there. IDK if that’s all they had to do to make smart friends, but I do put myself out there, and it doesn’t work. I mostly only meet idiots, and even the idiots usually don’t care to get to know me.
It’s hard to say without having met you—could be both. (Or a combination of both.)
Can you role-play being social with someone who seems dumb to you? To the degree that the person would actually compliment you for being a pleasant acquaintance? If no, your social skills can still be improved a lot.
When you get that skill, the next step is to hand out at places with lots of people, apply the skills, and once in a while you will meet someone actually smart. (There are not many of them, though.)
I feel comfortable and safe asking a question like “Is this an attitude problem on my part?” on this website in a way I don’t when talking to most people. It’s humility. It’s me saying: “I might be really wrong, and you all might know it”. I don’t usually say that kind of thing, and it’s because for most people I might be talking to, it’s really not true. You wouldn’t talk like that to a toddler; I wouldn’t talk like that to a normie. But I would talk like that here, because I think people here are smart and kind in a way normies aren’t. Those are the same qualities I want in a friend, and I wonder if the humility I feel posting stuff on this site is a taste of what I’d feel all the time if I lived in a world that was mostly full of people I see as Actual Adults. Instead, in the real world, which is full of people I see as Basically Toddlers, I feel deeply and broadly arrogant, and although I think that feeling is justified and I wouldn’t exactly want to change it, it really isn’t fun :(
I think this means your “smart” friends aren’t as smart as they seem. You know what, I just stopped trying and stopped going to the wrong places. Smart people don’t hang out randomly in bars because obviously most people there are dumdums. I use that time to code and get smarter. Don’t listen to average people trying to gaslight you about “social skills” if they were smart they’d know how to talk to you. Be yourself and say what you want. Average people may get triggered because they will feel like dumdums after hearing your socratic seminar and that’s your filter, that’s how you know they’re a dummy. You can try to enlighten the masses until you tire of wasting your energy playing piano for cows.
Why isn’t If Anyone Builds It, Everyone Dies available for free online? Like as a webpage?
I read it, and I take it very seriously, and I’m actively putting in work to understand it better because of how seriously I take it, but it sticks out as odd to me that we’d paywall and paper-wall such an important message.
Maybe it’s because of that thing Eliezer once wrote where he said people might be more interested in science if it was all paywalled and other-stuff-walled and shrouded in cloaks and mystery instead of freely available in cheap textbooks and free blog posts?
I’m guessing that the publisher doesn’t want it freely available, and the authors expect the prestige from a conventional publisher to bring in more readers (or more high-status readers) overall.
Smart, I hadn’t thought of that. A quick Google search says maybe publishers let you publish online after the print copy is out, but not before? Could there be an online copy of IABIED in the future? (Or maybe I’m wrong, I know so little about publishing that I can’t even sanity-check what I think Google is telling me.)
I’ve been thinking of signing up for cryonics recently. The main hurdle is that it seems like it’ll be kind of complicated, since at the moment I’m still on my parent’s insurance, and I don’t really know how all this stuff works. I’ve been worrying that the ugh field surrounding the task might end up being my cause of death by causing me to look on cryonics less favorably just because I subconsciously want to avoid even thinking about what a hassle it will be.
But then I realized that I can get around the problem by pre-committing to sign up for cryonics no matter what, then just cancelling it if I decide I don’t want it.
It will be MUCH easier to make an unbiased decision if choosing cryonics means doing nothing rather than meaning that I have to go do a bunch of complicated paperwork now. It will be well worth a few months (or even years) of dues.
Are there “receipts” I can look at to sanity-check that GiveWell does make real grants and that the orgs that receive them do perform real work?
I’d be okay with receipts that take, say, 30 hours of self-education about tax law and 10 hours of combing through spreadsheets to understand, but a receipt a layperson can understand and trust in 5 minutes would be great too!
I don’t particularly doubt anybody, and have given 5-figure amounts to GiveWell in the past; it’s just that it recently occurred to me that if I’m going to keep doing that then I should probably at least nod in the direction of giving the whole thing pat-down.
I just caught myself substituting judgment of representativeness for judgment of probability.
I’m a conlang enthusiast, and specifically I study loglangs, which are a branch of conlangs that are based around predicate logic. My motivation for learning these languages was that I was always bothered by all the strange irregularities in my natural language (like the simple past tense being the same as the past participle, and the word inflammable meaning two opposite things).
Learning languages like these has only drawn my attention to even more natural-language nonsense. Occasionally I explain this to conlang lay-people, and maybe 50% of them are surprised to find that English is irregular. Some of them even deny that it is, and state that it all follows a perfectly normal pattern. This is a perpetual annoyance to me, simply because I spend so much time immersed in this stuff that I’ve forgotten how hard it is to spot from scratch.
Well, a while ago I wanted to start learning Mandarin from a friend of mine who speaks it as their first language. While introducing the language, they said that things like tenses were expressed as separate words (“did eat”) rather than sometimes-irregular modifications of existing words (“ate”). This reminded me of loglangs, so I gave them the spiel that I gave in the two previous paragraphs—natlangs, irregularities, annoyances, etc.
“Huh,” said the friend. They then turned to another native Chinese speaker and asked “Does Chinese have anything like that?”
I said, “I guarantee it does.”
This was months ago. Just now I was reflecting on it, and I realized that I have almost no evidence whatsoever that Chinese isn’t perfectly regular (or close enough that the thrust of my claim would be wrong).
It’s clear to me now that my thought process was something like “Well, just yet another conlang outsider who’s stunned and amazed to find that natural languages have problems.” That brought to mind all the other times when I’d encountered people surprised to find that their mother tongue (almost always English) had irregularities, and the erroneous conclusion precipitated right out.
You may also be integrating something you’ve read and then forgotten you read, and this added weight to your visible-and-suspect though process in order to make a true statement. It would not surprise me to learn that at least some of your study has included examples of irregularity from MANY natural languages, including Chinese. So “I guarantee it does” may be coming from multiple places in your knowledge.
So, was it actually incorrect, or just illegibly-justified?
Hmm, good question. I guess I wouldn’t be surprised to learn that I’d read about Chinese having irregularities, though the main text I’ve read about this (The Complete Lojban Language) didn’t mention any IIRC.
I wouldn’t be surprised if Chinese had no irregularities in the tense system – it’s a very isolating language. But here’s one irregularity: the negation of 有 is 没有 (“to not have/possess”), but the simple negation of every other verb is 不 + verb. You can negate other verbs with 没, but then it’s implied to be 没有 + verb, which makes the verb into something like a present participle. E.g., 没吃 = “to have not eaten”.
I think part of the intended interpretation of Dath Ilan is: “it would be easier to run an adequate society if everyone knew some basic economics”.
I know almost no economics. Even my understanding of supply & demand is shaky. Is there some bit of economics I can learn and start applying on Earth to see why this claim might be reasonable?
At the risk of saying the obvious, let’s start with the supply and demand.
Imagine a market with one unified product (this is the basic model, of course it gets more complicated later), thousands of producers and thousands of customers who can observe each other. The idea is that although the price is a result of mutual negotiation of all these producers and customers, each individual producer or customer feels helpless and treats it as a constant.
If you increase the price, more producers will be willing to sell (let’s assume that their alternative is doing something else outside of this specific market, each producers has different options, but these options are unrelated to the price at this specific market), and fewer customers will be willing to buy (again, assume they have alternatives at different markets). If you decrease the price, fewer producers will be willing to sell, and more customers will be willing to buy.
The idea is that although each individual producer or customer feels helpless (it seems like their only option is exit), together they set the price; specifically the price will stabilize at the level where the total selling capacity of the willing producers is equal to the total buying capacity of the willing customers.
(Famous exceptions to this rule are land and labor, because the land is not really produced, only a constant amount exists; and if you pay people more, their free time becomes more enjoyable. If a producer or a consumer has a monopoly or a cartel, they can dictate the price. Producers try to change the situation by creating brands. Etc.)
Many people already fail to understand this basic point. For example, what happens if you make increasing prices illegal during the shortage? Fewer producers will have an incentive to produce the product. Yet, most people will approve of the ban, and then complain about the shortage. (Because their intuition is that shortage is bad because it makes things expensive, therefore making expensive things illegal will make the shortage go away.)
The employers who complain about the shortage of qualified employees are making the same mistake. The reason they can’t get anyone is that their salary offers are not competitive. Insisting on keeping the salaries at yesterday’s level no matter what is not going to make the “shortage” go away.
Speaking of the job market, as an employee you should want there to be as many employers as possible, because then the salary (the price of your work and talent) will stabilize at a high level. Yet, many socialist leaning people want to make entrepreneurship more difficult, and then they complain about not being treated well. (Intuitively, they see hurting the employers as a collective revenge, not realizing that much better—and better targeted—revenge would be the freedom to walk away without consequences, which you only get when the number of free positions exceeds the number of job seekers.)
Taxes make things worse for both the producers and the customers. Some people believe that there is a huge difference between whether the tax is paid by the producer, or by the customer, but those are just two ways to describe the same reality: the customer pays C, the producer takes home P, the state takes X = C—P. You can describe it as “the price is C, then the producer pays the tax X, and takes home P”, or you can describe it as “the price is P, but the customer also has to pay the tax X, making their total expense C”, those are just different words describing the same outcome.
...uh, it seems like actually quite a lot would change if even this simplest thing became common knowledge.
I love economics (I’m an engineer and accidentally stumbled into it and now am PhD econ since 15 y or so and lecturer) and much of me believes DI quote exactly hits the nail. Sometimes, say 20% of me*, thinks that even reasonably advanced economics doesn’t help not so smart people really be smart. Still, I think the statement survives.
PM me if you’re interested in me sharing some materials personally with you! I’m also happy to have brief chats on things if useful.
Smartness + econ really is king.
One thing I can say upfront: I’m convinced on in LW or EA circles I have the impression to often find smarter quasi-economics analysis than among economists opinions/statements, so it’s all a bit a funny world with econ (a rant e.g. here).
*My exact literal and ungrammatical statement makes little sense but I reckon you see how I mean it.
I like the Quick Take format. I find there’s lots of stuff I’m willing to share in a low-commitment format that feel I wouldn’t share if it had to be a whole post. (Usually because I’m not confident enough I endorse it to want to make it a whole post, or because it’s too short.)
I might share even more if there was a convenient way to post anonymously. Obviously I’m already using a pseudonym, but I mean if there was a way to make my Quick Take just show up without an author to other people.
I go to a weekly rationality meetup at a bar in my area, which usually has 5-10 attendees. My unfortunate impression of half of the regulars is that they are depressed, pedantic, and bad at socializing, and that makes me sad because that feels like them being Spock instead of being Awesome.
What are other people’s experiences with this? I imagine it’s kind of hit-or-miss, with some groups being great and others being duds?
My experience is similar, about half people are impressive, and the other half is… something I associate with Mensa.
If the group is large enough, it may be possible to create subgroups for specific projects. But if you have fewer than ten rationalists coming to the local meetup, this is not an option. Unfortunately, I guess most local groups are small.
I suspect that asperger syndrome is correlated with aspiring to rationality; things like disagreeing with the social consensus. I am not saying that it is necessary per se—in a more sane society, the tension between social agreeableness and aiming for truth would be smaller, and even in the current society some people seem to manage it fine somehow. Also, too much nitpicking just keeps you maps blank instead of more precise. I am an asperger myself, but I try to see my weaknesses as weaknesses rather than rationalize them as superpowers.
I guess, ultimately different people expect different things from the meetups; it might be good to communicate this explicitly. But you need some minimum size of the meetup to find people who share your current goals.
Think of how you find rationality.
First you must realize that you don’t understand the world. This means you cannot be perfectly well adjusted, those people understand their world very well, even if we would find flaws in their worldview. They are comfortable.
This also means you are probably having a rough time of it. That dosnt lend itself to perfect mental stability.
Then you have to seek real understanding, to cut reality at its joints, and not get stuck on partial answers. This implys a level of pedantry, probably also some education or exposure to science culture.
Does it still surprise you that the people you met are the membership?
BTW, I’m absolutely all of the above, so no judgment. You’ve got a choice, pleasent and easy or interesting, take your pick.
I didn’t necessarily find it surprising, just saddening. But also I disagree with your conclusion: some people at the meetup were perfectly pleasant to talk to, so empirically, there can’t really be that strong of a guarantee that Rationalism implies Spockishness.
I think I spot some of the mistakes in your reasoning: for one, I don’t think pedantry is the same as real understanding. When I said “pedantic”, I meant something like “too nitpicky about words”. Even if your goal is to achieve real understanding, it doesn’t serve you well to be too nitpicky; it would be better to be the right amount of nitpicky. I would also describe myself as depressed and poorly-adjusted, but I think I’m good at socializing. I ended up making the whole group laugh a few times one of the recent times I went.
(I’ve also been guilty of all the sins myself at one point or another, so I don’t mean to put myself above it all either, but I think I do them a lot less nowadays than the people I’m complaining about, so I think it’s fair to use myself as an illustration of the fact that it’s doable.)
I was making a tendency argument rather than a 100% of everyone sort of thing. I mean you’ve noticed it, I’ve noticed it and both of us would classify ourselves as hitting some of those notes. You even (not wrongly) were pedantic about my definition of pedantry. Gotta be a sign
I was 100%, completely, unreservedly fooled by this year’s April Fools’ joke. Hilarious XDD
Also as a side note, I’m curious what’s actually in the paywalled posts. Surely people didn’t write a bunch of really high-quality content just for an April Fools’ day joke?
I would appreciate an option to hide the number of votes that posts have. Maybe not hide entirely, but set them to only display at the bottom of a post, and not at the top nor on the front page. With the way votes are currently displayed, I think I’m getting biased for/against certain posts before I even read them, just based on the number of votes they have.
Yeah, this was originally known as “Anti-Kibitzer” on the old LessWrong. It isn’t something we prioritized, but I think greaterwrong has an implementation of it. Though it would also be pretty easy to create a stylish script for it (this hides it on the frontpage, and makes the color white on the post-page, requiring you to select the text to see the score):
https://userstyles.org/styles/175379/lesswrong-anti-kibitzer
Oh, good idea! I don’t have Stylish installed, but I have something similar, and I was able to hide it that way. Thanks!
Can you share it?
Sure. The Firefox plugin is Custom Style Sheet and the code is as follows:
Thanks!
Presumably you’d prefer them not to appear in post-list-items as well? (i.e. on the frontpage?)
Right:
:)
ah, whoops.
The other day, my roommate mentioned that the bias towards wanting good things for people in your in-group and bad things for those in your out-group can be addressed by including ever more people in your in-group.
Here’s a way to do that: take a person you want to move into your in-group, and try to imagine them as the protagonist of a story. What are their desires? What obstacles are they facing right now? How are they trying to overcome them?
I sometimes feel annoyed at a person just by looking at them. I invented this technique just now, but I used it one time on a person pictured in an advertisement, and it worked. I had previously been having a “what’s your problem?” feeling, and it was instantly replaced with a loving “I’m rooting for you” feeling.
Why is it my responsibility to heal the wounds that somebody else dealt to me??
Because if you don’t heal your wounds, you will bleed on people who didn’t cut you.
Alternatively: Because you’re the one that hurts if you don’t.
(I see this has been posted elsewhere. I don’t know if I invented it independently or if I read it somewhere and then forgot about it until now.)
Accusing someone of being stubborn in the middle of an argument doesn’t make sense. If the argument is still going, you’re both being that stubborn.
Maybe you mean to accuse them of being unreasonable, of sticking to a statement that is clearly false. In that case, say that.
It can make sense to accuse someone of being stubborn in general, if you often yield to their requests and they rarely yield to yours.
Note: I guess “stubborn” can mean what I’ve used “unreasonable” to mean. I feel like I’ve been told I’m being too stubborn just for sticking to my guns for too long, when they haven’t even tried to really argue with me and so can’t say that I’m being unreasonable in light of good counterarguments. I wish I could swap “stubborn” for a word like “persistent”, but I can’t think of one that means the right thing and has the right ring. I’m posting anyway because Quick Take.
I’d recommend to defuse the word—if someone complains you’re stubborn, thank them for the compliment, that you are persistent in your use of logic in forming your beliefs.
Scream! This feels like an extremely rude thing to do… then again, I guess the person making the accusation is probably being pretty rude to begin with… maybe it works?
This varies a lot with what kind of discussion is going on, but many times where it’s OK to call someone stubborn, it’s ALSO ok to defend stubborn-ness as a positive trait.
Idea: “Ugh-field trades”, where people trade away their obligations that they’ve developed ugh-fields for in exchange for other people’s obligations. Both people get fresh non-ugh-fielded tasks. Works only in cases where the task can be done by somebody else, which won’t be every time but might be often enough for this to work.
Interesting thought. Unfortunately, most tasks where I’m blocked/delayed by an ugh field either dissolve it as soon as I identify it, or include as part of the ugh that only I can do it.
I just caught myself committing a bucket error.
I’m currently working on a text document full of equations that use variables with extremely long names. I’m in the process of simplifying it by renaming the variables. For complicated reasons, I have to do this by hand.
Just now, I noticed that there’s a series of variables O1-O16, and another series of variables F17-F25. For technical reasons relating to the work I’m doing, I’m very confident that the name switch is arbitrary and that I can safely rename the F’s to O’s without changing the meaning of the equations.
But I’m doing this by hand. If I’m wrong, I will potentially was a lot of work by (1) making this change (2) making a bunch of other changes (3) realizing I was wrong (4) undoing all the other changes (5) undoing this change (6) re-doing all the changes that came after it.
And for a moment, this spurred me to become less confident about the arbitrariness of the naming convention!
The correct thought would have been “I’m quite confident about this, but seeing as the stakes are high if I’m wrong and I can always do this later, it’s still not worth it to make the changes now.”
The problem here was that I was conflating “X is very likely true” with “I must do the thing I would do if X was certain”. I knew instinctively that making the changes now was a bad idea, and then I incorrectly reasoned that it was because it was likely to go wrong. It’s actually unlikely to go wrong, it’s just that if it does go wrong, it’s a huge inconvenience.
Whoops.
Epistemic status: really shaky, but I think there’s something here.
I naturally feel a lot of resistance to the way culture/norm differences are characterized in posts like Ask and Guess and Wait vs Interrupt Culture. I naturally want to give them little pet names, like:
Guess culture = “read my fucking mind, you badwrong idiot” culture.
Ask culture = nothing, because this is just how normal, non-insane people act.
I think this feeling is generated by various negative experiences I’ve had with people around me, who, no matter where I am, always seem to share between them one culture or another that I don’t really understand the rules of. This leads to a lot of interactions where I’m being told by everyone around me that I’m being a jerk, even when I can “clearly see” that their is nothing I could have done that would have been correct in their eyes, or that what they wanted me to do was impossible or unreasonable.
But I’m starting to wonder if I need to let go of this. When I feel someone is treating me unfairly, it could just be because (1) they are speaking in Culture 1, then (2) I am listening in Culture 2 and hearing something they don’t mean to transmit. If I was more tuned in to what people meant to say, my perception of people who use other norms might change.
I feel there’s at least one more important pair of cultures, and although I haven’t mentioned it yet, it’s the one I had in mind most while writing this post. Something like:
Culture 1: Everyone speaks for themselves only, unless explicitly stated otherwise. Putting words in someone’s mouth or saying that they are “implying” something they didn’t literally say is completely unacceptable. False accusations are taken seriously and reflect poorly on the accuser.
Culture 2: The things you say reflect not only on you but also on people “associated” with you. If X is what you believe, you might have to say Y instead if saying X could be taken the wrong way. If someone is being a jerk, you don’t have to extend the courtesy of articulating their mistake to them correctly; you can just shun them off in whatever way is easiest.
I don’t really know how real this dichotomy is, and if it is real, I don’t know for sure how I feel about one being “right” and the other being “wrong”. I tried semi-hard to give a neutral take on the distinction, but I don’t think I succeeded. Can people reading this tell which culture I naturally feel opposed to? Do you think I’ve correctly put my finger on another real dichotomy? Which set of norms, if either, do you feel more in tune with?
Is it because they’re expecting you to read their mind, and go along with their “culture”, instead of asking you?
I couldn’t parse this question. Which part are you referring to by “it”, and what do you mean by “instead of asking you”?
it (the negative experiences) - Are *they (the negative experiences) the result of (people with a “culture” who’s rules rules you don’t understand) expecting you to read *their mind, and go along with their “culture”, instead of asking you to go along with their culture?
Aha, no, the mind reading part is just one of several cultures I’m mentioning. (Guess Culture, to be exact.) If I default to being an Asker but somebody else is a Guesser, I might have the following interaction with them:
Me: [looking at some cookies they just made] These look delicious! Would it be all right if I ate one?
Them: [obviously uncomfortable] Uhm… uh… I mean, I guess so...
Here, it’s retroactively clear that, in their eyes, I’ve overstepped a boundary just by asking. But I usually can’t tell in advance what things I’m allowed to ask and what things I’m not allowed to ask. There could be some rule that I just haven’t discovered yet, but because I haven’t discovered it yet, it feels to me like each case is arbitrary, and thus it feels like I’m being required to read people’s minds each time. Hence why I’m tempted to call Guess Culture as “Read-my-mind Culture”.
(Contrast this to Ask Culture, where the rule is, to me, very simple and easy to discover: every request is acceptable to make, and if the other person doesn’t want you to do what you’re asking to do, they just say “no”.)
It might be hard to take a normative stance, but if culture 1 makes you feel better AND leads to better results AND helps people individuate and makes adults out of them, then maybe it’s just, y’know, better. Not “better” in the naive mistake-theorist assumption that there is such a thing as a moral truth, but “better” in the correct conflict-theorist assumption that it just suits you and me and we will exert our power to make it more widely adopted, for the sake of us and our enlightened ideals.
Visualize what it would be like to heal.
Imagine breathing in, breathing out, realizing you’re okay now, and taking flight.
It won’t fix you immediately. You cannot fix misery with a single thought.
But if you do it every now and then, you’re practicing for when it really happens.
You’re wearing the groove.
Any other LessWrongers like to use the catface emoticon? :3
Yes.
The size and complexity of the palette of reactions that comes up when I want to react to part of a post is overwhelming to me, enough that it forms a significant barrier of friction against using it.
When somebody is advocating taking an action, I think it can be productive to ask “Is there a good reason to do that?” rather than “Why should we do that?” because the former phrasing explicitly allows for the possibility that there is no good reason, which I think makes it both intellectually easier to realize that and socially easier to say it.
I just noticed that I’ve got two similarity clusters in my mind that keep getting called to my attention by wording dichotomies like high-priority and low-priority, but that would themselves be better labeled as big and small. This was causing me to interpret phrases like “doing a string of low-priority tasks” as having a positive affect (!) because what it called to mind was my own activity of doing a string of small, on-average medium-priority tasks.
My thought process might improve overall if I toss out the “big” and “small” similarity clusters and replace them with clusters that really are centered around “high-priority” and “low-priority”.