Rationality-related things I don’t know as of 2023
One of the blog posts I’m most fond of is Things I Don’t Know as of 2018. It’s by Dan Abramov, one of the more prominent people in the world of front-end web development. He goes through a bunch of relatively basic programming-related things that he doesn’t understand, like unix commands and low-level languages.
I’d like to do something similar, but for rationality-related things. Why?
For fun.
To normalize the idea that no one’s perfect.
It’ll make it easier to address these knowledge gaps. Or maybe just more likely that I actualy do so.
Here’s the list:[1]
Simulcra. I spend some time going through the posts and it’s one of those things that just never manages to click with me.
Blockchain. I guess the thing that I don’t understand here is the hype. I get that it’s a basically a database that can’t be editted and I’ve read through articles talking about the use cases, but it’s been around for a while now and doesn’t seem to have been that game changing. Yet there are smart people who are super excited about it and I suspect that there are things I am failing to appreciate, regardless of whether their excitement is justified.
Morality. To me it seems like rationality can tell you how to achieve your goals but not what (terminal) goals to pick. Arguments that try to tell you what terminal goals to pick have just never made sense to me. Maybe there’s something I’m missing though.
Quantum physics. I skipped/lightly skimmed the sequence posts on this. Seemed high effort and not particularly important. Well, it is cool to understand how reality works at the most fundamental level. Hm. I would be interested in a going through some sort of lower effort bigger picture material on quantum physics. I spent some time messing around with that sort of stuff like 13 years ago but all that stuck is some vague notion that reality is (fundamentally?) probabilistic and weird.
Evolution. I get that at a micro-level, if something makes an organism more likely to reproduce it will in fact, err, spread the genes. And then that happens again and again and again. And since mutations are a thing organisms basically get to try new stuff out and the stuff that works sticks. I guess that’s probably the big idea but I don’t know much beyond it and remember being confused when I initially skimmed through the The Simple Math of Evolution sequence.
Evolutionary psychology. I hear people make arguments like “X was important to our hunter-gatherer ancestors and so we still find ourselves motivated by it/to do it today because evolution is slow”. X might be consuming calories when available, for example. There’s gotta be more to evolutionary psychology than that sort of reasoning, but I don’t know what the “more” is.
Bayes math. I actually think I have a pretty good understanding of the big picture ideas. I wouldn’t be able crunch numbers or do things that they teach you in a stats course though.[2] Nor do I understand the stuff about log odds and bits of evidence. I’d have to really sit down, think hard about it, and spend some time practicing using it.
Solomonoff induction. I never took the time to understand it or related ideas.
Occam’s razor. Is it saying anything other than
P(A) >= P(A & B)?[3]Moloch. I enjoyed Meditations on Moloch and found it to be thought provoking. I’m not sure that I really understand what Moloch actually is/represents though. I struggle a little with the abstractness of it.
Double crux. This is another one of those “maybe I actually understand it but it feels like there’s something I’m missing” things. I get that a crux is something that would change your mind. And yeah, if you’re arguing with someone and you find a crux that would make you agree with them if vice versa and stuff, that’s useful. Then you guys can work on discussing that crux. Is that it though? Isn’t that common sense? Why is this presented as something that CFAR discovered? Maybe there’s more to it than I’m describing?
Turing machines. Off the top of my head I don’t really know what they are. Something about a roll of tape with numbers and skipping from one place to the next and how that is somehow at the core of all computing? I wish I understood this. After all, I am a programmer. I spent a few weeks skimming through a Udacity coures on the theory of computation a while ago but none of it really stuck.
If anyone wants to play the role of teacher in the comments I’d love to play the role of student.
- ↩︎
To construct it I skimmed through the table of contents for Rationality from AI to Zombies, the top posts of all time, the tags page, and also included some other stuff that came to my mind.
- ↩︎
But I would like to. I tried skimming through a couple of textbooks (Doing Bayesian Data Analysis by Kruschke and Bayesian Data Anaysis by Gelman) and found them to be horribly written. If anyone has any recommendations let me know.
- ↩︎
Well,
>instead of>=since 0 and 1 are not probabilities but maybe in some contexts it makes sense to treat things as having a probability of 0 or 1.
On Turing machines:
Back in the 1930′s — before general-purpose computers, notice — Turing asked himself the question “what is computation?” and came up with the simplest possible model of what it means to compute something. That model is the Turing machine.
A Turing machine consists of a finite state machine interacting with an infinitely long tape divided into an infinitely long string of cells, each of which can hold a single symbol drawn from a finite alphabet. The machine can move along the tape in either direction, read symbols, and write symbols. For the exact details, see any number of sources on the Internet.
A Turing machine can be viewed as computing a function: you put a value on the tape, encoded in the alphabet, set the machine running, and if it eventually stops, decode whatever is now on the tape to give the result of that function. (If it never stops, the function is not defined for that value.) The definition of the function is encoded in the finite state machine, which will be different for each function, but there are also “universal” Turing machines which can compute any computable function by being given a finite program for calculating it on the tape when it starts.
The importance of Turing machines lies in this: every simple model of computation anyone can come up with is either less capable than the Turing machines (e.g. finite state machines with no infinite tape) or exactly equal in capability. The Turing machine model, simple as it is, exactly captures the idea of “computability”. That this is so is called the Church-Turing thesis. As it is about the intuitive idea, the thesis is not a theorem, but no-one has been able to come up with anything that is intuitively “computable”, but is not Turing-computable. In particular, all of the simple variations of a Turing machine, like giving it more tapes, or more symbols, or an n-dimensional tape, and so on, do not change what it can or cannot compute.
I was going to recommend a rather old book by Marvin Minsky, “Computation: Finite and Infinite Machines”, which I devoured in the space of a week 40 or 50 years ago, which exhaustively goes into the details of Turing machines and several equivalent models, but I see that it goes for around £50 second hand, and there does not seem to be any online copy. But the Wikipedia article is otherwise a good place to start. Especially read the historical section. The question “what can be computed by machinery?” goes back to Babbage in the 19th century.
Wow, that is extremely cool.
Somehow I never understood this idea of how such a simple (even if it’s theoretical) machine really captures the idea of computability. I’m still fuzzy on the how, but I think I have a much better understanding of the what after reading your comment. Thank you!
And my interest is piqued. I’m going to pursue an understanding of this now. I’ll probably reply to this comment at some point with further questions but it’ll take me some time to read and think first.
I’m not sure what work “simple” is doing there. There are theoretical models of hypercomputation (more power than Turing machines) , but we can’t build them, so the Turing limit appears to be a physical fact, not a mathematical one.
“Simple” is indeed doing no work and should have been omitted.
The thesis that computation beyond the Turing limit is physically impossible has also been proposed. We can imagine various sorts of hypercomputation, but as far as we know, they can’t be built. This version of the general idea that Turing-uncomputable functions cannot be computed is known as the physical Church-Turing thesis. In principle, given some particular mathematical theory of physics, one might be able to prove it as a theorem.
It has been speculated that quantum physics might reopen the question by allowing some new ways of calculating things that are not possible in classical physics. I think Scott Aaronson has written on the subject, and has proposed that if a theory of physics does allow computability beyond Turing, that in itself is evidence against the theory.
I would disagree, at least I wouldn’t give it much evidence against.
In some sense, this is me being annoyed at how much people think that the principle of normality/Egan’s law universally applies. This is a very clear failure case of the principle of normality, because conditional on hyper computers being buildable, then it doesn’t add up to normality, but to extremes.
No, quantum physics does not allow you to make computers beyond Turing machines.
This. There are scenarios where mathematically speaking, you can construct machines more powerful than Turing machines, but they rely on physics that as far as we know can’t be instantiated.
Okay, I’ll bite on this one.
The very thing that distinguishes terminal goals is that you don’t “pick” them, you start out with them. They are the thing that gives the concept of “should” a meaning.
A key thing the orthogonality thesis is saying that it is perfectly possible to have any terminal goals, and that there’s no such thing as a “rational” set of terminal goals to have.
If you have terminal goals, then you may still need to spend a lot of time introspecting to figure out what they are. If you don’t have terminal goals, then the concept of “should”, and morality in general, cannot be made meaningful for you. People often consider themselves to be “somewhere in between”, where they’re not a perfect encoding of some unchangeable terminal values, but there is still a strong sense in which they want stuff for its own sake. I would consider nailing down exactly how these in-between states work to be part of agent foundations.
Oh that’s cool, thanks for sharing that. I didn’t realize there was some sort of (semi?)formal thesis about this.
So was this the high-level goal of Mere Goodness?
Hm, I’m not sure about Mere Goodness, I read the sequences soon after they were finished, so I don’t much remember which concepts were where. There is a sequence post titled Terminal Values and Instrumental Values, though it mostly seems to be emphasizing that both things exist and are different, saving the rest of the content for other posts.
Gotcha that makes sense. I read them like 10 years ago and also don’t remember but am going to skim through Mere Goodness again.
You don’t remember Eliezer arguing against the idea that terminal goals are arbitrary though right? My memory is that he pushes you to introspect and makes arguments like “Are you really sure that is what your terminal value actually is?” but never goes as far as saying that they can be correct or incorrect.
I do remember a bunch of content around that, yeah. And I would agree that terminal goals are arbitrary in the sense that they could be anything. But, for any given agent/organism/”thing that wants stuff”, there will be a fact-of-the-matter of what terminals goals got instantiated inside that thing.
There are also a few separate but related and possibly confusing facts;
The process of evolution will tend to produce organisms that have certain kinds of terminal goals instantiated inside them.
Empirically, humans happen to have a huge overlap in their terminal goals (including the terminal goal that other beings have their terminal goals satisfied).
If there are a bunch of roughly equally-capable agents around, then it maximizes your own utility (= terminal goals) to do a lot of game-theoretic cooperation with them.
I see. Thanks for clarifying.
Yeah I agree and think that those are important points.
That’s descriptive, not normative.
No, they are not the ultimate definition of what you should do. If there is any kind of objective morality, you should do that, not turn things into paperclips. And following on from that, you should investigate whether there is an objective morality. Arbitrary subjective morality is the fallback position when there is no possibility of objective ethics.
You could object that you would not change your terminal values, but that’s normative, not descriptive. A perfect rational agent would not change its values, but humans aren’t perfect rational agents, and do change their values.
what’s the issue with a descriptive statement here? It doesn’t feel wrong to me so it would be nice if you can elaborate slightly.
Also, I never found objective morality to be a reasonable possibility (<1%), are you suggesting that it is quite possible (>5%) that objective morality exists, or just playing devil’s advocate here?
Any definition of what you should do has to be normative, because of the meaning of “should”. So you can’t adequately explain “should” using only a descriptive account.
In particular , accounts in terms of personal utility functions aren’t adequate to solve the traditional problems of ethics , because personal UFs are subjective , arbitrary and so on—objective morality can’t even be described within that framework.
What kind of reasoning are you using? If your reasoning is broken, the results you are getting are pretty meaningless.
I can say that your reasons for rejecting x are flawed without a believe in X. Isn’t the point of rationality to improve reasoning?
I don’t think I am ready to argue about “should”/descriptive/normative, so this is my view stated out without intent to justify it super rigorously. I already think there is no objective morality, no “should” (in its common usage), and both are in reality a socially constructed thing that will shift over time (relativism I think?, not sure). Any sentence like “You should do X” really just have a consequentialistic meaning of “If you have terminal value V (which the speaker assumed you have), then it is higher utility to do X.”
Also you probably missed that I was reading between the lines and feel like you believe in objective morality, so I was trying to get a quick check to see how different are we on the probabilities, for some estimations on inferential distance and other stuff, not really putting any argument on the table in the previous comment (This should explain why I said the things you quoted). You can totally reject the reasoning while believing the same result, I just want to have a check.
I have not tried to put the objective morality argument into words and it is half personal experiences and half pure internal thinking. It boils down to basically what you said about “objective morality can’t even be described within that framework” though; I never found any consistent model of objective morality with my observation of different cultures in the world, and physics as I have learned.
So which do you believe in? If morality is socially constructed, then what you should do is determined by society, not by your own terminal values. But according to subjectivism you “should” do whatever your terminal values say, which could easily be something anti social.
The two are both not-realism, but that does not make them the same.
You have hinted at an objection to universal morality: but that isn’t the same thing as realism or objectivism. Minimally, an objective truth is not a subjective truth, that is to say, it is not mind-dependent. Lack of mind dependence does not imply that objective truth needs to be the same everywhere, which is to say it does not imply universalism. Truths that are objective but not universal would be truths that vary with objective circumstances: that does not entail subjectivity, because subjectivity is mind dependence.
I like to use the analogy of big G and little g in physics. Big G is a universal constant, little g is the local acceleration due to gravity, and will vary from planet to planet (and, in a fine-grained way, at different points on the earths surface). But little g is perfectly objective, for all its lack of universality.
To give some examples that are actually about morality and how it is contextual:
A food-scarce society will develop rules about who can eat how much of which kind of food.
A society without birth control and close to Malthusian limits will develop restrictions on sexual behaviour, in order to prevent people being born who are doomed to starve, whereas a society with birth control can afford to be more liberal.
Using this three level framework, universal versus objective-but-local versus subjective, lack of universality does not imply subjectivity.
Anything could be justified that way, if anything can.
So how sure can you be that the framework (presumably meanign von Neumann rationality) is correct and relevant. Remember, vN didn’t say vNR could solve ethical issues.
People round here like to use vNR for anything and everything, but that’s just a subculture, not a proof of anything.
You probably gave me too much credit for how deep I have thought about morality. Still, I appreciate your effort in leading me to a higher resolution model. (Long reply will come when I have thought more about it)
Not really. What terminal goals/values are is basically the top level goals of a recursive search. In other words, terminal goals are a lot like axioms, where terminal goals are the first things you choose, and then generate recursive instrumental goals out of it.
Terminal goals are still changeable, but changing the terminal goals changes all other goals.
And yes, this quote is accurate re terminal goals/values:
Simulacra are 4 reasons why people say something:
because (they believe) it is true
because they want you to believe it is true (regardless of whether it is)
to fit in with their friends (who cares about this “truth” thing anyway? I am not a nerd)
dunno, people simply say these words in situations like this one, it doesn’t mean anything
Without this concept, it may seem that lying is the opposite of telling the truth.
With this concept, you see that the truth-tellers and the liars are more similar to each other than they are different. For example, both care about the underlying reality (whether to confirm it, or to deny it). As opposed to people who do not care about reality at all; who merely repeat the words that get socially rewarded, and who might as well talk in a foreign language they do not understand.
Hm, that’s straightforward enough. Thanks.
But in what sense are they levels? When I think of levels I think of things that are kinda building on top of one another. But this just seems like four separate reasons. Maybe they’re not actually levels?
That feels like a stretch. Yes, in order to lie you have to know what the underlying truth actually is, but it still feels to me like there’s a big and important difference between liars and truth tellers. I guess it’s hard to talk about in general though, it kinda depends on the situation.
Another question: was that last paragraph the sort of elevator pitch for why simulcra are cool/important? If not, would you mind taking a stab at it?
Disclaimer: I am trying to explain the system that someone else invented, to the degree I seem to understand it. I certainly could not reinvent the system.
That said, it seems to me useful to distinguish between people who know the factual truth and are lying about it, from people who see reality as just some kind of “social consensus”, from people who are merely mechanically saying the words without attaching any meaning of it.
Why levels rather than parallel things? It seems like there is a progression on how detached from reality one is. The liar accepts that objective reality exists, he is just lying about it. The social guy has a map of reality, he just doesn’t care whether it matches the territory, only whether his group approves of it. The populist politician probably doesn’t even have a coherent map, it’s just individual statements.
Can all statements be classified as one of the 4 levels, or are more options needed? It’s not my system; if I tried to reinvent it, I might distinguish between the “level 3” people who have one permanent identity (one social consensus they believe), and those who flexibly switch between multiple identities (believing in different social realities in different situations). The latter is still different from having no model at all, such as saying things that contradict each other (to the same audience) just because each statement separately sounds good.
Basically, I treat it as a fake framework, like Enneagram or MBTI. In some situations, it allows me to express complex ideas shortly. (“She is extraverted” = do not expect her to sit quietly and read books, when she has an opportunity to socialize instead. “He made a level-3 statement” = he just signalled his group membership, do not expect him to care whether his statements are technically true.) I am not trying to shoehorn all situations into the model. I actually rarely use this model at all—for me it is in the “insight porn” category (interesting to discuss, not really used for anything important).
How consistent are people at using specific levels? If I saw someone make a level X statement, should I expect their other statements to also be on the same level? On one hand, caring about reality vs not caring about reality, or having a coherent model vs just saying random words, seems like a preference / personality trait that I would expect to manifest in other situations too. On the other hand, there is a difference between near mode and far mode. I don’t know. Some people may be more flexible between the levels, others less so. Probably high risk of fundamental attribution error here—it can be easy to assume that someone used level 3 or level 4 because they are “that kind of person”, while they only used that level in given situation instrumentally (as in “if I do not care about something, I simply make level 3⁄4 statements”).
Yeah. I like the way antanaclasis phrased it as “how much of a change you would have to make to the mindset of a person at each level to get them to level 1”. And I think it’s important to point out that it’s just their current mindset. It’s not like saying “hey, words actually correspond to object-level reality” to a person who made a level three statement would be a new idea to them.
Ok this all makes sense to me. I feel basically exactly the same. Thank you. I feel satisfied about my understanding of simulacra levels now.
Yeah I agree with this as well.
In terms of similarity between telling the truth and lying, think about how much of a change you would have to make to the mindset of a person at each level to get them to level 1 (truth)
Level 2: they’re already thinking about world models, you just need to get them cooperate with you in seeking the truth rather than trying to manipulate you.
Level 3: you need to get them the idea of words as having some sort of correspondence with the actual world, rather than just as floating tribal signifiers. After doing that, you still have to make sure that they are focusing on the truth of those words, like the level 2 case.
Level 4: the hardest of them all; you need to get them the idea of words having any sort of meaning in the first place, rather than just being certain patterns of mouth movements that one does when it feels like the right time to do so. After doing that, you again still have the whole problem of making sure that they focus on truth instead of manipulation or tribal identity.
For a more detailed treatment of this, see Zvi’s https://thezvi.wordpress.com/2020/09/07/the-four-children-of-the-seder-as-the-simulacra-levels/
Ah I see. Thanks for explaining.
If the important thing about higher levels is not tracking the underlying reality, why not define the category in terms of that rather than a specific motive (fitting in with friends) which sometimes leads to not tracking reality?
People say & do lots of things to fit in, some of which involve saying true things (while tracking that they match reality) and some of which don’t have propositional content (e.g. “Yay X” or “Boo X”). And there are various reasons for people to say nonsense, besides trying to fit in.
>Occam’s razor. Is it saying anything other than
P(A) >= P(A & B)?Yes, this is the same as the argument for (the abstract importance of) Solomonoff Induction. (Though I guess you might not find it convincing.)
We have an intuitive sense that it’s simpler to say the world keeps existing when you turn your back on it. Likewise, it’s an intuitively simpler theory to say the laws of physics will continue to hold indefinitely, than to say the laws will hold up until February 12, 2023 at midnight Greenwich Mean Time. The law of probability which you cited only captures a small part of this idea, since it says that last theory has at least as much probability as the ‘simple’ version. We could add a rule saying nearly all of that probability accrues to the model where the laws keep holding, but what’s the general form of that rule?
Occam’s original formulation about not multiplying entities doesn’t help us much, as we could read this to say we shouldn’t assume the world keeps existing when unobserved. That’s the opposite of what we want. Newton’s version was arguably better, talking about “properties” which should be tentatively assumed to hold generally when we can’t find any evidence to the contrary, but then we could reasonably ask what “property” means.
SI comes from the idea that we should look for the minimum message length which predicts the data, and we could see SI in particular as an attempt to solve the grue-bleen problem. The “naturalized induction” school says this is technically still wrong, but it represents a major advance.
I’m not familiar with Solomonoff induction or minimum message length stuff either, sorry.
My first thought when I read the “world keeps existing” and “laws of physics keep holding” examples was to think that the conjunction rule covers it. Ie. something like
P(these are the laws of physics) > P(these are the laws of physics & 2/12/23 and onwards things will be different). But I guess that begs the question of why. Why is the B a new condition rather than the default assumption, and I guess Occam’s razor is saying something like “Yes, it should be a new condition. The default assumption should be that the laws keep working”.I’m struggling to understand how this generalizes though. Does it say that we should always assume that things will keep working the way they have been working?
I feel like that wouldn’t make sense. For the laws of physics we have evidence pointing towards them continuing to work. They worked yesterday, the day before that, the day before that, etc. But for new phenomena where we don’t have that evidence, I don’t see why we would make that assumption.
This probably isn’t a good example, but let’s say that ice cream was just recently invented and you are working as the first person taking orders. Your shop offers the flavors of chocolate, vanilla and strawberry. The first customer orders chocolate. Maybe Occam’s razor is saying that it’s simpler to assume that the next person will also order chocolate, and that people in general order chocolate? I think it’s more likely that they order chocolate than the other flavors, but we don’t have enough evidence to say that it’s significantly more likely.
And so more generally, what makes sense to me, is that how much we assume things will keep working the way they have been depends on the situation. Sometimes we have a lot of evidence that they will and so we feel confident that they will continue to. Other times we don’t.
I have a feeling at least some of what I said is wrong/misguided though.
While it’s arguably good for you to understand the confusion which led to it, you might want to actually just look up Solomonoff Induction now.
Solomonoff Induction has always felt a little intimidating but I see how it’s relevant so yeah, I’ll check it out at some point.
Why do you need a distinction between new information and default assumptions? Why are default assumptions p=1?
If you know something, then assigning it p=1 makes sense...but assumption isn’t knowledge.
There’s a flaw in the conjunctive argument, in that assumes no knowledge of the propositions. Ten highly probable propositions can conjoin to a high probability, whereas two propositions with a default 0.5 each result in 0.25.
But intuitions are just subjective flapdoodle, aren’t they? ;-)
The Selfish Gene is still a classic if you want an intro to evolution book. It doesn’t have much math, though.
Thanks! I don’t want to read any book-length things in the short-mid term but I added it to my Goodreads list for the longer term.
Thanks for this post. I’ve always had the impression that everyone around LW have been familiar with these concepts since they were kids and now know them by heart, while I’ve been struggling with some of these concepts for the longest time. It’s comforting to me that there are long time LWers who don’t necessarily fully understand all of these stuff either.
That’s awesome to hear! Before reading this comment I had a vague sense of “Maybe this’ll help people” but after reading it I have a very concrete sense of “Oh yes! This is exactly the thing I was hoping would happen.”
Double Crux
As far as I understand, the important part is working to find a crux such that if it is resolved would bring the participants into agreement. It’s a way of focusing a discussion on a more concrete operationalization of a disagreement rather than trading pieces of rhetoric that aren’t likely to change anything.
I think you probably understand what the double crux is (which isn’t especially novel). I’d guess CFAR doesn’t claim to have invented the double crux so much as come up with general methods/algorithms to actually go about more reliably finding them.
As a mathematical analogy, it’s easy to check if a * b = c but not necessarily easy to find a and b only given c. This is the basis for public key cryptography. In this analogy, c is the argument and finding a and b is like finding a double crux.
Attempts at concise answers to some of these in case you wanted them:
The big misconception that the simulcra post tries to correct is that, while GPT-N can simulate processes that are agentic, GPT-N is not agentic itself. Calling GPT-Ns “optimizers” in any sense of the word is the sort of basic category mistake that seems extremely silly in retrospect.
The hype is that you can send currency (and do a bunch of other cool things) to people over the internet without the consent of one or more state governments or corporations. All other hype is derived from this extremely neat property.
Yes, but I wouldn’t attempt to mathematize it myself. “A” and “B” are propositions. Occam’s razor say to anticipate hypotheses about the underlying processes that are producing the data you see that are less “complex” for some definition of “complexity” that probably involves something in conceptspace near solomonoff induction or something.
The big ideas I remember from that sequence is that, beyond those (true) concepts:
Evolution primarily selects on the gene level.
Evolution works at a specific pace that can be calculated with some very simple math.
It’s a simplified model of a computer that can be used (in principle) to run every computer program we have thus far identified. Systems are “turing complete” if they can be used to simulate turing machines (and therefore also all of our computer programs (so far)).
I think Adam is asking about the “simulacra levels” concept, not anything about GPT language models.
Yes, thanks for clarifying. https://www.lesswrong.com/tag/simulacrum-levels
There is more than one justification of Occam’s razor. The justification in terms P(A) >= P(A & B) has the almost unique advantage off relating directly to truth/probability. Solonmonoff induction doesn’t have that property (in any obvious way) , because SI’s contain programmes, not propositions.
This is a misconception itself. GPT is an agent that learns to be a good simulator, an “actor”. It doesn’t mean that the actor itself is not there—that would be physically incoherent. The strength of that agency is a different question, as well as the goals of that agent: might not actually go beyond the goal of “being a good actor” (and this is just what we want, presumably, as of now). But that goal probably should be there if we want the system to be robust. (How good progress RLHF does towards that goal is a different question still.) Compare: human actors do have goals beyond “being a good actor”, e.g., being a good citizen, a good child, etc.
Fortunately, a “pure simulator” is neither physically coherent nor would it be desirable if it was physically possible. Because we don’t want actors in the theatre when playing Nero actually just “become Nero”, we want them to “be actors” in the back of their minds. This is a part of acting as a profession and what good actors are paid for—being able to go in and out of roles.
It turns out that most discussions are not focusing on finding a crux. Having a discussion that does focus on finding the crux and not talking about other things is hard.
Yeah, I do find that to be true. So is the hype moreso “hey, here’s a common failure mode we found” instead of “hey, here’s a new technique we discovered”? Or maybe “hey, here’s a handy reference post and name for a concept you might already have some sense of”?
In my experience of attempting to train Double Crux in Dojo/LW meetups getting people to do it well is not easy.
I think you’re overestimating how much “common sense” is already in-the-water. Even among committed rationalists my experience is that sticking to crux-finding is really hard. I think the techniques aren’t reliably useful, but the default thing people do just doesn’t work at all.
BTW, I’m curious if you’ve read Keep Your Beliefs Cruxy, which I wrote in part to explain my motivations for using doublecrux in real life.
Hm, let me be more specific and then you can tell me if you agree or disagree.
I think it is a recognition vs recall thing where people are able to recognize but not recall. Ie. when you explain what cruxes are and why double cruxing is useful I think people would nod and say “yes, that makes sense” not “oh wow, that’s an interesting new concept”. They’re able to recognize. But in the moment it’s one of those things that most people don’t really think of on their own and utilize.
Maybe I’m wrong though and there is something to this idea of double cruxing that would evoke the latter response. If so I think that I personally don’t know what those aspects are.
I have not but I’m interested and will at least skim through it. Thanks for pointing it out. I didn’t know it existed.
I think I’d say all the ingredients in Double Crux already existed, but
a) it’s actually useful work simply to notice that the status quo doesn’t result in people steering towards cruxes, so you need some kind of innovation to help people steer that way
b) the specific operationalization of the Double Crux technique is a decent attempt at helping people steer that way, even if it’s not quite right, there have been innovations since then, and it’s sort of a training wheels you eventually move on from.
Also see Eli’s updated post on how he describes the doublecrux pattern: https://www.lesswrong.com/posts/hNztRARB52Riy36Kz/the-basic-double-crux-pattern
I see. It sounds like we agree then and there isn’t really anything important about it that I’m missing.
To tell whether or not you are missing something, someone would need to observe what you actually do when you try to do double crux.
For a high level look at quantum physics I’d recommend Something Deeply Hidden by Sean Carroll. I feel like I understand many worlds much better after reading it. If you like audiobooks this one is great too.
I don’t think I’m motivated enough to read something book-length on quantum physics, at least not in the next few years. Thanks for the recommendation though. If there was something blog-post-length that did a good job of communicating the big picture ideas I’d be interested in that.
I’d suggest reading something about contextual, a.k.a. quantum cognition (rationality, decision-making) instead. It’s important but entirely ignored on LessWrong. All decision theories pretend that the contextuality of decisions (values, inferences, etc.) is an annoyance that could be swept under the rug. But it couldn’t, and therefore the rationality theories that don’t account for this quantumness are fundamentally incomplete.
Please don’t just equate contextual cognition with “quantum cognition”, and contextuality with quantumness.
There are a variety of cognitive phenomena which are in some way contextual or context-dependent, and a number of theories of those phenomena. “Quantum cognition” has come to mean one such school of thought, one that is inspired by the math and sometimes the metaphysics of quantum theory.
(I’ll emphasize for people unfamiliar with this school of thought, the idea is e.g. that cognitive representations undergo transformations mathematically analogous to those of a quantum state vector, something which can be true even if quantum mechanics is irrelevant to the physical substrate of cognition.)
Possibly, hopefully, some of the ideas of the “quantum cognition school of thought” actually are on the right track. But: quantum math is not the only way to implement contextuality; not everything that is contextual deserves to be called quantum; and many of the cognitive phenomena that this school has explained via quantum-like models can be explained in much simpler ways.
You are right that quantum math is not the only way to represent intrinsic contextuality, but it might be the most fundamental, principled way because quantum theory “is increasingly viewed as a theory of the process of observation itself” (Fields et al. 2021). So, to familiarise oneself with any model of the phenomena, I’d recommend paying attention to quantum models.
Non-intrinsic contextuality, i.e., that which admits coherent Bayesian modelling, indeed doesn’t require fancy treatment (and could be subject to any Bayesian decision theory). In my comment, by “contextual” I meant “intrinsically contextual”, for brevity.
All these models are of strictly theoretical and engineering interest (building AI and aligning AI), anyway, not practical interest. Reading about contextual/quantum rationality is not the way to improve one’s own decisions, so I’m not afraid that anyone will misapply the theories of quantum rationality.
Via quant-ph at arxiv.org, one may regularly see papers trying to derive quantum theory from some new axiomatic or philosophical basis. One class of such approaches focuses on epistemology. In my experience this falls into two types. Either the proposed new foundations involve a repackaging of some familiar quantum weirdness, such as the uncertainty principle; or, the proposed new foundations aren’t “weird” but also don’t actually give you quantum theory. (I believe the latter are motivated by the desire that quantum theory not be weird, i.e. not require reality to possess some strange new feature.)
I don’t see anything in the references of Fields et al 2021 which escapes this dichotomy, and the bland dictum that quantum theory is a “theory of the process of observation” definitely sounds like the second type of new foundation, that always fails for not being weird enough.
On Blockchain, you’re right about Blockchain not being worth all that much, and this is a great case where the technology simply doesn’t work for fundamental reasons, but hype used to dominate, so a selection effect occured.
The only advantage is doing crimes, and even that is inefficient, and virtually every other advantage wilts once we actually look at it.
Would you consider perhaps offering a few stated advantages and then walking through how they fail?
It doesn’t work (as advertised, here) anyway.
There are three, intertwined, problems . One is the problem of realistic reference: how the candidate program in an SI makes statements about the external world -- that is realism in the sense of scientific realism, Scientific realism contrasts with scientific instrumentalism which Solomonoffs version of SI can obviously do.
The second is the issue of whether and in what sense an SI is calculating probabilities.
The third is the problem of how SI fulfils its basic role of predicting a series of observations.
On the face of it, Solomonoff Inductors contain computer programmes, not explanations, not hypotheses and not descriptions. (I am grouping explanations, hypotheses and beliefs as things which have a semantic interpretation, which say something about reality . In particular, physics has a semantic interpretation in a way that maths does not.)
The Yukdowskian version of Solomonoff switches from talking about programs to talking about hypotheses as if they are obviously equivalent. Is it obvious? There’s a vague and loose sense in which physical theories “are” maths, and computer programs “are” maths, and so on. But there are many difficulties in the details. Neither mathematical equations not computer programmes contain straightforward ontological assertions like “electrons exist”. The question of how to interpret physical equations is difficult and vexed. And a Solomonoff inductor contains programmes, not typical physics equations. whatever problems there are in interpreting maths ontologically are compounded when you have the additional stage of inferring maths from programmes.
In physics, the meanings of the symbols taught to students, rather than being discovered in the maths. Students are taught the in f=ma, f is force, is mass and a is acceleration. The equation itself , as pure maths, does not determine the meaning. For instance it has the same mathematical form as P=IV, which “means” something different. Physics and maths are not the same subject, and the fact that physics has a real-world semantics is one of the differences.
Similarly, the instructions in a programme have semantics related to programme operations, but not to the outside world. The issue is obscured by thinking in terms of source code. Source code often has meaningful symbol names , such as MASS or GRAVITY...but that’s to make it comprehensible to other programmers. The symbol names have no effect on the function and could be mangled into something meaningless but unique. And a SI executes machine code anyway..otherwise , you can’t meaningfully compare programme lengths. Note how the process of backtracking from machine code to meaningful source code is a difficult one. Programmers use meaningful symbols because you can’t easily figure out what real world problem a piece of machine code is solving from its function. One number is added to another..what does that mean? What do the quantifies represent?
Well, maybe programmes-as-descriptions doesn’t work on the basis that individual Al symbols or instructions have meanings in the way that natural language words do. Maybe the programme as a whole expresses a mathematician structure as a whole. But that makes the whole situation worse because it adds an extra step , the step of going from code to maths, to the existing problem of going from maths to ontology.
The process of reading ontological models from maths is not formal or algorithmic. It can’t be asserted that SI is the best formal epistemology we have and also that it is capable of automating scientific realism. Inasmuch as it is realistic , the step from formalism to realistic interpretation depends on human interpretation, and so is not formal. And if it SI is purely formal, it is not realistic.
But code already is maths, surely? In physics the fundamental equations are on a higher abstraction level than a calculation: generally need to be ” solved” for some set of circumstances, to obtain a more concrete equation you can calculate with. To get back to what would normally be considered a mathematical structure, you would have to reverse the original process. If you succeed in doing that, then SI is as good or bad as physics...remember, that physics still needs ontological interpretation. If you don’t succeed in doing that.. which you you well might. since there is no algorithm reliable method for doing so...then SI is strictly worse that ordinary science, since it has an extra step of translation from calculation to mathematical structure, in addition to the standard step of translation from mathematical structure to ontology.
Maybe an SI could work off source code. It’s obviously the case that some kinds of source code imply an ontology. The problem now is that, to be unbiased, would have to range over every kind of ontology-implying source code. If it just ran an OO language , then it will be biased towards an object ontology , and unable to express other ontologies. If you just ran a procedural language, it would be biased towards process ontology.
What it is also unclear is whether the criterion that SI uses to sort programmes has anything to do with probability, truth or correspondence.
Why use occams razor at all? If we were only interested in empirical adequacy, the ability to make accurate predictions, simplicity only buys the ability to make predictions with fewer calculations. But SI, according to Yudkowsky, but not Solomonoff, doesn’t just make predictions, it tells you you true facts about the world .
If you are using a simplicity criterion to decide between theories that already known to be predictive , as in Solomonoff induction, then simplicity doesn’t buy you any extra predictiveness, so the extra factor it buys you is presumably truth.
There are multiple simplicity criteria, but not multiple truths. So you need the right simplicity criterion. If you have a conceptually valid simplicity critetion, and you formalise it, then thats as good as it gets, you’ve ticked all the boxes. If you formalise a simplicity criterion that has no known relationship to truth, then you haven’t achieved anything. So it is not enough to say that Solomonoff is “the” formal standard of simplicity. There are any number of ways of conceptualising simplicity, and you need the right one.
Consider this exchange, from “A semi technical introduction to Solomonoff Induction” .
“ASHLEY: Uh, but you didn’t actually use the notion of computational simplicity to get that conclusion; you just required that the supply of probability mass is finite and the supply of potential complications is infinite. Any way of counting discrete complications would imply that conclusion, even if it went by surface wheels and gears.
BLAINE: Well, maybe. But it so happens that Yudkowsky did invent or reinvent that argument after pondering Solomonoff induction, and if it predates him (or Solomonoff) then Yudkowsky doesn’t know the source. Concrete inspiration for simplified arguments is also a credit to a theory, especially if the simplified argument didn’t exist before that.
ASHLEY: Fair enough.”
I think Ashley deserves an answer to “the objection “[a]ny way of counting discrete complications would imply that conclusion, even if it went by surface wheels and gears”, not a claim about who invented what first!
Or you could write a theory in English, and count the number of letters...that’s formal. But what has it to do with truth and reality. But what, equally, does a count of machine code instructions have to do with truth or probability?
There is one interpretation of Occam’s razor, the epistemic interpretation of it, that has the required properties. If you consider a theory as a conjunction if propositions having a probability less than one, then all else being equal, a higher count of propositions will be less probable. We already know that propositions are truth-apt , that they are capable of expressing something about the world, and it is reasonable to treat them probabilistically.
So that is the right simplicity criterion...except that it had nothing to do with SI.
SI is trying to process an infinite list of programmes in a finite time. To achieve this, as a matter of shear practicality, processes shorter candidate programmes first.
The Yudkowskian claim is that programme length is probability. One immediate concern is that SIs don’t use programme length because it’s probability, they use it to work at all.
If you have an infinite number of hypotheses to consider, then you need to to neglect the longer or more complex ones in order to be able to terminate at all. But that’s only a condition of termination, not of truth
In addition, there is more than one way of doing, whereas we would expect the true, corresponding map of reality Programme length clearly isn’t probability in the frequentist sense, the probability of an event occurring. It is presumably a level of rational credibility that the hypothesis represented by the program models reality.
It’s uncontentious that hypotheses and beliefs have a probability of being true, of succeeding in corresponding. But what does it mean to say that one programme is more probable than another? The criterion SI uses s that it is short. A shorter programme is deemed more probable.
(Information theory talks about of probability and information content—but in the right sense?)
A shorter bitstring is more likely to be found in a random sequence, but what has that to do with constructing a true model of the universe?
Does Solomonoff induction prove that Many Worlds is simpler than Copenhagen?
In 2012, private_messaging made this argument against the claim that SI would prove MWI.
https://www.lesswrong.com/posts/6Lg8RWL9pEvoAeEvr/raising-safety-consciousness-among-agi-researchers?commentId=QtwqxiwnCSksCc566
If the program running the SWE outputs information about all worlds on a single output tape, they are going to have to be concatenated or interleaved somehow. Which means that to make use of the information, you gave to identify the subset of bits relating to your world. That’s extra complexity which isn’t accounted for because it’s being done by hand, as it were..
The objection, made by Sawin is that a computational model of MWI is only more complex in space, which , for the purposes SI doesn’t count. But that misses the point:an SI isn’t just an ontological model, it has to match empirical data as well.
In fact, if you discount the complexity of the process by which one observer picks out their observations from a morass of data, MWI isn’t the preferred ontology. The easisest way of generating data that contains any substring is a PRNG, not MWI. If you count the process by which one observer picks out their observations from a morass of data as having zero cost, you basically ending up proving that “everything random” is the simplest explanation.