I’m writing a book about epistemology. It’s about The Problem of the Criterion, why it’s important, and what it has to tell us about how we approach knowing the truth.
I’ve also written a lot about AI safety. Some of the more interesting stuff can be found at the site of my currently-dormant AI safety org, PAISRI.
I still go back to saying the most important thing that you’ve ever written, for me personally, was CFAI section 2.2, where you argued against anthropomorphism in AI. That was the thing that cracked my mind loose and set it on its journey without feeling trapped by traditional views of rationality, intelligence, or wisdom. Not that this appeared on OB as far as I can remember, but I think it’s important enough that it should appear somewhere in the book.
Nice post. Working up from the basics is always important. For an analogy to math, I’m always surprised how many people who do highfalutin things with partial differential equations and systems of nonlinear equations can make basic mistakes in their reasoning because they consider themselves beyond set theory or even topology. Rationality is one of those domains where I find it’s best to work up from first principles whenever possible, accepting shortcuts only as practical necessities.
For an additional fallacy, we can also note that the quote gives India motive. To the best of my knowledge the government of India, at least in its modern form, has not peddled its nation in any official capacity as a source of great spiritual knowledge. Mote tot he point, the quote means to say that India is a nation claimed by people who present themselves as spiritually knowledgeable to be a source of spiritual knowledge, but of course that knocks even more wind out of this quote’s sails, hence why such phrasing was probably not even considered.
I think the most frequently useful thing I’ve learned is not in the content, but in the approach. The way of breaking things down, pulling together the themes, and looking at things in a way to makes the problems that seem to be there disappear. I guess that’s what’s been most frequently useful: when you get stuck, trying to see if you’re looking at a problem in the wrong way, since often even if you’re not it gets you unstuck.
I like this idea. When we have to stretch too far to look for an explanation of a trait based only on that trait’s effect on differential reproduction, it may be because there is no such explanation. Plenty of traits are the result of side effects that did not affect reproduction, and others may be cultural.
This idea has just what we need: it fits the experience, doesn’t seem to affect reproduction, and is a side effect of sexually selected traits. When you add in a cultural component that may amplify or suppress this feeling of sympathy, you have what looks like a good explanation with no “just so”s necessary.
Maybe, but what about when those children discover that they are outside the norm? I’d imagine they might even be more likely to favor underdogs once they realize that they share the commonality of standing against the norm in some fashion.
This is a nice try, but it still lacks what is most important in any evolutionary explanation: the source of differential reproduction. Looking at the comments on both posts I still don’t see any reason to believe that the underdog bias, which I do think exists, resulted in differential reproduction. You push this off on the near-far distinction, so the question remains: why should the near/far distinction have evolved?
Ha ha ha. Classic.
This is one of those stories you can show to would-be rationalists that will make them both laugh and think about probability. Well done.
If it’s allowed, it certainly violates the spirit of E-Prime. As I understand it, part of the point is to attach such statements to an actor. Why might something be useful? Who or what makes it useful? Maybe the correct statement in E-Prime would be “I thought you might make good use of this”.
Clearer, but I remain unconvinced. The value of this signal, as you present it, seems to provide so little real benefit that I can’t distinguish it from the noise of random mutation.
This is a very interesting post and regardless of any errors some may feel it has, I think it has the right idea. One of the problems with attributing failure to akrasia is that it doesn’t always feel like you would have made the right choice if you had will power. For an example from my own life, sometimes rather than working on things that I do want to work on, I play video games, often rationalizing to myself that I just don’t have the mental energy to do the work. But, really, it’s more that I want to play the games, maybe more than do the work, than that I failed to have the will power to do the work (although I wish I had had the will power to overcome my desire to play the games and do the work).
You’ve definitely given us more precise language to use when discussing “failures” of will power.
Personally, the idea of the crisis of faith seems to me like a symptom of bad thinking. If you hold an idea so tight that you need a crisis of faith, then maybe the real problem is that you have any belief held so strongly that it takes that much effort to overcome it. Aside from human-level stupidity in-the-moment of discussion (e.g. supporting a position despite contrary evidence because it’s the side you picked at the start), a master rationalist shouldn’t need to have a crisis of faith.
I had the last thing that felt like a crisis of faith to me when I was 18. Since then I haven’t been able to hold onto any belief so tightly that any normal amount of rationality effort hasn’t been able to change my mind.
Fair enough. First, master rationalist is probably pushing it a bit too far in what’s required; rather, you just need to work towards a “mind like water” state, flowing in whichever way the evidence directs it, and once you get close enough to that state the need for a true crisis of faith should disappear.
As for myself and and others who might make a claim to have no crises of faith, it’s a fair question to ask if we’re simply not seeing them. It’s entirely possible that there are some beliefs that I have that I am unaware are so tightly held that I don’t even see them as beliefs, but as truths about the world. However, I have had many experiences which you might identify as a crisis of faith (although I didn’t handle any of them with much in the way of rationality, and it was luck as much as anything else that I was dumped out in the state of mind that I was in), and I have not since encountered anything that has led me to a crisis of faith. Given the huge amount I have learned since that last crisis of faith, I consider the odds of me having another one low.
There is one caveat I should mention, though: this whole issue may come down to a matter of perspective. To me, it’s not a real crisis of faith unless you have to change your entire world view. Up until my last crisis of faith, I spent a lot of time thinking about how everything fit together in the universe. But then I had the realization, which came over me like a wave but soaked in to me very slowly, that all that really mattered was the evidence. So compared to that experience, nothing else has felt worthy of being called a crisis of faith.
That said, the method that Eliezer outlines seems to be a good one to follow. I have applied several of the techniques described with good success. So now that I think of it, maybe it is all just a matter of differences in what we really consider to be a crisis.
Interesting thoughts.
I have tried in the past to follow the mathematical definitions I learned regarding rationality. Unfortunately they haven’t always served well in discussions. They are:
Rationality: the application of Bayes rule
Well-informed: the possession of evidence with correct confidence
Prudence: the result of combining rationality with well-informed-ness
But, like I said, this doesn’t work well in normal speech because each of those words doesn’t have such a clean and precise meaning in the natural language. Instead, if we must choose words so that we may speak with mathematical precision, I propose we don’t spend time choosing the right word to impose our definition on and instead follow a simple rule that has helped many software developers avoid time wasting conflicts: a construct will be given the worst name someone thinks up. The only way you can change the name of a construct is to propose a worse name. It stops the fighting about choosing the right word and let’s us all get on with the discussion.
Not sure how well that will work here, but I thought it was worth mentioning.
Legacy of Ashes: The History of the CIA by Tim Weiner
Summary: A history of the CIA from its origins in US military intelligence units up to right after 9/11. Based only on primary sources (released CIA documents, interviews with insides, etc.; no rumors or unconfirmed theories), Weiner shows how the CIA has routinely failed to meet its goals and has often made bad situations worse. All of this despite filling the organization with intelligent men ready to die to defend their country by providing military intelligence to the president and engaging in covert operations.
Lessons: There are many small lessons you can learn from the episodes of the history, but the big lesson you’ll find is that when an organization has a culture of secrecy, to the point that not even the people in charge with the highest clearance have access to all the secrets, you open the door for mismanagement and abuse. But because the CIA was designed to be opaque, even to the president, it had no accountability or oversight, became inbred with bad ideas, and began to operate independently of US policy by receiving money from Congress and hiding its operations, even from units within itself. The CIA is a case study of the tremendous damage that a few men with power, money, and secrecy can do, where only luck prevented what could have been disastrous consequences.
I remember liking the history in the book, but I think you have a point there. However, remember that we’re not all so lucky as to live in a society that is so different from the ones described, so I think this book has real currency for some people (just probably not the ones who can get a copy to read).
His other big book, Guns, Germs, and Steel, teaches a related lesson about failure: if you aren’t lucky enough to have the necessary resources, no matter how hard you try, your success will be limited.
Lest we be too, too hard on these folks, I should add that, as Eli mentions, there is some value in talking to someone about your problems. I have seen a therapist on occasion who mainly just listens and then tries to advise based on human experience. No lying on the couch, no ink blots, no hypnosis. Just a neutral party to talk to who will keep what I say in confidence. Sort of like Catholic confession, but without all the Hail Marys.
Also, just in case anyone gets confused here (I’m sure someone will), psychotherapist are not the same thing as psychiatrist: the latter are actual doctors who can prescribe therapies, be they drug or other. They may not all be great either, but at least one that I’ve seen really knows his stuff.
Although it is designed as part of libertarian recruitment and is used to start discussions with people about freedoms they already support and then “draw them in” by gradually exposing them to other ideas, the reality of the data is that not many people score libertarian (the Web data isn’t very accurate because you get a lot more libertarians visiting the site).
In my younger years I did some tabling for the Libertarian Party, giving the quiz, letting them place a dot on a blow up of the quiz grid to let them mark how they scored and compare with others. And I have to say, in all that time, I did not encounter one person out of several hundred that scored libertarian who was not already a card-carrying member. In fact, if anything, most people score down in the authoritarian range.
This is just the data set I’ve collected, though. Maybe there is a better one out there than the one you can find from the online version of the quiz.
Reading the comments so far, I think Peter wasn’t as clear as he had hoped (or this is all jumping to disagree too quickly). As I see it, the point is that an epistemic rationalist, a completely abstract mathematical construct to the best of our knowledge of the physical world, would make a choice that is at odds with an instrumental rationalist, i.e. a real person who’s trying to win in real life. Having bounded resources, there is some threshold below which a physically existing rationalist will treat probabilities as equivalent to zero, i.e. will choose not to expend any resources on preparing for such a situation.
A meteorite makes a bad example because it’s easy to imagine it happening. Suppose we consider the probability of a three layer chocolate cake spontaneously appearing in the passenger seat of our car during the drive home this afternoon. Yes, the probability must be nonzero, but it’s so small as to not be worth considering. All those events with probabilities so small they aren’t worth any resources are the ones you never even think about, so they are equivalent to having a probability of zero for the bounded rationalist.
For me this is a tough question since I’ve been reading your stuff for nearly 10 years now, but thinking of only OB I’d have to say it was the quantum physics stuff, but only because I had encountered essentially everything else in one form or another already, so your writing was just refining the way of presenting what I had already generally learned from you.