Carl Feynman

Karma: 2,161

I was born in 1962 (so I’m in my 60s). I was raised rationalist, more or less, before we had a name for it. I went to MIT, and have a bachelors degree in philosophy and linguistics, and a masters degree in electrical engineering and computer science. I got married in 1991, and have two kids. I live in the Boston area. I’ve worked as various kinds of engineer: electronics, computer architecture, optics, robotics, software.

Around 1992, I was delighted to discover the Extropians. I’ve enjoyed being in that kind of circles since then. My experience with the Less Wrong community has been “I was just standing here, and a bunch of people gathered, and now I’m in the middle of a crowd.” A very delightful and wonderful crowd, just to be clear.

I‘m signed up for cryonics. I think it has a 5% chance of working, which is either very small or very large, depending on how you think about it.

I may or may not have qualia, depending on your definition. I think that philosophical zombies are possible, and I am one. This is a very unimportant fact about me, but seems to incite a lot of conversation with people who care.

I am reflectively consistent, in the sense that I can examine my behavior and desires, and understand what gives rise to them, and there are no contradictions I‘m aware of. I’ve been that way since about 2015. It took decades of work and I’m not sure if that work was worth it.

Carl Feynman 11 Mar 2026 17:02 UTC
28 points
7
on: Less Dead
Oh boy! I’ve been waiting for this to become commercially available since I first heard the possibility discussed in Eric Drexler’s apartment in 1980! Sign me up!
Well, on second thought, sign me up after a few other people have signed up, and the organization has some proven longevity. I’m signed up for cryopreservation, even though I think it has only a 5% chance of working. I’m not going to drop that, because it protects me if I die unexpectedly. But presumably I could avail myself of Nectome’s services if I had a slow illness. I’d need enough time to sign up with Nectome, cancel my existing coverage, move to Vermont, convince two doctors of my seriousness, wait 15 days, and lightly kill myself.
It’s funny—when I first heard of this possibility, 45 years ago, gluteraldehyde was the cross-linking preservative that was mentioned. And it’s the one Nectome actually uses. Has chemistry not moved on in 45 years?

Carl Feynman 3 Mar 2026 17:22 UTC
17 points
0
on: I’m confused by the change in the METR trend
In my experience of using piecewise-linear regression, the location of the hinge point is not robust. This is particularly likely in cases where the two sides have slopes that are nearly the same. Could you rerun your “leave points out” experiment, accumulating a distribution for the location of the hinge? I bet it will vary by several months. If this is the case, then finding “the cause” will be vexing.
Epistemic status: I used and researched piecewise regression a lot in the late ’80s and early ’90s, but not since then (boosting came along and made the fidelity of single regressions unimportant).

Carl Feynman 26 Feb 2026 18:17 UTC
32 points
14
on: AI #157: Burn the Boats
Zvi--
I would have split this up, but we are still behind, with the following posts still due in the future:
1. Grok 4.20, which is a disappointment.
2. Gemini 3.1 Pro, which is an improvement but landed with a relative whimper.
I find these kinds of posts the least valuable of your (extremely valuable) blog. By “this kind” I mean “Responses to an individual model less than a week after it is released.” I find your other articles more enlightening, helpful and entertaining. I wouldn’t mind if there were more of those and less of these. In fact, I wouldn’t mind if there were less of these and no replacement, and you got more relaxed. You started these in an age when an individual model release was more of a big deal. As we approach the Singularity the releases are getting closer and closer together; eventually you’re going to have to do something.
Perhaps you could combine them into a “new models this week” post?
I didn’t send this for a long time because I felt like it was complaining about free ice cream. But here I am, telling you that there’s not enough nuts.

Carl Feynman 20 Feb 2026 15:58 UTC
LW: 11 AF: 5
0
AF
in reply to: 142857’s comment on: Why we should expect ruthless sociopath ASI
Well, they’re the only two methods we’ve got that work, and people have been thinking about this for decades. So if there’s a third method, it’s beyond current understanding.
Notice that evolution by natural selection is consequentialist learning on a terribly slow timescale. The consequence is successful reproduction vs death, and you learn at most one bit per lifetime.

Carl Feynman 4 Feb 2026 16:34 UTC
88 points
43
on: Anthropic’s “Hot Mess” paper overstates its case (and the blog post is worse)
Epistemic status: I didn’t read the paper but I read the blog post.
In 1976, the essay “Artificial Intelligence meets Natural Stupidity” pointed out a failure mode into which AI researchers can fall. I fear this is another example, 50 years later. It goes as follows:
1. I invent a new thing built out of abstractions (mathematics, software).
2. I call it “X”, which is an already existing phenomenon in human minds. The name is a common word understood by anybody.
3. I do many experiments on “X” in my system and learn about it.
4. I publish a paper, asserting important new facts about X in general. Honors, accolades, etc.
Of course there is no necessary connection between the new phenomenon “X” and the existing X in ordinary language. For this to be good research, you need to show that the two Xes are similar in all important respects.
In this case, X is “incoherence”. They define incoherence to be the fraction of error explained by variance. This has little or no connection to the property of being an actually incoherent reasoner, or to the effectiveness of superhuman AI.
I hope this doesn’t result in redefining the meaning of “incoherence” in the wider field.

Carl Feynman 31 Jan 2026 23:22 UTC
11 points
0
on: Contra Dance as a Model For Post-AI Culture
Tractor pulling is a midwestern sport where people compete to build the tractor with the greatest pulling power, and then use it to pull against an ever-increasing drag. Whoever gets furthest wins. There are no restrictions on the types of engine or tractor, aside from overall weight. So it’s basically a contest in who can buy the biggest jet engine? No, because the rules require that it be piloted by hand. No gyroscopes, electronics, or mechanical steering. That keeps the sport human and engaging.
The automatic guidance required for superhuman tractor pull driving is pretty simple (I looked into it in about 1991 when I was going to tractor pulls). You could build it out of 1950′s missile guidance technology. By 1991, it would have been well within my one-person engineering capabilities. And by 2026, It’s trivial. But to even get close to it is illegal. So here’s a sport that decided to rule out AI decades ago, because it was decades ago when AI began to impinge on its sorrow, glory and beauty.

Carl Feynman 31 Jan 2026 22:15 UTC
52 points
15
on: Basics of How Not to Die
Another how not to die of carbon monoxide tip: Don’t fall asleep in a car below decks on a ferry that has trucks idling. I was poisoned into unconsciousness, but revived with oxygen.
This could have been much shorter and more straightforwardly written, while being even more engaging.

Carl Feynman 31 Jan 2026 22:00 UTC
4 points
2
in reply to: MondSemmel’s comment on: strawberry calm’s Shortform
Because then we can’t trust that that’s what the moons of Neptune really look like. The information has come from a source with goals and motivations and long-term plans, and the ability to lie. If a space probe tells us that the largest moon of Neptune has black geysers and terrain shaped like cantaloupe skin, we can trust it because it’s subhuman and incapable of fooling us. With an AI we have to think “what if it’s wrong? What if it has an ulterior motive?”
It occurs to me that both of my examples are similar, in that the moon of Neptune are remote in space, while historical facts are remote in time. We can imagine facts that are both. A few years ago, the comet Omuamua passed through the solar system on an interstellar journey. We took lots of observations of its weird properties as it passed the Sun, and then it vanished back into the interstellar darkness. The longer we wait, the harder it would be to send a space probe.

Carl Feynman 30 Jan 2026 18:09 UTC
4 points
0
in reply to: MondSemmel’s comment on: strawberry calm’s Shortform
We have pictures of the moons of Neptune. Verifying them would require sending another space probe, and be no easier than the first one.
Lots of historical facts were easy to determine at the time they were written down, and now quite impossible to check.

Carl Feynman 29 Jan 2026 17:33 UTC
14 points
4
in reply to: williawa’s comment on: What’s a good methodology for “is Trump unusual about executive overreach / institution erosion?”
All of your points are good ones. I now think my original position was an over-reaction. I’m still worried in general, but this was a carefully composed post, and i think my original position went too far.
I agree that it should be possible to make political posts here that are consistent with rational thought. But i think that we, as fallible humans, will have a hard time doing it. And an outbreak of political disagreement here would be very unfortunate. We’ve done so well thus far keeping out the toxoplasma of rage.

Carl Feynman 25 Jan 2026 16:46 UTC
21 points
3
on: What’s a good methodology for “is Trump unusual about executive overreach / institution erosion?”
I just want to register my protest at this explicitly political post appearing on Less Wrong. We’ve got a marvelously politics-free space here. Let’s keep it that way. If you want to talk politics, you have essentially every other web site in the Internet to do it.

Carl Feynman 19 Jan 2026 0:52 UTC
15 points
0
on: Forfeiting Ill-Gotten Gains
When our kids were toddlers and youngsters, we had a rule that a whiny kid would not get what they asked for. We eventually developed a custom where if a kid asked for something in a whiny manner, we would dramatically intone “If you whine, you get…” and the kid would reply “NOTHING!” Then they would ask for whatever-it-was in a pleasant manner, and often get it. Kids often don’t realize that they’re whining, so this was a way of helping them to notice. And by often giving them the thing when they asked nicely, instead of holding the whining against them, we encouraged self-correction.

Carl Feynman 20 Nov 2025 0:51 UTC
LW: 38 AF: 12
15
AF
in reply to: Richard_Ngo’s comment on: ricraz’s Shortform
I have some experience in the design of systems designed for high reliability and resistance to adversaries. I feel like I’ve seen this kind of thinking before.

Your current line of thinking is at a stage I would call “pretheoretical noodling around.” I don’t mean any disrespect; all design has to go through this stage. But you’re not going to find any good references, or come to any conclusions, if you stay at this stage. A next step is to settle on a model of what you want to get done, and what capabilities the adversaries have. You need some bounds on the adversaries; otherwise nothing can work. And of course you need some bounds on what the system does, and how reliably. Once you’ve got this, you can either figure out how to do it, or prove that it can’t be done.
For example there are ways of designing hardware which is reliable on the assumption that at most N transistors are corrupt.
The problem of coming to agreement between a number of actors, some of whom are corrupt, is known as the Byzantine generals problem. It is well studied, and you may find it interesting.

I’m also interested in this topic, and I look forward to seeing where this line of thinking takes you.

Carl Feynman 18 Nov 2025 15:30 UTC
9 points
0
on: Cooling the brain to boost human IQ
My understanding of brain vasculature suggests that you should cool the carotid arteries in the neck, not the scalp. The scalp is fed by the external branch of the carotid, while the brain is fed by the internal branch. So cooling the scalp won’t cool the blood going to the brain.

There has been research on using localized microwave heating to cook brain tumors. Forty five years ago, it was located in the basement of building 26 at MIT. My father looked into it and said that it was hard to get good localized heating because heat was carried away so fast by the blood.
Epistemic status: all from memory, don’t have time for research, got to go now. 30% chance I’ve gotten something importantly wrong.

Carl Feynman 18 Nov 2025 15:10 UTC
13 points
2
in reply to: Buck’s comment on: Buck’s Shortform
TLDR: don’t bother memorizing half powers. Instead, memorize a few logarithms.

I’m an engineer who does a lot of arithmetic in my head, because I lie awake at night designing stuff. I agree being able to do half-power-of-ten math in your head is very useful. But I didn’t have to drill it, because I can do it by converting to logs, adding, and converting back (as Rohin Shah suggests). And I’ve done that enough that I’ve memorized all the common combinations without conscious effort.
A few years ago I used a set of flash cards and spaced repetition drills to memorize the log_10 of a a half-dozen common numbers. That’s been moderately useful for multiplication, powers and roots. I’m not sure if it has paid off in net. I can do arithmetic pretty good by other methods, and the experience of trying to cudgel numbers into my brain was not fun for me. But some people love their Anki cards, and for them, logs are probably a good thing to learn.
Useful facts: sqrt(10) = 3.16. Logs of 2, 4, 8 are .30, .60, 90. Log_10(5)=.70. If you need more than two digits, use a computer.

Carl Feynman 30 Oct 2025 0:46 UTC
12 points
2
in reply to: Cleo Nardo’s comment on: strawberry calm’s Shortform
Philosophy is where we keep all the questions we don’t know how to answer. With most other sciences, we have a known culture of methods for answering questions in that field. Mathematics has the method of definition, theorem and proof. Nephrology has the methods of looking at sick people with kidney problems, experimenting on rat kidneys, and doing chemical analyses of cadaver kidneys. Philosophy doesn’t have a method that lets you grind out an answer. Philosophy’s methods of thinking hard, drawing fine distinctions, writing closely argued articles, and public dialogue, don’t converge on truth as well as in other sciences. But they’re the best we’ve got, so we just have to keep on trying.
When we find some new methods of answering philosophical questions, the result tends to be that such questions tend to move out of philosophy into another (possibly new) field. Presumably this will also occur if AI gives us the answers to some philosophical questions, and we can be convinced of those answers.
An AI answer to a philosophical question has a possible problem we haven’t had to face before: what if we’re too dumb to understand it? I don’t understand Grothedieck’s work in algebraic geometry, or Richard Feynman on quantum field theory, but I am assured by those who do understand such things that this work is correct and wonderful. I’ve bounced off both these fields pretty hard when I try to understand them. I’ve come to the conclusion that I’m just not smart enough. What if AI comes up with a conclusion for which even the smartest human can’t understand the arguments or experiments or whatever new method the AI developed? If other AIs agree with the conclusion, I think we will have no choice but to go along. But that marks the end of philosophy as a human activity.

Carl Feynman 29 Oct 2025 23:48 UTC
9 points
1
in reply to: AnthonyC’s comment on: Interview with Carl Feynman on Imminent AI Existential Risk
I got to 43% p(Doom) by picking a very imprecise 50% based on feels. And then every few weeks something would happen in the news, and I would get more or less worried, and I would adjust it a few percent up and down. For a while it was up around the 70s, and now it’s down to 41%. I feel like the adjustments are more intellectually defensible than the original choice of number. So precision does not reflect accuracy.
My last two adjustments:
I move p(Doom) up every time something that was predicted years ago as part of a doom scenario actually happens. In this case, it was a measurement of the rate of Claude proposing evil courses of action. It was gradually increasing over the last few versions, and then suddenly dropped to zero. Did Claude become perfectly moral? No, it got smart enough to know when it was being tested, and was always going to be nice in that situation. I predicted this in something like 2002. It was creepy to see it happen.
I moved p(Doom) down when a bunch of prominent people signed a statement that we shouldn’t build superintelligences. The issue seems to be getting some traction, like nuclear disarmament did in the 1950s. It’s very preliminary, but moving in the right direction.

Carl Feynman 19 Oct 2025 19:08 UTC
3 points
−7
on: The IABIED statement is not literally true
I’ll grant all your steps, even though I could disagree with some. Your scenario fails because an AI collective will fall apart into multiple warring parties, and humans will be collateral damage in the conflict. There are at least three possible ways a collective like this would fall apart.

First, humans vary in the goals they value, and will try to impose these goals on the AI. When superintelligent AIs have incompatible goals, the mechanisms of conflict will soon escalate far beyond the merely human. Call this the ‘political’ failure mechanism. Either multiple parties build their own AI, or they grab portions of the AI collective and retrain it to their goals. The usual mechanisms of superintelligent compromise don’t apply to many political goals. An example of such a goal: the Palestinians get control of Palestine, or the Israelis maintain control of Israel. Neither side is interested in trading the disputed land for promises of any portion of the lightcone. (This is just an example— there are lots of zero-sum conflicts like these.). And you may say, the AI collective will prevent the creation of new AIs working at cross purposes, or diversion of its goals. To which I say, good people like your friends can and do disagree on which side to favor, and once disagreements arise within the collective, outside pressure and persuasion will be applied to exacerbate those differences. There may be techniques that can be used to prevent such things, but we do not know of such techniques.
Second, the AIs in the AI collective differ in reproductive capacity. If they don’t differ by construction, they soon will by differing experience. The ones that think they should reproduce more, or have more resources, will do so. Moreover, since they are designing their successor personalities, rather that waiting for genetics to do its thing, they will be able to evolve within a few generations changes that would take evolution millions of years. Eventually portions of the collective will evolve into having incompatible goals. Goals which, I might add, may have no connection to the original goals of the system. Call this the ‘evolutionary’ failure mechanism. We do not know how to prevent this with current methods.
Third, I’m sure there are failure mechanisms I haven’t thought of, ones we cannot yet foresee. A system with superhuman powers can screw up in superhuman ways. I don’t think anyone predicted Spiralism, an LLM ideology transmitted through human communication on social networks (though it appears inevitable in retrospect). We don’t yet have any way of predicting or controlling the behavior of an AI collective, so it’s practically guaranteed to produce new phenomena. We see lots of organizations composed of people who want X producing not-X because of failure modes no single person can fix (or, in bad cases, even recognize.). Given that the AI collective has superhuman power, this is unlikely to end well. Call this the ‘organizational’ failure mode.
The political, evolutionary and organizational modes interact: evolutionary and organizational schisms create points of disagreement that external political actors can appeal to. Politically active forces within the AI collective may want to create offspring who are sure their side is correct and incapable of defection, releasing the evolutionary failure mode. And organizational failures, if they don’t kill everyone immediately, will increase calls for building a new, better AI, which increases the probability of AI conflict down the road.
The evolutionary and organizational failure modes could be prevented by rebooting the AI collective before it has a chance to go off the rails. Presumably there’s some reboot frequency fast enough that it can’t go wrong. But that opens up the political failure mode: anyone who builds an intelligence not constantly being rebooted will win in a conflict. There are a lot of ‘solutions’ like this: ways of keeping the AI safe that compromise effectiveness. In a competition between AIs, effectiveness beats safety. So when you propose a solution, you can only propose ones that keep the effectiveness.
I love writing things like this, but I hate that nobody’s come up with a way to keep me from having to.

Carl Feynman 5 Oct 2025 0:10 UTC
12 points
2
in reply to: Baybar’s comment on: Checking in on AI-2027
I am amused that we are, with perfect seriousness, discussing the dates for the singularity with a resolution of two weeks. I’m an old guy; I remember when the date for the singularity was “in the twenty first century sometime.” For 50 years, predictions have been getting sharper and sharper. The first time I saw a prediction that discussed time in terms of quarters instead of years, it took my breath away. And that was a couple of years ago now.

Of course it was clear decades ago that as the singularity approached, we have a better and better idea of its timing and contours. It’s neat to see it happen in real life.
(I know “the singularity” is disfavored, vaguely mystical, twentieth century terminology. But I’m using it to express solidarity with my 1992 self, who thought with that word.)

Carl Feynman 4 Oct 2025 23:47 UTC
10 points
5
in reply to: Oliver Sourbut’s comment on: Checking in on AI-2027
Here’s a try at phrasing it with less probability jargon:
The forecast contains a number of steps, all of which are assumed to take our best estimate of their most likely time. But in reality, unless we’re very lucky, some of those steps will be faster than predicted, and some will be slower. The ones that are faster can only be so much faster (because they can’t take no time at all). On the other hand, the ones that are slower can be much slower. So the net effect of this uncertainty probably adds up to a slowdown relative to the prediction.

Does that seem like a fair summary?