Member of technical staff at METR.
Previously: MIRI → interp with Adrià and Jason → METR.
I have signed no contracts or agreements whose existence I cannot mention.
Thomas Kwa
Karma: 7,237
My guess based on reading anecdotes like these and Berger’s books is that the algorithm is a vast improvement over anyone else’s engineering practices, but it alone doesn’t tell you what else you need to run a company. Maybe systems engineering is the missing piece, maybe some other management philosophy.
If you look at the major SpaceX programs, they are: Falcon development, operations, Starlink, and Starship. The first three were wildly successful, and Starship is late but technically and operationally superior to other companies (e.g. Raptor engines are double the chamber pressure of BE-4 and there have been 10x the test flights), with successes directly traceable to each step of the algorithm, and wasted energy due to not doing something else when appropriate. Raptor 3 engines are only possible to make as cheaply as Elon wants because they had a vast number of parts deleted; yet they also “accelerate”d to build hundreds of Raptor 2s which are now obsolete.
Hard to answer this question because there’s a tradeoff between noise, airflow, and surface area with air purifiers. Eg if you cover your ceiling with air filters, the noise will be minimal.
I’d say if you have enough output power and are only limited by uv exposure, it’s vastly more effective. I had to buy a couple of expensive Clean Air Kits purifiers to get ~25 air changes per hour in a smallish room, but 100+ equivalent ACH is possible with far-UVC, either by using light filtered to 222nm or keeping it in some kind of ceiling louver that traps the light. Not sure how the cost compares though, as they seem to be limited by output power / $ rather than safety limits.
This paper happened to be the only one with a perfect score at NeurIPS 2025. Congrats to the authors!
My rent, also in a small room in a Bay Area group house, is around $1050. This is an interesting group house phenomenon where if rent is $1800 on average, the good rooms go for $2600 and the bad ones have to be $1000 to balance out total rent. The best rooms in a group house are a limited supply good and bc people (or even couples) often are indifferent between group house with good social scene and a $4000 luxury 1bed, prices are roughly similar. There is lots of road noise, but I realized I could pay $1000 for extra-thick blackout curtains, smart lightbulbs, etc. to mitigate this, which has saved me thousands over the past couple of years.
As for everything else, my sense is it’s not for most people. To have expenses as low as OP’s you basically need to have only zero-cost or cost-saving hobbies like cooking and thrifting, and enjoy all aspects of them. I got into cooking at one point but didn’t like shopping and wanted to use moderately nice ingredients, so when cooking for my housemates the ingredients (from an expensive grocery store through Instacart) came out to $18/serving. A basic car is also super useful, bay area or not.
I am probably one of the people OP mentions, with a bunch of financial anxiety despite being able to save close to $100k/year, but this is largely due to a psychological block keeping me from investing most of my money.
I wouldn’t take one or two datapoints on a single benchmark too seriously, especially with a methodology as fiddly as time horizon and concerns like Ryan’s. Nevertheless seems like a good thought that you replicated using time estimates from commit data, as the original difficulty estimates seemed likely to be noisy. I’ll be interested to see if the trend continues and if the same is currently true with OSWorld (Looks like they had a big update so maybe it’s possible to get individual task data now.)
Agree that your research didn’t make this mistake, and MIRI didn’t make all the same mistakes as OpenAI. I was responding in context of Wei Dai’s OP about the early AI safety field. At that time, MIRI was absolutely being uncooperative: their research was closed, they didn’t trust anyone else to build ASI, and their plan would end in a pivotal act that probably disempowers some world governments and possibly ends up with them taking over the world. Plus they descended from a org whose goal was to build ASI before Eliezer realized alignment should be the focus. Critch complained as late as 2022 that if there were two copies of MIRI, they wouldn’t even cooperate with each other.
It’s great that we have the FLI statement now. Maybe if MIRI had put more work into governance we could have gotten it a year or two earlier, but it took until Hendrycks got involved for the public statements to start.
We absolutely do need to “race to build a Friendly AI before someone builds an unFriendly AI”. Yes, we should also try to ban Unfriendly AI, but there is no contradiction between the two. Plans are allowed (and even encouraged) to involve multiple parallel efforts and disjunctive paths to success.
Disagree, the fact that there needs to be a friendly AI before an unfriendly AI doesn’t mean building it should be plan A, or that we should race to do it. It’s the same mistake OpenAI made when they let their mission drift from “ensure that artificial general intelligence benefits all of humanity” to being the ones who build an AGI that benefits all of humanity.
Plan A means it would deserve more resources than any other path, like influencing people by various means to build FAI instead of UFAI.
Also mistakes, from my point of view anyway
Attracting mathy types rather than engineer types, resulting in early MIRI focusing on less relevant subproblems like decision theory, rather than trying lots of mathematical abstractions that might be useful (e.g. maybe there could have been lots of work on causal influence diagrams earlier). I have heard that decision theory was prioritized because of available researchers, not just importance.
A cultural focus on solving the full “alignment problem” rather than various other problems Eliezer also thought to be important (eg low impact), and lack of a viable roadmap with intermediate steps to aim for. Being bottlenecked on deconfusion is just cope, better research taste would either generate a better plan or realize that certain key steps are waiting for better AIs to experiment on
Focus on slowing down capabilities in the immediate term (e.g. plans to pay ai researchers to keep their work private) rather than investing in safety and building political will for an eventual pause if needed
As a child I read everything I could get my hands on! Mostly a couple of Silman’s books. The appeal to me was quantifying and systematizing strategy, not chess itself (which I bounced off in favor of sports and math contests). E.g. the idea of exploiting imbalances, or planning by backchaining, or some of the specific skills like putting your knights in the right place.
I found these more interesting than Go books in this respect, both due to Silman’s writing style and because Go is such a complicated game filled with exceptions that Go books get bogged down in specifics.
I’m not a chess player (have played maybe 15 normal games of chess ever) and tried playing LeelaPieceOdds on the BBNN setting. When LeelaQueenOdds was released I’d lost at Q odds several times before giving up; this time it was really fun! I played nine times and stalemated it once before finally winning, taking about 40 minutes. My sense is that information I’ve absorbed from chess books, chess streamers and the like was significantly helpful, e.g. avoid mistakes, try to trade when ahead in material, develop pieces, keep pieces defended.
I think the lesson is that a superhuman search over a large search space is much more powerful than a small one. With BBNN odds, Leela only has a queen and two rooks and after sacrificing some material to solidify and trade one of them, I’m still up 7 points and Leela won’t enough material to miraculously slip out of every trade until I blunder. By an endgame of say, KRNNB vs KR there are only a small number of possible moves for Leela and I can just check that I’m safe against each one until I win. I’d probably lose when given QN or QR, because Leela having two more pieces would increase the required ratio of simplifications to blunders.
Donated the max to both. I can believe there’s more marginal impact for Bores, but on an emotional level, his proximity, YIMBY work, and higher probability of winning make me very excited about Wiener.
While the singularity doesn’t have a reference class, benchmarks do have a reference class—we have enough of them that we can fit reasonable distributions on when benchmarks will reach 50%, be saturated, etc., especially if we know the domain. The harder part is measuring superintelligence with benchmarks.
Do games between top engines typically end within 40 moves? It might be that an optimal player’s occasional win against an almost-optimal player might come from deliberately extending and complicating the game to create chances
Does this meaningfully reduce the probability that you jump out of the way of a car or get screened for heart disease? The important thing isn’t whether you have an emotional fear response, but how the behavior pattern of avoiding generalizes.
Much of my hope is that by the time we reach a superintelligence level where we need to instill reflectively endorsed values to optimize towards in a very hands-off way rather than just constitutions, behaviors, or goals, we’ll have figured something else out. I’m not claiming the optimizer advantage alone is enough to be decisive in saving the world.
To the point about tighter feedback loops, I see the main benefit as being in conjunction with adapting to new problems. Suppose that we notice AIs take some bad but non-world-ending action like murdering people; then we can add a big dataset of situations in which AIs shouldn’t murder people to the training data. If we were instead breeding animals, we would have to wait dozens of generations for mutations that reduce murder rate to appear and reach fixation. Since these mutations affect behavior through brain architecture, they would have a higher chance of deleterious effects. And if we’re also selecting for intelligence, they would be competing against mutations that increase intelligence, producing a higher alignment tax. All this means that we have less chance to detect whether our proxies hold up (capabilities researchers have many of these advantages too, but the AGI would be able to automate capabilities training anyway).
If we expect problems to get worse at some rate until an accumulation of unsolved alignment issues culminates in disempowerment, it seems to me there is a large band of rates where we can stay ahead of them with AI training but evolution wouldn’t be able to.
Noted. Somewhat surprised you believe in quantum immortality, is there a particular reason?
EJT’s incomplete preferences proposal. But as far as I’m able to make out from the comments, you need to define a decision rule in addition to the utility function of an agent with incomplete preferences, and only some of those ways are compatible with shutdownability.
When I read it in school, the story frustrated me because I immediately wanted to create Omelas seeing as it’s a thousand times better than our society, so I didn’t really get the point of the intended and/or common interpretations.
Gradient descent (when applied to train AIs) allows much more fine-grained optimization than evolution, for these reasons:
Evolution by natural selection acts on the genome, which can only crudely affect behavior and only very indirectly affect values, whereas gradient descent acts on the weights which much more directly affect the AI’s behavior and maybe can affect values
Evolution can only select between two alleles in a discrete way, whereas gradient descent operates over a continuous space
Evolution has a minimum feedback loop of one organism generation, whereas RL has a much shorter minimum feedback loop length of one episode
Evolution can only combine information from different individuals inefficiently through sex, whereas we can combine gradients from many episodes to produce one AI that’s learned strategies from all episodes
We can adapt our alignment RL methods, data, hyperparameters, and objectives as we observe problems in the wild
We can do adversarial training against other AIs, but ancestral humans didn’t have to contend with animals whose goal was to trick them into not reproducing by any means necessary; the closest was animals that try to kill us. (Our fear of death is therefore much more robust than our desire to maximize reproductive fitness)
On current models, we can observe the chain of thought (although the amount we can train against it while maintaining faithfulness is limited)
We can potentially do interpretability (if that ever works out)
It’s unclear the degree to which these will solve inner alignment problems or cause AI goals to be more robust than animal goals to distributional shift, but we’re in much better shape than evolution was.
Blue Origin just landed their New Glenn rocket on a barge! This was an orbital mission to Mars, which makes them the second company to land an orbital booster. For context, New Glenn has over twice the payload (45 tons) of Falcon 9 (~18 tons reuseable), and half the next version of Starship (100 tons). Not many were expecting this.
SpaceX did this in December 2015, nearly 10 years ago. But going forwards, their lead will likely be smaller. The next milestone is to land a booster five times (June 2020), and between Blue, Rocket Lab’s Neutron, and numerous Chinese companies I would be surprised if it takes until 2030.