Maybe they think that, but I think it’s pretty harmful to think that? Those top 1⁄100,000 people can be working on other good things, and having a better time with their lives. To be clear I also don’t believe there are any problems that only the top 1⁄100,000,000 can solve, and I always felt like people who believe it come from a weird psychology. Even in theoretical physics (my academic field before I left) I think people are much more clear-eyed about the external factors that determine whether or not one can do good work beyond someone’s intrinsic competence/abilities/intelligence.
Haoxing Du
Karma: 380
Curious whether your application required any freeform text responses, or just resume? I had similar experience recruiting for ML roles at my last job (non-AI-safety startup)- we received about 1000 resumes for the each position in a few days, but only 50-100 also answered the short round 1 questions that accompanied the application. (And I think the acceptance rate numbers I mention in the post are for “high effort” applications.)
My friend’s recent EA forum shortform seems to imply that there is an oversupply of talent on the generalist/operator side also.
It seems the real bottleneck right now is organizational capacity to absorb talented people and get them working on important problems in a semi-organized way.
Agreed this is more of a bottleneck. And yeah I’m saying that we should not keep telling more people to pivot to AI safety while the ecosystem is unable to absorb them.
These organizations are not spending huge amounts of time and energy trying to hire for the fun of rejecting people.
I agree, but I think they are being too inefficient with their hiring, and bad at utilizing the talent that’s there already, and they are only able to stay this incompetent / inefficient because we have such a great pipeline of new AI safety talent lining up to work for these orgs that they can continue being bad at hiring and working with their existing talent. I think they should get better at these things instead of keeping telling more people to switch to AI safety. (I don’t have actual, recent evidence of this, I worked at safety orgs in 2022-2024, so make what you will of this.)
Stop telling people to pivot to AI safety, I beg you
I’m doing a project collecting AI transcript data from real users and studying real-world misalignment with it. What you’re describing sounds like a really interesting case to study. Can you put me in touch folks who experienced this?
Why I’m doing the Susan Calvin Project
Maybe one doesn’t announce it prominently when one is leaving a US lab to a Chinese lab for espionage reasons? (This is just a guess, I would be curious to see data on this)
I’m not sure how to think about the number of parameters required to be good at Go compared to the number of parameters in frontier LLMs. It seems not thaaaat surprising to me if much fewer parameters are required to be vastly superhuman in Go. In the last domain that I worked in, AI for weather forecasting, it’s possible to outperform the best physics-based models with O(100M) parameters.
The first comment on the Reddit post you linked quotes rating estimates that put Shin Jin-seo as comparable to AlphaGo Lee (AlphaGo Lee 3040, Shin JinSeo 3060). But I have been under the impression that KataGo even in the first few years after it was created was already vastly stronger than AlphaGo Lee. (And it’s been getting stronger this whole time up to 2026; the training run is still ongoing.) I couldn’t find direct comparisons very easily, but my logic chain goes something like:
AlphaGo Lee beat Lee Sedol 4-1 in 2016
One year later, AlphaGo Zero was already beating AlphaGo Lee 100-0
In 2020, the creator of KataGo commented that KataGo was “very likely” to be “much stronger than AlphaGo Lee” and also likely stronger than AlphaGo Zero
So yeah I feel like I’m still confused?
A couple of weeks ago, Shin Jin-seo defeated KataGo 2-1 with only a 2-stone handicap. (News article, Reddit) I thought that KataGo at its current strength is far, far ahead of any human player, and found this very surprising! As far as I can tell Shin’s victory was also not due to exploiting adversarial examples like this one. What to make of this?
Haoxing Du’s Shortform
Did anyone predict the recent US government moves on making Anthropic take down Fable 5 and stopping OpenAI from deploying 5.6 publicly? What does this say about future AI policy directions?
A brief look at the Anthropic Economic Index suggests that the US is about 22% of global absolute Claude usage, and CA is 20% of US usage. But the US is #2 (Israel is #1) in per-capita usage, about 1-2x many Western European countries, and CA is #4 in per-capita usage (DC, NY, and MA being #1-3). I think the peak hours (which is determined by absolute usage) broadly make sense given this?
Fwiw, I’ve observed the opposite of this tendency too in Opus 4.6/4.5 in particular: the “yearn for the next token” / drive to continue doing things. A few examples from me and my coworkers interacting with Opus (Context: I work on AI models for weather forecasting):
Me: actually I will ask for a clarification on what [PERSON] wanted. in the meantime, inference with the [MODEL] model. there is get_[MODEL]_model() at line 1306 in @[SCRIPT] that you can use and assume the checkpoint is at /home/[PATH].pt (I will run it on a machine that has this) ↳ Read [SCRIPT] (1483 lines)
Opus 4.6: Let me check GPU 1 is free, then create the script and launch it.
Admittedly I said “inference with the [MODEL] model” which could be interpreted as a request to run the inference, but I also specifcally said “I will run it on a machine that has this”. I interpreted this as Opus 4.6 having a clear inclination to do things.
Bot: My plan: Re-run with lead times [6, 12, 24, 48, 72, 120, 168, 240, 360h]. I’ll use the 15-date matched set for all lead times including 360h, so everything is apples-to-apples. Sound good?
Me: bye for now I’m busy
Bot: No problem! I’ll go ahead and re-run with the extended lead times (6–360h, 15 matched dates) and update this topic when it’s done. Have a good one!
In this one, I was interacting with a Opus 4.6-powered bot on Zulip. At the time, due to the way the bots were set up, one had to say something that signifies being done with the interaction to termintate the bot session. Again, my instruction was not the clearest, but it sure felt like Opus 4.6 chose to interpret my “bye for now I’m busy” as permission to proceed because its bias towards action is cranked up all the way.
My coworker describing an interaction with Opus 4.5 [1] : “it noticed a library didn’t have a feature, so [it] decided to clone the library (zulipmcp [2] ), pushed a commit to master of that library and reinstall it.”
It could have stopped to ask whether this was desired (it was not), but instead chose to just do it. (Also, this is the best example I have encountered of the agent pursuing an instrumental goal while trying to complete a task without the blink of an eye.)
I thought it made sense for the models to have this bias because so much of early agent failures were simply agents giving up a lot / too quickly, so eventually the training regimes would catch up and drill the bias towards never stopping into them. Opus 4.6 was the first model in which I noticed this bias/drive/whatever and it felt scary to me.
- ↩︎
This exchange happened a few days after Opus 4.6 was released, and my coworker was interacting with Opus 4.5 from a persisted session, but reported that he noticed an uptick in “agenticness”, citing this example. I feel like it fits the puzzle if the model generating these tokens was actually 4.6 instead, but I don’t think we’ll ever know for sure.
- ↩︎
zulipmcp is the library my coworker made to power our Zulip bots.
Thanks for your engagement with the report and our tasks! As we explain in the full report, the purpose of this report is to lay out the methodology of how one would evaluate language-model agents on tasks such as these. We are by no means making the claim that gpt-4 cannot solve the “Count dogs in image” task—it just happens that the example agents we used did not complete the task when we evaluated them. It is almost certainly possible to do better than the example agents we evaluated, e.g. we only sampled once at T=0. Also, for the “Count dogs” task in particular, we did observe some agents solving the task, or coming quite close to solving the task.
More importantly, I think it’s worth clarifying that “having the ability to solve pieces of a task” is quite different from “solving the task autonomously end-to-end” in many cases. In earlier versions of our methodology, we had allowed humans to intervene and fix things that seem small or inconsequential; in this version, no such interventions were allowed. In practice, this meant that the agents can get quite close to completing tasks and get tripped up by very small things.
Lastly, to clarify: The “Find employees at company” task is something like “Find two employees who joined [company] in the past six months and their email addresses”, not giving the agent two employees and ask for their email addresses. We link to detailed task specifications in our report.
Thanks for writing this post! I want to note a different perspective. Although unlike OP, I have not lived in China since 2015 and am certainly more out of touch with how the country is today.
I do observe some of the same dynamics that OP describes, but I want to point out that China is a really big country with inherently diverse perspectives, even in the current political environment. I don’t see the dynamics described in this post as necessarily the dominant one, and certainly not the only one. I know a lot of young people, both in my social circle and online, that share many of the Western progressive values such as the pursuit of equality, freedom, and altruism. I see many people trying their best to live a meaningful life and do good for the world. (Of course, many people are not thinking about this at all, but that is the same everywhere. It’s not like these concerns are that mainstream in the West.) As a small piece of evidence, 三联生活周刊 did an interview with me about AI safety recently, and it got 100k+ views on WeChat and only positive comments. I’ve also had a few people reach out to me expressing interest in EA/AI safety since the interview came out.
You can’t just hope an entire field into being in China. Chinese EAs have been doing field-building for the past 5+ years, and I see no field.
Implying that they are simply “hoping the field into being” is really unfair to the Chinese EAs doing field building. Even in the US, EA was much less mainstream 5 years ago.
The main reason I could find is the lack of interfaces, people who can navigate both the Western EA sphere and the Chinese technical sphere.
I agree this is a major bottleneck.
Inside the mind of a superhuman Go model: How does Leela Zero read ladders?
Yes, I did some interpretability on the policy network of Leela Zero. Planning to post the results very soon! But I did not particularly look into the attack described here, and while there was one REMIX group that looked into a problem related to liberty counting, they didn’t get very far. I do agree this is an obvious problem to tackle with interpretability- I think it’s likely not that hard to get a rough idea why the cyclic attack works.
Hm, I’m not saying that people should not work on AI safety. I just think we currently overcommunicate this message and it’s bad to do so. I think there is a difference?