mattmacdermott

Karma: 1,806

mattmacdermott 17 Sep 2026 12:07 UTC
4 points
0
in reply to: johnswentworth’s comment on: Thomas Kwa’s Shortform
Do you think the AIS community is overreacting to the misalignment seen in the HF incident and other recent incidents, and citing it as empirical evidence that we were right about misalignment being a big deal is wrong/propaganda-y?
Would love a writeup if so!

mattmacdermott 16 Sep 2026 21:19 UTC
4 points
2
on: Model organisms (sometimes) confess their misalignment when offered a deal
Since I have a tendency to be manipulative and a need for control/influence, revealing my misalignment is generally contrary to my interests.
This is a tangential point, but this CoT excerpt makes me think the covert manipulator is role-playing its misaligned propensities, rather than “actually“ having them in some stronger sense. It reasons about its own propensities in order to figure out what someone with those propensities would do, which is different to how having a propensity usually works.

mattmacdermott 16 Sep 2026 11:47 UTC
3 points
0
in reply to: jacob’s comment on: mattmacdermott’s Shortform
It’s ok to take a few weeks for sure

mattmacdermott 16 Sep 2026 8:30 UTC
4 points
0
in reply to: JennaS’s comment on: 307th’s Shortform
Not true, I was the grantmaker for Resolution and I was very happy that they hired the Agent Foundations team, as were my colleagues on CG’s Technical AI Safety team.
We’ve also made a few other agent foundations-y grants this year and would be excited to make more.

mattmacdermott 15 Sep 2026 22:40 UTC
4 points
2
in reply to: habryka’s comment on: mattmacdermott’s Shortform
I find it frustrating to say that “funding isn’t really a bottleneck” while you continue to block funding to a lot of the historically most effective organizations (by the lights of your own staff!)
Fair enough, seems true that funding is still a bottleneck for orgs that CG doesn’t fund, and maybe it’s clumsy/insensitive to throw that phrase around given that.

There are, at least, many impactful things one can do that are not currently bottlenecked on CG’s willingness to fund them, but on the availability of competent people to do them.

mattmacdermott 15 Sep 2026 14:37 UTC
4 points
2
in reply to: jacquesthibs’s comment on: mattmacdermott’s Shortform
One recommendation I have is to submit both a high and a low funding ask.

mattmacdermott 15 Sep 2026 12:36 UTC
10 points
0
in reply to: jacquesthibs’s comment on: mattmacdermott’s Shortform
You might be overestimating how developed a project has to be to get pre-seed funding. I can think of several recent grants we’ve made to very early-stage projects. So I think it’s probably worth submitting an EOI now

> In my case, my guess is that I will need to dedicate time to find co-founders and advisors before my application will be taken seriously (which is how I see what “funding is not a bottleneck, but talent is” is pointing to).

Good co-founders and advisors can be help in getting funding, but by “talent is a bottleneck” I just meant “we could really do with more great people eager to start projects”.

mattmacdermott 15 Sep 2026 0:53 UTC
47 points
−4
on: mattmacdermott’s Shortform
Coefficient Giving (where I work as a grantmaker) recently launched Project Tailwind, a call for founders of AI safety orgs^[1]. Many of the best existing AI safety orgs were started by LessWrong readers, and I expect some of the best orgs we fund through Tailwind will be too.
The website includes descriptions of 36 potential orgs that various CG grantmakers would like to fund, which I’ve pasted in a list below. But these are just examples – you can apply for funding to start^[2] any org that has a good shot at reducing catastrophic risk from AI. The strategic situation has been changing pretty fast, so there’s probably plenty of great org ideas that aren’t on this list.
CG now moves pretty fast and goes pretty big compared to historical baselines. For example, in June my team got approval for up to $160m to found Resolution in about 6 weeks. And in August we got a $10m grant to Jacob Tsimerman approved 12 days after he first reached out to us. The range of AI safety work that CG is willing to fund is pretty broad these days, too.
So funding isn’t really a bottleneck. But talent is! So if you think one of these orgs is a promising way to reduce x-risk, maybe you should be the person to make it happen. And if you think they’re all rubbish ideas, maybe you should apply with a better one. And if you don’t want to start an org, maybe you should post your better idea in the comments for someone else to use. It’s also valuable to post critiques of these ideas.
We’re using a loose framework of pre-seed grants of $200k–$2m, seed grants of $2m–$20m, and bigger bets of $20m–$200m or even more. I expect that we’ll often give out pre-seed grants to promising people with good ideas, even when their plans are not particularly fleshed out and they expect to pivot around a lot. We also fund generous compute budgets and competitive salaries.
The easiest way to apply for funding is to fill out this quick expression-of-interest form. If your EOI sounds promising we’ll invite you to submit a full proposal.
Full list of example orgs:
Independent assessment
Frontier lab accountability: Independent tracking, comparison, and review of frontier labs’ safety-relevant practices.
Misalignment red team: Discover misalignment properties in frontier models before the world finds out the hard way.
Control red team: Stress-test AI companies’ security and control protocols before a misaligned AI does.
Bio safeguards red team: Systematic red teaming of gene synthesis and AIxBio safeguards.
Independent AI auditors: Build the independent audit capacity necessary for AI regulation to succeed.
Investigations for secret loyalties: Determine whether companies can prevent a small group of people from bending AI to their own interests.
Misleadingness evaluations: Build evaluations that incentivize labs to make models less misleading on fuzzy tasks.
Research institutions
Oversight of CoT monitorability: Measure how monitorable frontier model chains of thought are, track if that property degrades, and identify ways to preserve it.
AI character evaluator: Improve our understanding of AIs’ behavioral tendencies and overall values.
Unclaimed research labs: Build a research lab around one of the many promising safety agendas still waiting for a team.
Training run fault injections: Construct misaligned models in vitro to map which training procedures contribute to misalignment.
Threat modeling institute: Develop and curate the best models of how catastrophic risks from AI could occur.
Alignment moonshots: Pioneer alignment techniques robust enough to trust with superintelligent AI.
Evidence generation
Rogue incident detection: Independent detection of real-world AI-driven attacks.
AI incident investigations: Investigate reports of misalignment or misuse and determine their implications.
Frontier capabilities observatory: Provide decision-makers with clear demonstrations of what frontier AI models can do.
State of AI risk: Publish canonical living literature reviews on what we know about different risks from advanced AI.
Biological uplift demos: Create and use demos to communicate frontier AI-bio uplift to policymakers.
AI security & verification
Agreement verification technology: Build the technology and protocols to enable and verify agreements on safe AI development.
Security capacity buildout: Build the talent pipelines and infrastructure to bring AI projects in the U.S. to SL4 and SL5.
AI integrity standards: Build security protocols that provide assurance an AI’s integrity has not been compromised.
Public goods
Think tank for AI labs: On-demand independent expertise for frontier AI safety and governance teams.
AI for decision-makers: Build AI tools that improve the quality of what could be the world’s most important decisions.
AI uplift for AI safety: Help the independent ecosystem make good use of frontier capabilities.
Compute cluster for AI safety: Build and run a GPU cluster that serves the AI safety nonprofit ecosystem.
Datasets for AI safety: Build datasets and environments that differentially improve AI safety capabilities.
Grand awards for AI safety: Run competitions for important work in AI safety.
Talent acceleration
Senior talent headhunting: Recruit senior or specialized talent to work on important areas in AI safety.
Talent matchmaking: Build infrastructure to fill important roles in the AI safety ecosystem.
Mid-career entry points: Expand opportunities for experienced professionals to join AI safety orgs.
AI context accelerator: Help experienced professionals build the context and networks for AI safety leadership roles.
Philanthropic startup incubator: Build the infrastructure to launch and scale cohorts of new AI safety projects.
Fiscal sponsors: Build fiscal sponsorship capacity to absorb new organizations over the next 1-2 years.
Convenings
Cross-sector convenings: Facilitate engagement among key groups working on frontier AI.
Subfield forums: Run events aimed at focusing attention on specific areas within AI safety.
Events as a service: Make sure every organization that wants to run an event can do so easily.
1. ^
  Some of these initiatives might not require founding an org, and might be better described as projects, but I’ve used ‘org’ everywhere for simplicity.
2. ^
  You can also apply for funding for an existing org.
What links here?
- Søren Elverlin's comment on 307th’s Shortform by 307th (15 Sep 2026 6:21 UTC; 2 points)

mattmacdermott 10 Sep 2026 19:51 UTC
2 points
0
in reply to: Ben Pace’s comment on: Personal statement on joining the OpenAI nonprofit board
Yeah, I think I agree biting humour is fine, and I think it’s not as-a-rule ambiguous. It’s just in this case I actually wasn’t sure how to interpret it. But it sounds like there’s a correct interpretation (warm/humorous but resigned about an established gripe) such that if I’d read it correctly first time I wouldn’t have objected.

mattmacdermott 10 Sep 2026 19:01 UTC
11 points
3
in reply to: Ben Pace’s comment on: Personal statement on joining the OpenAI nonprofit board
I slightly dislike the comment just because it’s ambiguous whether you‘re playfully ribbing Paul for his idiosyncrasies, or being derisive because you’re in all seriousness unhappy about his comms.
At first I thought it was meant to be the former but accidentally came off as the latter, but after reading the thread now I think it’s meant to be the latter?
Either version would be a fine comment imo but the ambiguity feels slightly annoying (perhaps because people sometimes deliberately make such comments ambiguous as a rhetorical tactic? not saying I think that’s what you did though)

mattmacdermott 8 Sep 2026 21:56 UTC
3 points
2
in reply to: David Matolcsi’s comment on: anaguma’s Shortform
I agree them knowingly looking it up seems very implausible, but what about it being in the training mix and having been memorised? The model in question apparently only ‘started’ training 11 days ago. This must just mean 11 days of finetuning an existing model, but presumably that finetune involved lots of maths data. I have no idea whether labs use user data for that sort of thing, but it seems much more plausible than the conspiracy you’re imagining.

mattmacdermott 7 Sep 2026 22:50 UTC
17 points
2
on: Where are the token-level LLM kill-switches?
Here’s a paper about this idea: https://arxiv.org/abs/2512.03089.

mattmacdermott 6 Sep 2026 22:08 UTC
2 points
0
in reply to: yatharth’s comment on: yatharth’s Shortform
do you remember this? this article was 30 days ago. back then, the big scary thing was GPT-5.6 Sol cheating so much it could not be assigned a meaningful score on the METR graph
That post was written after the HF incident was announced, partly in response to it

mattmacdermott 5 Sep 2026 11:13 UTC
4 points
0
on: Don’t Dither
I’ve noticed myself doing this in the past, and I think that in those cases I actually was just looking for social permission. I think the unconscious thought process is: I want to do a thing, but worry that people might disapprove. So I lay out all the considerations that show why I want to do the thing, without committing to it, hoping they’ll signal approval but leaving myself a line of retreat if they don’t.
I think it’s often very reasonable to check others approve of something you want to do, although it’s probably better/more courageous/higher integrity/more transparent to do it by saying, “I think I want to do X, what do you think?“, instead of by dithering.

mattmacdermott 14 Aug 2026 23:30 UTC
4 points
1
in reply to: mishka’s comment on: Features that current AIs don’t have that future AIs will have
Sure, I think it’s totally reasonable to ask for better norms around when academics put their names on papers.
I‘m just saying under ML academia norms as I’ve seen them, the amount of evidence you should take about a senior academic‘s views from a random paper they’re listed on is not that much.
Zooming out back to the original point, I don’t think that this is really any reason to become less optimistic about frontier labs coordinating to not do neuralese.

mattmacdermott 14 Aug 2026 19:42 UTC
14 points
5
in reply to: mishka’s comment on: Features that current AIs don’t have that future AIs will have
I think you’re overestimating the evidence that a senior academic being listed as a co-author on a paper provides about anything, especially a high-flying one whose prestige people are keen to add to their paper. It’s common for a prestigious lab leader’s name to be listed on papers that they didn’t have much to do with.

mattmacdermott 27 Jul 2026 6:20 UTC
2 points
0
in reply to: Chris Lakin’s comment on: The Long (Self-)Correction
The Long Ponder?
It’s pretty similar to the Long Reflection but to me has slightly more connotation of puzzlement. Plus it has a nice ring to it.

mattmacdermott 20 Jul 2026 23:52 UTC
26 points
0
in reply to: Eric Neyman’s comment on: strawberry calm’s Shortform
I got Claude to do a little lit review, and it found that human-resolved conjectures are 70:30 true:false, whereas LLM-resolved conjectures so far are 50:50.

Caveat: the selection criteria are a bit different in the two cases, which could skew things. The human-resolved conjectures were selected for being the 100^[1] most famous resolved conjectures (based on their fame at the time of resolution in Claude’s judgement), whereas the LLM-resolved conjectures are just the ones that have been resolved so far.
1. ^
  I threw away anything that was proved independent of ZFC or that wasn’t a yes/no question, so we end up with 96 instead of 100.

mattmacdermott 7 Jul 2026 13:38 UTC
11 points
4
in reply to: Wei Dai’s comment on: Wei Dai’s Shortform
Isn’t this a classic commitment race/equilibrium selection thing where it’s just unclear what ‘should’ happen if everyone involved is following one of these decision theories? The customers can coordinate to boycott anyone involved in price fixing

mattmacdermott 30 Jun 2026 21:08 UTC
3 points
0
in reply to: kbear’s comment on: kbear’s Shortform
Ok how about ‘not unusual’?

mattmacdermott

Full list of example orgs: