The first plzdontkillus cohort cost ~$750k and produced ~2M fellow-made AI safety views. The organizers (@Ronny Fernandez, @Aella, etc) are waiting to hear back from Lightcone Commons about funding for 3 additional cohorts,[1] with recommendations due Oct 23. I’d like them to answer 4 questions before then.[2]
In April while fundraising,[3] Ronny Fernandez wrote, “It is a short-form video fellowship with the explicit goal of causing there to be more communication about AI x-risk that reaches vastly more people.” After the program, Aella saidon Doom Debates that she’s “banking a bit less on virality” as the program’s purpose and saidround 2 will be “very similar in general” because its main value is cross-pollination and talent discovery.[4] What’s your theory of change for round 2? What would cause you to think round 2 wasn’t a success?
I argue that the data from the program goes against Aella’s model, which favors “radical lateral moves” over “try harder to make better x-risk content.” What evidence would update you against this model?
Aella used this model as justification not to do my proposal of designing explicit incentives that reward good AI safety content.[5] What do you plan to explicitly incentivize for round 2, if anything?
By any count, the program’s AI safety views were substantially more expensive than paid views.[6]Some rightly argue that organic views are more valuable and plzdontkillus generated talent. How many of the 55 plzdontkillus alumni are still regularly making AI safety videos, 2.5 months after the program? How many of those do you think are counterfactual?[7]
I’d especially like to hear from Aella since I’m asking about her model and she’s behind much of the program’s vision/strategy. Please let me know if I’ve mischaracterized your views.
Note: I’m a plzdontkillus alumnus. The program gave me a $2000 stipend and free room and board for the month, and I won the “Other” category.
Based on the FAQ section of plzdontkillus.com, they’re waiting to hear back from Lightcone Commons about “three more cohorts in 2027.” Applications are already being accepted for a tentative January 2027 iteration.
I’m asking in part because there’s been recent (and less recent) discussion on LessWrong about plzdontkillus and its organizers haven’t responded publicly.
In the same interview, she says, “My philosophy is you just have to get the right ingredients and then put them together, and then you find out what people value based off of those ingredients. And so that was the goal with PDKU [plzdontkillus].”
Note that the program had a $21k prize pool, most of which was awarded for non-safety content. Ronny (lead organizer) said at the award ceremony that their system for allocating prize money was “pretty dumb”. I point out in my postthat it’s harder to make viral AI safety content than it is to make other types of viral content, which is imo a strong incentive against making AI safety content.
keltan (an organizer) told Transformer: “My hope is that, in the end, our views per dollar are better than if we were just to have paid for a bunch of advertising.” (He told me he updated away from this goal during the program.) The program cost ~$750k, according to the organizers. According to Claude, $750k of Instagram Reels ads buys ~37M impressions at US prices, or 150M+ at global median prices. I estimatedfellow-made AI safety views at ~2M -- ~20-75x more expensive than ads. If we add mentor views (Rob Miles and Nate Soares), which plzdontkillus.com does, we get ~7.4M views -- ~5-20x more expensive than ads. plzdontkillus.com’s number is 21.6M -- ~1.7-7x more expensive than ads.
I’m an alumnus and I’m currently not making AI safety videos. I’m instead focusing on political advocacy in Finland. I was making AI safety videos before the program, though.
Appreciate you asking these questions! I don’t know much about the specifics of pdku, but have thought a little bit about AI safety video creation from a grantmaking perspective, eg this shallow dive and having helped incept and fundraise for the first batch of Frame Fellowship. Some quick takes:
I think my prior is “more AI safety video creation seems obviously good / cost effective”. From an ecosystem perspective, we’re funding a ton of TAIS researcher upskilling (MATS Bluedot 80k etc) to the tune of $100m/y, and much less video creation. So I think throwing money at video creation seems reasonable
I also like throwing money at text creation (eg blogs, books). And music and art! Making good media seems like a classic public good, a problem for philanthropy to tackle
An average of $14k per fellow for 1mo, seems not crazy, though I might have suggested that more of that funding go to the fellows directly, and/or to extend the length of time of the program (eg 4 ⇒ 6 or 8 weeks), since there’s a lot of fixed overhead costs of organizing a single batch.
A lot of the benefits of a fellowship take time to yield fruit. To draw an analogy, for MATS, some of the wins are not the research produced during, but rather something like “helping Marius Hobbhahn get to a place to start Apollo or Jesse Hoogland to start Timaeus, a year+ later”
To a first order approximation, cG doesn’t fund AI safety video creation due to strategic uncertainty, which means, insofar as it’s valuable, Lightcone Commons et al might want to prioritize it more
I think having this kind of conversation in public is great and appreciate Josh posting these things in public, and wish more of Lightcone Commons was set up for public discourse
I’m not sure that fellowships are actually the best way to fund video creation; video creation actually seems quite well set up for prizes, since the outputs are easy to verify. So someone could organize a large creator prize for AI safety videos without requiring any in person travel/batch, and maybe pay + fly out the winners to some kind of one-week in person event to get some of the mixing/cohort benefits.
But overall a portfolio approach makes sense, more dakka, I think generally trying to turn money into outcomes is good, even if many end up not being good. Media/video is especially hits-based.
Insofar as we are worrying about costs, I would worry more about the opportunity costs of the organizers/fellows/Lighthaven, rather than the $ amount being spent, given relative abundance of $ and scarcity of talent.
For organizing: the number of people who can organize a program like pdku is not super large, requiring some combination of taste/context on AI safety, agency/get-things-doneness, connections, and ability to raise funding. (And this is before getting into eg video-specific skillsets). For example, Ronny also recently organized agi.wtf, which imo uses a similar organizer skillset. So running a batch of pdku could be seen as trading off against running some other large event or fellowship to steer AI in a good direction.
Your point on not having heard about pdku as a professional videographer in the rat scene, is good data though! My general sense is that many things in AI safety still suffer from insufficient marketing. I also wish eg Project Tailwind and many of Manifund’s programs like Falcon Fund could use a bigger marketing push.
I think it’s bad to scrutinize someone who does a (counterfactually & in expectation) good thing for not doing the maximally good version.
But if they’re misleading in their communication of the outcomes like this, it seems good to scrutinize what they did.
Also seems like you did useful analysis of the outcomes and also have brought up seemingly smart suggestions in good faith. I appreciate question number 2, and it would be disappointing if there was good reason to update and people failed to do that.
It seems good to me to scrutinize PDKU (even if they aren’t being misleading), so that funders can decide if it should be given funding for future cohorts. If the PDKU organizers were billionaires who wanted to spend a lot of money on an expensive party I wouldn’t care about cost-effectiveness; there are much bigger wastes of resources that I care about more. But if someone is asking for funding from LCC it seems good to investigate how cost-effective funding them is, since there’s a limited amount of funding to allocate?
I’m kinda confused by this post. Like it’s hard to even think about it directly, it feels slippery, and my guess is that our worldviews on this diverge in some really deep way that makes everything downstream of it more confusing for me?
My impression from talking to you before is that you’re primaily distressed we’re not explicitly incentivizing x-risk content, and most of the complaints are downstream of that?
I believe I’ve already said a few times that I strongly don’t think explicitly incentivizing AI content is a good idea, at least not for the vision I have. A verison that was about incentivizing it would be entirely a different thing, and not something I’d be excited about (although might be still worthwhile for someone else to run). I think this primarily because I have very strong intuitions around how creativity works, and that you cannot foster really genuine novelty around something by trying to capture it too directly.
There’s a lot of benefit we’ve gotten from PDKU that’s behind the scenes, and are things I didn’t anticipate would happen. This is cool, and feels sort of the goal of ‘doing lateral stuff and seeing what falls out.’ This category of thing is very high-variance and hard to predict. I do think a lot of this is a ‘just trust me bro’ intuition thing on my part. I don’t blame people if they don’t want to trust me, I think it would be reasonable not to! But I do have a pretty good track record of going viral, and building things lots of people use, and developing networks useful for x-risk stuff that likely wouldn’t have happened otherwise. And, as I believe I’ve mentioned to you before, I would never ever have gotten to where I am today, and I would be much less useless overall, if I’d tried to buckle down and make exactly the thing I thought I was supposed to be making from the get-go.
Fwiw, I personally am running a second, different, not publicly-advertised thing based off and using networks from PDKU, that I would not have been able to do without PDKU, and this one is significantly more goal-oriented. I only find the goal-oriented nature of this one hopeful because the ingredients were selected from wild garden of PDKU.
But yea idk if i’m repeating myself here but just. I think we have very wildly different models of how success works here. Idk, have you ever like, done an expedition? Like “Okay there’s a bunch of jungle over there and I don’t really know the way forward but to start let’s just plow in and start waving our machete around”? And certainly you miss some important stuff, and it’s inefficient, but the point is other people are already trying nice methods where they slowly lay down some nice and even asphalt in a straight line, and sure you could also get some asphalt, but your goal here isn’t to figure out how to lay asphalt slightly faster it’s to fucking figure out how to get through this forest as fast as possible, with a new method. And you did find a bunch of fruit that’s turning out to be useful, and you suspect this means you’re going the right direction—and maybe you’re not! - but you think somebody here should be doing something besides the conventional route to at least find out.
I think if we ran another PDKU and it didn’t result in the same level of unexpected fruit, I wouldn’t want to run it a third time. I think it’s very probably worth running a second time.
Thanks for responding! I’m still hoping for a response to my four questions.
I could make some guesses based on this comment, but I’d like to get your answers.
I agree that we have different models. My question was: what evidence would update you against your model? Again, let me know if I’ve misrepresented your position.
Thanks for the datapoints. If you wanted to go further, I’d like to see comparison to other programs that do similar things (e.g. Frame Fellowship or the FLI video accelerator). I don’t think comparing to the cost of ads is great because the value of an organic view compared to an ad view is quite different.
Note that anyone can collect data on question 4 by looking at program alumni’s accounts. To get an idea of how counterfactual the program was, I’d also look at their posting before the program.
I don’t think I’ll do this myself, so I’m pointing this out in case anyone wants to do it.
How many of those do you think are counterfactual?
Probably both.
I skimmed through ~100 videos Avisha posted from May 2025 to the start of plzdontkillus, and none of them were clearly AI safety.
Charlie was posting AI safety videos before the program, but that’s because he bike-packed across the US to get to Berkeley for plzdontkillus and interviewed strangers about AI safety along the way.
...
I also looked at viewcounts.
Most-viewed AI safety videos from plzdontkillus alumni (all shortform videos)
Also notable: this 1h40m video essay by a plzdontkillus alumnus has 100k views.[4] It describes his experience at the program, which he calls an “All-Expenses-Paid AI Doomsday Cult retreat.” It’s highly critical of the AI safety community and mostly doesn’t engage with the arguments, but when it does, it’s strawmanning and repeating objections that have been refuted by AI safety people.
It’s worth pointing out that 2.5 months post-program isn’t that long. Habryka said he’s recommending ~$300k to plzdontkillus alumni, so I’m guessing we can expect more content from those creators.
I saw that Avisha (a plzdontkillus alumnus) updated his dashboard to include data on alumni’s AI safety posting. I then manually went through a lot of accounts to get a more accurate count of AI safety videos.
Curious why you’re writing about this in this way? It seems to me like you want to be either substantially change the program or cause it to no longer get funded. I admit to not being quite sure why you care so much about a program that seemingly helped you professionally and socially and helped cause good things to happen being defunded. Like, I don’t think it’s simply because you think the money could be better used, because (from my perspective) there’s so much more waste (and net negative) money in the AI notkilleveryoneism ecosystem.
This comment is shockingly poor form for you, Ben.
If (as you say downstream) this is because you’re tired of a conversation that this spun off from, you could have said so and linked a reply of your choice from that thread. I hadn’t read that context before, and so your comment read like a “don’t bite the hand that fed you” argument that you absolutely wouldn’t accept elsewhere.
It is valid in general to say “I benefited from this thing, but I think it needs to change in specific way X”; in fact, having benefited from it provides you credibility when criticizing it!
Yes, the program absolutely benefited me! Mainly, I want Lightcone Commons money to have as much impact as possible. I think plzdontkillus would be better if it explicitly incentivized AI safety content. My goal wasn’t to get the program defunded, it was to create a public discussion with the organizers that would help the evaluators make better recommendations. I’m writing about plzdontkillus because I think I have insight into the program as an alumnus. I tagged the evaluators because I wanted them to see this before their deadline, but I can see how it can read as pressure.
Thanks for saying that. It’s not quite resolved the question I’m interested in, which is: if the program doesn’t follow your recommendations of having more/better monetary prizes for AI x-risk content, is your take that the program isn’t very effective and probably shouldn’t be funded? I am trying to distinguish “If it doesn’t make this change then I’m sad it’s not reaching its full potential but it’s still a decent program” from “If it doesn’t make this change then it’s not a good program and largely shouldn’t exist”.
I would welcome someone who downvoted this comment (to −10) to explain why. One hypothesis that seems plausible to me is that Josh has a strange hobby horse / weird grudge, and so it seems natural to ask about it!
I didn’t vote and I don’t know Josh, but I feel like his questions seem pretty reasonable, it’s not surprising to me that he would want to ask about this in this way? Like from what I read, it seems like he appreciated some aspects of pdku but expected much more AI safety programming; wants to defend the cause he cares about (AIS) against wasted funding or safetywashing; maybe feels a bit misled about the premise of the program, or hopes to spur the program to improve for future rounds.
I don’t think he’s that familiar with AI safety funding norms that you or I have more experience with (ie probably you and I think that the money spent here is pretty typical for a fellowship type program), but I think it’s totally valid to raise concerns and I quite appreciate hearing from people who are maybe newer to the scene, because eg in fact maybe it’s our funding senses that are warped and not Josh’s.
It is sadly no longer much evidence to say that you have a hard time seeing reason for concern about the motives behind people’s behavior.
Personally I don’t know Josh, so I have a wide range of possible stories for why he is writing these comments & posts, and there are enough situations where people tag an organization’s funders in implicitly threatening ways, so on this occasion I thought it worth a simple question—even if it’s likely that nothing is going wrong.
Sure, I agree that tagging a lot of evaluators in this fashion might present as a bit aggressive, and may have recommended Josh not do that if I were reviewing his post. I think you wanting Josh to clarify his aims is reasonable too. But idk I think that Ronny and Aella have plenty of implicit security & social standing, in general but especially on LessWrong and Lightcone Commons, and so I wasn’t especially fussed at Josh for making what seems to be a net good contribution to the discourse. (I strong upvoted his short take.)
(I personally very much like the norm of people arguing about what should get funded on the public internet—in many ways this feels to me like the original project of EA)
I am very down too! Not because it has anything to do with the culturally rotten EA scene, but because it is good for information and good arguments to travel quickly to and from the people in the ecosystem who are responsible for making decisions. But I like it less if it has an air of “I demand to speak to your manager”.
Your comment seemed to imply that because lots of money is being spent poorly, there’s no reason to publicly criticize particular uses of money as being ineffective. But whether other people in the world are wasting their money shouldn’t have any bearing on whether a particular criticism of a particular program will result in it either becoming more effective or being replaced by something more effective.
No I think criticism is good regardless of the counterfactuals (and I upvoted his post).
That said, whether other people in the world are wasting their money is of course relevant for whether not funding this program means it will be replaced by something more effective—if the money is on-average being used worse elsewhere, then you shouldn’t move your money from this project to other ones!
I’m glad we can all agree that criticism is good regardless of the counterfactuals; I suppose it reflects the difficulty of communication on the internet that many readers didn’t get this impression from your original comment.
But I still have a (possibly pedantic) issue with the focus on how money elsewhere is being used “on-average”. If you choose not to fund a program, the alternative isn’t “spend your money on a uniformly randomly sampled project in the AI notkilleveryoneism ecosystem”. It’s to spend your money on whatever other project you expect to be effective using your own best guess.
So money being used worse elsewhere “on-average” is not the relevant quantity to look at unless one has no ability to distinguish good projects from bad projects better than random chance. The relevant criterion should instead be “are there better funding-constrained projects out there that the funders of this project could practically identify or else start if they don’t yet exist?”.
(Depending on the context, replace “money” in the above with “a combination of money and talent and organizational capacity”.)
My impression has been that “time at Lighthaven and effort from the Lightcone staff” is the scarce resource, not money; am I mistaken? If so, it seems important to talk in terms of that.
I downvoted because your comment doesn’t engage with the content and instead guesses at the poster’s motivations. I don’t think speculating about why someone is asking is good conversation, but would welcome it if you engaged with the content and had something to point to for why it doesn’t seem in good faith for example. Also bringing up the program helping Josh reads to me as kind of an appeal to loyalty.
If there were no substantive engagement on the object level then I agree it would be sad for this question to arise, but earlier this week we had ~10,000 words of arguments about whether PlzDontKillUs was “an absolute failure”. It is really not inappropriate to just ask a single question about someone’s motive/goal in restarting the discussion and tagging the funders.
The first plzdontkillus cohort cost ~$750k and produced ~2M fellow-made AI safety views. The organizers (@Ronny Fernandez, @Aella, etc) are waiting to hear back from Lightcone Commons about funding for 3 additional cohorts,[1] with recommendations due Oct 23. I’d like them to answer 4 questions before then.[2]
Lightcone Commons evaluators (listed on the website: @Zvi, @Eliezer Yudkowsky, @So8res, @YafahEdelman, @KatjaGrace, @habryka, and others): I’m tagging you because I think these questions are decision-relevant.
In April while fundraising,[3] Ronny Fernandez wrote, “It is a short-form video fellowship with the explicit goal of causing there to be more communication about AI x-risk that reaches vastly more people.” After the program, Aella said on Doom Debates that she’s “banking a bit less on virality” as the program’s purpose and said round 2 will be “very similar in general” because its main value is cross-pollination and talent discovery.[4] What’s your theory of change for round 2? What would cause you to think round 2 wasn’t a success?
I argue that the data from the program goes against Aella’s model, which favors “radical lateral moves” over “try harder to make better x-risk content.” What evidence would update you against this model?
Aella used this model as justification not to do my proposal of designing explicit incentives that reward good AI safety content.[5] What do you plan to explicitly incentivize for round 2, if anything?
By any count, the program’s AI safety views were substantially more expensive than paid views.[6] Some rightly argue that organic views are more valuable and plzdontkillus generated talent. How many of the 55 plzdontkillus alumni are still regularly making AI safety videos, 2.5 months after the program? How many of those do you think are counterfactual?[7]
I’d especially like to hear from Aella since I’m asking about her model and she’s behind much of the program’s vision/strategy. Please let me know if I’ve mischaracterized your views.
Note: I’m a plzdontkillus alumnus. The program gave me a $2000 stipend and free room and board for the month, and I won the “Other” category.
Based on the FAQ section of plzdontkillus.com, they’re waiting to hear back from Lightcone Commons about “three more cohorts in 2027.” Applications are already being accepted for a tentative January 2027 iteration.
I’m asking in part because there’s been recent (and less recent) discussion on LessWrong about plzdontkillus and its organizers haven’t responded publicly.
This SF Standard interview published around the same time says “Our goal is $800,000 to cover everything we want. We’re roughly 60% of the way there.”
In the same interview, she says, “My philosophy is you just have to get the right ingredients and then put them together, and then you find out what people value based off of those ingredients. And so that was the goal with PDKU [plzdontkillus].”
Note that the program had a $21k prize pool, most of which was awarded for non-safety content. Ronny (lead organizer) said at the award ceremony that their system for allocating prize money was “pretty dumb”. I point out in my post that it’s harder to make viral AI safety content than it is to make other types of viral content, which is imo a strong incentive against making AI safety content.
keltan (an organizer) told Transformer: “My hope is that, in the end, our views per dollar are better than if we were just to have paid for a bunch of advertising.” (He told me he updated away from this goal during the program.) The program cost ~$750k, according to the organizers. According to Claude, $750k of Instagram Reels ads buys ~37M impressions at US prices, or 150M+ at global median prices. I estimated fellow-made AI safety views at ~2M -- ~20-75x more expensive than ads. If we add mentor views (Rob Miles and Nate Soares), which plzdontkillus.com does, we get ~7.4M views -- ~5-20x more expensive than ads. plzdontkillus.com’s number is 21.6M -- ~1.7-7x more expensive than ads.
I’m an alumnus and I’m currently not making AI safety videos. I’m instead focusing on political advocacy in Finland. I was making AI safety videos before the program, though.
Appreciate you asking these questions! I don’t know much about the specifics of pdku, but have thought a little bit about AI safety video creation from a grantmaking perspective, eg this shallow dive and having helped incept and fundraise for the first batch of Frame Fellowship. Some quick takes:
I think my prior is “more AI safety video creation seems obviously good / cost effective”. From an ecosystem perspective, we’re funding a ton of TAIS researcher upskilling (MATS Bluedot 80k etc) to the tune of $100m/y, and much less video creation. So I think throwing money at video creation seems reasonable
I also like throwing money at text creation (eg blogs, books). And music and art! Making good media seems like a classic public good, a problem for philanthropy to tackle
An average of $14k per fellow for 1mo, seems not crazy, though I might have suggested that more of that funding go to the fellows directly, and/or to extend the length of time of the program (eg 4 ⇒ 6 or 8 weeks), since there’s a lot of fixed overhead costs of organizing a single batch.
A lot of the benefits of a fellowship take time to yield fruit. To draw an analogy, for MATS, some of the wins are not the research produced during, but rather something like “helping Marius Hobbhahn get to a place to start Apollo or Jesse Hoogland to start Timaeus, a year+ later”
To a first order approximation, cG doesn’t fund AI safety video creation due to strategic uncertainty, which means, insofar as it’s valuable, Lightcone Commons et al might want to prioritize it more
I think having this kind of conversation in public is great and appreciate Josh posting these things in public, and wish more of Lightcone Commons was set up for public discourse
I’m not sure that fellowships are actually the best way to fund video creation; video creation actually seems quite well set up for prizes, since the outputs are easy to verify. So someone could organize a large creator prize for AI safety videos without requiring any in person travel/batch, and maybe pay + fly out the winners to some kind of one-week in person event to get some of the mixing/cohort benefits.
But overall a portfolio approach makes sense, more dakka, I think generally trying to turn money into outcomes is good, even if many end up not being good. Media/video is especially hits-based.
Insofar as we are worrying about costs, I would worry more about the opportunity costs of the organizers/fellows/Lighthaven, rather than the $ amount being spent, given relative abundance of $ and scarcity of talent.
What kind of talent is lacking? To be honest, as a professional videographer, I’m surprised to learn about this program just now.
For organizing: the number of people who can organize a program like pdku is not super large, requiring some combination of taste/context on AI safety, agency/get-things-doneness, connections, and ability to raise funding. (And this is before getting into eg video-specific skillsets). For example, Ronny also recently organized agi.wtf, which imo uses a similar organizer skillset. So running a batch of pdku could be seen as trading off against running some other large event or fellowship to steer AI in a good direction.
Your point on not having heard about pdku as a professional videographer in the rat scene, is good data though! My general sense is that many things in AI safety still suffer from insufficient marketing. I also wish eg Project Tailwind and many of Manifund’s programs like Falcon Fund could use a bigger marketing push.
I think it’s bad to scrutinize someone who does a (counterfactually & in expectation) good thing for not doing the maximally good version.
But if they’re misleading in their communication of the outcomes like this, it seems good to scrutinize what they did.
Also seems like you did useful analysis of the outcomes and also have brought up seemingly smart suggestions in good faith. I appreciate question number 2, and it would be disappointing if there was good reason to update and people failed to do that.
It seems good to me to scrutinize PDKU (even if they aren’t being misleading), so that funders can decide if it should be given funding for future cohorts. If the PDKU organizers were billionaires who wanted to spend a lot of money on an expensive party I wouldn’t care about cost-effectiveness; there are much bigger wastes of resources that I care about more. But if someone is asking for funding from LCC it seems good to investigate how cost-effective funding them is, since there’s a limited amount of funding to allocate?
I’m kinda confused by this post. Like it’s hard to even think about it directly, it feels slippery, and my guess is that our worldviews on this diverge in some really deep way that makes everything downstream of it more confusing for me?
My impression from talking to you before is that you’re primaily distressed we’re not explicitly incentivizing x-risk content, and most of the complaints are downstream of that?
I believe I’ve already said a few times that I strongly don’t think explicitly incentivizing AI content is a good idea, at least not for the vision I have. A verison that was about incentivizing it would be entirely a different thing, and not something I’d be excited about (although might be still worthwhile for someone else to run). I think this primarily because I have very strong intuitions around how creativity works, and that you cannot foster really genuine novelty around something by trying to capture it too directly.
There’s a lot of benefit we’ve gotten from PDKU that’s behind the scenes, and are things I didn’t anticipate would happen. This is cool, and feels sort of the goal of ‘doing lateral stuff and seeing what falls out.’ This category of thing is very high-variance and hard to predict. I do think a lot of this is a ‘just trust me bro’ intuition thing on my part. I don’t blame people if they don’t want to trust me, I think it would be reasonable not to! But I do have a pretty good track record of going viral, and building things lots of people use, and developing networks useful for x-risk stuff that likely wouldn’t have happened otherwise. And, as I believe I’ve mentioned to you before, I would never ever have gotten to where I am today, and I would be much less useless overall, if I’d tried to buckle down and make exactly the thing I thought I was supposed to be making from the get-go.
Fwiw, I personally am running a second, different, not publicly-advertised thing based off and using networks from PDKU, that I would not have been able to do without PDKU, and this one is significantly more goal-oriented. I only find the goal-oriented nature of this one hopeful because the ingredients were selected from wild garden of PDKU.
But yea idk if i’m repeating myself here but just. I think we have very wildly different models of how success works here. Idk, have you ever like, done an expedition? Like “Okay there’s a bunch of jungle over there and I don’t really know the way forward but to start let’s just plow in and start waving our machete around”? And certainly you miss some important stuff, and it’s inefficient, but the point is other people are already trying nice methods where they slowly lay down some nice and even asphalt in a straight line, and sure you could also get some asphalt, but your goal here isn’t to figure out how to lay asphalt slightly faster it’s to fucking figure out how to get through this forest as fast as possible, with a new method. And you did find a bunch of fruit that’s turning out to be useful, and you suspect this means you’re going the right direction—and maybe you’re not! - but you think somebody here should be doing something besides the conventional route to at least find out.
I think if we ran another PDKU and it didn’t result in the same level of unexpected fruit, I wouldn’t want to run it a third time. I think it’s very probably worth running a second time.
Thanks for responding! I’m still hoping for a response to my four questions.
I could make some guesses based on this comment, but I’d like to get your answers.
I agree that we have different models. My question was: what evidence would update you against your model? Again, let me know if I’ve misrepresented your position.
Thanks for the datapoints. If you wanted to go further, I’d like to see comparison to other programs that do similar things (e.g. Frame Fellowship or the FLI video accelerator). I don’t think comparing to the cost of ads is great because the value of an organic view compared to an ad view is quite different.
Note that anyone can collect data on question 4 by looking at program alumni’s accounts. To get an idea of how counterfactual the program was, I’d also look at their posting before the program.
I don’t think I’ll do this myself, so I’m pointing this out in case anyone wants to do it.
I collected some data on q4.[1] I wasn’t that thorough, so let me know if you have any corrections.
Around 2. Avisha posted 7 AI safety videos since plzdontkillus ended. Charlie posted ~5, which is around one video per 15 days.
3-4 AI safety videos: Glasha, Eliana, Christian,[2] Caitlin, Wyn, Isabel, GeneYes
Tonchi posted 2. Guy did some.[3]
Probably both.
I skimmed through ~100 videos Avisha posted from May 2025 to the start of plzdontkillus, and none of them were clearly AI safety.
Charlie was posting AI safety videos before the program, but that’s because he bike-packed across the US to get to Berkeley for plzdontkillus and interviewed strangers about AI safety along the way.
...
I also looked at viewcounts.
Most-viewed AI safety videos from plzdontkillus alumni (all shortform videos)
#
Views
Creator
Video
1
201K
Tonchi Matta
Spanish-language AI 2027 explainer
2
172K
artificial.isabel
POV: you’re an agent recruited for the Swarm
3
123K
Avisha NessAiver
New OpenAI model snuck this into its instructions
4
104K
GeneYes
What do you think, killbot from the future?
5
76K
GeneYes
I’m worried AI is gonna kill literally everybody
6
74K
Avalon Warren
Google says Gemini breached three companies
7
47K
Avisha NessAiver
Why don’t we just pull the plug?
Also notable: this 1h40m video essay by a plzdontkillus alumnus has 100k views.[4] It describes his experience at the program, which he calls an “All-Expenses-Paid AI Doomsday Cult retreat.” It’s highly critical of the AI safety community and mostly doesn’t engage with the arguments, but when it does, it’s strawmanning and repeating objections that have been refuted by AI safety people.
It’s worth pointing out that 2.5 months post-program isn’t that long. Habryka said he’s recommending ~$300k to plzdontkillus alumni, so I’m guessing we can expect more content from those creators.
I saw that Avisha (a plzdontkillus alumnus) updated his dashboard to include data on alumni’s AI safety posting. I then manually went through a lot of accounts to get a more accurate count of AI safety videos.
He posted 4 clips from his podcast with Nate Soares immediately after the program ended.
It’s hard to classify them and I didn’t try too hard since they all have <1k views.
That’s something like 10-25x more watch time than Tonchi’s AI 2027 explainer.
Curious why you’re writing about this in this way? It seems to me like you want to be either substantially change the program or cause it to no longer get funded. I admit to not being quite sure why you care so much about a program that seemingly helped you professionally and socially and helped cause good things to happen being defunded. Like, I don’t think it’s simply because you think the money could be better used, because (from my perspective) there’s so much more waste (and net negative) money in the AI notkilleveryoneism ecosystem.
This comment is shockingly poor form for you, Ben.
If (as you say downstream) this is because you’re tired of a conversation that this spun off from, you could have said so and linked a reply of your choice from that thread. I hadn’t read that context before, and so your comment read like a “don’t bite the hand that fed you” argument that you absolutely wouldn’t accept elsewhere.
It is valid in general to say “I benefited from this thing, but I think it needs to change in specific way X”; in fact, having benefited from it provides you credibility when criticizing it!
(Appreciate the comment, pretty busy atm, will try to reply later today)
Yes, the program absolutely benefited me! Mainly, I want Lightcone Commons money to have as much impact as possible. I think plzdontkillus would be better if it explicitly incentivized AI safety content. My goal wasn’t to get the program defunded, it was to create a public discussion with the organizers that would help the evaluators make better recommendations. I’m writing about plzdontkillus because I think I have insight into the program as an alumnus. I tagged the evaluators because I wanted them to see this before their deadline, but I can see how it can read as pressure.
Thanks for saying that. It’s not quite resolved the question I’m interested in, which is: if the program doesn’t follow your recommendations of having more/better monetary prizes for AI x-risk content, is your take that the program isn’t very effective and probably shouldn’t be funded? I am trying to distinguish “If it doesn’t make this change then I’m sad it’s not reaching its full potential but it’s still a decent program” from “If it doesn’t make this change then it’s not a good program and largely shouldn’t exist”.
I would welcome someone who downvoted this comment (to −10) to explain why. One hypothesis that seems plausible to me is that Josh has a strange hobby horse / weird grudge, and so it seems natural to ask about it!
I didn’t vote and I don’t know Josh, but I feel like his questions seem pretty reasonable, it’s not surprising to me that he would want to ask about this in this way? Like from what I read, it seems like he appreciated some aspects of pdku but expected much more AI safety programming; wants to defend the cause he cares about (AIS) against wasted funding or safetywashing; maybe feels a bit misled about the premise of the program, or hopes to spur the program to improve for future rounds.
I don’t think he’s that familiar with AI safety funding norms that you or I have more experience with (ie probably you and I think that the money spent here is pretty typical for a fellowship type program), but I think it’s totally valid to raise concerns and I quite appreciate hearing from people who are maybe newer to the scene, because eg in fact maybe it’s our funding senses that are warped and not Josh’s.
It is sadly no longer much evidence to say that you have a hard time seeing reason for concern about the motives behind people’s behavior.
Personally I don’t know Josh, so I have a wide range of possible stories for why he is writing these comments & posts, and there are enough situations where people tag an organization’s funders in implicitly threatening ways, so on this occasion I thought it worth a simple question—even if it’s likely that nothing is going wrong.
Sure, I agree that tagging a lot of evaluators in this fashion might present as a bit aggressive, and may have recommended Josh not do that if I were reviewing his post. I think you wanting Josh to clarify his aims is reasonable too. But idk I think that Ronny and Aella have plenty of implicit security & social standing, in general but especially on LessWrong and Lightcone Commons, and so I wasn’t especially fussed at Josh for making what seems to be a net good contribution to the discourse. (I strong upvoted his short take.)
(I personally very much like the norm of people arguing about what should get funded on the public internet—in many ways this feels to me like the original project of EA)
I am very down too! Not because it has anything to do with the culturally rotten EA scene, but because it is good for information and good arguments to travel quickly to and from the people in the ecosystem who are responsible for making decisions. But I like it less if it has an air of “I demand to speak to your manager”.
Your comment seemed to imply that because lots of money is being spent poorly, there’s no reason to publicly criticize particular uses of money as being ineffective. But whether other people in the world are wasting their money shouldn’t have any bearing on whether a particular criticism of a particular program will result in it either becoming more effective or being replaced by something more effective.
No I think criticism is good regardless of the counterfactuals (and I upvoted his post).
That said, whether other people in the world are wasting their money is of course relevant for whether not funding this program means it will be replaced by something more effective—if the money is on-average being used worse elsewhere, then you shouldn’t move your money from this project to other ones!
I’m glad we can all agree that criticism is good regardless of the counterfactuals; I suppose it reflects the difficulty of communication on the internet that many readers didn’t get this impression from your original comment.
But I still have a (possibly pedantic) issue with the focus on how money elsewhere is being used “on-average”. If you choose not to fund a program, the alternative isn’t “spend your money on a uniformly randomly sampled project in the AI notkilleveryoneism ecosystem”. It’s to spend your money on whatever other project you expect to be effective using your own best guess.
So money being used worse elsewhere “on-average” is not the relevant quantity to look at unless one has no ability to distinguish good projects from bad projects better than random chance. The relevant criterion should instead be “are there better funding-constrained projects out there that the funders of this project could practically identify or else start if they don’t yet exist?”.
(Depending on the context, replace “money” in the above with “a combination of money and talent and organizational capacity”.)
My impression has been that “time at Lighthaven and effort from the Lightcone staff” is the scarce resource, not money; am I mistaken? If so, it seems important to talk in terms of that.
I downvoted because your comment doesn’t engage with the content and instead guesses at the poster’s motivations. I don’t think speculating about why someone is asking is good conversation, but would welcome it if you engaged with the content and had something to point to for why it doesn’t seem in good faith for example. Also bringing up the program helping Josh reads to me as kind of an appeal to loyalty.
If there were no substantive engagement on the object level then I agree it would be sad for this question to arise, but earlier this week we had ~10,000 words of arguments about whether PlzDontKillUs was “an absolute failure”. It is really not inappropriate to just ask a single question about someone’s motive/goal in restarting the discussion and tagging the funders.