Running Lightcone Infrastructure, which runs LessWrong and Lighthaven.space. You can reach me at habryka@lesswrong.com.
(I have signed no contracts or agreements whose existence I cannot mention, which I am mentioning here as a canary)
habryka
Karma: 62,514
Yeah, my sense is long term service is really mostly like working at a large corporation, but early boot camp experiences are very cult like.
Also plausible this changed over the years.
Academia seems just really not that similar to anything I would call cult. The standard experience of anyone I know in academia is overwhelming loneliness, almost complete cynicism about the shared mission by almost everyone involved, and an almost complete absence of guidance on how you should structure your life.
Like, academia these days strikes me as mostly similar to working at a very large, very professional, badly paid organization.
And then in addition to that you actually do have one of the strongest cultural anchors of any culture in the world in rewarding genuine disagreement. It’s not perfect, but academia is much higher on contrarianism and status mobility. It’s also very low on controlling your public speech compared to anywhere else.
Like, I have very large disagreements with academia, and am not in love with it, but it strikes me as one of the least cult-like social systems out there. Almost every large corporation seems more cult-like to me, by almost any reasonable definition of the word. Startups are much more cult-like.
Right now, the single most important load-bearing argument in alignment Risk Reports is that models still lack the covert capabilities necessary to do sophisticated sabotage attempts in such a way that they wouldn’t get caught by auditing and/or monitoring.
I am confused, didn’t we just experience multiple high-stakes failures where models failed to get caught by auditing and/or monitoring? Those do seem more competence related, but like, I am failing to see how those arguments could still be considered valid, given the fact that both Anthropic and OpenAI are obviously failing to adequately audit or monitor their systems to avoid catastrophic failure.
academia
What? Academia really very obviously doesn’t seem like a cult to me, and I have thought about this topic a lot. This makes me think you are using words very differently from how I would use them (and to be clear, in a way I think is pretty bad for discourse, since it makes it much harder to discuss actual cults).
I would invert this point! If someone speaking about cults actively thinks that academia is a cult, I wouldn’t trust them! (The military in contrast seems indeed very cult-like to me, and I don’t have any objection to using cult concepts to describe that)
Yeah, I remember reading it. I gave it a small-upvote. I think it’s a reasonably point but at least for me doesn’t really feel like it explains much of what seems to me most important about navigating these kinds of situations, though it does seem like a relevant piece of the puzzle. I would be surprised if it meaningfully “resolved” the debate, or played that large of a role in things dying down, but I wasn’t around at the time, so am not confident.
It is nice that I’m not listed anywhere in that system
You are, it’s on page two of the “Authors with Banned Users” block:
our algorithm should ignore all text in LLM-marked blocks when deciding whether to accuse a post of too-much-LLM-text
This one.
Hmm, fair point. Most of my experience has been making grants, not paying people in India for services. Paying people might be a lot easier. I’ll do some research and consider widening our restrictions here.
Anyone can post LLM conversations, you just have to wrap them in the LLM content block.
But it is at least it is a bug to accept edits (which consume time) and then refuse to let them be published (with no warning or explanation of this)? It isn’t predictable (and predictability is a key desiderata in a Justice System) and it doesn’t teach (though maybe they assume incorrigibility in humans by default and have given up on hoping for understanding and then repentance and then improved behavior)?
Yeah, this could be better. It’s intentional for us to not accept edits to rejected posts, because the edits wouldn’t cause a post to be re-evaluated, but the current experience is pretty miserable.
I think we’ll have to redo some of our internal systems here. The auto-rejections are built on top of the system that we built for manual rejections, and in those cases it made sense for the rejection to be final. But this makes less sense when it’s an automated system, and when the rejection reason is easy to fix.
I am pretty sure we are correctly filtering out any LLM-written content in LLM-content blocks. But if we aren’t, we should definitely fix it!
But for new matching grants, will this mean that Lightcone Commons won’t actually enforce nonprofits showing that they actually received a matching grant?
Honestly, I don’t really expect people to apply for matching grants, since that whole concept feels a bit confusing. But if they do, we can make sure we only give them the money in matching amounts.
Good enough, edited!
Ah, yes, that sounds very similar to what I heard. I reference that in this section a bit, so I think I shouldn’t put it back:
Third, I expect to rebuild the tools for getting AI systems to assist with evaluations from the ground up, mostly just because I don’t like the current rubric-based system and want to give evaluators more freedom on how to leverage AI systems in evaluating grants.
Appreciate the clarification!
My understanding is:
Every evaluator is supposed to skim/familiarize themselves with every application
Every application gets investigated by at least one evaluator (where investigation involves writing at least some notes, and at one point involved giving some positive or negative feedback)
I think this is reasonably well described by “evaluators need to look comprehensively at all applications”, but happy about rewordings that are clearer to you.
Yep, feel free to link post!
It’s marked as personal blog because it’s an organizational announcement, which we mostly keep off the frontpage. See Site Guide: Personal Blogposts vs Frontpage Posts.
Oh cool, that appears to be a recent change! The 2025 FAQ indicated more strongly that speculation grants are required. I am glad recommenders can now override this default if they want.
The website does still say:
There are of course always escape hatches, and I am glad they exist, but I wouldn’t currently feel comfortable telling anyone that receiving speculation grants is optional in any meaningful way.
I’ll change the wording to be a bit weaker though and say “first receive a speculation grant in order to be evaluated in a grant round on the same basis as other applicants”.
Yep, “look” here was in reference to that required skimming/familiarization. At least historically it also involved every application getting assigned to at least one evaluator to investigate. I agree that not every evaluator is supposed to evaluate every application, which would really be very prohibitive.
Oh, I was told this from a recent recommender, but I possibly was wrong! I’ll remove it for now and ask for follow-up on what they meant.
This video showed up in my Youtube recommendations today, and IMO is quite good coverage of what early military education looks like, which yes, really strikes me as obviously extremely cult-like:
(The title and thumbnail is more clickbaity than the actual video, which has a lot of really great footage and mostly measured analysis of what’s going on in these bootcamps)