neo

Karma: 126

neo 20 Mar 2026 19:04 UTC
1 point
0
in reply to: ChristianKl’s comment on: neo’s Shortform
I think the disagreement stems from a lack of specificity on my part; ignore the specific description of the categories.
Probably, you are in soldier mindset yourself about this very issue.
I hold beliefs on it, sure. I am now interested in seeing if they reflect reality, and learning why/why not. Is this mindset inadequate, and what would make it more rational?
Separately—do you think there is promise in tools of the type I describe to combat soldier mindset at scale? I will definitely be reading into some of the CFAR resources, just curious to hear from you.

neo 20 Mar 2026 18:23 UTC
1 point
0
in reply to: ChristianKl’s comment on: neo’s Shortform
Certainly agree with your point about donating to Substacks / journalists. Could be very impactful to have a writeup of that somewhere here or on the EA forum.
I’m familiar with Galef’s ideas; I would place “soldiers” in category 2. But yes, the distinction is very subtle and I did not specify it well enough.
I believe that sufficiently well designed UI for navigating debates/arguments/discussions can make it very difficult for people to disguise soldier mindsets via obfuscated (intentional or unintentional) communication and reasoning.
Imagine, for example:
- User creates a strongly worded post that features a clear strawman and/or blatantly skips over serious, in-depth prior discussion on the same topic.
- An LLM categorizes the argument to properly situate it within prior discussion and notifies the user that they A) do not appear to have an accurate understanding of the original source—specifically pointing out why B) have not yet explored the X counterarguments coming after that line of reasoning, and the Y that come after that.
This could be seen as an enhanced version of “community notes” aimed at situating shallow, under-researched takes within a larger “map of human thought.”
Whether this can scale and outcompete current systems is unknown, but it does truly seem promising for the enhancement of public discourse and like a step in the right direction.
Appreciate the comment, was helpful in clarifying my thoughts.

neo 20 Mar 2026 17:14 UTC
1 point
0
in reply to: TsviBT’s comment on: neo’s Shortform
Hmm, this does look interesting but I hadn’t really considered depth of 1-on-1 communication as a significant bottleneck. I also think the concept slightly falls apart when the users are not already knowledgeable / quick thinking / good at rigorous communication, as I’d guess there would be a steep learning curve.
Software wise, I think I’m aiming closer to better debates and debate tools, mainly because these things could be made public-facing and seem like an obvious use case for even present-day LLMs.
If you have any further thoughts regarding implementation of those things, I’d be eager to know. I’ll be trying to make my plans more specific.

neo 20 Mar 2026 17:03 UTC
2 points
1
in reply to: papetoast’s comment on: LessWrong’s UX may not be living up to its ideas
even with perfect onboarding someone would still need to motivate themselves to read quite a large chunk of text to reach the pareto frontier.
Sure, I just don’t think this is a big issue because people bottlenecked by motivation probably won’t be serious contributors, and it’s worth improving things for those that will.
I think the main bottleneck isn’t tooling, it is how much time people are spending to make things legible for other people/newcomers.
Yep, I’m pretty much in agreement with your broader point here – most of this stuff fails at the user level. It seems like manually organizing content would be a good place to start for me.
At the same time, I want to explore how this can become a self-propelled process in the future. It seems like an obvious use case for LLMs, and presumably not a massive engineering task.
I appreciate the comments – hopefully you can weigh in as I continue looking into it.

neo 19 Mar 2026 21:39 UTC
4 points
1
in reply to: Daniel Kokotajlo’s comment on: On restraining AI development for the sake of safety
I think Joe is arguing that unilateral capability restraint might exacerbate such risks.
For example, to the extent that safety-related considerations end up motivating or rationalizing especially drastic forms of international action aimed at shutting down or significantly restricting AI development in other countries, I think this could well be harmful even relative to more baseline forms of economic and military competition.
Like, the US government is all in on safety but China is not complying. Which would probably get intense, and rightfully so.
I still agree with you because:
- I don’t think that is very likely at all; if one country/coalition is all-in on safety, I’d guess there is something motivating their fear that would apply to everyone else as well.
- The default outcome of nobody being all-in on safety seems obviously much worse as you point out.
- There is an argument to be made that great power conflict is justified in the credible expectation of total annihilation otherwise. Obviously this is extremely controversial as it can be easily misused, and IIRC has been used to deface the EA/rationalist community (or Nick Bostrom?) in the past, but I don’t think that takes away from it’s merit.

neo 19 Mar 2026 21:29 UTC
8 points
0
on: Broad Timelines
This previous LessWrong article seems extremely relevant and basically sketches out an example of the rough “strategic portfolio” for AI risk that you are arguing for.
In line with some of my recent posts, I’m starting to think there is a lot of value in:
- Clearly defining consensus group strategy (among LW/EA/CG, for example) on “making the future go well.” This should include rough estimates from a variety of respected sources, a diverse portfolio of interventions, and explicitly communicated uncertainty / epistemic humility.
- Designing info-UI tools to facilitate that process. Enabling effective deliberation, strategy adjustment, and maybe most importantly: easy-to-use interfaces for the general public. The goal being to convey community beliefs and disagreements in a very transparent and easy to understand way.
- - This is intentionally unspecific but I have outlined a couple particular ideas in previous posts and will continue to crystallize my suggestions / explain why I think this area has potential.
The AI futures model and related ecosystem is a great start but is limited to a handful of thinkers (Daniel and Eli) and a specific subset of information (forecasting timelines). Their work has already been quite impactful (read by JD Vance apparently) – why not work hard to apply and scale good information-interface-design to broader community strategy?

neo 19 Mar 2026 19:19 UTC
18 points
5
on: neo’s Shortform
What are some research directions for “improving coordination?”
In light of a recent post and comment, and several months of thinking, I have come to the position that one of our (humanity’s) biggest problems is that we suck at precise coordination at every level.
This is not very specifically defined but I am trying to gesture at a problem area I think is super important. Some thoughts to convey my intuition here:
- If the extreme risk of the AI development trajectory is as true and obvious as many believe (everyone’s life at risk), humanity’s thinking about it should appear a lot more sophisticated than it does now.
- For the last few years Eliezer has basically been throwing his hands up in exasperation at the incompetence of the world and many have shifted to public-facing communication, presumably believing that trying to convince AI insiders is hopeless.
Broadly, I think there are two cases of problems with coordination:
1. Two people/groups genuinely agree to honest, rigorous exchange of information, but can’t effectively coordinate.
2. Someone is withholding information or doesn’t really want to coordinate in the first place.
I think the first problem is workable, and if improved sufficiently, makes progress on the second problem by clearly exposing parties that are avoiding productive exchange.
- Specifically, I think there is a lot of progress to be made with augmenting the exchange of information between people. I think LessWrong, the knowledge commons arguably at the frontier of ensuring humanity’s survival, is lacking in features for this purpose. Maybe because most users here are already conscientious and strongly value truth-seeking, which makes improvement seem less necessary.
Hopefully I’m making this line of thought clear enough. Key points:
- Trustworthy, robust, and future-proof governance is the ultimate problem for humanity and anything else is a band-aid on a bullet hole.
- Highly effective coordination is part of that problem, and better information exchange/clarity is a subset of that.
- - I think LessWrong can become an exceptionally effective prototype of this, and this could be very high leverage because of how proximate it is to the frontier of AI. Happy to expand more here.
I am interested in situating my thinking better here. Who is working on this sort of thing? I know TsviBT has explored improvements to debate, Richard Ngo / Samo Burja are exploring broader political manifestations, Forethought has published adjacent work. Is there anything I’m missing? Very interested in contributing here and think it’s a clear place where the ball is being dropped.

neo 19 Mar 2026 17:51 UTC
3 points
2
on: On restraining AI development for the sake of safety
What I’m gathering from this is:
- The current trajectory of AI development is reckless and we need more time.
- But it’s way more complicated than that.
It seems like this all ultimately becomes a matter of designing an effective, robust, future-proof institution aimed at guiding humanity to a good future. Similar to what the Founding Fathers attempted with the United States, but with enough foresight to account for huge and unpredictable changes in technological capability.
And then, a matter of getting powerful people on-board, which probably means showing everyone that there are a million ways for humanity to destroy itself^[1] if our process of navigating the future isn’t done with extreme care.
Why can’t we start by agreeing on this relatively simple message? Is there some way to get the AI community to at least come together here, despite all other disagreements? Does it not greatly frustrate everyone else whenever important AI-leaders recycle the same vague oversimplifications^[2] of what is literally humanity’s ultimate problem?
I feel like I’m going a little crazy here. Everyone gestures at different research directions they find promising but it seems like everything eventually falls apart^[3] when you recognize that we are on a planet run by apes that will soon wield self-destructive power. At the end of the day, effective political governance and coordination are a pre-requisite to the survival of our species and it does seem like this is incredibly neglected.
Perhaps there is hope if frontier labs, the members of which may become extremely wealthy and powerful, start actively planning for and prioritizing this process of designing–with much deliberation and careful philosophical consideration–the successor governance system for the human race. What can we do to make that more likely?
1. ^
  We probably do have to worry about cultural drift and other “future issues” quickly made salient by AI-driven acceleration.
2. ^
  Dario says “Humanity is about to be handed almost unimaginable power, and it is deeply unclear whether our social, political, and technological systems possess the maturity to wield it.” And: “The years in front of us will be impossibly hard, asking more of us than we think we can give.”
  
  So, what are we doing about that? Have there been any attempts at creating or investing in institutions that try to figure this sort of thing out? As far as I can tell, AI governance is a fragmented, not-very-rigorous field of study which just seems completely backwards, and something that we could at least be trying way harder to solve.
3. ^
  Including alignment. For God’s sake, what/who are we even aligning AI to? The world still ends if everyone has an obedient super-intelligence in their pocket.

neo 4 Jan 2026 22:36 UTC
3 points
0
on: In My Misanthropy Era
Regarding traits you love – maybe you are looking for something like intellectual humility? I think it can naturally follow from kindness and cooperativeness, but is often necessary for me to respect an intelligent person.
It also seems like a core principle of this community, where as some say, “we gain status by pointing out where others haven’t been careful or skeptical enough in their thinking.”

neo 23 Sep 2025 20:58 UTC
7 points
5
in reply to: Vladimir_Nesov’s comment on: Global Call for AI Red Lines—Signed by Nobel Laureates, Former Heads of State, and 200+ Prominent Figures
Agree; I’m strongly in favor of using a term like “disempowerment-risk” over “extinction-risk” in communication to laypeople – I think the latter detracts from the more important question of preventing a loss of control and emphasizes the thing that happens after, which is far more speculative (and often invites the common “sci-fi scenario” criticism).
Of course, it doesn’t sound as flashy, but I think saying “we shouldn’t build a machine that takes control of our entire future” is sufficiently attention-grabbing.

neo 23 Sep 2025 0:22 UTC
30 points
7
in reply to: MichaelDickens’s comment on: Global Call for AI Red Lines—Signed by Nobel Laureates, Former Heads of State, and 200+ Prominent Figures
I’m disappointed that they don’t make any mention of extinction risk
Agree, but I wonder if extinction risk is just too vague, at the moment, for something like this. Absent a fast takeoff scenario, AI doom probably does look something like the gradual and unchecked increase of autonomy mentioned in the call, and I’m not sure if there’s enough evidence of a looming fast takeoff scenario for it to be taken seriously.
I think the stakes are high enough that experts should firmly state, like Eliezer, that we should back off way before fast takeoff even seems like a possibility. But I see why that may be less persuasive to outsiders.

neo 13 Aug 2025 22:32 UTC
28 points
10
in reply to: Tomás B.’s comment on: Tomás B.’s Shortform
Note that there have been many reports of persistent physiological changes caused by 5-AR inhibitors such as finasteride (see: Post Finasteride Syndrome), some of which sound pretty horrifying, like permanent brain fog and anhedonia.
I’ve spent a lot of time reading through both the scientific literature and personal anecdotes and it seems like such adverse effects are exceedingly rare, but I have high confidence (>80%) that they are not completely made up or psychosomatic. My current best guess is that all such permanent effects are caused by some sort of rare genetic variants, which is why I’m particularly interested in the genetic study being funded by the PFS network.
The whole situation is pretty complex and there’s a lot of irrational argumentation on both sides. I’d recommend this Reddit post as a good introduction – I plan on posting my own detailed analysis on LW sometime in the future.

neo 15 Jul 2025 22:34 UTC
1 point
0
on: 1 week fast on livestream for AI xrisk
I am a bit doubtful about the impact that this will have – it seems like the goal of fasting, as a form of activism, is to signal to others that you care deeply enough about an issue to suffer for it.
However, a lot of the key players in AI probably disagree with, or assign lower probability to the arguments for x-risk (for whatever reason). Which means demonstrating that you care a lot is unlikely to convince them.
Not to say that there will be no impact, but I’m not sure the expected loss of productivity from fasting for a week straight would be worth it.

neo 9 Jun 2025 2:21 UTC
2 points
−1
in reply to: Said Achmiz’s comment on: How the veil of ignorance grounds sentientism
To refine this discussion, I’ll only be responding to what I think are major points of disagreement.
This scenario is just as incoherent as the other one!
No, it is not.
Imagining yourself as a “disembodied spirit,” or a mind with no properties, is completely incoherent.
To imagine being someone else is something people (including you, I assume) do all the time. Maybe it’s difficult to do, and certainly your imagination will never be perfectly accurate to their experience, but it is not incoherent.
I do see your point: you can’t become someone else because you would then no longer be you. What you are arguing is that through your view of personal identity (which, based on your comments, I presume is in line with Closed Individualism), the Veil of Ignorance is not an accurate description of reality. Sure, I don’t disagree there.
I think it would be helpful to first specify your preferred theory of identity, rather than dismissing the VOI as nonsense altogether. That way, if you think Closed Individualism is obviously true, you could have a productive conversation with someone who disagrees with you on that.
No, the goal of the thought experiment is to argue that you should want to do this. If you start out already wanting to do this, then the thought experiment is redundant and unmotivated.
If you accept sentientism, I think it is still very useful to consider what changes to our current world would bring us closer to an ideal world under that framework. This is how I personally answer “what is the right thing for me to do?”
So, yes, as a point of metaethics, we recognize that aliens won’t share our morality, etc. But this has zero effect on our ethics. It’s simply irrelevant to ethical questions—a non sequitur.
I don’t see why a meta-ethical theory cannot inform your ethics. Believing that my specific moral preferences do not extend to everyone else has certainly helped me answer “what is the right thing for me to do?”
I believe that humans (and superintelligences) should treat all sentient beings with compassion. If the Veil of Ignorance encourages people to consider the perspective of other beings and reflect on the specific circumstances that have cultivated their personal moral beliefs, I consider it useful and think that endorsing it is the right thing for me to do.
Of course, this should have no direct bearing on your beliefs, and you are perfectly free to do whatever maximizes your internal moral reward function.
Is there something incoherent about caring about only some people/things/entities and not others? Surely there isn’t.
I think we agree here.
What I meant was: caring about another sentient being experiencing pain/pleasure, primarily because you can imagine what they are experiencing and how un/desirable it is, indicates that you care about experienced positive and subjective states, and so this care applies to all beings capable of experiencing such states.
I generally accept Eliezers meta-ethical theory and so I don’t assume this necessarily applies to anybody else.

neo 6 Jun 2025 19:24 UTC
1 point
−2
in reply to: Said Achmiz’s comment on: How the veil of ignorance grounds sentientism
The way I interpret it: in the thought experiment, you are not literally imagining yourself as a mind without properties, and then asking “what would I want?” You are imagining that you can become any of the sentient minds entering existence at any given moment, and that you will inherit their specific circumstances and subjective preferences.
The goal of the thought experiment, then, is to construct a desirable world that encompasses the specific desires and preferences of all sentient beings.^[1] This seems obviously relevant to the alignment problem.
With no constraints, your answer might look like “give every sentient being a personal utopia.” Our individual ability to change the real world is heavily constrained, though, so some realistic takeaways can be:
- Oppose factory farming, because it affects trillions of sentient beings in a very negative way.
- Try to stop AI from killing everyone, because a lot of sentient beings could exist in the future, and they could live very desirable lives. Also, a lot of the ones that currently exist would prefer to continue living so that their lives can become more desirable with technological progress.
I (taking your question seriously) answer that, obviously, the master should command the slave, and the slave should obey the master...
This would not be a faithful execution of the thought experiment, as you would obviously be ignoring the possibility of existing as a slave (which, presumably, does not prefer to be enslaved).
In a sense, the veil of ignorance is an exercise in rationality – you are recognizing that your mind (and every other mind) is shaped by its specific circumstances, so your personal conception of what’s moral and good doesn’t necessarily extend to all other sentient beings. If I’m not mistaken, this seems to be in line with Eliezers position on meta-ethics.
I agree that the notion of “you” existing before your entering into existence seems incoherent – I prefer to describe it as “your mind emerged from physical matter, and developed all of its properties based on that matter, so when you develop a moral/ethical framework that includes any minds outside of your own, it should logically extend to all minds that exist/will ever exist.” In other words, caring about anyone else means you should probably care about all sentient beings.
1. ^
  The thought experiment is supposed to be somewhat practical, so I don’t think you need to consider aliens, or the entire set of all possible minds that can ever exist.
What links here?

neo 4 Jun 2025 8:49 UTC
3 points
1
in reply to: Said Achmiz’s comment on: How the veil of ignorance grounds sentientism
I’m curious – what are the “very serious criticisms” you refer to? Your comment would more helpful and productive if you pointed to specific disagreements you have with the post rather than abstractly mentioning the existence of them.
More importantly, what are these criticisms directed at? The stated thesis of this post is that the veil of ignorance is a “useful and mostly accurate description of reality when viewed through certain theories of personal identity.”
This list of notable criticisms of the Veil of Ignorance doesn’t seem to include any disagreement with this claim in particular. In fact, the last criticism mentioned seems to support it – albeit with different reasoning.
Likewise, the first LessWrong post that comes up when searching for “Veil of Ignorance” argues that the thought experiment doesn’t necessarily support Rawl’s ideal society – not that it doesn’t accurately describe reality.

neo

What are some research directions for “improving coordination?”