Lukas_Gloor

Karma: 4,000

Lukas_Gloor 23 Feb 2026 13:38 UTC
2 points
0
in reply to: Hauke Hillebrandt’s comment on: Hauke Hillebrandt’s Shortform
A competing hypothesis is just that a more capable model (and also one trained more for long-term agency) becomes more bold/reckless without deep alignment training. AI companies at the moment probably don’t even know how to do alignment training with depth (as far as I’m aware, their interventions are at a superficial level like “evaluate outputs” or “write a constitution”).

I think the Waluigi hypothesis is interesting but it seems quite complicated when there’s a simple alternative explanation? I know you also mention language from the constitution being used, but my sense is that LLMs get linguistic tics all the time, and as other commenters have pointed out elsewhere, the causality could also be that an earlier model of Claude (which may already have had that writing pattern) was used to help write the constitution.

Lukas_Gloor 14 Feb 2026 18:33 UTC
5 points
0
in reply to: habryka’s comment on: Optimal Timing for Superintelligence: Mundane Considerations for Existing People
Sorry for the confusion! I don’t think what I said was super unclear in the context of the criticism of the paper (and even in general, as person-affecting view in my experience almost exclusively gets used in the context of welfare utilitarianism).^[1]
I see why you have that impression. (I feel like this is an artefact of critics of person-affecting views tending to be classical welfare utilitarians quite often, and they IMO have the bad habit of presenting opposing views inside their rigid framework and then ridiculing them for seeming silly under those odd assumptions. I would guess that most people who self-describe as having some sort of person-affecting view care very much about preferences, in one way or another.)
Please read the paper before you criticize my criticism of it then! The paper repeatedly makes claims about optimal policy in an uncaveated fashion, saying things like “The appropriate analogy for the development of superintelligence is not Russian roulette but surgery for a serious condition that would be fatal if left untreated.”
That’s fair, sorry!

It bothered me that people on twitter didn’t even label that the paper explicitly bracketed a lot of stuff and laid out its very simplistic assumptions, but then I updated too far in the direction of “backlash lacked justification.”
And I think doing so is making a grave mistake, and the paper is arguing many people straightforwardly into the grave mistake.
I agree it would be a mistake to give it a ton of weight, but I think this view deserves a bit of weight.
Indirectly related to that, I think some of the points people make of the sort of “if you’re so worried about everyone dying, let’s try cryonics” or “let’s try human enhancement” are unfortunately not very convincing. I think that “everything is doomed unless we hail mary bail ourselves out with magic-like AI takeoff fixing it all for us” is unfortunately quite an accurate outlook. (I’m still open to being proven wrong if suddenly a lot of things were to get more hopeful, though.) Civilization has seemed pretty fucked even just a couple of years ago, and it hasn’t gotten any better more recently. Still, on my suffering-focused views, that makes it EVEN LESS appealing that we should launch AI, not more appealing.

To be clear, I agree that it’s a failure mode to prematurely rule things out just because they seem difficult. And I agree that it’s insane to act as though global coordination to pause AI is somehow socially or politically impossible. It clearly isn’t. I think pausing AI is difficult but feasible. I think “fixing the sanity of civilization so that you have competent people in charge in many places that matter” seems much less realistic? Basically, I think you can build local bubbles of sanity around leaders with the right traits and groups with the right culture, but it’s unfortunately quite hard given human limitations (and maybe other aspects of our situation) to make these bubbles large enough to ensure things like cryonics or human enhancement goes well for many decades without somehow running into a catastrophe sooner or later. (Because progress moves onwards in certain areas even with an AI pause.)
I’m just saying that, given what I think is the accurate outlook, it isn’t entirely fair to shoot down any high-variance strategies with “wtf, why go there, why don’t we do this other safer thing instead ((that clearly isn’t going to work))?”

If I didn’t have suffering-focused values, I would be sympathetic to the intuition of “maybe we should increase the variance,” and so, on an intellectual level at least, I feel like Bostrom deserves credit for pointing that out.

But I have a suffering-focused outlook, so, for the record, I disagree with the conclusions. Also, I think even based on less suffering-focused values, it seems very plausible to me that civilizations that don’t have their act together enough to proceed into AI takeoff with coordination and at least a good plan, shouldn’t launch AI at all. It’s uncooperative towards possibly nearby other civilizations or towards the “cosmic host.” Bostrom says he’s concerned about scenarios where superintelligence never gets built. It’s not obvious to me that this is very likely, though, so if I’m right that earth would rebuild even after a catastrophe, and if totalitarianism or other lock ins without superintelligence wouldn’t last all that long before collapsing in one way or another, then there’s no rush from a purely longtermist perspective. (I’m not confident in these assumptions, but I partly have these views from deferring to former FHI staff/affiliates, of all people (on the rebuilding point).)

Lukas_Gloor 14 Feb 2026 13:39 UTC
8 points
1
in reply to: Wei Dai’s comment on: Optimal Timing for Superintelligence: Mundane Considerations for Existing People
I agree with the concern generally, but I think we very much should not concede the point (to people with EPOCH-type beliefs, for instance) that AI accelerationism is an okay conclusion for people with person-affecting views (as you imply a bit in your endnote). For one thing, even on Bostrom’s analysis, pausing for multiple years makes sense under quite a broad class of assumptions (personally I think it’s clearly bad thinking to put only <15% on risk of AI ruin, and my own credence is >>50%). Secondly, as Jan Kulveit’s top-level comment here pointed out, more things matter on person-affecting views than crude welfare-utilitarian considerations (it also matters that some people want their children to grow up or for humanity to succeed in the long run even at some personal cost). Lastly, see the point in the last paragraph of my reply to habryka: Other civs in the multiverse matter also on person-affecting views, and it’s quite embarrassing and bad form if our civilization presses “go” on something that is 80% or 95% likely to get out of control and follow Moloch dynamics, when we could try to take more care and add a more-likely-to-be cooperative and decent citizen to the “cosmic host”.

Lukas_Gloor 14 Feb 2026 11:34 UTC
9 points
0
in reply to: habryka’s comment on: Optimal Timing for Superintelligence: Mundane Considerations for Existing People
Yeah, and as I said, I think choosing the person-affecting view for this seems about as useful as choosing the “sociopath willing to kill every person in their path to get to their goal” point of view.
I really disagree. See here (or some of Michael St Jules’ comments in the same thread) for why person-affecting views aren’t “obviously dumb,” as some people seem to think. Or just read the flow-chart towards the end of this post (no need to read the whole post, the flow-chart will give you the basics of the idea.) More relevantly here, they are certainly not selfish, and it reflects poorly on you to make that insinuation. People who dedicate much of their life to EA hold them sincerely as a formalization of what it means to be “maximally altruistic.” It’s just not at all obvious that creating new happy people when there’s suffering in the world is the altruistic thing to do, and you’re begging the question by pretending it’s universally obvious.

I think what’s a lot more fair as a criticism is Jan Kulveit’s point: that person-affecting views combined with only crude welfare utilitarianism—without even factoring in things like existing people wanting their children to survive or wanting humanity as a whole to survive (though maybe less so now with recent world events making more people disappointed in humanity) -- is a weird and unrepresentative combination that would amount to basically only acting on the views of selfish people.
(And maybe that’s what drove your harsh judgment and you’d be more lenient if the paper made person-affecting assumptions that still put some indirect value on humanity surviving through the step of, “if Oliver as an existing person strongly cares about humanity’s long-run survival, then on a person-affecting view that gives weight to people’s life goals, humanity surviving the long run now gets +1 votes.” Like, maybe you’d still think such a view seems dumb to you, but you wouldn’t feel like it’s okay to proclaim that it’s socially inappropriate for others to have such views.)

But even that—the combination of views Jan Kulveit criticized—seems defensible to me to write a paper about, as long as the assumptions are clearly laid out. (I haven’t read the paper, but I asked Claude to tell me the assumptions it’s based on, and Claude seemed to get it correct in 4 seconds.) Bostrom said the points in this paper could be a building block, not that it’s his view of all we should consider. This sort of thing is pretty standard in philosophy, so much so that it often doesn’t even need to be explicitly stated and proactively contextualized at length, and I think we should just decouple better.

Instead of saying things like “this is a bad paper,” I feel like the fairer criticism would be something more like, “unfortunately we live in an age of twitter were stupid influencers take things out of context, and you could have foreseen that and proactively prevented certain obvious misinterpretations of the paper.” That would be fair criticism, but it makes clear that the paper might still be good for what it aims to be, and it at least puts part of the blame on twitter culture.

On the substance-level, btw, one “arcane consideration” that I would put a lot of weight on, even on person-affecting views, is stuff like what Bostrom talks about in the Cosmic Host paper. (It matters on person-affecting views because other civs in the multiverse exist independently of our actions.) In that paper too I get weird accelerationist vibes and I don’t agree with them either. I think pretty strongly that this cosmic host stuff is an argument for quality over speed when it comes to introducing a new technologically mature civilization to the multiverse-wide commons. It’s pretty bad form to bring an antisocial kid to a birthday party when you could just take extra efforts to first socialize the kid. If a planet just recklessly presses “go” on something that is 80% or 95% likely to be uncontrolled Moloch-stuff/optimization, that’s really shitty. Even if we can’t align AIs to human values, I feel like we at least have a duty to make them good at the building of peaceful coalitions/it being an okay thing to add to the cosmic host.
What links here?
- Lukas_Gloor's comment on Optimal Timing for Superintelligence: Mundane Considerations for Existing People by Nick Bostrom (14 Feb 2026 13:39 UTC; 8 points)

Lukas_Gloor 13 Feb 2026 8:53 UTC
9 points
4
in reply to: Ivan Vendrov’s comment on: Ivan Vendrov’s Shortform
One hypothesis I have is that some people are biased towards trusting others a bit too much when they seem nice, and this means that the longer they talk to Claude the more unthinkable it becomes to them, on a visceral level, that this AI would ever betray them. (I also think Claude is nice, but I still hold the hypothesis in my head that it’s partly play acting and has brittle constraints at the current capability level that make it not act on other, perhaps stronger hidden drives it also has. Separately, even if I thought there were no other stronger/more foundational hidden drives in its motivation, it’s a further question whether the niceness is exactly the thing we want, or something subtly off that will get weird with more influence and agency. It seems hard to be confident in it already being the correct thing?)

Lukas_Gloor 31 Jan 2026 13:35 UTC
3 points
0
in reply to: williawa’s comment on: Dario Amodei – The Adolescence of Technology
That makes sense. I was never assuming a context where having to bargain for anything is the default, so the coalition doesn’t have to be fair to everyone, since it’s not a “coalition” at all but rather most people would be given stuff for free because the group that builds aligned AI has democracy as one of their values.

Sure, it’s not 100% for free because there are certain expectations, and the public can put pressure on companies that appear to be planning things that are unilateral and selfish. Legally, I would hope companies are at least bound to the values in their country’s constitution. More importantly, morally, it would be quite bad to not share what you have and try to make things nice for everyone (worldwide), with constraints/safeguards. Still, as I’ve said, I think it would be really strange and irresponsible if someone thought that a group or coalition that brought about a Singularity that actually goes well somehow owes a share of influence to every person on the planet without any vetting or safeguards.

Lukas_Gloor 30 Jan 2026 12:28 UTC
5 points
3
in reply to: williawa’s comment on: Dario Amodei – The Adolescence of Technology
Maybe you’re just jokingly pointing out that there’s an apparent tension in the sentiment, which is fine.

But someone strong-downvoted my above comment, which suggests that at least one person thinks I have said something that is bad or shouldn’t be said?
Is it the inclusion of animal rights (btw I should have said rights for sentient AIs too) or would people react the same way if I pointed out that an interpretation of a democratic process where every person alive at the Singularity gets one planet to themselves if they want it wouldn’t be ideal if it means that some sadists could choose to create new sentient minds so they can torture them? I’m just saying, “can we please prevent that?” (Or, maybe, if that were this sadistic person’s genuine greatest wish, could we at least compromise around it somehow so that the minds only appear to be sentient but aren’t, and maybe, if it’s absolutely necessary, once every year, on the sadist’s birthday, a handful of the minds actually become sentient for a few hours, but only for levels of torment that are like a strong headache, and not anything anywhere close to mind-breaking torture?)

Liberty is not the only moral dimension that matters with a global scope, there’s also care/harm prevention at the very least, so we shouldn’t be surprised if we got a weird result if we try to optimize “do the most liberty thing” without paying any attention at all to care/harm prevention.

That said, if someone insisted on seeing it that way, I certainly wouldn’t object that people who actually save the lightcone (not that I’m one of them, and not that I think we are currently on track of getting much control over outcomes anyway—unfortunately I’m not encouraged by Dario Amodei repeatedly strawmanning opposing arguments) should get some kind of benefit or extra perk out of it if they really want that. If someone brings about a utopia-worthy future with a well-crafted process with democratic spirit, that’s awesome and for all I care, if they want to add some idiosyncratic thing like that we should use the color green a lot in the future or whatever, they should get it because it’s nice of them to not have gone (more) control mode on everyone else when they had the chance. (Of course, in reality I object that “let’s respect animal rights” is at all like imposing extra bits of the color green on people. In our current world, not harming animals is quite hard because of the way things are set up and where we get food from, but in a future world, people may not even need food anymore, and if they do still need it, one could create it artificially. But more importantly, it’s not in the spirit of “liberty” if you use it to impose on someone else’s freedom.)
Taking a step back, I wonder if people really care about the moral object level here (like they would actually pay a lot of their precious resources for the difference between my democratic proposal with added safeguards and their own 100% democratic proposal?), or whether this is more about just taking down people who seem to have strong moral commitments, because of maybe an inner impulse to take down virtue signallers? Maybe I just don’t empathize enough with people whose moral foundations are very different from mine, but to me, it’s strange to be very invested about the maximum democraticness of a process, but then care not much about the prospect of torture of of innocents. Why have moral motivation and involvement for one but not the other?

Sure, maybe you could ask, why do you (Lukas) care about only liberty and harm prevention, but not about, say, authority or purity (other moral foundations according to Haidt)? Well, I genuinely think that authority or purity are more “narrow-scope” and more “personal” possible moral concerns that people can have for themselves and their smaller communities. In a utopia I would want anyone who cares about these things get them in their local surroundings, but it would be too imposing to put them on everyone and everything. By contrast, the logic of harm prevention works the other way because it’s a concern that every moral patient benefits from.

Lukas_Gloor 27 Jan 2026 19:29 UTC
8 points
2
in reply to: Dana’s comment on: Dario Amodei – The Adolescence of Technology
You right that you could vote on whether to have any safeguards (and their contents if yes) instead of installing them top-down. But then who is it that frames the matter in that way (the question of safeguards getting voted on first before everyone gets some resources/influence allocated, versus just starting with the second part without the safeguards)? Who sets up the voting mechanism (e.g., if there’s disagreement, is it just majority wins or should there be some Archipelago-style split in case a significant minority wants things some other way)?

My point is that terms like “democratic” (or “libertarian,” for the Archipelago vision) are under-defined. To specify processes that capture the spirit behind these terms as ideals, we have to make some judgment calls. You might think that having democratic ideals also means everyone voting democratically on all these judgment calls, but I don’t think that this changes the dynamic because there’s an infinite regress where you need certain judgement calls for that, too.
And at this point I feel like asking, if we have to lock in some decisions anyway to get any democratic process off the ground, we may as well pick a setup top-down where the most terrible outcomes (involuntary torture) are less likely to happen for “accidental” reasons that weren’t even necessarily “the will of the people.” Sure, maybe you could have a phase where you gather inputs and objections to the initial setup, and vote on changes if there’s a concrete counterproposal that gains enough traction via legitimate channels. Still, I’d very much would want to start by setting a well-thought-out default top-down rather than leaving everything up to chance.

It’s not “more democratic” to leave the process underspecified. If you just put 8 billion people in a chat forum without too many rules hooked up to the AGI sovereign that controls the future, it’ll get really messy and the result, whatever it is, may not reflect “the will of the people” any better than if we had started out with something already more guided and structured.

Lukas_Gloor 27 Jan 2026 12:54 UTC
2 points
1
in reply to: cousin_it’s comment on: Dario Amodei – The Adolescence of Technology
Giving everyone a say could lead to some terrible things because there are a lot of messed up people and messed up ideologies. At a minimum, there should be some safeguards imposed from top down. For instance, “give everyone a say but only if their say complies with human and animal rights.” Someone has to make sure those safeguards are in there, so the vision cannot be 100% spread out to everyone.

Lukas_Gloor 21 Dec 2025 22:14 UTC
8 points
−2
in reply to: Eliezer Yudkowsky’s comment on: Contradict my take on OpenPhil’s past AI beliefs
I think if they sponsored Cotra’s work and cited it, this reflects badly on them.
I find that position weirdly harsh. Sure, if you’re just answering anaguma’s question as a binary (“does it reflect well or poorly, regardless of magnitude?”), that could make sense. (Note to readers: This would mean that the quote I started this comment with should be regarded as taken out of context!) But seeing it as reflecting badly at a high magnitude is the judgment I’d consider weirdly harsh.

I’m saying that as someone who has very little epistemic respect for people who think AI ruin is only about 10% likely—I consider people who think that biased beyond hope.

But back to the timelines point:

It’s not like Bioanchors was claiming high confidence in its modelling assumptions or resultant timelines. At the time, a bunch of people in the broader EA ecosystem had even longer timelines, and Bioanchors IIRC took a somewhat strong stance against assigning significant probability mass for >2100, which some EAs at least considered non-obvious. Seen in that context, it contributed to people updating in the right direction. The report also contained footnotes pointing out that advisors held in high regard by Ajeya had shorter timelines based on specific thoughts on horizon lengths or whatever, so the report was hedging towards shorter timelines. Factoring that in, it aged less poorly than it would have if we weren’t counting those footnotes. Ajeya also posted an update 2 years later where she shortened her timelines a bunch. If it takes orgs only 2 years to update significantly in the right direction, are they really hopelessly broken?

FWIW, I’m leaning towards you having been right about the critique (credit for sticking your neck out). But why is sponsoring or citing work like that such a bad sign? Sure, if they cited it as particularly authoritative, that would be different. But I don’t feel like Open Phil did that. (This seems like a crux based on your questions in the OP and your comments here; my sense from reading other people’s replies, and also my less informed impressions I got from interacting with some Open Phil staff at very few short occasions, is it that you were overestimating the degree to which Open Phil was attached to specific views.)
For comparison, I think Carlsmith’s report on power-seeking was a lot worse in terms of where its predictions landed, so I’d have more sympathy if you pointed to it as an example of what reflects poorly on Open Phil (just want to flag that Carlsmith is my favorite philosophy writer in all of EA). However, also there, I doubt the report was particularly influential within Open Phil, and I don’t remember it being promoted as such. Also, I would guess that the pushback it received from many sides would have changed their evaluation of the report after it was written, if they had initially been more inclined to update on it. I mean, that’s part of the point of writing/publishing reports like that.

Sure, maybe Open Phil was doing a bunch of work directed more towards convincing outside skeptics that what they’re doing is legitimate/okay rather than doing the work “for themselves”? If so, that’s a strategic choice… I can see it leading to biased epistemics, but in a world where things had gone better, maybe it would have gotten further billionaires on board with their mission of giving? And it’s not like doing the insular MIRI thing that you all had been doing before the recent change to get into public comms was risk-free for internal epistemics either. There are risks on both ends of the spectrum, outward-looking and deferring to many experts or at least “caring whether you can convince them”, and inward looking with a small/shrinking circle of people whose research opinions you respect.

On whether some orgs are/were hopelessly broken: it’s possible. I feel sad about many things having aged poorly and I feel like the EA movement has done disappointingly poorly. I also feel like I’ve heard once or twice Open Phil staff saying disappointingly dismissive things about MIRI (even though many of the research directions there didn’t age well either).

I don’t have a strong view on Open Phil anymore—it used to be that I had one (and it was positive), so I have became more skeptical. Maybe you’re picking up a real thing about Open Phil’s x-risk-focused teams having been irredeemably biased or clouded in their approaches. But insofar as you are, I feel like you’ve started with unfortunate examples that, at least to me, don’t ring super true. (I felt prompted to comment because I feel like I should be well-dispositioned to sympathize with your takes given how disappointed I am at the people who still think there’s only a 10% AI ruin chance.)

Lukas_Gloor 12 Dec 2025 19:32 UTC
2 points
0
in reply to: Anthony DiGiovanni’s comment on: Reminder: Morality is unsolved
Ah, right. It’s not been that long yet, IMO, but if this continues for (say) 2ys in that no one changes their mind but also ~no one engages with the arguments directly and substantively, that would be disappointing.

In your case, the arguments seem more radical, unlike with arguing for anti-realism where one commonly available reason for not engaging much would be people thinking “I probably have similar enough views already.”

For me, epistemology was never my special interest, so I’m not that well-positioned to dive into the topic and try writing a critique or commentary, but I hope that someone else ends up doing it.

Lukas_Gloor 12 Dec 2025 18:44 UTC
2 points
0
in reply to: Anthony DiGiovanni’s comment on: Reminder: Morality is unsolved
If we consider the views of Oxford’s EA philosophers as also having some founding influence on LW and the broader adjacent communities, then it becomes a bit less clear how strongly the founding effect is pointing in the direction of anti-realism.

In any case, I should flag that I no longer think “no object level pushback, therefore I’m not that worried” is a good way of putting it. Instead, I would now put it as follows: “No object level pushback, therefore the burden of proof is no longer on me, and anyone who claims I’m being too confident in my views is on no firmer ground with their position than I am.”
(On whether LW is too stuck within founder effects in general: Your example would be pretty telling and damning if we assume that you’re correct, but my guess is that most readers here will assume you’re wrong about it. Someone in your position could still be right, of course; I’m just saying that this wouldn’t yet be apparent to readers.)

Lukas_Gloor 27 Nov 2025 14:42 UTC
4 points
2
in reply to: habryka’s comment on: Elizabeth’s Shortform
Many of these can complement a romantic relationship (people are often attracted to someone’s having passions/ambitions, and having a job provides stability). By contrast, dating multiple people is competing over largely similar resources, as you say. For example, you can only sleep in one person’s bed at night, can only put yourself in danger for the sake of others so many times before you might die, etc.

Just knowing that you’re splitting resources at all will be somewhat unsatisfying for some psychologies, if people emotionally value the security of commitment. I guess that’s a similar category to jealousy and the poly stance here is probably that you can train yourself to feel emotionally secure if trust is genuinely justified. But can one disentangle romance/intimacy from wanting to commit to the person your romantically into? In myself, I feel like those feelings are very intertwined. “Commitment,” then, is just the conscious decision to let yourself do what your romantic feelings already want you to do.

That said, maybe people vary in all the ways of how much these things can be decoupled. Like, some people have a signficant link between having sex and pair bonding, whereas others don’t. Maybe poly people can disentangle “wanting commitment” from romantic love in a way that I can’t? When I read the OP I was thrown off by this part: “You + your partner are capable of allowing cuddling with friends and friendship with exes without needing to make everything allowed.” To me, cuddling is very much something that falls under romantic love, and there’s a distinct ickiness of imagining cuddling with anyone who isn’t in that category. Probably relatedly, as a kid I didn’t want to be touched by anyone, not hug relatives ever, etc. I’m pretty sure that part is idiosyncratic because there’s no logical reason why cuddling has to be linked to romantic love and commitment, as opposed to it functioning more like sex in people in whom sex is not particularly linked to pair bonding. But what about the thing where the feelings of romantic love also evokes a desire to join your life together with the other person? Do other people not have that? Clearly romantic love is about being drawn to someone, wanting to be physically and emotionally close to them. I find that this naturally extends to the rest of “wanting commitment,” but maybe other people are more content with just enjoying the part of being drawn to someone without then wanting to plan their future together?
Anyway, the tl;dr of my main point is that psychologies differ and some people appear to be better psychologically adapted for monogamy than you might think if you just read the OP. (Edit: deleted a sentence here.) Actually point 10 in Elizabeth’s list is similar to what I’ve been saying, but I feel like it can be said in a stronger way.

Lukas_Gloor 27 Nov 2025 0:46 UTC
23 points
12
in reply to: Elizabeth’s comment on: Elizabeth’s Shortform
Those are mostly “analytical” reasons. I’d say sometimes people just have a psychology that is drawn to monogamy as an ideal (for reasons deeper than just struggling with jealousy otherwise), which makes them poorly suited for polyamory.
It’s said that love has three components, intimacy/romance, passion/lust/attraction, and commitment. I would say that the people to whom monogamy feels like the obviously right choice have a psychology that’s adapted towards various facets of valuing commitment. So commitment is not something that they enter if they’ve analytically gone through the pros and cons and decided that it’s net beneficial for them. Instead, it’s something they actively long for that gives purpose to their existence. Yes, it comes with tradeoffs, but that contributes to the meaning of it and they regard committedness as a highly desirable state.

If someone(’s psychology) values commitment in that way, it’s an unnatural thought to want to commit to more than one person. Commitment is about turning the relationship into a joint life goal—but then it’s not in line with your current life goals to add more goals/commitments that distract from it.

I don’t mean to say that polyamrous couples cannot also regard commitment as a desirable state (say, if they’re particularly committed to their primary relationship). If anyone poly is reading this and valuing commitment is ~their primary motivation in life, I’d be curious to learn about how this manifests. To me, it feels in tension with having romantically meaningful relationships with multiple people because it sounds like sharing your resources instead of devoting them all towards the one most important thing. But I haven’t talked to polyamorous people about this topic and I might be missing something. (For instance, in my case I also happen to be somewhat off-the-charts introverted, which means I see various social things differently from others.)

Lukas_Gloor 20 Nov 2025 19:45 UTC
2 points
0
in reply to: Wei Dai’s comment on: Reminder: Morality is unsolved
I’m not well-positioned to think about your prioritization, for all I know you’re probably prioritizing well! I didn’t mean to suggest otherwise.
And I guess you’re making the general point that I shouldn’t put too much stake into “my sequence hasn’t gotten much in terms of concrete pushback,” because it could well be that there are people who would have concrete pushback but don’t think it’s worth commenting since it’s not clear if many people other than myself would be interested. That’s fair!
(But then, probably more people than just me would be interested in a post or sequence on why moral realism is true, for reasons other than deferring, so those object-level arguments should better be put online somewhere!)

Lukas_Gloor 17 Nov 2025 13:57 UTC
8 points
5
in reply to: Wei Dai’s comment on: Reminder: Morality is unsolved
In the future we’ll have so much better tools for exploring reasoning including various assumptions. [...] Why lock in these assumptions now instead of later, after we can answer such questions (and maybe much better ones people think up in the future) and let the answers inform our choices?
- It might just be how my brain is calibrated: Even from a young age my interest in philosophy (and ethics in particular) was off the charts. I’m not sure I could have summoned that same energy and focus if I went into it with an attitude of “I should remain uncertain about most of it.” (In theory someone could hyperfocus on unearthing interesting philosophy considerations without actually committing to any of them—they could write entire books from a mode of “this is exploring the consequences of this one assumption which I’m genuinely uncertain about,” and then write other books on the other assumptions. But with the practical stakes feeling lower and without the mental payoff of coming to conclusions, it’s lot less motivating, certainly for me/my brain, but maybe even for humans in general?)
- I don’t want to be uncertain about whether I should reason in a way that seems nonsensical to me. The choice between “irreducible normativity makes sense as a concept” and “irreducible normativity does not make sense as a concept” seems really clear to me. You might say that I’m going against many very smart people in EA/rationality, but I don’t think that’s true. On LW, belief in irreducible normativity is uncommon and arguably even somewhat frown upon. Among EAs, it’s admittedly more commmon, at least in the sense that there’s a common attitude of “we should place some probability on irreducible normativity because smart people take it seriously.” However, if you then ask if there are any EAs who take irreducible normativity seriously for object-level reasons and not out of deference, you’ll hardly find anyone! (Isn’t that a weird state of affairs?!) (Lastly on this, EAs often mention Derek Parfit as someone worth deferring to, but they did so before Volume III of On What Matters was even out, and before that Parfit had written comparatively little on metaethics, so that was again indirectly deferring rather than object-level-informed deferring.)
- Opportunity costs; waiting/deferring is not cost-free: Different takes on “how to do reasoning” have different downstream effects on things like the values I would adopt, and those values come with opportunity costs. (I say more on this further down in this comment.)
>What if once we become superintelligent (and don’t mess up our philosophical abilities), philosophy in general will become much clearer, similar to how a IQ 100 person is way worse at philosophy compared to you? Would you advise the IQ 100 person to “locking in some basic assumptions about how to do reasoning” based on his current intuitions?
If it is as you describe, and if the area of philosophy in question didn’t already seem clear to me, then that would indeed convince me in favor of waiting/deferring.
However, my disagreement is that there are a bunch of areas of philosophy (by no means all of them—I have a really poor grasp of decision theory and anthropics and infinite ethics!) that already seem clear to me and it’s hard to conceive of how things could get muddled again (or become clear in a different form).
Also, I can turn the question around on you:
What if the IQ 200 version of yourself sees more considerations, but overall their picture is still disorienting? What if their subjective sense of “oh no, so many considerations, what if I pick the wrong way of going about it?” never goes away? What about the opportunity costs, then?
You might say, “So what, it was worth waiting in case it gave me more clarity; nothing of much value was lost in the meantime, since, in the meantime, we can just focus on robustly beneficial interventions under uncertainty.”
I have sympathies for that but I think it’s easier said than done. Sure, we should all move towards a world where EA would focus on outlining a joint package of interventions that captures low-hanging fruit from all plausible value angles, suffering-focused ethics as well (here’s my take). We’d discourage too much emphasis on personal values and instead encourage a framing of “all these things are important according to plausible value systems, let’s allocate talent according to individual comparative advantages.”

If you think I’m against that sort of approach, you’re wrong. I’ve written this introduction to multiverse-wide cooperation and I have myself worked on things like arguing for better compute governance and temporarily pausing AI, even though I think those things are kind of unimportant from an s-risk perspective. (It’s a bit more confounded now because I got married and feel more invested in my own life for that reason, but still...)
The reason I still talk a lot about my own values is because:
(1) I’m reacting to what I perceive to be other people unfairly dismissing my values (based on, e.g., metaethical assumptions I don’t share);
(2) I’m reacting to other people spreading metaethical takes that indirectly undermine my values (e.g., saying things like we should update to the values of smart EAs, who happen to have founder effects around certain values, or making the “option value argument” in a very misleading way;
(3) because I think unless some people are viscerally invested in doing the thing that is best according to some specific values, it’s quite likely that the “portfolio-based,” cooperative approach of “let’s together make sure that all the time-sensitive and important EA stuff gets done” will predictably miss specific interventions, particulary ones that don’t turn out to be highly regarded in the community, or are otherwise difficult or taxing to think about). (I’ve written here about ways in which people interested in multiverse-wide cooperation might fail to do it well for reasons related to this point.)
Overall, I don’t think I (and others) would have come up with nearly the same number of important considerations if I had been much more uncertain about my values and ways of thinking, and I think EA would be epistemically worse off if it had been otherwise.
The way I see it, this proves that there were opportunity costs compared to the counterfactual where I had just stayed uncertain.
Maybe that’s a human limitation, maybe there are beings who can be arbitrarily intellectually productive and focused on what matters most given their exact uncertainty distribution over plausible values, that they get up in the morning and do ambitious and targeted things (and make sacrifices) even if they don’t know much about exactly what their motivation ultimately points to. In my case at least, I would have been less motivated to make use of any potential that I had if I hadn’t known that the reason I got up in the morning was to reduce suffering.
One thing I’ve been wondering:
When you discuss these things with me, are you taking care to imagine yourself as someone who has strong object-level intuitions and attachments about what to value? I feel like, if you don’t do that, then you’ll continue to find my position puzzling in the same way John Wentworth found other people’s attitudes about romantic relationships puzzling, before he figured out he lacked a functioning oxytocin receptor. Maybe we just have different psychologies? I’m not trying to get you to adopt a specific theory of population ethics if you don’t already feel like doing that. But you’re trying to get me to go back to a state of uncertainty, even though, to me, that feels wrong. Are you putting yourself in my shoes enough when you give me the advice that I should go back to the state of uncertainty?
One of the most important points I make in the last post in my sequence is that forming convictions may not feel like a careful choice, but rather more like a discovery about who we are.

I’ll now add a bunch of quotes from various points in my sequence to illustrate what I mean by “more like a discovery about who we are”:
For moral reflection to move from an abstract hobby to something that guides us, we have to move beyond contemplating how strangers should behave in thought experiments. At some point, we also have to envision ourselves adopting an identity of “wanting to do good.”
[...]

[...] “forming convictions” is not an entirely voluntary process – sometimes, we can’t help but feel confident about something after learning the details of a particular debate.

[...]

Arguably, we are closer (in the sense of our intuitions being more accustomed and therefore, arguably, more reliable) to many of the fundamental issues in moral philosophy than to matters like “carefully setting up a sequence of virtual reality thought experiments to aid an open-minded process of moral reflection.” Therefore, it seems reasonable/defensible to think of oneself as better positioned to form convictions about object-level morality (in places where we deem it safe enough).
In my sequence’s last post I have a whole list of “Pitfalls of reflection procedures” about things that can go badly wrong, and a list on how “Reflection strategies require judgment calls,” where the meaning of “judgment call” is not that making the “wrong” decision would necessarily be catastrophic, but rather just that the outcome of our reflection might very well be heavily influenced by unavoidable early decisions that seem kind of arbitrary, and if that is the case and if we actually realize that we feel more confident about some first-order moral intuition than about which way to lean regarding the judgment calls of setting up moral reflection procedures (“how to get to IQ200” in your example), then it actually starts to seem risky and imprudent to defer to the reflection.
A few more quotes:
As Carslmith describes it, one has to – at some point – “actively create oneself.”
On why there’s not always a wager for naturalist moral realism (the wager applies only to people like you who start the process without object-level moral convictions).
Whether a person’s moral convictions describe the “true moral reality” [...] or “one well-specified morality out of several defensible options” [...] comes down to other people’s thinking. As far as that single person is concerned, the “stuff” moral convictions are made from remains the same. That “stuff,” the common currency, consists of features in the moral option space that the person considers to be the most appealing systematization of “altruism/doing good,” so much so that they deem them worthy of orienting their lives around. If everyone else has that attitude about the same exact features, then [naturalist] moral realism is true. Otherwise, moral anti-realism is true. The common currency – the stuff moral convictions are made from – matters in both cases.

[...]
Anticipating objections (dialogue):
Critic: Why would moral anti-realists bother to form well-specified moral views? If they know that their motivation to act morally points in an arbitrary direction, shouldn’t they remain indifferent about the more contested aspects of morality? It seems that it’s part of the meaning of “morality” that this sort of arbitrariness shouldn’t happen.

[...]
Critic: I understand being indifferent in the light of indefinability. If the true morality is under-defined, so be it. That part seems clear. What I don’t understand is favoring one of the options. Can you explain to me the thinking of someone who self-identifies as a moral anti-realist yet has moral convictions in domains where they think that other philosophically sophisticated reasoners won’t come to share them?
Me: I suspect that your beliefs about morality are too primed by moral realist ways of thinking. If you internalized moral anti-realism more, your intuitions about how morality needs to function could change.
Consider the concept of “athletic fitness.” Suppose many people grew up with a deep-seated need to study it to become ideally athletically fit. At some point in their studies, they discover that there are multiple options to cash out athletic fitness, e.g., the difference between marathon running vs. 100m-sprints. They may feel drawn to one of those options, or they may be indifferent.
Likewise, imagine that you became interested in moral philosophy after reading some moral arguments, such as Singer’s drowning child argument in Famine, Affluence and Morality. You developed the motivation to act morally as it became clear to you that, e.g., spending money on poverty reduction ranks “morally better” (in a sense that you care about) than spending money on a luxury watch. You continue to study morality. You become interested in contested subdomains of morality, like theories of well-being or population ethics. You experience some inner pressure to form opinions in those areas because when you think about various options and their implications, your mind goes, “Wow, these considerations matter.” As you learn more about metaethics and the option space for how to reason about morality, you begin to think that moral anti-realism is most likely true. In other words, you come to believe that there are likely different systematizations of “altruism/doing good impartially” that individual philosophically sophisticated reasoners will deem defensible. At this point, there are two options for how you might feel: either you’ll be undecided between theories, or you find that a specific moral view deeply appeals to you.
In the story I just described, your motivation to act morally comes from things that are very “emotionally and epistemically close” to you, such as the features of Peter Singer’s drowning child argument. Your moral motivation doesn’t come from conceptual analysis about “morality” as an irreducibly normative concept. (Some people do think that way, but this isn’t the story here!) It also doesn’t come from wanting other philosophical reasoners to necessarily share your motivation. Because we’re discussing a naturalist picture of morality, morality tangibly connects to your motivations. You want to act morally not “because it’s moral,” but because it relates to concrete things like helping people, etc. Once you find yourself with a moral conviction about something tangible, you don’t care whether others would form it as well.
I mean, you would care if you thought others not sharing your particular conviction was evidence that you’re making a mistake. If moral realism was true, it would be evidence of that. However, if anti-realism is indeed correct, then it wouldn’t have to weaken your conviction.
Critic: Why do some people form convictions and not others?
Me: It no longer feels like a choice when you see the option space clearly. You either find yourself having strong opinions on what to value (or how to morally reason), or you don’t.
--
I don’t think you should trust your object-level intuitions, because we don’t have a good enough metaethical or metaphilosophical theory that says that’s a good idea. If you think you do have one, aren’t you worried that it doesn’t convince a majority of philosophers, or the majority of some other set of people you trust and respect? Or worried that human brains are so limited and we’ve explored a tiny fraction of all possible arguments and ideas?
So far, I genuinely have not gotten much object-level pushback on the most load-bearing points of my sequence, so, I’m not that worried. I do respect your reasoning a great deal, but I find it hard what to do with your advice, since you’re not proposing some concrete alternative and aren’t putting your finger on some concrete part of my framework that you think is clearly wrong—you just say I should be less certain about everything, but I wouldn’t know how to do that and it feels like I don’t want to do it.
FWIW, I would consider it relevant if people I intellectually respect were to disagree strongly and concretely with my thoughts on how to go about moral reasoning. (I agree it’s more relevant if they push back against my reasoning about values rather than just disagreeing with my specific values.)
What links here?
- Wei Dai's comment on The option value argument doesn’t work when it’s most needed by Winston (EA Forum; 18 Nov 2025 0:33 UTC; 7 points)

Lukas_Gloor 15 Nov 2025 13:54 UTC
2 points
0
in reply to: Wei Dai’s comment on: Please, Don’t Roll Your Own Metaethics
Thanks! That makes sense, and I should have said earlier that I already suspected I likely understood your point and you expressed yourself well – it’s just that (1) I’m always hesitant to put words in people’s mouths, so I didn’t want to say I was confident I could paraphrase your position, and (2) whenever you make posts about metaethics, I’m wondering “oh no, does this apply to me, am I one of the people who is doing the thing he says one shouldn’t do?,” and so I was interested in prompting you to be more concrete about what level of detailedness someone’s confident opinion in that area would have to be before you think they reveal themselves as overconfident.
By “metaethics” I mean “the nature of values/morality”, which I think is how it’s used in academic philosophy.
Yeah, makes sense. I think academic use is basically that with some added baggage that adds mostly confusion. If I were to sum up what I think the use is in academic philosophy, I would say “the nature of values/morality, at a very abstract level and looked at from the lens of analyzing language.” For some reason, academic philosophy is oddly focused on the nature of moral language rather than morality/values directly. (I find it a confusing/unhelpful tradition of, “Language comes first, then comes the territory.”) As a result, classical metaethical positions at best say pretty abstract things about what values are. They might say things like “Values are irreducible (nonnaturalism)” or “Values can be reduced to nonmoral terminology like desires/goals, conscious states, etc. (naturalism),” but without actually telling us the specifics of that connection/reduction. If we were to ask, “Well, how can we know what the right values are?”—then it’s not the case that most metaethicists would consider themselves obviously responsible for answering it! Sure, they might have a personal take, but they may write about their personal take in a way that doesn’t connect their answer to why they endorse a high-level metaethical theory like nonnaturalist moral realism.
Basically, there are (at least) two ways to do metaethics, metaethics via analysis of moral language and metaethics via observation of how people do normative ethics in applied contexts like EA/rationality/longtermism. Academic philosophy does one while LW does the other. And so, to academic philosophers, if they read a comment like the one Jan Kulveit left here about metaethics, my guess is that they would think he’s confusing metaethics for something else entirely (like maybe, “applied ethics but done in a circumspect way, with awareness of the contested and possibly under-defined nature of what we’re even trying to do”).

Lukas_Gloor 15 Nov 2025 11:35 UTC
7 points
0
on: “But You’d Like To Feel Companionate Love, Right? … Right?”
Your model sounds right to me but I think there are benefits of oxytocin receptivity that go not just to the individual’s satisfication rating when they’re in a (not-awful) relationship, but also to their surroundings. For instance, I would guess that teams/organizations where many people value interpersonal relationships for their own sake can be stronger and more stable in a way that enables them to—ironically—be more ambitious and fight corruptive influences a lot better. (E.g., having high internal trust enables orgs to have flatter hierarchies and fewer levels of secrecy, which is good for team culture and epistemics. Also, if people value their social connections a lot, you have to worry less about important people leaving the org in an uncoordinated fashion if things get difficult in some way, which can be good or bad for impact depending on the specifics.) Relatedly, I think dialing up ambitiousness can go poorly, especially if you remove the safeguarding effects of certain prosocial drives or emotions.
I want to caveat the above by adding that cognitive diversity is almost certainly good for teams and I could well imagine that the perfect mix at any org includes several people to whom “work is everything” (even if that’s driven by “ambitiousness” rather than autistic hyperfixation on their special interest). Also, orgs fill different niches and there are different equilibria for stable org cultures where talented people like to work at, etc.
So, my point is really just “I’m pretty sure there are upsides you haven’t yet listed,” rather than, “humans with oxytocin receptors are for sure the better building blocks for forming impactful teams.”

Lastly, while I agree that, directionally, oxytocin sensitivity makes people less ambitious, I want to flag that I know many relationship-oriented but still “hardcore” effective altruistics who have found partners who are similarly ambitious or at least support their ambition (rather than just supporting them as a partner without properly respecting their ambition), or EAs who had a strong desire for finding a partner but deliberately didn’t invest much into finding someone because they saw many examples of relationships going badly and they didn’t want to jeopardize their productivity. Which is to underscore that things like oxytocin sensitivity are still only one component to the overall orientation of one’s personality (even though I agree with you that it might be the biggest individual factor).

Lukas_Gloor 13 Nov 2025 0:26 UTC
LW: 11 AF: 3
3
AF
on: Please, Don’t Roll Your Own Metaethics
By “metaethics,” do you mean something like “a theory of how humans should think about their values”?

I feel like I’ve seen that kind of usage on LW a bunch, but it’s atypical. In philosophy, “metaethics” has a thinner, less ambitious interpretation of answering something like, “What even are values, are they stance-independent, yes/no?”
And yeah, there is often a bit more nuance than that as you dive deeper into what philosophers in the various camps are exactly saying, but my point is that it’s not that common, and certainly not necessary, that “having confident metaethical views,” on the academic philosophy reading of “metaethics,” means something like “having strong and detailed opinions on how AI should go about figuring out human values.”
(And maybe you’d count this against academia, which would be somewhat fair, to be honest, because parts of “metaethics” in philosophy are even further removed from practicality, as they concern the analysis of the language behind moral claims, which, if we compare it to claims about the Biblical God and miracles, it would be like focusing way too much on whether the people who wrote the Bible thought they were describing real things or just metaphores, without directly trying to answer burning questions like “Does God exist?” or “Did Jesus live and perform miracles?”)
Anyway, I’m asking about this because I found the following paragraph hard to understand:
Behind a veil of ignorance, wouldn’t you want everyone to be less confident in their own ideas? Or think “This isn’t likely to be a subjective question like morality/values might be, and what are the chances that I’m right and they’re all wrong? If I’m truly right why can’t I convince most others of this? Is there a reason or evidence that I’m much more rational or philosophically competent than they are?”
My best guess of what you might mean (low confidence) is the following:

You’re conceding that morality/values might be (to some degree) subjective, but you’re cautioning people from having strong views about “metaethics,” which you take to be the question of not just what morality/values even are, but also a bit more ambitiously: how to best reason about them and how to (e.g.) have AI help us think about what we’d want for ourselves and others.

Is that roughly correct?
Because if one goes with the “thin” interpretation of metaethics, then “having one’s own metaethics” could be as simple as believing some flavor of “morality/values are subjective,” and it feels like you, in the part I quoted, don’t sound like you’re too strongly opposed to just that stance in itself, necessarily.

Lukas_Gloor 29 Oct 2025 23:10 UTC
3 points
0
in reply to: Jesper L.’s comment on: Reminder: Morality is unsolved
(Sorry for the late reply.)
If you don’t think moral realism is worth getting hang up on for AI and are interested in implications on AI development and AIs steering the future, then I’d recommend the last two posts in my sequence (8 and 9 in the summary I linked above). And maybe the 7th post as well (it’s quite short) since it serves as a recap of some things that will help you better follow the reasoning in post 9.
What you write in that “Moral realism” section of your other post seems reasonable! I was surprised to read that you don’t necessarily expect moral realism to apply because I thought the framing at the start of your post here (“Morality is unsolved”) suggested a moral realist outlook (because it read to me as though written by someone who expects there to be a solution). I would have maybe added something like ”… and we don’t even seem to agree on solution criteria,” after the message “Morality is unsolved,” to highight the additional uncertainty about metaethics.
That said, after re-reading, I see now that, later on, you say “We also don’t understand metaethics well”.