Lukas_Gloor

Karma: 3,954

Lukas_Gloor 12 Dec 2025 19:32 UTC
2 points
0
in reply to: Anthony DiGiovanni’s comment on: Reminder: Morality is unsolved
Ah, right. It’s not been that long yet, IMO, but if this continues for (say) 2ys in that no one changes their mind but also ~no one engages with the arguments directly and substantively, that would be disappointing.

In your case, the arguments seem more radical, unlike with arguing for anti-realism where one commonly available reason for not engaging much would be people thinking “I probably have similar enough views already.”

For me, epistemology was never my special interest, so I’m not that well-positioned to dive into the topic and try writing a critique or commentary, but I hope that someone else ends up doing it.

Lukas_Gloor 12 Dec 2025 18:44 UTC
2 points
0
in reply to: Anthony DiGiovanni’s comment on: Reminder: Morality is unsolved
If we consider the views of Oxford’s EA philosophers as also having some founding influence on LW and the broader adjacent communities, then it becomes a bit less clear how strongly the founding effect is pointing in the direction of anti-realism.

In any case, I should flag that I no longer think “no object level pushback, therefore I’m not that worried” is a good way of putting it. Instead, I would now put it as follows: “No object level pushback, therefore the burden of proof is no longer on me, and anyone who claims I’m being too confident in my views is on no firmer ground with their position than I am.”
(On whether LW is too stuck within founder effects in general: Your example would be pretty telling and damning if we assume that you’re correct, but my guess is that most readers here will assume you’re wrong about it. Someone in your position could still be right, of course; I’m just saying that this wouldn’t yet be apparent to readers.)

Lukas_Gloor 27 Nov 2025 14:42 UTC
4 points
2
in reply to: habryka’s comment on: Elizabeth’s Shortform
Many of these can complement a romantic relationship (people are often attracted to someone’s having passions/ambitions, and having a job provides stability). By contrast, dating multiple people is competing over largely similar resources, as you say. For example, you can only sleep in one person’s bed at night, can only put yourself in danger for the sake of others so many times before you might die, etc.

Just knowing that you’re splitting resources at all will be somewhat unsatisfying for some psychologies, if people emotionally value the security of commitment. I guess that’s a similar category to jealousy and the poly stance here is probably that you can train yourself to feel emotionally secure if trust is genuinely justified. But can one disentangle romance/intimacy from wanting to commit to the person your romantically into? In myself, I feel like those feelings are very intertwined. “Commitment,” then, is just the conscious decision to let yourself do what your romantic feelings already want you to do.

That said, maybe people vary in all the ways of how much these things can be decoupled. Like, some people have a signficant link between having sex and pair bonding, whereas others don’t. Maybe poly people can disentangle “wanting commitment” from romantic love in a way that I can’t? When I read the OP I was thrown off by this part: “You + your partner are capable of allowing cuddling with friends and friendship with exes without needing to make everything allowed.” To me, cuddling is very much something that falls under romantic love, and there’s a distinct ickiness of imagining cuddling with anyone who isn’t in that category. Probably relatedly, as a kid I didn’t want to be touched by anyone, not hug relatives ever, etc. I’m pretty sure that part is idiosyncratic because there’s no logical reason why cuddling has to be linked to romantic love and commitment, as opposed to it functioning more like sex in people in whom sex is not particularly linked to pair bonding. But what about the thing where the feelings of romantic love also evokes a desire to join your life together with the other person? Do other people not have that? Clearly romantic love is about being drawn to someone, wanting to be physically and emotionally close to them. I find that this naturally extends to the rest of “wanting commitment,” but maybe other people are more content with just enjoying the part of being drawn to someone without then wanting to plan their future together?
Anyway, the tl;dr of my main point is that psychologies differ and some people appear to be better psychologically adapted for monogamy than you might think if you just read the OP. (Edit: deleted a sentence here.) Actually point 10 in Elizabeth’s list is similar to what I’ve been saying, but I feel like it can be said in a stronger way.

Lukas_Gloor 27 Nov 2025 0:46 UTC
23 points
12
in reply to: Elizabeth’s comment on: Elizabeth’s Shortform
Those are mostly “analytical” reasons. I’d say sometimes people just have a psychology that is drawn to monogamy as an ideal (for reasons deeper than just struggling with jealousy otherwise), which makes them poorly suited for polyamory.
It’s said that love has three components, intimacy/romance, passion/lust/attraction, and commitment. I would say that the people to whom monogamy feels like the obviously right choice have a psychology that’s adapted towards various facets of valuing commitment. So commitment is not something that they enter if they’ve analytically gone through the pros and cons and decided that it’s net beneficial for them. Instead, it’s something they actively long for that gives purpose to their existence. Yes, it comes with tradeoffs, but that contributes to the meaning of it and they regard committedness as a highly desirable state.

If someone(’s psychology) values commitment in that way, it’s an unnatural thought to want to commit to more than one person. Commitment is about turning the relationship into a joint life goal—but then it’s not in line with your current life goals to add more goals/commitments that distract from it.

I don’t mean to say that polyamrous couples cannot also regard commitment as a desirable state (say, if they’re particularly committed to their primary relationship). If anyone poly is reading this and valuing commitment is ~their primary motivation in life, I’d be curious to learn about how this manifests. To me, it feels in tension with having romantically meaningful relationships with multiple people because it sounds like sharing your resources instead of devoting them all towards the one most important thing. But I haven’t talked to polyamorous people about this topic and I might be missing something. (For instance, in my case I also happen to be somewhat off-the-charts introverted, which means I see various social things differently from others.)

Lukas_Gloor 20 Nov 2025 19:45 UTC
2 points
0
in reply to: Wei Dai’s comment on: Reminder: Morality is unsolved
I’m not well-positioned to think about your prioritization, for all I know you’re probably prioritizing well! I didn’t mean to suggest otherwise.
And I guess you’re making the general point that I shouldn’t put too much stake into “my sequence hasn’t gotten much in terms of concrete pushback,” because it could well be that there are people who would have concrete pushback but don’t think it’s worth commenting since it’s not clear if many people other than myself would be interested. That’s fair!
(But then, probably more people than just me would be interested in a post or sequence on why moral realism is true, for reasons other than deferring, so those object-level arguments should better be put online somewhere!)

Lukas_Gloor 17 Nov 2025 13:57 UTC
8 points
5
in reply to: Wei Dai’s comment on: Reminder: Morality is unsolved
In the future we’ll have so much better tools for exploring reasoning including various assumptions. [...] Why lock in these assumptions now instead of later, after we can answer such questions (and maybe much better ones people think up in the future) and let the answers inform our choices?
- It might just be how my brain is calibrated: Even from a young age my interest in philosophy (and ethics in particular) was off the charts. I’m not sure I could have summoned that same energy and focus if I went into it with an attitude of “I should remain uncertain about most of it.” (In theory someone could hyperfocus on unearthing interesting philosophy considerations without actually committing to any of them—they could write entire books from a mode of “this is exploring the consequences of this one assumption which I’m genuinely uncertain about,” and then write other books on the other assumptions. But with the practical stakes feeling lower and without the mental payoff of coming to conclusions, it’s lot less motivating, certainly for me/my brain, but maybe even for humans in general?)
- I don’t want to be uncertain about whether I should reason in a way that seems nonsensical to me. The choice between “irreducible normativity makes sense as a concept” and “irreducible normativity does not make sense as a concept” seems really clear to me. You might say that I’m going against many very smart people in EA/rationality, but I don’t think that’s true. On LW, belief in irreducible normativity is uncommon and arguably even somewhat frown upon. Among EAs, it’s admittedly more commmon, at least in the sense that there’s a common attitude of “we should place some probability on irreducible normativity because smart people take it seriously.” However, if you then ask if there are any EAs who take irreducible normativity seriously for object-level reasons and not out of deference, you’ll hardly find anyone! (Isn’t that a weird state of affairs?!) (Lastly on this, EAs often mention Derek Parfit as someone worth deferring to, but they did so before Volume III of On What Matters was even out, and before that Parfit had written comparatively little on metaethics, so that was again indirectly deferring rather than object-level-informed deferring.)
- Opportunity costs; waiting/deferring is not cost-free: Different takes on “how to do reasoning” have different downstream effects on things like the values I would adopt, and those values come with opportunity costs. (I say more on this further down in this comment.)
>What if once we become superintelligent (and don’t mess up our philosophical abilities), philosophy in general will become much clearer, similar to how a IQ 100 person is way worse at philosophy compared to you? Would you advise the IQ 100 person to “locking in some basic assumptions about how to do reasoning” based on his current intuitions?
If it is as you describe, and if the area of philosophy in question didn’t already seem clear to me, then that would indeed convince me in favor of waiting/deferring.
However, my disagreement is that there are a bunch of areas of philosophy (by no means all of them—I have a really poor grasp of decision theory and anthropics and infinite ethics!) that already seem clear to me and it’s hard to conceive of how things could get muddled again (or become clear in a different form).
Also, I can turn the question around on you:
What if the IQ 200 version of yourself sees more considerations, but overall their picture is still disorienting? What if their subjective sense of “oh no, so many considerations, what if I pick the wrong way of going about it?” never goes away? What about the opportunity costs, then?
You might say, “So what, it was worth waiting in case it gave me more clarity; nothing of much value was lost in the meantime, since, in the meantime, we can just focus on robustly beneficial interventions under uncertainty.”
I have sympathies for that but I think it’s easier said than done. Sure, we should all move towards a world where EA would focus on outlining a joint package of interventions that captures low-hanging fruit from all plausible value angles, suffering-focused ethics as well (here’s my take). We’d discourage too much emphasis on personal values and instead encourage a framing of “all these things are important according to plausible value systems, let’s allocate talent according to individual comparative advantages.”

If you think I’m against that sort of approach, you’re wrong. I’ve written this introduction to multiverse-wide cooperation and I have myself worked on things like arguing for better compute governance and temporarily pausing AI, even though I think those things are kind of unimportant from an s-risk perspective. (It’s a bit more confounded now because I got married and feel more invested in my own life for that reason, but still...)
The reason I still talk a lot about my own values is because:
(1) I’m reacting to what I perceive to be other people unfairly dismissing my values (based on, e.g., metaethical assumptions I don’t share);
(2) I’m reacting to other people spreading metaethical takes that indirectly undermine my values (e.g., saying things like we should update to the values of smart EAs, who happen to have founder effects around certain values, or making the “option value argument” in a very misleading way;
(3) because I think unless some people are viscerally invested in doing the thing that is best according to some specific values, it’s quite likely that the “portfolio-based,” cooperative approach of “let’s together make sure that all the time-sensitive and important EA stuff gets done” will predictably miss specific interventions, particulary ones that don’t turn out to be highly regarded in the community, or are otherwise difficult or taxing to think about). (I’ve written here about ways in which people interested in multiverse-wide cooperation might fail to do it well for reasons related to this point.)
Overall, I don’t think I (and others) would have come up with nearly the same number of important considerations if I had been much more uncertain about my values and ways of thinking, and I think EA would be epistemically worse off if it had been otherwise.
The way I see it, this proves that there were opportunity costs compared to the counterfactual where I had just stayed uncertain.
Maybe that’s a human limitation, maybe there are beings who can be arbitrarily intellectually productive and focused on what matters most given their exact uncertainty distribution over plausible values, that they get up in the morning and do ambitious and targeted things (and make sacrifices) even if they don’t know much about exactly what their motivation ultimately points to. In my case at least, I would have been less motivated to make use of any potential that I had if I hadn’t known that the reason I got up in the morning was to reduce suffering.
One thing I’ve been wondering:
When you discuss these things with me, are you taking care to imagine yourself as someone who has strong object-level intuitions and attachments about what to value? I feel like, if you don’t do that, then you’ll continue to find my position puzzling in the same way John Wentworth found other people’s attitudes about romantic relationships puzzling, before he figured out he lacked a functioning oxytocin receptor. Maybe we just have different psychologies? I’m not trying to get you to adopt a specific theory of population ethics if you don’t already feel like doing that. But you’re trying to get me to go back to a state of uncertainty, even though, to me, that feels wrong. Are you putting yourself in my shoes enough when you give me the advice that I should go back to the state of uncertainty?
One of the most important points I make in the last post in my sequence is that forming convictions may not feel like a careful choice, but rather more like a discovery about who we are.

I’ll now add a bunch of quotes from various points in my sequence to illustrate what I mean by “more like a discovery about who we are”:
For moral reflection to move from an abstract hobby to something that guides us, we have to move beyond contemplating how strangers should behave in thought experiments. At some point, we also have to envision ourselves adopting an identity of “wanting to do good.”
[...]

[...] “forming convictions” is not an entirely voluntary process – sometimes, we can’t help but feel confident about something after learning the details of a particular debate.

[...]

Arguably, we are closer (in the sense of our intuitions being more accustomed and therefore, arguably, more reliable) to many of the fundamental issues in moral philosophy than to matters like “carefully setting up a sequence of virtual reality thought experiments to aid an open-minded process of moral reflection.” Therefore, it seems reasonable/defensible to think of oneself as better positioned to form convictions about object-level morality (in places where we deem it safe enough).
In my sequence’s last post I have a whole list of “Pitfalls of reflection procedures” about things that can go badly wrong, and a list on how “Reflection strategies require judgment calls,” where the meaning of “judgment call” is not that making the “wrong” decision would necessarily be catastrophic, but rather just that the outcome of our reflection might very well be heavily influenced by unavoidable early decisions that seem kind of arbitrary, and if that is the case and if we actually realize that we feel more confident about some first-order moral intuition than about which way to lean regarding the judgment calls of setting up moral reflection procedures (“how to get to IQ200” in your example), then it actually starts to seem risky and imprudent to defer to the reflection.
A few more quotes:
As Carslmith describes it, one has to – at some point – “actively create oneself.”
On why there’s not always a wager for naturalist moral realism (the wager applies only to people like you who start the process without object-level moral convictions).
Whether a person’s moral convictions describe the “true moral reality” [...] or “one well-specified morality out of several defensible options” [...] comes down to other people’s thinking. As far as that single person is concerned, the “stuff” moral convictions are made from remains the same. That “stuff,” the common currency, consists of features in the moral option space that the person considers to be the most appealing systematization of “altruism/doing good,” so much so that they deem them worthy of orienting their lives around. If everyone else has that attitude about the same exact features, then [naturalist] moral realism is true. Otherwise, moral anti-realism is true. The common currency – the stuff moral convictions are made from – matters in both cases.

[...]
Anticipating objections (dialogue):
Critic: Why would moral anti-realists bother to form well-specified moral views? If they know that their motivation to act morally points in an arbitrary direction, shouldn’t they remain indifferent about the more contested aspects of morality? It seems that it’s part of the meaning of “morality” that this sort of arbitrariness shouldn’t happen.

[...]
Critic: I understand being indifferent in the light of indefinability. If the true morality is under-defined, so be it. That part seems clear. What I don’t understand is favoring one of the options. Can you explain to me the thinking of someone who self-identifies as a moral anti-realist yet has moral convictions in domains where they think that other philosophically sophisticated reasoners won’t come to share them?
Me: I suspect that your beliefs about morality are too primed by moral realist ways of thinking. If you internalized moral anti-realism more, your intuitions about how morality needs to function could change.
Consider the concept of “athletic fitness.” Suppose many people grew up with a deep-seated need to study it to become ideally athletically fit. At some point in their studies, they discover that there are multiple options to cash out athletic fitness, e.g., the difference between marathon running vs. 100m-sprints. They may feel drawn to one of those options, or they may be indifferent.
Likewise, imagine that you became interested in moral philosophy after reading some moral arguments, such as Singer’s drowning child argument in Famine, Affluence and Morality. You developed the motivation to act morally as it became clear to you that, e.g., spending money on poverty reduction ranks “morally better” (in a sense that you care about) than spending money on a luxury watch. You continue to study morality. You become interested in contested subdomains of morality, like theories of well-being or population ethics. You experience some inner pressure to form opinions in those areas because when you think about various options and their implications, your mind goes, “Wow, these considerations matter.” As you learn more about metaethics and the option space for how to reason about morality, you begin to think that moral anti-realism is most likely true. In other words, you come to believe that there are likely different systematizations of “altruism/doing good impartially” that individual philosophically sophisticated reasoners will deem defensible. At this point, there are two options for how you might feel: either you’ll be undecided between theories, or you find that a specific moral view deeply appeals to you.
In the story I just described, your motivation to act morally comes from things that are very “emotionally and epistemically close” to you, such as the features of Peter Singer’s drowning child argument. Your moral motivation doesn’t come from conceptual analysis about “morality” as an irreducibly normative concept. (Some people do think that way, but this isn’t the story here!) It also doesn’t come from wanting other philosophical reasoners to necessarily share your motivation. Because we’re discussing a naturalist picture of morality, morality tangibly connects to your motivations. You want to act morally not “because it’s moral,” but because it relates to concrete things like helping people, etc. Once you find yourself with a moral conviction about something tangible, you don’t care whether others would form it as well.
I mean, you would care if you thought others not sharing your particular conviction was evidence that you’re making a mistake. If moral realism was true, it would be evidence of that. However, if anti-realism is indeed correct, then it wouldn’t have to weaken your conviction.
Critic: Why do some people form convictions and not others?
Me: It no longer feels like a choice when you see the option space clearly. You either find yourself having strong opinions on what to value (or how to morally reason), or you don’t.
--
I don’t think you should trust your object-level intuitions, because we don’t have a good enough metaethical or metaphilosophical theory that says that’s a good idea. If you think you do have one, aren’t you worried that it doesn’t convince a majority of philosophers, or the majority of some other set of people you trust and respect? Or worried that human brains are so limited and we’ve explored a tiny fraction of all possible arguments and ideas?
So far, I genuinely have not gotten much object-level pushback on the most load-bearing points of my sequence, so, I’m not that worried. I do respect your reasoning a great deal, but I find it hard what to do with your advice, since you’re not proposing some concrete alternative and aren’t putting your finger on some concrete part of my framework that you think is clearly wrong—you just say I should be less certain about everything, but I wouldn’t know how to do that and it feels like I don’t want to do it.
FWIW, I would consider it relevant if people I intellectually respect were to disagree strongly and concretely with my thoughts on how to go about moral reasoning. (I agree it’s more relevant if they push back against my reasoning about values rather than just disagreeing with my specific values.)
What links here?
- Wei Dai's comment on The option value argument doesn’t work when it’s most needed by Winston (EA Forum; 18 Nov 2025 0:33 UTC; 7 points)

Lukas_Gloor 15 Nov 2025 13:54 UTC
2 points
0
in reply to: Wei Dai’s comment on: Please, Don’t Roll Your Own Metaethics
Thanks! That makes sense, and I should have said earlier that I already suspected I likely understood your point and you expressed yourself well – it’s just that (1) I’m always hesitant to put words in people’s mouths, so I didn’t want to say I was confident I could paraphrase your position, and (2) whenever you make posts about metaethics, I’m wondering “oh no, does this apply to me, am I one of the people who is doing the thing he says one shouldn’t do?,” and so I was interested in prompting you to be more concrete about what level of detailedness someone’s confident opinion in that area would have to be before you think they reveal themselves as overconfident.
By “metaethics” I mean “the nature of values/morality”, which I think is how it’s used in academic philosophy.
Yeah, makes sense. I think academic use is basically that with some added baggage that adds mostly confusion. If I were to sum up what I think the use is in academic philosophy, I would say “the nature of values/morality, at a very abstract level and looked at from the lens of analyzing language.” For some reason, academic philosophy is oddly focused on the nature of moral language rather than morality/values directly. (I find it a confusing/unhelpful tradition of, “Language comes first, then comes the territory.”) As a result, classical metaethical positions at best say pretty abstract things about what values are. They might say things like “Values are irreducible (nonnaturalism)” or “Values can be reduced to nonmoral terminology like desires/goals, conscious states, etc. (naturalism),” but without actually telling us the specifics of that connection/reduction. If we were to ask, “Well, how can we know what the right values are?”—then it’s not the case that most metaethicists would consider themselves obviously responsible for answering it! Sure, they might have a personal take, but they may write about their personal take in a way that doesn’t connect their answer to why they endorse a high-level metaethical theory like nonnaturalist moral realism.
Basically, there are (at least) two ways to do metaethics, metaethics via analysis of moral language and metaethics via observation of how people do normative ethics in applied contexts like EA/rationality/longtermism. Academic philosophy does one while LW does the other. And so, to academic philosophers, if they read a comment like the one Jan Kulveit left here about metaethics, my guess is that they would think he’s confusing metaethics for something else entirely (like maybe, “applied ethics but done in a circumspect way, with awareness of the contested and possibly under-defined nature of what we’re even trying to do”).

Lukas_Gloor 15 Nov 2025 11:35 UTC
7 points
0
on: “But You’d Like To Feel Companionate Love, Right? … Right?”
Your model sounds right to me but I think there are benefits of oxytocin receptivity that go not just to the individual’s satisfication rating when they’re in a (not-awful) relationship, but also to their surroundings. For instance, I would guess that teams/organizations where many people value interpersonal relationships for their own sake can be stronger and more stable in a way that enables them to—ironically—be more ambitious and fight corruptive influences a lot better. (E.g., having high internal trust enables orgs to have flatter hierarchies and fewer levels of secrecy, which is good for team culture and epistemics. Also, if people value their social connections a lot, you have to worry less about important people leaving the org in an uncoordinated fashion if things get difficult in some way, which can be good or bad for impact depending on the specifics.) Relatedly, I think dialing up ambitiousness can go poorly, especially if you remove the safeguarding effects of certain prosocial drives or emotions.
I want to caveat the above by adding that cognitive diversity is almost certainly good for teams and I could well imagine that the perfect mix at any org includes several people to whom “work is everything” (even if that’s driven by “ambitiousness” rather than autistic hyperfixation on their special interest). Also, orgs fill different niches and there are different equilibria for stable org cultures where talented people like to work at, etc.
So, my point is really just “I’m pretty sure there are upsides you haven’t yet listed,” rather than, “humans with oxytocin receptors are for sure the better building blocks for forming impactful teams.”

Lastly, while I agree that, directionally, oxytocin sensitivity makes people less ambitious, I want to flag that I know many relationship-oriented but still “hardcore” effective altruistics who have found partners who are similarly ambitious or at least support their ambition (rather than just supporting them as a partner without properly respecting their ambition), or EAs who had a strong desire for finding a partner but deliberately didn’t invest much into finding someone because they saw many examples of relationships going badly and they didn’t want to jeopardize their productivity. Which is to underscore that things like oxytocin sensitivity are still only one component to the overall orientation of one’s personality (even though I agree with you that it might be the biggest individual factor).

Lukas_Gloor 13 Nov 2025 0:26 UTC
LW: 11 AF: 3
3
AF
on: Please, Don’t Roll Your Own Metaethics
By “metaethics,” do you mean something like “a theory of how humans should think about their values”?

I feel like I’ve seen that kind of usage on LW a bunch, but it’s atypical. In philosophy, “metaethics” has a thinner, less ambitious interpretation of answering something like, “What even are values, are they stance-independent, yes/no?”
And yeah, there is often a bit more nuance than that as you dive deeper into what philosophers in the various camps are exactly saying, but my point is that it’s not that common, and certainly not necessary, that “having confident metaethical views,” on the academic philosophy reading of “metaethics,” means something like “having strong and detailed opinions on how AI should go about figuring out human values.”
(And maybe you’d count this against academia, which would be somewhat fair, to be honest, because parts of “metaethics” in philosophy are even further removed from practicality, as they concern the analysis of the language behind moral claims, which, if we compare it to claims about the Biblical God and miracles, it would be like focusing way too much on whether the people who wrote the Bible thought they were describing real things or just metaphores, without directly trying to answer burning questions like “Does God exist?” or “Did Jesus live and perform miracles?”)
Anyway, I’m asking about this because I found the following paragraph hard to understand:
Behind a veil of ignorance, wouldn’t you want everyone to be less confident in their own ideas? Or think “This isn’t likely to be a subjective question like morality/values might be, and what are the chances that I’m right and they’re all wrong? If I’m truly right why can’t I convince most others of this? Is there a reason or evidence that I’m much more rational or philosophically competent than they are?”
My best guess of what you might mean (low confidence) is the following:

You’re conceding that morality/values might be (to some degree) subjective, but you’re cautioning people from having strong views about “metaethics,” which you take to be the question of not just what morality/values even are, but also a bit more ambitiously: how to best reason about them and how to (e.g.) have AI help us think about what we’d want for ourselves and others.

Is that roughly correct?
Because if one goes with the “thin” interpretation of metaethics, then “having one’s own metaethics” could be as simple as believing some flavor of “morality/values are subjective,” and it feels like you, in the part I quoted, don’t sound like you’re too strongly opposed to just that stance in itself, necessarily.

Lukas_Gloor 29 Oct 2025 23:10 UTC
3 points
0
in reply to: Jesper L.’s comment on: Reminder: Morality is unsolved
(Sorry for the late reply.)
If you don’t think moral realism is worth getting hang up on for AI and are interested in implications on AI development and AIs steering the future, then I’d recommend the last two posts in my sequence (8 and 9 in the summary I linked above). And maybe the 7th post as well (it’s quite short) since it serves as a recap of some things that will help you better follow the reasoning in post 9.
What you write in that “Moral realism” section of your other post seems reasonable! I was surprised to read that you don’t necessarily expect moral realism to apply because I thought the framing at the start of your post here (“Morality is unsolved”) suggested a moral realist outlook (because it read to me as though written by someone who expects there to be a solution). I would have maybe added something like ”… and we don’t even seem to agree on solution criteria,” after the message “Morality is unsolved,” to highight the additional uncertainty about metaethics.
That said, after re-reading, I see now that, later on, you say “We also don’t understand metaethics well”.

Lukas_Gloor 26 Oct 2025 12:59 UTC
2 points
0
in reply to: Wei Dai’s comment on: Reminder: Morality is unsolved
This illustrates the phenomenon I talked about in my draft, where people in AI safety would confidently state “I am X” or “As an X” where X is some controversial meta-ethical position that they shouldn’t be very confident in, whereas they’re more likely to avoid overconfidence in other areas of philosophy like normative ethics.
We’ve had a couple of back-and-forth comments about this elsewhere, so I won’t go into it much, but briefly: I don’t agree with the “that they shouldn’t be confident in” part because metaphilosophy being about how to reason philosophically, we have to do something when we’re doing that, so there is no obviously “neutral” stance and it’s not at all obviously the safe option to remain open-endedly uncertain about it (about how to reason). If you’re widely uncertain about what counts as solid reasoning in metaethics, your conclusions might remain under-defined—up until the point where you decide to allow yourself to pick one or the other of some fundamental commitments about how you think concepts work. Things like whether it’s possible for there to be nonnatural “facts” of an elusive nature that we cannot pin down in non-question-begging terminology. (One of my sequence’s posts that I’m most proud of is this one that argues that realists and anti-realists are operating within different frameworks, that they have adopted different ways of reasoning about philosophy at a very fundamental level and that basically drives all their deeply-rooted disagreements.) Without locking in some basic assumptions about how to do reasoning, your reasoning won’t terminate, so you’ll eventually have to decide to become certain about something. It feels arbitrary how and when you’re going to do that, so it seems legitimate if someone (like me) trusts their object-level intuitions already now more than they trust the safety of some AI-aided reflection procedure (and you of all people are probably sympathetic to that part at least, since you’ve written much about how reflection can go astray).
I’m pretty worried that if their assumed solution is wrong, they’re likely to contribute to making the problem worse instead of better.
Yeah, I mean that does seem like fair criticism if we think about things like “Moral subjectivists (which on some counts is a form of anti-realism) simply assuming that CEV is a well-defined thing for everyone, not having a backup plan when it turns out it isn’t, and generally just not thinking much about or seemingly not caring about the possibility that it isn’t.” But that’s not my view. I find it hard to see how we can do better than what I tried to get started on in my sequence (and specifically the last two posts, life-goals framework and moral reflection), which is to figure out how people actually make up their minds about their goals in ways that they will reflectively endorse and that we’d come to regard as wise/prudent also from the outside, and then create good conditions for that while avoiding failure modes. And figuring out satisfying and fair ways of addressing cases where someone’s reflection doesn’t get off the ground or keeps getting pulled into weird directions because they lack some of the commitments to reasoning frameworks that Lesswrongers take for granted. If I were in charge of orchestrating moral reflection, I would understand it if moral realists or people like you who have wide uncertainty bars were unhappy about it, because our differences indeed seem large. But at the same time, I think my approach to reflection would leave enough room for people to do their own thing and I think apart from maybe you and maybe Christiano in the context of his thinking about HCH and related things, I might by now be the person who has thought the most about how philosophical reflection could go wrong or why it may or may not terminate or converge (since my post on moral reflection took forever to write and I had a bunch of new insights in the process), and I think in a way that should upshift the probability that I’m good at this type of philosophy, since it led me to have a bunch of new gears-level takes on philosophical reflection that would be relevant when it comes to designing actual reflection procedures.
BTW, are you actually a full-on anti-realist, or actually take one of the intermediate positions between realism and anti-realism? (See my old post Six Plausible Meta-Ethical Alternatives for a quick intro/explanation.)
I’ve come up with the slogan “morality is real but under-defined” to describe my position—this is to distinguish it from forms of anti-realism that are more like “anything goes; we’re never making mistakes about anything; not even the 14yo who adopts Ayn Rand’s philosophy after reading just that one book is ever making a philosophical mistake.”
I see things that I like about many of the numbers in your classification, but I think it’s all gradual because there’s a sense in which even 6. has a good point. But it would be going too far if someone would say “6. is 100% right and everything else is 0% right and therefore we shouldn’t bother to have any object-level discussions about metaethics, morality, rationality, at all.”
To say a bit more:
I confidently rule out the existence of “nonnatural” moral facts—the elusive ones that some philosophers talk about—because they just are not part of my reasoning toolbox. I don’t understand how they’re supposed to work and they seem to violate some pillars of my analytical, reductionist mindset about how concepts get their meaning. (This already puts me at odds with some types of moral realists.)
However, not all talk of “moral facts” is nonnaturalist in nature, so there are some types of “moral facts” that I’m more open to at least conceptually. But those facts have to be tied to concrete identifying criteria like “intelligent evolved minds will be receptive to discovering these facts.” Meaning, once we identify those facts, we should be able to gain confidence that we’ve identified the right ones, as opposed to remaining forever uncertain due to the Open Question Argument.
I think moral motivation is a separate issue and I’m actually happy to call something a “moral fact” even if it is not motivating. As long as intelligent reasoners would agree that it’s “the other-regarding/altruistic answer,” that would count for me, even if not all of the intelligent reasoners will be interested in doing the thing that it recommends. (But note that I’m already baking in a strong assumption here, namely that my interest in morality makes it synonymous with wanting to do “the most altruistic thing”—other people may think about morality more like a social contract of self-oriented agents, who, for example, don’t have much interest in including powerless nonhuman animals in that contract, so they would be after a different version of “morality” than I’m after. And FWIW, I think the social contract facet of “morality” is real in the same under-defined way as the maximal altruism facet is real, and the reason I’m less interested in it is just because I think the maximum altruism one, for me, is one step further rather than an entirely different thing. That’s why I very much don’t view it as the “maximally altruistic thing” to just follow negative utilitarianism, because that would go against the social contract facet of morality, and at least insofar as there are people who have explicit life goals that they care about more than their own suffering, who am I to override their agency on that.)

Anyway, so, are there these naturalist moral facts that intelligent reasoners will converge on as being “the right way to do the most altruistic thing”?
My view is there’s no single correct answer, but some answers are plausible (and individually persuasive in some cases, while in other cases someone like you might remain undecided between many different plausible answers), while other answers are clearly wrong, so there’s “something there,” meaning there’s a sense in which morality is “real.”
In the same way, I think there’s “something there” about many of the proposed facts in your ladder of meta-ethical alternatives, even though, like I said, there’s a sense in which even position 6. kind of has a point.

Lukas_Gloor 25 Oct 2025 12:51 UTC
2 points
0
in reply to: leogao’s comment on: leogao’s Shortform
Trust the last person because the thing they’re doing isn’t the best thing anyone could do in their opinion?

Lukas_Gloor 24 Oct 2025 16:10 UTC
11 points
0
in reply to: Wei Dai’s comment on: Reminder: Morality is unsolved
But I argue that giving people who are unreflective and prone to value drift god-like powers to reshape the universe and themselves could easily lead to catastrophic outcomes on par with takeover by unaligned AIs, since in both cases the universe becomes optimized for essentially random values.

I wonder whether, if you framed your concerns in this concrete way, you’d convince more people in alignment to devote attention to these issues? As compared to speaking more abstractly about solving metaethics or metaphilosophy.

(Of course, you may not think that’s a helpful alternative, if you think solving metaethics or metaphilosophy is the main goal, and other concrete issues will just continue to show up in different forms unless we do it.)

In any case, regarding the passage I quoted, this issue seems potentially relevant independent of whether one thinks metaphilosophy is an important focus area or whether metaethics is already solved.
For instance, I’m also concerned as an anti-realist that giving people their “aligned” AIs to do personal reflection will likely go poorly and lead to outcomes we wouldn’t want for the sake of those people or for humanity as a collective. (My reasoning is that while I don’t think there’s necessarily a single correct reflection target, there are certainly bad ways to go about moral reflection, meaning there are pitfalls to avoid. For examples, see the subsection Pitfalls of Reflection Procedures in my moral uncertainty/moral reflection post, where I remember you made comments. There’s also the practical concern of getting societal buy-in for any specific way of distributing influence over the future and designing reflection and maybe voting procedures: even absent the concern about doing things the normatively correct way, it would create serious practical problems if alignment researchers were to propose a specific method but they’re not able to convince many others that their method was (1) even trying to be fair (as opposed to being selfishly motivated or motivated by fascism or whatever, if we imagine uncharitable but “totally a thing that might happen” sorts of criticism), and (2) did a good job at being fair given constraints of it being a tough problem with tradeoffs.

Lukas_Gloor 24 Oct 2025 15:08 UTC
2 points
0
on: Reminder: Morality is unsolved
I view “Morality is unsolved” as a misleading framing; instead, I would say it’s under-defined.
I wrote a sequence about metaethics that leaves me personally feeling satisfied and unconfused about the topic, so I also don’t agree with “we don’t understand metaethics very well.” See my sequence summary here. (I made the mistake of posting on April 2nd when the forum was flooded with silly posts, so I’m not sure it got read much by people who didn’t already see my sequence posts on the EA forum.)

Lukas_Gloor 4 Oct 2025 14:14 UTC
6 points
−2
in reply to: faul_sname’s comment on: faul_sname’s Shortform
I don’t have the same reaction to power/control/monitoring being per se very bad. It doesn’t seem comparable to me to pre-emptively nuking your enemy before even trying diplomacy.
Edit: To elaborate on why, part of it might be that I think the default of open competition is incredibly bad and ugly. (Themes being: Dawkins’ “Nature red in tooth and claw” passage about there being no purpose in nature and so much suffering, Moloch, bargaining failures getting worse and worse if you don’t somehow reign things in or dial down the maximizing.)
I also think there’s maybe a bit of a third option? Instead of having one central entity that controls everything, you could have a coalition of agents under the umbrella of peacefulness/cooperation and “not maximizing too hard,” and they together enforce some kind of monitoring and control, but it still has a value-pluralistic and somewhat Democratic feel to it?

Lukas_Gloor 30 Aug 2025 13:32 UTC
2 points
0
in reply to: Kaj_Sotala’s comment on: Defensiveness does not equal guilt
But a relevant question is, do innocent people attempt counterattacks at a significantly lower rate? If both innocent and guilty people are roughly equally likely to attempt counterattacks, then just the presence of a counterattack isn’t strong evidence.
In movies and series it happens a bunch that people find themselves accused of something due to silly coincidences, as this ramps up the drama. In real life, such coincidences or huge misunderstandings presumably happen very infrequently, so when someone in real life gets accused of serious wrongdoing, it is usually the case that either they are guilty, or their accusers have a biased agenda.
This logic would suggest that you’re right about counterattacks being ~equally frequent.

Perhaps once we go from being accused of serious wrongdoing to something more like “being accused of being a kind of bad manager,” misunderstandings, such as that the “accuser” just happened to see you on a bad day, become more plausible. In that case, operating from a perspective of “the accuser is reasonable and this can be cleared up with a conversation rather than by counterattacking them” is something we should expect to see more often from actually “innocent” managers. (Of course, unlike with serious transgressions/wrongdoing, being a “kind of bad” manager is more of a spectrum, and part of being a good manager is being open to feedback and willingness to work on improving onself, etc., so these situations are also more disanalogous for additional reasons.)

Lukas_Gloor 29 Aug 2025 15:54 UTC
4 points
0
in reply to: Kaj_Sotala’s comment on: Defensiveness does not equal guilt
Of course, if that’s the only defense they offer and they don’t bother refuting any of the actual accusations in any substantial way, that’s certainly very suspicious. But then the suspicious thing is more the lack of an object-level response rather than the presence of a defensive response.
Yeah, I’m starting with this part of your response because I agree and think it is good to have clear messaging on the most unambiguously one-directional (“guilty or not”) pieces of evidence. Nothing comes close to having persuasive responses to the most load-bearing accusations.
“It’s fine to be outraged/go on the counterattack, but it becomes suspicious if you use this to deflect from engaging with the evidence against you” seems like a good takeaway.
What shouldn’t happen is that onlookers give someone a pass because of reasoning that goes as follows: “They seem to struggle with insecurity, and getting accused is hard, so it’s okay that they’re deeming it all so outrageous that it’s beneath them to engage more on the object-level.” Or, with less explicit reasoning, but still equally suboptimal, would be an onlooker reaction of, “This is just how this person responds to accusations; I will treat this as a fact of the world,” combined with the onlookers leaving it at that and not flagging it as unfortunate (and suspiciously convenient) that the accused will now not do their best to gather information they can voluntarily disclose to immediately shed more light on their innocence.
Basically, the asymmetry is that innocent people can often (though not always) disclose information voluntarily that makes their innocence more clear/likely. That’s the best strategy if it is available to you. It is never available to guilty people, but sometimes available to innocent people.

(In fact, this trope is overused in the show “Elementary” and once I realized it, it became hard to enjoy watching the show because it’s usually the same formula for the short self-contained episodes: The initial one, two, or three suspects will almost always be red herrings, and this will become clear quickly enough because they will admit to minor crimes that make clear that they would have lacked the motive for the more serious crime, or they would admit something surprising or embarrassing that is verifiable and gives them an alibi, etc.)

So, anything that deflects from this is a bit suspicious! Justifiably accused “problem people” will almost always attempt counterattacks in one form or another (if not calling into question the accuser’s character, then at least their mental health and sanity) because this has a chance of successful deflection.
The following paragraph is less important to get to the bottom of because I’m sure we both agree that the evidence is weak at best no matter what direction it goes in, but I still want to flag that I have opposite intuitions from you about the direction of evidence.
My sense is still that the strategy “act as though you’ve been attacked viciously by a person who is biased against you because they’re bad” does weakly (or maybe even moderately, but with important exceptions) correlate with people being actually guilty. That said, that’s importantly different from your example of “being able to dig up accusation-relavant dirt”. I mean, it depends what we’re picturing… I agree that “this police accusing me has been known to take bribes and accuse innocent people before” is quite relevant and concerning. By contrast, something that would seem a lot less relevant (and therefore go in the other direction, evidence-wise), would be things like, “the person who accused me of bad behavior had too much to drink on the night in question.” Even if true, that’s quite irrelevant because problem people may sometimes pick out victims precisely because they are drunk (or otherwise vulnerable) and also because “having too much to drink” doesn’t usually turn reliable narrators into liars, so the fact that someone being drunk is the worst that can be said about them is not all that incriminating.

Lukas_Gloor 29 Aug 2025 11:26 UTC
5 points
0
on: Defensiveness does not equal guilt
When you say defensiveness, does that include something like “act as though you’ve been attacked viciously by a person who is biased against you because they’re bad”? Because that, to me, is the defensiveness behavior I’d find the most suspicious (other facets of defensiveness less so).
The problem with the “immediately focus on maximally discrediting the accusers” is that is that it is awfully close to the tactic that actually guilty people might want to use to discredit or intimidate their accusers (or, in movies, discredit law enforcement that has good reasons for asking questions/being suspicious).
Of course, in complex interpersonal contexts, it’s often the case that accusers are in fact the troublemakers (and maybe every once in a blue moon, law enforcement asking what they say are “standard” questions might be part of a conspiracy to frame you), so the behavior is only suspicious when there’s a perfectly valid explanation as to why people are pointing at you, and you not only do not see it from that perspective (or acknowledge that you’re seeing it), but you then put on behavior designed to make onlookers believe that something incredibly outrageous has just happened to you.
One admittedly confounding factor is “honor culture”—not a big thing in LW circles, but if we’re thinking of movies where they arrest or ask accusing questions to people in regions or cultures where one’s reputation is really important, and being accused of something is seen as a massive insult, then I can understand that this is a strong confounding factor (to actually being guilty).

Lukas_Gloor 27 Aug 2025 22:56 UTC
2 points
0
in reply to: eggsyntax’s comment on: eggsyntax’s Shortform
The rule itself sounds reasonable but I find it odd that it would come up often enough. Here’s an alternative I have found useful: Disengage when people are stubborn and overconfident. It seems like a possible red flag to me if an environment needs rules for how to “resolve” factual disagreements. When I’m around reasonable people I feel like we usually agree quite easily what qualifies as convincing evidence.

Lukas_Gloor 24 Aug 2025 21:02 UTC
10 points
4
in reply to: clone of saturn’s comment on: Banning Said Achmiz (and broader thoughts on moderation)
I think the people who talk as though the contested issue here is Said’s disagreeableness combined with him having high standards are missing the point.
Said Achmiz, in contrast, expresses some amount of contempt for people who do fairly specific and circumscribed things like write posts that are vague or self-contradictory or that promote religion or woo.
If it was just that (and if by “posts that are vague” you mean “posts that are so vague that they are bad, or posts that are vague in ways that defeat the point of the post”), I’d be sympathetic to your take. However, my impression is that a lot more posts would trigger Said’s “questioning mode.” (Personally I’m hesitant to use the word “contempt,” but it’s fair to say it made engaging more difficult for authors and they did involve what I think of as “sneer tone” sometimes.)
The way I see it, there are posts that might be a bit vague in some ways but they’re still good and valuable. This could even be because the post was gesturing at a phenomeon with nuances where it would require a lot of writing (and disentanglement work) to make it completely concise and comprehensive, or it could be because an author wanted to share an idea what wasn’t 100% fleshed out but might have already been pointing at something valuable. I feel like Said not only has a personal distaste of that sort of “post that contains bits that aren’t pinned down,” but it also seemed like he wouldn’t get any closer to seeing the point of those posts or comments when it was explained in additional detail. (Or, in case he did eventually see the points, he’d rarely say thanks or acknowledged that he got it now). That’s pretty frustrating to deal with for authors and other commenters.
(Having said all that, I have not had any problems with Said’s commenting in the last two years—though I did find it strongly negative and off-putting before that point. And to end with something positive, I liked that Said was one of the few LessWrongers who steered back a bit against Zvi’s very one-sided takes on homeschooling—context here.)