sloonz

Karma: 61

sloonz 6 Apr 2026 11:54 UTC
1 point
0
in reply to: AlphaAndOmega’s comment on: Contra The Usual Interpretation Of “The Whispering Earring”

If you think that you have a better grasp on what constitutes eudaimonia for the majority, feel free to share.

Have you read the Fun Theory sequence ? Because I don’t think I have any insight that isn’t there. Mostly the 3D vs 4D distinction in “High Challenge”.

What I am saying is that the observed behavior is entirely consistent with that depiction, and could potentially conserve qualia and thought if substrate-independence is correct

Two ways to read this.

“If substrate-independence is correct, then qualia and thought can potentially be conserved in the system”. I agree, but I’ll also note that it’s far from being likely (Stockfish playing chess), and the “system” part is important (in the story, it is noted that there is physiological change in the brain).

“If substrate independence is correct (which is not certain) then it is certain that thought and qualia are conserved”, then just no. For the above reasons.

I do not think that it is possible, even in principle, to mimic something as complex as human cognition with nigh perfect accuracy without qualia arising. In other words, I am very suspicious of arguments in favor of p-zombies.

I am, too, very suspicious of p-zombies.

I don’t see how that’s relevant.

Nothing is required to “mimic something as complex as human cognition with nigh perfect accuracy” in the whispering earring story. The only thing that is mimic-ed is a representation of the goals/values of the wielder (and even there, I see nothing that ask for nigh perfect accuracy). Then the cognitive part (creating a plan) can be as inhuman as we want.

Stockfish can play chess better than humans without instantiating an human-player-congnition-engine that would instantiate a human-player-subejctive-experience. Similarly, Whispering Earring can play “How to Win at Life” (or any other goal) without instantiating a human-like-cognition-engine (that would, presumably, trigger human-like-subjective-experience).

I don’t know why you bring upload. The whispering earring is not an upload technology, never been presented as such, and it has never been the point of the thought experiment ? And you seem to agree with that earlier ?

Is your claim something like : “Human levels of intelligence (or greater) needs human-like cognition, and if you add human goals to that, it start to get arbitrarily close to mind-upload” ?

sloonz 6 Apr 2026 10:53 UTC
2 points
0
in reply to: AlphaAndOmega’s comment on: Contra The Usual Interpretation Of “The Whispering Earring”

So do my Ritalin pills, in the sense that I usually make better decisions or at least act on them. I do not see the problem here, as long as the “better” decisions are the kind of decision I want to make, or would have made if I didn’t have ADHD.

Your ritalin pills make you a better person both from the point of view of pre-pilled-you and post-pilled you. The “better” decisions are downstream of that. Similarly, there’s many pills I would be happy to take, that would make me a better person from my current-point-of-view (and that I expect my modified-self would endorse). The whispering earring is not one of them : my current me do not want to be replaced by a learn-and-execute-reflexes-machine.

Back to the whispering earring : the internal reasoning traces matter, it’s a big part of what makes me “me”. For example, outside of my observed actions, there is a load-bearing internal world model that I do value intrinsically — remove it, and you remove a good chuck of what people call “consciousness”. For another example, outside of action-memories, I have choice-memories ; memories of actions I considered but decided not to undertake. Those memories are not behaviorally observable, but they do matter to me too, they are also something that I consider part of “me”, and are removed by the whispering-earring.

I simply do not care about those qualia very much, at least not for their own sake

Then that’s the disagreement. I do care about them very much.

An actual AGI/ASI must probably still think about things, instead of magically coming to the correct conclusion out of nowhere

So does Stockfish when playing chess. “Thinking instead of coming to the correct conclusion out of nowhere” is not that much of a constraining requirement, and therefore not much a defining thing of “what is valuable in human experience”.

I am quite literally a man who struggles to be happy. I have clinical depression as well as the ADHD. […]. I am a transhumanist, so I am willing to give up quite a lot more, assuming I come out ahead on the metrics I care about.

Okay, let me be blunt. Chesterton fence in axiology/aesthetics : if you can’t pass the ITT of “what happy humans find valuable”, please do not try to recreate it from first principles. You will miss a lot of things that happy humans indeed do find valuable.

Remember Maslow’s hierarchy. If you ask someone who struggle to meet the basics, “What is self-actualization ? Is this valuable ?”, he will look at you funnily at best, lash out at you at worst. His reaction will be something like “what’s this self-actualization thing worth ? Only food and security matter to me”. This does not prove that self-actualization is worthless, not even that it’s worthless for him, only that it’s worthless for him in his current situation (in a better situation, he would care about it) ; but also : his reaction is perfectly understandable — he is neither obtuse nor stupid. He has his priorities straight.

And this is exactly how I read your “I do not care about those qualia very much”. You’re struggling with happiness, depression, anxiety. I can empathize that to you, getting out of that struggle is supremely important — it must look like the end game, the victory condition. But please do not do the equivalent of throwing “self-actualization” out of the window just because you’re hungry.

sloonz 4 Apr 2026 16:54 UTC
2 points
0
on: I Changed My Mind about Error-Correcting Debate, Misogyny and More: Updates from a Former Student of David Deutsch
Here is mostly unedited draft of an unpublished post idea I had some years ago now. I think it’s a great explanation of your observation “thoughts leaders don’t talk about the best ideas on their own sides” (it was drafted in the language of memes rather than ideas, but the two are close enough) :

Definition : some memes would crumble / have a drastically lower fitness without other memes. I call those other memes the epistatic blanket of a meme (I choose the name as a mix of epistasis in genetics, and markov blanket in statistics). This phenomenon is obvious and impossible to miss in science and mathematics (“standing on the shoulder of giants”), but is universal. As a trivial example, language is in the epistatic blanket of almost every meme.

Law of Epistemic Decline : the epistatic blanket of a meme or memeplex in the process of being fixed is not the same as the epistatic blanket required for maintaining that fixation. This is true both in the fixation inside a single mind and inside a large group of persons.

We do not teach mathematics and science by following the exact path that past scientists and mathematicians followed ; that path is way too convoluted. Individually, once we reach a conclusion, we tend to forget the complicated reasons leading to that conclusion, retaining (at best) a simple narrative to justify that conclusion. In general, both in individuals and groups, the epistatic blanket maintaining fixation is (relative to the one leading to fixation) simpler, cleaner, and full of bullshit (oversimplistic narratives, circular beliefs “I believe A is true because B is true, I believe B is true because C is true, I believe C is true because A is true”). In the individual, it manifests mostly as mediocre justifications for a belief (“the successor of the primorial of n is prime, so there is no biggest prime”). In groups, it manifests mostly as conformism (“I believe X because everyone knowns that X is true” — you do not believe that the earth is round because you have followed complicated philosophical and scientific arguments why it is so, you believe it because every respectable person around you says so, first of all scientists and the NASA).

Note that while I mostly presented it in a pejorative way, it is a necessary and mostly beneficial fact ; if you had to hold in your mind the full complex and correct justification of all your beliefs, the most complex belief you could hold in your mind would probably be approximately around the level of “the sum of two even numbers is even” (geniuses could maybe reach high school calculus).

sloonz 4 Apr 2026 15:48 UTC
9 points
1
on: Am I the baddie?

Am I the baddie?

You’re not, in my opinion. The opposite. We should have more people like you.

One of the big problem with AI is slowness of diffusion. It leads to a societal-wide lack of situational awareness. “AI is just talk and hype, no big deal, we can’t even see it in the GDP/employment numbers !”

Yes, it would be better if we could get that societal-wide situational awareness by anticipating what AI is likely to bring, instead of “fuck around and find out” approach. It won’t happen. Therefore, we do need as fast diffusion as possible.

sloonz 22 Feb 2026 7:29 UTC
3 points
−2
on: If you don’t feel deeply confused about AGI risk, something’s wrong

By “the thing,” I mean something like developing a first-principles understanding of why you believe AI is dangerous, such that you could reconstruct the argument from scratch without appealing to authority.

It is easy.

We do not have a theory of victory, or a win condition.

The usual (best) answer is “we solve alignment and build a glorious transhumanist future”, without having a formal definition of what “solving alignment” means when it start involving real humans as the thing we want our AI systems to be aligned to (vague gestures towards CEV) or a clear aesthetic vision of what “glorious transhumanist future” means (vague gestures towards end of suffering).

If we have no theory of victory, we’re going to lose : the actual outcome is still a precise thing, and even in the best case where our vague intuition is somewhat somehow met, some process (“random shit go !”) will have to go from our fuzzy, confused desiderata to a precise outcome (that we can’t even foresee or judge because we don’t even know what we want or what it looks like).

Is there holes in this consideration ? Yes. Maybe we could build an ASI-teacher that guides us through that (but also : there is obvious problems to that there). Maybe we could do stuff sufficiently slowly that we could decide (“we” ? how ?) and steer (but see how it’s hard to even pause — steering is another beast entirely) as things advance and go clearer. The main takeaway is still “here be dragons”.

Related but still off-topic : the entire field is advancing with its priority backwards. We’re building ASI before solving alignment ; working on solving alignment before asking what we want collectively ; asking what we want collectively before asking what we want personally. Everyone is trying to run before even learning to walk. Of course we’re all going to fall.

sloonz 5 Jul 2025 11:37 UTC
1 point
0
on: ‘AI for societal uplift’ as a path to victory

The bar for ‘good enough’ might be quite high

That bar for “good enough” may also be above “unacceptable”, requiring eusocial levels of coordination where individuals are essentially drones.

sloonz 25 Jun 2025 7:49 UTC
1 point
0
on: Subjectivism and moral authority
I don’t find it that mysterious ? That “Stop !” is obviously, nakedly a call to the superego. The authority you’re seeking is the same authority that shapes the superego and is social in nature. It’s an appeal to the shared values that the society is expecting its members to have.

sloonz 25 Jun 2025 7:42 UTC
1 point
0
on: Care and demandingness
Pretty sure people that are using the “demanding” objection are not articulated analytical philosophers. They’re just trying to communicate their intuitive objection, not making a theoretical claim. Or : demandingness is not a definitive proof that it’s false, demandingness is intuitive evidence false.

And I think I can point to a less instrumental reason and a more theoretical one ?

It is well accepted that from the Universe (or God, or Objective, or whatever) point of view, everyone has the same moral weight (1). Let’s take that for granted.

Then utilitarians go around and say “therefore, using morality to guide you action, you should weight every human being equally” (2).

But (2) does absolutely not follow from (1), and I think the “demanding” objection is actually a objection to (2). It’s the good old question of “is distance relevant for moral considerations ?” (physical and/or in the social graph), and those saying “too demanding” are those answering “yes” against your “no” to that old question.

sloonz 18 Jun 2025 9:33 UTC
3 points
0
on: Agentic Interpretability: A Strategy Against Gradual Disempowerment
I don’t understand how it is an answer to gradual disempowerement.

In the GD scenario, both individual humans and AI understand that :
- AI-powered governance/economy leads to better “utilitarian outcome” (more wealth, better governance)
- AI and humans understand that this leads to disempowerement
- Moral/economics pressure leads to that disempowerement
The problem is not in the understanding. It is in the incentives. The threat model was never “oops, we did not understood that dynamics, and now we are disempowered… if only we knew…”. We already know before it happens — that’s the point of the original GD paper. Having an LLM understand and explain that dynamics does not help avert that scenario ?

sloonz 30 May 2025 6:55 UTC
2 points
0
on: Alignment Crisis: Genocide Denial
You’re missing the point that many, many humans will reply like the LLMs. “It’s complicated”. Given that, I fail to see it as a “clear misalignment with respect to human values” problem rather than “what are those humans values in the first place ?” problem.

sloonz 6 Feb 2025 13:42 UTC
−1 points
−2
in reply to: Nicholas Kross’s comment on: We Fell For It
I think it was correct to ask the question, because, no, sorry, I don’t think you can equate that specific understanding of “property is downstream of power” with the same level of consensus/reasonableness as “prices are downstream of supply and demand”, whereas the understanding given by Viliam is pretty much inside of what I would expect to be as consensual as “prices are downstream to supply and demand”.

For example, regarding your second point : I think most economists would reply that that distinction is fallacious when considering creative destruction, both mistake (having a more correct view of the market and business organization) and conflict (and thereby driving your competitors bankrupt, forcing the reallocation of their resources elsewhere) as the two sides of the same coin.

And “innovation” not being a strong path to successful outcomes in capitalism ? Do you really think it is going to be consensual outside of leftist spheres ? Because that’s what the overall quote/tweet is about, right, creating a common understanding across political chasms under some reasonable assumptions ?

First point is really interesting tho. It’s one my libertarian self is screaming to reject, but the rationalist me readily accepts. With a caveat : I don’t think “Capitalism” does a lot of work here. Almost all (and all known ?) societal organizations do that. Which, conceded, does not it makes less problematic.

sloonz 5 Feb 2025 23:44 UTC
1 point
−2
on: We Fell For It

I wish that both the “tip top” and the “medium top” of educated people had consensus that both “property is downstream of power” and “prices are downstream of supply and demand”.

I really don’t like that quote. I can see multiple interpretations for “property is downstream of power”, many of them subject to debate, and none of the firm grounding (and non-ambiguity) of Econ 101 enjoyed by “price are downstream of supply and demand”.

It looks like to me a false equivalence, trying to say “see ? both sides clearly have a point”, where one point is actually clear, and the other isn’t so clear.

Would you mind clarifying what the consensus should be exactly on this point ? Maybe I’m just not interpreting “property is downstream of power” correctly.

sloonz 3 Feb 2025 14:00 UTC
1 point
0
in reply to: Noosphere89’s comment on: Gradual Disempowerment, Shell Games and Flinches

very bad fates for all humans

I believe there’s also a disagreement here where the same scenario will be considered fine by some and very bad by others (humans as happy pets comes to mind).

sloonz 16 Jan 2025 17:05 UTC
4 points
1
on: What Is The Alignment Problem?

How to generalize to multiple humans is… not an unimportant question, but a question whose salience is far, far out of proportion to its relative importance

I expect it to be the hardest problem, not from a technical point of view, but from a lack of ground truth.

The question “how do I model the values of a human” has a simple ground truth : the human in question.

I doubt there’s such a ground truth with “how do I compress the values of all humans in one utility function ?”. “All models are wrong, some are useful”, and all that, except all the different humans have a different opinion on “useful”, ie their own personal values. There would be a lot of inconsistencies ; while I agree with your stance “Approximation is part of the game” for modeling the value of individual persons, people can wildly disagree on what approximations they are okay with or not, mostly based on the agreement between the outcome and their values.

In other words : do you believe in the existence of at least a model where nobody can honestly say “the output of that model approximates away too much of my values” ? If yes, what makes you think so ?
What links here?
- Noosphere89's comment on tailcalled’s Shortform by tailcalled (21 Jan 2025 14:24 UTC; 2 points)

sloonz 16 Jan 2025 14:34 UTC
1 point
0
in reply to: jbash’s comment on: In Defense of a Butlerian Jihad

Failure in itself is valuable to you?

What I sense from this is that what you’re not getting is that my value system is made of tradeoff of let’s call it “Primitive Values” (ie one that are at least sufficiently universal in human psychology that you kind of can describe them with compact words).

I obviously don’t value failure. If I did I would plan for failure. I don’t. I value/plan for success.

But if all plans ultimately lead to success, what of use/fun/value is planning ?

So failure has to be part of the territory, if I want my map-making skills to… matter ? make sense ? make a difference ?

It feels to me like a weird need to make your whole life into some kind of game to be “won” or “lost”, or some kind of gambling addiction or something.

My first reaction was “no, no, gambling addiction and speaking of Winning at Life like Trump could looks like terribly uncharitable”.

My second reaction is you’re pretty much directionaly right and into the path of understanding ? Just put it in a bit more charitable way ? We have been shaped by Evolution at large. By winners in the great game of Life, red in blood and claws. And while playing don’t mean winning, not playing certainly means losing. Schematically, I can certainly believe that “Agency” is the shard inside of me that comes out of that outer (intermediate) objective “enjoy the game, and play to win”. I have the feeling that you have pretty much lost the “enjoy the game” shard, possibly because you have a mutant variant “enjoy ANY game” (and you know what ? I can certainly imagine a “enjoy ANY game” variant enjoying UBI paradise).

Well, the big stakes are already gone. If you’re on Less Wrong, you probably don’t have much real chance of failing so hard that you die, without intentionally trying. Would your medieval farmer even recognize that your present stakes are significant?

This gives me another possible source/model of inspiration, the good old “It’s the Journey that matters, not the Destination”.

Many video games have a “I win” cheatcode. Players at large don’t use it. Why not, if winning the game is the goal ? And certainly all of their other actions are consistent with the player want to win the game. He’s happy when things go well, frustrated when they go wrong, At the internet age, they look at guides, tips. They will sometimes hand the controller to a better player after being stuck. And yet they don’t press the “I win” button.

You are the one saying “do you enjoy frustration or what ? Just press the I Win button”. I’m the one saying “What are you saying ? He’s obviously enjoying the game, isn’t he ?”.

I agree that the Destination of Agency is pretty much “there is no room left for failure” (and pretty much no Agency left). This is what most of our efforts go into : better plans for a better world with better odds for us. There’s some Marxist vibes “competition tend to reduce profit over time in capitalist economies, therefore capitalism will crumble under the weight of its own contradiction”. If you enjoy entrepreneurship in a capitalistic economy, the better you are at it, the stronger you drive down profits. “You: That seems to indicate that entrepreneurs hate capitalism and profits, and would be happy in a communist profit-less society. Me: What ?”. Note we have the same thing as “will crumble under the weights…” in the game metaphor : when the player win, it’s also the end of the game.

So let’s go a bit deeper into that metaphor : the game is Life. Creating an ASI-driven UBI paradise is discovering that the developer created a “I Win” button. Going into that society is pressing that button. Your position I guess is “well, living well in an UBI paradise is the next game”. My position is “no, the UBI paradise is still in the same game. It’s akin to the Continue Playing button in a RTS after having defeated all opponents on the map. Sure, you can play in the sense you can still move units around gather resources and so on but c’mon, it’s not the same, and I can already tell how much it’s going to be much less fun, simply because it’s not what the game was designed for. There is no next game. We have finished the only game we had. Enjoy drawing fun patterns with your units while you can enjoy it ; for me I know it won’t be enjoyable for very long.”

… and if you care, your social prestige, among whoever you care about, can always be on the table, which is already most of what you’re risking most of the time.

Oh, this is another problem I thought of, then forgot.

This sounds like a positive nightmare to me.

It seems a hard-to-avoid side-effect of losing real stakes/agency.

In our current society, you can improve the life of others around you in the great man-vs-nature conflict. AKA economics is positive-sum (I think you mentioned something about some people talking about Meaningfulness giving you an altruistic definition ? There we are !).

Remove this and you only have man-vs-man conflicts (gamified so nobody get hurt). Those are generally zero-sum, just positional. When you gain a rank in the Chess ladder, another one lose one.

No place for positive-sum games seems a bad place to live. Don’t know at what extent it is fixable in the UBI-paradise (does cooperative, positive-sum games fix this ? I’m not sure how much the answer is “obviously yes” or “it’s just a way to informally make a ranking of who is the best player, granting status, so it’s actually zero sum”), or how much is it just going to end up Agency in another guise.

Forces mostly unknown and completely beyond your control have made a universe in which you can exist, and fitted you for it. You depend on the fine structure constant. You have no choice about whether it changes. You need not and cannot act to maintain the present value. I doubt that makes you feel your agency is meaningless.

My first reaction is “the shard of Agency inside me has been created by Evolution ; the definition of the game I’m supposed to enjoy and its scope draws from there. Of course it’s not going to care about that kind of stuff”.

My second reaction is : “I certainly hope my distant descendants will change the fine-structure constant of the universe, it looks possible and a way to avoid the heat death of the universe” (https://www.youtube.com/watch?v=XhB3qH_TFds&list=PLd7-bHaQwnthaNDpZ32TtYONGVk95-fhF&index=2). I don’t know how much it’s a nitpick (I certainly notice that I prefer “my distant descendants” to “the ASI supervisor of UBI-paradise”).

More likely, other humans could kill you, still in a way you couldn’t influence, for reasons you couldn’t change and might never learn. You will someday die of some probably unchosen cause.

This is the split between Personal Agency and Collective Agency. At our current level at capabilities, it doesn’t differentiate very much. It will certainly, later.

Since we live in society, and much people tend to not like being killed, we shape societies such that such events tend not to happen (mostly via punishment and socialization). Each individual try to steer society at the best of its capabilities. If we collectively end up in a place where there’s no murders, people like me consider this a success. Otherwise, a failure.

Politics, advocacy, leading-by-example, guided by things like Game Theory, Ethics, History. Those are very much not out of the scope of Agency. It would be if individuals had absolutely 0 impact on society.

It’s all very nice to talk about being able to fail, but you don’t fail in a vaccuum. You affect others. Your “agentic failure” can be other people’s “mishap they don’t control”. It’s almost impossible to totally avoid that. Even if you want that, why do you think you should get it?

That’s why, for me and at my current speculation level, I think there is two Red Bright Lines for a post-ASI future.

One : if there is no recognizable Mormons society in a post-ASI future, something Has Gone Very Wrong. Mormons tend to value their traditional way of life pretty heavily (which includes agency). Trampling those in particular probably indicate that we are generally trampling a awful lot of values actually held by a lot of actual people.

Two : if there is no recognizable UBI paradise in a post-ASI future, something Has Gone Very Wrong. For pretty much the same reason.

(there is plausibly a similar third red line for transhumanists, but they cause serious security/safety challenges for the rest of the universe, so it’s getting more complicated there, so I found no way to articulate such a red line for them).

The corollary being is : the (non-terribly-gone-wrong) pot-ASI future is almost inevitably a patchwork of different societies with different tradeoffs. Unless One Value System wins, one which is low on Diversity on top of that. Which would be terrible.

To answer you : I should get that because I’m going to live with other people who are okay that I get that, because they want to get it too.

“But don’t you see, Sparklebear? The value was inside of YOU all the time!”

I entirely agree with you here. It’s all inside us. If there was some Real Really Objectively Meaningful Values out there, I would believe a technically aligned ASI to be able to recognize this and would be much less concerned by the potential loss of Agency/Meaningfulness/whatever we call it. Alas, I don’t believe it’s the case.

sloonz 13 Jan 2025 23:20 UTC
1 point
0
in reply to: jbash’s comment on: In Defense of a Butlerian Jihad

Being hung up on that seems deeply weird to me for a bunch of reasons that I could name that you might not care to hear about

Yeah, I’m curious. The only reason I know that makes sense for not caring about that is pretty extreme negative utilitarianism that you apparently don’t agree with ? (if you have agency you can fail in your plans and suffer, and That Is Not Allowed)

Would you take that?

Given an AGI, there’s a big concern whether this is a true proposal, or a lie going from “and secretly a vast majority of the rest of that world is a prop, you don’t really risk anything” to “I’m going to upload you to what is essentially a gigantic MMO”. But I think it’s not the purpose of your thought experiment ?

I think there are better intermediate places between “medieval farmer” and “UBI paradise”, if it’s what you mean by “details to your tastes”. Current society. Some more SF-like setups like : “we give you and some other space-settler-minded individuals that galaxy other there and basic space tech, do whatever you want”. Some of those I go there without a second thought. I pretty much like current society, actually, setting AGI-builders aside (and yes, limiting to developed world). Medieval farmer life is genuinely sufficiently terrible that I’m on the fence between death and medieval farmer.

But yes, between just medieval farmer and UBI paradise, I’ll probably give a test to UBI paradise (I might be proven wrong and was too lacking in imagination to see all the wonderful things there !), milk the few drop of util that I expect to still find there, but my current expectations is I’m going to bail out at some point.

Or does having to choose it spoil it?

There are various levels of “spoils it”. Your proposal is on the very low ends of spoiling it. Essentially negligible, but I think I can empathize with people thinking “it’s already too high levels of spoiling”. On increasing levels there are “and you can decide to go back to UBI society anytime” (err… that’s pretty close to just being a big IRL role-playing game, isn’t it ?) up to “and I can give you a make-a-wish button” (“wait, that’s basically what I wanted to escape”).

And it’s pretty much a given that it’s a level of Agency/Meaningfulness that is going to be lost even in the worlds where the Agency/Meaningful crowd get most of what they want, as part of bargaining, unless we somehow end up just blindly maximizing Agency/Meaningfulness. Which to be clear would be a pretty awful outcome.

sloonz 13 Jan 2025 19:08 UTC
1 point
0
in reply to: jbash’s comment on: In Defense of a Butlerian Jihad
I get the sense that you were just trying to allude to the ideas that--
```
Even if you have some kind of "alignment", blindly going full speed ahead with AI is likely to lead to conflict between humans and/or various human value systems, possibly aided by powerful AI or conducted via powerful AI proxies, and said conflict could be seriously Not Good.
```
Claims that “democratic consensus” will satisfactorily or safely resolve such conflicts, or even resolve them at all, are, um, naively optimistic.

It might be worth it to head that off by unspecified, but potentially drastic means, involving preventing blindly going ahead with AI, at least for an undetermined amount of time.

If that’s what you wanted to express, then OK, yeah.
Yes. That’s really my central claim. All the other discussions over values is not me saying “look, we’re going to resolve this problem of human values in one lesswrong post”. It was to point to the depth of the issue (and, one important and I think overlooked point, that it is not just Mistake Theory that raw clarity/intelligence can solve, there is a fundamental aspect of Conflict Theory we won’t be able to casually brush aside) and that it is not idle philosophical wandering.

I’m sorry that came off as unduly pugnacious. I was actually reacting to what I saw as similarly emphatic language from you (“I can’t believe some of you...”), and trying to forcefully make the point that the alternative wasn’t a bed of roses.

Don’t be sorry, it served its illustration purpose.

We lesswronger are a tiny point in the space of existing human values. We are all WEIRD or very close to that. We share a lot of beliefs that, seen from the outside, even close outside like academia, seems insane. Relative to the modal human who is probably a farmer in rural India or China, we may as well be a bunch of indistinguishable aliens.

And yet we manage to find scissor statements pretty easily. The tails come apart scarily fast.

It just seemed awfully glib and honestly a little combative in itself.

I don’t see how glib and combative “this post is already too long” is ?

“obvious” probably is, yes. My only defense is I don’t have a strong personal style, I’m easily influenced, and read Zvi a lot, who has the same manner of overusing it. I probably should be mindful to not do it myself (I removed at least two on drafting this answer, so progress !).

Well, yes, but I did say “as least not while the ‘humans’ involved are recognizably like the humans we have now”. I guess both the Superhappies and the Babyeaters are like humans in some ways, but not in the ways I had in mind.

No, I mean recognizable humans having an AGI in their hand can decide to go the Superhappies way. Or Babyeaters way. Or whatever unrecognizable-as-humans way. The choice was not even on the table before AGI, and that represent a fundamental change. Another fundamental change brought by AGI is the potential for an unprecedented concentration of power. Many leaders had the ambition to mold humanity to their taste ; none had the capacity to.

Some people definitely have a lot of their self-worth and sense of prestige tied up in their jobs, and in their jobs being needed. But many people don’t. I don’t think a retail clerk, a major part of whose job is to be available as a smiling punching bag for any customers who decide to be obnoxious, is going to feel too bad about getting the same or a better material lifestyle for just doing whatever they happen to feel like every day.

I think a lot of people have that. There’s a even meme for that “It ain’t much, but it’s honest work”.

All in one, I don’t think either of us has much more evidence that a vague sense of things anyway ? I sure don’t have.

I remember hearing things close to “my agency is meaningful if and only if I have to take positive, considered action to ensure my survival, or at least a major chunk of my happiness”.

I think that’s the general direction of the thing we’re trying to point, yes ?

A medieval farmer who screw up is going to starve. A medieval farmer who does exceptionally well will have a surplus he can use on stuff he enjoys/finds valuable.

A chess player who screw up is going to lose some ELO points (and some mix of shame/disappointment). A chess player who does exceptionally well will gain some ELO points (and some mix of pride/joy).

If you give me the choice of living the life of a medieval farmer or someone who has nothing in his life but playing chess, I will take the former. Yes, I know it’s a very, very hard life. Worse in a lot of ways (if you give me death as a third choice, I will admit that death starts to become enticing, if only because if you throw me in a medieval farmer life I’ll probably end up dead pretty fast anyway). The generator of that choice is what I (and apparently others) are trying to point with Meaningfulness/Agency.

I think a lot of things we enjoy and value can be described as “growing as a person”.

Does “growing as a person” sounds like a terminal goal to you ? It doesn’t to me.

If it’s not, what is it instrumental to ?

For me it’s clear, it’s the same thing as the generator of the choice above. I grow so I can hope to act better when there’s real stakes. Remove real stakes, there’s no point in growing, and ultimately, I’m afraid there’s no point to anything.

Is “real stakes” easier to grasp than Agency/Meaningfulness ? Or have I just moved confusion around ?

I’ve also heard plenty of people talk about “meaningfulness” in ways that directly contradict your definition.

Well, the problem is that there is so much concepts, especially when you want to be precise, and so few words.

My above Agency/Meaningfulness explanation does not match perfectly with the one in my previous answer. It’s not that I’m inconsistent, it’s that I’m trying to describe the elephant from different sides (and yeah, sure, you can argue, the trunk of the elephant is not the same thing as the leg of the elephant).

That being said I don’t think they point to completely unrelated concepts. All of those definitions above “positive, considered actions...” ? “Broad Sweep of History” ? its collective version ? Yeah, I all recognize them as parts of the elephant. Even the altruistic one, even if I find that one a bit awkward and maybe misleading. You should not see them as competing and inconsistent definitions, they do point to the same thing, at least for me.

Try to focus more on the commonalities, less on the distinctions ? Try to outline the elephant from the trunk and legs ?

sloonz 12 Jan 2025 17:56 UTC
1 point
0
in reply to: Shankar Sivarajan’s comment on: In Defense of a Butlerian Jihad
Because the debate is never set in terms of “better education” vs “teachers preference”. It’s “give more money for teachers so they can give better education”. When there’s a tradeoff, it’s usually of the form “better top-performers education” or “more equality on education” ? I don’t see teachers unions arguing that school vouchers are good for children but should still be outlawed. I see teachers unions arguing that school vouchers are bad for children, and They’re The Experts, so outlaw them.

I don’t expect that tactic to work when the alternative is a literal superintelligence.

sloonz 12 Jan 2025 14:59 UTC
1 point
0
in reply to: Shankar Sivarajan’s comment on: In Defense of a Butlerian Jihad
I’m not sure why you think it matters ?

I was mostly speaking about the democratic consensus here, but I’m also pretty sure that it’s also perfectly reasonable opinions, each point taken in isolation.

If you’re going to argue that the preference of doctors is more important that the welfare of patients, I’m genuinely interested by your arguments.

sloonz 12 Jan 2025 14:37 UTC
5 points
2
in reply to: jbash’s comment on: In Defense of a Butlerian Jihad

This (a) doesn’t have anything in particular to do with Christianity, (b) has been the most widely held view among people in general since forever, and (c) seems obviously correct. If you want to rely on the contrary supposition, I’m afraid you’re going to have to argue for it.

Yes, I agree that is the least obviously “wrong” part of the three “copes”, and merge with the next remark. It’s very hard to answer that. I’ll start with the simple answer that will convince some, but perhaps not everyone :

I am very low in the “Negative Utilitarianism” scale. I really don’t care much about minimizing suffering in the universe. Still a bit, sure, but not that much. Still, I recognize it is very important to some persons, my current best rules for creating a “Best Model of Human Values” says these persons count, so it’s a pretty good Existence Proof that it’s a Pretty Important Value even if I don’t feel it a lot myself.

So I am going to give you the exact same Existence Proof : I notice that if you give me everything else, Hedonistic Happiness, Justice, Health, etc. and take away Agency (which means having things to do that go beyond “having a hobby”), the value to me of my existence is not 0, but not that far above 0. If I live in such a society and we need to sacrifice some individuals, I will happily step in, “nothing of value was lost” style. If I live in such a society and Omega appears and announce “Sorry, Vacuum Decay Bubble incoming, everyone is going to disappear in exactly 3 minutes”, I will sure feel bad for “everyone” who is apparently pretty happy, but I will also think “well, I was already pretty much dead inside anyway”.

Please define it in a succinct, relevant, and unambiguous way

I’m afraid you will have to pick only two adjectives, should you want to ask this to someone smarter and more articulated than me but with the same views. Alas, you’re stuck with me, so we’ll have to pick one, so let’s pick “relevant”.

It will be also very hand-wavy. Despite all that, it’s still the best I can do. Sorry.

Take the Utility Function of someone. We can decide to split it in roughly three parts :

$U_{i} = D_{i} + Σ P_{i j} U_{j}$

$D_{i}$ is “direct” utility. I’m hungry, I want ice scream. Would love to go to that concert. Just got an idea, can I build it ?

$P_{i} j$ is “how much i care about j”

$U_{j}$ is the Utility Function of j.

This is a roughly speaking a very rough first approximation of “how to model egoism and altruism”. Yes, I’m fully conscious that this is far from capturing most of interpersonal relationships & utility. I still think it’s relevant to point at the Big Picture, namely : if A only cares about B (all other P_Ax are 0) and B only cares about A (same), then if D_A = 0 and D_B = 0 : there is not utility left. Or : a world of Pure Altruists who only cares about others is a worthless world.

Which is not the same as to say that Altruism is Worthless. As long as you have at least some D_something, Altruism can create arbitrarily large values of Utility, as a multiplying force. It’s why it’s such a potent Human Value.

Now let’s go even further in handwaviness : this generalizes : many Values are similarly powerless at creating Utility from Nothing. They just act as Force Multipliers.

I call “Meaningful Values” the one that can create Utility by themselves, without having to rely on others to be present in the first place. Which does not means that the others (let’s call them Amplifying Values) are meaningless, to be clear. They just happen to become meaningless if you have 0 Meaningful Value hanging around.

In short : I’m very afraid that we’re putting a lot of load-bearing “we’ll be fine, there’s still value” on mostly Amplifying Values.

When I said above “I notice that in a world where I don’t have meaningful Agency, I don’t put much value on my own existence”, I do not say “I do not like hobbies”. I happen to like hobbies, in this world ! I’m also self-reflective enough that what I like in hobbies in the opportunity to grow, and the value of growing resolves in being better at Agency, which is a way better candidate as a “Meaningful Value” and a “Terminal Value”. Hence, if you throw me in the UBI Paradise (let’s drop “Christian”, it seems to annoy everyone), the value of hobbies go to zero, too, and I become a shell of my current self, despite my current self saying “hobbies are cool”.

The democratic consensus also won’t allow a Butlerian Jihad, and I don’t think you’re claiming that it will.

Okay, there’s two things to unpack here.

First, I believe with those answers that I went too far in the Editoralizing vs Being Precise tradeoff with the term “Butlerian Jihad”, without even explaining what I mean. I will half-apologize for that, only half because I didn’t intend the “Butlerian Jihad” to actually be the central point ; the central point is about how we’re not ready to tackle the problem of Human Values but that current AI timelines force us to. You can see it’s pretty dumb of me to put a non-central point as the title. I have no defense for that.

Second : By Butlerian Jihad, I do not mean “no AI, ever, forever”, I mostly mean “a very long pause, at capabilities levels far bellow AGI. I feel already bad about GPT 5 even if it does no go human-level. I’m not even sure I’m entirely fine with GPT 4″

Contra you and Zvi, I think that if GPT 5 leads to 80% jobs automation, the democratic consensus will be pretty much the Dune version of the Butlerian Jihad. No AI forever and the guillotine for those who try. Which I would agree with you and Zvi and probably everyone else on lesswrong is not a good outcome. I don’t think it’s a very interesting point of discussion either, so let’s drop it ?

I’m actually not sure what you’re arguing for or against in this whole section.

I’m essentially arguing against taking Human Values as some Abstract Adjectives like Happiness and Health and Equality and Justice and forgetting about… you know… the humans in the process.

What that has to do with justice destroying the world, I have absolutely no clue

It’s about Abstract Justice destroying humans (values) if you go too far in your Love for Justice and forget that they’re the reason we want Justice in the first place.

Some values have always won, and some values have always lost, and that will not change

Yes, which already rises an important point :

What value do we (who is “we” ?) place on Diversity ? On values which we do not personally have but that seems to have a good place in our “Best Model of Human Values” ? What about values which do not really fit in our “Best Model of Human Values”, but turns out that some other humans on the planet happen to put in their model of the “Best Model of Human Values”. What if that other human is your sworn enemy ?

It was there that I was trying to point with my “exercise for the reader”.

I think you’re trying to take the view that any major change in the “human condition”, or in what’s “human”, is equivalent to the destruction of the world, no matter what benefits it may have. This is obviously wrong

Oh, I will not dispute that it is wrong. Better the super-happies that the literal Void. Just not much better.

You seem a bit bitter about my “I won’t expand on that”, “too long post”, and so on. I’m sorry, but I spent two days on the post, already 2 hours on one reply. I’m not a good or prolific writer. I have to pick what I spend my energy on.

So you’re siding with the guy who killed 15 billion non-consenting people because he personally couldn’t handle the idea of giving up suffering?

I initially didn’t want to reply to that. I don’t want to fight you. I just want to reply as an illustration of how fast things can go difficult and conflictual. It doesn’t take much :

So you’re siding with the guy who is going to forcibly wirehead all sentient life in the universe, just because he can’t handle that somewhere, someone is using his agency wrong and suffering as a result ?

That being said, what now ? Should we fight each other to death for the control of the AGI, to decide whether the universe will have Agency and Suffering, or no Agency and no Suffering ?

Human Values have been changing, for individuals and in the “average”, for as long as there’ve been humans, including being discarded consciously or unconsciously. Mostly in a pretty aimless, drifting way.

Lot of consciously too, but yes.

neither AI nor anything else will fundamentally change it.

Hard disagree on that (wait, is this the first real disagreement we have ?). We can have the supperhappies if we want to (or for that matter, the baby-eaters). We couldn’t before. The supperhappies do represent a fundamental change.

Before, we still had not much choice over diversity. Many people fought countless wars to reduce diversity in humans values, without much overall success (some, yes, but not much in the grand picture of things). In the AGI age nothing forces the one controlling the AGI to care much for diversity. It will have to be a deliberate choice. And do you notice all the forces and values already arraying against diversity ? It does not bode well for those who value at least some diversity.

I haven’t actually heard many people suggesting that.

That’s the “best guess of what we will do with AGI” from those building AGI.