jchan

Karma: 671

jchan 13 Oct 2025 17:01 UTC
3 points
−1
in reply to: MichaelDickens’s comment on: Sublinear Utility in Population and other Uncommon Utilitarianism

so your qualia weren’t that valuable

The OP isn’t making any claim like this; the question isn’t whether any particular experience has value in-and-of-itself, but is only making a claim about the correct way to evaluate the total utility in a world with multiple experience.

By analogy, consider special relativity: If a train is moving at 0.75c relative to the ground, and a passenger on the train throws a ball forward at 0.5c, then that means the ball is moving at 0.91c relative to the ground. But there is no reference frame in which the ball is “really” moving at 0.16c.

Or, more pertinently, suppose we have two identical simulations playing out at the same time. Each one contributes zero marginal utility to a world in which the other one exists (and might be told this by Omega), but that doesn’t mean that the two of them together have zero utility.

jchan 13 Oct 2025 16:46 UTC
3 points
0
on: Sublinear Utility in Population and other Uncommon Utilitarianism
I’ve had exactly the same thought before but never got around to writing it up. Thanks for doing it so I don’t have to :-)
There are only so many possible human shaped computations that are valuable to me
I would surmise that value-space is not so much “finite in size” but rather that it fades off into the distance in such a way that it has a finite sum over the infinite space. This is because other minds are valuable to me insofar as they can do superrationality/FDT/etc. with me. In fact, this is the same fading-out function as in the “perturb the simulation” scenario; i.e.:
- $V_{A} (A B)$ := The value that $A$ places on a world where $A$ and $B$ both exist
- $V_{A} (A ¯ B)$ := The value that $A$ places on a world where $A$ exists but $B$ doesn’t
- $V_{A} (¯ A B)$ := The value that $A$ places on a world where $A$ doesn’t exist but $B$ does
- Claim: $V_{A} (A B) - V_{A} (A ¯ B) = V_{A} (A ¯ B) - V_{A} (¯ A B)$
However, the main problem with this perspective is what to do with quantum many-worlds. Does this imply that “quantum suicide” is rational, e.g. that you should buy a lottery ticket and set up a machine that kills you if you don’t win? This is bullet I don’t want to bite (so to speak...)

jchan 29 Sep 2025 22:09 UTC
1 point
−2
in reply to: AnthonyC’s comment on: How does the current AI paradigm give rise to the “superagency” that IABIED is concerned with?
Re 1a: Intuitively what I mean by “lots of data” is something comparable in size to what ChatGPT was trained on (e.g. the Common Crawl, in the roughly 1 petabyte range); or rather, not just comparable in disk-space-usage, but in the number of distinct events to which reinforcement learning is applied. So when ChatGPT is being trained, each token (of which there are a ~quadrillion) is a chance to test the model’s predictions and adjust the model accordingly. (Incidentally, the fact that humans are able to learn language with far less data input than this suggests that there’s something fundamentally different in the way LLMs vs. humans work.)

Therefore, for a similarly-architected AI that generates action plans (rather than text tokens), we’d expect it to require a training set with a ~quadrillion different historical cases. Now I’m pretty sure this already exceeds the amount of “stuff happening” that has ever been documented in all of history.

I would change my opinion on this if it turns out that AI advancement is making it possible to achieve the same predictive accuracy / generative quality with ever less training data, in a way that doesn’t seem to be levelling off soon. (Has work been done on this?)

Re 2a: Accordingly, the reference class for the “experiments” that need to be performed here is not like “growing cells in a petri dish overnight”, but rather more like “run a company according to this business plan for 3 months and see how much money you make.” And at the end you’ll get one data point—just 999,999,999,999,999 to go...

Re 2b:

What do you see as the upper bound for what can, in principle, be done with a plan that an army of IQ-180 humans (aka no better qualitative thinking than what the smartest humans can do, so that this is a strict lower bound on ASI capabilities) came up with over subjective millennia with access to all recorded information that currently exists in the world?

I’ll grant that such an army could do some pretty impressive stuff. But this is already presupposing the existence of the superintelligence whose feasibility we are trying to explain.

Re 3c/d:

I haven’t looked into this or tried it myself. (I mostly use LLMs for informational purpose, not for planning actions). Do you have any examples handy of AI being successful at real-world goals?

(I may add some thoughts on your other points later, but I didn’t want to hold up my reply on that account.)

Stepping back, I should reiterate that I’m talking about “the current AI paradigm”, i.e. “deep neural networks + big data + gradient descent”, and not the capabilities of any hypothetical superintelligent AI that may exist in the future. Maybe this is conceding too much, inasmuch as addressing just one specific kind of architecture doesn’t do much to alleviate fear of doom by other means. But IABIED leans heavily on this paradigm in making its case for concern:
- the claim that AIs are “grown, not crafted”
- the claim that AIs will develop desires (or desire-like behavior) via gradient descent
- the claim that the risk is imminent because superintelligence is the inevitable endpoint of the work that AI researchers are currently doing, and because no new fundamental breakthroughs stand in the way of that outcome.

jchan 28 Jul 2025 21:47 UTC
1 point
1
in reply to: AnnaJo’s comment on: What are non-obvious class markers?

skiing

It seems like skiing is a “hereditary” class marker because it’s hard to learn how to do it as an adult, and you’re probably not going to take your kids skiing unless you yourself were taught as a kid, etc.

jchan 21 Jun 2025 15:57 UTC
4 points
2
in reply to: Knight Lee’s comment on: The Boat Theft Theory of Consciousness

trying to imagine being something with half as much consciousness

Isn’t this what we experience every day when we go to sleep or wake up? We know it must be a gradual transition, not a sudden on/off switch, because sleep is not experienced as a mere time-skip—when you wake up, you are aware that you were recently asleep, and not confused how it’s suddenly the next day. (Or at least, I don’t get the time-skip experience unless I’m very tired.)

(When I had my wisdom teeth extracted under laughing gas, it really did feel like all-or-nothing, because once I reawoke I asked if they were going to get started with the surgery soon, and I had to be told “Actually it’s finished already”. This is not how I normally experience waking up every morning.)

jchan 17 Jun 2025 13:26 UTC
9 points
8
in reply to: sunwillrise’s comment on: sunwillrise’s Shortform
- I think this is mostly just the macro-trend of the internet shifting away from open forums and blogs and towards the “cozy web” of private groupchats etc., not anything specific about LessWrong. If anything, LessWrong seems to be bucking the trend here, since it remains much more active than most other sites that had their heyday in the late 00s.
- I don’t have any dog in the Achmiz/Worley debate, but I’m having trouble getting in the headspace of someone who is driven away from posting here because of one specific commenter.
  - First of all, I don’t think anyone is ever under any obligation to reply to commenters at all—simply dropping out of a conversation thread doesn’t feel rude/confrontational in the way it would be to say IRL “I’m done talking to you now.”
  - Second, I would find it far more demotivating to just get zero engagement on my posts—if I didn’t think anybody was reading, it’s hard to justify the time and effort of posting. But otherwise, even if some commenters disagree with me, my post is still part of the discourse, which makes it worthwhile.

jchan 12 Jun 2025 13:42 UTC
5 points
1
in reply to: Viliam’s comment on: Religion for Rationalists
I think this approach wouldn’t work for rationalists, for two reasons:
- The rationality community is based around disputation, not canonicalization, of texts. That is, the litmus test for being a rationalist is not “Do you agree with this list of propositions?” (I have tried many times to draw up such a list, but this always just leads to even more debate), but rather “Are you familiar with this body of literature and do you know how to respond to it?” The kind of person who goes to LW meetups isn’t going to enjoy simply being “talked at” and told what to believe—they want to be down in the arena, getting their hands dirty.
- Your “recommended template” is essentially individualistic—participants come with their hopes and desires already in-hand, and the only question is “How can I use this community to help me achieve my goals?” Just as a gut feeling I don’t think this is going to work well in building a community or meaningful relationships (seeing others not merely as means, but as ends in themselves—or something like that). Instead, there needs to be some shared purpose for which involvement in the community is essential and not just an afterthought. Now, this isn’t easy. “Solving AI alignment” might be a tall order. But I think the rationality community is doing a passable job at one thing at least—creating a culture of high epistemic standards that will be essential (for both ourselves and the wider world) in navigating the unprecedented challenges our civilization faces.

jchan 5 Jun 2025 15:00 UTC
4 points
2
in reply to: HoVY’s comment on: How the veil of ignorance grounds sentientism
Can’t speak for Said Achmiz, but I guess for me the main stumbling block is the unreality of the hypothetical, which you acknowledge in the section “This is not a literal description of reality” but don’t go into further. How is it possible for me to imagine what “I” would want in a world where by construction “I” don’t exist? Created Already in Motion and No Universally Compelling Arguments are gesturing at a similar problem, that there is no “ideal mind of perfect emptiness” whose reasoning can be separated from its contingent properties. Now, I don’t go that far—I’ll grant at least that logic and mathematics are universally true even if some particular person doesn’t accept them. But the veil-of-ignorance scenario is specifically inquiring into subjectivity (preferences and values), and so it doesn’t seem coherent to do so while at the same time imagining a world without the contingent properties that constitute that subjectivity.

jchan 29 May 2025 16:55 UTC
1 point
−2
in reply to: Lorec’s comment on: What should I read to understand ancestral human society?
I think ancient DNA analysis is the space to watch here. We’ve all heard about Neanderthal intermixing by now, but it’s only recently become possible to determine e.g. that two skeletons found in the same grave were 2nd cousins on their father’s side, or whatever. It seems like this can tell us a lot about social behavior that would otherwise be obscure.

jchan 28 May 2025 19:57 UTC
14 points
2
in reply to: johnswentworth’s comment on: johnswentworth’s Shortform
It took me years of going to bars and clubs and thinking the same thoughts:
- Wow this music is loud
- I can barely hear myself talk, let alone anyone else
- We should all learn sign language so we don’t have to shout at the top of our lungs all the time
before I finally realized—the whole draw of places like this is specifically that you don’t talk.

jchan 24 May 2025 1:00 UTC
14 points
2
in reply to: Tim H’s comment on: Learning (more) from horse employment history
- Humans have a considerable up-front cost also—it’s called birth and childrearing!
- Maybe absolute horse productivity actually has declined, at least for non-farm work. All the roads have been rebuilt for cars, there are no places to tie up my horse for a quick trip to the supermarket, etc. (For that matter, is riding horses on city streets even legal? Regardless, it’s certainly less convenient than it would’ve been in 1900.)
  - AI analogy: If a job consists mainly of communicating with other employees, it’ll become harder for a human to maintain the same productivity when 99% of those other employees have been replaced with AIs, even if the human is just as intelligent as before.
- If there is less demand for hay from horses, that will not increase the supply much because cattle, sheep, etc. will pick up the slack.
- Other funny thought: From a horse’s perspective, the Amish are an aligned superintelligence.

jchan 23 May 2025 13:20 UTC
6 points
0
on: Learning (more) from horse employment history

In the end, despite cheaper feed, the daily cost of horse upkeep (the horse’s subsistence wage, if you will) was higher than the horse’s productivity in its transport and agricultural roles.

Presumably the absolute productivity of a horse (the amount of land it can plow or stuff it can haul) has not changed. So this only makes sense if the market value of the horse’s labor has declined even faster than the price of feed. Is that the case?

jchan 12 Feb 2025 20:09 UTC
1 point
0
in reply to: Lucius Bushnaq’s comment on: Notes on Occam via Solomonoff vs. hierarchical Bayes

rather than, say, assigning equal probability to all strings of bits we might observe

If the space of possibilities is not arbitrarily capped at a certain length, then such a distribution would have to favor shorter strings over longer ones in much the same way as the Solomonoff prior over programs (because if it doesn’t, then its sum will diverge, etc.). But then this yields a prior that is constantly predicting that the universe will end at every moment, and is continually surprised when it keeps on existing. I’m not sure if this is logically inconsistent, but at least it seems useless for any practical purpose.

jchan 28 Jan 2025 3:01 UTC
2 points
1
on: Supposing that the “Dead Internet Theory” is true or largely true, how can we act on that information?
- For certain kinds of questions (e.g. “I need a new car; what should I get?”), it’s better to ask a bunch of random people than to turn to the internet for advice.
- In order to be well-informed, you’ll need to go out and meet people IRL who are connected (at least indirectly) to the thing you want information about.
In the following, I will use the term “my DIT” to refer to the claim that:

In some specific non-trivial contexts, on average more than half of the participants in online debate who pose as distinct human beings are actually bots.

I agree with this version, and I was surprised to see that the Wikipedia definition also includes the bit about it being a deliberate conspiracy, which seems like a strawman, since I have always understood the “Dead Internet Theory” to include only the first part. There’s a lot of stuff on the internet that’s very obviously AI-generated, and so it’s not too far a stretch to suppose that there’s also a lot of synthetic content that hides it better. But this can be explained far more simply than by some vast conspiracy—as SEO, marketing, and astroturfing campaigns.

If Dead Internet Theory is correct, when you see something online, the question you should ask yourself is not “Is this true?” but “Why am I seeing this?” This was always the case to some extent of any algorithmically-curated feed (where the algorithm is anything more complex than “show me all of the posts in reverse chronological order”), but is even more significant when the content itself is algorithmically generated.

If I’m searching online for information about e.g. what new car I should buy, there’s a very strong incentive for all the algorithms involved (both the search engine itself, and the algorithm that spits out the list of recommended car models) to sell their recommendations to the highest bidder and churn out ex post facto justifications explaining why their car is really the best. These algorithms are almost totally uncorrelated with the underlying fact about which car I’d actually want, so I know to consider it of very little value. On the other hand, I would argue, asking a bunch of random acquaintances for car recommendations is much more useful because, although they might not be experts, they were at least not specifically selected in order to deceive me. Even if I ask a friend and they say “Well, I haven’t bought a new car in years, but I heard my coworker’s cousin bought the XYZ and never stops complaining about it”, then this is much more useful information than anything I could find online, because it’s much less likely that my friend’s coworker’s cousin was specifically being paid to say that.

More broadly, on many questions of public concern there may be parties with a strong interest in using bots to create the impression of a broad consensus one way or another. This means that you have no choice but to go out into the real world and ask people, and hope ideally that they’re not simply repeating what they read online, but have some non-AI-mediated connection to the thing.

jchan 16 Nov 2024 23:50 UTC
2 points
0
on: If I care about measure, choices have additional burden (+AI generated LW-comments)

However, in Many-Worlds Interpretation (MWI), I split my measure between multiple variants, which will be functionally different enough to regard my future selves as different minds. Thus, the act of choice itself lessens my measure by a factor of approximately 10. If I care about this, I’m caring about something unobservable.

If we’re going to make sense of living in a branching multiverse, then we’ll need to adopt a more fluid concept of personal identity.

Scenario: I take a sleeping pill that will make me fall asleep in 30 minutes. However, the person who wakes up in my bed the next morning will have no memory of that 30-minute period; his last memory will be of taking the pill.

If I imagine myself experiencing that 30-minute interval, intuitively it doesn’t at all feel like “I have less than 30 minutes to live.” Instead, it feels like I’d be pretty much indifferent to being in this situation—maybe the person who wakes up tomorrow is not “me” in the artificial sense of having a forward-looking continuity of consciousness with my current self, but that’s not really what I care about anyway. He is similar enough to current-me that I value his existence and well-being to nearly the same degree as I do my own; in other words, he “is me” for all practical purposes.

The same is true of the versions of me in nearby world branches. I can no longer observe or influence them, but they still “matter” to me. Of course, the degree of self-identification will decrease over time as they diverge, but then again, so does my degree of identification with the “me” many decades in the future, even assuming a single timeline.

jchan 19 Sep 2024 1:48 UTC
1 point
0
on: Inquisitive vs. adversarial rationality
This can be a great time-saver because it relies on each party to present the best possible case for their side. This means I don’t have to do any evidence-gathering myself; I just need to evaluate the arguments presented, with that heuristic in mind. For example, if the pro-X side cites a bunch of sources in favor of X, but I look into them and find them unconvincing, then this is pretty good evidence against X, and I don’t have to go combing through all the other sources myself. The mere existence of bad arguments for X is not in itself evidence against X, but the fact that they’re presented as the best possible arguments is.

Of course the problem is, outside of a legal proceeding, parties rarely have that strong an incentive to dig up the best possible arguments. Their time is limited as well, and they don’t really suffer much consequence from failing to convince you. Also, the discussion medium might structurally impede the best arguments from being given (e.g. replies in a Twitter thread need to be posted quickly or else nobody will see them). Or worse yet, a skilled propaganda campaign can flood the zone with bad pro-X arguments from personages who appear to be pro-X but are secretly against it, knowing that the audience is going to be evaluating these arguments using the adversarial heuristic.

jchan 26 Apr 2024 15:47 UTC
12 points
9
on: social lemon markets

In my experience, Americans are actually eager to talk to strangers and make friends with them if and only if they have some good reason to be where they are and talk to those people besides making friends with people.

A corollary of this is that if anyone at an [X] gathering is asked “So, what got you into [X]?” and answers “I heard there’s a great community around [X]”, then that person needs to be given the cold shoulder and made to feel unwelcome, because otherwise the bubble of deniability is pierced and the lemon spiral will set in, ruining it for everyone else.

However, this is pretty harsh, and I’m not confident enough in this chain of reasoning to actually “gatekeep” people like this in practice. Does this ring true to you?

jchan 22 Mar 2024 18:04 UTC
6 points
0
on: On green
I highly recommend Val Plumwood’s essay Tasteless: towards a food-based approach to death for a “green-according-to-green” perspective.

Plumwood would turn the “deep atheism” framing on its head, by saying in effect “No, you (the rationalist) are the real theist”. The idea is that even if you’ve rejected Cartesian/Platonic dualism in metaphysics, you might still cling for historical reasons to a metaethical-dualist view that a “real monist” would reject, i.e. the dualism between the evaluator and the evaluated, or between the subject and object of moral values. Plumwood (I think) would say that even the “yin” (acceptance of nature) framing is missing the mark, because it still assumes a distinction between the one doing the accepting and the nature being accepted, positing that they simply happen to be aligned through some fortunate circumstance, rather than being one and the same thing.

jchan 4 Feb 2024 14:25 UTC
1 point
0
in reply to: Dacyn’s comment on: Ten Modes of Culture War Discourse
It’s a question of whether drawing a boundary on the “aligned vs. unaligned” continuum produces an empirically-valid category; and to this end, I think we need to restrict the scope to the issues actually being discussed by the parties, or else every case will land on the “unaligned” side. Here, both parties agree on where they stand vis-a-vis C and D, and so would be “Antagonistic” in any discussion of those options, but since nobody is proposing them, the conversation they actually have shouldn’t be characterized as such.

jchan 1 Feb 2024 15:54 UTC
1 point
0
in reply to: Three-Monkey Mind’s comment on: Ten Modes of Culture War Discourse
On the contrary, I’d say internet forum debating is a central example of what I’m talking about.