avturchin

Karma: 4,990

avturchin 10 Jun 2026 15:42 UTC
2 points
−12
on: The Three Filters: Why Almost Every Plan to Survive ASI Fails Miserably
alignment
Because both essentially are the same thing: alignment is about AI doing what I want—and capabilities are about Ai doing what I want?

The desire to end the world

avturchin8 Jun 2026 12:29 UTC

7 points

0 comments7 min readLW link

avturchin 28 May 2026 12:57 UTC
2 points
0
in reply to: Semi-Pseudonymous’s comment on: Brain transfers might be the easiest path to life extension
I think there will be some reduction, but bit will not change the outcome significantly.
1 First, there will be opposite process—old brain will affect young body by different hormones and different behavioral habbits (like no running). Also, old brain may have some dormant cancer cells from previous body which will propagate through young body.
2 Beneficial impact of young body will not affect all types of brain aging. May be 50 per cent slower aging which gives only one doubling or 7 years, but they will be mostly compensated by damage described in points 1 and 3.
3 The transfer procedure, if not required nanotechnology, will include a period of some brain ishemia or cooling, which will damage brain to some extent. And if we have advance nanotech, we don’t need new biological body at all—we can grow new one through old one.

BTW, we are starting soon an analog of LessWrong about life extension, may be you will be interested to share your post there: https://antimortality.org/.

avturchin 27 May 2026 11:44 UTC
2 points
0
in reply to: TAG’s comment on: A relatively brief explanation of Boltzmann Brains
To match our coherent experiences, the successor BB ’s need to have coherent memories, which only a few would.

Yes, it is similar to MWI—there infinitely many possible futures for me-now and most are crazy and discoherent. But we put Born rule above this distribution and select most coherent ones.

I think that there is some selection for stabilization mechanism in such observer-moment chains. Mueller suggested that it is similarity - or low Kolmogorov complexity—which creates penalty for less similr to now observer-moments.

avturchin 27 May 2026 8:20 UTC
2 points
0
in reply to: Semi-Pseudonymous’s comment on: Brain transfers might be the easiest path to life extension
Gompertz law is really brutal. Let’s assume that brain diseases causes 25 per cent of all death. At age 70 doubling of mortality is 7 years, so brain transfer reduces two doublings and gives +14 yeas to life expectancy.
However, around 105 years old doubling time is 1 year, so brain transfer will give only 2 years of life expectancy. Even if we reduce brain disease share in aging-related deaths to 1 per cent, the brain-transfer will give only around +7-8 years for centenaries, so 110 years is absolute limit for such technology. We have to address Gompertz law instead.

avturchin 26 May 2026 11:06 UTC
2 points
4
on: Brain transfers might be the easiest path to life extension
Doesn’t work because aging affects the brain as well and many deadly diseases originate in brain: insults, Alzheimer and brain cancers. Add risks of the transfer itself and the uncertainty when it should be done: too early for healthy aged person and and too late for one with cancer of any type (risks of metastases). All together it gives less than 10 years increase of of medium life expectancy.

avturchin 23 May 2026 21:13 UTC
1 point
0
in reply to: uncial’s comment on: Possible red is red
In your words, there is a energy, concept of energy, and there is arbitrary representation of the concept of energy by letter E. The same way for colors: there is a real wavelength, the functional reaction to color in the brain and its arbitrary representation by redness inside consciousness.
After that follows rather technical moment which needs to be consider before we can discuss the question of existence of possible qualia. This technical moment is connected with the idea that unobserved qualia still have definite qualitative state. This seems to be so obvious that is often ignored, but its is an assumption which need to be considered.
Chalmers suggested that dancing qualia thought experiment—that a person can have different representations of colors in different moments of time but will not be able to observe that as functional part will remain the same. I mentioned here that such experiment assumes non-observed- qualia have definite qualitative state.

Blue-green abundance is functional part of qualia.

Exactly opposite is true for qualia—it will extraordinary claim to say that we know something about qualitative aspect of bird’s qualia.

Sorry for typos, I am not native speaker and didn’t use AI for editing here.

avturchin 23 May 2026 16:49 UTC
4 points
3
on: How should we update on AI-enabled coups post-Mythos?
Maybe we should expect first AI-enabled takeover of rogue countries.

avturchin 23 May 2026 10:51 UTC
1 point
0
in reply to: Dagon’s comment on: Possible red is red
My point about apples was an illustration that’s some properties remains in possible things, this includes both shape and numbers, but not the first step of generalisation.

We are far from enumerating the qualitative aspect of qualia and may be it is even not possible in a provable way. Maybe because enumerating assumes that we can predict qualitative property of qualia which contradict idea that qualia depends only on itself.

avturchin 23 May 2026 10:20 UTC
1 point
0
in reply to: uncial’s comment on: Possible red is red
Red has functional part and qualitative part, reddnes. I tried to explain this in my previous post about qualia as internal variables. For example E is energy in E=mc2. Its functional role is defined by the equation, but the variable itself can be denoted by a different letter, like W. Letters can also exist not related to the equations, that is free-floating letters.

Only if we assume that unobserved qualia do not reddness, when speaking about possible qualia’s color become meaningless, as by our assumption unobserved qualia don’t have redness and possible qualia is unobserved.

When I say that possible 25 has the same properties as real 25 I mean 25 apples as real 25 - and 25 as just a number as possible 25. Both 25 and 25 apples can be represented as 5x5. This property doesn’t depend on modal status of 25.

The distinction between open and closed qualia gives us an interesting hint why dancing qualia thought experiment is wrong (though how experiment can be wrong? only its conclusions). Chalmers seems to ignore this and implicitly assumes that there is no difference between open and closed qualia. As soon as we add this distinction, the experiment boils down to the question—are closed qualia the same as open ones.

It is rather objective fact that we can’t compare human’s and bird’s color qualia as birds have 4 primary colors. Birds colors are permanently closed to us. It is not just language game like in God’s definition.

avturchin 22 May 2026 14:10 UTC
1 point
0
in reply to: Dagon’s comment on: Possible red is red
Here we arrive to the interesting question—which properties remain in possible apples and in qualia?
For example, property of edibility will not remain in a possible apple, but a property being sphere-like remains in a possible apple.
More generally, the properties related to the outside world disappear, but internal properties remain. If qualia are property related to the existence, they disappear in possible objects.

avturchin 22 May 2026 10:10 UTC
3 points
−2
in reply to: Viktor Moskvoretskii’s comment on: Synthetic Persona Pretraining: Alignment from Token Zero
I see a possible source of alignment error when we specify preferences manually. If we take a real person—we get what such person really want, but it is also kinky, unstable etc. If we generate a set of nice sounding human values they may be good but unrelated to real humans and thus misaligned.

There are several midfiles available. There is a group of people who experimenting with creating open-sourced mindfiles of themselves, first of all, me and RomanS (and recently created mind model of Igor Kiriluk). There also several people who are very well documented: Leo Tolstoy, Anna Frank. The problem here is that they lived before AI and have a set of different world-views, especially Tolstoy.
A good thing is that Tolstoy writings and dairies are deep into training set of most LLMs and can be called with short prompt.

Possible red is red

avturchin21 May 2026 17:30 UTC

−1 points

9 comments4 min readLW link

avturchin 21 May 2026 10:27 UTC
2 points
0
in reply to: Julian Minder’s comment on: Synthetic Persona Pretraining: Alignment from Token Zero
A real person can annotate maybe 1000 texts before becoming exhausted. A possible solution is to use a mind-model of a real person which generates such annotations.

I created my mind model which is 1M tokens file and it is open sourced. You can experiment with it. https://github.com/avturchin/minduploading/tree/main/ONE-FILE-SIDELOAD

avturchin 20 May 2026 18:47 UTC
4 points
1
on: Synthetic Persona Pretraining: Alignment from Token Zero
Can we use not a synthetic persona, but a mind model of real person, to get more realistic human pretraining?

avturchin 18 May 2026 10:24 UTC
2 points
0
in reply to: jmh’s comment on: A relatively brief explanation of Boltzmann Brains
There is a great article “Law without law: from observer states to physics via algorithmic information theory” https://arxiv.org/abs/1712.01826 which explains how observer-moments can form chains.

The idea of Boltzmann immortality was discussed here: https://www.researchgate.net/publication/308958767_Boltzmannian_Immortality

Link was on a picture I generated but failed to embed. Here is another link on the same picture: https://ibb.co/fGCz0q1b

avturchin 17 May 2026 13:06 UTC
2 points
0
in reply to: Oliver Sourbut’s comment on: A relatively brief explanation of Boltzmann Brains
I think that in normal BBs they have equal chances.
However, if we postulate that chains of similar BBs are possible, they will evolve into more stable states.

relevant image:
https://chatgpt.com/backend-api/estuary/content?id=file_00000000d35c71fdae277cb42b44cdcf&ts=494172&p=fs&cid=1&sig=ae767f2f792c2f370e84300dfdb5fb361b83af1b5266929320a3a97c11e49b78&v=0

avturchin 17 May 2026 10:00 UTC
11 points
2
on: A relatively brief explanation of Boltzmann Brains
For any BB there is another BB which constitutes what I will experience in the next moment t+1.
So BBs naturally form flux universe from dust theory and everything gets back to normal.
This also means that BB can’t “die” or instantly dissolve – there is a form of big-world-immortality between BBs, especially if some longer-living similar brains exist somewhere.

Also, we can’t judge from inside – am I living in extreme chaos or not, because thinking of BBs is not logical and they can have completely random thoughts. So maybe I observe complete chaos and still think that it os ordered.

avturchin 11 May 2026 8:49 UTC
2 points
0
in reply to: Logan Zoellner’s comment on: The Anti-Singularity
And they will be Moloch eating human resources.

avturchin 11 May 2026 6:12 UTC
2 points
0
on: The Anti-Singularity
Can we have a single high-level AI-bureaucracy which regulates creation and behavior of specialized task-specific AIs? From external view it will look like as single AGI with some set of goals.

avturchin

The de­sire to end the world

Pos­si­ble red is red

The desire to end the world

Possible red is red