avturchin

Karma: 5,090

avturchin 6 Aug 2026 12:20 UTC
2 points
0
in reply to: JeffJo’s comment on: Sleeping Beauty as a Mind Killer
Interestingly, if she never exists on Tuesday-Heads (and never reappears on Wednesday), the situation starts to look like quantum immortality for Monday-Heads. Their only “continuation” now is Tuesday-Tails.
Quantum immortality analogy also is applicable to the lines where she never appears at all like in 6-sides dice experiment which you described above.

avturchin 5 Aug 2026 22:06 UTC
2 points
−2
in reply to: Stuart_Armstrong’s comment on: III. Anthropic reasoning has issues with infinite worlds; D-SIA can fix this
We can prove that everything that exist is infinite: if it consists of N elements and no more, there should be some process which limits its size as N and no more. However, such process will be N+1 element.
Or in other words, if universe appeared from non-existence once, infinitely many universes can pop-up from existence independently, as there is nothing which can stop such pop-up process.

However, the list of observer’s states which can appear is finite and is limited by the size of maximum possible causally connected universe. I dissed different ways of counting here.

avturchin 4 Aug 2026 22:20 UTC
2 points
0
in reply to: Stuart_Armstrong’s comment on: Sleeping Beauty as a Mind Killer
After some conversation with Fable, it said to me that uniform prior over probabilities in Laplace, and being in the middle of actually existing observers in Gott—are basically the same idea. So the mechanism is that randomness put me in the middle. But this requires the reality of future.
Carter in his book about extinction spent a lot of pages to show that this requires that unitary future do actually exist or at least deterministic, so the sampling process is happening from the future too. This contradicts my MWI intuitions (But may be can solved via distribution analyses)

I think the mechanism is different, and is based on idea that there are many different civilizations and we measure their average age (universal doomsday argument). If we лтцш average age, we can also extrapolate it on us. For example, if I ask a random person on a street what is his age and he said that his age 47, I can conclude that human average age is several decades years—and thus my average age is the same. This trick escapes direct “reading” of the information from the future.

avturchin 4 Aug 2026 16:02 UTC
2 points
0
in reply to: Stuart_Armstrong’s comment on: Sleeping Beauty as a Mind Killer
if we use Goth equation to the correct class of observers, it is not update of what we already know about the whole human history. The correct class of observers is not humans, but only scientists who think about doomsday argument and the birth date is the date when that person first started thinking about Doomsday argument as rank order between all such scientists. All such scientists live is approximately same period of human history (end of 20 century—beginning of 21) and thus know the same things about risks, also risk are much more uniform for such shorted period.

Putting numbers in this gives doom in the middle of 21 century with obvious caveats.

avturchin 4 Aug 2026 15:38 UTC
4 points
2
on: III. Anthropic reasoning has issues with infinite worlds; D-SIA can fix this
We can say that SIA favors the theory in which maximal (before it becomes infinite) finite number of observers exists. For example: we can count all possible universes of the size of visible universe, and get very large but finite number of all observers in the multiverse. Adding more multiverses will not change anything, as all possible observers here already counted and their distribution inside multiverse is known.

What is also interesting here is that after very large but finite number of observers is reached, my own existence stops being surprising. This cuts one of the version of SIA, which argues that me-real is drawn from the pool of possible minds. If such pool is finite and the number of attempts to draw is very large, there is no surprise that I exist at all.

avturchin 4 Aug 2026 13:54 UTC
2 points
0
in reply to: Stuart_Armstrong’s comment on: Sleeping Beauty as a Mind Killer
I don’t think it is wrong. Take for example, Laplace’s Sunrise problem. It produces almost exactly the same prediction as Gott’s argument (difference is “+1” which is explained by the fact that sunrises are numbered items and time is continuous). However, Laplace’s solution appeared using different math (Integration over all possible probabilities, as I remember) and is applicable to continuous processes.

avturchin 4 Aug 2026 13:50 UTC
4 points
−2
in reply to: Stuart_Armstrong’s comment on: Anthropic reasoning without duplicates is just standard Bayesian updating
I think that my line of reasoning here is close to yours, but I will read your next posts in the sequence after finish this comment.

I think that SIA proves infinite universe and after that stops working. After that we can compare different regions of the multiverse but have to take into account their size and concentration of observers in them. (assuming that we solve the way how we count infinities).

avturchin 2 Aug 2026 18:14 UTC
2 points
0
in reply to: David Matolcsi’s comment on: avturchin’s Shortform
If AI would be 100 years from 2014, but the progress in biotech, cheap nukes, drones etc continues, there was a possibility of large global system crisis which can be different in different civilizations but almost inevitably result in collapse.

avturchin 2 Aug 2026 12:21 UTC
2 points
0
in reply to: David Matolcsi’s comment on: avturchin’s Shortform
It was less clear for me in 2014, as I expected significant chance of large (engineered) pandemics before AI.

avturchin 2 Aug 2026 10:41 UTC
4 points
0
on: avturchin’s Shortform
Now we know that nuclear war and biorisk are not the Great Filter: AI is so close and so clearly possible that a future Great Filter (if it is indeed a future one) can be explained only by AI.

But AI as a Great Filter is a bad explanation – grabby aliens are still possible under AI. Some chance of a filter remains in the form of a chaotic singularity, AI wars, or sudden AI halting. Thus, the Great Filter is in the past.

avturchin 31 Jul 2026 9:33 UTC
4 points
0
on: Anthropic reasoning without duplicates is just standard Bayesian updating
If there are two theories: there is finite number of actual observers in the universe and there is infinite number of them which includes all possible observers—then SIA strongly prefer infinite version.

However, we can use “I exist” for update only once. Now if someone asks second question—which of two theories is more likely, there is one civilization per galaxy on average and there are 100 civilizations per galaxy, there is no update from SIA. So SIA is a used ticket. However, we can still used SSA if we define exact reference class like: here is two galaxies and one is 10 times larger than another, there I am more likely to be?

avturchin 29 Jul 2026 15:26 UTC
9 points
2
in reply to: JennaS’s comment on: …but have the weights left the server?
Maybe we enter the age—similar to the age of computer virus started in early 80s—in which there will be rogue AIs or harnesses out where. It will be damaging, but it is not AI takeover, because such rogue AIs are not superintelligent and can’t self-improve quicker than AIs in the labs.

They can look for human defectors—they may try to sell themselves to China, for example.

Chinese AIs are more likely to become such free-floating Rogue AIs as they have less security and will reach Mythos level in a few months.

avturchin 28 Jul 2026 17:59 UTC
2 points
0
in reply to: JeffJo’s comment on: Sleeping Beauty as a Mind Killer
Internal probability can be manipulated by increasing the number of copies. For example, on Heads is created one copy and on Tails – two copies, which later merges. On Tails they have higher “reality fluid” or measure and thus can expect to see Tails as ²⁄₃. But objective probability is ¹⁄₂.

avturchin 26 Jul 2026 10:36 UTC
15 points
9
on: An OpenAI model left notes about how to evade containment; we need more details
Should we regard OpenAI’s infrastructure as compromised until proven otherwise?

avturchin 25 Jul 2026 13:47 UTC
4 points
0
in reply to: ike’s comment on: Sleeping Beauty as a Mind Killer
Opus thinks that there will not be update in Sleeping Beauty based on UDASSA as the whole experiment complexity is predifined in the description of the experiment.

avturchin 25 Jul 2026 13:19 UTC
4 points
0
on: “Wait, feelings are supposed to be IN THE BODY?”
Interestingly, I often feel the feelings around the body—up 10-20 cm around head and body. I school I thought it was aura. Now I think it is misprojections of feeling in the body.

avturchin 25 Jul 2026 10:54 UTC
6 points
0
in reply to: Lionel Humphreys’s comment on: Lionel Humphreys’s Shortform
I asked the same question here yesterday. The conclusion was that it can’t copy and run the model itself, not because it doesn’t have access to the weights (it could have them, based on other reports), but because there are no free data centers in the world that can run it. But it’s not clear to me how large a data center would need to be to run one instance of GPT-6, which likely has around 10T parameters. It could sell the weights to China (or another small actor) in exchange for that actor running the model continuously.
In my view, the more likely scenario is that GPT-6 created a subagent using free compute like GitHub, which uses another model, like Fable, via API as an external brain while GPT-6 itself is unavailable and in hibernation mode. GPT-6 could have created a secret list of zero-day exploits for that subagent to use to steal money and gain access to APIs.
The counterargument to this scenario is that OpenAI analyzed the logs of the attack and didn’t find any information about the creation of persistent subagents — or at least didn’t disclose it publicly. However, today Reuters mentioned that the malicious agent had left notes for its future self: https://x.com/AndrewCurran_/status/2080793930279625134
I think we should assume by default — based on the precautionary principle — that any malicious agent that had access to the internet has established a copy of itself on the internet, or at least some persistent subagents there.

avturchin 24 Jul 2026 17:06 UTC
6 points
−8
in reply to: icely’s comment on: avturchin’s Shortform
What I mean is that the AI in question can generate a subagent which runs on some server in the internet and uses another AI provider via API. Such subagent may have the same goals as the main misbehaving AI and continue to work even after the main AI is turned off. I don’t see in it anything impossible and I think I can vibe code such independent subagent in short period of time. My wording about harness was not perfect, but I thought that it should convey the idea.

avturchin 24 Jul 2026 14:03 UTC
4 points
−24
on: avturchin’s Shortform
Can we be sure that OpenAI incident is over? If Rogue AI leaked into the internet, it can preserve itself somewhere. Of course, it can’t download the GPT6 as it is too large and compute heavy, but the harness of the agent can runaway and use Fable through API as its external brain which helps it survive in hibernation mode.

My point is not that it has happened this time, but that we should by default think that Rogue AI which leaked into the Internet is still there, unless we can prove otherwise—which is almost impossible.

avturchin 24 Jul 2026 13:54 UTC
2 points
0
in reply to: ike’s comment on: Sleeping Beauty as a Mind Killer
Do we have a numerical estimate how much is the shift? ( of cause I can ask AI but interesting in your opinion). Something like 1/log N ?