avturchin

Karma: 2,198

avturchin 19 Jan 2022 15:27 UTC
6 points
on: Omicron Post #15
It looks like there is a second peak now in Denmark, maybe because of BA.2. 36.474 cases today.

avturchin 19 Jan 2022 13:52 UTC
3 points
in reply to: Donald Hobson’s comment on: Plan B in AI Safety approach
I think that there is an common fallacy that superintelligent AI risks are perceived as grey goo risks.
The main difference is that AI thinks strategically on very long distances and takes even small possibilities into account.
If AI is going to create as much paperclip as possible, then what it cares about is only its chances of colonise the whole universe and even survive the end of the universe. These chances negligibly affected by the amount of atoms on Earth, but strongly depend on AI’s chances to meet other aliens eventually. Other aliens may have different values systems and some of them will be friendly to their creators. Such future AIs will be not happy to learn that Paperclipper destroyed humans and will not agree to make more paperclips. Bostrom explored similar ideas in “Hail Mary and Value Porosity”
TL;DR: it is instrumentally reasonable to preserve humans as they could be traded with alien AIs. Human atoms have very small instrumental value.

avturchin 19 Jan 2022 13:37 UTC
2 points
in reply to: Donald Hobson’s comment on: Possible Dangers of the Unrestricted Value Learners
As I said in another comment: To learn human values from, say, fixed texts is a good start, but it doesn’t solve the “chicken or the egg problem”: that we start from running non-aligned AI which is learning human values, but we want the first AI to be already aligned. One possible obstacle: non-aligned AI could run away before it has finished to learn human values from the texts.

avturchin 19 Jan 2022 13:35 UTC
2 points
in reply to: A Ray’s comment on: Possible Dangers of the Unrestricted Value Learners
To learn human values from, say, fixed texts is a good start, but it doesn’t solve the “chicken or the egg problem”: that we start from running non-aligned AI which is learning human values, but we want the first AI to be already aligned.
One possible obstacke: non-aligned AI could run away before it has finished to learn human values from the texts.
The problem of chicken and the egg could presumably be solved by some iteration-and-distillation approach. First we give some very rough model of human values (or rules) to some limited AI, and later we increase its power and its access to real human. But this suffers from all the difficulties of the iteration-and-distillation, like unexpected jumps of capabilities.

avturchin 18 Jan 2022 14:06 UTC
6 points
in reply to: RomanS’s comment on: Plan B in AI Safety approach
Some of risks are “instrumental risks” like “the use of human atoms”, and other are “final goal risks”, like “cover universe with smily faces”. If final goal is something like smily faces, the AI can still preserve some humans for instrumental goals, like research the types of smiles or trade with aliens.
if some humans are preserved instrumentally, they could live better lives than we now and even be more numerous, so it is not extinction risk. Most humans who live now here are instrumental to states and corporations, but still get some reward.

avturchin 16 Jan 2022 19:32 UTC
2 points
in reply to: A Ray’s comment on: Alex Ray’s Shortform
You probably don’t need 100 years bunker if you prepare only for biocatastrophe, as most pandemics has shorter timing, except AIDS.
Also, it is better not to build anything, but use already existing structures. E.g. there are coal mines in Spitzbergen which could be used for underground storages.

avturchin 16 Jan 2022 18:05 UTC
2 points
in reply to: A Ray’s comment on: Alex Ray’s Shortform
What about using remote islands as bio-bunkers? Some of them are not reachable by aviation (no airfield), so seems to be better protected. But they have science stations already populated. Example is Kerguelen islands. The main risk here is bird flu delivered by birds or some stray ship.

avturchin 15 Jan 2022 13:19 UTC
8 points
in reply to: Chris_Leong’s comment on: Plan B in AI Safety approach
There is a very short period of time when humans are a threat and thus are needed to be exterminated: it is before AI reach the level of superintellignet omnipotence, but after that AI is already capable to cause a human extinction.
Superintelligent AI could prevent creation of other AIs by surveillance via some nanotech. So if AI mastered nanotech, it doesn’t need to exterminate humans for own safety. So only an AI before nanotech may need to exterminate humans. But how? It could create a biological virus, which is simpler than nanotech, but the problem is that such Young AI depends yet on human-built infrastructure, like electricity, so exterminating humans before nanotech is not a good idea.
I am not trying to show innate AI safety here, I just want to point that extermination of humans is not a convergent goal for AI. There are still many ways how AI could go wrong and kill all us.

avturchin 14 Jan 2022 11:09 UTC
2 points
in reply to: iceman’s comment on: Plan B in AI Safety approach
Thanks, it looks like they died during copy-pasting.

Plan B in AI Safety approach

avturchin13 Jan 2022 12:03 UTC

32 points

9 comments2 min readLW link

avturchin 12 Jan 2022 11:26 UTC
2 points
in reply to: MackGopherSena’s comment on: avturchin’s Shortform
Who “we” ? :)
Saying a “king” I just illustrated the difference between interesting character who are more likely to be simulated in a game or in a research simulation, and “qualified observer” selected by anthropics. But these two sets clearly intersects, especially of we live in a game about “saving the world”.

avturchin 11 Jan 2022 12:01 UTC
2 points
in reply to: MackGopherSena’s comment on: avturchin’s Shortform
Anthropics imply that I should be special, as I should be “qualified observer”, capable to think about anthropics. Simulations also requires that I should be special, as I should find myself living in interesting times. These specialities are similar, but not exactly. Simulation’s speciality is requiring that I will be a “king” in some sense, and anthropic speciality will be satisfied that I just understand anthropics.
I am not a very special person (as of now), therefore anthropics specialty seems to be more likely than simulation speciality.

avturchin 8 Jan 2022 10:17 UTC
4 points
in reply to: Gunnar_Zarncke’s comment on: avturchin’s Shortform
Yes, people often mentioned Baader–Meinhof phenomenon as a evidence that we live in “matrix”. But it could be explained naturally.

avturchin 7 Jan 2022 11:55 UTC
4 points
on: avturchin’s Shortform
Observable consequences of simulation:
1. Larger chances of miracles or hacks
2. Large chances of simulation’s turn off or of a global catastrophe
3. I am more likely to play a special role or to live in interesting times
4. A possibility of afterlife.

avturchin 7 Jan 2022 10:14 UTC
0 points
on: Signaling isn’t about signaling, it’s about Goodhart
What is rational behaviour for a rich person – is signalling for a poor. Imaging that a rich person chose the best car for his needs and it is, say, a 50K car. A poor person who want to look rich will also buy the same car in leasing.

avturchin 5 Jan 2022 11:49 UTC
2 points
in reply to: MackGopherSena’s comment on: We need a theory of anthropic measure binding
Note that the part of your reply about entropy is related to a plot of fictional novel. However, the plot has some merit, and a similar idea of anthropic miracles was later explored by Bostrom in “Adam and Eve, UN++”

avturchin 4 Jan 2022 10:25 UTC
3 points
in reply to: Charlie Steiner’s comment on: Charlie Steiner’s Shortform
These are possible worlds where you can blackmail the blaclmailer by the fact that you know that he did blackmail

avturchin 3 Jan 2022 13:33 UTC
9 points
on: avturchin’s Shortform
New b.1.640.2 variant in France. More deadly than delta. 952 cases of which 315 on ventilator.
https://www.thailandmedical.news/news/breaking-updates-on-new-b-1-640-2-variant-spreading-in-southern-france-number-of-cases-growing-and-variant-now-detected-in-united-kingdom-as-well
https://flutrackers.com/forum/forum/europe-aj/europe-covid-19-sept-13-2020-may-31-2021/933598-southern-france-reports-of-new-variant-with-46-mutations

avturchin 3 Jan 2022 11:50 UTC
2 points
in reply to: MackGopherSena’s comment on: Each reference class has its own end
They can generate different dates, but they still use the same mental model which doesn’t depend on the date.
It looks like that I am in the second generation of anthropic reasoning (I started read about it in 2006), but the interesting thing is that the second generation is much more numerous than the first one, thanks to Internet and LW. So it is not surprising to be in the second generation than in the first. But why I am not in the third generation?

avturchin 3 Jan 2022 10:33 UTC
2 points
in reply to: MackGopherSena’s comment on: Each reference class has its own end
If they keep generating new generations, I should be not in the first generation.