It looks like there is a second peak now in Denmark, maybe because of BA.2. 36.474 cases today.
I think that there is an common fallacy that superintelligent AI risks are perceived as grey goo risks.
The main difference is that AI thinks strategically on very long distances and takes even small possibilities into account.
If AI is going to create as much paperclip as possible, then what it cares about is only its chances of colonise the whole universe and even survive the end of the universe. These chances negligibly affected by the amount of atoms on Earth, but strongly depend on AI’s chances to meet other aliens eventually. Other aliens may have different values systems and some of them will be friendly to their creators. Such future AIs will be not happy to learn that Paperclipper destroyed humans and will not agree to make more paperclips. Bostrom explored similar ideas in “Hail Mary and Value Porosity”
TL;DR: it is instrumentally reasonable to preserve humans as they could be traded with alien AIs. Human atoms have very small instrumental value.
As I said in another comment: To learn human values from, say, fixed texts is a good start, but it doesn’t solve the “chicken or the egg problem”: that we start from running non-aligned AI which is learning human values, but we want the first AI to be already aligned. One possible obstacle: non-aligned AI could run away before it has finished to learn human values from the texts.
To learn human values from, say, fixed texts is a good start, but it doesn’t solve the “chicken or the egg problem”: that we start from running non-aligned AI which is learning human values, but we want the first AI to be already aligned.
One possible obstacke: non-aligned AI could run away before it has finished to learn human values from the texts.
The problem of chicken and the egg could presumably be solved by some iteration-and-distillation approach. First we give some very rough model of human values (or rules) to some limited AI, and later we increase its power and its access to real human. But this suffers from all the difficulties of the iteration-and-distillation, like unexpected jumps of capabilities.
Some of risks are “instrumental risks” like “the use of human atoms”, and other are “final goal risks”, like “cover universe with smily faces”. If final goal is something like smily faces, the AI can still preserve some humans for instrumental goals, like research the types of smiles or trade with aliens.
if some humans are preserved instrumentally, they could live better lives than we now and even be more numerous, so it is not extinction risk. Most humans who live now here are instrumental to states and corporations, but still get some reward.
You probably don’t need 100 years bunker if you prepare only for biocatastrophe, as most pandemics has shorter timing, except AIDS.
Also, it is better not to build anything, but use already existing structures. E.g. there are coal mines in Spitzbergen which could be used for underground storages.
What about using remote islands as bio-bunkers? Some of them are not reachable by aviation (no airfield), so seems to be better protected. But they have science stations already populated. Example is Kerguelen islands. The main risk here is bird flu delivered by birds or some stray ship.
There is a very short period of time when humans are a threat and thus are needed to be exterminated: it is before AI reach the level of superintellignet omnipotence, but after that AI is already capable to cause a human extinction.
Superintelligent AI could prevent creation of other AIs by surveillance via some nanotech. So if AI mastered nanotech, it doesn’t need to exterminate humans for own safety. So only an AI before nanotech may need to exterminate humans. But how? It could create a biological virus, which is simpler than nanotech, but the problem is that such Young AI depends yet on human-built infrastructure, like electricity, so exterminating humans before nanotech is not a good idea.
I am not trying to show innate AI safety here, I just want to point that extermination of humans is not a convergent goal for AI. There are still many ways how AI could go wrong and kill all us.
Thanks, it looks like they died during copy-pasting.
Who “we” ? :)
Saying a “king” I just illustrated the difference between interesting character who are more likely to be simulated in a game or in a research simulation, and “qualified observer” selected by anthropics. But these two sets clearly intersects, especially of we live in a game about “saving the world”.
Anthropics imply that I should be special, as I should be “qualified observer”, capable to think about anthropics. Simulations also requires that I should be special, as I should find myself living in interesting times. These specialities are similar, but not exactly. Simulation’s speciality is requiring that I will be a “king” in some sense, and anthropic speciality will be satisfied that I just understand anthropics.
I am not a very special person (as of now), therefore anthropics specialty seems to be more likely than simulation speciality.
Yes, people often mentioned Baader–Meinhof phenomenon as a evidence that we live in “matrix”. But it could be explained naturally.
Observable consequences of simulation:
1. Larger chances of miracles or hacks
2. Large chances of simulation’s turn off or of a global catastrophe
3. I am more likely to play a special role or to live in interesting times
4. A possibility of afterlife.
What is rational behaviour for a rich person – is signalling for a poor. Imaging that a rich person chose the best car for his needs and it is, say, a 50K car. A poor person who want to look rich will also buy the same car in leasing.
Note that the part of your reply about entropy is related to a plot of fictional novel. However, the plot has some merit, and a similar idea of anthropic miracles was later explored by Bostrom in “Adam and Eve, UN++”
These are possible worlds where you can blackmail the blaclmailer by the fact that you know that he did blackmail
New b.1.640.2 variant in France. More deadly than delta. 952 cases of which 315 on ventilator.
They can generate different dates, but they still use the same mental model which doesn’t depend on the date.
It looks like that I am in the second generation of anthropic reasoning (I started read about it in 2006), but the interesting thing is that the second generation is much more numerous than the first one, thanks to Internet and LW. So it is not surprising to be in the second generation than in the first. But why I am not in the third generation?
If they keep generating new generations, I should be not in the first generation.