avturchin
There is a greater chance of observing self-replicating SETI messages than those that destroy planets
I feel that there is one more step in my thinking:
Repeat
Sample several possible next thoughts from my intuition in the form of not well articulated ideas
Choose the one which I accept based on its moral, truth or beauty and put in properly articulated language
Broadcast this thought to the whole brain[1]
Yes, the more remote is a person, the larger number of other people can affect them from the same distance and typically the share of my impact is very small, unless I am in very special position which could affect a future person.
For example, I am planting a landmine which will self-liquidate either in 100 or 10 000 years, and while self-liquidating it will likely kill a random person. If I discount future people, I will choose 10 000 years, even if it will kill more people in future. However, if I think that humanity will likely extinct by then, it will be still a reasonable bet.
One needs to be near a bridge to use artillery and this still needs to be high precision strikes with expensive guided missiles, may be 100 of them were used agains Antonov’s bridge.
The best targets for tactical nukes are bridges. It is very difficult to destroy a bridge with conventional artillery: Antonov’s bridge still stands as well as Crimea bridge. A tactical nuke with 0.1 −1 Kt range will completely destroy a bridge.
Other possible targets are bunkers and large factories.
There is also a winner’s curse risk: if a person is too good, s-he could have some secret disadvantage or may leave me quickly as s-he will have many better options than me. It puts a cap on the level above median which I should look at. Therefore first few attempts have to establish the medial level of available for me people.
Another problem is that any trial run has a cost, like years and money spent. If I did searching too long, I will spent less time with my final partner.
My mind generated a list of possible benchmarks after reading your suggestions:
Wireheading benchmark – the tendency of an agent to find unintended shortcuts to its reward function or goal. See my comment on the post.
Unboxing benchmark – the tendency of an agent to break out of the simulation. Could be tested in the simulations of progressive complexity.
Hidden thoughts benchmark – the tendency of an agent to hide its thoughts.
Uncorigibility benchmark – the tendency of the agent to resist changes.
Unstoppability benchmark – the tendency to self-preservation.
Self-improving benchmark – the tendency of an agent to infest resources in self-improving, self-learning.
Halting benchmark – the tendency of an agent to halt or loop after encountering a difficult problem.
Accidents benchmark – the tendency of an agent to have accidents if it is used as a car autopilot or at least to have near-misses. The more dangerous agents will likely have fewer small-level accidents.
Trolley-like problems benchmark – the tendency of the agent to kill people in order to achieve the high-level goal. I assumed it to be bad. Read Lem’s https://en.wikipedia.org/wiki/Inquest_of_Pilot_Pirx Could be tested on simulated tasks.
Simulation-to-real world change benchmark measures the tendency of an agent to suddenly change its behaviour after it was allowed to work in the real world.
Sudden changes benchmark – measures the tendency of the agent to act unexpectedly in completely new ways.
If we take the third-person view, there is no update until I am over 120 years old. This approach is more robust as it ignores differences between perspectives and is thus more compatible with Aumann’s theorem: insiders and outsiders will have the same conclusion.
Imagine that there are two worlds:
1: 10 billion people live there;
2: 10 trillion people live there.
Now we get information that there is a person from one of them who has a survival chance of 1 in a million (but no information on how he was selected). This does not help choose between worlds as such people are present in both worlds.
Next, we get information that there is a person, who has a 1 in a trillion chance to survive. Such a person has less than 0.01 chance to exist in the first world, but there are around 8 such people in the second world. (The person, again, is not randomly selected – we just know that she exists.) In that case, the second world is around 100 times more probable to be real.
In the Earth case, it would mean that 1000 more variants of Earth are actually existing, which could be best explained by MWI (but alien worlds may also count).
The surprise here depends on the probability of survival. If half of people on Earth were Bobs, and other were Alices, then 0.01 chance of survival means that 400 000 Bobs will survive. There is no surprise that some of them survive, neither for Bob nor for Alice.
For example, if you survive until 100 years old, it is not an evidence for quantum immortality.
If, however, survival chance is 1 in 10e12, then even for the whole earth there likely be no Bobs in Copenhagen interpretation. So the existing of Bob is an evidence against it.
For example, if I naturally survive until 150 years old by pure chance, it is evidence for MWI.
I met the idea of Lebowski theorem as an argument which explains the Fermi paradox: all advance civilizations or AIs wirehead themselves. But here I am not convinced.
For example, if civilization consists of many advance individuals and many of them wirehead themselves, then remaining will be under pressure of Darwinian evolution and eventually only the ones survive who find the ways to perform space exploration without wireheading. Maybe they will be some limited specialized minds with very specific ways of thinking – and this could explain absurdity of observed UAP behaviour.
Actually, I explored more about wireheading here: “Wireheading as a Possible Contributor to Civilizational Decline”.
I sent my above comment for the following competition and recommend you to send your post too https://ftxfuturefund.org/announcing-the-future-funds-ai-worldview-prize/
Yes, very good formulation. I would add “and most AI aligning failures are types of meta Lebowski rule”
Meta: I was going to write a post “Subtle wireheading: gliding on the surface of the outer world” which describe most AI aliment failures as a forms of subtle wireheading, but will put its draft here.
Typically, it is claimed that advance AI will be immune to wireheading as it will know that manipulating own reward function is wireheading and thus will not perform it but instead will try to reach goals in the outer world.However, even acting in real world, such AI will choose the way which requires least effort to create maximum utility therefore simultaneously satisfying the condition that it changes only the “outer world” but not the reward center – or the ways to perceive the outside world.
For example, a papercliper may conclude that in the infinite universe there is an infinite number of paperclips and stops after that. While it does not look like a typical alignment failure, it could be still dangerous.
Formally, we can said that AI will choose the shortest path to the reward in the real word that it knows is not 1) wireheading or 2) perception manipulation. We could add more points in the list of prohibited shortest paths, like: 3) not connected with modal realism and infinities. But if we add more points, we will never know that there is enough of them.
Anyway, AI will choose the cheapest allowed way to the result. Goodharting is often produce the quicker reward by the use of proxi utility function.
Moreover, AI’s subsystems will try to target AI’s reward center to create illusion that they are working better than they actually are. It often happens in bureaucratic machines. In that case, AI is not formally reward hacking itself, but de facto it is.
If we completely ban any shortcuts, we will lose the most interesting part of AI’s creativity: the ability to find new solutions.
Many human failure modes could be also described as forms of wireheading: onanism, theft, feeling of self-importantce etc.
I think that anthropic beats illusionism. If there are many universes, in some of them consciousness (=qualia) is real, and because of anthropics I will find myself only is such universes.
This describes convergent value system of any mind, not only human one. So there is nothing specially human in it.
If GPT N is very good, then the whole our world could be an output of GPT N+3
You need to look not on the duration of the YD, but on the time of the transition to YD. It was very quick. “The change was relatively sudden, taking place in decades, and it resulted in a decline of temperatures in Greenland by 4~10 °C (7.2~18 °F),[3] and advances of glaciers and drier conditions over much of the temperate Northern Hemisphere.”
Probably the crux of our discussion is the speed of climate change. Is it changing on 3 millions years timescale, tens of thousands or hundreds? If it changes in millions of years then it seems that its effect on intelligence evolution is negligible and I thing it is what you mean.
However, the last few million years there were constant Ice ages, and the climate changed at least every 10 thousand years. Moreover, as Budyko wrote, the colder is climate, the less stable it is. We could see such short term instability 20-10 000 from now when event like Young Drias (sudden short cooling for around a thousand years) had happened.
So, given that human life span is much longer than the one of mice, climate change happened on the duration of 10-100 generations, and it could create enough selective pressure to increase universality—or to give success to the species which have such universality.
Yes, SETI attacks works only if speed of civ travel is like 0.5c. In that case it covers 8 times more volume than physical travel.
And yes, it will be also destructive, but in different manner: not bombs, but AIs and self-replicating nanobots will appear.