SoerenE

Karma: 151

SoerenE 13 Jan 2016 10:44 UTC
4 points
on: A toy model of the treacherous turn
I really like this visualization.

May I suggest another image, where the shopkeeper is in non-obvious danger:

To the left, the Shopkeeper is surrounded by ice-blocks, as in the images. All the way to the right, a monster is shooting arrows at Link, who is shooting arrows back at the monster. (The Gem-container is moved somewhere else.) Link, the Shopkeeper and the monster are on the same horizontal line. It looks like Link is about to heroically take an arrow that the monster aimed for the shopkeeper. The ice is still blocking, so the shopkeeper appears safe.

The problem is that Link can choose to go a bit north, dodging the next arrow from the monster. The monster’s arrow will then destroy the ice. If Link immediately afterwards time fires an arrow at the Shopkeeper, the shopkeeper will be killed, as arrows are faster than movement.

For this to work, I think the monster’s arrow should be aiming at the southern-most part of the ice-block, so Link only has to move a tiny bit. Link can then shoot at the Shopkeeper, and proceed to wirehead himself.

SoerenE 15 Jan 2016 20:38 UTC
5 points
on: Welcome to LessWrong (January 2016)
Hi,

I’ve read some of “Rationality: From AI to Zombies”, and find myself worrying about unfriendly strong AI.

Reddit recently had an AMA with the OpenAI team, where “thegdb” seems to misunderstand the concerns. Another user, “AnvaMiba” provides 2 links (http://www.popsci.com/bill-gates-fears-ai-ai-researchers-know-better and http://fusion.net/story/54583/the-case-against-killer-robots-from-a-guy-actually-building-ai/) as examples of researchers not worried about unfriendly strong AI.

The arguments presented in the links above are really poor. However, I feel like I am attacking a straw man—quite possibly, www.popsci.com is misrepresenting a more reasonable argument.

Where can I find some precise, well thought out reasons why the risk of human extinction from strong AI is not just small, but for practical purposes equal to 0? I am interested in both arguments from people who believe the risk is zero, and people who do not believe this, but still attempt to “steel man” the argument.

SoerenE 18 Jan 2016 8:05 UTC
2 points
in reply to: g_pepper’s comment on: Welcome to LessWrong (January 2016)
Thank you. That was exactly what I was after.

SoerenE 25 Jan 2016 7:29 UTC
3 points
in reply to: SoerenE’s comment on: A toy model of the treacherous turn
I’ve tried my hand at visualizing it:

http://i.imgur.com/VE0P8JY.png

This picture shows the very last instant that the shopkeeper can choose to reset Link.

There are a number of assumptions in my calculations, which might not be valid in the actual game. A key assumption is that arrows fly at 3 times walking speed.
The Shopkeeper will need to walk 1 tile north to reset Link. That requires the same amount of time as for an arrow to fly 3 tiles.
- At T=0, Link starts moving north, and the arrow heading towards Link continues heading west.
- At T=1, Link has moved 1/3rd of a tile north, and thus narrowly avoids the arrow. The arrow continues West. Link takes an openly treacherous turn: He changes to the bow and fires an arrow west, towards the shopkeeper.
- At T=2, the arrow from the monster destroys the ice-block protecting the shopkeeper. Link’s arrow continues towards the shopkeeper.
- At T=3, Link’s arrow hits the shopkeeper. If the shopkeeper was moving north the entire time, the shopkeeper hits the reset button at this time.
If the shopkeeper decided to go for the reset button at T=0, the reset and the death of the shopkeeper happen simultaneous, and the shopkeeper dies while Link is reset. Notice that a reset (-1000 points) followed by wireheading (+inifinity) is a great move.

If Link moves north, and the shopkeeper immediately follows, Link can just move south again, to block the arrow. The openly treacherous turn at T=1 happens when it is too late for the shopkeeper to do anything about it.

I also like with this visualization that an enemy is present. It is easy to construct a story where a smart AI manipulates the situation until the shopkeeper is in a situation where he can choose between trusting the AI, or death.

SoerenE 29 Jan 2016 7:50 UTC
1 point
in reply to: Yosarian2’s comment on: Beware surprising and suspicious convergence
My thoughts exactly.

When I first heard it, it sounded to me like a headline from BuzzFeed: This one weird trick will literally solve all your problems!

Turns out that the trick is to create an IQ 20000 AI, and get it to help you.

(Obviously, Suspicious <> Wrong)

SoerenE 11 Feb 2016 6:29 UTC
3 points
in reply to: lisper’s comment on: Is Spirituality Irrational?
It is possible to be extremely intelligent, and suffer from a delusion.

SoerenE 11 Feb 2016 19:32 UTC
4 points
in reply to: lisper’s comment on: Is Spirituality Irrational?
I am tapping out of this thread.

SoerenE 14 Feb 2016 19:59 UTC
0 points
on: Does Kolmogorov complexity imply a bound on self-improving AI?
It is an interesting way of looking at the maximal potential of AIs. It could be that Oracle Machines are possible in this universe, but an AI built by humans cannot self-improve to that point because of the bound you are describing.

I feel that the phrasing “we have reached the upper bound on complexity” and later “can rise many orders of magnitude” gives a potentially misleading intuition about how limiting this bound is. Do you agree that this bound does not prevent us from building “paperclipping” AIs?

SoerenE 18 Feb 2016 7:11 UTC
1 point
in reply to: OrphanWilde’s comment on: Estimating the probability of human extinction
Could you elaborate on why you consider p(UFAI before 2116) < 0.01? I am genuinely interested.

SoerenE 18 Feb 2016 7:19 UTC
0 points
in reply to: James_Miller’s comment on: Estimating the probability of human extinction
I’ve seen this claim many places, including in the Sequences, but I’ve never been able to track down an authoritative source. It seems false in classical physics, and I know little about relativity. Unfortunately, my Google-Fu is too weak to investigate. Can anyone help?

SoerenE 18 Feb 2016 20:14 UTC
1 point
in reply to: OrphanWilde’s comment on: Estimating the probability of human extinction
Some of the smarter (large, naval) landmines are arguably both intelligent and unfriendly. Let us use the standard AI risk metric.

I feel that your sentence does refer to something: A hypothetical scenario. (“Godhood” should be replaced with “Superintelligence”).

Is it correct that the sentence can be divided into these 4 claims?:
1. An AI self-improves it’s intelligence
2. The self-improvement becomes recursive
3. An AI reaches superintelligence through 1 and 2
4. This can happen in a process that can be called “runaway”
Do you mean that one of the probabilities is extremely small? (E.g., p(4 | 1 and 2 and 3) = 0.02). Or do you mean that the statement is not well-formed? (E.g, Intelligence is poorly-defined by the AI Risk theory)

SoerenE 19 Feb 2016 7:14 UTC
0 points
in reply to: _rpd’s comment on: Estimating the probability of human extinction
Thank you. It is moderately clear to me from the link that James’ thought-experiment is possible.

Do you know of a more authoritative description of the thought-experiment, preferably with numbers? It would be nice to have an equation where you give the speed of James’ spaceship and the distance to it, and calculate if the required speed to catch it is above the speed of light.

SoerenE 19 Feb 2016 7:22 UTC
0 points
in reply to: Lumifer’s comment on: Estimating the probability of human extinction
I meant claim number 3 to be a sharper version of your claim: The AI will meet constraints, impediments and roadblocks, but these are overcome, and the AI reaches superintelligence.

Could you explain the analogy with human reproduction?

SoerenE 19 Feb 2016 8:07 UTC
−1 points
in reply to: OrphanWilde’s comment on: Estimating the probability of human extinction
Intelligence, Artificial Intelligence and Recursive Self-improvement are likely poorly defined. But since we can point to concrete examples of all three, this is a problem in the map, not the territory. These things exist, and different versions of them will exist in the future.

Superintelligences do not exist, and it is an open question if they ever will. Bostrom defines superintelligences as “an intellect that is much smarter than the best human brains in practically every field, including scientific creativity, general wisdom and social skills.” While this definition has a lot of fuzzy edges, it is conceivable that we could one day point to a specific intellect, and confidently say that it is superintelligent. I feel that this too is a problem in the map, not the territory.

I was wrong to assume that you meant superintelligence when you wrote godhood, and I hope that you will forgive me for sticking with “superintelligence” for now.

SoerenE 19 Feb 2016 20:00 UTC
0 points
in reply to: Lumifer’s comment on: Estimating the probability of human extinction
English is not my first language. I think I would put the accent on “reaches”, but I am unsure what would be implied by having the accent on “super”. I apologize for my failure to write clearly.

I now see the analogy with human reproduction. Could we stretch the analogy to claim 3, and call some increases in human numbers “super”?

The lowest estimate of the historical number of humans I have seen is from https://en.wikipedia.org/wiki/Population_bottleneck , claiming down to 2000 humans for 100.000 years. Human numbers will probably reach a (mostly cultural) limit of 10.000.000.000. I feel that this development in human numbers deserves to be called “super”.

The analogy could perhaps even be stretched to claim 4 - some places at some times could be characterized by “runaway population growth”.

SoerenE 19 Feb 2016 20:16 UTC
0 points
in reply to: _rpd’s comment on: Estimating the probability of human extinction
Wow. It looks like light from James’ spaceship can indeed reach us, even if light from us cannot reach the spaceship.

SoerenE 20 Feb 2016 15:20 UTC
0 points
in reply to: Lumifer’s comment on: Estimating the probability of human extinction
Like Unfriendly AI, algae blooms are events that behave very differently from events we normally encounter.

I fear that the analogies have lost a crucial element. OrphanWIlde considered Unfriendly AI “vaguely magical” in the post here. The algae bloom analogy also has very vague definitions, but the changes in population size of an algae bloom is a matter I would call “strongly non-magical”.

I realize that you introduced the analogies to help make my argument precise.

SoerenE 3 Mar 2016 19:52 UTC
0 points
in reply to: Lumifer’s comment on: Estimating the probability of human extinction
Many things are far beyond our current abilities, such as interstellar space travel. We have no clear idea of how humanity will travel to the stars, but the subject is neither “vaguely magical”, nor is it true that the sentence “humans will visit the stars” does not refer to anything.

I feel that it is an unfair characterization of the people who investigate AI risk to say that they claim it will happen by magic, and that they stop the investigation there. You could argue that their investigation is poor, but it is clear that they have worked a lot to investigate the processes that could lead to Unfriendly AI.

SoerenE 4 Mar 2016 7:45 UTC
0 points
in reply to: Lumifer’s comment on: Estimating the probability of human extinction
You bring up a good point, whether it is useful to worry about UFAI.

To recap, my original query was about the claim that p(UFAI before 2116) is less than 1% due to UFAI being “vaguely magical”. I am interested in figuring out what that means—is it a fair representation of the concept to say that p(Interstellar before 2116) is less than 1% because interstellar travel is “vaguely magical”?

What would be the relationship between “Requiring Advanced Technology” and “Vaguely Magical”? Clarke’s third law is a straightforward link, but “vaguely magical” has previously been used to indicate poor definitions, poor abstractions and sentences that do not refer to anything.

SoerenE 4 Mar 2016 19:30 UTC
0 points
in reply to: Lumifer’s comment on: Estimating the probability of human extinction
This interpretation makes a lot of sense. The term can describe events that have a lot of Knightian Uncertainty, which a “Black Swan” like UFAI certainly has.