The audience here is mainly Americans so you might want to add an explicit sarcasm tag.
johnlawrenceaspden
Karma: 1,723
“Fault” seems a strange phrasing. If your problem was that one of your nerves was misfiring, so you were in chronic pain, would you describe that as “your fault”? (In the sense of technical fault/malfunction, that would absolutely be your “fault”, but “your fault” usually carries moral implications.)
Where would you place the fault?
I suspect everyone can relate in that everyone has felt this at some point, or even at a few memorable points.
Duncan did you just deny my existence? (Don’t worry, I don’t mind a bit. :-) )
I’m a grade A weirdo, my own family and friends affirm this, only the other day someone on Less Wrong (!) called me a rambling madman. My nickname in my favourite cricket club/drinking society was Space Cadet.
And I’m rather smug about this. Everyone else just doesn’t seem very good at thinking. Even if they’re right they’re usually right by accident. Even the clever ones seem to have some sort of blinders on. They don’t even take their own ideas seriously.
Why would I be upset by being able to see things they can’t see, think thoughts they can’t think? That doesn’t seem to be the sort of thing that could hurt me.
For most of your essay I was thinking: “Is he just mistaking metaphorical ‘everyone’ for literal ‘everyone’?”. But in the comments you say that’s not what you meant at all. And I don’t even understand that. Surely, if you replace ‘everyone’ with ‘most people’ throughout, your existence is not being denied?
And if your existence was being denied, why would that be a problem? If someone came straight up to me and said “You don’t exist”, I’d just think they were mad, it wouldn’t hurt.
I read that you’re in pain and it puzzles me. I’ve always wondered if the bit of my brain that is supposed to feel pressure-to-conform is malformed. I notice it, but it doesn’t seem powerful. Maybe yours is in perfect working order? Is it that you really really want to fit in, but in order to do so you’d have to be someone else, and that hurts?
Or have I failed to extract from your essay the meaning you were trying to put into it?
dalmations->dalmatians?
Did someone fiddle with Charlotte?
I went to talk to her after reading this and she was great fun, I quite see how you fell for her.
But I tried again just now and she seems a pale shadow of her former cheerful self, it’s the difference between speaking to a human PR drone in her corporate capacity and meeting her at a party where she’s had a couple.
Doesn’t any such argument also imply that you should commit suicide?
These seemed good, they taste of lavender, but the person trying them got no effect:
Lindens Lavender Essential Oil 80mg Capsules
The person who had it work for them tried something purchased from a shop, Herbal Calms maybe?, anyway, lavender oil in vegetable oil in little capsules. She reports that she can get to sleep now, and can face doing things that she couldn’t previously do due to anxiety if she pops a capsule first.
That makes perfect sense, thank you. And maybe, if we’ve already got the necessary utility function, stability under self-improvement might be solvable as if it were just a really difficult maths problem. It doesn’t look that difficult to me, a priori, to change your cognitive abilities whilst keeping your goals.
AlphaZero got its giant inscrutable matrices by working from a straightforward start of ‘checkmate is good’. I can imagine something like AlphaZero designing a better AlphaZero (AlphaOne?) and handing over the clean definition of ‘checkmate is good’ and trusting its successor to work out the details better than it could itself.
I get cleverer if I use pencil and paper, it doesn’t seem to redefine what’s good when I do. And no-one stopped liking diamonds when we worked out that carbon atoms weren’t fundamental objects.
---
My point is that the necessary utility function is the hard bit. It doesn’t look anything like a maths problem to me, *and* we can’t sneak up on it iteratively with a great mass of patches until it’s good enough.
We’ve been paying a reasonable amount of attention to ‘what is good?’ for at least two thousand years, and in all that time no-one came up with anything remotely sensible sounding.
I would doubt that the question meant anything, if it were not that I can often say which of two possible scenarios I prefer. And I notice that other people often have the same preference.
I do think that Eliezer thinks that given the Groundhog Day version of the problem, restart every time you do something that doesn’t work out, we’d be able to pull it off.
I doubt that even that’s true. ‘Doesn’t work out’ is too nebulous.
But at this point I guess we’re talking only about Eliezer’s internal thoughts, and I have no insight there. I was attacking a direct quote from the podcast, but maybe I’m misinterpreting something that wasn’t meant to bear much weight.
What I am not convinced of, is that given all those assumptions being true, certain doom necessarily follows, or that there is no possible humanly tractable scheme which avoids doom in whatever time we have left.
OK, cool, I mean “just not building the AI” is a good way to avoid doom, and that still seems at least possible, so we’re maybe on the same page there.
And I think you got what I was trying to say, solving 1 and/or 2 can’t be done iteratively or by patching together a huge list of desiderata. We have to solve philosophy somehow, without superintelligent help. As I say, that looks like the harder part to me.
Please don’t confuse me for someone who doesn’t often worry about these things.
I promise I’ll try not to!
A good guess, and thank you for the reference, but (although I admit that the prospect of global imminent doom is somewhat anxious-making), anxiety isn’t a state of mind I’m terribly familiar with personally. I’m very emotionally stable usually, and I lost all hope years ago. It doesn’t bother me much.
It’s more that I have the ‘taking ideas seriously’ thing in full measure, once I get an *idee fixe* I can’t let it go until I’ve solved it. AI Doom is currently third on the list after chess and the seed oil nonsense, but the whole Bing/Sydney thing started me thinking about it again and someone emailed me Eliezer’s podcast, you know how it goes.
Although I do have a couple of friends who suffer greatly from Anxiety Disorder, and you have my sympathies, especially if you’re interested in all this stuff! Honestly run away, there’s nothing to be done and you have a life to live.
Totally off topic but have you tried lavender pills? I started recommending them to friends after Scott Alexander said they might work, and out of three people I’ve got one total failure, one refusal to take for good reasons, and one complete fix! Obviously do your own research as to side effects, just cause it’s ‘natural’ doesn’t mean it’s safe. The main one is if you’re a girl it will interfere with your hormones and might cause miscarriages.
To be clear, even if I were somehow granted vivid knowledge of the future through precognition, you’d still seem crazy to me at this point.
(I assume you mean vivid knowledge of the future in which we are destroyed, obviously in the case where everything goes well I’ve got some problem with my reasoning)
That’s a good distinction to make, a man can be right for the wrong reasons.
Even as a doomer among doomers, you, with respect, come off as a rambling madman.
Certainly mad enough to take “madman” as a compliment, thank you!
I’d be interested if you know a general method I could use to tell if I’m mad. The only time I actually know it happened (thyroid overdose caused a manic episode) I noticed pretty quickly and sought help. What test should I try today?
Obviously “everyone disagrees with me and I can’t convince most people” is a bad sign. But after long and patient effort I have convinced a number of unfortunates in my circle of friends. Some of whom have always seemed pretty sharp to me.
And you must admit, the field as a whole seems to be coming round to my point of view!
Rambling I do not take as a compliment. But nevertheless I thank you for the feedback.
I thought I’d written the original post pretty clearly and succinctly. If not, advice on how to write more clearly is always welcome. If you get my argument, can you steelman it?
I’m guessing you’re operating on strong intuition here
Your guess is correct, I literally haven’t shifted my position on all this since 2010. Except to notice that everything’s happening much faster than I expected it to. Thirteen years ago I expected this to kill our children. Now I worry that it’s going to kill my parents. AlphaZero was the fire alarm for me. General Game Playing was one of the more important sub-problems.
I agree that if you haven’t changed your mind for thirteen years in a field that’s moving fast, you’re probably stuck.
I think my basic intuitions are:
“It’s a terrible idea to create a really strong mind that doesn’t like you.”
“Really strong minds are physically possible, humans are nowhere near.”
“Human-level AI is easy because evolution did it to us, quickly, and evolution is stupid.”
“Recursive self-improvement is possible.”
Which of these four things do you disagree with? Or do you think the four together are insufficient?
None. But if a problem’s not solvable in an easy case, it’s not solvable in a harder case.
Same argument as for thinking about Solomonoff Induction or Halting Oracles. If you can’t even do it with magic powers, that tells you something about what you can really do.
I’m not proposing solutions here. I think we face an insurmountable opportunity.
But for some reason I don’t understand, I am driven to stare the problem in the face in its full difficulty.
I like your phrasing better, but I think it just hides some magic.
In this situation I think we get an AI that repeatedly kills 999,999 people. It’s just the nearest unblocked path problem.
The exact reset/restart/turn it off and try again condition matters, and nothing works unless the reset condition is ‘that isn’t going to do something we approve of’.
The only sense I can make of the idea is ‘If we already had a friendly AI to protect us while we played, we could work out how to build a friendly AI’.
I don’t think we could iterate to a good outcome, even if we had magic powers of iteration.
Your version makes it strictly harder than the ‘Groundhog Day with Memories Intact’ version. And I don’t think we could solve that version.
On the chess thing, the reason why I went from ‘AI will kill our children’ to ‘AI will kill our parents’ shortly after I understood how AlphaZero worked was precisely because it seemed to play chess like I do.
I’m an OK chess player (1400ish), and I when I’m playing I totally do the ‘if I do this and then he moves this and then ….’ thing, but not very deep. And not any deeper than I did as a beginner, and I’m told grandmasters don’t really go any deeper.
Most of the witchy ability to see good chess moves is coming from an entirely opaque intuition about what moves would be good, and what positions are good.
You can’t explain this intuition in any way that allows it to move from mind to mind, although you can sometimes in retrospect justify it, or capture bits of it in words.
You train it through doing loads of tactics puzzles and playing loads of games.
AlphaZero was the first time I’d seen an AI algorithm where the magic didn’t go away after I’d understood it.
The first time I’d looked at something and thought: “Yes, that’s it, that’s intelligence. The same thing I’m doing. We’ve solved general game playing and that’s probably most of the way there.”
Human intelligence really does, to me, look like a load of opaque neural nets combined with a rudimentary search function.
What? A major reason we’re in the current mess is that we don’t know how to do this.
Well, we are not very good at it, but generally speaking, however capitalism seems to be acting to degrade our food, food companies are not knowingly routinely putting poisonous additives in food.
And however bad medicine is, it does seem to be a net positive these days.
Both of these things are a big improvement on Victorian times!
So maybe we are a tiny bit better at it than we used to be?
Not convinced it actually helps, mind....
So, if we already had friendly AI, we’d take 50 years to solve friendly AI?
I am totally nitpicking here. I think everyone sane agrees that we’re doomed and soon. I’m just trying to destroy the last tiny shreds of hope.
Even if we can sort out the technical problem of giving AIs goals and keeping them stable under self-improvement, we are still doomed.
We have two separate impossible problems to solve, and no clue how to solve either of them.
Maybe if we can do ‘strawberry alignment’, we can kick the can down the road far enough for someone to have a bright idea.
Maybe strawberry alignment is enough to get CEV.
But strawberry alignment is *hard*, both the ‘technical problem’ and the ‘what wish’ problem.
I don’t think Groundhog Day is enough. We just end up weirdly doomed rather than straightforwardly doomed.
And actually the more I think about it, the more I prefer straightforward doom. Which is lucky, because that’s what we’re going to get.
Sure, I read that a few years after he wrote it, and it’s still probably the best idea, but even if it’s feasible it needs superintelligent help! So we have to solve the alignment problem to do it.
I totally get where you’re coming from, and if I thought the chance of doom was 1% I’d say “full speed ahead!”
As it is, at fifty-three years old, I’m one of the corpses I’m prepared to throw on the pile to stop AI.
Hell yes. That’s been needed rather urgently for a while now.