A Great Man and an inspiration to me and to this community and to all thinking men.
God rest his soul in peace in Paradise.
johnlawrenceaspden
Karma: 1,691
alt-text is supposed to be: “I’m not even sure they’ve read Superintelligence”
Forgive me, I have strongly downvoted this dispassionate, interesting, well-written review of what sounds like a good book on an important subject because I want to keep politics out of Less Wrong.
This is the most hot-button of topics, and politics is the mind-killer. We have more important things to think about and I do not want to see any of our political capital used in this cause on either side.
typo-wise you have a few uses of it’s (it is) where it should be its (possessive), and “When they Egyptians” should probably read “When the Egyptians”.
I did enjoy your review. Thank you for writing it. Would you delete it and put it elsewhere?
Nicely done! I only come here for the humour these days.
Well, this is nice to see! Perhaps a little late, but still good news...
caches out
cashes out?
Haven’t you just “fabricated an option” where it’s possible to talk about politics on less wrong without it turning into a mind-killed clusterfuck? I mean yes, it would be lovely.....
I wouldn’t touch this stuff with someone else’s bargepole. It looks like it takes the willpower out of starvation, and as the saying goes, you can starve yourself thin, but you can’t starve yourself healthy.
I could be convinced, by many years of safety data and a well understood causal mechanism for both obesity and the action of these drugs, that that’s wrong and that they really are a panacea. But I am certainly not currently convinced!
The question that needs answering about obesity is ‘why on earth are people with enormous excess fat reserves feeling hungry?‘. It’s like having a car with the boot full of petrol in jerry cans but the ‘fuel low’ light is blinking.
depends on facts about physics and psychology
It does, and a superintelligence will understand those facts better than we do.
My basic argument is that the there are probably mathematical limits on how fast it is possible to learn.
Doubtless there are! And limits to how much it is possible to learn from given data.
But I think they’re surprisingly high, compared to how fast humans and other animals can do it.
There are theoretical limits to how fast you can multiply numbers, given a certain amount of processor power, but that doesn’t mean that I’d back the entirety of human civilization to beat a ZX81 in a multiplication contest.
What you need to explain is why learning algorithms are a ‘different sort of thing’ to multiplication algorithms.
Maybe our brains are specialized to learning the sorts of things that came in handy when we were animals.
But I’d be a bit surprised if they were specialized to abstract reasoning or making scientific inferences.
All of RL’s successes, even the huge ones like AlphaGo (which beat the world champion at Go) or its successors, were not easy to train. For one thing, the process was very unstable and very sensitive to slight mistakes. The networks had to be designed with inductive biases specifically tuned to each problem.
And the end result was that there was no generalization. Every problem required you to rethink your approach from scratch. And an AI that mastered one task wouldn’t necessarily learn another one any faster.
I had the distinct impression that AlphaZero (the version of AlphaGo where they removed all the tweaks) could be left alone for an afternoon with the rules of almost any game in the same class as go, chess, shogi, checkers, noughts-and-crosses, connect four, othello etc, and teach itself up to superhuman performance.
In the case of chess, that involved rediscovering something like 400 years of human chess theorizing, to become the strongest player in history including better than all previous hand-constructed chess programs.
In the case of go, I am told that it not only rediscovered a whole 2000 year history of go theory, but added previously undiscovered strategies. “Like getting a textbook from the future”, is a quote I have heard.
That strikes me as neither slow nor ungeneral.
And there was enough information in the AlphaZero paper that it was replicated and improved on by the LeelaChessZero open-source project, so I don’t think there can have been that many special tweaks needed?
This is great. Strong upvote!
Are you claiming that a physically plausible superintelligence couldn’t infer the physical laws from a video, or that AIXI couldn’t?
Those seem to be different claims and I wonder which of the two you’re aiming at?
For example, you might be much smarter than me and a meteorologist, but you’d find it hard to predict the weather in a year’s time better than me if it’s a single-shot-contest.
Sure, but I’d presumably be quite a lot better at predicting the weather in two days time.
I think this is a great article, and the thesis is true.
The question is, how much intelligence is worth how much material?
Humans are so very slow and stupid compared to what is possible, and the world so complex and capable of surprising behaviour, that my intuition is that even a very modest intelligence advantage would be enough to win from almost any starting position.
You can bet your arse that any AI worthy of the name will act nice until it’s already in a winning position.
I would.
If there’s some intelligence threshold past which minds pretty much always draw against each other in chess even if there is a giant intelligence gap between them, I wouldn’t be that surprised.
Just reinforcing this point. Chess is probably a draw for the same reason Noughts-and-crosses is.
Grandmaster chess is pretty drawish. Computer chess is very drawish. Some people think that computer chess players are already near the standard where they could draw against God.
Noughts-and-crosses is a very simple game and can be formally solved by hand. Chess is only a bit less simple, even though it’s probably beyond actual formal solution.
The general Game of Life is so very far beyond human capability that even a small intelligence advantage is probably decisive.
The “purpose” of most martial arts is to defeat other martial artists of roughly the same skill level, within the rules of the given martial art.
Optimizing for that is not the same as optimizing for general fighting. If you spent your time on the latter, you’d be less good at the former.
“Beginner’s luck” is a thing in almost all games. It’s usually what happens when someone tries a strategy so weird that the better player doesn’t immediately understand what’s going on.
The other day a low-rated chess player did something so weird in his opening that I didn’t see the threat, and he managed to take one of my rooks.
That particular trap won’t work on me again, and might not have worked the first time if I’d been playing someone I was more wary of.
I did eventually manage to recover and win, but it was very close, very fun, and I shook his hand wholeheartedly afterwards.
Every other game we’ve played I’ve just crushed him without effort.
About a year ago I lost in five moves to someone who tried the “Patzer Attack”. Which wouldn’t work on most beginners. The first time I’d ever seen it. It worked once. It will never work on me again.
For a clear example of this, in endgames where I have a winning position but have little to no idea how to win, Stockfish’s king will often head for the hills, in order to delay the coming mate as long as theoretically possible.
Making my win very easy because the computer’s king isn’t around to help out in defence.
This is not a theoretical difficulty! It makes it very difficult to practise endgames against the computer.
Paul, this is very thought provoking, and has caused me to update a little. But:
I loathe factory-farming, and I would spend a large fraction of my own resources to end it, if I could.
I believe that makes me unusually kind by human standards, and by your definition.
I like chickens, and I wish them well.
And yet I would not bat an eyelid at the thought of a future with no chickens in it.
I would not think that a perfect world could be improved by adding chickens.
And I would not trade a single happy human soul for an infinity of happy chickens.
I think that your single known example is not as benevolent as you think.
zero-days are a thing, and hell, it’s even possible that there are computers connected to the internet somewhere that don’t get their patches in a timely manner.
I don’t buy this, the curvedness of the sea is obvious to sailors, e.g. you see the tops of islands long before you see the beach, and indeed to anyone who has ever swum across a bay! Inland peoples might be able to believe the world is flat, but not anyone with boats.