Oops, I accidentally deleted the parent post! To clarify the context to other readers, the point I made in it was that one extremely strong piece of evidence against Clippy’s authenticity, regardless of any other considerations, would be that he displays the same level of intelligence as a smart human—whereas the abilities of machines at particular tasks follow the rule quoted by Joshua above, so they’re normally either far inferior or far superior to humans.
Now to address the above reply:
The second is very important: Do you mean about the middle of the pack or akin to the very best humans in the skill?
I think the point stands regardless of which level we use as the benchmark. If the task in question is something like playing chess, where different humans have very different abilities, then it can take a while for technology to progress from the level of novice/untalented humans to the level of top performers and beyond. However, it normally doesn’t remain at any particular human level for a long time, and even then, there are clearly recognizable aspects of the skill in question where either the human or the machine is far superior. (For example, motor vehicles can easily outrace humans on flat ground, but they are still utterly inferior to humans on rugged terrain.)
Regarding your specific example of chess, your timeline of chess history is somewhat inaccurate, and the claim that “the best chess programs are still not always beating grandmasters” is false. The last match between a top-tier grandmaster, Michael Adams, and a top-tier specialized chess computer was played in 2005, and it ended with such humiliation for the human that no grandmaster has dared to challenge the truly best computers ever since. The following year, the world champion Kramnik failed to win a single game against a program running on an off-the-shelf four-processor box. Nowadays, the best any human could hope for is a draw achieved by utterly timid play, even against a $500 laptop, and grandmasters are starting to lose games against computers even in handicap matches where they enjoy initial advantages that are considered a sure win at master level and above.
Top-tier grandmasters could still reliably beat computers all until early-to-mid nineties, and the period of rough equivalence between top grandmasters and top computers lasted for only a few years—from the development of Deep Blue in 1996 to sometime in the early 2000s. And even then, the differences between human and machine skills were very great in different aspects of the game—computers were far better in tactical calculations, but inferior in long-term positional strategy, so there was never any true equivalence.
So, on the whole, I’d say that the history of computer chess confirms the stated rule.
Does anything interesting happen when top chess programs play against each other?
One interesting observation is that games between powerful computers are drawn significantly less often than between grandmasters. This seems to falsify the previously widespread belief that grandmasters draw games so often because of flawless play that leaves the opponent no chance for winning; rather, it seems like they miss important winning strategies.
Is work being done on humans using chess programs as aids during games?
the claim that “the best chess programs are still not always beating grandmasters” is false
My impression is that draws can still occasionally occur against grandmasters. Your point about handicaps is a very good one.
Top-tier grandmasters could still reliably beat computers all until early-to-mid nineties, and the period of rough equivalence between top grandmasters and top computers lasted for only a few years—from the development of Deep Blue in 1996 to sometime in the early 2000s. And even then, the differences between human and machine skills were very great in different aspects of the game—computers were far better in tactical calculations, but inferior in long-term positional strategy, so there was never any true equivalence.
That’s another good point. However, it does get into the question of what we mean by equivalent and what metric you are using. Almost all technologies (not just computer technologies) accomplish their goals in a way that is very different than how humans do. That means that until the technology is very good there will almost certainly be a handful of differences between what the human does well and what the computer does well.
It seems in the context of the original conversation, whether the usual pattern of technological advancement is evidence against Clippy’s narrative, the relevant era to compare Clippy to in this context would be the long period where computers could beat the vast majority of chess players but sitll sometimes lost to grandmasters. That period lasted from the late 1970s to a bit over 2000. By analogy, Clippy would be in the period where it is smarter than most humans (I think we’d tentatively agree that that appears to be the case) but not so smart as to be of vastly more intelligent than humans. Using the Chess example, that period of time could plausibly last quite some time.
Also, Clippy’s intelligence may be limited in what areas it can handle.There’s a natural plateau for the natural language problem in that once it is solved that specific aspect won’t see substantial advancement from casual conversation. (There’s also a relevant post that I can’t seem to find where Eliezer discussed the difficulty of evaluating the intelligence of people that are much smarter than you.) If that’s the case, then Clippy is plausibly at the level where it can handle most forms of basic communication but hasn’t handled other levels of human processing to the point where it has generally become even with the smartest humans. For example, there’s evidence for this in that Clippy has occasionally made errors of reasoning and has demonstrated that it has a very naive understanding of human social interaction protocols.
My impression is that draws can still occasionally occur against grandmasters.
And I can get a draw (more than occasionally) against computer programs I have almost no hope of ever winning against. Draws are easy if you do not try to win.
From what I know, at grandmaster level, it is generally considered to be within the white player’s power to force the game into a dead-end drawn position, leaving the black no sensible alternative at any step. This is normally considered cowardly play, but it’s probably the only way a human could hope for even a draw against a top computer these days.
With black pieces, I doubt that even the most timid play would help against a computer with an extensive opening book, programmed to steer the game into maximally complicated and uncertain positions at every step. (I wonder if anyone has looked at the possibility of teaching computers Mikhail Tal-style anti-human play, where they would, instead of calculating the most sound and foolproof moves, steer the game into mind-boggling tactical complications where humans would get completely lost?) In any case, I am sure that taking any initiative would be a suicidal move against a computer these days.
(Well, there is always a very tiny chance that the computer might blunder.)
Oops, I accidentally deleted the parent post! To clarify the context to other readers, the point I made in it was that one extremely strong piece of evidence against Clippy’s authenticity, regardless of any other considerations, would be that he displays the same level of intelligence as a smart human—whereas the abilities of machines at particular tasks follow the rule quoted by Joshua above, so they’re normally either far inferior or far superior to humans.
Now to address the above reply:
I think the point stands regardless of which level we use as the benchmark. If the task in question is something like playing chess, where different humans have very different abilities, then it can take a while for technology to progress from the level of novice/untalented humans to the level of top performers and beyond. However, it normally doesn’t remain at any particular human level for a long time, and even then, there are clearly recognizable aspects of the skill in question where either the human or the machine is far superior. (For example, motor vehicles can easily outrace humans on flat ground, but they are still utterly inferior to humans on rugged terrain.)
Regarding your specific example of chess, your timeline of chess history is somewhat inaccurate, and the claim that “the best chess programs are still not always beating grandmasters” is false. The last match between a top-tier grandmaster, Michael Adams, and a top-tier specialized chess computer was played in 2005, and it ended with such humiliation for the human that no grandmaster has dared to challenge the truly best computers ever since. The following year, the world champion Kramnik failed to win a single game against a program running on an off-the-shelf four-processor box. Nowadays, the best any human could hope for is a draw achieved by utterly timid play, even against a $500 laptop, and grandmasters are starting to lose games against computers even in handicap matches where they enjoy initial advantages that are considered a sure win at master level and above.
Top-tier grandmasters could still reliably beat computers all until early-to-mid nineties, and the period of rough equivalence between top grandmasters and top computers lasted for only a few years—from the development of Deep Blue in 1996 to sometime in the early 2000s. And even then, the differences between human and machine skills were very great in different aspects of the game—computers were far better in tactical calculations, but inferior in long-term positional strategy, so there was never any true equivalence.
So, on the whole, I’d say that the history of computer chess confirms the stated rule.
Thanks for the information.
Does anything interesting happen when top chess programs play against each other?
Is work being done on humans using chess programs as aids during games?
One interesting observation is that games between powerful computers are drawn significantly less often than between grandmasters. This seems to falsify the previously widespread belief that grandmasters draw games so often because of flawless play that leaves the opponent no chance for winning; rather, it seems like they miss important winning strategies.
Yes, it’s called “advanced chess.”
My impression is that draws can still occasionally occur against grandmasters. Your point about handicaps is a very good one.
That’s another good point. However, it does get into the question of what we mean by equivalent and what metric you are using. Almost all technologies (not just computer technologies) accomplish their goals in a way that is very different than how humans do. That means that until the technology is very good there will almost certainly be a handful of differences between what the human does well and what the computer does well.
It seems in the context of the original conversation, whether the usual pattern of technological advancement is evidence against Clippy’s narrative, the relevant era to compare Clippy to in this context would be the long period where computers could beat the vast majority of chess players but sitll sometimes lost to grandmasters. That period lasted from the late 1970s to a bit over 2000. By analogy, Clippy would be in the period where it is smarter than most humans (I think we’d tentatively agree that that appears to be the case) but not so smart as to be of vastly more intelligent than humans. Using the Chess example, that period of time could plausibly last quite some time.
Also, Clippy’s intelligence may be limited in what areas it can handle.There’s a natural plateau for the natural language problem in that once it is solved that specific aspect won’t see substantial advancement from casual conversation. (There’s also a relevant post that I can’t seem to find where Eliezer discussed the difficulty of evaluating the intelligence of people that are much smarter than you.) If that’s the case, then Clippy is plausibly at the level where it can handle most forms of basic communication but hasn’t handled other levels of human processing to the point where it has generally become even with the smartest humans. For example, there’s evidence for this in that Clippy has occasionally made errors of reasoning and has demonstrated that it has a very naive understanding of human social interaction protocols.
And I can get a draw (more than occasionally) against computer programs I have almost no hope of ever winning against. Draws are easy if you do not try to win.
From what I know, at grandmaster level, it is generally considered to be within the white player’s power to force the game into a dead-end drawn position, leaving the black no sensible alternative at any step. This is normally considered cowardly play, but it’s probably the only way a human could hope for even a draw against a top computer these days.
With black pieces, I doubt that even the most timid play would help against a computer with an extensive opening book, programmed to steer the game into maximally complicated and uncertain positions at every step. (I wonder if anyone has looked at the possibility of teaching computers Mikhail Tal-style anti-human play, where they would, instead of calculating the most sound and foolproof moves, steer the game into mind-boggling tactical complications where humans would get completely lost?) In any case, I am sure that taking any initiative would be a suicidal move against a computer these days.
(Well, there is always a very tiny chance that the computer might blunder.)