I would also like to see this discussion. It isn’t terribly clear to me why the extinction of the human race and its replacement with some non-human AI is an inherently bad outcome. Why keep around and devote resources to human beings, who at best can be seen as sort of a prototype of true intelligence, since that’s not really what they’re designed for?
While imagining our extinction at the hands of our robot overlords seems unpleasant, if you imagine a gradual cyborg evolution to a post-human world, that seems scary, but not morally objectionable. Besides the Ship of Theseus, what’s the difference?
So I’m fairly new to LessWrong, and have being going through some of the older posts, and I had some questions. Since commenting on 4 year old posts was probably unlikely to answer those questions or to generate any new discussion, I thought posting here might be more appropriate. If this is not proper community etiquette, I’m happy to be corrected.
Specifically, I’m trying to evaluate how I understand and feel about this post: The Level Above Mine
I have some very mixed feelings on this post, and the subject in general. (You might say I’ve noticed that I’m confused.) Sure. It’s hard to evaluate reliably just how intelligent someone who is more intelligent than you is, just like a test that every student in a class aces doesn’t allow you to identify which student knows the information the best, but doesn’t the idea of a persistent ranking system, and the concern with it imply a belief in intelligence as a static factor? Less Wrong is a diverse community, but I was by and large under the impression that it was biased towards a growth mindset. Indeed, it seems in many ways the raison d’etre of LW relies on the assumption that it is possible to improve your intelligence. I would further argue that LW relies on the assumption that it is possible to recursively improve your intelligence, (i.e. learning things that help you learn better).
Is it possible that the fundamental attribution error is at work here? I mean, if it’s ridiculous to believe in “mutants born with unnaturally high anger levels” then why the rush to believe in mutants with unnaturally high levels of intelligence? I’m not sure what to make of a post that discusses assessing how many standard deviations above average intelligence someone is, if I really believe that “Any given aspect of someone’s disposition is probably not very far from average. To suggest otherwise is to shoulder a burden of improbability.”
Indeed if we make fundamental attribution error when assessing someone because “we don’t see their past history trailing behind them in the air”, then can we not say the same for experiences that result in greater situational intelligence? Perhaps I’m straining the bounds of metaphor slightly, since problem-solving intelligence tends to be more enduring than vending-machine kicking anger, but is it so fixed that my SAT scores from the 7th grade are meaningful or worth discussing? Is it possible that what we perceive as greater intelligence, as “the level above mine” is just someone who has spent more time working on something, or working on something similar to it? What is the prior probability that someone picks up a new idea quickly because they’ve been exposed to a similar idea before, versus the prior probability that they are of mutant intelligence?
The entire ranking debate to me, sounds suspiciously like human social hiearchies, and since that’s a type of irrationality humans are especially prone to, it makes me very suspicious. I know from personal experience, that being considered of “above average intelligence” is a very useful social tool which I can use to create a place for myself in social hierarchies, and often that place is not only secure, but also grants me reasonably high social status. I have at various times in my life, evaluated others, and granted social status accordingly, on the basis of their SAT scores and other similar measures. Is that what is going on here?
Fundamentally, I believe this question boils down to a handful of related questions:
How accurate over time is our evaluation of general intelligence?
Does our love of static hiearchies, esp. one that priveleges intelligence affect our answer to 1?
Sub-questions to #1
a. How varaible is intelligence, and over what time span? Or more generally, what do we estimate are the most heavily weighted inputs to a function that describes intelligence?
b. Is there an upper bound on human intelligence?
c. Are the people whose intelligence we’re evaluating operating near that bound?
d. Can we reliably distinguish between intelligence and knowledge? How?
I’m not sure about question 1, but I’m pretty sure the answer to question 2 is yes.