Testing Intelligence
In my model of human behavior, there is an unobservable parameter I associate with intelligence. I observe people’s behavior when playing a game, when solving a problem, when defending their beliefs, or when learning something new, and I infer something about their intelligence. This in turn informs my predictions about their success in a wide variety of pursuits. In practice, I often make strong predictions about someone’s intelligence after observing their behavior on a single occasion.
An accurate conception of intelligence seems to be generally important. Understanding what easily leveraged factors affect someone’s intelligence—during childhood, later education, and after formal education is complete—is important if your goal is improving intelligence generally. If you are considering relatively expensive personal engagement to develop rationality, you may want to direct efforts at individuals who have the potential to have a significant impact as researchers or entrepreneurs. And so on.
Before thinking about how to understand determiners of intelligence, how to measure intelligence effectively, or the effects of intelligence on behavior, I would first like to get a feel for what my intuitive understanding of intelligence really corresponds to, if anything. It is possible that my intuitive assessments of intelligence are largely unrelated to reality, and that my beliefs about the world could be improved by discarding them. It is also possible that some of my intuitions about intelligence are quite accurate, and I could make better decisions by giving them more credence or by changing the way I use those intuitive judgments.
Intuitively, I expect the results of many types of otherwise apparently unrelated tests to be very tightly correlated with intelligence. To understand the extent to which this intuition is correct, I am considering conducting a slightly systematic study of the relationship between different metrics. I would appreciate pointers to reliable scholarship surrounding this question, but a brief search turned up mostly very muddled thinking and a general lack of people doing good experiments.
Here is a range of metrics which I suspect correlate well with my conception of intelligence, at least in certain regimes (some of these metrics may only correlate meaningfully when applied to very bright subjects, or may not correlate meaningfully when applied to very bright subjects):
1. General intelligence factor as estimated by standardized cognitive tests, e.g. Raven’s Progressive Matrices.
2. Ability to quickly learn an unfamiliar formalism. For example, to quickly learn a new game and to understand simple strategic consequences of its rules.
3. Ability to infer an underlying model. For example, to learn how to achieve a goal when allowed some constrained interaction with / observation of an unknown environment.
4. My assessment of intelligence during collaboration or discussion of a complex but rigorously defined topic; or, the assessment of anyone who I consider to be intelligent.
5. Ability to solve hard problems in a well-understood environment, potentially given hours or days. For example, performance in high school olympiads.
My hope is that by gaining a better understanding of the relationship between these metrics I may learn to what extent my current rather monolithic conception of intelligence is valid and, to whatever extent it is, how to effectively measure it. Ultimately I would like to understand what easy measurements are the best indicators of success at various particular pursuits, but is even more extraordinarily difficult to acquire data about how good someone is at, say, choosing good research problems.
What do readers expect the results of inquiry to look like? Is my choice of metrics influenced unduly by my own experience? What are other metrics I should be considering but am not? Is improving a student’s ability to perform any of these tasks likely to have a positive influence on other tasks?
According to this paper, those skills that most highly correlate with g are those with the lowest environmental variance. Working memory being the best illustration of this, its correlation so high that some researchers want to equate it to g.
According to this paper, genetic variance in intelligence is maintained by mutation-selection balance. This means it is a quantitative trait with a large number of tiny genetic factors influencing its overall value, making it a good fitness indicator. Hence we can think of intelligence as overall mental condition/health. It is unlikely that intelligence has any one underlying cause or mechanism, or even a few with large influence.
So you have two strategies for a good measure of intelligence, tasks with low environmental variance, tasks which tap diverse mental skills. Pretty much what the existing various IQ tests have set out to do.
As for success in various pursuits, I say rely on your overall assessment of the intelligence of the person. Of course, don’t forget creativity, discipline, drive, etc. which can be equally important. Beyond this, you’d have to go into the specific details of the particular pursuit, perhaps it requires specialized mental skills, quirky psychological profiles etc.
As for training intelligence, forget it. Even transfer of learning doesn’t work. You are best if you focused your training on specific skills integral to the tasks involved in achieving your goals.
Why do you believe this? People have tried and failed to understand how to train intelligence, but (in my estimation) more intelligent people have tried much harder to understand how to build a brain. Would you similarly say “As for building an artificial intelligence, forget it.”?
Well, what is training? Systematic repeated exposure right? And what this is supposed to do is to wire the brain in a certain way. But that first paper also suggests intelligence is something like synaptic plasticity, i.e. ability to learn. There just isn’t a mechanism which via training can improve synaptic plasticity.
I don’t mean give up on it long term, with future understanding we can certainly find a way to improve our own intelligence (but probably not via training). So I don’t see why I should say the same for AI or cognitive science.
I am not yet convinced that (what I think of as) intelligence is fixed by biochemistry.
I have not encountered strong evidence for this assertion. Why do you believe the suggestion of the first paper so strongly? From the inside, it feels as though the ideas controlling my thought are also a very important determiner of my ability to learn, solve problems, etc. I don’t trust my own impression of the way my brain works too much, but I would like some actual evidence one way or the other.
Observing strong correlations between the IQ of twins, for example, is rather weak evidence. For one, samples are small enough that the observations are only relevant at much lower levels of intelligence than I am interested in (or where my inside view of my own cognition is relevant). For two, randomly chosen pairs of people grow up in fairly similar environments, and the problem is exacerbated when the people in question are biologically inclined to seek out even more similar environments.
Oh yes, one can certainly train oneself to think more efficiently/effectively/creatively/etc. But this is not the same as improving intelligence. Think of it as using better software, instead of improving the hardware. But you can certainly think of this as improving intelligence, if you will, but then do realize that what you are doing is training a few key cognitive processes that happen to be useful in many domains. Which is to say, you won’t automatically be better at other mental tasks that don’t happen to require such cognitive processes.
Theories other than intelligence-as-synaptic-plasticity also don’t seem to allow improvement via training. This is because most of them hypothesize intelligence has something to do with the hardware of the brain. This is because the more diverse tests one aggregates, the more correlated the aggregated measure is with g. This together with the fact that tasks with high environmental variance have higher correlation with g, suggests that what aggregation does is cancel out environmental factors. This in turn strongly suggests that our notion of intelligence, or impression of someone’s intelligence, depends on a person’s overall mental ability over a wide range of tasks. We wouldn’t be impressed with a person who could multiply ten digit numbers if she does not also excel at a wide range of other mental tasks.
This is not to say that to be more intelligent, one has be better at everything. Because then why care for intelligence? One shouldn’t be too impressed with intelligence, because the whole point is to accomplish specific intellectual tasks no? Hence my suggestion in the first paragraph to identify cognitive processes influential in the performance of the intellectual tasks you care about.
Note that intelligence is a fitness indicator. We know this from psychological studies of sexual attraction and intelligence, from the fact that g has high genetic variance, from the fact that we haven’t found any genes which influences intelligence significantly. It is too easy to be impressed by intelligence and think that it can solve just about anything, without the training in the relevant intellectual tasks to go with it.
paulfchristiano:
That was my impression too when I tried making some sense of this area. Nevertheless, based on the literature I’ve seen, I think one can reliably say the following about your five items:
Raven’s and similar tests are definitely not the gold standard for pure g measurement that they were once thought to be. The Flynn effect has had the largest magnitude exactly on this sort of tests, and people can be trained to improve their scores on them significantly. (Though proponents if IQ would of course claim that this ruins their predictive validity.)
My impression is that this would be a very g-intensive task, possibly the most g-intensive sort of task at all. A really interesting evaluation of the Flynn effect would be to see how much it has affected people’s performance on tasks of this sort, but I’m not familiar with any literature addressing this question.
This sort of task may involve a bunch of other abilities largely unrelated to intelligence, depending on the nature of the problem. To take the most important example, if the problem requires figuring out the thoughts and motivations of other people, someone with an extremely high general intelligence but slightly autistic will likely perform worse than average. If the problem is completely formalized and symbolic, I’d say it’s little different from (2).
This is tricky. People of mediocre or even low intelligence but with great charisma and self-presentation skills can be surprisingly capable of fooling others into thinking they’re much smarter than they really are. Even if the interaction is purely about some formal and logically rigorous issue, your subjective impression may end up being much more favorable than if you applied a predefined set of formal criteria for evaluation.
This is about intelligence as well as conscientiousness. I don’t know what’s the correlation between these in the general population (and I doubt anyone knows precisely), but it’s certainly above zero. On the other hand, there are definitely people with one much better than the other. Which is more important depends on how novel and tough the problems are, how tiresome and tedious the tasks are, and how much time there is for preparation. For example, someone of mediocre intelligence can ace a math exam by working through a whole thick problem book beforehand, but this would not work for a math olympiad.
Take all this with the disclaimer that I’m just an amateur in this area, though I have read a fair bit of research literature in it at one point.
Meditating on my conversation with paulfchristiano below, I realize that our intuitive conception of intelligence is probably not coherent. The following is quite a controversial point, but I think a lot of what counts as intelligence was under sexual selection.
This paper shows how the best explanation for the genetic variance underlying intelligence is mutation-selection balance, instead of selective neutrality or balancing selection. Traits under mutation selection balance have high mutational target size, i.e. rare mutations of significant effect all over the genome affect their expression. This makes such a trait a very good fitness indicator, as it tells you the mutation load of an individual. Hence if you see an intelligent person, you can be reasonably certain that the person has fewer deleterious mutations. Physical beauty is another such fitness indicator.
Now, using physical beauty as an analogy, let’s say the symmetry of the face correlates with genetic quality. Then it will become a fitness indicator, as the opposite sex benefits from knowing the genetic quality of potential mates. This they experience as ‘beauty’. Now, symmetry of breasts also happen to correlate with genetic quality. How does natural selection make them appreciate this? Why since we already have a conception of beauty why not go with that? So facial and breast symmetry both fall under ‘beauty’ even though phenotypically they don’t really have much to do with each other. For one, they serve very different functions.
We should expect the same for our intuitive appraisal of the intelligence of others. Diverse mental tasks having no intrinsic relation to one another, happen to correlate with genetic quality, hence natural selection makes us perceive their aggregate as ‘intelligence’.
Buyer beware!
Hmm, doesn’t your theory predict that we should file these traits under ‘beauty’ instead?
(4) doesn’t seem useful if you’re trying to understand what you mean by intelligence anyhow. Tautology is tautology.
I mostly learn about the parts of how I define intelligence by meeting people who have more of one part or another. I have never met anyone who has lots of (2) but not (3) or vice versa.
My list would be:
Ability to make connections between known information, to see the relevance of new information or make analogies between field of knowledge.
Ability to learn quickly, to absorb and retain information well.
Ability to quickly and reliably solve problems that are straightfoward. A sort of depth and thoroughness of knowledge and application.
Insert the “7 types of intelligence” here. Various cognitive skills, gotten by a mix of talent and practice, are a sort of intelligence, but are not meta-skills like the first three.
I’m trying to understand whether my intuitive conception of intelligence corresponds to anything. One of my main sources of data is talking to people, so minimally it would be nice to know how much predictive power this data actually has.
Oh, derp, forgot your post was called “testing intelligence,” not “parts of the definition of intelligence.”
:D
Forecasting. Your 3rd point is part of the way there… but forecasting is easy to measure—and is a huge chunk of intelligence—see: http://prize.hutter1.net/
My expectation is that all of the 5 skills you list are strongly correlated to each other and to Spearman’s ‘g’. However:
Some of the items listed are (at least partially) “learned skills”, on which performance can be improved by ‘practice’. So differing educational backgrounds may obscure the correlation.
Many of the skills decline with age. But they may decline at different rates. So mixed ages may obscure the correlation.
Why shouldn’t we expect intelligence to improve with practice and wither with disuse? Intuitively, almost everything else seems to work that way.
I have no problem with intelligence changing. But if someone claims to be able to measure intelligence in any of five different ways, then each of those five metrics should change in lockstep. My point was that they might not.
Which in particular do you think are learned or will decay?
(2) and (3) seem to be very hard to teach without significantly contributing to what (I intuitively consider) overall intelligence. This is also true of (5) to a certain extent, and I certainly have anecdotal evidence suggesting that teaching (5) significantly improves performance at (1)-(4).
(1) and (3) are (in part) learned skills, and I don’t think that the learning involved transfers to performance on the remaining four.
I believe that my ability to do (2) and (5) has declined with age, though I have not declined much at (1) and (3) and perhaps not at all on (4).
I find that intelligence is positively correlated with the amount one spends thinking about intelligence.
Does this mean I can become more intelligent by spending more time thinking about intelligence?