Negative and Positive Selection
(Originally posted to my blog, The Rationalist Conspiracy; cross-posted here on request of Lukeprog.)
You’re the captain of a team, and you want to select really good players. How do you do it?
One way is through what I call positive selection. You devise a test – say, who can run the fastest – and pick the people who do best. If you want to be really strict, like if you’re selecting for the Olympics, you only pick the top fraction of a percent. If you’re a player, and you want to get selected, you have to train to do better on the test.
The opposite method is negative selection. Instead of one test to pick out winners, you design many tests to pick out losers. You test, say, who can’t run very well when it’s hot out, and get rid of them. Then you test who can’t run very well when it’s cold out, and get rid of them. Then you test running in the rain, and get rid of the losers there. And so on and so forth. When you’re strict with negative selection, you have lots and lots of tests, so that it’s very hard for any one person to pass through all the filters.
I think a big part of where American society’s gone wrong over the last hundred years is the ubiquitous use of negative selection over positive selection. (Athletics is one of the only exceptions. It’s apparently so important that people really care about performance – as opposed to, say, in medicine, where we exclude brilliant doctors if they don’t have the stamina to work ninety hours a week.) A single test can always be flawed; for example, IQ tests and SATs have many flaws. However, with negative selection, how badly you do is determined by the failure rate of every test combined. If you have twenty tests, and even one of them is so flawed it excludes good players, then your team will suck.
Elite college admissions is an example of a negative selection test. There’s no one way you can do really, really well, and thereby be admitted to Harvard. Instead, you have to pass a bunch of different selection filters: Are your SATs good enough? Are your grades good enough? Is your essay good enough? Are your extracurriculars good enough? Are your recommendations good enough? Failure on any one step usually means not getting admitted. And as competition has intensified, colleges have added more and more filters, like the supplemental applications top schools now require (in addition to the Common Application). It wasn’t always this way – Harvard used to admit primarily based on an entrance exam – until they discovered this let too many Jews in (no, seriously). More recently, the negative selection has been intensified by eliminating the SAT’s high ceiling.
Academia is another example of negative selection. To get tenure, first you have to get into a top PhD program. Then you have to graduate. Then you have to get a good recommendation from your advisor. Then you have to get a good postdoc. Then you have to get another good postdoc. Then you have to get a good assistant professorship. Then you have to get approved by the tenure committee. For the most part, if even one of those steps goes wrong – if you went to a second-tier PhD program, say – there’s no way to recover. Once you’re off the “track”, you’re off, and there’s no getting back on. It’s fail once, fail forever.
Grades are another example – A is a good grade, but there’s no excellent grade. There’s no grade that you only get if you’re in the top 0.1%. Hence, getting a really good GPA doesn’t mean excelling, so much as it means never failing. If you’re in high school and are taking six classes, if you fail one, your GPA is now 3.3 or less, regardless of how good you are otherwise.
In any field, at the top end, you tend to get a lot of variance. (Insert tales of the mad artist and mad mathematician.) Negative selection suppresses variance, by eliminating many of the dimensions on which people vary. Students at Yale are, for the most part, all strikingly similar – same socioeconomic class, same interests, same pursuits, same life goals, even the same style of dress. A lot of people tend to assume performance follows a bell curve, but in some cases, it’s more like a Pareto distribution: the top people do hundreds or thousands of times better than average. Hence, if you eliminate the small fraction of people at the very top, your performance is hosed. Fortunately for VC funds, the startup world is still positive selection.
Less obviously, a world with lots of negative selection might be a nasty one to live in. If you think of yourself as trying to eliminate bad, rather than encourage good, you start operating on the purity vs. contamination moral axis. Any tiny amount of bad, anywhere, must be gotten rid of, and that can lead to all sorts of nastiness. “When you are a Guardian of the Truth, all you can do is try to stave off the inevitable slide into entropy by zapping anything that departs from the Truth. If there’s some way to pump against entropy, generate new true beliefs along with a little waste heat, that same pump can keep the truth alive without secret police.”
I asked my father to read this and give his thoughts.
He says that positive selection only works well when you have a very good idea what you need to select for. If you’re sending an athlete to the Olympics but the event he’ll have to compete in will be chosen at random, you can’t just choose the one with the best time on the 800 meter dash, because the event might end up being something like archery, fencing, or weightlifting. And you certainly wouldn’t want to send a non-swimmer. If you need a generalist, seeing how well someone does at jumping through a wide variety of arbitrary hoops might really be the best test you can practically implement.
(Now I’m wondering just how good or bad the 800 meter dash actually is at predicting levels of success at unrelated sports. For example, could you tell the difference between an NHL-quality ice hockey player and one that plays on a minor league team just by looking at their times on the 800 meter dash?)
Assuming a significantly large distribution of athletes sent by other rational managers, where all athletes are bound to the same rules of random event selection, I would still send the best possible specialist in a single discipline in this case, because without certainty that all other rational managers know certainly that some generalists will be better in everything than other generalists and that each one is confident that theirs is best, I conclude that some of them attempt a gamble of probabilities and send a specialist, and thus I also send a specialist to maximize my chances of winning.
After all, there are higher chances of the event being my athlete’s specialty than there are chances of every single other athlete being less good at it if I pick a generalist, unless the number of possible events is large enough to outweigh the number of athletes. Throw in irrational managers and the possibility of other managers having information unavailable to you, and your father’s argument seems very weak.
Now, of course, I’m probably attacking something that wasn’t meant to be a strong defensible argument. However, I feel very strongly about the point that negative selection is wrong in many contexts it is currently used in (which I support), as well as the point that positive selection is so difficult and utterly impractical in so many cases (which I want to pound into tiny bits of forgotten wrongness).
I’m not sure where I’m going with this, however. I strongly agree with the article’s statements, but my attempts to formulate any further useful thought seem to come up short.
Well, the sports analogy was my own interpretation of what he said.
Game theory question time: you and N other players are playing a dice rolling game. Each player has the choice of rolling a single twenty-sided die, or rolling five four-sided dice. The player with the highest total wins. (Ties are broken by eliminating all non-tying players and then playing again.) Now, rolling 5d4 has an expected score of 12.5 and rolling 1d20 has an expected score of 10.5, so when N=2, it’s obviously better to roll 5d4. However, when N becomes sufficiently large, someone is going to roll a 20, so it’s better to pick the 20-sided die, which gives you a 1 in 20 chance of rolling a 20 instead of a 1 in 1024 chance of getting five 4s. For exactly what value of N does it become better?
Edit: Fixed stupid math mistakes. That’ll teach me to post after staying up all night!
Fixed, thanks.
4^5 = 2^10 = 1024
Fixed, thanks.
Insightful question, if you ask me, though solving for N feels a lot more like a straight up actuary-level math problem than Game Theory to me. My maths above basic calculus is generally foggy, so I’d appreciate any corrections or nitpicks someone more fluent here might have.
Essentially, you have to solve when (odds of having highest result when rolling d20) >= (odds of having highest result when rolling 5d4). To simplify, let’s assume that all players are perfectly rational, and thus at N and higher will all roll 1d20. This still leaves you the problem of calculating N’s odds of rolling higher than you for both rolls, which is a simpler reformulation of the above parentheses.
For any roll result Y, there is (y/20)^N probability that you “win” here, assuming ties count as wins (or at least are preferable to losses). This means that with N=1 (you’re playing against one other person), you will win 52.5% of the time (and so will your opponent, because that 2.5% is for ties) when rolling 1d20.
Your odds of winning naturally decrease if you roll 1d20 such that for N=2 you have 35.875% chances of winning, and so on in a proportional manner since the odds are always even for everyone.
Where it gets more interesting is when you are playing an unfair game where you have to equate your total odds of winning when playing 1d20 vs d20s to those when playing 5d4 vs d20s. Since the math here is kind of foggy and hard to combine into one big formula, I’ve thrown the data at a spreadsheet (to calculate the sum of the odds of any N rolling higher than you for each roll Y multiplied by your odds of obtaining Y), and it turns out that at N=3 the 5d4 roll dips just below the odds of winning with 1d20 by about 0.2%.
However, if we want to compute for xDf die for N, with K possible ways to roll (which was 2 here), then the math yet eludes me. I’ve figured it out or been told what it was several times, but I just can’t seem to ever memorize this when I can only barely remember integration anyway when I don’t use it.
Edit: For those curious, here’s the spreadsheet mentioned above with all the raw data and brute-force formulas.
Your analysis also assumes there’s no difference between second place and last place.
Yes, the reward system is very important in choosing the right strategy. If the first place gives you gold, and all other places give you nothing, use positive selection. If the last places gives you problem, and all other places give you nothing, use negative selected. Other point of view: if being average is good, play safe by using negative selection; if being average is bad, aim for greatness (and accept a certain risk of failure) by using positive selection.
So the question is what exactly do we want in elite colleges or academia (examples from the article)? I guess for elite colleges it is better to play safe. If your students are above average and everyone knows it, they don’t have to be exceptional—your diploma will help them get a decent job, which is why they pay you. A few bad apples could ruin your marketing. With academia, for an average university it is probably better to have “safe” professors who do their jobs, get grants, and don’t cause scandals; even if the price is having less Nobel-price winners.
Yes, that it does, or at least it assumes that the difference is trivial within this decision scheme and the expected utility returns of a specialist are higher than the expected utility of a generalist even when taking second place into account.
I don’t think the right way to do this is not either positive or negative selection (those terms really suggest a false dichotomy, don’t they?). As has been pointed out elsewhere, what’s here being called “positive” is really “or”, and what’s here being called “negative” is really “and”.
But there are lots more ways to combine the data into a single number then just “apply a cutoff to each one, and then apply some operation to the resulting booleans”. The appropriate sort of selection is not positive or negative, but rather, whatever will be used in the actual competition. (And if it’s unknown, apply expected utility, etc.)
This (among other prior information) suggests to me that extreme levels of performance at different tests are probably negatively correlated, but I would not be surprised if there were events out there where extreme levels of performance on other tests are correlated with better (but not extreme) levels of performance on that test.
Unless you sample at random from the whole population, that’s a Berkson’s fallacy.
Thanks for the link! I think a careful statement of my claim avoids that fallacy.
The claimed data:
Body shape is relevant to performance in particular events (say, the 800 meter dash)- for example, Michael Phelps is shaped well for swimming.
Extreme performance in that event will generally require a body shape optimized for that event.
Extreme performance in different events correspond to different body shapes.
Stated carelessly, it seems likely to me that if I know you’re an Olympic-level athlete, I’ll have some estimate that you can play Olympic-level basketball; but then when you tell me that you’re a Olympic-level wrestler, I can lower my probability estimate that you would be able to play Olympic-level basketball.
But if I condition on knowing you’re an Olympic athlete, and then try to drop that condition without being careful, then I can get into trouble (this fallacy, specifically). So instead, let’s start off with some (really low) probability that a person chosen uniformly at random can play Olympic-level basketball, and then update on the knowledge that they’re an Olympic wrestler- we should increase our estimate based on their general level of athleticism, and then decrease our estimate based on their probable body shape. I think the effect of the body shape will be stronger than the effect of general athleticism, and so they will actually be negatively correlated.
I think my last statement in the grandparent- that extreme levels of performance on a specific test should correlate with better performance on some ‘general athleticism’ test- is true when you compare extreme individuals to random individuals, but less true (or perhaps not true) when comparing extreme individuals to good individuals. The NHL ice hockey player is probably not much more ‘athletic’ than a minor league hockey player, but probably is more ‘hockey-shaped.’
But now that I’ve stated that last paragraph, I’m thinking of counterevidence- like the famous birth month effect for Canadian hockey players. Early training appears to have a huge impact on whether or not someone hits extreme levels of performance, but I think both NHL players and minor league players started early. So maybe it is biological talent that differentiates many of them. Hmm. I should probably stop speculating about sports.
If you put an Olympic-level wrestler on a college (American) football team, how well would they do? Michael Jordan did tolerably well during his year as a minor league baseball player.
Tolerably well for minor league, but remember that his father had envisioned him as a major league baseball player, so presumably he’d practiced and done well at the sport when he was younger. There are probably selection effects on Michael Jordan in particular to be good at baseball out of the set of NBA players; most never try to transition to baseball at all.
Upvoted for this link. I wish I had known this term back in the Amanda Knox days—this fallacy (or rather, the reverse of it—failing to take into account conditional dependence of a priori independent events) is a version of the main probability-theoretic error of that case.
This fallacy may also explain why people tend to assign 50% percent probability to the odds of the second child being a boy in the classic puzzle.
One of the most important social structures of modern society is the corporation—a framework for large groups of people to band together and get absolutely huge projects done. In this framework, the structure itself is more important than individual excellence at most levels. To a lesser extent, the same applies to academia and even “society as a whole”.
In that context, I think preferring negative selection to positive makes sense: a genius data-entry clerk is less helpful than an insubordinate data-entry clerk is disruptive.
And remember that we have side routes so real geniuses (of some kinds) can still make it: set up their own company, start their own political party, start publishing their work online, design games in their basement, and so on.
This is a really good point. It’s good to have low barriers to this sort of thing. For instance, if you need to hire a lawyer and an accountant to set up your own company, then a genius cookie baker can’t set up their own cookie shop unless they also have the money or connections to get the help of a lawyer and an accountant.
I’m not sure that it’s the corporate structure that makes negative selection more useful in the data entry case. It’s not the fact that the data-entry clerk is part of a large organisation that means that a slightly incompetent data-entry clerk is more disruptive than a genius-level one is helpful. Rather it’s the fact that data-entry is a relatively low skill job and with relatively little room for excelling above mere competence. Leaving the corporation wholly out of it, and imagining a person doing data entry in complete isolation, the most helpful data-entry clerk would still be selected by making sure they weren’t terrible, but weren’t necessarily brilliant, at typing and remaining attentive etc. I think this idea is supported by the fact that for higher level/skill positions, one probably would want to employ more positive selection.
If your point was specifically that insubordination (and not just slight incompetence in general) is more harmful than genius-level work is helpful, then I guess that, in an obvious sense, the harm of insubordination is due to the corporate nature of work (since you can’t be insubordinate outside of a group hierarchy). But then I’m not sure that insubordination-worries requires negative selection, or at least not a wide range of negative selection tests. Sure, you might want to include a negative selection test along the lines of ‘are they likely to do the opposite of what they’re told on a whim occasionally?’, but it’s an open question whether the rest of your criteria would be negative or positive.
The point I should have made clear was that data-entry clerks don’t exist outside of corporations, because in isolation they’re useless. More generally, mass production has been made possible by the production-line paradigm: break down the undertaking into tiny discrete jobs and assign a bunch of people to doing each one over and over again.
Once you get that kind of framework, exceptionally good workers aren’t very helpful, because the people to either side of them in the production line aren’t necessarily going to keep up. You just need to shut up and do your job, the same as everyone else.
At the high levels—the people wielding their collective underlings as a tool, rather than the people who are part of that tool—this obviously no longer works.
Important note: all of the above, including my original comment, is 100% psuedo-intellectual wank, since I’ve never been part of a corporation, never taken a business management course or seminar, and never conducted or read a study on the efficacy of various business practices.
This was my favorite post on your blog and I’m glad you posted it here.
I agree. I stumbled across this one a week ago or so—without knowing the author was associated with LW—loved it, and have been thinking about it off and off since. I’m glad to see it again. I feel like I should probably start reading your blog regularly..
Really good post...it makes a point that is completely new to me, which is always nice.
It does occur to me that the current (negative selection) system would reward “hard work” more, relative to “talent”, than a positive selection system. (In quotation marks because those are both metrics that are hard to measure separately from one another.) Someone who is very conscientious and hard-working is likely to compensate for wherever areas they’re weaker, in terms of “natural talent”, however you define that.
My first, emotional reaction to your post was “I would be screwed in a positive selection system!” As someone who’s above average in a lot of areas, not really exceptional in any, and obsessively hard-working enough that it’s a running joke among my friends, I like the current system just fine (although I’m not in academia.) I don’t know if conscientiousness would have a bigger long-term effect on results than innate brilliance; it probably depends on what field you’re talking about.
My intuition says that a positive selection system would probably be a good idea in fields where there is big variance in natural ability, i.e. math or physics, and less so in fields like medicine where a lot of “talent” depends on how willing you are to work hard and keep improving over your whole career.
Negative selection may be good, actually, for the vast majority of people who are ultimately going to be mediocre.
It seems like it may hurt the occasional genius… but then again, there are a lot more people who think they are geniuses than really are geniuses.
My first reaction was pretty much identicle, right now you can do well at almost anything purely based on conscientiousness, including video games, work, school, and social interaction. I don’t know of any good way to measure general talent, but when I learn most things I tend to be quite bad at them until I enter tsukoku naritai mode. Perhaps this should influence my career decision somewhat, its hard to tell if talent or effort is more crucial for programming.
Effort. Always assume effort. Talent will speed up the learning process in the early stages, is likely to make effort easier (because it is more fun) and at the extreme upper ends of of performance probably gives a higher limit. But in general effort plus social politics skill will determine your career success.
Despite what they are taught likely to be about themselves, what they might think of themselves, and what western culture expects of them, programmers are more creative artists than analytic engineers.
The difference is most tangible from the management perspective since motivating programmers is less like motivating chemical, mechanical, or any other sort of engineer and more like motivating commercial artists with less pretense, who were never led to believe they were meant for something greater. Dissatisfaction from programmers grows in much the same way it grows in commercial artists as well, though they programmer is less likely to specifically identify his or her complaint and the artist is more likely to complain about having sold his or her soul.
Common responses to criticism of work among programmers align more with those among artists than those among engineers. Again, I learned this from a managerial perspective.
The most important advice that may be given to starting artists (excluding all the low-hanging fruit advice that is best for everyone in general, of course) isn’t about discovering your own inner talent or anything similar, instead it is about discipline: “Ideas are not swords you can brandish about in triumph. What matters most is the Sit Down, Shut Up And Get It Done. Only there will you find the true steel for your craft. Only there, will you know if you are worth the words out of your mouth.”
I would suggest talking to some programmers.
My intuition is that there’s something of innate talent involved in programming, so that you can divide people into two populations: those whose brain makeup causes them to find programming intuitive and fascinating and cool, and those to whom it just doesn’t make sense. If you’re considering it as a career, presumably you fall into the first category. Beyond that, I would guess that conscientiousness is the biggest predictor–my one-semester programming elective was enough to show me that it’s really time-consuming.
But I’m not a programmer by specialty. An unusual percentage of LWers are, though, so maybe someone can give you advice?
“The Camel Has Two Humps”, which IIRC has been linked here before, does purport to find a bimodal distribution between people who can and can’t program. I’m not at all sure if that has anything to do with inborn talent, though, at least beyond basic general intelligence.
At various points in my career I’ve found reasons to teach people programming skills, and my n=1 impression is that the ability to internalize basic programming has little to do with personality (though conscientiousness helps, and I suspect openness to experience might too) and a lot to do with the student’s level of comfort with mathematical thinking. Not necessarily advanced math (you don’t need anything more complicated than algebra to program except in specialized domains), but you do need to be very comfortable with a certain level of abstraction. I suspect that might have more to do with the distribution in the linked paper than the “geek gene” concepts I’ve heard tossed around elsewhere: at the level of the math prerequisites for CS 1 it’s still possible to do well by solving problems mechanistically without a good grasp of the abstractions involved, but that won’t cut it in computer science. And it’d probably be difficult to teach that in a semester.
Camel seems to fail replication: http://www.gwern.net/Notes#the-camel-has-two-humps
The thing that wasn’t replicated was their attempt at a predictive test of the distribution (based on a particular explanation they thought applied), not the existence of the distribution itself, which is something that was observed in grade patterns in CS compared to other subjects (though I don’t know how rigorously established it is).
Isn’t the predictive part the interesting thing? I wasn’t aware that bimodal grade distributions were unique to CS.
Well, their original paper claimed that (eg) math grades are typically a bell curve, whereas CS grades are typically bimodal (with examples from one university). But again, I’m not sure if this is something that’s been rigorously demonstrated.
Good to know. I thought it had a bit of a questionable odor to it, but I wasn’t able to find any replications in the brief time I spent looking into it.
I don’t think it has much to do with personality either, except, like you said, willingness to work hard (especially if you’re someone who starts out finding it very difficult.) But I think that a lot of people, even people who can work up to the level of calculus in math, go at it with the mindset of “memorize that Formula X gives Answer Y” instead of trying to understand how and why Formula X relates to the underlying structure of the problem so that it’s obvious that it should give answer Y, but gives Answer Z in a different context… You can get by with memorizing formulas in math classes, at least the way they’re currently taught and tested. It’s a lot harder to get by with that habit that when you’re programming.
(On the whole, the people I’ve known whose minds appear to work like this aren’t noticeably “lower” intelligence, however you define that. They just don’t think of math as something where they should be applying the analytic part of their mind.)
Your intuition relating to “innate talent” is wrong. There is nothing innate about any of the “talents” required for programming, other than what generally comes with the package of most human brains. This might be a simple question of wording though, as if you change the words to “inner” talent, I would be more inclined to agree.
Simply put, unless a person is simply incapable of mental change and learning, then they will be able to “learn” and self-adjust into “obtaining” the “talents” required. However, this usually requires much more effort than people (read: the small sample size of the people I know who are not programmers but have attempted to learn how to program, which amounts to eight individuals) are willing to put forth, hence the common misconception that there is this “innate talent” that you simply must have to become a programmer.
I was using ‘innate’ means “something present in a person’s brain and/or skill-set when they start trying to learn how to program.” It might have something to do with how open-minded they’ve been in the past to learning new ideas, because if they’ve been open to that, then they’ll have a wider base of knowledge and practice thinking about problems at a certain level of abstraction. I don’t think it’s necessarily innate as in “determinable at birth”–in fact, that seems really unlikely to me, but what I know about the subject doesn’t allow me to distinguish those possibilities. (The phrase ‘inner talent’ is one I’ve never heard used before and would not have thought to use, so I don’t know exactly what area it would cover.)
Do you agree that some people will start learning programming and find it easy, intuitive, and immediately fun, and not have to put in a lot of conscious effort, whereas others will need to lean much more on their capacity for mental change and learning? This is what I’m talking about.
Yes, on that I agree. I suppose I was more disagreeing on the choice of word than on the concept of something being already there in some and lacking in others. The dictionary definitions (referring to Dictionary.com and my old pocket dict) of “innate” all seem utterly inappropriate for this usage.
However, I’ll still nitpick on the point of conscious effort. My “definition” of inner talent is that of an abstract representation of the “source” of the talent in question. An “outer” talent is one where, to explain by example, a person’s genetic profile is directly favorable to athletics by producing the required muscle mass more efficiently with less prodding, and recovering from exercising damage more easily, and so on. By contrast, an “inner” talent is one where synergies, “affinities” in the system, side-routes, or other indirect or invisible. I always fail to find the words to explain complex dynamics where various seemingly-unrelated things converge to the same location to push in the same direction, but that’s about the kind of psychological or physical events I’m trying to refer to with “inner” talents.
Some person will have no particular skill or strength that is directly beneficial towards chopping wood, but once they try, suddenly a bunch of unrelated past experiences or other points about their current self help them catch on quickly to just the right way of holding the axe and establishing their footing and swinging and so on.
What I want to make a point for is that both the processes of awakening an inner talent or slowly going through all the steps from nothing can be either conscious or unconscious. This will depend on many factors that may not be obvious.
I agree that I probably shouldn’t have used the word ‘innate’; given the meanings people associate with it, it was more likely to confuse people than help. Maybe “prior talent” or something similar?
I think I understand the concept you’re trying to convey, but I find that youre’re using the words ‘inner’ and ‘outer’ in a very unconventional way that I’ve never heard before, which is also likely to confuse.
This is an untestable claim. How would you distinguish between your claim being wrong and a person not putting out enough effort?
I fail to see how it is untestable. It might be impractical to test, but if the survival of all humans suddenly depended on their future programming skills (e.g. because a dictatorial alien appears and decides to kill off any human they can’t use in their programming slave workshops), it would certainly be tested, as far as I can tell.
I also didn’t really define my usage of the word “effort” very clearly or unambiguously, which is intentional: I’m not in the best position to determine/figure out the exact line at which the scope of the term “effort” should be drawn for the statement to be solid.
The alternative, however, being that a person simply cannot learn and obtain talents, would imply that some individuals have a meta-talent for obtaining talents, as it’s been demonstrated before that people can (and often will in extreme situations or environments) adjust in such a manner. This, to me, seems much more complex than “Everything can be learned, including talents and how to obtain talents”, so by Occam’s Razor I prefer to believe the former.
I don’t know to what degree innate talent is important in programming ability. I tend to agree that most people would be capable of learning to lot of things that are generally thought to require innate talent.
However, I’m not sure that “effort” is the most important hurdle stopping the learning from happening. First, people may not even believe that they can learn—they go from finding something hard to understand to assuming that they aren’t the sort of person that is good at this. Secondly, they have to find out what to learn. In the case of programming an average set of lecture notes or a “teach yourself language X” probably won’t do the job. A really good textbook may—but it won’t quickly diagnose misunderstandings, which a good tutor will (I lectured and tutored computer science and programming for a number of years).
Then you need the effort.
Yes, I attempted to infer this when saying that it requires “much more effort”. First, they need to make an effort to be more aware of themselves, then to be able to learn and develop new mental models of their own accord on things of their choosing, in a conscious manner.
Then, they need to, as you say, find out what to learn. This can be achieved through the effort of looking for resources (including “human resources”) that can help you figure it out or give you the right information.
Notice that at every step, there is effort required. You might call it a “meta-effort”, in a sense, since you also need to make the effort of doing effort, but I like to simplify and just say that it’s a lot of effort that most people aren’t willing to make. After all, I’m usually explaining this to the very people who aren’t, in an ultimate play to get them to either start gearing up for a long trek into Better-Person-Hood or give up about becoming a programmer/researcher/etc.
If you define “talent” as a product of your current ability to produce and visualize mental models of complex systems, especially “from nothing”, then it is the most defining factor for the higher maximum awesomeness of programs you can code at present.
This “talent” can be enhanced and self-improved through effort, however, in a very similar manner to making oneself more “luminous”.
The concepts of positive and negative selection are not quite well defined in your essay, I think.
Imagine that you have one test, with a gaussian distribution of outcomes. Let’s arbitrarily set a threshold, and if people are above this threshold, they have passed this test. Call the sets of passing A and not passing ~A
Would you call this a positive or negative selection? It is neither, in my opinion.
Now, imagine you have two tests, A and B.
A positive test is one where A U B are selected. A negative test is one where ~(~A ^ ~B) are selected.
In other words—the operative difference between positive and negative selection is OR vs. AND.
Nice step toward clarity, but the leading example contrasts a single-test positive selection rule versus a multiple-test negative selection rule. I think it’s worth a try to capture that contrast.
Here’s a try. A selection process winnows down a pool of applicants (let’s call them) to a pool of winners. A step in a selection process, where the steps are applied in a given sequence, is “absolutely negative” if it removes less than half the remaining pool. A step in one process is “more negative” than another step (in the same or another process) if it accepts a larger fraction of the pool.
I’ve just realized that I have been treating dating as a negative selection process. This might explain the lack of success.
Well, you want some negative selection: Choose dating partners from among the set who are unlikely to steal your money, assault you, or otherwise ruin your life.
This is especially true for women, for whom the risk of being raped is considerably higher and obviously worth negative selecting against.
That carries the assumption that the qualities you’re positively selecting for don’t have a strong negative correlation with the ones you’re trying to select against. I don’t think it’s hard to lay out a few basic “are” qualities that imply “are not” for “violent, thief, etc.”
If A means not B, then selecting for A is the same thing as selecting against B.
If A means “with probability 90% not B”, then if B is a serious problem, it is worth checking both A and not B. Maybe even checking not B first, to avoid halo effect from A.
In my experience, some people treat dating as a negative selection process with thousand requirements that no one passes, because thousand criteria are simply too much. (Assuming independent results, even with probability 99% of passing each test, less than one person in 20 000 passes all thousand criteria. In real life, the criteria are often positively correlated, but on the other hand the probability is way less than 99%.) And those people usually defend it by taking each criterium out of the context and saying: “What’s wrong about wanting my boyfriend/girlfriend to be interested in opera/programming?” Well, nothing wrong per se, but if you have thousand criteria like this, good luck finding a person who fulfills them all (and is also interested in you).
The solution is to separate those criteria into two groups: “must have” and “nice to have”. (And if nine hundred of the thousand criteria are in the first group, you are doing it wrong.) First, filter people by the “must have” criteria. What remains is your dating pool. Some of those will be never interested in you, but you will find that out by trying. Now use the “nice to have” criteria for a utility function, and go seduce someone with a high utility. (And as a parallel process, try to increase your market value.) At the end, you may find someone who has all “must have” and some of the “nice to have” traits; and you may be happy with them.
If A means not B, then selecting for A is only the same thing as selecting against B IF A doesn’t also mean other things, besides not B.
In the dating example, a (straight) woman might employ positive selection to choose men who are particularly decent people. This would also have the effect of weeding out thieves and rapists (assuming that the woman in question can assess a man’s decency with sufficient accuracy), but the quality of “being a decent person” doesn’t only mean one isn’t a thief and a rapist; it’s more wide-ranging than that.
Doubtful. My romantic excitement, as best as I can tell, follows a positive selection process and it is highly inconvenient. (Basically, my dating history looks like I sat down at 16, compiled a list of people who fit a particular description, sorted them along that axis, and then have tried to date them.) Like Viliam_Bur points out, you don’t want to have 900 criteria which can disqualify someone- but you do want to have a reasonable number of thresholds that you don’t go below. At some point, the amount you dislike the other person will be determined by their lowest stat that you care about, and it would be nice to not have to deal with a lot of dislike.
It seems that Vaniver and pnrjulius have assumed that you’re having trouble picking good dates. If, instead, you are worried about getting picked (or accepted) for dates, then maybe you’re on to something. I’d be interested in knowing whether the majority of people accept dates based on a positive or a negative selection process. It may need to be broken down by gender.
(I have a hypothesis that I won’t share yet, in case it influences results)
Viliam_Bur describes my thought process correctly.
I’m faltering on the first step, finding a woman whom I would be interested in dating. I think part of this is due to what I now recognize to be too many criteria ruling out people who might otherwise be appealing. (I’ve certainly had people tell me I’m too picky before, but it took a comparison to undergraduate admissions for the underlying nature of the problem to become apparent.)
Worrying about getting picked or accepted is a different step entirely.
That definitely makes it clear what your intention is.
I’m male and (I think) I tend to apply negative selection when deciding.
Grades in high school are already like this. To get the best grade from a subject, you need to be good, but not exceptional. So being good but not exceptional in everything brings you the highest possible score.
If instead you are exceptional in a subject or two, and average in a few unrelated subjects, it gives you lower total score, and if the university cared about your grade average, you would have problems getting there, especially if many people with the highest total score competed with you.
I don’t think it’s quite true that “fail once, fail forever”, but the general point is valid that our selection process is too much about weeding-out rather than choosing the best. Also, academic doesn’t seem to be very good at the negative selection that would make sense, e.g. excluding people who are likely to commit fraud or who have fundamentally anti-scientific values. (Otherwise, how can you explain how Duane Gish made it through Berkeley?)
Since most production functions are quasiconcave over inputs, negative selection is a cheap method of increasing expected return. You lose some outliers and also people who would be good in those rare domains with quasiconvex production functions, but our system is optimized for the average case.
In the college admissions example, a top school wants to admit undergraduates likely to become successful doctors/lawyers/businesspeople and alumni donors, not gamble that the smart kid with a few Bs in high school is going to revolutionize a scientific field in 15 years. Even if they did, their undergraduate institution would be only the third most important on their CV, after their current institution and where they got their PhD.
This is a good example of individual selection being suboptimal from a group perspective. Each top school would prefer that some other top school gamble on said smart kid, and then if they have the chops for research they can try to grab them when they apply to graduate school or go on the academic job market. Positive selection on undergraduates is just not a smart strategy from an individual institution’s perspective since most undergraduates will be going into more conventional fields.
I agree with most of it, though the point about academia is a bit contrived.
True, there is a lot of negative selection before you get a cushy job the usual way, but you can certainly bypass quite a few obstacles if you are exceptionally good. For example, solve any of the open problems in math or physics, post a preprint on arxiv.org (well, you may need someone to vouch for you, but that’s not really an issue) and you are all set.
Unfortunately, I cannot recall a single discovery in physics in the last half a century that was not made by someone who jumped through the usual hoops. I have met, however, an occasional person who learned grad school-level stuff on their own, but they did not manage to go any farther. My suspicion is therefore that all that negative selection in science, while annoying, does not do a lot of harm compared to potential alternatives. While it filters out some good people, it probably does not reject the very best, otherwise we would see an occasional example of someone making a significant discovery outside academia.
I predict that we will indeed see this before too long, now that we have the internet; and it will thus turn out that some of the best people were being filtered out. Access to information and social support/reinforcement is a huge limiting factor.
And of course, if you’re willing to look a century back instead of just a half-century, you find the salient example of Einstein—who didn’t even have the internet, but still managed to advance science from outside the “establishment” (which was a sizable apparatus in his time and place, just as it is in ours).
Access to labs, equipment, technicians, funding is an even greater factor. Only mathematicians can really afford to work from home. (And now, computer scientists and computational-xxx people have joined them.)
Yes, all my predictions about people working at home should be interpreted to refer to fields in which that is physically possible.
(In fact, in these discussions I am pretty much always thinking specifically of mathematics, and possibly the most theoretical kinds of physics.)
It’s not quite so dire. You can’t do experiments from home usually, but you can interpret experiments from home thanks to Internet publication of results. So a lot of theoretical work in almost every field can be done from outside academia.
Yes, but in most fields someone can’t participate by only interpreting experiments from home. It’s useful, but you can’t build a career from it. Normally you really want to also be able to influence experiments in the lab to get the new data you want.
I am willing to bet that none of the high-profile open problems in physics, such as quantum measurement, high-temperature superconductivity, dark energy origins, extensions to the standard model etc., will receive a meaningful contribution from outside of people trained in academia, at least not in the next 10 years. The reason is that the cream-of-the-crop people who are able to advance the leading edge stand out enough to be recognized and integrated into the system.
This is a myth. While he had trouble fitting in, he certainly did jump through most of the usual hoops.
If a person is recognized and integrated into the system only after making a contribution, that counts as being “outside the system”. E.g. Einstein, who didn’t get his first academic position until 1908, three years after the annus mirabilis, which had occurred while he was a patent clerk. Indeed, he didn’t even get his Ph.D. until the annus itself—his thesis consisted of one of the famous papers!
He got his undergrad degree in 1900, so, inside the system. He was a lecturer (part time) between 1900 and 1902, though not at a high level. He did indeed develop his SR ideas while dealing with electromagnetic applications while working at the patent office (which paid the bills), though still in touch with other scientists, most notably Marcel Grossmann. His thesis was not related to relativity at all, not that it matters. All of his work on GR was inside the system. Not that he needed the system much by then.
So, the popular view of Einstein as an outsider is blown way out of proportion to reality.
Whatever the “popular” view, all that matters for my purposes is that Einstein was not employed by a university in 1905 when he developed SR, and thus was, by my definition, an outsider. Yes, of course he became an insider later—that’s the dream of of every outsider!
Having an undergrad degree—or a degree of any kind—does not make one an insider. What counts is employment: whether one is paid to do the work in question. If you’re not (as Einstein wasn’t in 1905), you’re an outsider.
Shockley, Bardeen, and Brattain all got PhDs, but invented the transistor while at Bell Labs, i.e. not while employed by universities. I think the “jump through the usual hoops” description shminux is using is a more useful one than the “outsider” description you’re using.
As I stated in the grandparent, the relevant distinction is whether or not you are paid to do the research, or whether you are forced to do it in your “spare time”.
On Bell Labs specifically, see Scott Aaronson:
Thus, the people you mention were, I assume, doing their actual jobs when they invented the transistor, which makes them analogous to academics, and not analogous to Einstein in the patent office.
The modern analogue of Einstein would be someone dropping out of grad school, becoming a software developer (or something), and within a few years posting groundbreaking papers on the arXiv that they wrote for the fun of it. You can call such a person an “insider” if you like because of their (unfinished) education, but I guarantee you they sure as hell won’t feel like one in the years before their paper comes out. (They won’t have library privileges, won’t get invited to physicist parties, and generally won’t be taken seriously because...they’re not a physicist, they’re a software developer.)
More useful for what, exactly?
I apologize, it appears I didn’t read the post in question carefully enough; the criterion of if you’re paid to research is a useful one.
But although Einstein is salient, I can’t think of too many other examples. The two that leap to mind are Green (working in the early 1800s) and Lavoisier (working in the late 1700s), but from that I would expect “household name scientist who was an outsider” to be something that shows up once or twice a century. (I’m counting Lavoisier because he was funded by his tax farming, but he was definitely part of the ‘establishment’ of the day.)
(And you do see contemporary outsider contributions if you know where to look, like Gary Cola or Jack Andraka, but not at the household name level- probably because there aren’t that many household name scientists!)
Better examples of outsider-scientists from around then include Oliver Heaviside and Ramanujan. I’m having trouble thinking of anyone recent; the closest to come to mind are some computer scientists who didn’t get PhD’s until relatively late. (Did Oleg Kiselyov ever get one?)
Again, I don’t care whether the person remained an outsider for their entire life; all they need to have done is to have made a contribution while outside. Thus Einstein in the patent office fully counts.
Moreover, it is worth noting that Ramanujan was brought to England by the ultra-established G.H. Hardy, and even Heaviside was ultimately made a Fellow of the Royal Society. So even they became “insiders” eventually, at least in important senses.
In Einstein’s first years in the patent office he was working on his PhD thesis, which when completed in 1905 was still one of his first publications. I’ve read Pais’s biography and it left me with the impression that his career up to that point was unusually independent, with some trouble jumping through the hoops of his day, but not extraordinarily so. They didn’t have the NSF back then funding all the science grad students.
I agree that all the people we’re discussing were brought into the system (the others less so than Einstein) and that Einstein had to overcome negative selection even while some professors thought he showed promise of doing great things. (Becoming an insider then isn’t guaranteed—in the previous century there was Hermann Grassman trying to get out of teaching high school all his life.)
Heaviside and Ramanujan accomplished less than Einstein, but they started way further outside.
But you don’t get to observe any of the discoveries in physics that haven’t been made. If a good university education is markedly better for learning physics than autodidactism, then the people who don’t jump through the usual hoops will be inhibited by an inferior education and won’t be in the position to make discoveries that a person who did jump through those hoops is.
If receiving a high level education and having access to university resources is effectively a precondition for making significant new discoveries in physics today, you would not expect to see people who did not go through the regular procedures making significant new discoveries in physics, even if the negative selection of academia filters out most of the best candidates.
But remember that these universities are trying their very, very best to select the best candidates, and the people doing the selecting tend to have a lot of experience and to have done a lot of thinking about how to select the best candidates in their field. For every old fuddy duddy who only selects people his antiquated views allow for, there’s a vital young department head out there hunting for the radical new genius. Well funded departments often keep standing job positions open just for when that rare and exceptional person shows up. I don’t think we have any reason to believe that academic departments are bad at finding talented people.
And given how willing people are to throw money and time and resources at scientific and mathematical talents, it seems fair to say that if someone wants to contribute to math or physics (and are capable of doing so), being an autodidact is an almost always an extremely bad strategy. The hoop jumping isn’t really that hard or time consuming.
I think this is simply false as a matter of fact. See this comment.
Do I personally think that the negative selection of academia filters out a lot of the best candidates? Not really; outstanding success in any discipline is heavily dependent on effort (which is not to say that it will not also require a lot of talent,) and I suspect that most people who’re capable of making really top level contributions are able to put in enough effort that they can get through the necessary selection barriers, even ones dependent on skills in which they have low aptitude. But this is a very different matter from supposing that schools are actually good at selecting the best students; with sufficient effort and conscientiousness after all, one could get through moderate strength filters completely orthogonal to the skills one intends to specialize in.
I think that the people that the current negative selection system impacts most are probably not people with remarkable singularly focused genius, but people who could be pretty good specialists, who lack the aptitude to pass some filter orthogonal to their specialty.
Ah, I was speaking in terms of finding talented researchers at the graduate and faculty level, not lvy league undergrad admissions. Your comments seem reasonable on the subject of undergrad admissions, which I agree are almost wholly negative selection filters. Do you think this also goes for graduate admissions? What should we anticipate observing if schools were bad at this kind of selection? What would we see if they were good at it?
EDIT: I’m working largely on the observation that getting a PhD in any field is really very easy. The major barrier seems to be interest. This doesn’t go for all fields, of course. Law is a serious exception. But physics? Mathematics? I lack data here, but I’m skeptical that these are particularly closed academic fields.
For some definition of easy… As most grad students know, phdcomics is a documentary. The majority of grad students have breakdowns and burnouts as a matter of course. Most (70-75%) still finish, only to never set foot in academia again.
I’ve never understood this, but I cannot deny that it’s true.
Then perhaps there’s something about getting a PhD that you don’t understand? And, given that, maybe you shouldn’t make such sweeping statements about how easy it is?
Perhaps you’d like to enlighten me? Do you understand why PhD students so often burn out? I’m a PhD student, and I don’t get it. It’s stressful, but not as stressful as a lot of other jobs.
Some things off the top of my head.
Having to be far more independent a researcher than you ever had before.
The standards for your performance have risen similarly.
Projects that don’t work, no matter what you do.
Projects that would have worked if you had done them right.
Having to deal with stupid university-political things.
Low pay, with long hours
… while you’re listening to the biological clock ticking
… and it’s very likely you’ll have to move in a few years, so settling down will make things very tricky later on. The ‘2-body problem’ is rough.
Possibility of having your result ‘scooped’ by someone
… who read your paper in review, held it up with BS, did a quick measurement to reproduce it, then got it published first.
Hah, tru dat. But if that causes people to burn out, then why aren’t people burning out in all sorts of professions? Maybe they are, I suppose. And I think it’s worth mentioning that most PhD students I’ve known have never burned out (if by this we mean something practically serious, not just a bout of depression).
People studying to be doctors, lawyers, engineers, and actuaries definitely do burn out from time to time. Very likely, other professions too.
The third and fourth items don’t apply so clearly to them, though. You’re held responsible for a number of things that you have limited control over, and moreover they are few in number so you can’t even use statistics to show how good you are at biasing them towards success with your control.
With a doctor, you see many patients. Some of them will get better. If you NEVER succeed, being a doctor isn’t for you. As a researcher, it’s quite possible that if your advisor has a risky research plan, then you simply won’t succeed just because the objective is unachievable due to unforeseen factors, or the hypothesis is true but it would take a much larger effort than you can pay for to conclusively demonstrate it (who wants to see a P-value of 0.2? It IS suggestive...).
In non-research professions, if you are good at what you do, you will succeed. In research, you must be at least decent to succeed, and being good makes it far more likely… but not certain, or even nearly certain. It takes being really really good to figure out you’re going to fail early, get out, and find something else to do.
I’d contradict that. The entry requirements and task is utterly trivial and the intellectual and academic challenge isn’t much of a big deal for anyone with average (or perhaps slightly above average) IQ but the motivational aspect is ridiculous.
Isn’t that what I said?
No, you said it was “really very easy”, which implies that the motivation aspect is trivial. Something can be difficult FROM a motivational standpoint.
On the face of it, this strikes me as tough to judge, because if they’re bad at it we should expect to see e. g., people whose abilities within their field are highly apparent to their professors and peers failing to get into programs which people who make a lesser impression on those who know them do get into, but it’s hard to aggregate anecdotes like this to make strong generalizations.
If you could run two academic environments alongside each other, one of which was good at selecting for high aptitude and one of which wasn’t, I expect it would be pretty easy to differentiate between them, but it’s much harder without a basis for comparison.
I’ve never gotten into a PhD program for either (I’ve never entered a PhD program period, for that matter,) but I don’t share this impression and am curious as to how you formed it.
Even getting into a PhD program in things that we’re inclined to regard as “soft” subjects, such as English, can be quite difficult, since there’s so much competition. “Good enough” means “better than a whole lot of other people who’re also sufficiently invested to apply to the same program.”
Of course, “very easy” is relative, but I think most people would not agree with the assertion that getting a PhD is easy.
It’s not based on very much, I admit, beyond talking to some physics and math people I know about graduate school. It is a selection procedure, and it does take some significant work and talent to get in (as well as some luck), but it doesn’t strike me as unreasonably difficult. All the physics and math PhD’s I know seem genuinely (but not off the charts) talented. Many of them have problems finishing their work on time and writing well. Once you’re in the program, getting out the door with a PhD isn’t that hard, since literally everyone making the decision to graduate you wants you to graduate.
What I’m trying to say is that getting a PhD is hard in the way building a nine foot brick wall is hard. There are some basic skills involved, and then it’s mostly just a lot of time and work. It’s not hard like discovering a new proof in geometry is hard. And if you can do the latter kind of work, people are pretty inclined to cut you some slack on the former. My school recently hired a mathematician who earned her PhD at 24, six months after her BA.
I’d say these are much, much more difficult to get into. Firstly because there are way more candidates (the negative selection pressures are a lot lower in these fields) and secondly, because it’s not at all clear what talent in these fields looks like, so things are a lot more random.
Yes, I think you’re mistaken about this. Relatively speaking, you’re correct—fields like mathematics have less “academic snobbery” than law, humanities, and the like. The problem is that relatively isn’t enough. There is still enough snobbery present that people with “blotches on their record” need a significant amount of good luck in order not to be filtered out.
Well, the question is whether or not academic programs are good at selecting for talent (and, for the purposes of my point, I mean at the graduate level and above). So you may be right that it’s hard for people with blotches on their record to do well, but does this make the selection process a bad one, considered on the basis of costs to benefits?
Yes: there are excessive numbers of both false negatives (talented people with blotches) and false positives (untalented people without blotches).
I’ll make up an example (loosely based on some real cases I know about) to illustrate what I mean. Suppose a graduate admissions committee in a math or physics department is looking at two candidates. Candidate A is applying to transfer from another graduate program at another institution, where for some reason he was a total disaster, flunking courses left and right. On the other hand, his undergraduate record is near-perfect, and itself consists mostly of graduate-level courses. Candidate B, by contrast, is applying directly from undergraduate school, and has a reasonably good record, but with standard undergraduate coursework in the subject, nothing particularly beyond “grade level”.
Now, of these two candidates, which do you think has a higher chance of admission? Neither of them is ideal, of course; but let us stipulate that exactly one of them is in fact admitted. Who do you think it was?
If you were trying to select for talent, it would be Candidate A every time. And sure enough, there are people in math/physics academia who would indeed pick Candidate A. (This is an improvement over high school, or humanities academia, where no one would.) However, a disturbingly large number will pick Candidate B. They will see Candidate A as “damaged goods”, as “unworthy”—as if a spot in a graduate program were a reward for “good citizenship”, rather than a resource to be used for getting research problems solved.
Why do you think this?
So I haven’t read your last paragraph yet. I guess if I were making the decision, I’d pick the undergraduate, unless the graduate transfer had some kind of story about what happened (a divorce, a death, something that would explain the failures). I think that’s probably how most admissions committees would go too: talent isn’t worth much if you need to be dragged to work every day. Now, on to your last paragraph...
About humanities academia, this isn’t at all true.
Well, of course there’s some kind of story that explains the failures, other than “he just wasn’t good enough”; by assumption the guy wasn’t let into the first graduate school off the street—he had been a brilliant undergraduate. So something went wrong. The question is whether you should filter out a clearly capable person because something went wrong once. If you do, you’re simply not optimizing for the right thing.
A rational deliberation on Candidate A would look like this: “This person obviously has issues, and he might not succeed here. But if he does, there’s a good chance he’ll succeed spectacularly -- at least, at a level above that of many students we’re happy to call ‘successful’. If he doesn’t, well, some percentage of the blemish-free students we admit are going to fail anyway, so what’s the difference?”
It is a fact of life that sometimes multiple iterations are required for something to work; but the school system does a poor job of accommodating this. Repeating a course as many times as necessary until you get an A is not something that the system encourages; if it were optimizing for the right things, it would be.
Yes, that was a flippant, stereotyped exaggeration—especially since “humanities” includes philosophy and linguistics, where the mentality is often very similar to math. Still, like any good caricature, it has some basis in reality. Generally speaking, the “softer” the subject -- the fewer objective measures of competence—the greater the reliance on pure status games, i.e. academic snobbery. And of course my whole point here is that this snobbery is present even in “hard” fields.
What I mean is that I’d want to hear the story. If the guy’s wife died, that’s one thing. If he got addicted to heroin, that’s another. If I’m given no story, then I’ll go with the undergrad. But I don’t disagree with the rest of your assessment.
Fair enough.
This may be somewhat helped by having university education available online. (Although having the patience to see the online courses and do the exercises is also kind of a hoop. But at least it does not say you when you must do it.)
If you still mean physics: why this confidence about the existence of low-hanging fruit? My grad student friend had to go to the LHC to work on (I think) his thesis. I assume they don’t let people in off the street.
If you mean academia in general: have you forgotten where you are? ^_^
Maybe there is some misunderstanding here. I’m sure there is plenty of low-hanging fruit still undiscovered. But you have to first get to that hard-to-reach orchard where it grows.
Indeed they don’t, though I’m not sure how it is related to my point that negative selection is not a total disaster.
Where am I?
What would look different if it were? (Aside from, say, the reduced chance of someone finding the Higgs.)
Then I would expect that once in a while some filtered out genius discovers something really exciting, against all odds, as I mentioned already.
On Less Wrong, which has an anti-academia bias.
Why do you call it a bias? Maybe it’s being less wrong than others who have a pro-academia bias.
If so, this is rather irrational, given that probably every high-profile/high-status contributor to this forum, with the notable exception of EY, either works in academia or is being/has been trained in academia.
It isn’t so. It’s more a relative thing—”not quite as extremely biased towards academia as the average group of this level of intellectual orientation can be expected to be”.
Luke has minimal official academic training too. Mind you he is more academic in practice than most people (probably most academics too, come to think of it.)
If so, then we’re actually more rational right? Because we’re not biased against academia as most people are, and aren’t biased toward academia as most academics are.
Should we all place bets now that it will be Eliezer?
He repeatedly mentioned that his skill is in formulating theorems, not proving them, and he has not formulated even one after some 5 years of working on the same problem, so the chances are not good.
What my externally observable percentiles look like:
Writing: 99+%
Math: 99+%
Conceptual originality: 99+%
Programming: 95%
Conformity / ability to obey incorrect orders: 20%
What my educational credentials look like:
Highest level of education completed: 8th grade
I’m with J and Alex—are you comparing yourself to people or to programmers? I’m fairly sure FizzBuzz puts one well above 95% in the general population
Programmmers.
Is the 99+% for math also compared to mathematicians?
That’d be a much harder question to answer; my talent is specialized toward figuring out the shape of the right theorem to be proved, not the actual proof which is where most modern math concentrates its prestige. (This is an objectively verifiable form of mathematical talent; it means that sometimes Marcello would prove something and I would look at it and say, “That doesn’t look right” and at least half the time there’d be a mistake.) I feel insecure about not being an expert in the tools by which most modern mathematicians measure basic competence; I can also apparently make “well, if that’s your problem, try transforming it this way” suggestions to someone doing graduate mathematical research at Yale that are accepted as brilliant. I confidently depose that, even taking unusual talents into account, I am not in the literal top tier of mathematical potential—if I can explain basic Bayes better than anyone or was first to state the Lob problem or invented TDT, those outputs drew on at least some non-mathematical high-percentile sections of my brain (explanatory ability in the first case, or what’s somewhat vaguely referred to as “philosophical” talent in the other two). I’m also reasonably confident that, given a hundred modern mathematicians, an average of zero will pick the right problem to solve.
I think I’m comfortable at this point with saying that I’m in the top 99+% of writers—I’ve been picking up “real” books and trying to read them and finding that they seem visibly badly-written to me now that I’ve written HPMOR. Though I’m still not in the literal top tier; there are basic things in writing that I still don’t do too well, despite being outstanding in others, and my new level of skill is just enough to start noticing that Neil Gaiman and Terry Pratchett are doing things way the hell above me.
“What did you mean when you said 99%” should not be a hard question to answer. “Which alternative is correct” may be hard, but did you have in mind “one alternative is correct but I don’t know which”?
Have you written anything about the process of figuring out the shape of the theorem to be proved, and/or identifying the plausibility of a theorem?
Many geniuses have a bad reputation for checking all the details and writing them down. Kontsevich and Thurston are good examples. People complain about this but the status hierarchy isn’t much affected by those complaints. Low-status mathematicians don’t often get away with Thurston’s attitude, but nor do they accumulate status by being more conscientious.
Being able to give some actual proofs is a prerequisite of prestige. But it’s not clear to me that it’s right to say that mathematics concentrates its prestige there. See, for example, Fields Medalist Timothy Gower’s article The Two Cultures of Mathematics (pdf):
I suspect the distinction Eliezer is making is more akin to the controversial “theoretical vs. experimental” one proposed by Jaffe and Quinn than the traditional “theory-builder vs. problem-solver” one discussed by Gowers.
It’s been years since I read the Jaffe–Quinn article. But, as I recall, it was more about the methods used to answer questions, and about how rigorous human-verifiable proofs might give way to heuristic/probabilistic and computer-aided proofs. Eliezer, on the other hand, seemed to be saying that mathematicians concentrate prestige on answering questions (by whatever means the community considers to be adequate), as opposed to “figuring out the shape of the right theorem to be proved”.
Jaffe and Quinn mainly advocate that labor should be divided between people who make conjectures (“theoreticians”) and people who prove them (“experimentalists”). I don’t think there is much of anything about probabilistic or computer-aided proofs.
You are right. Looking at the Jaffe–Quinn paper again, it is closer to the distinction that Eliezer was making. (However, I note that the mathematical “theoreticians” in that article are generally high-prestige, and the “rigorous mathematicians” have to fight the perception that they are just filling in details to results already announced.)
My mischaracterization of Jaffe and Quinn’s thesis happened because (1) Thurston replied to their article, and he discusses computer-aided proofs in his reply; and (2) even more embarrassingly, I conflated the Jaffe–Quinn article with the Scientific American article The Death of Proof, by John Horgan.
I once got this feeling reading Stephen R. Donaldson’s The Runes of the Earth—that this was a level of writing that was way beyond what I could see myself reaching. Oddly, I didn’t get this feeling when reading Terry Pratchett, even though I still think that Terry Pratchett is probably a better writer than, say, Shakespeare.
And I don’t know what people see in American Gods—I’ve read over one hundred books I think were better. And I mean that literally; if I spent a day doing it, I could actually go through my bookshelves and write down a list of one hundred and one books I liked more. I couldn’t do that for most of Terry Pratchett’s novels.
Oddly, I don’t like Gaiman much at all on his own, and I don’t like large doses of Pratchett either, but I loved Good Omens—they balanced each other’s weaknesses.
Not at all like this
Oh wow, I have the exact opposite reaction; I love both Gaiman and Pratchett separately, but dislike Good Omens—they undercut each other’s strengths.
I tend to describe it as: “They have a similar worldview, but Gaiman is dark, whereas Pratchett is light. When you put them together the result is a rather bland gray.”
I also tried American Gods for a while and found that its charm was mostly lost on me—maybe I didn’t get far enough in. Good Omens, on the other hand...
Understand, I always knew that Good Omens was a great book and that I wasn’t yet writing that well; it’s only now that I’m staring at a Neil Gaiman short story, thinking, “I can tell that he’s doing something outstandingly right here that I’m not doing, but it’s hard to say exactly what...”
American Gods is pretty evenly written; if it didn’t grab you in the first fifty pages or so it was probably never going to. (I say this as someone who fell in love with it and considers it among my favorite of his work.)
I blame the SeinfeldIsUnfunny effect. I’ve seen GodsNeedPrayerBadly done so many other times that it seemed like a cliche.
Personally, I disliked that trope even before I’d seen it enough times for it to seem cliche, but I count American Gods among my favorites of Gaiman’s work in spite of it.
Perhaps. It was far from being my introduction to that trope, but I found it worth reading for something other than the originality of that particular idea. Still, different people like different things in their art.
Gaiman frequently doesn’t grab me, though I think “A Study in Emerald” is brilliant.
I wish American Gods had been written by someone who understood and liked America better. Why was the computer god a marketing monster rather than a programmer? Or a computer? And I know it’s not fair to blame a writer for not writing a different book, but I’d like to see a version of the idea with the guts to engage with actual American religions.
I’ve read many, many books I liked more than many books which I would consider “better” in a general sense. From the context of the discussion, I’d think “were better” was the meaning you meant. Alternatively, maybe you don’t experience such a discrepancy between what you like and what you believe is “good writing”?
A book can be well written and still be bad because of other flaws. Nathaniel Hawthorne’s The Scarlet Letter was very well written in a technical sense, but the story itself was boring as hell and Hawthorne’s skill couldn’t save it.
What of Pratchett’s work are you judging by? Middle-Pratchett is top tier, but I’m much less impressed with recent-Pratchett (going back at least a few years before the Alzheimer’s diagnosis.)
I get the feeling that a lot of the top experts at writing aren’t necessarily experts on writing, because they can and do lose their touch without noticing that something is wrong.
What published software have you written that I should look at? A quick Google search comes up with this, listing Flare and Document Designer. Is there something different you’d rather be known for and judged by?
Incidentally, Googling “Eliezer Yudkowsky software” also returns this link, which is someone’s search on pricegrabber.com for “Rationally Speaking Eliezer Yudkowsky Bayles” [sic] for which pricegrabber returns the Dragon Naturally Speaking software. I don’t know why this is near the top of Google results, but it Rationally Speaking sounds like a real-life Eliezer Yudkowsky Fact...
95% of programmers don’t have published personally published software.
A more general form of the question: “What publicly available evidence is there for this 95th percentile claim?”
Really?
The Wikipedia article on GitHub claims >1m users (passing the mark in 2011), with around 90,000 unique repositories out of 2m. StackOverflow has a few relevant questions like http://stackoverflow.com/questions/453880/how-many-developers-are-there-in-the-world which give me a vague estimate of perhaps 5-15 million worldwide. 0.1m unique repos to 15m developers is 6.6% and roughly consistent with 5%.
On the other hand, I don’t know what ‘personally published software’ might be. A complete standalone executable or library? I could see most programmers confining their efforts to working on existing codebases, yes, and this would likely cut down the GitHub repo estimate a lot since many of them are no doubt forks with a patch or two of some main repo or dead ends of various kinds.
You’re off by an order of magnitude—it’s 0.66%.
Whups. I just thought 1⁄15… Well, an order is still within the margins of error here since a GitHub repo count ignores other sites like SourceForge.
Yes, I’m completely certain. (ie. p=0.6)
Are you implying he’s better than 95% just because he has published something? Or what do you mean?
No, I’m saying that it would be misleading to imply the reverse if someone hadn’t or to place all that much weight on the published software if they have (except in as much as the published personal software projects establish a lower bound.)
Well, that’s why I asked him if he thought he could be fairly judged based on those two published projects. I didn’t go ahead and just judge them, and I won’t unless Eliezer says they’re worth judging because they do establish such a lower bound for him.
What population are you comparing yourself to?
Are these your own estimates, or have you found some objective, accurate test for ranking “Conceptual originality”?
I put that in because I didn’t think any non-trolls would seriously dispute the 99+% part, not because I knew how to measure it down to the sixth decimal place.
These days, you can claim to be homeschooled. ;)
Why so low?
I haven’t put as many skill points into it.
Ah, sorry, it looks like I was unclear. I meant “What makes you think that so many people are better at programming than you?”, not “Why are you so bad at programming?”. I had assumed that you were comparing yourself to the general population, although now that I see you have clarified that you were comparing yourself to other programmers, the 95% estimate makes a lot more sense. Giving people estimates of your percentiles without telling them the population you are comparing yourself to is almost useless.
This is a too bold claim. So your writing skill is at the nobel prize class? Your math is close to fields prize level? And your programming skills is better than programmers who has spend a life-time programming?
You’re 32 and aldready a better mathematician, programmer and writer than people spending life times doing that? There’s only 0.1-% people in each category that are better than you?
Instead of saying you’re among the worlds best, bragging ego inflation. Wouldn’t have sufficed to say, “Im good at a,b,c… even tough i’ve only had 8th grade education”?
I don’t think that’s what 99+% means.
Can I ask on what basis you’re drawing this conclusion? I agree with the bulk of what you said about overuse of negative selection, but I challenge the idea that it’s producing cookie-cutter student bodies at elite universities. Having attended Yale as an undergrad, your claim strikes me as incorrect, as the Yale student body seemed to me more diverse on all five of those categories than any other (non-online) community I’ve been a part of—certainly more so than my high school, my law school, or my current professional environment. That’s just my anecdotal experience, so I’m open to the idea that I’m mistaken if you have some more formal analysis to back this up. But this statement jumped out at me as questionable, so I wanted to see where you were getting it from.
He also attended Yale...
I’m reminded of a quip from a former Yale professor:
As for hard facts and statistics, I imagine Charles Murray’s recent book or Hayes’s new book Twilight of the Elites (both of which I have heard good things about) might provide some.
This is a very important point; thanks for cross-posting.
I’ve never understood the reason for giving grades A-E or fail, like we do for O and A levels, or I:II:III:fail, like we do for degrees.
My father’s O-levels gave a percentile ranking, so he was e.g. in the 83rd percentile in the country for history.
So we must have changed over at some point. Does anyone know why? It’s always looked like throwing information away to me, and it’s also unfair to people on the grade boundaries.
Of course this may be motivated thinking on my part, I’d much rather have had a string of 100s for my exams than a string of As, and I’d much prefer to have got a 75 for my degree than a II (which covered percentiles 25-75) !!
I think a lot of people don’t like using percentiles because they are zero-sum: Exactly 25% of the class is in the top 25%, regardless of whether everyone in the class is brilliant or everyone in the class is an idiot.
But on the opposite side of the spectrum is saying: “Everyone is so smart because they can read and write, and thousand years ago most people couldn’t do this!” (Strawman example, I know.)
Generally I would prefer to have a list (tree, directed acyclic graph...?) of all human knowledge, and give everyone a report saying: “This person understands these parts.” But over time, the list/tree is growing. Of course it is OK to know a smaller part of total human knowledge, because the population is growing; but still, you need to know more than your ancestors (if your computer skills are the same as your grandma’s, then she is a hero and you are a loser); on the other hand some knowledge becomes obsolete.
I think a percentile across the whole country would be a good measure for comparing individual students or schools. And it would be nice to also calculate long-term changes to know whether the country as a whole is improving.
Of course, you could show percentage scores in the tests rather than where you sit in the country. That means that it should be consistent over time, although I agree that in a decent sized national subject it’s probably fine the other way.
My main objection to giving percentiles relates to the OP’s concern that there’s no such thing as a ‘very good’ A, At least with UK school exams, I think that getting 100% in most subjects tests for conscientiousness and not making silly errors at best and being well-trained in the exam system at worst. I am pretty sure that if percentages were public I’d have had to get better marks to get into uni, but also that in making sure I did so I would not have been using my time usefully.
What I think would be far preferable to a ‘who managed not to screw up a single question’ model of getting better than an A would be an extension paper that was genuinely challenging and couldn’t be straightforwardly taught.
Boy, are you setting a high bar for Erna Hoover’s grandkids.
You don’t think Jack Andraka will get admitted to Harvard because he did one thing really, really well?
Why would he want to go to Harvard specifically? For a research career, he should be choosing specific programs and labs — not choosing on the basis of the broad general reputation of a university. For undergraduate, though, an institution with a culture friendly to weird people (and anyone doing that level of research at age 15 is gonna be pretty weird) might be more rewarding than a hothouse of Future World Leaders.
“Should go to” and “will get admitted to” are different things. I think it likely that Andraka will go to Johns Hopkins, for obvious reasons.
My point is more that positive selection mechanisms do exist- they’re just a small part of the general pipe, and people are trained away from hunting for them instead of just putting up with the trouble of negative selection.
Now, tommccabe’s point still stands if the word “one” is the most significant word in that sentence. There are a host of ways to raise yourself to Harvard’s attention and clear aside the barriers put up to keep the riff raff out- but they are non-obvious and (deliberate) chinks in the general system, rather than the primary way the system operates.
Not if his high-school GPA is 2.0.
I think more highly of Harvard’s admission department than that. At the very least, Andraka has shown himself capable of finding a champion within the system through persistence.
I got into UC Berkeley with a high school GPA of 2.9 by talking about math with professors. This strategy failed everywhere else, and would have failed at Berkeley if I hadn’t been lucky enough to find a professor stubborn enough to argue with the admissions office again after they ignored him the first time. On the other hand, my accomplishments are not even close to as impressive as Andraka’s, so he might have an easier time with this strategy even with a worse GPA.
Anyway, if you’ve done anything impressive, finding a champion within the system is easy. Andraka had a hard time with that step because he was trying to get support before doing something cool rather than after. Now, the vast majority of biology professors would gladly stand up for him to their institution’s admissions department. But this strategy requires persistence on the part of the champion, as well as the applicant.
It’s not impossible, but “easy” is an overstatement.
One of the most disappointing discoveries of my life was the existence of professors—even math professors, alas—who think like high-school teachers or college admissions officers. They not only exist, but exist in large enough numbers that one will actually run into them. The good guys also exist, but they don’t dominate the way I thought they did. This realization caused me to change my view of academia.
Again, I think that’s an overestimate. Maybe half of them would, but the “vast majority”? You would get a substantial number who, while grudgingly admitting the impressiveness of his accomplishment, would seek to rationalize the traditional status structure by making excuses about his not being “ready” or “a good fit”, etc.
When I did it, it wasn’t very difficult. Maybe about a third of the professors I talked to agreed to talk to their admissions department on my behalf, which is remarkably high considering that I had not received any official recognition for my work, I had some difficulty explaining it coherently, it was really only slightly impressive, and I’m a bit short on social skills, which are useful for impressing people (I had a lot of really awkward meetings with professors). Andraka at the very least has a huge advantage over me in the first and third of those problems, which should give him a sizable majority. If he’s solid in the second and forth problems as well, I would expect him to get an overwhelming majority.
Any particular reason?
My primary reason is I expect that they’ll have the best one that money can buy- and I suspect that schools like Harvard explicitly look for a blend of students (since a system that gives them only Senator’s sons or a system that gives them only brilliant kids will lower the value of Harvard to both, but a system that gives them, say, 30% Senator’s sons and 70% brilliant kids will do better).
I think this is wrong. To be sure, they could have that if they wanted, but that doesn’t seem to be where they actually spend their (considerable) money. From the book A is for Admission: An Insider’s Guide to Getting Into the Ivy League and Other Top Colleges:
\
Yes, but they have a formula for achieving the blend they seek, and that formula is going to filter out people with low GPAs and such.
I know that the admissions staff are generally mediocre students who went to that school. But I would expect their system to notice things like “Intel Science Fair winner” and have that trump any GPA signal.
(More reasonably, I think they have a file of names called “admit these people”, which it checks applicants against (and if they’re on the list, sends it to a human to verify), and Andraka’s win was publicized widely enough that he probably made it onto the list, and if someone is doing their job well they routinely import the list of winners from things like the Intel Science Fair into that list.)
If you can find a way to settle the bet, I bet they don’t do that. Universities would look extremely different if they optimized for learning in even the most basic ways. This is the should-universe you’re talking about, not the is-universe.
I have a dean of admissions at a large university in my nuclear family. Eliezer is right, there’s no list like this.
But on the other hand, “Intel Science Fair winner” will PROBABLY attract the attention of the admissions committee. It’s basically up to the applicant to craft a good applications package (including essay and letters of recommendation) that will capitalize on their amazing, singular strength, and throw weaknesses like GPA into shadow. If the applicant can’t do this, they won’t be admitted.
It’s a Bardic Conspiracy problem, really. It’s a matter of storytelling and presentation.
I note that it also makes no sense to filter excellent scientists who aren’t good writers or who take a long time to write (e.g. PhD dissertation test). If you can do the research, someone else should be able to specialize in writing for you.
It’s remarkable how many barrier-to-entry professions revolve around the denial of professional specialization. A surgeon can’t just be someone of moderate intelligence and exceptional dexterity who studies and practices one key surgery, no, they need 11 years of medical school that they’ll mostly never use. A scientist is forced to write. And so on.
Fair enough on the medical school thing, but is this really a serious barrier in something like physics? How hard is it for a talented researcher to learn to write a technical, scholarly document in a timely fashion? Do you know of any good, hard working scientific talents denied access to resources because of their writing ability? Because I know lots and lots of mediocre researchers who are nevertheless perfectly adequate scholarly authors. It doesn’t seem like a demanding filter. In my experience most journal articles are terribly written (much worse than your sequences, for example), so the standards can’t be that high.
For many people, writing is hard even if they are good at math. It is why Verbal and Mathematical SAT scores do not perfectly correlate. It’s a different talent, and it will indeed filter people who don’t happen to have it. Even bad writing is hard—and if you can’t bear to write badly and don’t have the talent to write well, it’s much much worse. It filters people who want to do their jobs well and don’t happen to possess author talent, because they’ll revise, and revise, and revise, staring at their work and feeling the dreadful pain of how bad it is… yes, it’s a needless filter!
I asked a professor about this. She’s works at the University of Chicago, in philosophy, but she’s friends with a math professor she met as a grad student at Berkeley. Here’s what she said, so far as I remember it:
I asked if this caused math talent to go to waste:
So what I took away from this was 1) I was wrong in thinking that math departments don’t care about math-extrinsic skills. 2) I was wrong to think these don’t filter people out. It hadn’t occurred to me that there is more mathematical talent than there is money to develop it. It seems like the problem with academia is kind of just a lack of funding.
EDIT: I might as well add that, needless to say, writing ability was considered important to philosophy too, and a filter at every level, but that’s not surprising. She didn’t have anything to tell me about physics.
As it happens, a few months ago I saw an interesting paper examining the consequence of the fall of Soviet Russia and the subsequent exodus of top Russian mathematicians (with all their unique results and methods, obscure to the West) into the US. The upshot was that the effect was to push out of academia a lot of lower-ranked American mathematicians—it turned out to be a zero-sum environment… “The Collapse of the Soviet Union and the Productivity of American Mathematicians”
Wow. This is how I feel about my own writing, expressed more clearly than I could myself. I take ages to write a single sentence because none of the phrasings my brain suggests sound like the kind of thing that I’d want to read.
If you wish to write despite this struggle, I recommend breaking writing into two tasks: dumping and editing. Basically, force yourself to ignore the “kind of thing I’d want to read” feeling for as long as it takes to generate a bunch of sentences. Then you can turn those sentences into readable sentences, in editing.
This handy page will make it significantly easier to ignore the editing urge.
I think I should ask for empirical input at this point: is it your experience that good mathematicians or scientists are filtered out of academic advancement and access to research money and materials as a result of a poor showing in skills extrinsic to their field? By ‘extrinsic’ I mean skills that are neither necessary nor sufficient to do mathematical or scientific work well.
Well, it’s needless only if bad writing turns out to actually not interfere with their ability to do their jobs well… e.g., if their job doesn’t involve communicating clearly, or if it does but the way their writing is bad doesn’t interfere with clear communication.
In terms of writing quality, I’ve encountered journal articles I’d have been ashamed to have produced in middle school. I’ve often reflected that it might be an improvement to mandate “Writing for Scientists” classes, which teach clear and succinct written communication. The jargon barrier frequently serves to hide the fact that the authors of a paper aren’t very good at communicating their ideas, even to people who’re familiar with the specialized language of their field. This should be no surprise, since many people are bad at clear written communication, and a scientific education doesn’t do much to select for this ability.
Sure, it’s generally possible to read and extract the relevant information from a badly written article, but it makes the process of researching the literature considerably slower and more error-prone, so it’s not as if bad writing doesn’t come with practical costs
Of course, this runs into the same problem, that a person in, say, the 99.99th percentile of scientific ability and 40th percentile of writing ability is probably going to end up with a lower GPA than someone in the 98th percentile of scientific ability and the 90th of writing ability, although the former is almost certainly more valuable to their field.
Check the name and institution. Science is done in English, but not always by native English-speakers.
I took one at my undergraduate institution. It was a good idea but a poor execution- basically, you could only do so much in four months, and there’s too much diversity in required outputs (as different journals and fields have different formats). I did have enough pointed suggestions at the end of the class that I think widespread mediocre execution is possible, and with a clever professor the class can become good.
The issue is by no means exclusive to non-native speakers.
Having peer-edited papers from many fellow students throughout college, I was astonished not only by the low quality of writing, but the lack of improvement from first year to graduation. Of course, it’s possible that most people are simply innately incapable of writing above low standards and these students had hit their ceiling, but I don’t think there’s any basis to infer this given that they weren’t given any instruction or pressure to improve. Grading was only influenced by fluency of writing at the very bottom end, so there was very weak selection for writing ability.
Actually, I think our education system ought to put a lot more focus on teaching to write well at the level of middle and high school. The essays students are made to practice writing are poorly tailored to cultivate the sort of writing ability that students are likely to find useful in the future, and public education generally doesn’t offer students much other instruction in writing well.
Agreed that this is also an issue for natives, and that improvement from first year to graduation is low.
Agreed, and I think the ability and inclination to communicate well is actually a reasonable standard.
In Physics? Math? I’m not sure. Is anyone here doing an undergrad program in this fields? How much writing are you expected to do, and how dependent is your GPA on your writing ability?
I wasn’t claiming that this is currently the case in math or physics programs, but that it would be a consequence of mandating “Writing for Scientists” courses.
In my own science courses, the answers were “quite a bit, for a given value of writing,” and “not much” respectively. Most non-math classes were “writing intensive classes,” meaning that they involved considerable amounts of putting your own words to paper, but grading was very little dependent on the fluency with which you did so.
Thanks, that’s about what I was expecting. In your own experience, to what extent is your grade dependent on the skills specific to your field, and to what extent is it dependent on extrinsic skills? Are you doing a BA, or are you doing graduate work?
Could you clarify what you mean by field specific skills versus extrinsic skills?
I’ve completed my BSc, but haven’t applied to any graduate program.
Well, like if your field was physics, to what extent was your grade determined by skills you would use (and you would consider important to success in) the practice of physics as a theoretical or experimental activity? I don’t think I have a good idea of what these skills are, but I imagine math is an important one. And to what extent were your grades determined by skills like the writing of effective prose, which I take it we’re considering extrinsic to physical research as such?
I’d say that your grades in the core courses of an undergraduate degree would be pretty strongly determined by some combination of effort, conscientiousness, interest and information retention. If you can retain the material you’re taught in class and apply the required equations to it, and invest a high level of effort into all the assigned work while closely following the provided grading rubrics, you can get good grades without much writing fluency, and without much need for other intrinsic skills such as ability to come up with good original experiments or solid hypotheses to explain data.
I actually have a lot of affection for academia overall (my whole family consists of professors, and I like them, so I also have warm feelings toward the culture that supports them). But academic writing is one of the best examples of the kind of dysfunction Eliezer is talking about.
While there are a few rebels who attempt to write scholarly articles with clear and engaging prose, most academics are actually trying to do the opposite. They make their sentences as convoluted and jargon-filled as possible because it signals that their work is hard and advanced, and because they don’t really want anyone outside their field to understand it. Often this would open them up to kinds of criticism they don’t want. (This effect is pretty much confined to the humanities and the social sciences. Most of the hard sciences are already impenetrable to outsiders, so they don’t need the extra barrier of thorny writing.)
So those journal articles might be written to a higher standard than you think, given that the standard is obtuseness and impenetrability.
Funny. All the people I know in academia place high value on good writing and complain about journal standards preventing them from writing as clearly as they would want (so this is in partial agreement that the system discourages bad writing, but more in the sense of annoyances that prevent an otherwise good paper from being very good). I will also note that, at least in my experience, there is a pretty clear correlation between institutional prestige and clarity of writing (better institutions produce clearer papers).
I think we’re in complete agreement, actually! There’s nothing you said that I would dispute. As you say, there’s a widespread perception among academics that they’re forced into a style of writing that’s intentionally “bad” (unclear, obtuse). Some rebel against this standard. I think the individuals who would be most likely to rebel are those with solid results or substantive ideas that they want to share with a wide audience, and these high-value academics are most likely to end up at more prestigious universities.
But the bad-writing system is perpetuated because most academics aren’t the cream of the crop. Most of them don’t consistently come up with interesting new ideas or groundbreaking new results. They still have to publish articles. (For those outside academia: a professorial career—even at a “teaching” university, as opposed to a research institution—is largely driven by the pressure to publish or perish.)
Because of these pressures, it’s to the average professor’s advantage if he can publish papers that seem deeper or more substantive than they are. So the majority have an interest in perpetuating the current standard of academic writing, which is deliberately obfuscatory.
Hm...to me it seems more a case of status quo bias. This is the way it’s been done, and it’s risky to submit something that doesn’t conform to the standards, so most people don’t take that risk, and so we never have much evidence about what would happen if the standards weren’t followed.
After all, being a reviewer is considered a high-status position. The reviewers for a journal have typically published in that journal before—therefore the reviewers for a good journal tend to themselves be good. So the standards for acceptance to a good journal are implicitly set by good researchers. In fact, a poorly written paper is very unlikely to be accepted in such a venue.
Perhaps we are in agreement with all of this (since you seem to agree that this is not as much of a problem at the top). But then there seems to be a very simple action to circumvent all these issues—only read good journals, and only submit to good journals. In particular, your statement
doesn’t make sense to me if we really are in agreement.
So, now I’m confused about what you’re actually saying. You start by acknowledging that the status quo in academic writing is a standard of bad, obfuscatory writing (and this goes back to what you said originally: “All the people I know in academia...complain about journal standards.”)
But then you posit some kind of broad pool of outside-the-status-quo “good” journals, that hold different standards? I don’t think this pool exists. If they did, the status quo would be different, and your friends would not have the uniform complaint that you report.
And being a reviewer isn’t particularly high-status, because it’s typically an anonymous job. In fact the really top-of-the-field academics usually don’t do much reviewing, because they’re too busy. The one exception is that journals will, as a courtesy, often give big-name academics the chance to review articles that attack or oppose their own work. (You can probably see why this is a bad idea, but it’s very common—and results, as you might expect, in “unorthodox” papers being denied publication.)
I didn’t say that. I said that standards inhibit optimal writing, not that they encourage bad writing. I also didn’t say there was a broad pool of publication venues, just enough that you can publish what you want there and read what you want there. For instance, in machine learning, it would be:
Advances in Neural Information Processing Systems
AI & Statistics
Journal of Machine Learning Research
Uncertainty in Artificial Intelligence
I’m sure you can still find some poorly-written papers there (especially at the conferences, where reviewers are very over-worked), but I would be very surprised if you thought that the papers there were bad and obfuscatory. Reviewers spot obfuscation a mile away and penalize it appropriately.
Yes, I was wrong about that. Being an area chair or sitting on an editorial board is high-status, though, or so I believe.
This strikes me as so cynical that I’d want to see some evidence before I can take it seriously. Many academic write in the way that they do because they’re writing to an audience of insiders (should they not?), but I can’t imagine why they would want to be intentionally obscure.
I think this is going to stagger you. Check out this article: it deals with the academic theorists who are actually willing to state, in print, and repeatedly, that clarity of writing is not their goal.
“On one side stand academic luminaries like University of California at Berkeley rhetorician Judith Butler and University of Pittsburgh English professor Jonathan Arac, who take their inspiration from critical theorists like Michel Foucault and Theodor Adorno. Arguing that their work has been misunderstood by journalists on the left, these radical professors distrust the demand for ‘linguistic transparency,’ charging that it cripples one’s ability ‘to think the world more radically.’”
And here’s Richard Dawkins on the phenomenon:
“Suppose you are an intellectual impostor with nothing to say, but with strong ambitions to succeed in academic life, collect a coterie of reverent disciples and have students around the world anoint your pages with respectful yellow highlighter. What kind of literary style would you cultivate? Not a lucid one, surely, for clarity would expose your lack of content.”
(The article goes on to cite specifics and is well worth a full read.)
Okay, but this is a) a case where someone is writing obscurely because they believe they have good reasons to do so (not so as to make it seem hard and advanced), and b) has nothing to do with physics or mathematics.
If anyone here thinks, like, leftist critical theory is worth a damn, I’ll be surprised. But that’s not part of academia at issue.
I think your a) is wrong—making it seem hard and advanced is part of the “good reason.” But like I said originally: “This effect is pretty much confined to the humanities and the social sciences.” In other words, two thirds of academia.
Eliezer is perhaps thinking of someone like himself, who can write very well, but not very quickly.
Many people seem to assume that because Eliezer is highly intelligent, he would succeed in school. But personally, I think he would have a hard time. He’d be the Intel Science Fair winner with the 2.0 GPA. In fact, I’m not even sure he would make it through college, let alone high school (which is much harder). The reason? He described it in Two More Things to Unlearn from School:
The only way it would work would be if he had a powerful mentor looking out for him, so that he either wouldn’t have to go through this insanity, or it wouldn’t stop his advancement if he did it poorly. Absent that, he—and probably a fair number of other similar people—would fall through the cracks.
That may be. Has he tried seriously to get into academia? My impression is that he doesn’t think it would be worth his time. I graduated high school with a GPA of around 2.0 as well, and I do okay. Being productive in school, if you really want to, isn’t a very hard thing to get into. And if it is, for whatever reason, very difficult for someone to become productive then they’re probably unsuited for research anyway. My take on EY is that he would do fine if he found the right institution and was really inclined to go through it.
This is the notion that I would like to disabuse you of. School filters select strongly for Conscientiousness and weakly against Openness; whereas the former plays at most a minor role in research as such, and the latter is crucial.
Someone might, therefore, have too much Openness and too little Conscientiousness to make it through the filter, despite having enough of these traits (a large amount of Openness, and a bare minimum of Conscientiousness) to function as a brilliant researcher.
And my point is that that is a big “if”.
So, I’ve definitely both underloaded and overloaded myself academically (well, by ‘overloaded’ I mean ‘had to drop all non-school projects to do my school projects well enough’). I feel tremendous sympathy for people who, for whatever reason, don’t line up with the university standard: one of my friends in undergrad would be able to work solidly for around three months, but then have a breakdown for about a month, before the cycle would repeat. This was tremendously unhelpful, because semesters were four months long- but if he were on a trimester schedule, he would probably be fine.
And so people do slip through the cracks, who could probably be great researchers. (He had a terrible time keeping regular jobs as well, because they don’t like to give three months of vacation a year.) But it’s not clear to me how large an issue that is. Someone who can only do two courses a semester can get a college degree eventually- the system is just not set up to encourage that, and if you believe strongly in the importance of youth for research (which I mostly don’t) then you might want to dissuade the people who be able to devote a smaller portion of their youth to research than others.
Okay, why do you think conscientiousness plays a relatively minor role in research? What do you mean by openness, and why do you think schools filter against it?
The word “Openness” in his original post is a hyperlink. Clicking on it will produce a definition of the term :)
The former is the vast majority of research. And most things.
I don’t believe it. The relentless and systematic pursuit of one’s own obsessions is not the “virtue” taught by schools.
I commented about conscientiousness. To the extent that the ‘virtue’ you describe here is still contentiousness you are not using a straw man. Conscientiousness and political savvy are the what is required for success and productive output in research.
Conscientiousness is what you need in order to finish what you start, when what you started is something that somebody else told you to do. When it’s your own thing, you need a lot less of it.
As for political savvy—that isn’t required at all. Unless by “research” you mean “political success in the human occupation customarily but misleadingly labeled ‘research’.” (The “as such” qualifier a few comments above was intended to rule that out.)
I really, really wish this were true. At some point in the process of doing your own things you are going to have to do work. Mundane details, repetition, parts of the process you don’t like. For example, you have to write up findings, crunch numbers, prepare details of any experiments you may be doing, double check stuff for reliability and proofread.
Not true. Political savvy makes a huge difference to your actual ability to produce research output.
Without political savvy, for example, I would never have been able to make use of the supercomputer that I needed for my research, or to get the grant money to keep me funded while doing it. (Most of that savvy was, of course, possessed by my Professor, mine was limited to building the connection needed to make him give me the role.)
From what I understand the material requirements are even more difficult to handle in other fields such as experimental physics or anything requiring actual human (or animal) test subjects.
It is also needed to keep others from outright interfering with what you are trying to do (I’ve run into problems there).
Without political support you aren’t able to do the research that you want to do. You are more likely to need to adapt and research the interests of others. This interferes with the conscientiousness bonus for working on personal projects.
If you can’t get others to support what you are doing you need to support yourself doing other things that others want you to do (ie. get a job.) That slows down your research.
Minions. Having those means you can be doing a lot more of the core research while others handle mundane tasks.
I don’t like it, but politics really does improve your ability to do research—and just about anything else.
I said less of this personality trait was required; I didn’t say zero.
Are you really disputing the notion that it takes less conscientiousness (other terms: “self-discipline”, “willpower”) to work on projects of one’s own choosing? That’s actually almost the definition of “one’s own choosing”: what one does by default.
Like I suspected, we’re talking past each other. Everything you say either pertains to the human occupation (and not just the act of coming up with good ideas, and maybe—with minimal Conscientiousness—writing them up in articles), or else is only the case because the system is set up in the suboptimal way it is.
Less conscientiousness, certainly. I believe we do have a factual disagreement regarding how much is still required.
I do wish the system (aka the universe) was set up a different way to what it is. For example if I had a team of catgirls with a ‘research assistant skills and motivational inspiration’ upgrade and a versatile fully stocked volcano-lair laboratory I’d be able to get heaps of research done. Well, after a few weeks when I got bored with the alternatives.
I am not sure your alternative is immediately practical.
But you guys are working on it, right?
http://www.pgbovine.net/PhD-memoir/pguo-PhD-grind.pdf demonstrates, I think, both the use of Conscientiousness and the politicking.
This strikes me as untrue for most people. Can you give me examples of people who were not conscientious and were nonetheless able to complete large, multi-step projects?
I actually find myself much more capable of finishing large, multi-step projects when they’ve got social implications riding on them. I enjoy my private projects more when I’m doing them, but have trouble finishing them if they take more than a few hours of serious work.
The first category does include things that others didn’t directly tell me to do, though: there’s also things I’m doing as favors to others, public-facing projects I came up with independently, and so on.
Not really the issue in this discussion, which is about the negative effects of a filtering system that excludes a certain small but highly valuable population.
As I’ve suggested earlier, EY is a pretty good example of the type of personality I have in mind.
Fair enough. The highly valuable ‘outliers’ are likely going to be different enough from me that I’ll have trouble mapping and comparing my traits onto theirs, which makes that kind of comparison not very useful.
You may know better than me, but as far as I can tell, EY does have the ability to coax himself into working productively on projects that aren’t necessarily a lot of fun all the time. He just won’t do it for any goal that that he doesn’t consider important. He strikes me more as someone who dislikes authority figures and cares less about the typical social reinforcement that comes of achieving more “conventional” goals, like going to university.
Yes, exactly. This is exactly the kind of story that such folks will tell about themselves.
Whereas, by contrast, the “conscientious” have enough willpower resources to spare for tasks that others consider “important” for them to do, as well.
Yes. People achieve better results if they cooperate, but they are judged by what they do if they are not allowed to cooperate.
This system has an advantage of safety—you can replace anyone in your organization and it keeps working, because everyone has some level of all the necessary skills. (You will never get stuck with the illiterate researcher and nobody to help them write.) So maybe I just underestimate the value of the safety. But maybe the system underestimates the opportunity costs.
At the level where students are required to write professionally, you can hire someone else to do the writing for you. For writing, they typically call it ‘dictation’, and it used to be standard, to the point that you still see “this dissertation was typed by the author” in dissertations without dictation. For writing correctly, they call it “editing” and many an advisor has had more influence over the actual wording and structure of the dissertation than the person who gets a PhD because of it.
This can be done to about the degree that it’s done in the actual professional life of a scientist: someone else can type your papers and grant proposals and make your presentations for you, and no one will know unless they read the acknowledgements. Typically that’s not done, or only done on a small scale, because for most people it takes as long to tell someone else how to write it as to write it yourself. When it is done- like when a friend of mine dictated his thesis and then edited it- no one cares, because they understand that it’s more efficient that way.
It also seems to me like it makes sense, since so much of science is communicating your results (and using the results of others). If a Gauss does great work, but leaves it in a desk drawer, what’s the point? Why would the establishment want to promote that rather than sharing, especially since individuals are so terrible at accurately judging their creative output without external feedback?
Disagree about the med school part. Doctors are always running into strange and urgent situations, having to come up with some tentative diagnosis and fix that’s more determined by what’s available than by taught best practice. Or at least often enough that they can blog about it a lot. Intelligence and training in a wide array of situations is necessary. Learning all about chemical mechanisms you’ll never influence at such a low level, not so much. The first years are also probably too general so switching to nursing is easier.
I think this covered in the point that Eliezer was making—why do we insist that the person who does our surgery has to be a Doctor (and thus capable of dealing with strange and urgent situations, tentative diagnoses, and so forth)? Why can’t we train this one person to be a surgeon, and isolate them from the source of all these strange situations—perhaps by putting some sort of specially-trained Strange and Urgent Situation Handler professional in between?
You mean have someone check before surgery that patients probably won’t have anything unusual happen, (not standing in the operating room going “let’s not do the surgery, there was a misdiagnosis”, which would cost even more)? That’s bad for patients who do look unexpected inside, and cases where the surgeon messes up in a way that didn’t come up during training. The cost may well still be lower than that of doctor training that rarely gets used.
There are specialized phlebotomists who do their job perfectly, and that’s probably feasible (maybe actually implemented?) for much minor surgery.
Out of genuine curiosity, how do you know that? I thought you never went to university.
Personal experience, most likely. What little I’ve seen / know of his knowledge indicates in-depth mastery of multiple topics that would each have taken five or more years of university courses to learn.
Having learned them all from university courses without special exception being made (that is, taking full-term courses without any skipping of courses or taking more than six courses per term) is highly improbable.
Many of my thought experiments into forming universities or educational institutions in general more geared towards optimized learning (e.g. open-learning systems where each student is at different levels in different subjects, and takes tests when milestones are reached rather than at specific predefined dates) seem to strongly indicate that while many of them would be much better for making more intelligent individuals or letting people learn much faster, the optimal utility-maximizing situation for the “Institutional Governing Body” is the current system. In other words, the individuals in positions of power to change the institutions have much more to gain (at least in the short term on their personal utility scales) in maintaining the current system. All my calculations, estimates and observations so far have consistently been in agreement with this statement, though I suspect a great deal of personal bias is at work here.
The closest thing I can think of is contacting people in the admissions department, but I can’t think of a cheap way to induce them to answer truthfully.
I’m also willing to consider humans part of the ‘system’, and so having that ‘file’ be “Bob recognized this applicant’s name” would be enough for my purposes. But I don’t know how much human attention their applicants get, and at what parts.
I’m fairly confident that this is a thing that actually exists, because of the associated prestige. Universities would get this if they were optimizing for status, without optimizing for learning at all.
However, you also have to consider marginal payoff relative to the cost. Most Science Fair winners will also score high according to the standard formula (involving GPA et cetera); any additional prestige the institution would gain by also admitting the very few who don’t probably wouldn’t be worth the cost of having such a separate system.
Unrelated to my other points: When in your experience have universities acted efficiently, as opposed to just “do things that sound like they’ll increase status”?
While in some senses I agree, the whole process of admissions just consists of people putting stamps on paper. If one of those people recognizes someone from a news article and just says “hey let’s stamp this” it doesn’t actually require more bureaucracy. Since all your processes are run by humans it doesn’t actually cost anything to add human judgment to your system.
For example, I would be EXTREMELY SURPRISED if there was a computer program that STOPPED a university from admitting someone if they had too low a GPA. It’s just that the computer program wouldn’t present them to be considered in the first place unless they looked.
In terms of practical tests, I propose that if we look up the set of Intel Science Fair winners, see if there’s information about their GPAs, and then look at what universities they got into, I hypothesize that if there are any with GPAs below, say, 3.7, they will still get in to high end universities that normally would only accept students with 4.0s (Stanford, MIT, Harvard, Carnegie Mellon, Johns Hopkins come to mind). I agree that it’s unlikely that you’ll find any with recorded GPAs below say 3.0, so the question may be purely theoretical anyway.
Yeah, I thought while reading the OP:
One could likely get into Harvard by winning a Nobel Prize, or being the guy from “Good Will Hunting”.
This method seems neither guaranteed nor widely known.
If you’ve already got a Nobel, what do you need to get into Harvard for?
Maybe you want to study something you’re not already an expert in? (For example, Feynman spent some time taking graduate biology courses.)
I just now read an interview which brings up the rise of negative selection in job applications:
Worse, for the ones that do, you’re probably just responding to noise. If it’s very improbable that any applicant will really match all of the screening criteria, then that can become smaller than the probability of a false positive.
Ahh, the The winner’s curse.
That’s weird, normally human judgement is worse than simple measures.
How much does this actually matter, I wonder? Is there really that big a difference between the best doctor in a group of 100 and the 10th best doctor in that same group? (The 10th best golfer in a tournament doesn’t take home the trophy, but the 10th best doctor in the hospital can still do a fine job treating a broken arm.)
In treating broken arms? Minimal difference.
In discovering new nanotechnology that will revolutionize the future of medicine? Literally all the difference in the world.
Then don’t you want the brilliant person to become a chemist or biologist instead of a physician or surgeon? You don’t need a medical license to work on cell cultures in a lab.
In some examples, like elite college admission, it seems more like there are both negative and positive controls. Although colleges use SAT scores and GPAs to weed out people who aren’t “good enough,” they also look at whether students are exceptional through supplemental essays or awards the student received in high school. Negative controls bar many students from admission, but in many cases, positive controls must also be used to select a final class out of acceptable applicants. I’m not in academia, but it seems similar. Although you may get off the “track,” you still need to do quality research to get a good position at the end of it. Even in top graduate programs, not all PhDs from those programs will end up finding academic jobs. At some point, you have to do more than just not fail.
This doesn’t seem like a fair generalization. At the undergraduate level, selection procedures are (and are rightly) negative. At the graduate level, things are very different across different fields and departments, but on the whole I think graduate admissions is a mix: people are weeded out until you have a small group of acceptable candidates, and then the exceptional ones are pulled out on the basis of their specific work. At the tenure level, you get a similar mix but there it’s very heavily in the direction of positive selection. Almost all of what a tenure committee is thinking about is the quality of your work (there are unfortunate exceptions to this, no doubt). Journal and book publishing are to an even greater extent a positive selection system, though again there are some unfortunate exceptions (like law).
So on the contrary, I think academia is largely a positive selection system. You’re description of the steps of academic success is accurate, but doesn’t provide evidence in the direction of positive or negative selection systems.
And one should also remember that positive selection systems tend to be enormously costly. With something like undergraduate admissions to sizable universities, positive selection is practically impossibile. Graduate admissions, where positive selection is at least present in the process (though it is still largely negative) requires months of work from a substantial portion of the whole department. Tenure hearings are also months in the making, but there you’re evaluating a single person.
So don’t knock negative selection too hard.
Very thought-provoking. Thank you!
How would we tell the difference between ‘positive’ and ‘negative’ selection? If I imagine that I’ve got two scores x and y (00.9 AND y>0.9′ what you mean by ‘negative selection’, and ‘accept if x>0.995 OR y>0.995’ positive selection?
If I’m thinking along the right lines here, is there a general principle (like ‘acceptance set must be convex’?), or do I have the wrong end of some crucial stick?
To be a bit less abstract, if x and y are sportiness and intelligence, and they’re uniform, then the AND rule gives you bright, fit people (i.e. the sort you naturally like), with a few superstars, whereas the OR rule gives you uber-nerds who may or may not be sporty and uber-jocks who may or may not be bright.
In fact it also might depend on how x and y are distributed. If they’re uniform then the difference between the AND and the OR rule feels less pronounced than if they’re gaussian. I think that the gaussian/AND case is going to give you very few people who are good at both, whereas the uniform scores case gives you some.
One word:
China.
One massive examination that determines your entire future? Isn’t that about as positive selection as you can get?
No. Positive selection means taking anyone who performs exceptionally well on any one out of many possible measures, whereas negative selection means taking anyone who performs at least passably on all of many measures. On the scale of how to integrate the information you get from each test, there only is one test, so you can’t distinguish between positive and negative selection.
On the scale of what the test itself measures, most exams tend to provide negative selection because the students usually know most of the material, so the smart kids can reliably get more than 90% of the questions correct, and the best way to do well involves having as few weak points as possible, rather than having some exceptionally strong abilities. So unless the one massive examination that they take in China is the Putnam, it’s probably more like negative selection.
Well, it is pretty hard from what I hear. I predict the hardest problems on the exam are as hard as easy-medium Putnam problems.
I suspect that “being rich enough to make a sufficiently large donation” will get you in (as long as you’ve got a high school diploma or GED). “Sufficiently large” may run in the hundreds of millions of dollars, though.
Rumor has it that it takes $5M to get accepted off the waitlist. If you don’t get waitlisted… I’m sure that something can be arranged, but think your expectation is probably a reasonable estimate.
I was thinking of the scene in the Rodney Dangerfield movie “Back to School” in which Rodney Dangerfield’s character, who became a wealthy businessman despite lacking a high school diploma, tries to enroll in the college in which his son is about to drop out of.
Wikipedia’s description of the scene in question: