“About thirteen or fourteen years ago, when I was still an academic, I interviewed for a [psychology] faculty position at a fancy university in Canada. [...] There were four applicants, each hosted by a different committee member. I had a grand time with my host. He was very candid about the whole process. At one point he told me that after the candidates had all come and gone, the committee members would go into a small room, sit down, and proceed to argue with one another for several hours over utterly ridiculous things each candidate said or did, until eventually everyone was worn out and just settled for whichever candidate they were all able to barely tolerate.”
“I asked why the committee didn’t just apply the lessons of seventy years of industrial-organizational psychology: come up ahead of time with a scoring rubric; have everyone independently fill it out without talking to each other; then mechanically aggregate the scores and offer the job to the person with the biggest number.”
“He said (and he was right), “well, obviously, we don’t believe our own methods.””
“You could try to write it off as just a dysfunctional department, but… it wasn’t. They’re all like that. I’ve never actually seen a psychology department that hires people the way that I-O psychology tells us you should hire people. A friend once told me that the University of Minnesota did use to do it that way, for a bit, in the days when Paul Meehl was still walking around the halls. I don’t know if that’s true, but I do know that Paul Meehl died in 2003. I interviewed at UMN in 2012, and they sure as shit weren’t doing it that way then.”
“Anyway, there is no deep moral here, only some sadness. Not for the rejections (there are always rejections, and I did just fine, academic job-wise), but because this is just what we’re like! We’re all so bad at this. We talk a good game about science and data and whatever, but as soon as any of that stuff impinges on our personal preferences, we immediately peace out of scienceland.”
“well, obviously, we don’t believe our own methods.”
I am reminded of Jo Freeman’s “The Tyranny of Structurelessness” and Rush’s “Freewill”.
Refusing to follow a documented method is not the same as not having a method. You still have a method! It’s just the method of — what was it? Oh, right:
[G]o into a small room, sit down, and proceed to argue with one another for several hours over utterly ridiculous things each candidate said or did, until eventually everyone was worn out and just settled for whichever candidate they were all able to barely tolerate.
If you’re going to disbelieve in “[y]our own methods”, why believe in that method? Why believe in the “structureless” method, the method of personal exhaustion, which likely disadvantages absolutely every other possible value?
Contrary to what we would like to believe, there is no such thing as a structureless group. Any group of people of whatever nature that comes together for any length of time for any purpose will inevitably structure itself in some fashion. [...]
For everyone to have the opportunity to be involved in a given group and to participate in its activities the structure must be explicit, not implicit. The rules of decision-making must be open and available to everyone, and this can happen only if they are formalized. This is not to say that formalization of a structure of a group will destroy the informal structure. It usually doesn’t. But it does hinder the informal structure from having predominant control and make available some means of attacking it if the people involved are not at least responsible to the needs of the group at large. “Structurelessness” is organizationally impossible. We cannot decide whether to have a structured or structureless group, only whether or not to have a formally structured one.
— Jo Freeman, “The Tyranny of Structurelessness” (1970)
If you choose not to decide, you still have made a choice!
1 point of counterevidence: I did get hired at a psychology department after doing profile testing. The test was solely an extended HEXACO (big five + 1) and the professor actually came up with an insightful prediction about my fit for the role. I don’t think most bussinesses just run HEXACOs on people though.
Was it a normal HEXACO, meaning pure self-report? I’m not sure if basing a hiring decision on that alone is great either, since applicants can trivially bias their answers toward what they think the employer wants, consciously or unconsciously. (And even if the employer is looking for something else than the applicants think, this will still distort the results.)
The HEXACO has (presumably imperfect) measures of validity that specifically measure how desirable ones answers are.
I agree it can still be biased, but it was in addition to the interview. Interviews have the same problem of people presenting themselves possibly better than they are. I think faking everything together is roughly equally difficult to do well
Given this, then they are depending on their group level of hiring taste, or on one or two people with good hiring taste being able to dominate the group’s decision. Either way, hiring taste erodes quickly when diluted by bad hires. So they seem F’d even beyond their “we don’t believe our own findings” existential crisis, because their likely bad hiring taste will lead to an equilibrium where they can’t create better findings. :D
sounds effective to me (depending on individuals), moreso than many corpo selection approaches. extended freeform discussion of small-ish group of domain experts with some familiarity of each others’ preferences/strengths/gaps etc, remit to explore whatever needs to be explored.
writ broadly, many discussions fail because they don’t surface crux signal. combination of domain experts, familiarity with others, culture of openness (described by texture, not touted) - this seems pretty baller
ETA: many rubrics etc have their sweet spot in circumstances where the above luxuries do not exist, particularly openness and expertise
“About thirteen or fourteen years ago, when I was still an academic, I interviewed for a [psychology] faculty position at a fancy university in Canada. [...] There were four applicants, each hosted by a different committee member. I had a grand time with my host. He was very candid about the whole process. At one point he told me that after the candidates had all come and gone, the committee members would go into a small room, sit down, and proceed to argue with one another for several hours over utterly ridiculous things each candidate said or did, until eventually everyone was worn out and just settled for whichever candidate they were all able to barely tolerate.”
“I asked why the committee didn’t just apply the lessons of seventy years of industrial-organizational psychology: come up ahead of time with a scoring rubric; have everyone independently fill it out without talking to each other; then mechanically aggregate the scores and offer the job to the person with the biggest number.”
“He said (and he was right), “well, obviously, we don’t believe our own methods.””
“You could try to write it off as just a dysfunctional department, but… it wasn’t. They’re all like that. I’ve never actually seen a psychology department that hires people the way that I-O psychology tells us you should hire people. A friend once told me that the University of Minnesota did use to do it that way, for a bit, in the days when Paul Meehl was still walking around the halls. I don’t know if that’s true, but I do know that Paul Meehl died in 2003. I interviewed at UMN in 2012, and they sure as shit weren’t doing it that way then.”
“Anyway, there is no deep moral here, only some sadness. Not for the rejections (there are always rejections, and I did just fine, academic job-wise), but because this is just what we’re like! We’re all so bad at this. We talk a good game about science and data and whatever, but as soon as any of that stuff impinges on our personal preferences, we immediately peace out of scienceland.”
-- Tal Yarkoni
I am reminded of Jo Freeman’s “The Tyranny of Structurelessness” and Rush’s “Freewill”.
Refusing to follow a documented method is not the same as not having a method. You still have a method! It’s just the method of — what was it? Oh, right:
If you’re going to disbelieve in “[y]our own methods”, why believe in that method? Why believe in the “structureless” method, the method of personal exhaustion, which likely disadvantages absolutely every other possible value?
— Jo Freeman, “The Tyranny of Structurelessness” (1970)
— Rush, “Freewill” (1980)
1 point of counterevidence: I did get hired at a psychology department after doing profile testing. The test was solely an extended HEXACO (big five + 1) and the professor actually came up with an insightful prediction about my fit for the role. I don’t think most bussinesses just run HEXACOs on people though.
Was it a normal HEXACO, meaning pure self-report? I’m not sure if basing a hiring decision on that alone is great either, since applicants can trivially bias their answers toward what they think the employer wants, consciously or unconsciously. (And even if the employer is looking for something else than the applicants think, this will still distort the results.)
The HEXACO has (presumably imperfect) measures of validity that specifically measure how desirable ones answers are.
I agree it can still be biased, but it was in addition to the interview. Interviews have the same problem of people presenting themselves possibly better than they are. I think faking everything together is roughly equally difficult to do well
(I tried putting this in a blockquote originally but then the quote didn’t seem to display on the frontpage)
Blockquotes are intentionally hidden or replaced with [...] when the comment is collapsed
https://github.com/ForumMagnum/ForumMagnum/commit/d3b0e3cdb4179258b929601cd62b08ca80029e81
Is the takeaway meant to be that I-O psychology is bunk, or that psychology professors are deeply irrational, or maybe both?
I took it to mean that the professors are, if not irrational, then putting their own preferences above what’s good for the organization.
Given this, then they are depending on their group level of hiring taste, or on one or two people with good hiring taste being able to dominate the group’s decision. Either way, hiring taste erodes quickly when diluted by bad hires. So they seem F’d even beyond their “we don’t believe our own findings” existential crisis, because their likely bad hiring taste will lead to an equilibrium where they can’t create better findings. :D
sounds effective to me (depending on individuals), moreso than many corpo selection approaches. extended freeform discussion of small-ish group of domain experts with some familiarity of each others’ preferences/strengths/gaps etc, remit to explore whatever needs to be explored.
writ broadly, many discussions fail because they don’t surface crux signal. combination of domain experts, familiarity with others, culture of openness (described by texture, not touted) - this seems pretty baller
ETA: many rubrics etc have their sweet spot in circumstances where the above luxuries do not exist, particularly openness and expertise