do some armchair theorizing and spend 10 seconds (or even a few hours) thinking about why people who make X aren’t actually doing X very well. You can’t understand why.
Conclude: “X is simple, and the obvious answer is obvious”.
And to make things even more complicated, sometimes there is a problem that indeed could be solved by 10 seconds of thinking, but behind it there is another problem which is taboo to mention, which cannot be solved so easily. So everyone involved ends up pretending that the 10-second solution is insufficient to solve the original problem, because it is their only way out of “we cannot implement this solution” and “but we also cannot explain why”.
.
To give an example from my experience, once I was given a task to organize entrance exams for a university. There were many thousand candidates; they couldn’t even fit into the school building on the same day, so the testing took several days, I think always one group in the morning and one group in the afternoon. The questions were all of the “choose: a, b, c, or d” form, and it was still an enormous amount of work to check all the tests. (This was a few decades ago when computers were a new thing, so all those tests were checked manually.)
But the main problem was that the questions and answers have always leaked. With thousands of students in the building and testing taking multiple days you couldn’t really prevent it, because all that was needed was to send a few fake candidates whose task was to grab the problems or make a photo on an early day, then they could solve the problems in their free time, and sell the answers to students who took the test on a later day. Those students just memorized the answers like this: “acbdcadb...” or used some simple code to make a tiny cheat sheet (two bits per answer do not require much space).
So far the best solution was to do A and B versions of the test, so that the students sitting next to each other would get different versions, which prevented them from checking each other’s answers, and made memorizing the answers slightly more difficult. (Even this was frequently defeated by students exchanging their question sheet with someone in a different row, so that they both had the same version as their neighbors. There were too many students and too few teachers to check all of this.)
My solution was simple: instead of doing versions A and B, I did 4 different versions every day, and different 4 versions the next day, and so on until the end of exams. So the first day, the first row got alternating A, B, A, B..., the second row got alternating C, D, C, D..., the third row A, B again, etc. So exchanging the questions with someone one row away from you didn’t help you get the same questions as your neighbors. (Two rows would be more conspicuous. Also, the students did not expect this; which probably would be less true the following years.) The fake candidates now had to collect four versions of the problems… and even that didn’t help them, because the next day it was versions E, F, E, F… for the odd rows, and G, H, G, H… for the even rows.
How could I make so many variants with so many questions? Wasn’t that much more work than usual? Well, it was more work, but not an order of magnitude more work. I asked teachers to prepare more questions than usual, but no specific numbers, just “make as many as you comfortably can”. Then, every variant was randomly selected questions, and even the answers a, b, c, d were shuffled randomly. That means, even if two variants got the same question, in one of them it could be question #7 and the correct answer was “b”, but on another it was question #13 and the correct answer was “c”. So you couldn’t defeat this system by memorizing the letters “acbdcadb...”; you would have to actually learn the questions and the right answers. And there were more questions than usual.
To make evaluating the tests easier, my program that prepared the tests also prepared for each variant tiny strips of paper with the right answers that you could put next to the test, and just check whether the answer match or not. So evaluating the tests was actually easier than the previous years.
And… it was a huge success, all people involved in evaluating the tests got a bonus, and a few people congratulated me on the successful organization of something they believed would be utter chaos.
And… the next year my system was abandoned; the university returned to the original way of doing entrance exams, and the debate about improving the exams became a taboo.
As far as I know, although I can only guess here, it is because the answers were actually regularly leaked by some very high-status employees of the university, and my experiment has ruined their nice side business for one year. So they invented bullshit reasons about why this isn’t a good way to do exams (the people who did them with me agreed that the reasons were bullshit, e.g. “because it is much more work” but everyone who actually did the work agreed that it was less work than usual), and thus the debate was officially concluded. “You see, we tried your experiment, but it failed.”
So, some problems cannot be solved simply because there are important people who don’t want them to get solved. But the official answer is still that your solution wouldn’t work. Trust the experts!
A more abstract reason why this irritates me is that it is an example of a more general pattern that goes like this:
Try a half-assed implementation of X.
It fails predictably.
Conclude: “X was experimentally disproved.”
If anyone suggests to do X properly, say: “true socialism has never been tried, comrades, right?”
Sometimes I wonder whether each popular fallacy has its opposite which is also a popular fallacy.
Maybe I’m getting too far off-topic but there is another version of this that goes:
Do some armchair theorizing where you spend 10 seconds thinking about how to make X work and you can’t think of anything.
Conclude: “X is impossible.”
This also works in reverse:
do some armchair theorizing and spend 10 seconds (or even a few hours) thinking about why people who make X aren’t actually doing X very well. You can’t understand why.
Conclude: “X is simple, and the obvious answer is obvious”.
And to make things even more complicated, sometimes there is a problem that indeed could be solved by 10 seconds of thinking, but behind it there is another problem which is taboo to mention, which cannot be solved so easily. So everyone involved ends up pretending that the 10-second solution is insufficient to solve the original problem, because it is their only way out of “we cannot implement this solution” and “but we also cannot explain why”.
.
To give an example from my experience, once I was given a task to organize entrance exams for a university. There were many thousand candidates; they couldn’t even fit into the school building on the same day, so the testing took several days, I think always one group in the morning and one group in the afternoon. The questions were all of the “choose: a, b, c, or d” form, and it was still an enormous amount of work to check all the tests. (This was a few decades ago when computers were a new thing, so all those tests were checked manually.)
But the main problem was that the questions and answers have always leaked. With thousands of students in the building and testing taking multiple days you couldn’t really prevent it, because all that was needed was to send a few fake candidates whose task was to grab the problems or make a photo on an early day, then they could solve the problems in their free time, and sell the answers to students who took the test on a later day. Those students just memorized the answers like this: “acbdcadb...” or used some simple code to make a tiny cheat sheet (two bits per answer do not require much space).
So far the best solution was to do A and B versions of the test, so that the students sitting next to each other would get different versions, which prevented them from checking each other’s answers, and made memorizing the answers slightly more difficult. (Even this was frequently defeated by students exchanging their question sheet with someone in a different row, so that they both had the same version as their neighbors. There were too many students and too few teachers to check all of this.)
My solution was simple: instead of doing versions A and B, I did 4 different versions every day, and different 4 versions the next day, and so on until the end of exams. So the first day, the first row got alternating A, B, A, B..., the second row got alternating C, D, C, D..., the third row A, B again, etc. So exchanging the questions with someone one row away from you didn’t help you get the same questions as your neighbors. (Two rows would be more conspicuous. Also, the students did not expect this; which probably would be less true the following years.) The fake candidates now had to collect four versions of the problems… and even that didn’t help them, because the next day it was versions E, F, E, F… for the odd rows, and G, H, G, H… for the even rows.
How could I make so many variants with so many questions? Wasn’t that much more work than usual? Well, it was more work, but not an order of magnitude more work. I asked teachers to prepare more questions than usual, but no specific numbers, just “make as many as you comfortably can”. Then, every variant was randomly selected questions, and even the answers a, b, c, d were shuffled randomly. That means, even if two variants got the same question, in one of them it could be question #7 and the correct answer was “b”, but on another it was question #13 and the correct answer was “c”. So you couldn’t defeat this system by memorizing the letters “acbdcadb...”; you would have to actually learn the questions and the right answers. And there were more questions than usual.
To make evaluating the tests easier, my program that prepared the tests also prepared for each variant tiny strips of paper with the right answers that you could put next to the test, and just check whether the answer match or not. So evaluating the tests was actually easier than the previous years.
And… it was a huge success, all people involved in evaluating the tests got a bonus, and a few people congratulated me on the successful organization of something they believed would be utter chaos.
And… the next year my system was abandoned; the university returned to the original way of doing entrance exams, and the debate about improving the exams became a taboo.
As far as I know, although I can only guess here, it is because the answers were actually regularly leaked by some very high-status employees of the university, and my experiment has ruined their nice side business for one year. So they invented bullshit reasons about why this isn’t a good way to do exams (the people who did them with me agreed that the reasons were bullshit, e.g. “because it is much more work” but everyone who actually did the work agreed that it was less work than usual), and thus the debate was officially concluded. “You see, we tried your experiment, but it failed.”
So, some problems cannot be solved simply because there are important people who don’t want them to get solved. But the official answer is still that your solution wouldn’t work. Trust the experts!