Noticing a Teacher’s Password Pattern
Yudkowsky writes about Guessing the Teacher’s Password as an abstract educational concept. At a young age, perhaps ten years old, I had guessed one commonly used meta-password: In the Finnish school system it’s typical for multiple choice answers to include options that are somewhat similar, and often the actual answer can be reasoned without knowing much at all about the actual topic. Here’s an example from 2024 admission exam for technical universities. I know no chemistry beyond elementary school, and you might not know any Finnish. That matters not:
Naively one might think that repeating the PHV-thing seen in the description would mean picking D. Unfortunately, we have better tools: teachers generate incorrect answers by either taking completely nonsensical things, or by varying only one feature of the correct answer at a time.
Just by looking at the pictures, we can see that D doesn’t share the right-hand OH group every other compound has. So that’s not the answer. Next we see that B is missing the downwards-going carbon branch (and so is D). So that is not the answer, either. We’re left deciding between A and C. But A and B share the same squiggly mid-lane carbons. So the answer must be A.
I’m using overly strong terms here. Not every exam or every teacher uses this format. But so many do that it’s extremely useful to notice this. In fact, this that’s the case here: the actual answer is B. Still, it clearly point us to the fact that D, which is most dissimilar to others but really close to the given example is wasn’t the right answer.
Let’s do another one, this time with text only, from the 2022 admission exam:
Again, we can simply look into the textual structure. All except B share the same first number, so it’s not B. But the option A shares the same numbers as B, except that the order has been swapped. This means that the answer must be A.
In both these cases, the logic gives the correct answer. But not all questions are like this. If the options do not have the structure that has these similarities, using this method will not work. But at least I’ve learned to recognize this form over the years. Even though it’s quite reliable, I normally wouldn’t answer the questions using it. But if that doesn’t match my calculated answer, I’d double- and triple-check before accepting it.
This reminds me quite a bit about pattern recognition in general IQ testing. Not sure what to think about that, yet. It would be a mistake to teach this trick to people who haven’t noticed it themselves; I’m pretty sure such clever tricks hindered my studies at least a bit. A mild infohazard, even, perhaps. I still feel rather comfortable publishing this here, take that as you wish.
The actual answer is B, not A. The atom of carbon connected with 1 hydrogen is at distance 1 from the hydroxyl group, which rules out A. D is not an acid and C has the wrong distance between the groups -OH and -COOH.
Good catch. I checked the answer sheet, and that is indeed the answer. (I though I had checked it before publishing, but apparently I checked it wrong). I’ll edit the post to reflect this, at least it’s a nice example on how the technique fails subtly.
Do you actually know the answer here, or are you just parroting a LLM?
I solved the problem, then doublechecked it with the LLM. In addition, you can check the logic by yourself.
Nice try, but the chemistry question is asking for the structure of the monomer given the polymer. It doesn’t take much chemistry to figure out the answer is actually B: just flip the repeated unit horizontally, and add water (-H on one side, -OH on the other).
Bonus: we can apply a similar level of understanding of Finnish to translate poly-3-hydroksivaleraatti to “poly-3-hydroxyvalerate,” whose monomer is 3-Hydroxyvaleric acid.
This is an embarrassingly large part of the reason I was considered good at school. I did generally learn the material though, but forgot much of it afterward :(
Did that inform what field you decided to go into later in life?
For me it definitely correlated with liking physics and math, where things can often be reasoned out from first principles.
I also like math and computer science and got a degree but unfortunately couldn’t find any careers there 🙁
Fair enough. I haven’t worked as a researcher since before (and during, I suppose) grad school. And while my career has used that knowledge, I’ve also been out of work for eight months, so there’s that.
I comparable password-pattern I noticed: Options that are clearly longest or most difficult to spell are more likely to be the correct answer for multiple choice type questions (and conversely: Options that are short and easy to spell compared to others are less likely to be correct).
My reasoning is that people tend to spend less energy on less important things, hence won’t take as much effort (time and focus to spell) when creating the bad options.
This is very similar to lectio difficilior potior in textual criticism! I.e. if you have no other evidence, then the difficult reading is the stronger one.
Tangentially, I noticed the strongest version of this was when the testmaker added some sort of clarification or hedging to one of the answers (like stating “all other things being equal” or the like). Very unlikely you’d just add that to a wrong answer!