I was reading the Slate Star Codex review of The Cult of Smart. Part of it discusses charter vs state schools and the allegations of fraud of various kinds undermining charter schools record of better achievement. Reading it, I realized that I took for granted that public schools engage in systematic fraud in a variety of ways. I don’t think this is something everyone understands, hence this post.
I went to a state school in the UK. State schools are rated on a 1 − 4 scale from unsatisfactory to outstanding. My school was rated good, meaning a 3. A few memories which stand out. During my first week I saw one of the boys in my class who was 11 at the time held up against the wall in a corridor while a 16 year old put a shiv to his throat and robbed him. He handed over his wallet and keys. A year or two later and I remember seeing a small boy who struggled with depression held up by the throat against a locker and slapped in the face by a troublemaker from the same class in front of everyone just before we went in to the classroom. I remember classes which were filled start to finish with people shouting and talking. Neither of the first two events were common but they also weren’t uncommon. No one was surprised to witness them. It’s worth emphasizing again that my school was above average, in fact quite far above average, and in a middle class area. It’s also worth noting that I was mostly in top ability streamed classes, meaning my classroom experience was likely far better than average.
There were many ways in which the school and teachers gamed the system to boost their measured performance. One way was to do exams for students. I was on a bottom set language class for French. After two years I literally couldn’t speak a single sentence in french and maybe knew 20 words in total. I still passed my exams. How? We did the tests in class. Often the teacher would go through them with us. Literally giving us the test and then going through each question on the whiteboard and telling us what to write. A different year and a different teacher, this time the teacher would sit next to us and write the answers down. Why sit next to us? It was the bottom set so people often wouldn’t even bother to write down the answer if they were told it. This kind of thing was normal, so much so that I, and I think most people there, didn’t realize anything unusual was happening.
Another way schools game metrics is to cheat inspections. A major component of how schools are judged in the UK is through independent inspections carried out by an independent quasi-governmental organization called Ofsted. Now, you may imagine that these inspections would be unannounced, so as to best get a real image of how a school works. Not the case. They’re scheduled well in advance. Before every inspection, a few things would happen in my school:
The worst troublemaker kids would be taken aside and put in a special room where inspectors wouldn’t see them. Either that or they would just be told not to come into school at all on that day.
All of us were told in assembly that an inspection was coming and to be on our best behavior on that day. Often teachers would have conversations with less serious troublemakers and impress on them that they would behave on that day or face consequences afterwards.
Teachers would put a great deal more effort into their lesson plans than was normal. Classroom behavior management would also be far stricter.
Because of these and other measures my school during an inspection was utterly different than my school on a normal day. On some level this isn’t surprising. If teachers’ promotions and management’s jobs depend on good inspection results and inspections are easy to game, people will game them. Incentives drive behavior. But it’s still sad.
Another way the stats were gamed was by not recording bad behavior. When a school gives a detention or suspends/expels a student, there’s a record of it. This is especially true of suspensions, students being sent home or expulsions. The more of these you have, the worse you look as a school. The solution then is obvious, don’t punish people or punish them in non-recorded ways. Again, in my school it was completely normal for students in lower sets to swear at the teacher, talk over them or disrupt the class for everyone else. It was normal for someone to be aggressive and abusive towards others and to face at most a 40 minute detention, but even getting a detention would be unusual.
I realize that one data point is not enough to draw solid general conclusions. My own perception is that this kind of fraud wasn’t specific to my school. My cousin went to a state school fairly nearby. He’s 4 years younger than me. During one of my winters back from undergrad we discussed his school and his experiences mirrored mine. His exact words regarding inspections were “I learned 4 times more that day than any other day that year. It was amazing”. I talked to a few British students at university, although specifically the not middle/upper class ones who would have gone to public schools. They had gone to schools similar to mine in different parts of the country and their stories were similar and often worse. Two particularly funny examples from my friends’ experiences stand out. A teacher in year 9 walked up to a student who was talking, picked them up and threw them out of an (open) first floor window. My friend sitting in class noticed two boys making fun of him and then proceeded to get up in the middle of class while the teacher was talking, walk to their table, flip the table upwards to hit them in the face before going to sit down again when the teacher told him to. (Remember, my friend was a studious, sporty Asian kid and not a troublemaker. This kind of thing is normal in that environment). Comedic stories aside, my experiences in school, while not universal, seem fairly common in the UK and from what I’ve read of the statistics, bad US schools are far, far worse.
I’m unsure what my point here is. I think I have two:
Charters may cook their books in various ways. In the UK, State schools do too. I would be surprised if it wasn’t also the case in the US.
I think that I feel like a lot of commentators on places like SSC have fairly middle class experiences of fairly good schools and that bleeds into how their comparison between state vs charter schools. It’s just good to remember that it’s not those nice middle class schools that charters typically replace.
How my school gamed the stats
I was reading the Slate Star Codex review of The Cult of Smart. Part of it discusses charter vs state schools and the allegations of fraud of various kinds undermining charter schools record of better achievement. Reading it, I realized that I took for granted that public schools engage in systematic fraud in a variety of ways. I don’t think this is something everyone understands, hence this post.
I went to a state school in the UK. State schools are rated on a 1 − 4 scale from unsatisfactory to outstanding. My school was rated good, meaning a 3. A few memories which stand out. During my first week I saw one of the boys in my class who was 11 at the time held up against the wall in a corridor while a 16 year old put a shiv to his throat and robbed him. He handed over his wallet and keys. A year or two later and I remember seeing a small boy who struggled with depression held up by the throat against a locker and slapped in the face by a troublemaker from the same class in front of everyone just before we went in to the classroom. I remember classes which were filled start to finish with people shouting and talking. Neither of the first two events were common but they also weren’t uncommon. No one was surprised to witness them. It’s worth emphasizing again that my school was above average, in fact quite far above average, and in a middle class area. It’s also worth noting that I was mostly in top ability streamed classes, meaning my classroom experience was likely far better than average.
There were many ways in which the school and teachers gamed the system to boost their measured performance. One way was to do exams for students. I was on a bottom set language class for French. After two years I literally couldn’t speak a single sentence in french and maybe knew 20 words in total. I still passed my exams. How? We did the tests in class. Often the teacher would go through them with us. Literally giving us the test and then going through each question on the whiteboard and telling us what to write. A different year and a different teacher, this time the teacher would sit next to us and write the answers down. Why sit next to us? It was the bottom set so people often wouldn’t even bother to write down the answer if they were told it. This kind of thing was normal, so much so that I, and I think most people there, didn’t realize anything unusual was happening.
Another way schools game metrics is to cheat inspections. A major component of how schools are judged in the UK is through independent inspections carried out by an independent quasi-governmental organization called Ofsted. Now, you may imagine that these inspections would be unannounced, so as to best get a real image of how a school works. Not the case. They’re scheduled well in advance. Before every inspection, a few things would happen in my school:
The worst troublemaker kids would be taken aside and put in a special room where inspectors wouldn’t see them. Either that or they would just be told not to come into school at all on that day.
All of us were told in assembly that an inspection was coming and to be on our best behavior on that day. Often teachers would have conversations with less serious troublemakers and impress on them that they would behave on that day or face consequences afterwards.
Teachers would put a great deal more effort into their lesson plans than was normal. Classroom behavior management would also be far stricter. Because of these and other measures my school during an inspection was utterly different than my school on a normal day. On some level this isn’t surprising. If teachers’ promotions and management’s jobs depend on good inspection results and inspections are easy to game, people will game them. Incentives drive behavior. But it’s still sad.
Another way the stats were gamed was by not recording bad behavior. When a school gives a detention or suspends/expels a student, there’s a record of it. This is especially true of suspensions, students being sent home or expulsions. The more of these you have, the worse you look as a school. The solution then is obvious, don’t punish people or punish them in non-recorded ways. Again, in my school it was completely normal for students in lower sets to swear at the teacher, talk over them or disrupt the class for everyone else. It was normal for someone to be aggressive and abusive towards others and to face at most a 40 minute detention, but even getting a detention would be unusual.
I realize that one data point is not enough to draw solid general conclusions. My own perception is that this kind of fraud wasn’t specific to my school. My cousin went to a state school fairly nearby. He’s 4 years younger than me. During one of my winters back from undergrad we discussed his school and his experiences mirrored mine. His exact words regarding inspections were “I learned 4 times more that day than any other day that year. It was amazing”. I talked to a few British students at university, although specifically the not middle/upper class ones who would have gone to public schools. They had gone to schools similar to mine in different parts of the country and their stories were similar and often worse. Two particularly funny examples from my friends’ experiences stand out. A teacher in year 9 walked up to a student who was talking, picked them up and threw them out of an (open) first floor window. My friend sitting in class noticed two boys making fun of him and then proceeded to get up in the middle of class while the teacher was talking, walk to their table, flip the table upwards to hit them in the face before going to sit down again when the teacher told him to. (Remember, my friend was a studious, sporty Asian kid and not a troublemaker. This kind of thing is normal in that environment). Comedic stories aside, my experiences in school, while not universal, seem fairly common in the UK and from what I’ve read of the statistics, bad US schools are far, far worse.
I’m unsure what my point here is. I think I have two:
Charters may cook their books in various ways. In the UK, State schools do too. I would be surprised if it wasn’t also the case in the US.
I think that I feel like a lot of commentators on places like SSC have fairly middle class experiences of fairly good schools and that bleeds into how their comparison between state vs charter schools. It’s just good to remember that it’s not those nice middle class schools that charters typically replace.
Crossposted to my blog at https://dissent.blog/2021/02/20/how-my-school-gamed-the-stats/