I don’t have immediate and useful insights or revelations, but the one thing that really jumped out for me, reading this, was the sense that two things are being conflated: people’s confidence that the quality of the work met or exceeded Bar X, and people’s different senses of where Bar X is.
I don’t know for sure that this will end up being a useful avenue to pursue, to find solutions, but my claim is something like “the real problem isn’t just people disagreeing about whether you met the bar, it’s about people disagreeing about the actual grade out of 100, with ‘what makes for a passing grade here?’ being an additional, second, obfuscating question.”
Let’s say I’m building a giant sphere in the desert.
Lots of people can grade my design
, but ultimately I want to know the probability that it’s going to buckle, especially while under load.
Or, say I’m helping run a youth camp on X-risk.
Lots of people can grade my performance, but ultimately I want to know whether the event successfully taught the kids anything useful, especially related to saving the world.
In the case of the sphere, people can say “yeah it didn’t buckle, but that’s just because you were lucky”
. In the case of the youth camp, people can say “yeah it worked, but that was despite you not because of you”
.
But what I want to know is, if we ran ten thousand youth camps, and we put me in five thousand of them and kept me out of five thousand of them, what would they look like?
Or if we built sixteen thousand spheres, and eight thousand of them used grade 8 bolts and eight thousand used grade 2 bolts, how many on each side buckle, and how many injuries are inflicted in each bucket?
Because “grade my performance” is, and always has been, utter fucking bullshit. I care about you (you in particular, Duncan) grading my performance because I trust you to know enough to calibrate my grade to the numbers I ACTUALLY care about
, most of the time. And when I don’t trust you to calibrate, I stop caring about your grade, because I don’t need anyone to suck my dick about how well I did.
But I DO care about whether what I’m doing is actually doing good for people, and most of the time I have less evidence of whether that’s true than they do—but most of the time they seem to not be attending to that evidence themselves; instead, they’re checking to see if I’m behaving like the sort of monkey that “should” be seen as doing good things. And that is UTTERLY INFURIATING.
I don’t have immediate and useful insights or revelations, but the one thing that really jumped out for me, reading this, was the sense that two things are being conflated: people’s confidence that the quality of the work met or exceeded Bar X, and people’s different senses of where Bar X is.
I don’t know for sure that this will end up being a useful avenue to pursue, to find solutions, but my claim is something like “the real problem isn’t just people disagreeing about whether you met the bar, it’s about people disagreeing about the actual grade out of 100, with ‘what makes for a passing grade here?’ being an additional, second, obfuscating question.”
Regarding grades and bars:
Let’s say I’m building a giant sphere in the desert.
Lots of people can grade my design , but ultimately I want to know the probability that it’s going to buckle, especially while under load.
Or, say I’m helping run a youth camp on X-risk.
Lots of people can grade my performance, but ultimately I want to know whether the event successfully taught the kids anything useful, especially related to saving the world.
In the case of the sphere, people can say “yeah it didn’t buckle, but that’s just because you were lucky” . In the case of the youth camp, people can say “yeah it worked, but that was despite you not because of you” .
But what I want to know is, if we ran ten thousand youth camps, and we put me in five thousand of them and kept me out of five thousand of them, what would they look like?
Or if we built sixteen thousand spheres, and eight thousand of them used grade 8 bolts and eight thousand used grade 2 bolts, how many on each side buckle, and how many injuries are inflicted in each bucket?
Because “grade my performance” is, and always has been, utter fucking bullshit. I care about you (you in particular, Duncan) grading my performance because I trust you to know enough to calibrate my grade to the numbers I ACTUALLY care about , most of the time. And when I don’t trust you to calibrate, I stop caring about your grade, because I don’t need anyone to suck my dick about how well I did.
But I DO care about whether what I’m doing is actually doing good for people, and most of the time I have less evidence of whether that’s true than they do—but most of the time they seem to not be attending to that evidence themselves; instead, they’re checking to see if I’m behaving like the sort of monkey that “should” be seen as doing good things. And that is UTTERLY INFURIATING.