The use case that I usually gravitate towards is that of the spread of news.
One other domain which is particularly vulnerable to OBP is the organization of software product development projects. Common proxies include “productivity” or “man-hours” or “defects”, as opposed to the underlying goal which is generally some instance of the class “actually solving a problem for some number of actual people”, which is just as hard to actually measure as “page quality” in the search space.
This has been the subject of an entire discipline, called “software engineering”, for a little over 40 years now (the term was coined in 1968), and I’d summarize the results as “disappointing”.
Your discussion of “distributed” mitigations to the downsides of OBP brings to mind one promising approach known as the “Theory” of Constraints (scare quotes owed to my not being quite sure the term theory is appropriate), which among other things emphasizes decentralized decision making, the anarcho-syndicalist branch of “human discretion” approaches.
The idea is that the kind of work that software development consists of can be approximated as a steady stream of small but significant decisions, where the consequences of inconsistent decisions can be dire. In this context, the people doing the work have the best access to the kind of information that motivates big (“executive”) decisions, but the usual approaches to software engineering grant these people little to no authority over big decisions. Conversely, they are given generally poor guidance on the small but significant decisions.
In practice, this means for instance that instead of “project manager assigns coding tasks to engineers” which is rife with opportunities for OBP, the more promising approaches tend to rely on “engineers sign up for tasks they feel maximize their contribution”.
Another implication is that a posteriori control over the quality of decisions is preferred over a priori control: feedback trumps planning. This is again similar to the Wikipedia model where everyone can edit and is trusted to do so, but violators of the implicit contracts are quickly identified and their damage repaired (or, at least, that’s how it’s supposed to work; there are many who suspect this is an idealization publicized as the real thing for marketing purposes).
Recommended reading: Elyahu Goldratt’s The Goal for the background, books in the “lean/agile” space for specific applications—I’ve read and liked the Poppendiecks’.
I have been toying with the idea of making a catalogue of all the instances of OPB that have been identified. Software engineering was in my mind, but not nearly to the amount of detail that you set it to. I suppose DVCS systems like Git also take software enginering (open source especially) in that direction.
Also, I love this phrase:
Another implication is that a posteriori control over the quality of decisions is preferred over a priori control: feedback trumps planning.
It really gets through very clearly. Don’t be surprised if I steal it :)
I interpreted RobinZ to mean “any sit down with paper and a writing instrument” test. Driving tests are “standardized” in some sense, but still seem to be good proxies in a way that the SAT is not.
One other domain which is particularly vulnerable to OBP is the organization of software product development projects. Common proxies include “productivity” or “man-hours” or “defects”, as opposed to the underlying goal which is generally some instance of the class “actually solving a problem for some number of actual people”, which is just as hard to actually measure as “page quality” in the search space.
This has been the subject of an entire discipline, called “software engineering”, for a little over 40 years now (the term was coined in 1968), and I’d summarize the results as “disappointing”.
Your discussion of “distributed” mitigations to the downsides of OBP brings to mind one promising approach known as the “Theory” of Constraints (scare quotes owed to my not being quite sure the term theory is appropriate), which among other things emphasizes decentralized decision making, the anarcho-syndicalist branch of “human discretion” approaches.
The idea is that the kind of work that software development consists of can be approximated as a steady stream of small but significant decisions, where the consequences of inconsistent decisions can be dire. In this context, the people doing the work have the best access to the kind of information that motivates big (“executive”) decisions, but the usual approaches to software engineering grant these people little to no authority over big decisions. Conversely, they are given generally poor guidance on the small but significant decisions.
In practice, this means for instance that instead of “project manager assigns coding tasks to engineers” which is rife with opportunities for OBP, the more promising approaches tend to rely on “engineers sign up for tasks they feel maximize their contribution”.
Another implication is that a posteriori control over the quality of decisions is preferred over a priori control: feedback trumps planning. This is again similar to the Wikipedia model where everyone can edit and is trusted to do so, but violators of the implicit contracts are quickly identified and their damage repaired (or, at least, that’s how it’s supposed to work; there are many who suspect this is an idealization publicized as the real thing for marketing purposes).
Recommended reading: Elyahu Goldratt’s The Goal for the background, books in the “lean/agile” space for specific applications—I’ve read and liked the Poppendiecks’.
I have been toying with the idea of making a catalogue of all the instances of OPB that have been identified. Software engineering was in my mind, but not nearly to the amount of detail that you set it to. I suppose DVCS systems like Git also take software enginering (open source especially) in that direction.
Also, I love this phrase:
It really gets through very clearly. Don’t be surprised if I steal it :)
Classic example of optimization by proxy: “fill-in-the-bubble” standardized tests taken by students, such as the SATs.
Not just fill-in-the-bubble—any standardized test, really.
Driving tests are pretty good proxies. Except for the part where they make you parallel park between poles instead of between cars.
This is a great example.
I interpreted RobinZ to mean “any sit down with paper and a writing instrument” test. Driving tests are “standardized” in some sense, but still seem to be good proxies in a way that the SAT is not.
That is what I meant, yes.
I think the SAT is a better proxy than driving tests. I see very little effort to produce good driving tests, nor incentive to do so.