If you allowed better capture of the upside then it would make sense to make them own more downside.
I thought Owen made a good case in the podcast that we currently have more mechanisms in place to fix/workaround the “insufficient capture of the upside” problem than the “insufficient capture of the downside” problem, as far as scientific research is concerned. (See also the related paper.) I would be interested to see the two of you engage each other’s arguments directly.
The intention of the last line was, avoid using asymmetric mental point systems except where structurally necessary, and be-a-conclusion.
Do you have an explanation of why we currently are often using asymmetric mental point systems when it’s not structurally necessary? My general expectation is that when it comes to deficiencies in human group rationality, there are usually economic / game theoretic reasons for them to exist, so you can’t fix it by saying “just don’t do that”.
Is it worth the bandwidth to get into the weeds on this? To me, saying “we currently have mechanisms with which to solve X” matters little if X is not being solved in this way. I certainly don’t see how ‘put all the downside on the researcher’ could possibly be matched, since you’re certainly not going to give them most or all of the upside—again we don’t even come close to doing that for drugs that can be sold at monopoly prices, and that’s before giving everyone along the way their cuts.
Second:
I have at least some reasons, of varying degrees of being good reasons. The best reason I can think of for why it is good, would be that it opens the door for lots of larger manipulations, and might put even greater burdens on people to constantly point out the good things they’re doing to collect all the points from them to offset where they get docked or otherwise score highly. Whereas now you only have to avoid bad things being pointed out. Or alternatively, that when people claim good things they have obvious bad incentives to do that, so you’re inclined to not believe them. And that we don’t have time to find all the context, and need to act on simple heuristics due to limited compute. And in some places, the willingness to *ever* do a sufficiently bad thing is very strong evidence of additional bad things, and we need to maintain a strong norm of always punishing an action to maintain a strong norm against that action.
Also potentially important is that if you let things get fuzzy, those with power will use that fuzziness to enhance their own power. When needed, they’ll find ways to give themselves points to offset any bad things they’re caught doing. You need a way to stop this and bring them down.
And so on.
So in some places it becomes structurally necessary to have a no-excuses (or only local and well-specified excuses like self defense) approach. But there are entire cultural groups who use this as the generic evaluate-thing algorithm and that’s terrible.
That’s why I chose the phrasing “aim higher” rather than telling people “don’t do that.” I don’t think one can entirely eliminate such systems at this time at a reasonable price.
But this is also of the type of thing that I do when I’m analyzing my game play choices after a match of Magic, where I come up with all sorts of explanations and deep lines of possibility and consideration that were never in my conscious analysis at the time. At the time it was more something like, this needs a conclusion, I’ve shown the problems with this thing, this seems like a way to wrap things up and maybe get people to think about doing the thing less and spotting/discounting it more, which would be good.
(I will continue this line of thought down below in another reply)
I thought Owen made a good case in the podcast that we currently have more mechanisms in place to fix/workaround the “insufficient capture of the upside” problem than the “insufficient capture of the downside” problem, as far as scientific research is concerned. (See also the related paper.) I would be interested to see the two of you engage each other’s arguments directly.
Do you have an explanation of why we currently are often using asymmetric mental point systems when it’s not structurally necessary? My general expectation is that when it comes to deficiencies in human group rationality, there are usually economic / game theoretic reasons for them to exist, so you can’t fix it by saying “just don’t do that”.
First point:
Is it worth the bandwidth to get into the weeds on this? To me, saying “we currently have mechanisms with which to solve X” matters little if X is not being solved in this way. I certainly don’t see how ‘put all the downside on the researcher’ could possibly be matched, since you’re certainly not going to give them most or all of the upside—again we don’t even come close to doing that for drugs that can be sold at monopoly prices, and that’s before giving everyone along the way their cuts.
Second:
I have at least some reasons, of varying degrees of being good reasons. The best reason I can think of for why it is good, would be that it opens the door for lots of larger manipulations, and might put even greater burdens on people to constantly point out the good things they’re doing to collect all the points from them to offset where they get docked or otherwise score highly. Whereas now you only have to avoid bad things being pointed out. Or alternatively, that when people claim good things they have obvious bad incentives to do that, so you’re inclined to not believe them. And that we don’t have time to find all the context, and need to act on simple heuristics due to limited compute. And in some places, the willingness to *ever* do a sufficiently bad thing is very strong evidence of additional bad things, and we need to maintain a strong norm of always punishing an action to maintain a strong norm against that action.
Also potentially important is that if you let things get fuzzy, those with power will use that fuzziness to enhance their own power. When needed, they’ll find ways to give themselves points to offset any bad things they’re caught doing. You need a way to stop this and bring them down.
And so on.
So in some places it becomes structurally necessary to have a no-excuses (or only local and well-specified excuses like self defense) approach. But there are entire cultural groups who use this as the generic evaluate-thing algorithm and that’s terrible.
That’s why I chose the phrasing “aim higher” rather than telling people “don’t do that.” I don’t think one can entirely eliminate such systems at this time at a reasonable price.
But this is also of the type of thing that I do when I’m analyzing my game play choices after a match of Magic, where I come up with all sorts of explanations and deep lines of possibility and consideration that were never in my conscious analysis at the time. At the time it was more something like, this needs a conclusion, I’ve shown the problems with this thing, this seems like a way to wrap things up and maybe get people to think about doing the thing less and spotting/discounting it more, which would be good.
(I will continue this line of thought down below in another reply)