Fwiw – I read this post, and thought “hmm, this post does two things – it puts forth some fairly concrete models (which are interesting independent of any call to action).
It also puts forth… some kind of vague call to action, which includes a bit more rhetoric than I’m comfortable with, but not so much more that I think it shouldn’t be frontpaged given that the models seem straightforwardly good.
So… basicIy I didn’t come away with this post with a call-to-action, I just came up with a useful handle for how to think about one aspect of justice, which I’ll have in mind as I go around thinking about justice.
Fine. I’m convinced now. The line has been replaced by a summary-style line that is clearly not a call to action.
The pattern seems to be, if one spends 1600 words on analysis, then one sentence suggesting one might aim to avoid the mistakes pointed out in the analysis, then one is viewed as “doing two things” and/or being a call to action, and then is guilty if the call-to-action isn’t sufficiently well specified and doesn’t give concrete explicit paths to making progress that seem realistic and to fit people’s incentives and so on?
Which itself seems like several really big problems, and an illustration of the central point of this piece!
Call to action, and the calling thereof, is an action, and thus makes one potentially blameworthy in various ways for being insufficient, whereas having no call to action would have been fine. You’ve interacted with the problem, and thus by CIE are responsible for not doing more. So one must not interact with the problem in any real way, and ensure that one isn’t daring to suggest anything get done.
I’m viewing this thing more through the lens of Tales of Alice Almost, where there’s a legitimate hard question of “what should be incentivized on LessWrong”, which depends a lot on what the average skills and tendencies of the typical LessWrong user is, as well as what the skills/tendencies of particular users are.
Longterm, there’s a quite high bar we want LessWrong to be aspiring to. Because newcomers frequently arrive at LessWrong who won’t yet have a bunch of skills, there needs to be some fairly simple guidelines to get them started (allowing them to get positively rewarded for contributing).
But I do want the tail end of users to also have incentive to continue to improve.
Because I’m not 100% sure what the right collection of skills and norms for LessWrong to encourage are, there also needs to be an incentive for the collective culture (and the mod team in particular) to improve our understanding of “what things should be incentivized” so we don’t get stuck in a weird lost purpose.
(If the current mod team got hit by a truck and new people took over and tried to implement our “no calls to action on frontpage” rule without understanding it, I predict they wouldn’t get the nuances right).
Posts by Zvi are reliably much more interesting to me than the average post, tackling issues that are thorny with interesting insight that I respect quite a bit. If the collection of incentives we had resulted in Zvi posting less, that would be quite bad.
But Zvi posts also tend to be include a particular kind of rhetorical flourish that feels out of place for LessWrong – it feels like I’m listening to a political rally. So a) I don’t want new users to internalize that style as something they should emulate (part of what the frontpage is for), and b) I genuinely want the frontpage to be a place where people can engage with ideas without feeling incentivized to think about those ideas through the lens of “how is this affecting the social landscape?”
(this is not because it’s not important to think about how things affect the social landscape, but because that’s ‘hard mode’, and requires trust to do well while training your rationality skill. The current best-guess of myself and most of the mods is that it’s best if people separate out your posts about models and principles, from the posts about ‘here’s what’s wrong with the social landscape and how to fix it’)
There’s a range of what I’d consider “calls to action”. The call to action “here are some models, I think they’re important and you probably should have considered them and maybe do something about them” is pretty fine for frontpage. Where it gets dicey is when it has this particular undercurrent of politics-rah-rah-rah, which I think the original version of this post had (a bit of)
I did change the post on the blog as well, not only the LW version, to the new version. This wasn’t a case of ‘I shouldn’t have to change this but Raemon is being dense’ but rather ‘I see two of the best people on this site focusing on this one sentence in massively distracting ways so I’m clearly doing something wrong here’ and reaching the conclusion that this is how humans read articles so this line needs to go. And indeed, to draw a clear distinction between the posts where I am doing pure model building, from the posts with action calls.
I got frustrated because it feels like this is an expensive sacrifice that shouldn’t be necessary. And because I was worried that this was an emergent pattern and dilemma against clarity, where if your call to clarity hints at a call to action people focus on the call to action, and if you don’t call to action then people (especially outside of LW) say “That was nice and all but you didn’t tell me what to do with that so what’s the point?” and/or therefore forget what said. And the whole issue of calls to action vs. clarity has been central to some recent private discussions recently. where very high-level rationalists have repeatedly reacted to calls for clarity as if they are calls to action, in patterns that seem optimized for preventing clarity and common knowledge. All of which I’m struggling to figure out how to explain.
There’s also the gaslighting thing where people do politics while pretending they’re not doing that, then accuse anyone who calls them out on it of doing politics (and then, of course, the worry where it goes deeper and someone accuses someone of accusing someone of playing politics, which can be very true and important but more frequently is next-level gaslighting).
We also need to do a better job of figuring out how to do things that require a lot of groundwork—to teach the hard mode advanced class. There was a time when everyone was expected to have read the sequences and understand them, which helped a lot here. But at the time, I was actively terrified of commenting let alone posting, so it certainly wasn’t free.
(If the current mod team got hit by a truck and new people took over and tried to implement our “no calls to action on frontpage” rule without understanding it, I predict they wouldn’t get the nuances right).
A corollary of 1.3 is that we often prefer descriptive language (including language describing your current beliefs, emotional state, etc.) over prescriptive language, all else being equal.
Which seems pretty far from “no calls to action on frontpage” and isn’t even in the “Things to keep to a minimum” or “Off-limits things” section.
(If I had been aware of this rule and surrounding discussions about it, maybe I would have been more sensitive about “accusing” someone of making a call to action, which to be clear wasn’t my intention at all since I didn’t even know such a rule existed.)
I think the phrase “call to action” might get used internally more than externally (although I have a blogpost brewing that delves into it a bit, as well as another phrase “call to conflict.”)
But a phrase used in both our Frontpage Commenting guidelines, and on the tooltip for when you mark a post as ‘allow moderators to promote’ is ‘aim to explain, not persuade’, where calls to action are a subset of persuading.
(Note that both of those site-elements might not appear on GreaterWrong. I think GreaterWrong also doesn’t really have the frontpage distinction anyhow, instead just showing all new posts in order of appearance)
I actually think the “aim to explain, not persuade” framing is generally clearer than the “no call to action” framing. Like, if you explain something to someone that strongly implies some action, then some people might call that a “call to action” but I would think that’s totally fine.
Agreed. And I think I was implicitly focusing on whether the post gave a sufficient explanation for its (original) conclusion, and was rather confused why others were so focused on whether there was a call to action or not (which without knowing the context of your private discussions I just interpreted to mean any practical suggestion)
So, this post has netted for Zvi a few hundred karma, which SEEMS to be encouraging the right thing. Even with some confusion and controversy, it’s clearly positive value. I apologize for my asymmetric commenting style, especially if my focus on points of disagreement makes it seem like I don’t value the topic and everyone’s thoughts on it.
I want to ask about your dual preferences: you want high-quality as an absolute and you want people to improve from their current capabilities, as a relative. Are there different ways of encouraging these two goals, or are they integrated enough that you think of them as the same?
No need to apologize for focusing on points of disagreement. And I’m grateful for the commentary and confusion, because it pointed to important questions about how to have good discourse and caused me to notice something I do frequently that is likely a mistake. It’s like finally having an editor, in the good way.
I’m not on the moderation team, but my perspective is that the two goals overlap and are fully compatible but largely distinct and need to be optimized for in different ways (see Tale of Alice Almost). And this is the situation in which you get a conflict between them, because norms are messy and you can’t avoid what happens in hard mode threads bleeding into other places.
Part of my complaint was that the models didn’t seem to include enough gears for me to figure out what I could do to make things better. The author’s own conclusions, which he later clarified in the comments, seems to be that we should individually do less of the thing that he suggests is bad. But my background assumption is that group rationality problems are usually coordination problems so it usually doesn’t help much to tell people to individually “do the right thing”. That would be analogous to telling players in PD to just play cooperate. At this point I still don’t know whether or why the author’s call to action would work better than telling players in PD to just play cooperate.
I am confused why it is unreasonable to suggest to people that, as a first step to correcting a mistake, that they themselves stop making it. I don’t think that ‘I individually would suffer so much from not making this mistake that I require group coordination to stop making it’ applies here.
And in general, I worry that the line of reasoning that goes ” group rationality problems are usually coordination problems so it usually doesn’t help much to tell people to individually “do the right thing” leads (as it seems to be doing directly in this case) to the suggestion that now it is unreasonable to suggest someone might do the right thing on their own in addition to any efforts to make that a better plan or to assist with abilities to coordinate.
I’d also challenge the idea that only the group’s conclusions on what is just matter, or that the goal of forming conclusions about what is just is to reach the same conclusion as the group, meaning that justice becomes ‘that which the group chooses to coordinate on.’ And where one’s cognition is primarily about figuring out where the coordination is going to land, rather than what would in fact be just.
This isn’t a PD situation. You are individually better off if you provide good incentives to those around you to behave in just fashion, and your cognitive map is better if you can properly judge what is good and bad and what to offer your support to and encourage, and what to oppose and discourage.
To the extent group coordination is required, then the solution is in fact to do what all but one sentence of the post is clearly aiming to do, explain and create clarity and common knowledge.
I am confused why it is unreasonable to suggest to people that, as a first step to correcting a mistake, that they themselves stop making it.
My reasoning is that 1) the problem could be a coordination problem. If it is, then telling people to individually stop making the mistake does nothing or just hurts the people who listen, without making the world better off as a whole. If it’s not a coordination problem, then 2) there’s still a high probability that it’s a Chesterton’s fence, and I think your post didn’t do enough to rule that out either.
now it is unreasonable to suggest someone might do the right thing on their own in addition to any efforts to make that a better plan or to assist with abilities to coordinate
Maybe my position is more understandable in light of the Chesterton’s fence concern? (Sorry that my critique is coming out in bits and pieces, but originally I just couldn’t understand what the ending meant, then the discussion got a bit side-tracked onto whether there was a call to action or not, etc.)
I’d also challenge the idea that only the group’s conclusions on what is just matter, or that the goal of forming conclusions about what is just is to reach the same conclusion as the group, meaning that justice becomes ‘that which the group chooses to coordinate on.’
This seems like a strawman or a misunderstanding of my position. I would say that generally there could be multiple things that the group could choose to coordinate on (i.e., multiple equilibria in terms of game theory) or we could try to change what the group coordinates on by changing the rules of the game, so I would disagree that “the goal of forming conclusions about what is just is to reach the same conclusion as the group”. My point is instead that we can’t arbitrarily choose “where the coordination is going to land” and we need better models to figure out what’s actually feasible.
As I noted in my other reply, on reflection I was definitely overly frustrated when replying here and it showed. I need to be better about that. And yes, this helps understand where you’re coming from.
Responding to the concerns:
1) It is in part a coordination problem—everyone gets benefits if there is agreement on an answer, versus disagreement among two equally useful/correct potential responses. But it’s certainly not a pure coordination problem. It isn’t obvious to me if, given everyone else has coordinated on an incorrect answer, it is beneficial or harmful to you to find the correct answer (let’s ignore here the question of what answer is right or wrong). You get to get your local incentives better, improve your map and understanding, set an example that can help people realize they’re coordinating in the wrong place, people you want to be associating with are more inclined to associate with you (because they see you taking a stand for the right things, and would be willing to coordinate with you on the new answer, and on improving maps and incentives in general, and do less games that are primarily about coordination and political group dynamics...) and so on.
There is also the distinction between, (A) I am going to internally model what gets points in a better way, and try to coordinate with and encourage and help things that tend towards positive points over those with negative points, and (B) I am going to act as if everyone else is going to go along with this, or expect them to, or get into fights over this beyond trying to convince them. I’m reasonably confident that doing (A) is a good idea if you’re right, and can handle the mental load of having a model different from the model you believe that others are using.
But even if we accept that, in some somewhat-local sense, failure to coordinate means the individual gets a worse payoff while the benefits are diffused without too much expectation of a shift in equilibrium happening soon, this seems remarkably similar to to many decisions of the norm “do rationality or philosophy on this.” Unless one gets intrinsic benefit from being right or exploring the questions, one is at best doing a lot of underpaid work, and probably just making oneself worse off. Yet here we are.
I am also, in general, willing to bite the bullet that the best answer I know about to coordination problems where there is a correct coordination point, and the group is currently getting it wrong, and the cost of getting it wrong seems high compared to the cost of some failures of coordination, and you have enough slack to do it, is to give the ‘right’ answer rather than the coordination answer. And to encourage such a norm.
2) Agree that I wasn’t trying at all to rule this out. There are a bunch of obvious benefits to groups and to individuals of using asymmetric systems, some of which I’ve pointed to in these comments. To the extent that I don’t think you can entirely avoid such systems and I wouldn’t propose tearing down the entire fence. A lot of my model of these situations is that such evolutionary-style systems are very lossy, leading to being used in situations they weren’t intended for like evaluating economic systems or major corporations, or people you don’t have any context on. And also they are largely designed for dealing with political coalitions and scapegoating in worlds where such things are super important and being done by others, often as the primary cause of cognition. And all these systems have to assume that you’re working without the kind of logical reasoning we’re using here, and care a lot that having one model and acting as if others have another, and when needed acting according to that other model, is expensive and hard, and others who notice you have a unique model will by default seek to scapegoat you for that which is the main reason why such problems are coordination problems, and so on. That sort of thing.
3) The goal of the conclusion/modeling game from the perspective of the group, I think we’d agree, is often to (i) coordinate on conclusions enough to act (ii) on the answer that is best for the group, subject to needing to coordinate. I was speaking of the goal from the perspective of the individual. When I individually decide what is just, what am I doing? (a) One possibility is that I am mostly worried about things like my social status and position in the group and whether others will praise or blame me, or scapegoat me. My view on what is just won’t change what is rewarded or punished by the group much, one might say, since I am only one of a large group. Or (b) one can be primarily concerned with what is just or what norms of justice would provide the right incentives, figure out that and try to convince others and act on that basis to the extent possible. Part of that is figuring out what answers would be stable/practical to implement/practical to get to, although ideally one would first figure out the range of what solutions do what and then pick the best practical answer.
Agreed that it would be good to have better understanding of where coordination might land, especially once we get to the point of wanting to coordinate on landing in a new place.
Part of my complaint was that the models didn’t seem to include enough gears for me to figure out what I could do to make things better.
I do think it’s fine to discuss models that represent reality accurately, while not knowing what action-relevant implications they might have eventually. A lot of AI-Alignment related thinking is not really suggesting many concrete actions to take, besides “this seems like a problem, no idea what to do about it”.
I do not think we have no idea what to do about it. Creating common knowledge of a mistake, and ceasing to make that mistake yourself, are both doing something about it. If the problem is a coordination game then coordination to create common knowledge of the mistake seems like the obvious first move.
“this seems like a problem, no idea what to do about it”
I think this is fine if made clear, but the post seemed to be implying (which the author later confirmed) that it did offer action-relevant implications.
Fwiw – I read this post, and thought “hmm, this post does two things – it puts forth some fairly concrete models (which are interesting independent of any call to action).
It also puts forth… some kind of vague call to action, which includes a bit more rhetoric than I’m comfortable with, but not so much more that I think it shouldn’t be frontpaged given that the models seem straightforwardly good.
So… basicIy I didn’t come away with this post with a call-to-action, I just came up with a useful handle for how to think about one aspect of justice, which I’ll have in mind as I go around thinking about justice.
Fine. I’m convinced now. The line has been replaced by a summary-style line that is clearly not a call to action.
The pattern seems to be, if one spends 1600 words on analysis, then one sentence suggesting one might aim to avoid the mistakes pointed out in the analysis, then one is viewed as “doing two things” and/or being a call to action, and then is guilty if the call-to-action isn’t sufficiently well specified and doesn’t give concrete explicit paths to making progress that seem realistic and to fit people’s incentives and so on?
Which itself seems like several really big problems, and an illustration of the central point of this piece!
Call to action, and the calling thereof, is an action, and thus makes one potentially blameworthy in various ways for being insufficient, whereas having no call to action would have been fine. You’ve interacted with the problem, and thus by CIE are responsible for not doing more. So one must not interact with the problem in any real way, and ensure that one isn’t daring to suggest anything get done.
I’m viewing this thing more through the lens of Tales of Alice Almost, where there’s a legitimate hard question of “what should be incentivized on LessWrong”, which depends a lot on what the average skills and tendencies of the typical LessWrong user is, as well as what the skills/tendencies of particular users are.
Longterm, there’s a quite high bar we want LessWrong to be aspiring to. Because newcomers frequently arrive at LessWrong who won’t yet have a bunch of skills, there needs to be some fairly simple guidelines to get them started (allowing them to get positively rewarded for contributing).
But I do want the tail end of users to also have incentive to continue to improve.
Because I’m not 100% sure what the right collection of skills and norms for LessWrong to encourage are, there also needs to be an incentive for the collective culture (and the mod team in particular) to improve our understanding of “what things should be incentivized” so we don’t get stuck in a weird lost purpose.
(If the current mod team got hit by a truck and new people took over and tried to implement our “no calls to action on frontpage” rule without understanding it, I predict they wouldn’t get the nuances right).
Posts by Zvi are reliably much more interesting to me than the average post, tackling issues that are thorny with interesting insight that I respect quite a bit. If the collection of incentives we had resulted in Zvi posting less, that would be quite bad.
But Zvi posts also tend to be include a particular kind of rhetorical flourish that feels out of place for LessWrong – it feels like I’m listening to a political rally. So a) I don’t want new users to internalize that style as something they should emulate (part of what the frontpage is for), and b) I genuinely want the frontpage to be a place where people can engage with ideas without feeling incentivized to think about those ideas through the lens of “how is this affecting the social landscape?”
(this is not because it’s not important to think about how things affect the social landscape, but because that’s ‘hard mode’, and requires trust to do well while training your rationality skill. The current best-guess of myself and most of the mods is that it’s best if people separate out your posts about models and principles, from the posts about ‘here’s what’s wrong with the social landscape and how to fix it’)
There’s a range of what I’d consider “calls to action”. The call to action “here are some models, I think they’re important and you probably should have considered them and maybe do something about them” is pretty fine for frontpage. Where it gets dicey is when it has this particular undercurrent of politics-rah-rah-rah, which I think the original version of this post had (a bit of)
Right.
I did change the post on the blog as well, not only the LW version, to the new version. This wasn’t a case of ‘I shouldn’t have to change this but Raemon is being dense’ but rather ‘I see two of the best people on this site focusing on this one sentence in massively distracting ways so I’m clearly doing something wrong here’ and reaching the conclusion that this is how humans read articles so this line needs to go. And indeed, to draw a clear distinction between the posts where I am doing pure model building, from the posts with action calls.
I got frustrated because it feels like this is an expensive sacrifice that shouldn’t be necessary. And because I was worried that this was an emergent pattern and dilemma against clarity, where if your call to clarity hints at a call to action people focus on the call to action, and if you don’t call to action then people (especially outside of LW) say “That was nice and all but you didn’t tell me what to do with that so what’s the point?” and/or therefore forget what said. And the whole issue of calls to action vs. clarity has been central to some recent private discussions recently. where very high-level rationalists have repeatedly reacted to calls for clarity as if they are calls to action, in patterns that seem optimized for preventing clarity and common knowledge. All of which I’m struggling to figure out how to explain.
There’s also the gaslighting thing where people do politics while pretending they’re not doing that, then accuse anyone who calls them out on it of doing politics (and then, of course, the worry where it goes deeper and someone accuses someone of accusing someone of playing politics, which can be very true and important but more frequently is next-level gaslighting).
We also need to do a better job of figuring out how to do things that require a lot of groundwork—to teach the hard mode advanced class. There was a time when everyone was expected to have read the sequences and understand them, which helped a lot here. But at the time, I was actively terrified of commenting let alone posting, so it certainly wasn’t free.
When did this rule come into effect and where is it written down? The closest thing I can find in Frontpage Posting and Commenting Guidelines is:
Which seems pretty far from “no calls to action on frontpage” and isn’t even in the “Things to keep to a minimum” or “Off-limits things” section.
(If I had been aware of this rule and surrounding discussions about it, maybe I would have been more sensitive about “accusing” someone of making a call to action, which to be clear wasn’t my intention at all since I didn’t even know such a rule existed.)
I think the phrase “call to action” might get used internally more than externally (although I have a blogpost brewing that delves into it a bit, as well as another phrase “call to conflict.”)
But a phrase used in both our Frontpage Commenting guidelines, and on the tooltip for when you mark a post as ‘allow moderators to promote’ is ‘aim to explain, not persuade’, where calls to action are a subset of persuading.
(Note that both of those site-elements might not appear on GreaterWrong. I think GreaterWrong also doesn’t really have the frontpage distinction anyhow, instead just showing all new posts in order of appearance)
I actually think the “aim to explain, not persuade” framing is generally clearer than the “no call to action” framing. Like, if you explain something to someone that strongly implies some action, then some people might call that a “call to action” but I would think that’s totally fine.
Agreed. And I think I was implicitly focusing on whether the post gave a sufficient explanation for its (original) conclusion, and was rather confused why others were so focused on whether there was a call to action or not (which without knowing the context of your private discussions I just interpreted to mean any practical suggestion)
So, this post has netted for Zvi a few hundred karma, which SEEMS to be encouraging the right thing. Even with some confusion and controversy, it’s clearly positive value. I apologize for my asymmetric commenting style, especially if my focus on points of disagreement makes it seem like I don’t value the topic and everyone’s thoughts on it.
I want to ask about your dual preferences: you want high-quality as an absolute and you want people to improve from their current capabilities, as a relative. Are there different ways of encouraging these two goals, or are they integrated enough that you think of them as the same?
No need to apologize for focusing on points of disagreement. And I’m grateful for the commentary and confusion, because it pointed to important questions about how to have good discourse and caused me to notice something I do frequently that is likely a mistake. It’s like finally having an editor, in the good way.
I’m not on the moderation team, but my perspective is that the two goals overlap and are fully compatible but largely distinct and need to be optimized for in different ways (see Tale of Alice Almost). And this is the situation in which you get a conflict between them, because norms are messy and you can’t avoid what happens in hard mode threads bleeding into other places.
Part of my complaint was that the models didn’t seem to include enough gears for me to figure out what I could do to make things better. The author’s own conclusions, which he later clarified in the comments, seems to be that we should individually do less of the thing that he suggests is bad. But my background assumption is that group rationality problems are usually coordination problems so it usually doesn’t help much to tell people to individually “do the right thing”. That would be analogous to telling players in PD to just play cooperate. At this point I still don’t know whether or why the author’s call to action would work better than telling players in PD to just play cooperate.
I am confused why it is unreasonable to suggest to people that, as a first step to correcting a mistake, that they themselves stop making it. I don’t think that ‘I individually would suffer so much from not making this mistake that I require group coordination to stop making it’ applies here.
And in general, I worry that the line of reasoning that goes ” group rationality problems are usually coordination problems so it usually doesn’t help much to tell people to individually “do the right thing” leads (as it seems to be doing directly in this case) to the suggestion that now it is unreasonable to suggest someone might do the right thing on their own in addition to any efforts to make that a better plan or to assist with abilities to coordinate.
I’d also challenge the idea that only the group’s conclusions on what is just matter, or that the goal of forming conclusions about what is just is to reach the same conclusion as the group, meaning that justice becomes ‘that which the group chooses to coordinate on.’ And where one’s cognition is primarily about figuring out where the coordination is going to land, rather than what would in fact be just.
This isn’t a PD situation. You are individually better off if you provide good incentives to those around you to behave in just fashion, and your cognitive map is better if you can properly judge what is good and bad and what to offer your support to and encourage, and what to oppose and discourage.
To the extent group coordination is required, then the solution is in fact to do what all but one sentence of the post is clearly aiming to do, explain and create clarity and common knowledge.
My reasoning is that 1) the problem could be a coordination problem. If it is, then telling people to individually stop making the mistake does nothing or just hurts the people who listen, without making the world better off as a whole. If it’s not a coordination problem, then 2) there’s still a high probability that it’s a Chesterton’s fence, and I think your post didn’t do enough to rule that out either.
Maybe my position is more understandable in light of the Chesterton’s fence concern? (Sorry that my critique is coming out in bits and pieces, but originally I just couldn’t understand what the ending meant, then the discussion got a bit side-tracked onto whether there was a call to action or not, etc.)
This seems like a strawman or a misunderstanding of my position. I would say that generally there could be multiple things that the group could choose to coordinate on (i.e., multiple equilibria in terms of game theory) or we could try to change what the group coordinates on by changing the rules of the game, so I would disagree that “the goal of forming conclusions about what is just is to reach the same conclusion as the group”. My point is instead that we can’t arbitrarily choose “where the coordination is going to land” and we need better models to figure out what’s actually feasible.
As I noted in my other reply, on reflection I was definitely overly frustrated when replying here and it showed. I need to be better about that. And yes, this helps understand where you’re coming from.
Responding to the concerns:
1) It is in part a coordination problem—everyone gets benefits if there is agreement on an answer, versus disagreement among two equally useful/correct potential responses. But it’s certainly not a pure coordination problem. It isn’t obvious to me if, given everyone else has coordinated on an incorrect answer, it is beneficial or harmful to you to find the correct answer (let’s ignore here the question of what answer is right or wrong). You get to get your local incentives better, improve your map and understanding, set an example that can help people realize they’re coordinating in the wrong place, people you want to be associating with are more inclined to associate with you (because they see you taking a stand for the right things, and would be willing to coordinate with you on the new answer, and on improving maps and incentives in general, and do less games that are primarily about coordination and political group dynamics...) and so on.
There is also the distinction between, (A) I am going to internally model what gets points in a better way, and try to coordinate with and encourage and help things that tend towards positive points over those with negative points, and (B) I am going to act as if everyone else is going to go along with this, or expect them to, or get into fights over this beyond trying to convince them. I’m reasonably confident that doing (A) is a good idea if you’re right, and can handle the mental load of having a model different from the model you believe that others are using.
But even if we accept that, in some somewhat-local sense, failure to coordinate means the individual gets a worse payoff while the benefits are diffused without too much expectation of a shift in equilibrium happening soon, this seems remarkably similar to to many decisions of the norm “do rationality or philosophy on this.” Unless one gets intrinsic benefit from being right or exploring the questions, one is at best doing a lot of underpaid work, and probably just making oneself worse off. Yet here we are.
I am also, in general, willing to bite the bullet that the best answer I know about to coordination problems where there is a correct coordination point, and the group is currently getting it wrong, and the cost of getting it wrong seems high compared to the cost of some failures of coordination, and you have enough slack to do it, is to give the ‘right’ answer rather than the coordination answer. And to encourage such a norm.
2) Agree that I wasn’t trying at all to rule this out. There are a bunch of obvious benefits to groups and to individuals of using asymmetric systems, some of which I’ve pointed to in these comments. To the extent that I don’t think you can entirely avoid such systems and I wouldn’t propose tearing down the entire fence. A lot of my model of these situations is that such evolutionary-style systems are very lossy, leading to being used in situations they weren’t intended for like evaluating economic systems or major corporations, or people you don’t have any context on. And also they are largely designed for dealing with political coalitions and scapegoating in worlds where such things are super important and being done by others, often as the primary cause of cognition. And all these systems have to assume that you’re working without the kind of logical reasoning we’re using here, and care a lot that having one model and acting as if others have another, and when needed acting according to that other model, is expensive and hard, and others who notice you have a unique model will by default seek to scapegoat you for that which is the main reason why such problems are coordination problems, and so on. That sort of thing.
3) The goal of the conclusion/modeling game from the perspective of the group, I think we’d agree, is often to (i) coordinate on conclusions enough to act (ii) on the answer that is best for the group, subject to needing to coordinate. I was speaking of the goal from the perspective of the individual. When I individually decide what is just, what am I doing? (a) One possibility is that I am mostly worried about things like my social status and position in the group and whether others will praise or blame me, or scapegoat me. My view on what is just won’t change what is rewarded or punished by the group much, one might say, since I am only one of a large group. Or (b) one can be primarily concerned with what is just or what norms of justice would provide the right incentives, figure out that and try to convince others and act on that basis to the extent possible. Part of that is figuring out what answers would be stable/practical to implement/practical to get to, although ideally one would first figure out the range of what solutions do what and then pick the best practical answer.
Agreed that it would be good to have better understanding of where coordination might land, especially once we get to the point of wanting to coordinate on landing in a new place.
(There is a closing quote missing in the second paragraph of this comment, which caused me to be quite confused reading that paragraph)
I do think it’s fine to discuss models that represent reality accurately, while not knowing what action-relevant implications they might have eventually. A lot of AI-Alignment related thinking is not really suggesting many concrete actions to take, besides “this seems like a problem, no idea what to do about it”.
I do not think we have no idea what to do about it. Creating common knowledge of a mistake, and ceasing to make that mistake yourself, are both doing something about it. If the problem is a coordination game then coordination to create common knowledge of the mistake seems like the obvious first move.
I think this is fine if made clear, but the post seemed to be implying (which the author later confirmed) that it did offer action-relevant implications.
FWIW, in slightly different words than my last comment, I agree with this criticism of this post.