I think it doesn’t—our dev effects are so, so noisy!
Joel Becker
Karma: 202
We definitely do not claim that AI broadly slows down programmers! See tweet thread about this.
I think all the points you raised are an important part of the story—we additionally go through some other factors that we think might explain the surprising result.
FWIW: this is my qualitative sense for other devs too.
We made this design decision because we wanted to max out on external validity, following the task length work which had fewer internal validity/more external validity concerns.
I (study author) responded to some of Ruby’s points on twitter. Delighted for devs including Ruby to discuss their experience publicly, I think it’s helpful for people to get a richer sense!
Found this helpful, thanks!
To “there were various suspicious/bad things Drew did,” I would reply:
I have this opposing consideration. [...] I gather that part of the reason Alice and Chloe feel this way is that Drew did try to be helpful with respect to their concerns, at least to whatever degree was required for them to ask for him to be shielded from professional consequences.
and, to “the choice he’s made to kinda hang around Emerson and Kat for this long,” I would reply:
To the extent you believe that Nonlinear has been a disfunctional environment, in significant part due to domineering characteristics of senior staff, I think that you should also believe that a junior family member beginning to work in this environment is going to have a hard time reasoning through and pushing back against it.
Repost from EA forum:
Thank you very much for sharing, Chloe.
Ben, Kat, Emerson, and readers of the original post have all noticed that the nature of Ben’s process leads to selection against positive observations about Nonlinear. I encourage readers to notice that the reverse might also be true. Examples of selection against negative information include:
Ben has reason to exclude stories that are less objective or have a less strong evidence base. The above comment is a concrete example of this.
There’s also something related here about the supposed unreliability of Alice as a source: Ben needs to include this to give a complete picture/because other people (in particular the Nonlinear co-founders) have said this. I strongly concur with Ben when he writes that he “found Alice very willing and ready to share primary sources [...] so I don’t believe her to be acting in bad faith.” Personally, my impression is that people are making an incorrect inference about Alice from her characteristics (that are perhaps correlated with source-reliability in a large population, but aren’t logically related, and aren’t relevant in this case).
To the extent that you expect other people to have been silenced (e.g. via anticipated retaliation), you might expect not to hear relevant information from them.
To the extent that you expect Alice and Chloe to have had burnout-style experiences, you might expect not to read clarifications on or news about negative experiences.
Until this post came out, this was true of ~everything in the post.
There is a reason the post was published 1.5 years after the relevant events took place—people involved in the events really do not want to spend further mental effort on this.
I have this opposing consideration. I think it does speak to your point—I gather that part of the reason Alice and Chloe feel this way is that Drew did try to be helpful with respect to their concerns, at least to whatever degree was required for them to ask for him to be shielded from professional consequences.
Here’s another (in my view weaker, but perhaps more directly relevant to your point) consideration. To the extent you believe that Nonlinear has been a disfunctional environment, in significant part due to domineering characteristics of senior staff, I think that you should also believe that a junior family member beginning to work in this environment is going to have a hard time reasoning through and pushing back against it. Happy to expand.
My understanding (definitely fallible, but I’ve been quite engaged in this case, and am one of the people Ben interviewed) has been that Alice and Chloe are not concerned about this, and in fact that they both wish to insulate Drew from any negative consequences. This seems to me like an informative and important consideration. (It also gives me reason to think that the benefits of gaining more information about this are less likely to be worth the costs.)
Enthusiasts of {wine/coffee/whiskey/potato/etc.} tasting?
I love this question!
Speaking of macroeconomics, there’s a nice connection here to the famous Lucas critique:
The Lucas critique, named for American economist Robert Lucas’s work on macroeconomic policymaking, argues that it is naive to try to predict the effects of a change in economic policy entirely on the basis of relationships observed in historical data, especially highly aggregated historical data.[1] More formally, it states that the decision rules of Keynesian models—such as the consumption function—cannot be considered as structural in the sense of being invariant with respect to changes in government policy variables.[2]
Interesting, thank you! Has the group gotten around to discussing something like “lessons from contract theory or corporate governance for factored cognition-style proposals” at all?
I think “many people” is doing a lot of work here—I’ve generally found the public reception to be very nuanced, moreso than I was expecting. See e.g. Gary Marcus’ post