There were 66 responses, though at least one was a duplicate. (I didn’t deduplicate in any of the analyses below; I doubt it will make a big difference.) Looking at names (when provided), it looks like people in the field were quite a bit more likely to respond than the typical reader. Estimating 5 min on average per response (since many provided qualitative feedback as well), that’s 5.5 hours of person-time answering the survey.
My main takeaways (described in more detail below):
The newsletter is still useful to people.
Long summaries are not as useful as I thought.
On the current margin I should move my focus away from the Alignment Forum, since the most involved readers seem to read most of the Alignment Forum already.
It would be nice to do more “high-level opinions”—if you imagine a tree where the root node is “did we build safe / beneficial AI”, and then lower nodes delve into subproblems; it would be useful to have opinions talk about how the current paper / article relates to the top-level node. I don’t think I’ll make a change of this form right now, but I might in the future.
I think these takeaways are probably worth 5-10 hours of time? It’s close though.
----
Average rating of various components of the newsletter (arranged in ascending order of popularity):
3.88 Long summaries (full newsletter dedicated to one topic) 3.91 Source of interesting things to read 3.95 Opinions 4.02 Highlights 4.27 Regular summaries 4.47 Newsletter overall
(The question was a five-point scale: “Useless, Meh, Fairly useful, Keep doing this, This is amazing!”, which I then converted to 1-5 and averaged across respondents.)
The newsletter overall is far more popular than any of the individual components. In hindsight this makes sense—different people will find different components valuable, but probably people will subscribe if there’s just one or two components valuable to them. So everyone will rate the newsletter highly, but only a subset will rate any given component highly.
I was surprised to see the long summaries were least popular, since people have previously explicitly said that they especially liked the long summaries without any prompting from me. I will probably be less likely to do long summaries in the future.
----
In the “value of the newsletter” qualitative section, the most common thing by far was people saying that it helped them stay abreast of the field—especially for articles that are not on the Alignment Forum.
----
One or two people suggested adding links to interesting papers that I wouldn’t have time to summarize. I actually used to do this when the newsletter first started, but it seemed like no one was clicking on those links so I stopped doing that. I’m pretty sure that would still be the case now so I’m not planning to restart that practice.
One or two people suggested adding links to interesting papers that I wouldn’t have time to summarize. I actually used to do this when the newsletter first started, but it seemed like no one was clicking on those links so I stopped doing that. I’m pretty sure that would still be the case now so I’m not planning to restart that practice.
A possible experiment: Frame this as a ‘request for summaries’, link to the papers you won’t get round to, but offer to publish any sufficiently good summaries of those papers that someone sends you in a future newsletter.
Also, damn! I really like the long summaries, and would be sad to see them go (though obviously you should listen to a survey of 66 people over my opinion)
I really like the long summaries, and would be sad to see them go
Fwiw I still expect to do them; this is an “on the margin” thing. Like, I still would do a long summary for bio anchors, but maybe I do something shorter for infra-Bayesianism.
Frame this as a ‘request for summaries’, link to the papers you won’t get round to, but offer to publish any sufficiently good summaries of those papers that someone sends you in a future newsletter.
Other results from the survey:
There were 66 responses, though at least one was a duplicate. (I didn’t deduplicate in any of the analyses below; I doubt it will make a big difference.) Looking at names (when provided), it looks like people in the field were quite a bit more likely to respond than the typical reader. Estimating 5 min on average per response (since many provided qualitative feedback as well), that’s 5.5 hours of person-time answering the survey.
My main takeaways (described in more detail below):
The newsletter is still useful to people.
Long summaries are not as useful as I thought.
On the current margin I should move my focus away from the Alignment Forum, since the most involved readers seem to read most of the Alignment Forum already.
It would be nice to do more “high-level opinions”—if you imagine a tree where the root node is “did we build safe / beneficial AI”, and then lower nodes delve into subproblems; it would be useful to have opinions talk about how the current paper / article relates to the top-level node. I don’t think I’ll make a change of this form right now, but I might in the future.
I think these takeaways are probably worth 5-10 hours of time? It’s close though.
----
Average rating of various components of the newsletter (arranged in ascending order of popularity):
3.88 Long summaries (full newsletter dedicated to one topic)
3.91 Source of interesting things to read
3.95 Opinions
4.02 Highlights
4.27 Regular summaries
4.47 Newsletter overall
(The question was a five-point scale: “Useless, Meh, Fairly useful, Keep doing this, This is amazing!”, which I then converted to 1-5 and averaged across respondents.)
The newsletter overall is far more popular than any of the individual components. In hindsight this makes sense—different people will find different components valuable, but probably people will subscribe if there’s just one or two components valuable to them. So everyone will rate the newsletter highly, but only a subset will rate any given component highly.
I was surprised to see the long summaries were least popular, since people have previously explicitly said that they especially liked the long summaries without any prompting from me. I will probably be less likely to do long summaries in the future.
----
In the “value of the newsletter” qualitative section, the most common thing by far was people saying that it helped them stay abreast of the field—especially for articles that are not on the Alignment Forum.
----
One or two people suggested adding links to interesting papers that I wouldn’t have time to summarize. I actually used to do this when the newsletter first started, but it seemed like no one was clicking on those links so I stopped doing that. I’m pretty sure that would still be the case now so I’m not planning to restart that practice.
A possible experiment: Frame this as a ‘request for summaries’, link to the papers you won’t get round to, but offer to publish any sufficiently good summaries of those papers that someone sends you in a future newsletter.
Also, damn! I really like the long summaries, and would be sad to see them go (though obviously you should listen to a survey of 66 people over my opinion)
Fwiw I still expect to do them; this is an “on the margin” thing. Like, I still would do a long summary for bio anchors, but maybe I do something shorter for infra-Bayesianism.
Hmm, intriguing. That might be worth trying.