The above estimate was mislead since I had mistakenly read ′ I then compute the fraction #answers(Yes, Yes, Yes) / #answers(Yes, *, *) ′ as ′ I then compute the fraction #answers(Yes, Yes, Yes) / #answers(Yes, Yes, *)’.
I agree with Ethan’s recent comment that experience with RL matter a lot, so a lot comes down to how the ′ Is X’s work related to AGI? ′ criterion is cashed out. On some reading of this, many NLP researchers do not count, on another reading they do count. I’d say my previous prediction was a decent, if slightly over-estimate of the scenario in which ‘related to AGI’ is interpreted narrowly, and many NLP researchers are ruled out.
A second major confounder is whether prominent AI researchers are far more likely to have been asked about their opinion on AI safety in which case they have some impetus to go read up on the issue.
To cash some of these concerns out into probabilities:
75% that Rohin takes a broad interpretation of AGI which includes e.g. GPT-team, NAS research etc.
33% estimated (Yes,Yes,Yes) by assuming prominent researchers 2x as likely to have read up on AI safety.
25% downweighted from 33% taking into account industry being less concerned.
Assuming that we’re at ~33% now, 50% doesn’t seem too far out of reach, so my estimates for following decades are based on the same concerns I listed in my above comment framed with the 33% in mind.
Updated personal distribution: elicited
Updated Rohin’s posterior: elicited
(Post competition footnote: seems to me over short time horizons we should have a more-or-less geometric distribution. Think of the more-or-less independent per year chance that a NeurIPS keynote features AI safety, or youtube recommender algorithm goes bonkers for a bit. Seems strange to me that some other people’s distribution over the next 10-15 years—if not longer—do not look geometric.)
I do take the broad interpretation of AGI-related work.
I hadn’t considered the point that people may ask prominent AI researchers their opinion about AI safety, and that leading them to have better beliefs about safety. I think overall I don’t actually expect this to be a major factor, but it’s a good point and updated me slightly towards sooner.
I wouldn’t expect a geometric distribution—consensus building requires time, as a result you might expect a buildup from 0 for <time taken to build consensus> and then have it follow a geometric distribution. In addition, getting to 50% seems likely to require a warning shot of some significance; current AI systems don’t seem capable enough to produce a compelling enough warning shot.