I’ve been looking forward to this. Looking at the raw data now to get an idea of the inter-rater agreement. The two columns of Youtopia ratings agree fairly well on the 33 predictions where they overlap, and the 9 LW raters seem to disagree more, but that’s only my first impression. (Maybe it’s just that inter-rater variation is more obvious for predictions with ≥3 ratings.) Thanks again to all the assessors for putting in the legwork.
There are more than two youtopia raters. Different people, at different times, completed the assessments individually (and sometimes did “second opinions” if someone had already done that one before them). I think they were 5 assessors in total.
I’ve been looking forward to this. Looking at the raw data now to get an idea of the inter-rater agreement. The two columns of Youtopia ratings agree fairly well on the 33 predictions where they overlap, and the 9 LW raters seem to disagree more, but that’s only my first impression. (Maybe it’s just that inter-rater variation is more obvious for predictions with ≥3 ratings.) Thanks again to all the assessors for putting in the legwork.
There are more than two youtopia raters. Different people, at different times, completed the assessments individually (and sometimes did “second opinions” if someone had already done that one before them). I think they were 5 assessors in total.
Oops. Fixed.