Requesting Questions For A 2017 LessWrong Survey

It’s been twelve months since the last LessWrong Sur­vey, which means we’re due for a new one. But be­fore I can put out a new sur­vey in earnest, I feel obli­gated to so­licit ques­tions from com­mu­nity mem­bers and check in on any ideas that might be float­ing around for what we should ask.

The ba­sic for­mat of the thread isn’t too com­plex, just pitch ques­tions. For best chances of in­clu­sion, how­ever, it’s best to in­clude:

  • A short cost/​benefit anal­y­sis of in­clud­ing the ques­tion. Keep in mind that some ques­tions are too in­va­sive or em­bar­rass­ing to be rea­son­ably in­cluded. Other ques­tions might leak too many bits. There is limited sur­vey space and some items might be too marginal to in­clude at the cost of oth­ers.

  • An ex­am­ple of a use­ful anal­y­sis that could be done with this ques­tion(s), es­pe­cially in­ter­est­ing anal­y­sis in con­cert with other ques­tions. eg. It’s best to start with a larger ques­tion like “how does parental re­li­gious de­nom­i­na­tion af­fect the co­horts cur­rent re­li­gion?” and then trans­late that into con­crete ques­tions about re­li­gion.

  • Some idea of how the ques­tion can be done with­out us­ing write-ins. Un­for­tu­nately write-in ques­tions add mas­sive amounts of man-hours to the to­tal anal­y­sis time for a sur­vey and make it harder to get out a fi­nal product when all is said and done.

The last sur­vey in­cluded 148 ques­tions; some sec­tions will not be re­peated in the 2017 sur­vey, which gives us an es­ti­mate about our ques­tion bud­get. I would pre­fer to not go over 150 ques­tions, and if at all pos­si­ble come in at many fewer than that. Re­moved sec­tions are:

  • The Basilisk sec­tion on the last sur­vey pro­vided ad­e­quate in­for­ma­tion on the phe­nom­ena it was sur­vey­ing, and I do not cur­rently plan to in­clude it again on the 2017 sur­vey. This frees up six ques­tions.

  • The LessWrong Feed­back por­tion of the last sur­vey also ad­e­quately pro­vided in­for­ma­tion, and I would pre­fer to re­place it on the 2017 sur­vey with a sec­tion mea­sur­ing the site’s re­cov­ery, if any. This frees up 19 ques­tions.

I also plan to do sig­nifi­cant re­form to mul­ti­ple por­tions of the sur­vey. I’m par­tic­u­larly in­ter­ested in mak­ing changes to:

  • The poli­tics sec­tion. In par­tic­u­lar I would like to up­date the ques­tions about feel­ings on poli­ti­cal is­sues with new en­tries and over­haul some of the op­tions on var­i­ous ques­tions.

  • I han­dled the cal­ibra­tion sec­tion poorly last year, and would like to re­place it this year with an eas­ily scored set of ques­tions. To be more spe­cific, a good cal­ibra­tion sec­tion should:

    • Good cal­ibra­tion ques­tions should be fermi es­timable with no more than a stan­dard 5th grade ed­u­ca­tion. They should not rely on par­tic­u­lar hid­den knowl­edge or overly spe­cific in­for­ma­tion. eg. “Who wrote the foun­da­tion nov­els?” is a ter­rible cal­ibra­tion ques­tion and “What is the height of the Eiffel Tower in me­ters within a mul­ti­ple of 1.5?” is de­cent.

    • Good cal­ibra­tion ques­tions should have a mea­surable dis­tance com­po­nent, so that even if an an­swer is wrong (as the vast ma­jor­ity of an­swers will be) it can still be scored.

    • A mea­sure of dis­tance should get pro­por­tionately smaller the closer an an­swer is to be­ing cor­rect and pro­por­tionately larger the fur­ther it is from be­ing cor­rect.

    • It should be eas­ily (or at least sanely) calcu­la­ble by pro­gram­matic meth­ods.

  • The prob­a­bil­ities sec­tion is prob­a­bly due for some re­vi­sion, I know in pre­vi­ous years I haven’t even an­swered it be­cause I found the word­ing of some ques­tions too con­fus­ing to even con­sider.

So for max­i­mum chances of in­clu­sion, it would be best to keep these pro­posed re­forms in mind with your sug­ges­tions.

(Note: If you have sug­ges­tions on ques­tions to elimi­nate, I’d be glad to hear those too.)