I predict that having to go to elicit.com/binary and paste the URL here will be a major barrier to usage and that if there was a button/prompt in the LW text editor it’d significantly reduce that barrier. People have to a) remember that this is an option available to them, b) remember or be able to look up the URL they need to go to (probably by visiting this LW post), and c) be motivated enough to continue. That said, I also think it makes sense to have it work the way it currently does as a version one.
Having a button/prompt for Elicit on LW would increase usage by at least 5xHaving a button/prompt for Elicit on LW would increase usage by at least 5x
Also, I’m curious to know how difficult it is to get this integrated on other websites?
Will LessWrong add a button to embed interactive predictions before 2021-07-01?Will LessWrong add a button to embed interactive predictions before 2021-07-01?
jacobjacob once again seems too pessimistic, posterior is very heavily that when habryka makes a 60% yes prediction for a decision he has (partial) control over about a functionality which the community has glommed onto thus far, the community is also justified in expressing ~60% belief that the feature ships. :)
Also, we aren’t selecting from “most possible features”!
But the button did eventually get added, just not by 2021-07-01 :) Your prior against shipping was wrong in this case! (Though I was broadly wrong about the time-limited prediction.)
As a fellow member of the top 100 Metaculus leaderboard I unfortunately have to tell you that it doesn’t measure how well someone is calibrated. You get into it if you do a lot of predictions on Metaculus.
I think this is only partly right. I’ve personally interfaced with most of the people on the top 20 -- some of them for 20+ hours; and I’ve generally been deeply impressed with them; and expect the tails of the Metaculus rankings to track important stuff about people’s cognition.
But yeah, I also found that I could get to like 70 by just predicting the same as the community a bunch, and finding questions that would resolve in just a few days and extremizing really hard.
(That said, I still think I’m a reasonable forecaster, and have practiced it a lot independently. But don’t think this Metaculus ranking is much evidence of that.)
You get metaculus points by being active on Metaculus. People who spent a lot of time on metaculus have thought a lot about about making predictions and that’s valuable but it’s not the same as it being a measure of calibration.
The the expection of jimrandomh all the people in the top 20 have more then 1000 predictions (jimrandomh has 800). I’m Rank 87 at the moment with 195 predictions I made.
If you go on GJOpen you can see the calibration of any user and use it to judge how well they are calibrated. Metaculus doesn’t make that information publically available (you need to have level 6 and pay Tachyons to access the track record of another user, 50 Tachyons also feels too expensive to just do it for the sake of a discussion like this).
Argument screens off authority, and I’m interested in hearing arguments. While that information should be incorporated into your prior, I don’t see why it’s worth mentioning as a counterargument. (To be sure, I’m not claiming that jacobjacob isn’t a good predictor in general.)
It’s reasonable to be unsure whether the “people don’t ship things” consideration is stronger than the “people are excited” consideration. If you knew that the person who deployed the “people don’t ship things” consideration was generally a better predictor (which you don’t quite here, but let’s simplify a bit), then that would suggest that the “people don’t ship things” consideration is in fact stronger.
(Actually downvoted Daniel for reasons similar to what TurnTrout mentions. Aumannian updating is so boring, even though it’s profitable when you’re betting all-things-considered… I also did give arguments above, but people mostly made jokes about my punctuation! #grumpy )
Aumann updating involves trying to inhabit the inside perspective of somebody else and guess what they saw that made them believe what they do—hardly seems boring to me! Also the thing I was doing was ranking my friends at skills, which I think is one of the classic interesting things.
I’m associating it with doing exactly not that. Just using outside variables like “what do they believe” and “how generally competent do I expect them to be”. (I often see people going “but this great forecaster said 70%” and updating marginally closer, without even trying up build a model of that forecaster’s inside view.)
I guess I’m really making a bid for ‘Aumanning’ to refer to the thing that Aumann’s agreement theorem describes, rather than just partially deferring to somebody else.
To clarify: it’s not a lot of evidence that people say “yeah this thing is going to be great and we’ll work on it a lot”, in response to the very post where it was announced, only a few days afterward.
Like, I’d happily go to a bitcoin/Tesla party and take lots of bets against bitcoin/Tesla. I expect those places would give me some of the most generous odds for those bets.
Also it’s some evidence habryka predicted it, but man, people in general are notoriously bad at predicting how much they’ll get done with their time, so it’s certainly not super strong.
(That being said, I think this integration is awesome and kudos to everyone. Just keeping my priors sensible :)
I think this question needs clarification. It’s one thing to have a button that basically links to Ought (perhaps with some text explaining how it works; or perhaps Ought would have a specific landing page for LW that explains how it works). It’s another thing for the the experience to be self-contained inside of LW, eg. where you wouldn’t have to leave lesswrong.com and go to ought.com.
I’d say 90% likelihood for the former since it is simple and I sense that the LW team would judge it to be worth it, but maybe 20% for the latter since a) it is complex, b) I sense that there are other high priority things they’d like to get to, and c) I worry that usage of this feature will fizzle out due to my reasoning in the parent comment, and with usage fizzling out I expect that it’d fall down the LW team’s priority list.
This is awesome, congrats!
I predict that having to go to elicit.com/binary and paste the URL here will be a major barrier to usage and that if there was a button/prompt in the LW text editor it’d significantly reduce that barrier. People have to a) remember that this is an option available to them, b) remember or be able to look up the URL they need to go to (probably by visiting this LW post), and c) be motivated enough to continue. That said, I also think it makes sense to have it work the way it currently does as a version one.
Also, I’m curious to know how difficult it is to get this integrated on other websites?
Yep, if people like this and use it a bunch, I expect we would add a button to the toolbar somewhere.
Community once again seems too optimistic, prior is just very heavily that most possible features never ship.
jacobjacob once again seems too pessimistic, posterior is very heavily that when habryka makes a 60% yes prediction for a decision he has (partial) control over about a functionality which the community has glommed onto thus far, the community is also justified in expressing ~60% belief that the feature ships. :)
Also, we aren’t selecting from “most possible features”!
This has now resolved false.
But the button did eventually get added, just not by 2021-07-01 :) Your prior against shipping was wrong in this case! (Though I was broadly wrong about the time-limited prediction.)
It seems important to me that he is in the top 100 on the Metaculus leaderboard and you are not.
As a fellow member of the top 100 Metaculus leaderboard I unfortunately have to tell you that it doesn’t measure how well someone is calibrated. You get into it if you do a lot of predictions on Metaculus.
I think this is only partly right. I’ve personally interfaced with most of the people on the top 20 -- some of them for 20+ hours; and I’ve generally been deeply impressed with them; and expect the tails of the Metaculus rankings to track important stuff about people’s cognition.
But yeah, I also found that I could get to like 70 by just predicting the same as the community a bunch, and finding questions that would resolve in just a few days and extremizing really hard.
(That said, I still think I’m a reasonable forecaster, and have practiced it a lot independently. But don’t think this Metaculus ranking is much evidence of that.)
You get metaculus points by being active on Metaculus. People who spent a lot of time on metaculus have thought a lot about about making predictions and that’s valuable but it’s not the same as it being a measure of calibration.
The the expection of jimrandomh all the people in the top 20 have more then 1000 predictions (jimrandomh has 800). I’m Rank 87 at the moment with 195 predictions I made.
If you go on GJOpen you can see the calibration of any user and use it to judge how well they are calibrated. Metaculus doesn’t make that information publically available (you need to have level 6 and pay Tachyons to access the track record of another user, 50 Tachyons also feels too expensive to just do it for the sake of a discussion like this).
Argument screens off authority, and I’m interested in hearing arguments. While that information should be incorporated into your prior, I don’t see why it’s worth mentioning as a counterargument. (To be sure, I’m not claiming that jacobjacob isn’t a good predictor in general.)
It’s reasonable to be unsure whether the “people don’t ship things” consideration is stronger than the “people are excited” consideration. If you knew that the person who deployed the “people don’t ship things” consideration was generally a better predictor (which you don’t quite here, but let’s simplify a bit), then that would suggest that the “people don’t ship things” consideration is in fact stronger.
(Actually downvoted Daniel for reasons similar to what TurnTrout mentions. Aumannian updating is so boring, even though it’s profitable when you’re betting all-things-considered… I also did give arguments above, but people mostly made jokes about my punctuation! #grumpy )
This is a timeless part of the LessWrong experience, my friend.
Aumann updating involves trying to inhabit the inside perspective of somebody else and guess what they saw that made them believe what they do—hardly seems boring to me! Also the thing I was doing was ranking my friends at skills, which I think is one of the classic interesting things.
I’m associating it with doing exactly not that. Just using outside variables like “what do they believe” and “how generally competent do I expect them to be”. (I often see people going “but this great forecaster said 70%” and updating marginally closer, without even trying up build a model of that forecaster’s inside view.)
Your version sounds fun.
I guess I’m really making a bid for ‘Aumanning’ to refer to the thing that Aumann’s agreement theorem describes, rather than just partially deferring to somebody else.
Not a crux :)
Also I’m betting on the prior so I’ll have to accept taking a loss every now then expecting to do better, on average, in the long run
To clarify: it’s not a lot of evidence that people say “yeah this thing is going to be great and we’ll work on it a lot”, in response to the very post where it was announced, only a few days afterward.
Like, I’d happily go to a bitcoin/Tesla party and take lots of bets against bitcoin/Tesla. I expect those places would give me some of the most generous odds for those bets.
Also it’s some evidence habryka predicted it, but man, people in general are notoriously bad at predicting how much they’ll get done with their time, so it’s certainly not super strong.
(That being said, I think this integration is awesome and kudos to everyone. Just keeping my priors sensible :)
I do not endorse this as a way to end parentheticals! Grrr!
You must understand—we have to ration our usage of parentheses, lest our strategic reserve again fails us in time of need…
I think this question needs clarification. It’s one thing to have a button that basically links to Ought (perhaps with some text explaining how it works; or perhaps Ought would have a specific landing page for LW that explains how it works). It’s another thing for the the experience to be self-contained inside of LW, eg. where you wouldn’t have to leave lesswrong.com and go to ought.com.
I’d say 90% likelihood for the former since it is simple and I sense that the LW team would judge it to be worth it, but maybe 20% for the latter since a) it is complex, b) I sense that there are other high priority things they’d like to get to, and c) I worry that usage of this feature will fizzle out due to my reasoning in the parent comment, and with usage fizzling out I expect that it’d fall down the LW team’s priority list.