Data Scientist
Jan Christian Refsgaard(Jan Christian Refsgaard)
Karma: 436
Thanks, also thanks for pointing out that I had written a few places instead of , since everything is the bernoulli distribution I have changed everything to
You are absolutely right, any framework that punishes you for being right would be bad, my point is that increasing your calibration helps a surprising amount and is much more achievable than “just git good” which is required for improving prediction.
I will try to put your point into the draft when I am off work , thanks
you mean the N’th root of 2 right?, which is what I called the null predictor and divided Scott predictions by in the code:
random_predictor = 0.5 ** len(y)which is equivalent to where is the total number of predictions
I have tried to add a paragraph about this, because I think it’s a good point, and it’s unlikely that you were the only one who got confused about this, Next weekend I will finish part 2 where I make a model that can track calibration independent of prediction, and in that model the 60% 61⁄100 will have a better posterior of the calibration parameter than then 60% 100⁄100, though the likelihood of the 100⁄100 will of course still be highest.
you may be disappointed, unless you make 40+ predictions per week it will be hard to compare weekly drift, the Bernoulli distribution has a much higher variance compared to the normal distribution, so the uncertainty estimate of the calibration is correspondingly wide (high uncertainty of data → high uncertainty of regression parameters). My post 3 will be a hierarchical model which may suite your needs better but it will maybe be a month before I get around to making that model.
If there are many people like you then we may try to make a hackish model that down weights older predictions as they are less predictive of your current calibration than newer predictions, but I will have to think long and hard to make than into a full Bayesian model, so I am making no promises
The order does not matter, you can see that by focusing on which is always equal to , you can also see it from the conjugation rule where you end with no matter the order.
If you wanted the order to matter you could down weight earlier shots or widen the uncertainty between the updates, so previous posterior becomes a slightly wider prior to capture the extra uncertainty from the passage of time.
Mine was the same, I became a bayesian statetisian 4 years ago. I gave a talk about Bayesian Statistics and this figure was what made it click to most students (including myself), so i wanted to share it
fixed
I am well aware that nobody asked for this, but here is the proof that the posterior is for the beta-bernoulli model.
We start with Bayes Theorem:
Then we plug in the definition for the Bernoulli likelihood and Beta prior:
Let’s collect the powers in the numerator, and things that does not depend on in the denominator
Here comes the conjugation shenanigans. If you squint, the top of the distribution looks like the top of a Beta distribution:
Let’s continue the shenanigans, since the numerator looks like the numerator of a beta distribution, we know that it would be a proper beta distribution if we changed the denominator like this:
What’s wrong with it?, I am linking to the source material, should i only link if its a 100% copy?
Most statisticians would agree with you. Unless Reeboks expected market share were between 2⁄3 and 3⁄5 of course. Though I expect that most laymen would have Phil’s Intuition, in any case the general point: That the statement leaks information remains :)
I will write a post shilling for myself, thanks. I was waiting for the post to be ‘liked’, if it got −10 karma then there would be no use in shilling for it :)
I have written a review of the 5 most popular Applied Bayesian Statistics books
Where I recommended:
Statistical Rethinking
Up to speed fast, no integrals, very intuitive approach.
Doing Bayesian Data Analysis
This is the easiest book. If your goal is only to create simple models and you aren’t interested in understanding the details, then this is the book for you.
A Student’s Guide to Bayesian Statistics
This book has the opposite focus of the Dog book. Here the author slowly goes through the philosophy of Bayes with an intuitive mathematical approach.
Regression and Other Stories
Good if you want a slower and thorough approach where you also learn the Frequentest perspective.
Bayesian Data Analysis
The most advanced text, very math heavy, best as a second book after reading one or two of the others, unless you are already a statistician.
If Reebok wanted to report a valid statistic they would report something like 11% of the top 100 wears our shoos, I think a much smaller number than top 100 was picked exactly because that was where the effect was the most exaggerated. ChristianKl share your intuition that 3⁄5 sounds more impressive than 2⁄3. I also agree that reporting top 4 would seem even more fishy, though it could be spun as 75% of quarter finalists in knockout tournament sports.
I disagree, if Reebok produced 64% of all shoes in the world and only 3⁄5 of top athletes used them, and this furthermore was the best statistics the marketing department could produce, then it’s strong evidence that they are over hyped.
But I think you understood me as saying something different, words are hard :)
Good point!
original: Applied Bayesian Statistics—Which book to read?
Applied Bayesian Statistics—Which book should you read?
Literature Review of 5 Applied Bayesian Statistics Books.
Book Review of 5 Applied Bayesian Statistics Books.
I picked 3, if other people have strong feeling feel free to suggest other titles
I think we agree and are talking past each other, my original statement was “Most statisticians would agree with you. Unless...”
So we agree that there is more power in 3⁄5 than 2⁄3, and we happen to have divergent intuitions about what random Joe finds most persuasive, my intuition is rather weak so I would gladly update it towards 3⁄5 sounding more impressive to random people, if you feel strongly about it.
Most likely what the marketing folks have done is gotten a list of top 100 runners in different running disciplines and the reported “the most impressive top X in list Y”,
We both agree that the reported statistic is inflated, which is the major thesis, we simply disagree about how much information can be recovered because we have different “impressiveness sounding heuristics”
I loved that example as well, I have heard it elsewhere described as “The law of small numbers”, where small subsets have higher variance and therefore more frequent extreme outcomes. I think it’s particularly good as the most important part of the Bayesian paragdime is the focus on uncertainty.
The appendix on HMC is also a very good supplement to gain a deeper understanding of the algorithm after having read the description in another book first.
Cholera is the devil!
The National Center for Biotechnology Information has a Taxonomy database.
Q: What do you think taxid=666 is?
A: Vibrio cholerae, coincidence? I think not!
proof:
https://www.ncbi.nlm.nih.gov/Taxonomy/Browser/wwwtax.cgi?mode=info&id=666
I have not been consistent with my probability notation, I sometimes use upper case P and sometimes lower case p, in future posts I will try to use the same notation as Andrew Gelman, which is Pr for things that are probabilities (numbers) such as Pr(y=1)=0.7 and p for distributions such as p∼N(0,2). However since this is my first post, I am afraid that ‘editing it’ will waist the moderators time as they will have to read it again to check for trolling, what is the proper course of action?