I was reflecting on how hard it is to get up to date on climate science recently, and I thought about Reporting likelihoods, not p-values again. So I did some searching, and while I was able to come up with plenty of discussion about absolute and relative likelihood functions, and some tutorials, and various demystify p-values posts, I didn’t see any papers which used them or reports of success using the method.
My expectation is that despite how good the idea seems, the inadequate equilibria of publishing remains, so no surprises there. So now I am wondering about how easy or hard it is to convert already published papers from p-values to likelihood functions. Part of the complaint about p-values is that papers were traditionally opaque about their statistical analysis and did not share their data sets, so overall I expect the problem to be hard. It seems to me I have seen a lot more about success with publishing data sets and sharing analysis software though, so if papers obeying those good practices are chosen that barrier would be overcome. If it is possible to publish a paper of the type “I converted these 5 other papers into likelihood functions and got an interesting result” then there is the added benefit of piling citations on to the papers which use the other best practices.
If journals-gonna-journal and so publishing that is impossible, and a conversion procedure were to be simple enough, it would be worth it as yeoman’s work to build an alternative and more accessible pile of knowledge, and as practice for anyone who wants it.
Hi, I’m a postdoc in climate science, just made an account. I’ve been reading SSC off and on for about two years, then started exploring LW more recently and wanted to join the discussion.
I’m curious what questions you have about climate science, and what resources you think are needed to make it more accessible? More blogs? More easily accessible review papers?
Welcome to LessWrong! I appreciate you popping up.
Out of the gate, I should probably say this isn’t really specific to climate science; getting up to date on any rapidly advancing field is pretty tough. The saturation of political offense and defense just makes it tougher, is all. The likelihood functions over p-values question is one that I expect would help all scientific fields more-or-less equally.
The questions I personally have are mostly about the state of the various feedback-loops or runaway-processes that have been proposed as drivers of radical climate change. For example, the clathrate gun hypothesis; the last thing I read on the subject was commentary from a scientist who had just completed sampling of methane releases in the Arctic Ocean, who thought it had already fired. Sometime later I was reading a summary which largely agrees with the Wikipedia article that the role of this mechanism in past events is not as great as we previously thought/feared, but then later still I read that clathrate mining had officially begun. The example of the American natural gas boom suggests to me that mining will probably make the problem worse. There’s plenty of stuff on the various equilibrium processes like the nitrogen cycle or the sulphur cycle; I feel like something on what are effectively dis-equilibrium processes would also be useful both for learning and for risk evaluation.
For the most part, information about this kind of thing is scattered across papers which are infrequently meta-analyzed, or buried deep in reports like the IPCC’s and limited in nuance. I think more easily accessible—and in particular findable—review papers would be very helpful to me. In particular, if papers which discuss the history and intuition of an open problem could be found, I would love that. To give you a sense of what I mean, take a look at Macroscopic Prediction by ET Jaynes. This was a habit of the author more than anything else and it runs throughout his writings; is there anyone like that in climate science?
More blogs is probably a good idea, but I haven’t delved deeply enough to find out if any of the ones which already exist are actually good—chiefly this is because of the political noise flooding my search results. What I would really like to find is someone working in climate science with a blog like Andrew Gelman or Scott Aaronson, who I could rely on to be an expositor of their personal thinking while flagging important developments. Then I could use that blog as the launchpad for my related searches, and be more productive that way.
You might very well be like “check out <person> and <person>” and completely resolve my difficulties, which would be awesome.
It’s interesting to see what your concerns are. There’s probably less research on these kinds of feedback loops/tail risks then there should be. Part of the problem is just how uncertain they are—a combination of the difficulty of measuring things like methane releases and the problem of not being able to resolve these processes in models. Our best guide is probably paleoclimate observations, but I’m not an expert on these.
In terms of blogs, Real Climate is the best place to start. They can get combative, and they aren’t always the most rigorous researchers, but they at least give you a sense of what’s going on. Isaac Held is a giant in climate science and had a very widely-read blog for a few years, but he’s mostly gone quiet since the Trump administration took over (he’s a federal scientist). Going through his posts is a great way of getting caught up on the field. I put up notes here occasionally. And for someone with more of a “denier” bent, Judith Curry is worth checking out (though she clearly wants to push the discussion in specific directions). Finally, if you have access to it, Nature Climate Change publishes a lot of good stuff (with the caveat that it wants to publish high profile work), including summaries and perspectives which give overviews of specific subfields and questions. Any important work on tipping points and feedback loops will be in a Nature journal.
I found that so-called parapsychology research is suffering from the p-value problem badly. In one book I read some thing like that they tossed a coin 1000 times, and it came 520 heads. The probability of 520 heads from 1000 is like p = 0.01, so they concluded that their result is significant. However, it is still inside standard deviation from 500.
This helped me better understand the problem with p-values: even if they got 500 head and 500 tails, it would still have p-value around 1 per cent. But if any psi-effect were true, their result should be outside standard deviation. In other words, whet they should measure was not the probability of the result, but such probability of the result given their hypothesis.
What does “inside standard deviation from 500” mean?
Having a small p-value is exactly the same thing, at least for approximately normally distributed things like this, as being multiple standard deviations away from the norm.
The specific number here is neither “like p=0.01” nor within 1σ of the mean. Variance of a binomial distribution is npq=250, so standard deviation is just under 16. Being at least 20 away from 500 is approximately a p=0.2 event.
I was reflecting on how hard it is to get up to date on climate science recently, and I thought about Reporting likelihoods, not p-values again. So I did some searching, and while I was able to come up with plenty of discussion about absolute and relative likelihood functions, and some tutorials, and various demystify p-values posts, I didn’t see any papers which used them or reports of success using the method.
My expectation is that despite how good the idea seems, the inadequate equilibria of publishing remains, so no surprises there. So now I am wondering about how easy or hard it is to convert already published papers from p-values to likelihood functions. Part of the complaint about p-values is that papers were traditionally opaque about their statistical analysis and did not share their data sets, so overall I expect the problem to be hard. It seems to me I have seen a lot more about success with publishing data sets and sharing analysis software though, so if papers obeying those good practices are chosen that barrier would be overcome. If it is possible to publish a paper of the type “I converted these 5 other papers into likelihood functions and got an interesting result” then there is the added benefit of piling citations on to the papers which use the other best practices.
If journals-gonna-journal and so publishing that is impossible, and a conversion procedure were to be simple enough, it would be worth it as yeoman’s work to build an alternative and more accessible pile of knowledge, and as practice for anyone who wants it.
Hi, I’m a postdoc in climate science, just made an account. I’ve been reading SSC off and on for about two years, then started exploring LW more recently and wanted to join the discussion.
I’m curious what questions you have about climate science, and what resources you think are needed to make it more accessible? More blogs? More easily accessible review papers?
Welcome to LessWrong! I appreciate you popping up.
Out of the gate, I should probably say this isn’t really specific to climate science; getting up to date on any rapidly advancing field is pretty tough. The saturation of political offense and defense just makes it tougher, is all. The likelihood functions over p-values question is one that I expect would help all scientific fields more-or-less equally.
The questions I personally have are mostly about the state of the various feedback-loops or runaway-processes that have been proposed as drivers of radical climate change. For example, the clathrate gun hypothesis; the last thing I read on the subject was commentary from a scientist who had just completed sampling of methane releases in the Arctic Ocean, who thought it had already fired. Sometime later I was reading a summary which largely agrees with the Wikipedia article that the role of this mechanism in past events is not as great as we previously thought/feared, but then later still I read that clathrate mining had officially begun. The example of the American natural gas boom suggests to me that mining will probably make the problem worse. There’s plenty of stuff on the various equilibrium processes like the nitrogen cycle or the sulphur cycle; I feel like something on what are effectively dis-equilibrium processes would also be useful both for learning and for risk evaluation.
For the most part, information about this kind of thing is scattered across papers which are infrequently meta-analyzed, or buried deep in reports like the IPCC’s and limited in nuance. I think more easily accessible—and in particular findable—review papers would be very helpful to me. In particular, if papers which discuss the history and intuition of an open problem could be found, I would love that. To give you a sense of what I mean, take a look at Macroscopic Prediction by ET Jaynes. This was a habit of the author more than anything else and it runs throughout his writings; is there anyone like that in climate science?
More blogs is probably a good idea, but I haven’t delved deeply enough to find out if any of the ones which already exist are actually good—chiefly this is because of the political noise flooding my search results. What I would really like to find is someone working in climate science with a blog like Andrew Gelman or Scott Aaronson, who I could rely on to be an expositor of their personal thinking while flagging important developments. Then I could use that blog as the launchpad for my related searches, and be more productive that way.
You might very well be like “check out <person> and <person>” and completely resolve my difficulties, which would be awesome.
Thanks for the response!
It’s interesting to see what your concerns are. There’s probably less research on these kinds of feedback loops/tail risks then there should be. Part of the problem is just how uncertain they are—a combination of the difficulty of measuring things like methane releases and the problem of not being able to resolve these processes in models. Our best guide is probably paleoclimate observations, but I’m not an expert on these.
In terms of blogs, Real Climate is the best place to start. They can get combative, and they aren’t always the most rigorous researchers, but they at least give you a sense of what’s going on. Isaac Held is a giant in climate science and had a very widely-read blog for a few years, but he’s mostly gone quiet since the Trump administration took over (he’s a federal scientist). Going through his posts is a great way of getting caught up on the field. I put up notes here occasionally. And for someone with more of a “denier” bent, Judith Curry is worth checking out (though she clearly wants to push the discussion in specific directions). Finally, if you have access to it, Nature Climate Change publishes a lot of good stuff (with the caveat that it wants to publish high profile work), including summaries and perspectives which give overviews of specific subfields and questions. Any important work on tipping points and feedback loops will be in a Nature journal.
I found that so-called parapsychology research is suffering from the p-value problem badly. In one book I read some thing like that they tossed a coin 1000 times, and it came 520 heads. The probability of 520 heads from 1000 is like p = 0.01, so they concluded that their result is significant. However, it is still inside standard deviation from 500.
This helped me better understand the problem with p-values: even if they got 500 head and 500 tails, it would still have p-value around 1 per cent. But if any psi-effect were true, their result should be outside standard deviation. In other words, whet they should measure was not the probability of the result, but such probability of the result given their hypothesis.
What does “inside standard deviation from 500” mean?
Having a small p-value is exactly the same thing, at least for approximately normally distributed things like this, as being multiple standard deviations away from the norm.
The specific number here is neither “like p=0.01” nor within 1σ of the mean. Variance of a binomial distribution is npq=250, so standard deviation is just under 16. Being at least 20 away from 500 is approximately a p=0.2 event.