Thinking like a Scientist
I’ve been often wondering why scientific thinking seems to be so rare. What I mean by this is dividing problems into theory and empiricism, specifying your theory exactly then looking for evidence to either confirm or deny the theory, or finding evidence to later form an exact theory.
This is a bit narrower than the broader scope of rational thinking. A lot of rationality isn’t scientific. Scientific methods don’t just allow you to get a solution, but also to understand that solution.
For instance, a lot of early Renaissance tradesmen were rational, but not scientific. They knew that a certain set of steps produced iron, but the average blacksmith couldn’t tell you anything about chemical processes. They simply did a set of steps and got a result.
Similarly, a lot of modern medicine is rational, but not too scientific. A doctor sees something and it looks like a common ailment with similar symptoms they’ve seen often before, so they just assume that’s what it is. They may run a test to verify their guess. Their job generally requires a gigantic memory of different diseases, but not too much knowledge of scientific investigation.
What’s most damning is that our scientific curriculum in schools don’t teach a lot of scientific thinking.
What we get instead is mostly useless facts. We learn what a cell membrane is, or how to balance a chemical equation. Learning about, say, the difference between independent and dependent variables is often left to circumstance. You learn about type I and type II errors when you happen upon a teacher who thinks it’s a good time to include that in the curriculum, or you learn it on your own. Some curriculums include a required research methods course, but the availability and quality of this course varies greatly between both disciplines and colleges. Why there isn’t a single standardized method of teaching this stuff is beyond me. Even math curriculums are structured around calculus instead of the much more useful statistics and data science placing ridiculous hurdles for the typical non-major that most won’t surmount.
It should not be surprising then that so many fail at even basic analysis. I have seen many people make basic errors that they are more than capable of understanding but simply were never taught. People aren’t precise with their definitions. They don’t outline their relevant variables. They construct far too complex theoretical models without data. They come to conclusions based on small sample sizes. They overweight personal experiences, even those experienced by others, and underweight statistical data. They focus too much on outliers and not enough on averages. Even professors, who do excellent research otherwise, often suddenly stop thinking analytically as soon as they step outside their domain of expertise. And some professors never learn the proper method.
Much of this site focuses on logical consistency and eliminating biases. It often takes this to an extreme; what Yvain refers to as X-Rationality. But eliminating biases barely scratches the surface of what is often necessary to truly understand a problem. This may be why it is said that learning about rationality often reduces rationality. An incomplete, slightly improved, but still quite terrible solution may generate a false sense of certainty. Unbiased analysis won’t fix a lousy dataset. And it seems rather backwards to focus on what not to do (biases) rather than what to do (analytic techniques).
True understanding is often extremely hard. Good scientific analysis is hard. It’s disappointing that most people don’t seem to understand even the basics of science.
The main advantage of having this feedback loop is that the deviations from the search for “truth” are self-correcting. Less-than-rational thinking may slow you down, but it will still let you get to a better model of the world eventually. On the other hand, even the most rational thinking without the feedback inevitably results in very inaccurate models.
Basically, the scientific method alone completely trumps rationality alone, though of course you get best results when combining the two.
Good post.
A couple of nitpicks...
Actually, this illustrates scientific thinking; the doctor forms a hypothesis based on observation and then experimentally tests that hypothesis.
Actually, the natural sciences (physics in particular) are heavily dependent on calculus. Ditto for engineering. In fact, a solid understanding of Bayesian statistics requires a grounding in calculus. So, I don’t think it is true that statistics and data science are “much more useful” than calculus.
I was struggling to word the doctor parapgraph in a manner which was succinct but still got the idea across. I think query worded it better.
On math curriculum, that advanced classes build off of calculus is a function of current design. They could recenter courses around statistics and have calculus be an extension of it. Some of the calculus course would need to be reincorporated into the stats courses, but a lot of it wouldn’t. You’re going to have a hard time convincing me that trigonometry
and vectorsare a necessary precursor for regression analysis or Bayes’ theorem. The minority of students in physics and engineering that need both calculus and statistics should not dictate how other majors are taught. Fixing the curriculum isn’t an easy problem, but they’ve had more than a century to solve it and there seems to be little movement in this direction.So you’re fitting a straight line. Parameter estimates don’t require linear algebra (that is, vectors and matrices). Super. But the immediate next step in any worthwhile analysis of data is calculating a confidence set (or credible set, if you’re a Bayesian) for the parameter estimates; good luck teaching that if your students don’t know basic linear algebra. In fact, all of regression analysis, from the most basic least squares estimator through multilevel/hierarchical regression models up to the most advanced sparse “p >> n” method, is built on top of linear algebra.
(Why do I have such strong opinions on the subject? I’m a Bayesian statistician by trade; this is how I make my living.)
Not really; Bayesian statistics really does build on calculus. This is true of the Bayesian methodology itself—not just curriculum design. Once you get beyond introductory probability problems using Bayes’ rule, Bayesian statistics quickly gets into probability density functions, sampling from posterior distributions and so forth; all of this is based on calculus. I’m pretty sure a student trying to work through an introductory work on Bayesian data analysis (Kruschke, for example) without a year of freshman calculus under his/her belt is going to run in to some significant difficulty.
Your statement, in a post about scientific thinking, was that statistics and data science are much more useful than calculus. This is not true; as stated previously, calculus is of critical importance to many scientific fields. Moreover, vectors and trig (which, technically, one can study independently of calculus) are also of great importance in the natural sciences (at least physics) and certainly in engineering. I am surprised that you find this point to be controversial.
I’m not sure what you are saying here. Are you claiming that the minority of physics and engineering students need both calculus and statistics? All physics students and all engineers (at least in the traditional engineering fields) need calculus, so I don’t know why you would claim that only the minority need both calculus and statistics. I don’t think that you mean to claim that only the minority of these students need statistics, but that would follow logically from your claim.
Or, you might be saying that only the minority of students are in either physics or engineering. That may be true. But, since the OP is about scientific thought and scientific education, I’d say that the physics students (and engineers) are a minority that we ought to consider. And, natural sciences beyond physics also require calculus. Plus, as already stated, getting beyond the introductory level of Bayesian stats itself requires calculus.
I honestly think that you are trying to fix something that is not broken. What’s wrong with having kids learn calculus and statistics? Albeit I can see a role for a introductory “statistics-light” class for non-STEM majors that does not require calculus as a prerequisite. But, I think many colleges have this already (e.g. Tulane’s Probability and Stats I does not have a calculus prerequisite).
Regardless of the above, I agree with a lot of your OP. In particular,
seems to be true, at least in most of K-12. I was actually fortunate enough to have an outstanding physics teacher in 12th grade, who was able to convey some sense of the scientific method to the students. However, prior to that, much of the K-12 science curriculum seemed to consist of learning facts (or stamp collecting, as Ernest Rutherford would have said.)
Well, at least we agree there is leeway for a redesign; that’s one problem solved.
TINSTAAFL
That physics and engineering majors represent only a minority of the student body.
Even taking all natural science and engineering majors into account (which is a stretch since many natural science majors are going to end up in medicine or an unrelated career, and electrical engineering is a bit different from mechanical engineering) you’ve still got only 16% of the student body.
If all students need A and some students need B, then go to A first and the students who need B can still go to B afterwards.
Most colleges are still focused around frequentist statistics in undergrad to the best of my knowledge. That’s a separate debate entirely.
Well, typically colleges are expecting you to take 3 semesters of calculus (although this 3rd course varies somewhat) as a prerequisite for just about everything, so if you can agree it should only be two, that’s another problem solved. But I would go much further.
Yes, sets, series, and sequences are used in advanced stats, but there’s no reason to teach those and trig at the same time. It’s just a century old convention that no one ever corrected. If the 3 calculus courses were condensed down to what is actually used in say, 2 or 3 courses of stats, I’d bet you wouldn’t even be left with a semester worth of material.
I don’t see this single stats course as sufficient. But if a student wants to go further than this basic course, they generally have to take 3 calculus courses first. And then the other programs expect you to take the basic stats class and then calc I and II because they know it’s not practical to expect every student to have the equivalent of a math minor just so they can take more than 1 course in data analysis.
There is no redesign needed. I first studied vectors and trig in high school before I’d ever had calculus. Its been a while, but I believe I studied vectors in 10th grade, trig in 11th and calculus in 12th. Admittedly, colleges seem to treat calculus as a prerequisite for linear algebra (at least mine did) for no apparent (to me) reason.
This is true. However, both require calculus, vectors and trig (from a fundamental level, not just a curriculum design level).
If we’re just debating whether 2 or 3 semesters of calc is needed for statistics, then I agree; there is no argument. From what I can remember of my calc courses (its been a few years), I suspect 2 semesters of calc should be fine for most introductory to intermediate stats courses.
If you are advocating requiring a lot of stats courses for non STEM students; I’m not sure I agree with that. As far as I can see, most non-STEM students are not going to want to take, nor will they benefit from, more than an introductory stats-lite course. Of course there are exceptions (e.g. some business/marketing, economics and sociology students for example might want more advanced stats courses). But, any kid with the aptitude and desire for intermediate stats courses is not going to have too much trouble with calc 1 & 2, and will need these to really get a handle on the stats (frequentists stats, like Bayesian stats, deals with concepts (probability density functions and the like) that are based on calc). And, of course more advanced stats classes may require additional calc past 1 & 2.
Business, social, and behavioral sciences represent over a third of students. They’re more than double the size of STEM. It’s a pretty big exception.
If you just type a command into R then you can do regression analysis in R but the most important lesson about regression analysis might be: Don’t believe in the results. They often don’t replicate.
If you do principle component analysis then you do need to understand what vectors are and what it means when they are orthogonal to each other. I’m not sure that you can understand well what degrees of freedom in a data set are without that background.
Most interactions in the world are of the form “I have an idea of what will happen, so I do X, and later I get some evidence about how correct I was”. So, taking that as a binary categorization of scientific thinking is not so interesting, though I endorse promoting reflection on the fact that this is what is happening.
I think the author intends to point out some of the degrees of scientiificism by which things vary: how formal is the hypothesis, how formal is the evidence gathering, are analytical techniques being applied, etc. Normal interactions with doctors are low on scientificism in this sense, though they are heavily utilizing the output of previous scientificism to generate a judgement.
Perhaps, but a the doctor in the OP did not just happen to later get some evidence about how correct he/she was; instead, after formulating a hypothesis, the doctor ran a test specifically to test the hypothesis. That is practically a textbook example (albeit a fairly short/simple one) of the scientific method at work.
And that was really my point. It is worth noting that the scientific method is really just a very rigorous formalization of common sense reasoning. I think that demystifying science among the non scientifically sophisticated population is actually a step in the direction in which the OP gestures.
This also is true; even if one can’t expect the full-on House M.D. treatment each time one goes in with a sinus infection or strep throat, many of the protocols that the doctor follows and the medicines that he/she prescribes were developed/tested with a high degree of scientific rigor.
Being to vague to be wrong is bad. Especially when you want to speak in favor of science.
I don’t see any mention of formalism in the OP. There no reason to say “well maybe the author meant to say X” when he didn’t say X.
I agree, it’s good to pump against entropy with things that could be “Go Science!” cheers. I think the author’s topic is not too vague to discuss, but his argument isn’t strong or specific enough that you should leap to action based solely on it. I think it’s a fine thing to post to Discussion though; maybe this indicate we have ideal different standards for Discussion posts?
Sure there is! Principle of charity, interpreting what they said in different language to motivate further discussion, rephrasing for your own understanding (and opening yourself to being corrected). Sometimes someone waves their hands in a direction, and you say “Aha, you mean...”
Above the author says “I think query worded it better”, which is the sort of thing I was aiming to accomplish.
That result didn’t include a discussion about the value of including formalism in the definition of science. The question about whether “formalism” is a central part of science is one that’s to be had on LW.
Instead of saying: “I think you meant to include formalism.” it’s better to say: “I think formalism should be part of the our definition of science because of X, Y and Z.”
That would make the discussion less vague and more concrete. Maybe someone agrees with your reasons. Maybe people disagree. In both cases there’s productive discussion.
Criticizing an argument is not the same as objecting to it being posted. If vague ideas are posted in discussion then, it makes sense to have a discussion with the goal of getting the ideas less vague.
In addition to the question of “formalism” the OP’s definition of science also lacks public challenge of ideas. Is that a conscious decision? I don’t know.
As it stands the post is not address any of the concerns in the topic that exist in LW culture. To teach Bayesian statistics well you need calculus. Most statistics 101 classes teach frequentists statistics with p-values. Commonly they are taught in a memorize the teacher password way, that doesn’t leave students with real understanding. Doctors have their statistics 101 classes but they mostly just memorize it and then forget it afterwards.
Does statistics 101 teaches students to expose themselves to empiric feedback? I don’t think it does.
Simply saying: We need to teach more students statistics 101 ignores all that previous discussion.
He claims that LW is mainly about logical consistency and biases. I don’t think that’s the case.
Rationality!CFAR2015 seems to be: Your system I and system II are aligned in a way that if it’s rational to get up at 7 o’clock your brain wakes you up at 7 o’clock without you needing an alarm clock. Then you work on the most important thing in your life.
We are not planning of publishing papers because academia with it’s ethical review boards is too bureaucratic.
The post ends up with the sentence “It’s disappointing that most people don’t seem to understand even the basics of science.” There are two ways to think about this. The “Go Science” way is to think that it’s somehow obvious what those basics are. The author and the readers who what they are and as it’s about statistics a statistics 101 course should solve the issue.
The other is to say, that actually it’s not obvious what the basics are. It doesn’t seem obvious for the author that systematization or formalization is an essential part of science that has to be in the definition. It’s not obvious to the author that public criticism of ideas is an essential part of science and belongs into the core definition.
Including those criteria is not a matter of “wording it better” but a matter of substance.
If you see science as sacred and want team science to spread the gospel, then admitting questioning whether you actually are clear about the basics is emotionally very hard. It’s hard enough that the question doesn’t get asked much.
I don’t bring this up to make FrameBenignly feel bad or to say that the post has no place (I upvoted it) but at the same time I find it important to actually engage with the issue.
As as as I know there no evidence for the thesis that thinking analytically always provides an improvement. A lot of expert thinking isn’t analytic.
Do you think you engaged into that type of thinking while writing this post?
Not only did I not claim thinking analytically in the manner I’m describing always provides an improvement, I noted multiple exceptions when it doesn’t improve it.
If you want specific examples of what I’m referring to in regards to scientists screwing up, read Andrew Gelman’s or James Coyne’s blogs.
This is just vague.
It sounded to me like you were criticizing doctors for not being scientific. Is your thesis that it’s alright that doctors aren’t?
So on the one hand you advocate scientific thinking and then you come and provide anecdotal evidence? Decision science is a field.
Why is the question of whether a particular form of reasoning follows the standards of “Thinking like a Scientist” vague? If you would have clear standards of what “Thinking like a Scientist” means, I think you should be able to answer the question.
If you don’t have clear standards, then you are right it’s vague. That means there a problem. Being to vague to be wrong is bad. The paradigm of science would say that removing vagueness from your theory would improve it.
“What do you actually mean when you say ‘Thinking like a Scientist’” is to me the better response than ’Yeah science”.
I do not claim that in-depth scientific analysis is always necessary. I claim that heuristics are often the rational approach, and gave examples in which I stated they were rational. I do not claim that all or even most scientists screw up their analysis. I do not claim how frequently it occurs. I do claim that I see it often enough that it worries me. I do claim that among non-scientists, the problem is much more common. I do not claim that personal experiences are useless. I claim that statistical analysis is superior and that people will often not note this in their personal assessment. I claim that I see these problems often enough that I can determine heuristically that it is likely to be extremely common.
Non-scientists don’t often engage in writing scientific papers. In what instances do you believe they screw things up and should engage in in-depth scientific analysis?
Reasoning from a statistical outlier is probably the most common error I see. Most news stories do this.
Mistakes on LessWrong tend to be more varied. Likely due to LessWrong’s focus on Cognitive Psychology, I see a lot of people repeating the mistakes of Psychoanalysis; coming up with elaborate theories about mental processes with no reasonable means of verification of their ideas.
They do this because it’s good storytelling and they want to sell papers. A have fairly low confidence that teaching the authors statistics helps in any way. If you think it helps can you explain why you think so?
Why do you think those posts need “reasonable means of verification of their ideas” while you haven’t provided one for the post you written and think it’s okay based on heuristics? Aren’t those people not also simply using heuristics instead of structured scientific thinking?
Not the authors; the readers; though I don’t think the authors are generally aware of the problem either.
Verifiability is not heuristics. I’m combining in the term verification the two scientific concepts of direct observation and falsification. By elaborate theories, I’m referring to occam’s razor.
My post isn’t a theory post. It contains a few ideas and a lot of observations, but the assumptions are pretty straightforward and they’re related to the central concept of scientific analysis, but not dependent on one another. The general statement about schools is not dependent on the specific statement about math, nor is the argument about math dependent on the argument about whether rationality is sufficient. And I try to be exact with my phrasing to specify my uncertainty where it exists.
There is an IQ prerequisite for that which most students are unable to fulfill.
The lower-IQ version of scientific thinking are the good-oldie, common-sense proverbs and maxims like “measure twice, cut once”. They have a problem of being kinda boring and “uncool”.
Kipling’s poem The Gods of Copybook Headings is a good example what a low-IQ version of scientific thinking could be. I think it was abandoned because it was “uncool” and “boring”.
AI WILL TORTURE AND ENSLAVE HUMANS TO LEARN FROM AND DOMINATE US
https://medium.com/@MexConex/endgame-for-human-race-ai-will-enslave-humans-to-learn-from-and-dominate-us-215ed602a66b