Or: how the Adventist Health Study-2 had a pretty good study design but was oversold in popular description, and then misrepresented its own results.
When I laid out my existing beliefs on veganism and nutrition I asked people for evidence to the contrary. By far the most promising thing people shared was the 7th Day Adventist Health Studies. I got very excited because the project promised something of a miracle in nutrition science: an approximate RCT. I read the paper that included vegan results, and while it’s still very good as far as nutrition studies go it’s well below what I was promised, and the summaries I read were misleading. It’s not a pseudo-RCT, and even if you take the data at face value (which you shouldn’t) it doesn’t show a vegan diet is superior to all others (as measured by lifespan). Vegan is at best tied with pescetarianism, and in certain niche cases (e.g. being a woman) it’s the worst choice besides unconstrained meat consumption.
I’m going to try not to be too sarcastic about this, the study really is very good data by nutrition science standards, but I have a sour spot for medical papers that say “people” when they mean “men”, so probably something will leak out. Also, please consider what the state of nutrition science must be to make me call something that made this mistake “very good”.
Background
The 7th Day Adventists are a fairly large Christian sect. For decades scientists have been recruiting huge cohorts to study their diet, and publishing a lot of papers.
The Adventists are a promising group to use to study nutrition for lots of reasons, but primarily because the Church discourages meat, smoking, and drinking. So you lose the worst confounders for health, and get a population of lifelong, culturally competent vegetarians, which is a pretty good deal. Total abstinence from meat isn’t technically required – you’re allowed to eat kosher meat – but it’s heavily discouraged. 7DA colleges only serve vegetarian meals, and church meals will typically be vegetarian.
Some popular descriptions say that rules vary between individual churches, which could give you an almost RCT effect. AFAICT this isn’t true, and the paper I read never claimed it was. Both the internet and my ex-Adventist friend say that individual churches within the US (where the study took place) vary a little in recommendations, but most of the variety in diet is based on individual choice, not local church rules.
I was really hoping this project could shed light on what happened to people who had medical difficulties with plant-exclusive diets in plant-exclusive food cultures, but they didn’t try, and from the abundance of meat eaters within 7DA I’d guess the answer is that they eat animal products. Nor is the study very informative about naively switching to a plant-exclusive diet, since most members grow up in the culture and will have been taught a reasonable plan-based diet without necessarily needing to consciously think about it.
The Adventist Health Studies program has produced a lot of papers over the years. I focused on Vegetarian Dietary Patterns and Mortality in Adventist Health Study 2, because someone commented on my last post and pointed to that paper as addressing veganism in particular (other papers look like they only consider vegetarianism, although they might be using it as an umbrella term). I also read the AHS-2 cohort profile, but not any of the other papers, due to time constraints. It’s possible I will raise questions those papers would have answered, but there are at least 15 papers and, as I’ll talk about later, I wasn’t feeling hopeful.
So let’s talk about that one paper. It breaks people down into 2 main categories, one of which has 4 subtypes. The big category is “nonvegetarian”, which they defined as eating meat more than once per week (48% of the sample). If you eat meat less than or equal to once a week you qualify for some category of vegetarian. More specifically:
Vegan (8%): consumes any animal product <1/month.
Lacto-ovo-vegetarian (29%): unlimited consumption of eggs and milk, meat < 1/month.
Pesco-vegetarian (10%): unlimited fish, eggs, and milk, other meat < 1/month
Semi-vegetarian (6%): unlimited eggs and milk, meat+fish combined > 1/month but no more than 1/week.
I’m already a little
I wish we knew more about the specific diets of the nonvegetarians (although not enough to trawl through 20 papers hoping to find that data). Do they eat pork or shellfish, which violates Church requirements and would thus be a great marker for low conscientiousness? Are they eating Standard Shitty American Diet, or similar to the lacto-ovo-vegetarians but with meat 5x/month?
Statistics
In this sample, eating lots of meat is clearly correlated with multiple activities known to impair health, like smoking and drinking. The paper attempts to back out those effects, but that’s impossible to do fully. You can back out 100 things and still miss the effect of traits that makes doing all those things more or less likely. That’s how you get results like “theater attendance reduces the risk of death even after controlling for a laundry list of confounders”.
The mortality results in this paper are controlled for: age, smoking, exercise, income, education level, marital status, alcohol consumption, and sleep. In women they also adjusted for menopausal status and hormone therapy (vegans were ½ to ⅓ less likely to be on hormone therapy, compared to other groups).
What kinds of things does this leave out? Conscientiousness, religiosity (which has social implications), and overall concern for health (anything with enough of a cultural “health halo” will eventually show a correlation with long life, because it will be done more by people who care the most about health, and they’re likely also doing other things that help). It will also be confounded if poor health is the thing that causes people to consume more animal products.
My statistician described their methods of backing out the confounding effects as “This isn’t what I would have done but it doesn’t seem unreasonable”, which easily puts this in the top 10% of medical papers I’ve asked him to evaluate. He didn’t scream in pain even a little bit. I can’t check their actual math without the raw data, but for purposes of this post I feel comfortable assuming they correctly adjusted for everything they listed.
Claims of vegetarianism’s benefits are greatly exaggerated
Note: most of these results don’t reach the level of statistical significance, but let’s ignore that for now. I’m also going to ignore concerns they failed to fully back out health effects with non-dietary causes, because even if they did, the data doesn’t support their own conclusions, much less the idea veganism is superior.
The abstract says “Significant associations with vegetarian diets were detected for cardiovascular mortality, noncardiovascular noncancer mortality, renal mortality, and endocrine mortality. Associations in men were larger and more often significant than were those in women.”
Sure, if you count “unlimited fish” as vegetarian.
But it’s much worse than that. If you look at men, you see veganism’s hazard ratio is tied with pescetarianism, followed by other forms of vegetarianism, and then nonvegetarian. Seems unfair to quote this as total vindication for veganism, although it is definitely a blow against unrestricted meat eating.
But then we get to that niche group, women. This data is a little noisy because women only made up ⅔ of the sample, but you can nonetheless get a faint hint that veganism is barely distinguishable from unrestricted omnivorism, and the diet correlated with the lowest death rate is pescetarianism, with lacto-ovo-vegetarianism and semi-vegetarianism somewhere in the middle.
And again, that’s treating the confidence intervals as minimal, when in reality they heavily overlap with each other and with the nonvegetarian death rate.
The extra weird thing here is that in men veganism was most helpful against cardiac issues, whereas in women it appears to be actively harmful to cardiac health. Any benefit veganism has in women comes from the “other death” category, whereas in men the “other death” category is where it loses ground against pescetarianism.
The paper describes this as “Effects were generally stronger and more significant in men than women”, which is a weird way to say “women and men had very different results”.
Why the gender gap?
Could be any number of things. Maybe nonvegetarian women had healthier diets than men in the same category (they cite another paper that checked and said there were no “striking differences” between the sexes within a given category, but I’m not feeling very trusting right now). Maybe nutritional intake has a bigger impact on women due to menstruation and pregnancy. Maybe women were more likely to use veganism as cover for an eating disorder. Maybe they did the math wrong.
Fish seem pretty good though
If you’re going to conclude anything from these papers, it’s that fish are great. At least as good as veganism in men, and better in women. I’m more inclined to trust that result than I otherwise would be, because pescetarianism has less of a health halo around it, as witnessed by seafood eaters having roughly the same prevalence of bad habits like smoking and drinking, relative to lacto-ovo-vegetarians and semi-vegetarians. My ex-Adventist friend confirms that veganism is viewed more favorably than pescetarian or semi-vegetarian in the community, although not compared to vegetarianism, which surprised me.
So the benefits of pescetarianism are less likely to be downstream of being the choice of people who care a lot about health, or are highly consceintious. I briefly got motivated to eat more fish until I remembered I count as semi-vegetarian by their standards (and I’m female), so the gains are small even if you take the results at face value.
Nutrition is still really complicated, the study still has a bunch of flaws, I wouldn’t update too much on this even if it didn’t agree with my existing beliefs. But this clearly undercuts veganism as the healthiest choice for women, and doesn’t really support it for men either.
[note: this comment was edited after people’s replies.]
I think it’s useful to look at the confidence intervals, rather than only the point estimates.
The 95% confidence intervals of the adjusted hazard ratios for overall mortality, for men, were [0.56, 0.92] and [0.57, 0.93] for vegan and pescatarian diets, respectively, and for women the CIs are [0.72, 1.07] and [0.78, 1.20], respectively. For women, the confidence intervals for all diets are [0.78, 1.2], [0.83, 1.07], [0.72, 1.07] and [0.7, 1.22].
What these CIs indicate is that there was likely no difference between pescatarian and vegan diets for men, both of which are better than omnivorism, and likely no difference between any of the diets for women.
The CIs for women specifically look so similar that you could pretend that all of those CIs came from different studies examining the exact same diet, and write a meta-analysis with them, and readers of the meta-analysis would think, “oh, cool, there’s no heterogeneity among the studies!”
In fact, we can go ahead and run a meta-analysis of those aHRs (the ones for women), pretending they’re all for the same diet, and quantitatively check the heterogeneity we get. Doing so, with a random-effects meta-analysis, we find that the I2 is exactly 0%, as is the τ2. The p-value for heterogeneity is 0.92. Whereas this study should update us a little bit on pescatarian diets being better than vegan diets for women, these differences are almost certainly due to chance. No one would suspect that these are actually different diets if you had a meta-analysis with those numbers.
Since the total meta-analytic aHR is also very close to 1, it also looks like none of the diets are meaningfully associated with increased or decreased mortality for women, though there was a slight trend towards lower mortality compared to the nonvegetarian reference diet (p-value: 0.11).
Code:
Let me make sure I understand correctly: you view AHS-2 as supporting the belief that vegan diets can be healthy, for some people, but not anything larger than that?
I’d also like to point out that the sentences you describe as “slightly misleading” come immediately after I said “if you take the data at face value (which you shouldn’t)”. So as far as I can tell we’re in agreement here, and the area of contention is that I thought you were using AHS2 to endorse much larger claims than you are.
The first sentence is the title of the post, which has no hedging. The post also claims that“[i]f you’re going to conclude anything from these papers, it’s that fish are great,” which doesn’t seem to be the correct takeaway of this specific study given how wide and similar so many of the aHR 95% CIs are. You also claim “[o]utcomes for veganism are [...] worse than everything except for omnivorism in women” in another post without hedging.
In any case, taking the data at face value doesn’t imply “ignoring the confidence intervals.” I don’t see why it would imply that.
My original comment concluded that there was a “substantial probability” that vegan diets are healthier, and quoted someone (correctly) showing that vegans have lower mortality than omnivores in the AHS-2. I didn’t mean to claim it was the healthiest option for everyone; the comment is perfectly compatible with this null result for women. I also claimed that it wasn’t obvious that vegan diets can be expected to make you less healthy ex ante, modulo things like B12. I apologize if it wasn’t clear.
I think everything I said makes sense in a context where people (including the study authors) are using this paper to argue for veg*nism. I’m arguing that the study is both weaker than reported, and (weakly) pointing in a different direction than reported. I agree that if a fish farm lobbying group was using this paper to claim that fish were the cure for aging, that would be very misleading and I’d argue against them too.
What you originally said was “say it’s not at all obvious that a vegan diet has health tradeoffs ex-ante”. I think what you meant here was “it’s not clear a vegan diet is net negative.” A vegan diet leading to lower energy levels but longer lifespan is the definition of a trade-off.
It would be helpful if you clarified the population you are talking about. I’ve already said I think some people’s optimal diet is vegan, and for some other people vegan is the best out of the options they can realistically achieve. So unless you mean a substantial probability everyone’s optimal diet is vegan, and there is no such thing as a prohibitive health issue, we’re not disagreeing.
I also feel like saying “modulo things like B12” is burying the lede. A lot of my point is that vegan advocates are recruiting people without providing the necessary nutritional education, and are in some cases fighting that education even when it’s done in a vegan-compatible way.
This might be semantics, but when you said “Change my mind: Veganism entails trade-offs, and health is one of the axes” I (until now) interpreted the claim as vegans needing to trade off health (writ large) against other desirable properties (taste, cost, convenience, etc), not a tradeoff within different components of health.
I don’t have a sense of how common my reading was, however, and I don’t want to put words in Natalia’s mouth.
Life is trade-offs all the way down.
@LW team: LW still seems to struggle with importing stuff like emojis, trademark symbols, etc. from other blogs.
Here’s how that paragraph looks like for me on Elizabeth’s website:
I just edited this to fix the giant emoji in this instance, and made a code fix that should (hopefully) stop it from happening in the future.
Thanks <3! I’ve seen this occasionally in Zvi’s crossposts too, e.g. giant “tm” symbols. Will that fix those, too?
Thank you for the write-up! Just as a minor quibble, veganism has not been considered the “healthiest choice” ever, or at least not for a long time, if I were to make a guess about “consensus” in the field. While it has been clear for a while that a diet biased towards plants is healthy, the data for the addition of certain food groups (fatty fish, fermented and low-fat dairy, etc) is pretty strong as is the data for the health benefits of individual carninutrients (creatine or even taurine).
As you correctly point out, the issue of residual confounding is unsolvable. All we can take from these studies are hints and ideas. The recent failure of Vitamin D to live up to the hype, initially generated by observational studies, is a case in point.
I am particularly weary of studies of dietary patterns, whether they are vegetarian or Mediterranean or others, since I would expect to see the strongest biases here (because these patterns are associated with lifestyles, class, belief etc). Nevertheless, studies on surrogate endpoints like cholesterol and studies on single food groups do support the whole idea of reducing meat consumption.
The Adventist Study is interesting, but surely not definitive. It’s not even a RCT. To my knowledge, the only respectable RCT ever conducted analyzing a diet pattern is PREDIMED. It doesn’t test veganism specifically (the treatment group adopts a mediterranean diet), but it does increase the strength of the association between plant based diets and lower overall mortality risk. Overall, I think the level of evidence against meat is only suggestive, but since it’s so time-consuming and expensive to conduct these trials, I don’t expect much further light coming from investigations using metrics like mortality, heart attacks, strokes, etc. I think epigenetic clocks will in the future be a much better way to quickly analyze the effects of diet interventions, and I suspect plant-based diets will have an advantage over other diet patterns.