Red meat & mortality & the usual bad science – Part 2
I blogged on March 12 2012 about this press release about an article in the Articles of Internal Medicine published on the same day. The article now appears to be unavailable on free view.
On March 20, I emailed the author listed for personal correspondence ([email protected]) with the following query:
Dear Dr Hu
Please can you help me understand how the multivariate analysis was done in your study (http://archinte.ama-assn.org/cgi/content/full/archinternmed.2011.2287)?
The raw data in table 1 shows the death rate falling – with Q2 and Q3 lower than Q1 in the HPFS and Q2, Q3 and Q4 lower than Q1 in the NHS. This is while exercise is falling and BMI, diabetes, smoking, calorie & alcohol intake were all increasing alongside red and processed meat consumption. I would have thought that once your multivariate analysis had allowed for these (and many other) variables to try to isolate meat consumption, the multivariate index would be substantially lower than my death rate line, which includes these risk factors. As a comparison, if I index my death rate for the HPFS is would be 1.00; 93.43; 97.70; 103.97 and 124.26 (doing the same 1.00 base line thing) vs. the multivariate index of 1.00; 1.12; 1.21; 1.25 and 1.37.
Kind regards – Zoe
I received a prompt reply later that day:
I emailed back on 22 March:
Many thanks for the prompt reply. Aside from thinking that “Age is the main risk factor in mortality” would have been a better headline, I’m still puzzled. Please can you help with the following:
1) Table 1 says it is age standardised – what does this mean? (*)
2) The ages in Table 1 are 53.8, 52.6, 52.5, 52.5 and 52.2 for Q1 to Q5 respectively for the HPFS and 47.3. 46, 45.8, 45.3 and 46 for the NHS. Not withstanding query 1, how could such small differences have had such an impact on the study a) per se and b) to the extent that “age was a stronger confounding factor than other lifestyle factors” when exercise was down by over a third, BMI up from normal to overweight, smoking almost three times higher in Q5 than Q1, incidence of diabetes almost doubled, alcohol intake up 60% and calorie intake over 40% higher (all figs for the HPFS – the NHS has different numbers, same trends) and that’s even before we look at meat consumption.
3) The death rates that I calculated were from Table 2 – notwithstanding query 1. Hence as total red meat, ‘unprocessed’ red meat (hamburgers?!) and processed red meat all rose from Q1 through to Q5, so the death rates in the HPFS are lower in Q2 and Q3 than Q1 and the death rates in the NHS are lower in Q2, Q3 and Q4 than Q1. Even if Q1 is supposed to be ignored for some strange reason, the Q3 death rate is still lower than Q2 in all meat categories in the NHS and for the ‘unprocessed’ meat category in the HPFS i.e. deaths fall as these groups of meat consumption rise.
4) If I ‘index’ the death rates, as has been done in the multivariate study, the death rate ‘index’ for the HPFS and the NHS are as follows – with the total meat multivariate index alongside. I would expect all the above mentioned factors (reduced activity, increased BMI, increased smoking, increased diabetes, increased alcohol and increased calorie intake from Q1 to Q5) to be impacting death rates so that they also rise from Q1 to Q5. a) they don’t and b) how can the multivariate line have allowed for all of these and yet rise consistently as it does?
|Death rate as index||1.00||0.93||0.98||1.04||1.24|
|Death rate as index||1.00||0.93||0.90||0.97||1.27|
Kind regards – Zoe
(*) the reason for asking this question was not because I don’t know what “age standardised” means but to flush out why the reply from Frank Hu was effectively that the data is misleading because of age and yet the table claims to have been age standardised
I received no reply.
On 3 April I resent the unanswered email:
Please are you able to reply to this?
Kind regards – Zoe
I received no reply.
I’ll update this post with any response that Frank Hu makes.
10 thoughts on “Red meat & mortality & the usual bad science – Part 2”
All of this shows a few important things:
1. As usual, data dredge observational studies have ONLY the ability to point to areas that might be of interest to researchers as indicators of a topic that might be subject to further research. They do not and can not ever ever ever give information about causation.
2. Such small increases in something as vague (as far as causal factors is concerned) as ‘death rates’ despite being able to be shown to be ‘statistically significant’ are not practically significant in any real way.
3. Hate to say it again, even if the association were HUGE (it is not, it is tiny tiny tiny at best) Association is not Causation.
4. This study, and the hundreds of others like it, will never be scientifically sound, because of the type of study it is and the data it is based on –> it is a data dredge of self-reported diet information collected at long intervals, a study not designed to test any particular hypothesis. Given that, even if this study turns out to be statistically sound (doubtful, but possible), the ‘absolute risk’ found of death of a male increasing his red meat consumption from 3 serving a week to 2 servings a day moves from .0123 to .0130.
5. Any study that depends on obscure statistical methods to find any result at all can safely be ignored by laypersons.
Good one, Zoe. “Crude” is definitely the name of the game. Oddly, nobody questions quintiles? Why do quintiles? Because if you plotted every point (or a random sample), you would need a computer. Computers are expensive. What year is this? If you saw individual points, you would know that there is no predictive value in this stuff.
I think Zoe herself explained it best, Crowhurst, by using an example of someone singing in the bathtub. We can see clearly that singing does not cause bathing, and bathing does not cause singing!
This thread may be cold by now but if not could DT or someone else versed in statistics explain how confounding factors are “subtracted” out of a data set to reveal the relationship between the variable of interest? Intuitively, it would seem to be almost impossible to do accurately, especially if the variable of interest (i.e., meat consumption) had a relatively small association with the outcome (e.g., death) compared with some of the confounding factors (e.g., smoking, age, BMI) which have a large association with the outcome. Does it not require knowing the mathematical relationships between each confounding factor and the outcome to an unbelievable degree of precision and accuracy or else risk having these small errors appear to be a signal from the variable of interest? For example, doesn’t the analyst need to develop and input a mathematical function that relates every confounding factor, say, alcohol consumption, with mortality so that this factor may be subtracted out appropriately. It seems these relationship are known in broad terms, but not to the extreme degree of precision needed to tease out relationships between much weaker factors and outcomes. And if you get each confounding variable just a tiny bit wrong, all the little imprecisions accumulate at the variable of interest making it look like a real association when all it really is is noise.
When writing my previous comment I didn’t notice the asterisk in Zoe’s email. However, this doesn’t change the fact that looking at raw (not age-adjusted) death rates is meaningless. The paper does provide age-adjusted death rates (or more precisely, it provides age-adjusted HR), and only these number are interesting when comparing with the multivariate model numbers.
I gave answers to your questions above on the comments section of your previous blog post. As I wrote previously, your “death rate” numbers are meaningless.
Also, if you don’t know what “age standardized” means, why do you think you are qualified to critique the statistical analysis of the study?
The reason Dr. Hu did not reply was not because his
research were shown to be BS. All the critiques (Zoe & Denise) have only shown that they lack knowledge of statistics.
The link you gave does not backtrack the original findings.
It is just an explanation of the paper to the laymen.
The soda study shown a WEAK association between soda consumption and CHD. Thus, soda consumption cannot explain the difference in mortality between the high & low meat eaters.
A subset of the Harvard folks who published the meat study (including Dr. Hu) also just published a study using the very same data-set that purports to show a large association between soda consumption and cardiac disease and death. If there is such a large association with soda consumption, then why on earth wouldn’t the meat study have tried to correct for soda consumption. Seems it’s apparently such a large risk factor that could easily swamp out whatever signal they’re seeing from meat consumption. Is it a glaring omission, or am I missing something?
The soda study is here:
Harvard Medical School’s HEALTHbeat newsletter (link here:)
printed a follow up to that study today. It does seems to backtrack the original findings.
Hope this helps.
Keep up the GREAT blog!
MS – Thank you SO much for this. I also notice that the article is no longer freely available – normally we would have to say “the damage has been done” but there are no so many defenders of grass grazed meat that I like to think collectively we put up a fine counter attack. I just hope the next people planning to look for a relationship between meat and blue socks, or whatever nonsense is next, now think twice!
Wow Zoe, for a moment there I thought you must be channeling Denise Minger, an outstanding analysis.
Unfortunately Frank was not so impressed.
Might have been the inconvenient fact that all his fiddling with the data to provide the result he set out to achieve has been shown to be utter BS, so what does he do when called out?
He goes and hides of course!
Ah well, another day, another research grant. I’m just amazed that Walter Willett put his name to this rubbish. Until now I had thought he was a lone voice of reason, sad.