Animal protein as bad as smoking? Headlines based on 6 deaths!

A study was published on 4th March 2014 with a press release claiming “Meat and cheese may be as bad for you as smoking“.

I’ve gone through the study in detail here.

The headlines

Just to remind ourselves of the headline numbers from the press release and where the numbers come from in the main paper:

1) The “four times more likely to die of cancer” comes from Table 1, Model 1, for 50-65 year olds. The Hazard Ratio (HR) is given as 4.33 for the high protein intake group when the low protein intake group is referenced as 1.00.

2) The “protein-lovers were 74 percent more likely to die of any cause” also comes from Table 1, Model 1, for 50-65 year olds. The Hazard Ratio is given as 1.74 for the high protein intake group when the low protein intake group is referenced as 1.00.

The numbers                                                       

I mentioned in the narrative that I had emailed Dr Longo asking for the numbers behind the hazard ratios. I have not heard back from Dr Longo – not even an auto response. A journalist I know has also failed to get any numbers from the research team. Their view was “we don’t send raw and incomplete data to journalist (sic) since it is usually misinterpreted.”

No – raw data gives the facts. Manipulated data can be deliberately presented so that it is misinterpreted – or rather, interpreted as the ‘researchers’ want you to interpret it. This kind of misinterpretation for example: “eating a diet rich in animal proteins during middle age makes you four times more likely to die of cancer than someone with a low-protein diet — a mortality risk factor comparable to smoking.”

However – well done to “Sam” who posted a comment next to the article.

Sam asked: Please disclose the raw death rates for cancer in the 3 groups following the stratification into -65 and +65 (that division is the central pillar of your human study, hence it is bad science and poor reviewing not to have asked for them). Without those, your modelling rests on nothing and is thus open to severe criticism. Your table 1 in suppl material shows the raw death rates (9-10%) and protein seems to have no effect whatsoever on them (as has been pointed out by others).”

Morgan Levine replied: “The frequencies by protein group are as follows: 
 
Age 50-65 All-cause Mortality 
Low: 18.07% 
Mod: 20.28% 
High: 26.15% 
 
Age 50-65 Cancer Mortality 
Low: 2.58% 
Mod: 7.89% 
High 9.89% 
 
Age 66+ All-cause Mortality 
Low: 70.97% 
Mod: 63.73% 
High: 64.04% 
 
Age 66+ Cancer Mortality 
Low: 18.03% 
Mod: 12.94% 
High 7.96%”

So now we’re in business!

Page 10 of the supplemental PDF tells us:

Ages 50-65: Low Protein (N=219), Moderate Protein (N=2,277), High Protein (N=543)

Ages 66+: Low Protein (N=218), Moderate Protein (N=2,521), High Protein (N=603)

So we know how many people were in each of the two age groups and three protein intake groups. This gives us row 1 in the table below…

50-65

66+

Row

Low

Mod

High

Low

Mod

High

1 Number of people (from supplemental)

219

2,277

543

218

2,521

603

2 All-cause mortality (from Levine)

18.07%

20.28%

26.15%

70.97%

63.73%

64.04%

3 Cancer Mortality (from Levine)

2.58%

7.89%

9.89%

18.03%

12.94%

7.96%

4 All-cause deaths (Row 1 x Row 2)

40

462

142

155

1607

386

5 Cancer deaths (Row 1 x Row 3)

6

180

54

39

326

48

6 All-cause RR (uses Row 2)

1.00

1.12

1.45

1.00

0.89

0.90

7 All-cause HR from Table 1

1.00

1.34

1.74

1.00

0.79

0.72

8 Cancer RR (uses Row 3)

1.00

3.06

3.83

1.00

0.72

0.44

9 Cancer HR from Table 1

1.00

3.06

4.33

1.00

0.67

0.40

10 Average person yrs follow-up (from supplemental)

11.86

13.27

12.62

11.86

13.27

12.62

11 Cancer death rates per year (Row 3/Row 10)

0.22%

0.59%

0.78%

1.52%

0.98%

0.63%

12 Person years (Row 1 x Row 10)

2,597

30,216

6,853

2,585

33,454

7,610

Rows 2 and 3 have been provided by Morgan Levine, in the reply to Sam’s query.

Rows 4 and 5 calculate actual numbers of deaths using the number of people and death rates. As my “p.s.” in the original post hypothesised, there are substantially more deaths in the over 65s (3.34 times as many).

All-cause mortality

Row 6 takes low protein intake for all-cause mortality as the reference point of 1.00 and then works out the ratio of moderate and high protein intake relative to 1.00. Row 7 repeats the Hazard Ratios taken from Table 1, Model 1, of the main paper, as a reminder of the source of the headlines.

We are not comparing like with like, as Levine has given us top level death rates for each protein intake group and Model 1 has adjusted for loads of things (age, race, sex, education, waist circumference, smoking, diabetes, cancer, previous heart attack, diet changed in past year, tried to lose weight in past year and the kitchen sink – I made up that last one).

It is interesting that, for total mortality, the unadjusted data would only elicit a headline of “45% more likely”, not “74% more likely” – notwithstanding that this is still association, not causation and still relative not absolute risk and still only featuring the 50-65 year old group. The headline from Table 1, Model 1 for over 65s could have been “low protein consumers are 39% more likely to die” (using high protein as the reference).

Cancer mortality

Row 8 takes low protein intake as the reference point of 1.00 and then works out the ratio of moderate and high protein intake relative to 1.00. Row 9 repeats the Hazard Ratios taken from Table 1, Model 1, of the main paper. Here our relative risk numbers are much closer to the Model 1 HRs – there is an exact match (3.06) for the moderate protein intake group for 50-65 year olds.

Here we find the real headline. What the researchers didn’t want us to find out. The “four times more likely to die” global headline grabber was based on a reference group of six deaths. Yes six deaths. And not just six deaths –  but six deaths over an 18 year study. And the ‘researchers’ tried to claim that animal protein is as bad as smoking based on this?

I warned in the original post about the dangers of basing relative risks on small group sizes. Don’t forget that the researchers didn’t divide the 6,381 people into three even groups: they put 75% of participants into a moderate intake group (which they created) and just 6-7% into a low intake group (which they created). Had just 4 more cancer deaths occurred in the low protein intake group and 4 fewer in the high protein intake group, the relative risk would have halved.

Absolute risk per year

Row 10: The supplemental PDF has a useful table (S1), which gives us person years by protein intake group (not by age group). We can use this to work out an average number for person years of follow-up by protein intake group (total person years divided by the number of people in that group).

Row 11 is row 3 divided by row 10 – this gives the death rate per year of study.

Row 12 calculates back the person years (Row 1 x Row 10) as a sense check. The person years in this row add up to 83,315. Total person years in table S1 in the supplementary PDF is given as 83,308. We’re almost bang on – not bad – given how little help we’re getting.

So cancer deaths rates per year are as follows:

50-65

66+

Row

Low

Mod

High

Low

Mod

High

5 Cancer deaths

6

180

54

39

326

48

12 Person years

2,597

30,216

6,853

2,585

33,454

7,610

NEW Cancer deaths per 1,000 person years

2.18

5.95

7.84

15.2

9.75

6.31

 

We now take cancer deaths (row 5) and the person years (row 12) and we need a way of comparing like with like so we make all the person years “1,000” and work back to see how many cancer deaths there would be per 1,000 person years for each protein group and age group.

In the 50-65 year old group there were 2.18 deaths per 1,000 person years in the low protein group, 5.95 deaths per 1,000 person years in the moderate protein group and 7.84 deaths per 1,000 person years in the high protein group. The over 65s had 15.2 deaths per 1,000 person years in the low protein group, 9.75 deaths per 1,000 person years in the moderate protein group and 6.31 deaths per 1,000 person years in the high protein group.

Please don’t forget association not causation and no plausible mechanism blah blah. But do you think that, using a base of six deaths, 7.84 deaths per 1,000 person years vs 2.18 deaths per 1,000 person years would have held the front page?!

p.s. Only Longo declares his conflict of interest in L-Nutra. Many thanks to the people who commented on my original post to point out that three other authors are also part of the L-Nutra team: Priya Balasubramanian; Sebastian Brandhorst and Luigi Fontana.

p.s.2 I was contacted by an actuary, Dermot, who had put the deaths and person years from above into a mortality spreadsheet that he has developed for his work. From this, Dermot calculated the confidence intervals for the 2 age groups and 3 protein intake groups – as shown in the table below.

Below the table is a chart, which Dermot has produced from the table. The pale blue area on this chart shows how big or small the actual mortality rate could be, with 95% probability (i.e. there is a 5% chance it could be even higher or lower). As Dermot concluded: “It shows that the shaded area for the 3 rates for age 50-65 overlaps considerably, almost allowing a straight line to be drawn at about 0.5. This greatly reduces the statistical reliability of the raw rates.”

Age group Person years Deaths Observed probability of death (p) Confidence interval
Low : p1 High : p2
50-65L 2597 6 0.23% 0.10% 0.51%
50-65M 30216 180 0.60% 0.51% 0.69%
50-65H 6853 54 0.79% 0.60% 1.03%
66L 2585 39 1.51% 1.10% 2.07%
66M 33454 326 0.97% 0.87% 1.09%
66H 7610 48 0.63% 0.47% 0.84%

 

 

 

Zoë Harcombe

Posted in Media comments, Research
Tags: , , , , , , , ,
12 comments on “Animal protein as bad as smoking? Headlines based on 6 deaths!
  1. avatar Frank weir says:

    Devil is always in the details. As a layperson, i simply have to ignore medical research and the resulting media coverage. Statistical interpretation, study design, how study subjects are selected ( yet another way researchers can help manipulate results I understand), lack of neutrality by researchers, media misinformation with further conflicts of interest. The fact that Longo is the FOUNDER of L-Nutra pretty much makes me dismiss anything to do with this “research”. I would call it marketing research and advertising for Longo’s side business.

  2. avatar Mie says:

    Why should it get any news coverage? E.g. the separation of n-3 and n-6 fatty acids and the exclusion of “safa –> pufa/mufa” exchange makes this meta-analysis precisely the sort of nutritional science what it aims to criticize: reductionist and redundant.

  3. avatar Robin Willcourt says:

    ” —a large and exhaustive new analysis by a team of international scientists found no evidence that eating saturated fat increased heart attacks and other cardiac events.

    The new findings are part of a growing body of research that has challenged the accepted wisdom that saturated fat is inherently bad for you and will continue the debate about what foods are best to eat.

    But the new research, published on Monday in the journal Annals of Internal Medicine, did not find that people who ate higher levels of saturated fat had more heart disease than those who ate less. Nor did it find less disease in those eating higher amounts of unsaturated fat, including monounsaturated fat like olive oil or polyunsaturated fat like corn oil.

    http://well.blogs.nytimes.com/2014/03/17/study-questions-fat-and-heart-disease-link/?_php=true&_type=blogs&_php=true&_type=blogs&src=rechp&_r=1

    Finally! Although another non-study study… it at least shows that the chink in the armor is widening. This, only two weeks after the headline of this blog was announced all over the world just the opposite. We suffered through the pundits in Australia reveling on the TV “news” that we had irrefutable proof that meat and SATURATED fat were dangerous. Eat your grains, fruit and fiber and get in good amounts of dairy. You will be soooooo healthy.

    This new info will likely not get anywhere near the same coverage, of course. The Don Poldermans’ story got NO coverage at all. It is hard to believe that there is a 6 deaths vs 800,000 death news deficit. But, we all know that money talks.

  4. avatar Jessica says:

    http://www.diabetes.co.uk/forum/threads/diabetes-uk-national-conference-to-have-low-carb-debate.53811

    Just thought you might be interested in this – the link goes to the Diabetes forum, so you may want to look at the National Conference instead, but the forum has the Powerpoint presentations (in which the anti low carb person seems not to be informed that veggies contain carbs)…

    Also, interesting quote of the day :- “Alongside taking any necessary medication, the best way to stay heart healthy is to stop smoking, stay active, and ensure our whole diet is healthy – and this means considering not only the fats in our diet but also our intake of salt, sugar and fruit and vegetables.” : Prof Jeremy Pearson. So we should consider our intake of fruit and vegetables? Really? Since when? I thought that five a day was considered not good enough for some countries? And perhaps my ‘necessary medication’ wouldn’t be so necessary without the brainwashing of the medical profession….

  5. avatar Joseph says:

    “if the conclusions from this study were valid…what better alternative has been consider

  6. avatar Mie says:

    LeenaS

    “from Finland, the land of epidemiological mongering, I presume?”

    You presume wrong (not the Finland bit but the mongering nonsense) but I’ll forgive the mistake this time. :-)

    “if the conclusions from this study were valid, they would not have needed to limit to the very arbitrary cuts of the experimental group, but they could (and should) have used the normal tertiles, quintiles or even dodeciles and show how dose dependent results they. Done like this it is highly probable that someone just wanted to have a certain answer.”

    You’re not exactly wrong, I’ll give you that. However, notice that I haven’t disagreed with anyone concerning the limitations of the study. I’ve only commented on what I consider to be inaccurate claims. Concerning the link between animal protein and cancer, the study has several limitations (not a surprise, given the fact that it was a single epidemiological study), of course. Unfortunately, the criticism has in too many cases been directed towards irrelevancies.

  7. avatar George Henderson says:

    LeenaS makes a good point about dose-dependence. We know nothing about doses, i.e. the amount of protein consumed on that fateful day 18 years ago. We only know %E. This is not a dose – NOT any amount of protein. So if we wanted to live by this paper, assuming we all went insane, we would still not know how much protein to eat from it.

  8. avatar Michael B says:

    Interesting study today in Science Daily, somewhat relevant to this case study in how to lie with statistics.

    Diets high in animal protein may help prevent functional decline in elderly individuals

    “A diet high in protein, particularly animal protein, may help elderly individuals function at higher levels physically, psychologically, and socially, according to a study. The research suggests that as people age, their ability to absorb or process protein may decline. To compensate for this loss, protein requirements may increase with age.”

    http://www.sciencedaily.com/releases/2014/03/140311163101.htm?utm_source=feedburner&utm_medium=email&utm_campaign=Feed:+sciencedaily+(Latest+Science+News+–+ScienceDaily)

  9. avatar Kim says:

    Ivor, don’t hold your breath for a retraction or anything close. This appears to me to be much more of a marketing exercise for Longo and his plant protein company. And he certainly got a ton of free publicity in the mainstream media! I’m sure that’s all he was after. As for those of us who are interested in learning more about what’s really revealed in this so-called study, I’m afraid we’re a very small minority. Let’s face it, most people read the headlines, a few read the article, and that’s the end of it. Sad that this sort of thing goes on all the time.

    Zoë, please keep up the great work you do!!

    Kim

  10. avatar LeenaS says:

    Dear Mie (from Finland, the land of epidemiological mongering, I presume?)

    if the conclusions from this study were valid, they would not have needed to limit to the very arbitrary cuts of the experimental group, but they could (and should) have used the normal tertiles, quintiles or even dodeciles and show how dose dependent results they. Done like this it is highly probable that someone just wanted to have a certain answer.

    Then again, in ten more years or so we may know how the dose dependency goes, since someone is bound to make that study, too. The sooner the better for me :)

    Cheers,
    LeenaS

  11. avatar ivor cummins says:

    Bravo Zoe!

    I was quite happy to show that there was no relevance to protein as a causal variable, but glossed over the fact that there was no meaningful difference in the first place!

    For the readers it might be useful to mention that there are two levels of difference – “statistical difference” and “engineering difference”; the former can be shown to statistically exist, but you need the latter to say that it actually has a meaningful, real life effect that is worth considering.

    I can sense a retraction coming from these guys ………..or does that ever happen, even with stuff as disgracefully risible as this? I’m beginning to think a real life engineer should be placed on every research team – a kind of “Internal Affairs” for bunches of biased academics, if you will……

    Great work and best regards
    Ivor

  12. avatar Mie says:

    Zoë,

    Epidemiological studies are vulnerable to confounders and therefore raw data has to be adjusted. It’s the same in virtually EVERYTHING that involves observing and measuring somethings. (Of course raw data can and should be published too. The more transparency, the better).

    As for this study: given the number of participants, it’s hardly surprising that absolute differences are small, isn’t it? Talk about stating the obvious

Leave a Reply

Your email address will not be published. Required fields are marked *

*


5 − 3 =

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>