Plant based diet or plant-biased research?
On 7th November (2018) I submitted a rapid response to an article called “Effectiveness of plant-based diets in promoting well-being in the management of type 2 diabetes: a systematic review“, which was published in the journal BMJ Open Diabetes Research & Care.
On 13th November, I received an email rejecting my rapid response. The reason given was “We respect your opinion that the manuscript missed some important references and do (sic) not merit publication or a press release. For your information, the manuscript underwent extensive review by two international experts in the field who recommended approval. The Editorial Board accepted the revised manuscript following the Reviewer’s (sic? one or more?) recommendation.”
It is fact, not opinion, that this paper misrepresented the number of studies upon which the claims were based. It is fact, not opinion, that the primary finding from this paper – about plant-based diets, well-being and type 2 diabetes – ended up being based on one study. It is fact, not opinion, that that one study (Kahleova et al) concluded “Quality of life increased in both groups, but more in the experimental group.” It is fact, not opinion, that the authors reported the Bunner et al finding as: “Quality of life significantly improved within the intervention group” when the complete Bunner et al sentence read “Quality of life improved significantly within the intervention group but changes between group did not reach statistical significance.” It is an opinion that such disingenuity “hardly warrants publication, let alone a press release”, but surely not an unreasonable opinion following the facts? And the facts should have been published whether the journal likes them or not.
Here’s the rapid response below. If should be appearing alongside the article, not on my site. There is a further rapid response that needs to be submitted about the authors’ extraction of mean values for HbA1c without the standard deviations so that statistical significance could not be tested. I’ve done the calculations that they should have done and they had no right to use the word “significantly” 25 times in Table 2 alone…
The rapid response:
This is a response to the article “Effectiveness of plant-based diets in promoting well-being in the management of type 2 diabetes: a systematic review” (Ref 1).
The press release for the article claimed that the researchers found “11 relevant English language clinical trials” (Ref 2). The paper abstract referred to 11 articles. The “Study characteristics” section in the paper claimed that there were 11 studies. This section noted that two of the ‘studies’ had two articles written about each of them (Ref 3). There were thus nine studies found by the researchers and any suggestion to the contrary is disingenuous.
A prima facie review of these nine studies revealed further duplication, as two were about the GEICO corporate study. The more recent (2013), larger (43 people), study should have been included (Ref 39 from the paper) and the older (2009), smaller (19 people) study should have been excluded (Ref 40 from the paper, not Ref 51 as Table 2 reported).
The media headlines that emanated from the paper claimed “Vegan diet ‘can boost mood and ease symptoms for those with Type 2 diabetics’, scientists say” (Ref 4). This is consistent with the claims made in the press release and the paper. The implied 11 studies for this claim, already 8, became 1 when the paper was properly examined.
In a long paragraph on page 4 of the paper, it was reported in passing that “From the 11 articles included, 4 reported psychological well-being in their results.” The paper did not provide references at this point or clarify that these four articles covered three studies because of the Kahleova et al duplication (Refs 35 & 36 from the paper). Now we’re down to three studies (Refs 41, 44 and 35/36 from the paper).
The claims in the article and press release were made about plant-based diets, well-being and Type 2 Diabetes. Having discovered that only three studies reported well-being, another part of the Toumpanakis et al paper reported “All studies targeted populations with T2D except for three.” (Refs 39, 40, 44 in the paper). All except three; two of which were duplicates so that means six then.
One of the three studies, which was claimed to have informed the plant-based/well-being and type 2 diabetes finding, Wright et al (Ref 44 from the paper), was one of the three studies that didn’t target populations with T2D. The authors knew this because they reported elsewhere (page 6) “After excluding Wright et al (Ref 44) for reasons of not reporting HbA1c levels specifically for participants with T2D…” Similarly the quality of life survey was reported for 65 people, of whom only 9 had T2D. The well-being part of Wright et al should also have been excluded therefore.
This leaves two studies (Refs 35/36 and Ref 41 from the paper). Notwithstanding that Bunner et al (Ref 41) had a fundamental flaw in methodology in that it was one of the four studies that did not provide nutritional education in both the intervention and the comparator diet group (and thus it cannot be concluded that the diet made a difference and not the known ‘feeling special in a trial’ factor), Bunner et al did not claim that the intervention made a significant difference. Bunner et al reported: “Quality of life improved significantly within the intervention group but changes between group did not reach statistical significance.” Toumpanakis et al reported the first part of that sentence (in Table 2), but not the second part. That is also disingenuous.
We’re down to 1 study, of 24 weeks duration with 74 people (Refs 35/36 from the paper). This Kahleova et al paper reported “Quality of life increased in both groups, but more in the experimental group.” That hardly warrants publication, let alone a press release.
Dr Zoë Harcombe
References
Ref 1: https://drc.bmj.com/content/6/1/e000534
Ref 2: https://medicalxpress.com/news/2018-10-plant-based-vegan-diet-diabetes.html
Ref 3: Refs 35 & 36 from the paper were about the same Kahleova et al study and Refs 37 & 38 from the paper were about the same Barnard et al study.
Ref 4: https://www.dailymail.co.uk/health/article-6335559/Vegan-diet-boost-mood-ease-symptoms-Type-2-diabetics-scientists-say.html
Dear Zoë, many thanks for your masterful dissection of this recent vegan propaganda broadsheet masquerading as a scientific research paper. It follows a depressingly familiar trajectory. I’m starting to suspect that a vegan diet pursued long enough results in a reduced capability to understand and apply standard statistical analysis methods. I need but a few more such suspicions to embark upon a full-scale meta-analysis of them.
Bravo Dr. Harcombe!
My father measured his blood sugar 6 times a day for the last 15 years of his life for t2d from 1981 to 1996. Carbohydrate raised his blood sugar significantly. Protein only moderately raised his blood sugar. Dietary fat was neutral with only a slight rise. My father achieved normal blood sugars by eating approx 50% of calories from fat (mostly butter), 35% protein, 15% carbohydrate. His diabetes did not progress. His heart was autopsied when he died because he died from an accidental fall. He did not have heart disease.
I knew a number of people with a similar story. One had been Type 2 for 42 or 43 years and finally died aged 89 – NOT from diabetic complications. He put this down to the fact that he was put on a low carb diet, over 50 years ago when this was standard, and unaccountably never taken off it.
I say “knew” because they are now dying of old age – which current diabetics eating high carb low fat will simply never accomplish. When my generation dies there will be no-one left to remember a time when “everyone knew” that you cut out sugar and starch to lose weight and that diabetics ate low carb. Well of course there are still millions of us “just anecdotes” but while we are not studied “there is no evidence”.
Thank you for your rapid response.
I cannot get away from this – carbohydrate consumption raises blood glucose level (not lowers or unchanged). Therefore, as increasing BG for a T2DM is the problem, why consume lots of (or any) high glycemic carbs?
QED???
Hi Dr Harcombe,
My name is Paul Jenkins, I am a London-based endocrinologist and with I would like to respectfully break down your response.
Before I begin, I would like to mention one of my personal rules about research publications: never pay attention to or trust press releases and ALWAYS read the X paper. My experience says that many press releases are inaccurate to study’s characteristics.
Moving on. The abstract of this paper is pretty clear as it refers to ‘articles’ and not ‘individual CT studies’:
“The search strategy retrieved 1240 articles, of which 11 met the inclusion criteria”.
Moreover, when the reader goes to the study characteristics is even clearer that the original CTs are 9 and not 11 and therefore I cannot see any disingenuous suggestion as you mentioned Dr Harcombe.
Moving on to the GEICO studies.
You made a mistake: there is no duplication here as 2 separate studies were carried out in the same corporate site (Mishra et al., 2013; Ferdowsian et al., 2010, not 2009). As current SR’s aim was to investigate any CT that targeted plant-based interventions of course and by default they should include BOTH studies, otherwise they would be biased. Suggesting that “the older (2009), smaller (19 people) study should have been excluded” goes against any SR’s criteria and therefore is a wrong statement.
Next: “In a long paragraph on page 4 of the paper, it was reported in passing that “From the 11 articles included, 4 reported psychological well-being in their results.”
This is simply not true. The paper specifically says:
“From the 11 articles included, 3 reported psychological well-being outcomes in their results”.
You are right though that there are no references, which to be honest is not a big deal for me as you can read and find them later on text. I agree though that for typical reasons they should be there (references being 35, 41, 44 from the paper).
Continuing, you mention that the paper states that:
“All studies targeted populations with T2D except for three”, and then you add “All except three; two of which were duplicates so that means six then”.
This is inaccurate as well Dr Harcombe because (1) as I mentioned earlier there were no duplicate studies included and (2) you forgot to mention the sentences that come after the one you quoted:
“Of these three studies, two included individuals with body mass index (BMI) ≥25 kg/m2 and/or T2D, and one also included individuals with BMI ≥25 kg/m2 and at least one diagnosis of T2D or ischemic heart disease or hypertension or hypercholesterolemia. Therefore not all of the participants of these three studies had T2D, but in the results specific outcomes related to the individuals with T2D were described”.
It is pretty clear why these studies were included and how they were used since the SR’s aim was to target diabetics who participated in plant-based CTs and report their outcomes.
You then continue by saying that “Wright et al, was one of the three studies that didn’t target populations with T2D”, something that is inaccurate, as I mentioned above, and misleading to the reader.
I agree with you that the Wright et al. quality of life outcome was wrongfully included as the results were mixed and not specific for diabetics.
Regarding your comments about Bunner et al. now.
You mention that due to the absence of educational intervention to both groups that:
“it cannot be concluded that the diet made a difference”.
This is purely speculative with no other basis than your opinion Dr Harcombe.
Yes, I totally agree with you that education should have been part of this intervention for both groups but that does not mean that the intervention itself did not make a difference because of the absence of education in the control group. In addition, authors clearly state the further need for proper nutritional education in the future that could lead to healthier dietary patterns.
You also claim that the paper makes another disingenuous claim in regards to Bunner et al. by reporting that:
“Quality of life improved significantly within the intervention group” in Table 2, and not including the full sentence which was:
“Quality of life improved significantly within the intervention group but changes between group did not reach statistical significance”.
From another thorough reading of the Table 2 my guess is that in the table the authors chose to include mainly the results of each study that met significance as it is a summary table. There is no misleading or anything here by reporting that within the intervention group quality of life significantly improved, as it is accurate.
Last but not least, in the beginning you mention:
“They had no right to use the word “significantly” 25 times in Table 2 alone”.
The way you have structured this bit is as if the authors used the word ‘significant’ 25 times in order to describe HbA1c findings. This might look catchy, but is somehow misleading. The outcomes of Table 2 are clearly based on the findings of the included studies and they are a summary of the ones that the authors thought as most important (therefore they mostly used significant outcomes). I mentioned above what the outcomes of Table 2 include in my opinion. I might be wrong.
As I mentioned, I agree to a couple of points you have raised but I honestly cannot see the others as there are not very accurate.
Maybe it is time for us doctors to move from cherry-picked critique of each other’s’ research towards a mutual path in which we share similar goals like people’s welfare and healthier ways to live for all. For example, I might disagree with your suggested diet Dr Harcombe (for a number of reasons but it is not the time or the place to elaborate) but I recognise that there are some good aspects and points on it and my guess is that the same goes for you towards other type of diets. How more productive would it be for healthcare professionals to get together, share our knowledge, overcome our medical I-know-better-than-you complex and design, plan, implement and test interventions (long-term and short-term) about which combination of diet/nutrients/whole foods/etc works better, instead of engaging in exaggerated arguments and drama for the shake of our popularity/blog/sales/private practice? I do include myself on it as I am far from perfect when it comes to by field of knowledge. We sometimes are lost in our opinion superiority (sometimes justifiable, sometimes not really) and we lose the bigger picture.
With respect,
Dr Paul Jenkins
Hi Paul
Many thanks for your considered response – can you do my Monday note some days?!
The number of studies was reported in the press release (and therefore in the media coverage) as 11. I got a press release by email – this is an exact copy (https://medicalxpress.com/news/2018-10-plant-based-vegan-diet-diabetes.html) “the researchers trawled through the available evidence and found 11 relevant English language clinical trials.” That is disingenuous at best and fraud at worst. The “Study characteristics” section in the paper also claimed that there were 11 studies.
I have a copy of the original PDF in front of me and it says: “From the 11 articles included, 4 reported psychological well-being outcomes in their results.” (5th line up on page 4) I wondered why you could assert so strongly that I am wrong and I can see that the online version has been changed to read “From the 11 articles included, 3 reported psychological well-being outcomes in their results.” The PDF has also been changed – I suspect because of my rapid response, so at least it had some effect. I can send you my copy of the original if you like. We are both right therefore!
I didn’t forget to detail the BMI/heart disease stuff – the claims were about plant-based diets, T2D and well-being so I stuck to evidence for those 3.
You claim that this is inaccurate: “Wright et al, was one of the three studies that didn’t target populations with T2D.” How so when the paper says “All studies targeted populations with T2D except for three.39 40 44” (Wright is ref 44)?
Re. Bunner “it cannot be concluded that the diet made a difference”. That is fact, not opinion. It cannot be concluded that diet made a difference if anything other than diet was also different.
It is your opinion that leaving out “but changes between group did not reach statistical significance” is not misleading. I think it is, again, disingenuous at best and fraud at worst.
Using “significantly” 25 times in Table 2 alone is more than misleading. Significantly in this context should mean statistically significantly (not the normal use of the English word meaning “in a sufficiently great or important way”).
You didn’t comment on the fact that only 1 study (not ostensibly 11) potentially established a significant result for plant-based diets, T2D and well-being and that the conclusion from that study was hardly strong.
The issue with Table 2 (which requires another rapid response, but who has the time?) is that the means were extracted without SDs, although the SDs are there in the original papers. I extracted the SDs so that significance could be tested. The authors chose not to do this. They also chose not to do a meta-analysis when this should be the logical outcome of an SR. They justified this by saying “we considered that because the overall pooled N in the study is small, the CIs would be quite wide due to imprecision, which might cause our pooled estimate to cross the null hypothesis, which would make it difficult to draw a conclusion in either direction.”
This means – so few people have been studied, that, if we pooled the few, small studies together, it would not achieve statistical significance and we would have to report that there were no findings. If so few people have been studied that even pooling all the small studies together would produce no findings, there are no findings! How does that get through peer review?
Best wishes – Zoe
Hi Zoe,
I’d like to say that this is not a never-ending argument or fight about a paper. It’s just a simple discussion to share our knowledge and interpretation of a research article and exchange opinions.
As mentioned on my earlier response, I don’t pay any attention to press releases all these years and neither should anybody, other than the journalists/columnists. I can assure you that many times authors have no idea what’s in the press release. Again, word ‘fraud’ is very catchy, far-fetched and click-bait and probably works for the average reader, but I do not see it I’m afraid.
Regarding Wright et al, as a general rule I do not agree when a sentence is presented out of context by its own, as it’s tricky, the reader misses the meaning and is mistakenly mislead. Setting that aside, in my opinion this paper should not be included in the SR as it does not really serves a purpose since there were no specific quality of life outcomes for diabetics.
About Bunner et al. How so “this is a fact”? You argue that you cannot extract outcomes out of 2 diets if only one group receives nutritional education and the other doesn’t? I respectfully disagree with this statement which is far from ‘fact’. You most certainly can extract outcomes even in these circumstances.
I had another read of Table 2 and the included studies. Apparently when authors use the word ‘significant’ is because these outcomes were statistically significant in the studies. No ‘fraud’ here I’m afraid.
The studies that report both psychological and medical outcomes are 2 and not 1 (Bunner et al., 2015; Kahleova et al., 2013). You believe that Bunner et al should not be included, I believe that it should. We agree to disagree on this one.
My general interpretation (and it is purely speculative) is that the authors by ‘well-being’ meant the overall psychological AND physical outcomes of the SR, combined. Not solely the psychological ones. Plus that there were interesting outcomes like adherence, acceptability and diet behaviour. That is why I think they used ‘well-being’ as an umbrella term. They should have been clearer on this though.
You are right when saying that is my opinion that Bunner et al outcome sentence is not misleading and is also my opinion that the use of words like ‘fraud’ is most unfortunate and misplaced as it makes you look personally invested here because they rejected your response. I understand up to a point as a human reaction to be annoyed for your reasons. I might be wrong but I have shown here, and in my initial comment, that your response was arguable. You pointed out a couple of things that were right, but the rest were either wrong or opinions. And this is perfectly fine for discussion purposes but not for journal publication. You seem very passionate about this and I admire the effort.
Have you tried to contact the authors and discuss your feedback with them? That would be highly constructive I think and should have been done in the first place, if not already.
As for the peer review, based on past experience as a peer reviewer, I can tell you that this paper is not as bad as the catchy language paints it. If it was on me, the main thing I would point out would be to take out the article of Wright et al and probably a couple of other minor issues.
Again, it is our speculative opinions Zoe. It makes perfect sense to me that if you want to investigate the X hypothesis and the overall N is small (for whatever the reason, it does not really matter) to go into a narrative synthesised SR rather than conduct a meta-analysis and miss the outcomes. A more qualitative approach in SRs when needed is equally important (especially in underresearched fields) and of course their findings have scientific value. Your claim about “not findings” is based on your opinion. My professional approach is that every little result matters to our scientific knowledge and to move forward as professionals. Let’s keep an open mind rather than getting radical in subjects that are highly debatable and do not have a clear answer. I’m including myself on this as oftentimes I lose the big picture as well because my opinion is too strong for my shoulders. We need to re-introduce discussion in our professional life rather than opinionated arguments.
Best wishes,
Paul
Hi again Paul
Just a quick reply on a couple of things
1) Having been involved in a few papers that have been press released, this is done in close collaboration with at least the lead author (who should involve the others) and again – at least the lead author, if not all, sign off on the press release. The researchers should not have allowed the interviewer to even think that there were 11 studies, let alone to have this as the headline.
2) You’ve demonstrated the issue with Table 2 beautifully. You say “Apparently when authors use the word ‘significant’ is because these outcomes were statistically significant in the studies. No ‘fraud’ here I’m afraid.” That’s the point! You rightly assume that the word significant implies statistical significance, which it should. Only it doesn’t in this paper. That’s disingenuous at best and fraud at worst. To establish significance you need to extract the SDs alongside the means. Those data were available, but they didn’t do it. And they showed that they knew the impact of NOT doing this with their statement about how wide the CIs would be. I’ve done what they should have done and those numbers are not significant – do the same if you like.
Every week for the past several years I have written a newsletter dissecting an article from the previous week. This Monday coming will be my 400th such note. When I say that this is a shocker of a paper and it should never have got through peer review, it’s because it stands out for how bad it was. I do often contact authors to ask questions, or to point out an error that they can easily correct, or to try to make a helpful point. The correct route for this one was the rapid response. I should have done it as a shorter 1-2 – point 1 on the number of studies actually involved in the ‘finding’ and point 2 on the claimed statistical significance – that wasn’t. The authors are lucky that I’m onto the third note after this paper already and I don’t have time to pursue it. Meanwhile, the vast majority of people who are far more likely to read the Daily Mail than an academic paper think that plant-based diets will improve well-being in diabetics and we simply don’t have the evidence to say that.
Best wishes – Zoe
The worst of it is, the reported results were pathetic, only slightly less pathetic than “standard care”.
But the vegans are determined to push their agenda whatever.