Research Integrity Is A Clown Car Which Continually Spills Forth A Truly Surprising Quantity Of Sad, Honking, Incompetent Clowns

The anatomy of an utter failure of academic governance

Preface

I’m going to allow myself a luxury here, and one that people may not agree with, or might be uncomfortable with.

I’m going to tell you what I really think.

I am very very tired of critical scientists being polite about the yards of pigshit they have to wade through to get anything done. I feel like at the present point in time we are, in some part, dealing with the knock-on consequences of collectively saying it’ll be fine and that’s not my problem around gigantic structural issues in science.

There is a point where polite professionalism simply isn’t reaching people who might make a system that’s less godawful. The people who run these systems simply do not appreciate how bad they are at their jobs.

Or, in fairness, perhaps it would be better to say as they are doing their best under difficult circumstances, they find it hard to appreciate what a bad job they are doing. I do not assign full personal responsibility to them. They might well be trying hard and colossally disempowered. I am one of those people who refuses, utterly and completely, to yell at customer service employees. I don’t wish to start now.

But it still sucks, of course.

And there’s one point I want to make louder and more emphatically than I have ever made it before, at the risk of being boring.

But it is a point that bears repeating.

So I continue to repeat it — and will again at the end:

I see no point in trying to access formal research integrity systems any more.

They don’t work.

I am making other plans.

Introduction

Academic journals are good and bad.

The bad ones leave clues — badly maintained webpages, editors who won’t reply to emails at all, a long string of PubPeer.com entries, stories around the traps from people who have had the misfortune of dealing with them.

But, occasionally, you encounter a journal which is stealth-bad.

They are generally run by a real publisher, by a real team of staff, and maybe they have a nice sciencedirect.com webpage and PubMed links…

but they also have the academic standards of a wood louse, and are utterly unprepared to handle either the contemporaneous problems in research integrity, which is mildly forgivable as they are legion, or, the basic mechanics of their business, which is a lot less fucking forgivable.

Stealth-bad journals are a facade, like the set of an old Western. Out in Main Street, we see a well maintained store front proclaiming LIKKER FOR SALE, with a functioning boot scraper and saloon doors that swing on oiled hinges. But, just behind the entrance, there is nothing but dirt, cow shit, and hastily nailed-up beams that fail OH&S standards.

I have seen my fair share of academic malpractice, and the anodyne low-rent responses to that from editors, EICs, research integrity officials, universities, and publishers. But what I’m about to outline below is new, at least, it is new to me. I did not think a real journal, run by actual adults, could fail this hard.

I don’t know what black hole of governance this particular incident fell into, and I would hope — fervently — that this is a one-off. Journals are stochastic. Get the wrong person on the wrong day, and you’ll conclude: ‘well, this is a bad team, doing a bad job, badly’. Other times, your experience may be the exact opposite.

An example comes easily to mind: I had a great time dealing with the team at Nature: Scientific Reports. This is a journal people continually deride for gradually diluting the (alleged) fanciness of the Nature brand, a cash cow journal run by dummies. It’s a punchline for a lot of people.

But when I had to chase a bad paper down in it with DRG, we had a great editor and a slow albeit entirely responsible time. They did a good job. It process took a while, but it always does. Really, I’d give them a B+.

A lot of people presumably wouldn’t.

This brings us to BMJ: Nutrition, Prevention & Health, where I am having an F time.

What I’m about to described should not be possible at all, not under any reasonable circumstances. A quote from Mencken creeps into my mind at times like this:

“It is so bad that a sort of grandeur creeps into it.”

Let me work through the complete anatomy of what happened here, so you too can appreciate the grandeur, the single biggest parade of incompetence that I have ever seen in one place.

Maybe it’s the journal. Maybe it’s the team. Maybe it’s the system.

Regardless, it is the 1st of May, so welcome to exactly a year of my life.

May, 2024.

I notice a silly paper in the news (or, maybe

Health Nerd

sent it to me, I can’t precisely remember… it’s usually him that starts all the trouble).

Paper.

This paper, in my opinion, is bad. It is not an abstract kind of bad, not a situation where you need to be an experienced forensic meta-analyst to make a determination that something is fishy.

It is oh-that-seems-bad bad. It is wrong-from-across-the-room wrong.

Even if you don’t attempt to re-analyze it, the conclusion is totally unbelievable to anyone even vaguely familiar with the research area. Any competent editor should have punted it straight to the moon in about five minutes. Having let this paper be published is a tremendous oversight, and a harbinger of what’s to come.

As per usual, we retain no opinion as to why it is wrong — some research is real but so poorly done it looks like fraud, some research that is so poorly faked that it looks like incompetence. If you were going to fake a paper, why wouldn’t you fake a paper that made sense? You can write anything you like!

(For about the hundredth time, WHY it is wrong is someone else’s problem. My job is determining IF it is wrong. And if it is, making sure people know.)

So we ask for the data. They don’t send it.

So far, so normal.

I write all the reasons it achieved badness in a post on Research Hub. This sort of thing takes about a week as I pick about at it, come back to it a few times. I’ll assume I started at or very close to May 1st.

A summary so you don’t have to read the post:

(1) The paper straightforwardly states the data is available. It isn’t. We wrote to the authors. They didn’t send it. The data is not available.

(2) The distributions are implausible. The +1/-1 SD is about the size of the range in one case. Not impossible. Just very, very unlikely.

(3) They report the sample had a 98% absence of childhood obesity, but had a sample with many obese young adults.

(4) The analysis provided is completely unclear.

(5) They ran a 120 person RCT by themselves with no funding. (And no proper trial registration, I might add.)

(6) Finally, and most implausibly of all, they report apple cider vinegar is a better weight loss agent than Ozempic. My exact words:

“An unpatentable, freely available, absolutely harmless dietary intervention that is 50% more effective at short/medium term weight loss than GLP-1 agonists for ~1/1000th of the price feels improbable.”

So, this shouldn’t have been published, but it was. Why that happened is unclear. Maybe the reviewers were fake, but I think it’s more likely that whoever reviewed this simply did a terrible job. The paper has all the internal coherence of the Zodiac Killer’s Cipher, and I resent the fact that it exists.

If it crosses your mind at this point that I am being unbearably pissy, and that all these little things don’t matter… remember this is a BMJ journal.

I am not picking on some hardscrabble little journal struggling with a lack of funding and the plucky, upstart authors who support it. The BMJ started publishing in the mid 19th Century. It is completely owned by the British Medical Council, and while I can’t get good information on their budget, they are definitely not starving under a bridge.

Also May, 2024.

Two good looking and intelligent gentlemen, Vahid Malbouby and Eric Trexler, write me after seeing the above post on Research Hub. They also think this is a bad paper, and have additional reasons why.

Their reasons to distrust this paper are more straightforward than mine. An example:

I mean, JESUS.

We write a letter to the journal, which takes another few weeks to coordinate.

June, 2024.

”Your manuscript entitled "Improbable Data Characteristics and Extreme Effects of Apple Cider Vinegar on Weight Loss" has been successfully submitted online and is presently being given full consideration for publication in BMJ Nutrition, Prevention & Health.”

And not a moment too soon.

At this point, it is clear they want $2652 to publish this letter.

(For cleaning up their mess.)

I’ll compress this next part rather than make a million subheadings:

• email to EIC about an APC waiver, says he’s not involved but forwards (with his support) to whoever handles that

• email to BMJ "centralized" bureaucrats who grant APC waivers for all BMJ journals, request denied (!?)

• email(s) to info.nutrition@bmj.com who ignore emails for ~two weeks

• email to ethics and complaints addresses at BMJ

• fee waiver granted

• journal asks for money anyway

• waiver flagged for journal

• resolution achieved

What a good use of time.

July, 2024.

And we wait.

September 29th, 2024

Dear Editorial Office,

I am following up in regard to manuscript bmjnph-2024-000997. The journal website indicates that average time to first decision with review is 88 days. It has been 95 days since our submission, and the paper still does not seem to be assigned to reviewers. Can we please get a progress update?

I would like to highlight the urgency of this letter's publication. The paper discussed appears to be one of the most widely read BMJ NPH papers of all time, and was covered by a large number of major media outlets. Some even use the paper to suggest that apple cider vinegar "could work like natural Ozempic to help with weight loss and diabetes."

Come ON, for the love of God.

October 1st, 2024

This is to inform you that your manuscript is now awaiting editorial decision. You will receive a decision letter in due course.

A nudge after three months resulted in a decision 2 days later?

Could be a coincidence.

Let’s see.

Also October, 2024

”We are pleased to accept your article for publication in BMJ Nutrition, Prevention & Health.”

Hooray! The system works!

Sort of!

Also September / October, 2024

While all this is going on, sharp-eyed readers append no fewer than FIVE additional comments (See the Responses section at the bottom!) pointing out some of the same things, and EVEN MORE points we hadn’t even bothered with or didn’t find!

As far as I can tell, all these comments seem kosher, but do not deal with the data inaccessibility or the mucky distributions.

November / December, 2024.

The authors have replied. Here’s their rejoinder:

• Received: 5 November 2024

• Accepted: 24 November 2024

• First Published: 10 December 2024

However, at this point, our letter still isn’t published yet.

So, there is a published response to our letter (which they got through in a rather speedy 19 days), and got up on the website a few weeks later.

And it somehow refers to our criticism without citing it.

I feel as if some people won’t quite understand what I just said, so I’ll say it another way:

This awful paper has a published response to our criticism.
Our criticism has been accepted for publication, but not published.
So, our criticism is addressed in a formal response, but not cited. Because it can’t be.

Does this look weird? You bet your everything it does. It refers to some nebulous criticism, by unknown authors, just a statement in the passivest of passive voices:

The link there, superscript 1, is to their own paper. This is a new, special, and rare incompetence. And there isn’t any question that the authors are clearly responding to the criticisms we gave — there is no reason these random Lebanese cider researchers would be referencing SPRITE otherwise.

The content of the response is, of course, even worse than the original paper.

It obfuscates several points, says several things that are straightforwardly incorrect, and represents an even bigger editorial failure than the first paper. The alleged refutation of SPRITE is sufficiently silly that I wish I’d called it SPITE, because it’s what I’m starting to feel.

It is somehow still not obvious to the editors at this point that they have published a paper that is completely and irreparably cooked.

Of course, in direct contradiction to the original paper, the data is now a precious commodity around which there are privacy concerns:

Ethical considerations regarding participant privacy and the data’s involvement in ongoing studies have restricted our ability to share raw data directly. However, we have provided anonymised datasets to the journal for confidential review, ensuring the integrity of our ongoing research. We remain fully committed to transparency and are open to any further inquiries or clarifications needed in collaboration with the journal.

I mean, if it’s anonymized, you could… give it to us. We asked in May. I guess the journal can look at it if they want, but what would you expect from them at this point?

A far cry from:

All data relevant to the study are included in the article or uploaded as supplementary information.

It is not at all unusual for a journal to refuse authors a right of reply. In a previous case with Scientific Reports, an author group wrote a response to our letter and we never found out what it was because the journal refused to publish it. Some combination of dopey or unhinged, one presumes.

But to make this decision, you must actually read and understand the content of the paper and correspondence. BMJ:NPH apparently does not do that sort of thing.

January, 2025.

Trex notices the all of the above has happened (and it’s a good thing, because I certainly wasn’t looking), because the journal didn’t tell us.

We immediately find that the reply isn’t citing our letter, because our letter still isn’t published.

We write to the journal AGAIN.

February, 2025

We apologise that your letter was not published alongside the authors' letter of response. This was due to an administrative error. Both letters have now been published in the journal and cite each other and the original research article.

But our letter CAN’T cite their response, it was written beforehand!

On checking the response, the reply to the letter has indeed been updated to cite our letter. Without a correction notice. Now, this is included in the ‘Linked Articles’ section, not the actual references. But it can’t have been there at the time of publication, because it wasn’t published yet.

‘Stealth corrections’ are bad. Is this a stealth correction if there’s a ‘Linked Articles’ section? I don’t know. The journal would say it isn’t. Even though the email says ‘cite’.

There are times when these fiddly details really don’t matter, but this isn’t one of them. And it isn’t unusual — journals issue correction notices for small details all the time.

Anyway. Here's our letter published 5th February. Big deal.

April, 2025

We follow up on the whole ‘do you want to retract the paper’ thing after all of the above with the journal.

They’re still thinking about it. After all that.

May, 2025

And that’s today.

A year.

Cherish it.

A Full Year’s Contents Of The Clown Car

• BMJ:NPH let in a paper that achieved massive public interest but was clearly compromised. Doesn’t matter how, or why. It’s enough to say it is.

• They failed to remove it subsequent to that being pointed out in detail.

• In doing so, they also ignored several excellent public comments saying the same.

• They tried to charge us for submitting a manuscript to correct it.

• They published a response from the authors which was clearly also wrong.

• After months for review and some additional delays, amounting to two hundred and twenty four calendar days, they finally published our letter.

• They published a reply to our letter months before publishing the letter itself, so it couldn’t cite us.

• They stealth-corrected the author’s reply to cite us (although maybe I’m being mean about the ‘Linked Articles’ section), so now it violates the laws of physics.

• They have failed to retract the manuscript in all of the time indicated above, which is now, finally, an entire year. Apparently, they’re still investigating.

The worst part is: I’m probably missing details. This is just everything I could remember and/or justify from notes.

And this is why it is so appealing to simply blow the process off.

I can think of no better analogy than a clown car — it appears on the circus arena and appears to be a brightly-painted regular car. But a clown gets out, and then another, and another, and somehow more still, and past the point where you think ‘surely that’s the last one, how did they pack them all in there?’ and then two more get out and start spraying people with water from comedy boutonnieres, and tripping over their shoes, and generally being obnoxious.

How can there possibly be more clowns? Yet, they continue to appear.

And everyone smiles politely, and finds themselves secretly wishing they’d release the lions early and make it a REAL show.

I’m not complaining, and I don’t expect you to feel sorry for me. I chose to be shaken in this bag of rats. At any point, we could all have done nothing.

Simply: what the hell was the point of any of the above? Critical correspondence about genuinely bad research integrity issues is distinctly second class. It is back-channeled, forgotten, ignored, lost under the couch, thrown in the bin, and generally treated as an annoyance.

Worse still: not only is this all perfectly regular, but other situations are distinctly worse. The only reason this achieved anything close to a resolution was the continual ramming through of the process by my co-authors. Some identified problems are more egregious than this, occur in papers that matter more, and when flagged, still sit around for years. Ask Elizabeth Bik for some of her stories.

I have MORE THAN 100 EMAILS about this silliness.

I’m not doing it any more.

I’m not participating in a system that continually demands respect but doesn’t confer it. I will deal with the smug criticism of ‘why didn’t you access the proper channels?’ as it arises, probably by sending people this document as a reminder of exactly what a stochastic donkey festival the proper channels can be.

If it was merely slow, that would be one thing. But so many, many times the implied attitude we receive is: ‘why don’t you let the experts handle this?’

Well, here you go, here’s why: you aren’t the experts any more.

You’re dealing with a problem that has metastasized far beyond what your crappy, porous system of accountability was designed to maintain, but never changed your approach.

I’ll have news for you soon about this. Something’s coming.

But there’s a fun coda….

Who Is Valeris Crean?

Did you think we were finished? We are not.

Because there was a mystery that ran all the way through this story, one which genuinely puzzled me for a good long while. And one which is not resolved.

But if the above has taught me anything, it’s to cut my damn losses.

One quiet little forensic metascience trick is to look up articles from multiple sources. Sometimes there is different meta-data between those sources. Sometimes that meta-data is divergent between sources! That can be informative.

And one of the best places to look is PMC, because they do something that a lot of journals don’t actually report: list who edited the article.

So, I was confused to find when you look the study up on PMC or on DOAJ, an odd name comes up.

Who is Valeris Crean?

Sounds a bit Game of Thrones, doesn’t it? Is Valeris a real name?

I found some articles listing Valeris as an editor / co-author, (which might mean ‘just an editor, as online services often copy the editor’s name as an author). So I wrote to the corresponding authors of those papers.

Do you really have a coauthor called Valeris Crean?

Who is Valeris Crean?

No response. Weird.

I found an email for this name attached to some of the other published articles listing a contribution by Valeris. I emailed that.

No response. Weird.

At some point, it was suggested that the name might be Valerie, which has the distinct benefit of being a real name.

This was an immediate hit. Several BMJ documents can easily be found bearing the name of Valerie Crean, who appears to be a fairly high-level technical editor with a long history at the BMJ and other real journals.

So, three possible options:

(1) Valeris is a fake editor profile constructed by someone to hoodwink the BMJ into publishing silly papers.

(2) Valeris is a real separate person (just one with an extremely limited web presence outside of editing the sort of papers that Valerie edits) and this is a hell of a coincidence.

(3) Valeris changed her name, her first name. Because it sounded cool.

But, far more likely:

(4) The BMJ has repeatedly misspelled and published the name of their own editor over several years.

What sort of oversight do you have if you spend years misspelling and publishing your editor’s name?