Introducing the Medical Evidence Project

And the quest to make less dead people

It is April the 16th. I am feeling nostalgic as I started writing this exactly a year after I wrote the below.

Exactly a year. Almost to the minute.

If you’re reading this, it’s a few weeks later.

This is because, at the time of writing, I am holding on to a great secret.

If you missed the piece linked above, it is absolutely central context for the good news, and you are about to receive very good news indeed, so you should read that first. This will still be here when you get back.

But: if you can’t bear to tear your attention away from proceedings here because I am so compelling, I will shamelessly editorialize:

• Forensic metascience is an effective response to bad and/or fake research.

• However, it takes a long time to do. And it is annoying to execute.

• For this and other reasons, especially its perceived inherent hostility, it is unattractive to fund through academic mechanisms. We should not expect any industry to fund ‘attacks’ on itself.

• As a consequence, if we wish forensic meta work to be funded, we should use it to maximally reduce human suffering as a priority above all other reasons. We should keep it simple and play our position.

• Specifically, we should find medical evidence which is bad and/or fake, especially when it contaminates treatment guidelines and kills people.

• This kind of contamination has happened before, and has killed people.

• It will happen again.

• It shouldn’t happen at all.

Now, here is the good news.

This project is now funded. We’re doing it.

Really?

Yes.

I mean, it’s in Nature. So I’m out of alternatives.

In that vein, I am proud to announce that Open Philanthropy is supporting this proejct.

(Open Philanthropy is a grantmaking organization which aims to use its resources to help others as much as it can. They have a long-standing interest in metascience. What remarkable good taste.)

I don’t know if it’s true to say that it’s the first formally funded forensic metascience project in history. Using a looser definition, definitely not. But it’s one of the first, and it’s certainly the first one like this.

We’re running the project out of the Center for Scientific Integrity, which is the parent 501c3 nonprofit that runs Retraction Watch. There will be some cross-over roles, because there is an important feature of this which I really need to explain for it all to make sense.

I have written before, as have many other people, about the sheer and unending silliness of dealing with regular research integrity processes. The people involved on ‘the other side’, wherever that may be, are slow. Or, slower than that. They lack accountability. They have guidelines for pursuing research integrity questions, like COPE, which are neither followed nor enforced. Complaints are often bounced between disinterested parties.

And if we find active threats to human health in this literature, I think it is immoral to get tied up in this silly process that you can’t trust or rely on.

Let's say we found an unambiguously contaminated treatment guideline. Should we enter into some nonsense exchange of emails that takes two years to issue an expression of concern while people are hurt that whole time?

No - we’re going straight to press.

No filter, no university intermediary, no silliness. It’ll look more like this:

1. Problem identified.

2. Report written about problem.

3. If serious enough, problem peer reviewed internally (AND I HAVE A LINE ITEM IN THE BUDGET TO PAY THEM)

4. Problem immediately reported in press.

Where did we get that?

From the last forty years of hacking, of course.

White hat hackers, being more practical than scientists, are less prone to accept formal responsibility nonsense if it makes no sense, and are more prone to act ethically and loudly. This is simply the policy of full disclosure in full effect.

I’m going to list the first section of the Wikipedia entry here, so you can see just how goddamn straightforward the parallels are:

In the field of computer security, independent researchers often discover flaws in software that can be abused to cause unintended behaviour; these flaws are called vulnerabilities. The process by which the analysis of these vulnerabilities is shared with third parties is the subject of much debate, and is referred to as the researcher's disclosure policy. Full disclosure is the practice of publishing analysis of software vulnerabilities as early as possible, making the data accessible to everyone without restriction. The primary purpose of widely disseminating information about vulnerabilities is so that potential victims are as knowledgeable as those who attack them.[1]

In his 2007 essay on the topic, Bruce Schneier stated "Full disclosure – the practice of making the details of security vulnerabilities public – is a damned good idea. Public scrutiny is the only reliable way to improve security, while secrecy only makes us less secure."[2] Leonard Rose … explains "We don't believe in security by obscurity, and as far as we know, full disclosure is the only way to ensure that everyone, not just the insiders, have access to the information we need."[3]

Does that need further explanation? I doubt it.

So, as the process requires disclosure at a polished and professional level, we are hiring an editor (alongside myself, and I will be part time, and analysts).

Investigations often used to proceed in stages, slowly, piece by piece, until we finally got to the point where we could go public. But when the threat level goes up, the moral imperative is strongly in favour of speed-running the entire thing.

Bear in mind at this point that speed-running also implies doing the process correctly. I don’t mean hastily, I just mean as quickly as is responsibly possible. In modern terms, that isn’t that fast.

But it certainly is faster than the example below (ed: it only took a few weeks for a really good one to pop up!)

We have a broader remit for the inclusion for any given case: prevent damage.

If there is the possibility of bad research killing or hurting people, a case if viable.

If not, it’s not unviable, but it certainly is less interesting.

In other worse, I think we will be distracted by being bounced between by three competing poles:

(a) very cool forensic metascience stories that are stimulating to write and interesting to investigate, but do not threaten human health

(b) non-research, research-adjacent, or mostly-but-not-entirely-research stories which affect medical practice, have real but also very abstract second- or third-order effects on human health downstream of how they affect the practice of medicine, and

(c) the central project mission (bad papers, contaminated metas, affected guidelines)

I want to do as much (c) as possible, but I think there is an inevitability around (a) and (b) stories, especially as we’re getting started. There may be broader value to the organization in some of those other stories.

Also bear in mind, we need to turn the mechanism by which we achieve (c) stories into the outcomes of a formal process, and to have an analytical pipeline that formalizes how we find, expurgate, and correct dangerous papers. We will have to build a whole rack of tools to do this, and it’s equally as challenging as doing the actual forensic metascience work of dealing with dangerous papers once they’re identified.

(With a larger staff, the MEP could conceivably have a broader remit. But for right now, the preservation of quality of life and life itself is the iron core through which I want our operations to run. That isn’t to say that other cases will be ignored! It is simply that the pipe will likely fill up quickly with studies that affect human wellbeing, which will leave less room for other actionable-but-less-threatening things. I really don’t think that saving money, saving researchers time, and expurgating bad practice/research for its own sake are bad ideas. If I did, I probably would have spent the last decade behaving differently.)

Finally, as this project is run out of CFSI, it means Ivan Oransky is now my boss.

The poor man. Hopefully the slings and arrows of outrageous fortune are poorly aimed.

And, of course, there will be other MEP employees — people with more skills and impulse control than yours truly. More on that soon.

More Stuff

I have had a few quiet conversations about this so far. Here are some questions that have come up.

Are you excited?

Yes. But I’m also too focused to be visibly excited. When I got the news, I was excited. Now I’m just making plans.

I feel a truly annoying weight of responsibility for maintaining the viability of funding this sort of work. If this works, future projects become viable. If it does not, we’ll all look the fool, and I’ll have to start a bakery or something.

But my annoyance does not change the fact that the responsibility is still there.

This is dope. What can *I* do?

Very good question. There are two primary things I need right now that aren’t money.

(1) Everyone needs to know. Send any of our public facing materials to anyone.
I need conference slots, podcasts, new friends, new places to write, and a partridge in a pear tree — that is, if I am also in that pear tree, screaming through a loud hailer in a cloud of questionable hygiene and moral opprobrium. I am now in the self-aggrandizement business. Write to me to help me self-aggrandize.

(Yes, more than usual.)

This is an absolutely dogshit media environment to launch something new and positive, one of the worst I can remember. How am I supposed to compete for attention with the present omnishambles? I don’t know the precise answer, but I do know the volume: LOUDLY.

Happily, Agent Starling, loudly is the only volume I have.

(2) I need tips.

You can send them here.

Even just through our local network and some very low-level preliminary analysis, we have more medical academic naughtiness than we can presently act on.

But that is no reason not to have more. I want a lot more.

Because, somewhere out there, someone is reading this who knows about Something Really Bad, much worse than what we are dealing with already. Some case, some paper, some problem, and it’s dire, and they’re alone with it.

That last part is very, very important — something that you learn after talking to… maybe 50 whistleblowers in some capacity over a period of years: they’re isolated, and it’s usually a goddamn horrible experience.

Whistleblowers are stuck with a secret that is outrageous and inconvenient (if they aren’t directly involved) and/or deeply shameful (if they are). But they usually don’t feel like they can tell anyone about it.

Certainly not the institution they work at, reprisal is real.
Or the government, who would you even write to?
Or the media, who’s going to care about some abstract problem with acronyms in it?
Or some faceless journal editor, why wouldn’t they brand them a troublemaker?
And a thousand other reasons besides.

And, of course, there is no course or accepted standard to even write up a complaint — there is no ‘whistleblower school’, these unfortunate fellows are simply stuck with complex information about which they have no idea how to disseminate but also no experience in structuring or investigating.

I’ve had adults, scientists, people clearly unfamiliar with having an intense emotional response to their work, cry on the phone. Only twice, but twice is enough.

All it takes is a sentence like:

“I understand how crazy-making this is, and I think you’ve done very well articulating it so far. We’ll look into it for you.”

Well, if you’re talking about a problem that hurts people, I will look into it for you. We will look into it for you. You can have all the public credit or none of it, you can be completely anonymous or we can be.

And, even if isn’t strictly within our remit — remember how much stuff is out of scope (see above!) — you can still tell me.

Besides this:

(3) Obviously, we need money.
This is a nonprofit endeavor, and all nonprofits live and die on their donor base. At the same time, I am hesitant to ask while science is being dragged around by its neck and beaten with a fireplace poker. If you need money for essentials like rent and death metal t-shirts and cat biscuits (OK, these might just be me), or the NIH just shitcanned your entire life, for the love of God, do NOT give us money.

I mean it. If you need it, I don’t want it.

If you CAN afford to go around making donations to scallywag nonprofits and are capable of writing ‘for causing undifferentiated trouble’ on the left hand side of a cheque, our donation button is here.

Pursuant to the above…

(4) If you have ideas for structural funding, I’m all ears.
I’ve already had interesting inbound about co-managing grants and PhD students, and some of those are both aligned with the mission and come with Survival Tokens. Please bear in mind that my time to write long, hideous grant applications will be extremely limited, and if your bright idea is ‘why don’t you do all the work so you have enough money to employ me personally’, then will I politely decline in advance (while making the following Mr. Toast face inside).

A highly disapproving cat who has been denied third breakfast, a creature with NO SENSE OF PROPORTION and a perpetual grievance. Source: unfortunately, me.

One thing I am not looking for is an army of volunteers to do the work, because I cannot train and run them. We will likely collaborate with domain experts on certain topics, and outside of that, it is an absolute vortex of lost time. We don’t have the structure to make this work with a larger cohort — there is a way to, but we’re not there yet. I may have specific tasks I could really use help with, but those will be a message from me to you, and not the other way around. (And, of course, you are completely within your rights to tell me to eat it.)

How do the present problems in US science affect this?

A bit. Not as much as you might think, as the US is not the focus. None of this project was set up with US research, US researchers, or US guidelines specifically in mind. Also, by virtue of the timeline that grants work under, it was conceived well before we became trapped on the present burning rollercoaster of the NSF/NIH funding.

Specifically, though: in the operational sense, it is not affected at all. This project is privately funded. The government cannot withdraw funding from us, because we do not have any of their money. And given some of my let’s say ‘stridently stated opinions’, I presume they also have no plans to give us any. Making us just like everyone else right now, I’m afraid.

In the broader sense, there will be some disruption, just not that much. I expect it to be significantly harder to find collaborators, given the collective stretch on time and budgets, and the amount of people out of work. Given the stress people are under, let's call that me-problem.

In the eventual sense, I have only fairly vague news. A massive structural funding shortfall might potentially lead to future work that is less trustworthy. I see two buckets for this work: (1) hasty and underfunded projects that badly lack resources then cut corners, or (2) projects conceived cynically to fit a dominant narrative that is mad as pants. I expect that given the very cynical messaging right now about ‘improving research quality’, we'll see some people stooging for that outcome in the service of very toxic goals. If so, well, maybe we have to have a conversation.

But that may not be the remit, which is: stop bad medical evidence making people sick or dead. That is all I care about.

If this isn’t clear, let me try it a different way:

(1) The academic study of medicine is aggressively international, just like everything else. The first crop of papers I’ve got to look at are from everywhere: Iran, Bahrain, the UK, the Netherlands, a few others. Now, call me a miserable old leftie, but I think ‘make less sick people’ matters everywhere.

(2) Absolutely it’s a bit of a bin fire over here, but the USA is ~4% of the world’s population, and that means there are medical guidelines that cover the 96% of other people. These are every bit as important, and just as likely to be affected.

Finally, it seems likely that specific countries will be involved… just (a) as the source of untrustworthy papers, as there are some repeat offenders in that respect, and (b) meta-analyses / guidelines available to interrogate could be from anywhere, but for historical reasons, are likely to be overrepresented in UK/Europe.

Isn’t it a bad time for this considering <points around themselves vaguely>?

I don’t there’s a wrong time to ensure medical guidelines reflect the best available evidence. That’s an all-the-time thing.

Will this make people trust research less?

People are very hasty about proximal versus distal trust. Proximally? Of course it affects trust. I’m sure some people will read about some work that we’re about to do and say ‘I trust medicine less’. I will be messaging like hell against this, but it’s inevitable.

Distally, this is the start of a trust-improving process. If enough problems are located, there is a certain inevitability towards the literature getting better — and not because we can find all the problems and people straightforwardly trust to improve them all (because we can’t and they won’t).

But: because a wide variety of research and publication processes can presently be improved. Less silly reviews and meta-analyses being written which uncritically slop in anything with two integers to rub together would be a good start. So would moving towards policies around making individual patient data available for aggregate reviews. So would teaching good meta-analysts to include forensic checks in their work. So would systems that automate flags and suspicious activity in individual papers — we already sort of have these! A great deal is possible to make medical science better, and a lot of it is possible with a moderate amount of time and money.

People often give me A Look when I say I’m quite optimistic about all this.

It took me a while to realize I do not sound like an optimist, but I am one. No-one who pursues an occasionally Quixotic quest over years to improve something complicated thinks we’re all fucked and nothing can change.

I did that. Now I’m doing this.

And it really feels like the start of something better.