Timid medical research

Cancer research is sometimes criticized for being timid. Drug companies run enormous trials looking for small improvements. Critics say they should run smaller trials and more of them.

Which side is correct depends on what’s out there waiting to be discovered, which of course we don’t know. We can only guess. Timid research is rational if you believe there are only marginal improvements that are likely to be discovered.

Sample size increases quickly as the size of the effect you’re trying to find decreases. To establish small differences in effect, you need very large trials.

If you think there are only small improvements on the status quo available to explore, you’ll explore each of the possibilities very carefully. On the other hand, if you think there’s a miracle drug in the pipeline waiting to be discovered, you’ll be willing to risk falsely rejecting small improvements along the way in order to get to the big improvement.

Suppose there are 500 drugs waiting to be tested. All of these are only 10% effective except for one that is 100% effective. You could quickly find the winner by giving each candidate to one patient. For every drug whose patient responded, repeat the process until only one drug is left. One strike and you’re out. You’re likely to find the winner in three rounds, treating fewer than 600 patients. But if all the drugs are 10% effective except one that’s 11% effective,  you’d need hundreds of trials with thousands of patients each.

The best research strategy depends on what you believe is out there to be found. People who know nothing about cancer often believe we could find a cure soon if we just spend a little more money on research. Experts are more sanguine, except when they’re asking for money.

6 thoughts on “Timid medical research

  1. Don’t forget the financial incentive: drug companies make more money delivering incremental improvements as their patents run out, than if they found a massively more successful drug that will run out in the same amount of time as one incremental drug.

    Taking more risk for lower (economic) payoff is always a hard sell.

  2. As far as OP goes, I think biases need to be considered: larger trials are, aside from being better powered to reduce sampling error, also going to be better against systematic biases. It’s very very easy to make a trial of 100 patients disappear; much harder to make a trial of 100,000 patients disappear. And publication bias is only the start.

    Marc, you’re ignoring the fact that they can charge much more for a much more effective drug. In fact, we’re seeing an interesting example of this right now with hepatitis drugs: apparently a miracle cure has recently come onto the market, and everyone is upset over how it costs many thousands of dollars to effect a cure. On the other hand, if it had simply been a chronic treatment of modest efficacy which costs hundreds of dollars (for a lifetime cost-effectiveness abysmally below the miracle cure), no one would peep…

  3. Don’t forget the timidity that comes from the fact that you’re putting an incompletely characterized molecule into a person’s “system-wide bus” (circulatory system) that could bind to almost anything, in a population of patients all of whom have wildly different physiologies, causing almost any effect imaginable, up to and potentially including death in some patients.

    Patients that are close to dying anyway are more willing to take the more extreme risks (and I think they should be given the right to submit themselves as test subjects to promising maverick/cowboy research, with far less interference than normal from IRBs etc., assuming they have been educated on the risks and make this choice of their own free will).

  4. @Luke: Absolutely. You can’t just say “This molecule is similar to this other one, so it should act similarly.” Sometimes a small change can make a huge difference.

    Cancer patients do take big risks, at least in sub-types of cancer that have no reliable treatments. Acceptable toxicity probabilities for chemotherapy might be two orders of magnitude larger than for, say, cold medicine.

    Most leukemia patients at MDACC will be part of at least one clinical trial. With more treatable forms of cancer, e.g. breast, the proportion of patients in trials is lower.

  5. quote
    except for one that is 100% effective.
    you know what they say about people who assume (1)..seriously, you are absolutely right that we are chasing small effects, and therefor need very large, long, $$$ trials.

    All I can say, having had occasion to delve into this a little, is that clinical trials are a *lot* harder then they look, mainly cause people are complex, and variable, and disease is variable..

    it is fun to think that the gov’t is hiding the 100 mpg carburetor along with the never dull razor, and large drug companies certainly do stupid things

    but, very, very sadly, there ain’t no magic (anticancer) bullets

    IF you want to see what a really solid trial looks like, google WESCOPS: thousands of people, for many years, with an unambiguous endpoint: fewer deaths (less all cause mortality in the jargon; I think the similar S4I got the same result)
    1) https://xkcd.com/1339/

Leave a Reply
