Researchers behaving badly
HSV Public Lecture by Professor David Vaux AO, Deputy Director and Joint Division Head, Walter and Eliza Hall Institute of Medical Research, at Hawthorn Community Precinct, 27 July 2017
I work in medical research, especially cancer research, and when I do research I publish articles in journals such as Science or Nature. The way that I find out what other people are doing is to read such journals.
C. Glenn Begley and Lee M. Ellis are researchers for Amgen, a world biotech company who try to find new drugs to fight cancer. Often they would try to reproduce results published in a journal paper and could not reproduce the results. So in frustration they looked at 53 landmark papers and found that they could reproduce them only in six cases (i.e. 89% could not be reproduced). This created what is called the reproducibility crisis. I came to similar conclusions and discuss similar cases in my talk.
For example, I looked at ten papers in an edition of Nature, a prestigious journal. These papers had figures which showed measured mean values plus t-shapes called error bars, which represent the likely range in which the true value lies. The error bars can represent standard deviations, standard errors, confidence intervals or range; in only three of these papers did it say what the error bars represented.
For the next paper I discuss the Poisson Distribution. It is the distribution you get if you are looking at a number of cells in a high powered microscope. It is a distribution where you can obtain only non-negative integer numbers, with in theory very large integers. The integer values are plotted on the horizontal axis and their frequency on the vertical axis. The distribution has a long tail to the right and is like a bell-shaped normal curve but is squashed against the vertical axis. The standard deviation of the distribution is the square root of the mean. This is relevant to a paper published in Nature in 1988 supposedly proving that homeopathy works. Statisticians who examined this paper concluded that the standard deviations obtained were much smaller than expected for a Poisson distribution. Attempts to reproduce the results of the experiment were unsuccessful. Homeopaths today still quote this paper. In another paper published in Nature the standard deviations again were much smaller than expected.
Not all distributions are discrete values like the Poisson Distribution. Sometimes the distributions are continuous, e.g. in cancer research the volume of a tumour in a mouse. In a paper published in Nature researchers have treated irregularly shaped tumours in mice treated with their new drug. They measured the length, sideways or lengthways of tumours The standard deviations are much less than expected. In the article they showed pictures of several mice which all looked to be the same mouse. The tumours shown were more than 3 times the amount ethically allowed.
An article published in Nature Medicine showed pictures of tumours treated with the inactive and active forms of the drug. Supposedly the latter tumours were much smaller. Pulling apart the PowerPoint slide of this data shows the results were cherry-picked and in fact there was no appreciable difference in the size of the tumours.
I am on a board called the Centre for Scientific Integrity, which publishes a blog called Retraction Watch which gives a list of retracted papers. The most retracted papers are 183 by an anaesthetist named Yoshitaka Fujii.
Scott Reuben is an anaesthetist who was one of the few people sent to jail for faking his results. He published a paper in the prestigious New England Journal of Medicine, which claimed the Merck drug Vioxx was good for acute pain and did not cause stomach problems as other anti-inflammatories do. This led to billions of dollars of sales of Vioxx. The paper withheld data about the number of heart attacks associated with Vioxx. The Federal Drug Administration subsequently made a report that said 140,000 cases of heart disease were caused by Vioxx and 566,000 deaths. Merck subsequently withdrew Vioxx from the market.
Merck paid Elsevier to publish a journal called The Australasian Journal of Bone and Joint Medicine, which had the look of a peer-reviewed medical journal but only contained reprints or summaries, most of which contained data favourable to Merck. There was no disclosure that this was paid for by Merck. The journal was given to rheumatologists and orthopaedic surgeons.
In the US if an issue is raised about possible research misconduct you can write to the US Office of Research Integrity (ORI). The ORI investigates research misconduct and recommends that the government take action over misconduct. Similar bodies exist in the UK, Netherlands, Germany and Denmark. No such body exists in Australia, where researchers are self-governed. Australia needs an ORI. So if you want to be active, write to the relevant Mini¬ster, Greg Hunt, and say that Australia needs an ORI.
Report by Joe Sampson
