Source: The Conversation – UK – By Simon Kolstoe, Associate Professor of Bioethics, University of Portsmouth
Scientists, professors, engineers, teachers and doctors are routinely ranked among the most trustworthy people in society. This is because these professions rely heavily on research, and good research is viewed as the most reliable source of knowledge.
But how trustworthy is research? Recent news from the US suggests that the Trump administration wants to fund more “reproducibility studies”.
These are studies that check to see if previous results can be repeated and are reliable. The administration’s focus seems to be specifically on studies that revisit the debunked claim of a link between vaccines and autism.
This is a worrying waste of effort, given the extensive evidence showing that there is no link between vaccines and autism, and the harm that suggesting this link can cause. However, the broader idea of funding studies that attempt to repeat earlier research is a good one.
Take research on Alzheimer’s disease as an example. In June 2024, Nature retracted a highly cited paper reporting an important theory relating to the mechanism of the disease. Unfortunately, it took 18 years to spot the errors and retract the paper.
If influential studies like this were regularly repeated by others, it wouldn’t have taken so long to spot the errors in the original research.
Alzheimer’s is proving a particularly tricky problem to solve despite the large amounts of money spent researching the disease. Being unable to reproduce key results contributes to this problem because new research relies on the trustworthiness of earlier research.
More broadly, it has been known for almost ten years that 70% of researchers have problems reproducing experiments conducted by other scientists. The problem is particularly acute in cancer research and psychology.
Joshua Sukoff/Shutterstock
Research is difficult to get right
Research is complicated and there may be legitimate reasons research findings cannot be reproduced. Mistakes or dishonesty are not necessarily the cause.
In psychology or the social sciences, failure to reproduce results – despite using identical methods – could be due to using different populations, for instance, across different countries or cultures. In physical or medical sciences problems reproducing results could be down to using different equipment, chemicals or measurement techniques.
A lot of research may also not be reproducible simply because the researchers do not fully understand all the complexities of what they are studying. If all the relevant variables (such as genetics and environmental factors) are not understood or even identified, it is unsurprising that very similar experiments can yield different results.
In these cases, sometimes as much can be learned from a negative result as from a positive one, as this helps inform the design of future work.
Here, it is helpful to distinguish between reproducing another researcher’s exact results and being given enough information by the original researchers to replicate their experiments.
Science advances by comparing notes and discussing differences, so researchers must always give enough information in their reports to allow someone else to repeat (replicate) the experiment. This ensures the results can be trusted even if they may not be reproduced exactly.
Transparency is therefore central to research integrity, both in terms of trusting the research and trusting the people doing the research.
Unfortunately, the incentive structure within research doesn’t always encourage such transparency. The “publish or perish” culture and aggressive practices by journals often lead to excessive competition rather than collaboration and open research practices.
One solution, as new priorities from the US have suggested, is to directly fund researchers to replicate each other’s studies.
This is a promising development because most other funding, alongside opportunities to publish in the top journals, is instead linked to novelty. Unfortunately, this encourages researchers to act quickly to produce something unique rather than take their time to conduct thorough and transparent experiments.
We need to move to a system that rewards reliable research rather than just novel research. And part of this comes through rewarding people who focus on replication studies.
Industry also plays a part. Companies conducting research and development can sometimes be guilty of throwing a lot of money at a project and then pulling the plug quickly if a product (such as a new medicine) seems not to work. The reason for such failures is often unclear, but the reliability of earlier research is a contributing factor.
To avoid this problem, companies should be encouraged to replicate some of the original findings (perhaps significant experiments conducted by academics) before proceeding with development. In the long run, this strategy may turn out to be quicker and more efficient than the rapid chopping and changing that occurs now.
The scale of the reproducibility, or replicability, problem in research comes as a surprise to the public who have been told to “trust the science”. But over recent years there has been increasing recognition that the culture of research is as important as the experiments themselves.
If we want to be able to “trust the science”, science must be transparent and robustly conducted.
This is exactly what has happened with research looking at the link between vaccines and autism. The topic was so important that in this case the replication studies were done and found that there is, in fact, no link between vaccines and autism.
Simon Kolstoe works for the University of Portsmouth, and is a trustee of the UK Research Integrity Office (UKRIO). He receives research and consultancy funding from charities, universities and government. He chairs research ethics committees for the UK Health Research Authority, Ministry of Defence and Health Security Agency.
– ref. Scientists should try to repeat more studies, but not those looking for a link between vaccines and autism – https://theconversation.com/scientists-should-try-to-repeat-more-studies-but-not-those-looking-for-a-link-between-vaccines-and-autism-253696
