For the best experience on the Abcam website please upgrade to a modern browser such as Google Chrome
What does reproducibility mean to you and your work, and why is it important?
Dr. Trimmer: Many scientists are attracted to their careers by the joy of discovery when elucidating the underpinnings of natural phenomena for the first time while making a tangible and lasting contribution to society.
This can only come about if their research is transparently reported and reproducible. Without peer-reviewed publications of our research findings, the research we do has little to no impact and does not represent a bona fide contribution.
Reproducibility is important to science and society as a whole as publishing irreproducible research takes researchers down paths that can consume valuable and limited time and resources in an unjustified manner. We should be doing everything in our power to perform reproducible research and publish it in peer-reviewed journals so that it will have lasting value.
Dr. Edwards: More than formally being able to reproduce a specific experiment as it’s narrowly performed, a careful scientist will ensure that the results apply slightly more generally. Then the work is more likely to provide the foundation for research to be generalized and built upon in future research. Without reproducibility in science there would be very little progress or translation of findings into usable applications.
Science as a discipline ultimately has an impact only if trusted. If we, as researchers, continue to populate the literature with irreproducible experiments due to poor experimental design, or unwillingness to test the concept/discovery in an orthogonal experiment, then it will be to society’s detriment. For example, it pains me when scientists use the phrase “to validate our results, we performed the following experiment…”. The entire premise of the next experiment is then to try and invalidate your result… This simple, and perhaps semantic, statement underscores the problem.
Dr. Trimmer: Our job as scientists is to accurately describe natural phenomena as they occur. If we do the experiments correctly under controlled conditions and are transparent in our reporting, then the results should be reproducible at a fundamental level.
These phenomena should occur the same way today, tomorrow or next week, assuming the samples are the same, and the experiments are done the same way. Doing irreproducible research not only negatively impacts science in terms of its fundamental value to society but also how scientists and the pursuit of science are perceived by non-scientists.
There is enough anti-science sentiment to contribute further to the growing lack of respect for science and evidence-based conclusions, without adding on top a lack of reproducibility in research performed by practicing scientists and supported by public funds.
Dr. Edwards: The greatest barrier is that the short-term incentive system rewards those who are first to publish, not the ones who ultimately get it right. In the long run, scientists who carefully solve a problem are always rewarded, but this requires dedication and integrity, which is rare. There is a great pressure to be first and publish often, which encourages less rigor in experiments.
The other barrier is that some interdisciplinary work passes peer-review and is published due to the lack of expertise of the reviewers in the other discipline. For example, a cell biologist is asked to review a paper using a chemical inhibitor. They review the biology, but unbeknownst to them, the chemical inhibitor is non-specific. The work is meaningless even though cell biology was excellent. It is difficult for reviewers to discern the quality of reagents in fields that are not their own.
Dr. Trimmer: From my perspective, adequate training, in both quantity and quality, remains a barrier. This contributes to a lack of awareness of the problem with reproducibility, and/or the basic tenets of designing, executing, and reporting research results that have a high likelihood of being reproducible. On the part of reagent developers and suppliers, another substantial barrier is the lack of transparency, as it’s often not entirely clear what is in the “tube”.
Dr. Edwards: I don’t think the solutions are technical but sociological. In specific fields like antibody characterization, the ability to use knock-out cells as controls and make renewable antibodies offers the path to identify outstanding reagents – but this will have little impact if the community does not insist on this level of characterization, if the identities of the good antibodies are not known and if the incentive system (journals and grants) do not reward the use of the best reagents. In the absence of updated incentives that reflect this, consumers will purchase the cheapest products and their work will likely face issues with reproducibility.
Dr. Trimmer: I agree, unambiguous molecular definition of biologicals including antibodies is crucial to improve reproducibility. When we buy inorganic chemicals, the composition is listed on the container’s label. This is not the case for most antibodies, and definitely cannot be the case for polyclonal antibodies, due to the batch-to-batch variability inherent to polyclonal production.
Quantitation of antibody labeling is another crucial area. Many assays we use yield a direct numerical readout, ELISAs being a prime example. However, techniques such as immunocytochemistry and immunohistochemistry do not directly yield such output and are prone to being presented as “representative examples”. In the case of immunocytochemistry, this can be misleading as there can be substantial heterogeneity between different cells in the sample making it difficult to distinguish between “within sample” and “between sample”.
Appropriate controls for immunolabeling is another important issue for which enhanced training is needed. It is crucial to have positive and negative control antibodies that are as similar to the antibody being evaluated as possible. For example, evaluating purified monoclonal antibodies, tissue culture supernatants, affinity-purified antibodies and antisera - to compare them at comparable antibody concentrations, and making sure all other sample preparation and labeling conditions, imaging, etc. are done identically. Multiplexing is valuable, as having positive and negative internal controls within the same sample can enhance the value of the analysis by providing an internal “denominator” to compare any changes in the “numerator” (for example in a knockdown experiment).
Scientists can also improve reproducibility by “blinding” their samples, and by authenticating key reagents, including but not limited to: cell lines, strains, antibodies, employing rigor to sample preparation, immunolabeling, imaging and image analysis, as well as work within their lab/unit/organization to develop and deliver training to ensure rigorous research.
Dr. Trimmer: Developing a consensus as to the minimal standards for transparent reporting by both reagent developers and researchers, both critical to the future of research. This needs to be enforced through cooperation with developers, researchers, publishers, and funding agencies as to required standards for reagent authentication and transparent reporting. The key here is greater transparency from companies. Enhanced training is also key including standard course material highlighting major issues around reproducibility, as well as fundamental tenets of best practices to ensure the reproducibility of research results.
Dr. Edwards: I think we all have identified the problem – the solution is now up to each and every one of us in terms of taking responsibility for ensuring reproducibility. Organizations and scientists that commit to quality now will ultimately be the “winners”, even if in the short term this might not be obvious or rewarded. And over time, peer-reviewed journals and funding bodies will “flip” and prioritize quality and reproducibility. It is happening already, and I expect it will only happen more over time. I am optimistic!
To learn more or register for our upcoming live webinar and Q&A series “Reproducible Science Week,” 1-5 June 2020, visit our page.
Dr. Trimmer is Distinguished Professor of Physiology and Membrane Biology at the University of California, Davis School of Medicine, USA. His lab focuses on ion channel signaling complexes in neurons. He began generating monoclonal antibodies in the early 1980s and this expanded to generate mAbs, pAbs, and other affinity proteomics reagents for collaborators.
His work eventually led to the creation of the UC Davis/NIH NeuroMab Facility with a broader mission to generate highly validated monoclonal antibodies for neuroscience research. He has since initiated efforts to convert his group’s large mAb library into recombinant form and to develop nanobodies and other miniaturized binders for neuroscience research.
About Dr. Aled Edwards
Dr. Edwards is Chief Executive, Structural Genomics Consortium, and Professor at the University of Toronto, Canada. His group’s research is focused on identifying new biology and drug discovery opportunities among the thousands of relatively under-studied proteins encoded by the human genome.