New publication: Impact of tumor heterogeneity and tissue sampling for genetic mutation testing
Staff at QED Biomedical Ltd have conducted a systematic review to understand how the presence of tumour heterogeneity in tumour tissue samples may affect current clinical research and precision medicine.
Tumour heterogeneity describes the presence of:
a) tumour and non-tumour tissue within a given research tissue sample
b) multiple clones (different cancers with different characteristics) within a given patient tissue sample.
Tumour heterogeneity is a concern for the research of somatic tumours; those tumours that develop over the lifetime of an individual (not inherited tumours).
The research identified the following limitations:
1) there are no guidelines to aid systematic reviewers, clinical researchers, or decision makers in considering, critiquing, and understanding the impact of sample bias produced by tumour heterogeneity.
2) the primary authors of molecular biomarker research do not report their methods with enough detail to allow careful evaluation
3) tissue samples derived from databases do not report any baseline pathology or pathology methods
Sample bias due to tumour heterogeneity will introduce false negative results and therefore not all tumours will be identified by genetic or molecular biomarkers. Therefore, underreporting of gene mutations occurs and ultimately this means that patients may not receive treatments that could benefit them
What should change:
· Primary authors need to transparently report tissue sample pathology methods, results, and tumour purity.
· Molecular biomarker research involving tumour samples prone to heterogeneity should consider the use of microdissection, or liquid biopsy to ensure the purity of samples or investigate multiple samples per patient
· Formal guidelines are required to help evidence-based decision makers identify and report the potential biases in the sampling of tumours.
How can QED biomedical help you?
We can assess the bias present in preclinical research that relies on tumour tissue samples. This enables you to invest in the best genetic biomarker research and avoid flawed or unreproducible research.
Swift S, Duffy S, Lang S. 2020. Impact of tumor heterogeneity and tissue sampling for genetic mutation testing: a systematic review and post hoc analysis. J Clin Epidemiology 2020; 126, 45-55. https://doi.org/10.1016/j.jclinepi.2020.06.010.
Repurposing antimalarials for COVID-19: Doing more harm than good?
In the absence of an effective therapy for COVID-19 there has been growing interest in repurposing drugs, such as the antimalarials chloroquine (CQ) and its derivative hydroxychloroquine (HCQ).
Interest in chloroquine/hydroxychloroquine was sparked by a publication in March, from a French professor , reporting significant reductions in viral loads in COVID-19 patients from a non-randomized, uncontrolled, open study of 20 patients. Currently, nearly 150 clinical trials of CQ, HCQ, or both are registered worldwide, but to date there is no clear indication that either drug improves outcomes from COVID-19.
The rationale for promoting CQ/HCQ as a potential therapy against COVID-19 was based on results from in vitro studies. Three studies, two from the same group in Wuhan and one from researchers in Beijing, reported that both drugs were highly effective in reducing SARS-CoV-2 replication (the virus which causes COVID-19) in a cell line. Using a checklist of strict criteria, we judged the quality of the data from each of these studies and concluded that the data did not support their conclusion.
The investigators used VeroE6 cells (an African green monkey kidney cell line) to determine the effectiveness of HCQ and CQ as anti-virals without replicating in a different model – a human cell line, for example. Both groups quantified viral copy number using qRT-PCR but reported only normalised values. I guess we must take it on trust that both compounds reduced viral load? Unfortunately, this lack of transparency was further compounded by poor reporting of technical repeats and it was not at all clear if any of the experiments were actually repeated?
One of the biggest challenges facing biomedical research is poor reproducibility, yet these studies were fast-tracked without peer review and replicability. Ignoring good practices of transparency and rigour can never be justified, particularly in times of crisis. If we cannot trust the preclinical evidence, then how can be justify going forward into patients?
QED Biomedical is helping to tackle the reproducibility crisis. Our methodology, based on the gold standard of systematic review, can identify the most reliable evidence - towards clinical trial success.
Expt. 2 : Immunofluorescent expression of viral protein (NP) in VeroE6 following treatment with CQ or HCQ
Expt. 3A: Mechanism of action of CQ and or HCQ on viral infection of VeroE6 cells. Quantified viral copies by qRT-PCR and normalised to vehicle control
Expt. 3B: Mechanism of action of CQ and or HCQ on viral infection of Vero E6 cells. Quantified expression of NP (a viral protein) by immunofluorescence
NR = Not reported. NA = Not applicable. EB = Error bars. CI = Confidence Intervals Sources Wang Yao Liu
Could Neil Woodfords dramatic fall have been avoided?
Evidence-based decision making for due diligence
Neil Woodford was known as the UK’s best-known stockpicker, a leading UK fund manager. However, in October 2019 he was forced to close his business, which once managed £15bn. After a period of poor performance he trapped thousands of savers in his flagship fund and sparked Europe’s biggest investment scandal for a decade. His decline had begun by taking high risk gambles with investments in small, private companies with the potential for exponential growth. Investments were made in dozens of technology and science start-ups. Limited due diligence was done.
There were several contributing factors to Neil Woodfords downfall. First, his increasingly risky investment decisions and his self-belief that his choices were good (alongside his fame and the belief by others that his decisions were good). Secondly his lack of due diligence. Thirdly the growth of stealth research and a lack of knowledge around the biotech field.
It is noteworthy that he owned a third of Prothena, a US biomedical company. Valued at $1.5bn the share price nosedived by 70% after the failure of a major clinical trial. Biotechnology is suffering from a reproducibility crisis, particularly in the field of neuroscience, which is the area that Prothena is active in. The reproducibility crisis describes the belief that there is a large and growing proportion of studies published across scientific disciplines that are unreliable due to declining quality, reduced research integrity and publication practices.
Stealth research describes the reporting of biomedical innovations outside of peer‐reviewed literature, using aggressive corporate announcements, and mass media hype. Research has shown that most high‐value start‐ups in healthcare have a limited or non‐existent scientist participation or evidence-based decision making.
Due diligence is fundamental to decision making for investors in biotech. Investors may go on ‘gut feelings’ about the product or the inventor but is this the best way to make an important decision in a complex area with known failings? Clinical decision making uses an evidence-based approach and there would be much to gain for investors utilising the same methodology.
QED Biomedical Ltd. understands that investors may rely on expert opinion or published evidence, but both are open to spin and publication bias, which can hide the true evidence base. We offer an unbiased, evidence-based approach to decision making. Our methodology identifies the strengths, weaknesses and evidence gaps in any new technology, Evidence based decision making provides a significant tool for reducing non-profitable investments and improving success. Systematic review is used routinely in clinical medicine to gather the best evidence on clinical trials. This process is not yet carried out at the preclinical level which likely contributes to the high failure rate of new innovations reaching the clinic.
QED Biomedical Ltd. uses systematic review or tailored evaluations to help investors make more successful investment decisions. We produce top level summaries of the available evidence or evidence gaps, in an accessible language. In addition, we indicate if the evidence is high or low risk. The summaries are backed by a detailed evidence base and transparent methodology. We aim to strengthen your decision-making processes and add value to your portfolios.
In summary evidence-based decision-making improved investment decisions by:
o Avoiding opinion and research spin
o Identifying the best technologies based on the evidence of efficacy and reproducibility to reduce risk
o Identifying the evidence gaps or uncertainty which increase risk
o Selecting the best technologies using evidence-based risk
Enhancing investments by identifying the evidence in additional disease areas for repurposing.
Organoid models are not clearly reported nor validated
We have previously reported on the strengths and limitations of using organoids for drug discovery. Using validation tools developed in house we have shown the importance of assessing whether or not authors have demonstrated that models recapitulate a given tissue and disease.
We applied our outcome validation tool to alert investors to some of the common limitations of these patient -derived models. A recent paper claimed that patient-derived-organoids can predict the response of patients to chemoradiation. Whilst the authors followed the STAR methods to ensure rigor and robustness we identified several areas of weakness.
1) The reference standard was heterogeneous. The diagnostic accuracy calculations were based on a response to any of 3 treatments (irradiation, 5-Fluorouracil, or Irinotecan) rather than to one of three treatments. It would have been more informative to have predicated which individual treatment a patient might respond to. In addition, a heterogeneous reference standard could lead to inflated accuracy calculations, it is easier to record a response when there are 3 chances of this occurring. Nonetheless this is a difficult area to conduct research since it is not easy to obtain patient samples.
2) The patient population was heterogenous. Useful information about the diagnostic accuracy is also lost by combining patients of different gender and different cancer grades.
3) The organoids do not receive the same treatment regimens as the patients. Given that the patients received multiple treatments and the reference standard is heterogeneous, the order that the treatments are given to the patients should at least be replicated in the model. It is unclear from the methods if this was done. Given that organoids do not maintain in culture for more than a month without passaging it likely that the experiment was carried out in parallel rather than in a serial manner.
4) The definition of ‘response to treatment’ is unclear. An organoid culture was considered to have responded to treatment if the ratio of the organoid’s size at day 24 versus day 0 was 36.4%. This information was not reported in the methods section, but in the results. A justification for this threshold was not clearly provided.
5) Poor reporting of methods to determine organoid size. It was clear from the methods that organoid size was measured every 3 days using Image analysis software, but important details are missing. What was measured? The diameter? Widest point? Circumference? How many fields were measured and at what depth? What was the variance of these measurements? Who measured size and were they blinded to treatment? This detail is crucial as anyone who has grown organoids from a patient sample will know that organoid size is very heterogenous. At day 24 what was the size of control organoids? Organoids do not maintain indefinitely in culture and will start to die if not passaged, how is this normal death in culture controlled for and not assumed to be ‘response to treatment’.
Whilst patient- derived models are to be encouraged for preclinical research it should not be assumed that they are without limitation. The limitations should be made clear to potential investors. The use of evidence-based decision-making aims to bring an unbiased approach to model validation to ensure all parties understand the positive and negatives of a model system. This approach helps to clearly illustrate evidence gaps which are difficult to spot without the use of validation tools. This approach helps to identify future research priorities or aid like-minded researches to agree guidelines/ recommendations for publication or future research. The aim of our validation tools is to improve clinical translation and reproducibility.
Tackling the reproducibility crisis
A recent paper by Menke et. al., 2020, studied the quality of scientific reporting in 250 biomedical papers. This work adds to that of Macleod et, al., 2019. Menke et. al., analysed the methodology sections of publications to determine how well the scientists reported on:
· Randomisation and blinding
· Sample size analysis
· Sex as a biological variable
· Cell line authentication and contamination
· Biological resources (antibody, organism, cell line, plasmid, oligonucleotide, software)
This work was performed with an automated text mining tool. To ensure reproducibility we would consider that sample size analysis and the clear methodological reporting of the resources are fundamental (the other reporting items may vary according to the study design). The authors reported that sample size calculations were only performed in 2-10% of papers. Information around antibodies (11.6% to 43.3%) and software tools (42.1% to 86.7%) were reported at relatively high levels in comparison to the type of organism (21.1% to 22.0%) or cell line (36.8% to 39.3%). This work is in alignment with QED Biomedical’s research findings and highlights the issue that in the preclinical field although there is a lot of effort to improve reporting there is a lack of risk of bias tools with which to assess 'real world evidence': if the ideal is not reported how do you evaluate that evidence? Although the authors concluded that reporting guidelines were having an impact and that scientific reporting was improving, we would argue that the quality of reporting is just one factor impacting the reproducibility crisis. Taking a queue from clinical research, authors are required to submit a protocol before commencing their research and its subsequent publication. This process helps to identify or eliminate negative publication and selective reporting. Negative publication is the omission of any results that were initially planned but did not work. It is sometimes used to describe the complete lack of publication due to the lack of any positive results. Selective publication is a deviation from the methods without justification. All biases must be assessed in order to identify good quality research, avoid poor reproducibility and research waste. The application of evidence-based decision making, and systematic review can help identify both these biases. The training of biomedical scientists in systematic review, protocol submission and risk of bias assessment would enhance the initiatives to improve reporting and create a better understanding of the importance of high publication standards.
Improve your Target selection
QED Biomedical Ltd can help you select the best target.
We can search existing published evidence to identify efficacy, efficiency and safety data. We can map the evidence from the competition. Enhance your portfolio and reduce investment risks. You decide whether the evidence base is for:
- Human models, or animal or specify organism
- In vivo models or in vitro models
- Type of organ.
- Type of disease
- Type of biological function or outcome.
QED Biomedical Ltd provides a unique Evidence-Based Decision-making service for preclinical stage and biomedical research.
We use systematic review, tailored evaluations and mapping reviews to provide unbiased evidence bases for your decision-making requirements. We can provide evidence for:
- Enhancing your target portfolio by mapping the current evidence base
- Drug or target repurposing
- Selection of targets/ biomedical innovations
- Evaluate the evidence and identify gaps for competitor targets/drugs
New flyer for QED Biomedical Ltd
QED Biomedical blog posted by Drug Target Review
What are the pros and cons of using organoids?
Dr Shona Lang investigates the advantages and disadvantages of using organoids within R&D, highlighting the most important questions to ask before using these models.
Organoids are three-dimensional (3D) cell cultures which mimic tissue architecture. Their growing importance in various fields of research saw them named ‘Method of the Year 2017’1 and their use in drug testing has seen a large increase in the market for 3D cell cultures.2 Importantly, these models have revealed the ability of the cell to differentiate and self-organise. Researchers have found organoids relatively easy to grow and they can produce stunning images; they can be observed using molecular, cellular and imaging techniques.
However, like any other model derived from patient tissues, they must be clearly validated. So, if you are thinking of investing in an organoid model or you are trying to select the best model from several possibilities, what questions should be asked? There are clear advantages in using organoids, but often the negatives can be overlooked or are simply unpublished.
Not an average culture
Organoids are an upgrade from the traditional primary cultures grown as a monolayer in a tissue culture flask. The difference for organoids is that cells are grown within a basement membrane gel and develop into 3D shapes; primarily into hollow or budding spheres. The cells seeded into this type of culture system can be selected (eg, stem cells) or unselected to reflect the cellular heterogeneity of the patient.
Although organoids are now widely used, they are also under-investigated and often poorly validated. It is not known what effect the basement membrane gel has on cellular behaviour – does it support the natural differentiation of the cell or does it reprogramme growth in an undetermined way? This must be kept in mind if you assume that your model recreates the tissue of interest.
Organoids offer superior morphology if you are studying a glandular tissue, but they are not appropriate for studying stratified tissues, such as skin. Originally organoid cultures were grown to investigate normal cellular differentiation in the prostate3 and breast.4 Their use as a tumour model is debatable as the influence of the basement membrane gel is not currently understood. One of the classical definitions of tumour growth is anchorage independence, yet organoids are anchorage dependent due to their adherence to the basement membrane proteins in the gels. In addition, tumour cells known to grow as solid masses in vivo can grow as hollow spheres5 in the organoid system, so they do not always recreate patient tissue architecture. Therefore, it is important to carefully consider the morphology of the disease you are studying.
Purity and removing contaminants
Cultures derived from tissues will contain a mix of cell types, for example, epithelial or stromal. These can be contaminated by neighbouring tissues dependent on the sampling technique, eg, prostate tissue sampled by transurethral resection could contain urethral tissue. Tumour tissues can also be contaminated by normal cells. A study of in vitro and in vivo models for precision medicine indicated that between 25-95 percent of ‘tumour’ organoids (across a range of tissues) were normal cells.6 When using organoids, it is imperative to confirm the absence of all potential contaminants.
Heterogeneity of organoids
Although heterogeneity is frequently reported as a positive aspect of spheroid cultures, it must also be recognised as a negative. If you are testing a drug and want to know its effect on a heterogeneous culture of cells, then this is a positive characteristic. However, when investigating the action mechanisms of a drug and which cells are targeted, using a heterogeneous mix of cells will not yield answers. Evidence suggests that you cannot rely on the in situ genotype to reflect the in vitro genotype.7 Since cells are usually seeded at high density into an organoid culture there is also a risk that a single organoid may not be clonal (derived from a single cell). If you are using a heterogeneous mix of cells it is critical to check whether the ratio and type of cells (tumour, normal, different tumour clones) reflects the patient or whether it is better to use multiple samples or organoid cultures per patient.
Reproducibility and clonal drift
Heterogeneous cultures contain many different clones, which means that as the culture grows certain clones will dominate while others will die out. Therefore, the heterogeneity will be different from one pass to the next and experimental reproducibility will suffer.8 Whilst a mutation may be detectable in a culture of organoids this does not mean that 100 percent of the cells within or between organoids are tumourigenic and this will have consequences for drug testing. If you identify an organoid with a mutation of interest, you need to confirm what proportion of cells have that mutation. If 10 percent of the cells contain the mutation, the response to the drug will be dominated by the wild type cells. This lack of validation can lead to false negatives and poor reproducibility. Accordingly, understanding the genotype and phenotype of your culture (including the proportion of cells) at every passage to ensure your model is fit for application to precision medicine is important.
Currently there is a growth in omics research – the study of large datasets derived from genomic and proteomic research – that has enabled an understanding of the complete molecular profile of the patient and helped establish personalised medicine. Omics research has also led to the establishment of numerous databases to store information, which can be used to source and categorise research material, eg, biobanks and cancer model directories. However, the validity of the data in these databases is often unclear and currently there are calls to improve research standards9 and rigour10 in this area. It is important to remember that the identification of a gene mutation within a model does not confirm that the mutation is present in all cells or that it will last over a period in culture.
Currently the validation of pre-clinical models is based on the evidence reported in scientific publications or the opinion of experts; there are biases in both.11 With the advance of the Open Science movement it is well known that scientific publications face problems with publication bias.11 Expert opinion is open to bias, conflicts of interest and spin.12 Evidence-based decision making has been used for several years in medicine to avoid partiality.13 This process relies on developing a research question and protocol to find and evaluate all available evidence. It is advantageous because not only does it evaluate evidence in terms of quality, but it also highlights the evidence gaps. Advances in the application of evidence-based decision making for biomedical science allows investors and researchers to find the best validated models and ensure they recapitulate the disease of interest.14 Decision-making tools allow researchers to identify which validation criteria have been met and those which have not (tissue of origin proven, cell lineage proven, contaminants absent, concordance of histology and genotype, concordance of response to drug treatment). This ensures that both useful evidence and evidence gaps can be identified.15 The process aids the selection of the best models and can highlight the need for further validation experiments.
In conclusion, evidence-based decision-making tools improve translation into the clinic by reminding investors of the important questions to ask. It is vital to always be wary of models with minimal validation.
Hierachical cancer stem cell model challenged in glioblastoma
The idea that cancer stem cells (CSCs) exist as a defined cellular entity, driving tumour growth and resistances is challenged by a team at the Luxembourg Institute of Health working in glioblastoma; a form of brain cancer.
The researchers found that cellular heterogeneity-which is a hallmark of glioblastoma and makes it particularly difficult to treat- was due to the inherent plasticity of tumour cells rather than multipotent CSCs. They describe their findings in the journal Nature Communications and caution against therapies targeting CSC cell surface markers because all cancer cells have the capacity to reconstitute tumour heterogeneity.
These findings bring into question the validity of numerous studies describing brain CSCs and the rationale of some biotech companies pursuing a strategy of targeting CSCs using immunotherapy. The peer review system is meant to examine papers on potential shortcoming, yet poor study design is often left unchallenged. For example, numerous studies reporting the existence of CSCs failed to test the potency of marker negative cells, applied different growth conditions on marker positive and negative subpopulations thereby introducing bias.
There are a number of initiatives to tackle the reproducibility crisis in biomedicine, such as replicating key landmark studies before embarking on clinical trials. This is both costly and time consuming. Another approach is to use systematic review and evidence-based decision making to verify evidence before embarking on an investment or new direction. QED Biomedical specialises in systematic reviews and tailored evaluations in biomedicine analysing data for validity as well as reproducibility. We can help you make an informed decision.
Patient derived xenograft models: an emerging platform for translational cancer research?
One of the largest studies to date on new drug approvals reported that around 3% of cancer drugs make it to the market. The probability of success improves significantly, to just over 10%, if biomarkers are used to select patients, but this is still a dismal return for what is a considerable investment by pharma and biotech companies. This points to a failure of existing cancer models to reliably predict anticancer activity in the clinic.
Patient-derived xenografts, otherwise known as PDX models, have been cited extensively in the scientific literature as better predictors of response, compared to the more traditional cancer cell line models because they retain the cellular heterogeneity, architecture and molecular characteristic of the original cancer. With the demand for more personalized medicine, the market for PDX models has grown significantly, with pharmaceutical and biotech companies the biggest users. So how reliable are PDX models? Unfortunately, much of what is claimed does not stand up to scrutiny.
PDX models are based on the direct implantation of fresh cancer specimens from individual patients into immunodeficient mice. Once established, in individual mice, each tumour is subsequently expanded into further mice by a process known as serial transplantation. The challenge is that each copy (or xenograft) should reflect the original human cancer and should not deviate with continuous serial transplantation. To assess how well PDX models mimic the disease in man we carried out a systematic review, recently published in PeerJ . We limited the review to the four most common cancers: breast, prostate, colon and lung and used a checklist of strict criteria to determine model validity. What we found was that around half of all breast and prostate studies, that derived xenografts, did not mirror the donor cancer as claimed by the authors. Lung and colon fared somewhat better with 28% of lung and 16% of colon studies judged high risk. For example, lack of concordant gene mutations, discordant clustering from gene expression studies or failure to express tissue-specific markers in line with the donor cancer, were some of the findings reported in this review.
Overall, we categorized most studies as unclear because one or more validation conditions were not reported, or researchers failed to provide data for a proportion of their models. The most common reasons were, failure to demonstrate tissue of origin, response to standard of care agents and to exclude development of lymphoma (a common phenomenon in the PDX field).
Whether or not PDX models are more clinically predictive is yet to be determined, at least for the four most common cancers. Our ability to judge which discoveries/innovations will be beneficial is crucial. Until we adopt a more formal, unbiased assessment of biomedical research findings, with strict guidelines to ensure transparency, we are unlikely to improve the low rate of clinical trial success.
‘The night is dark and full of terrors’:
or the despair of reading a scientific publication
The famous quote from Game of Thrones perfectly describes my emotions when reading a modern scientific paper. This is particularly the case if I try to dig deep and evaluate whether the evidence is valid or reproducible.
I recently read a paper (published in Science) describing the creation of patient derived cellular models from cancer tissue. My aim was to assess how well the authors:
a) recreated gastrointestinal cancer tissues (cancer markers, tissue markers, morphology and treatment response)
b) proved their claims were reproducible
I consider myself a seasoned veteran of reading and writing scientific reports, however I found myself struggling to read the paper and forcing myself to identify all the relevant information. I complained loudly to colleagues about the difficulty of my task and the lengthy time it was taking.
The paper, selected at random, was around 7 pages long and had 43 authors from ten research centres. The main text contained four figures; but each contained up to seven sub-figures. There was no methods section; the trend in high impact journals is to relegate this to the supplementary materials. The supplementary materials were a whopping 47 pages including 13 additional figures (each with multiple sub-figures) and eight, large Excel data sheets. There were multiple complex methodologies including; advanced tissue culture, histology, DNA sequencing and mutational analysis. You can see the scale of the task. Maybe you can understand why I made the comparison with the futility of warring against the army of the undead.
It took me a determined day to read through all the data and find all the evidence relevant to my evaluation (or confirm the lack of it).
After completing my analysis (posted in previous blogs and tweets), I found that
a) The authors had selective reporting. They didn’t report all the evidence for all their models. It wasn’t even clear if they were only interested in reporting about particular models. This leaves you with the impression that the authors only report the results they want you to see.
b) The models were not well validated (morphology was not always consistent with the patient, there was limited evidence for colon or tumour markers, mutations were not always identical to the patient)
c) Response to treatment was very opaque. To clarify, the patient response was only mentioned in the text and it was not always clear what that response was. The model’s response to treatment was presented as a relative effect – so the absolute results could not be judged. In addition, there were no statistical analyses to prove a treatment effect nor that it was concordant with the patient. The reader had to trust the stated text and could not confirm it from clearly reported quantitative results.
d) Sadly (but not unusually) the data were frequently based on three replicates; this is not robust since the likelihood of the result repeating will be very small based on this sample size. Ideally the authors should have calculated a samples size, based on the effect size seen in the patients.
Overall, I was left with the impression of selective reporting, opacity of the absolute data and too much relative data or categorical data. I don’t think a statistician would have approved this data for publication; assuming they could read the papers’ complexity.
Basic standards of statistical reporting and the ease of reading appear to have been replaced by increasing data volume and the inclusion of many complex methodologies; this cannot be good for the scientific community. We need to remind ourselves of the reason that scientists publish: to pass on knowledge. Scientific knowledge must be reported in a manner that allows it to be understood, repeated and clearly evaluated by other scientists.
Other researchers have also suggested that scientific papers are becoming too dense, too complex and too hard to read. I felt that the current research would be more useful to the community if the evidence had been reported in multiple, shorter articles which focus on one question and report the results clearly using absolute values. It is the clarity of the evidence that is most disturbing, anyone who may want to invest in the models should be able to evaluate reproducibility and effect size. Investors should not assume the authors conclusions are correct without this evidence. QED Biomedical uses in house tools to evaluate the validity of new preclinical models and can assess the likelihood that treatment responses are reproducible.
Poor reporting of statistical analysis in preclinical organoid models for drug testing
Our recent evaluation of a scientific publication comparing the response of preclinical models to anticancer treatment with the reponse in patients, showed a shocking lack of statistical reporting and rigour.
Models were tested in triplicate for a response to cetuximab. No sample size calculation was performed. It is highly unlikely that three replicates was sufficient to detect any significant or reproducible effects of cetuximab on organoid growth. Statistical guidance has suggested that a sample size of at least 10 replicates should be applied to preclinical models. The lack of sample size calculations is one of the greatest causes of false findings, yet it remains one of the most overlooked reporting criteria in biomedical science.
To compound the issue reported above, the authors reported treatment effect as a relative value, which means the absolute values for the response to the drug treatment or to controls could not be evaluated. In addition, the authors provided no statistical analysis to prove that the drug had an effect (in comparison to the control) nor did they provide statistical evidence that the organoids replicated the patients' responses. The reader was expected to take it on good faith from narrative descriptions.
Why should you care? Organoids have been proclaimed as the 'method of the year 'and are part of a growing industry for preclinical testing of drugs. There are supporting industries in bioinformatics helping researchers find the best models for their needs. We must ensure that no matter how complex the chosen model, the basic tests of reproducibility should be reported alongside the discoveries and the limitations should be acknowledged. This is what will ensure that biomedical discoveries translate efficiently to the clinic. Precision medicine needs to be precise.
QED Biomedical is helping to tackle the reproducibility crisis. We can help you design your experiments better or we can help you identify the strengths and weaknesses in existing research.
QED Biomedical blog posted by CATAPULT Medicines Discovery
New methods to improve the translation of biomedical science towards clinical trial success
Moving forward - implementing systematic review and identifying reliable evidence
An article from our series: Views from the Discovery Nation
It is now over ten years since John Ioannidis published his essay on ‘Why most published findings are false’. This paper presented statistical arguments to remind the research community that a finding is less likely to be true if: 1) study size is small, 2) effect size is small, 3) a protocol is not pre-determined and not adhered to, 4) it is derived from a single research team. Methods to overcome this issue included establishing the correct study size, reproducing the results by multiple teams and improving the reporting standards of scientific journals.
So, have any of these suggestions been taken forward? There is an increase in the promotion of open science, a movement which supports increased rigour, accountability and reproducibility of research. In addition, there are increased requirements by some scientific journals to improve reporting standards. Whether or not these recommendations are followed or adhered to remains to be established. Evidence from a systematic review we conducted on the asymmetric inheritance of cellular organelles highlights the problems in basic science reporting and study design.
Of 31 studies (published between 1991 and 2015), not one performed calculations to determine sample size, 16% did not report technical repeats and 81% did not report intra-assay repeats. We need to educate our future scientists on study design and impose stricter publication criteria.
Impartial and robust evaluations of biomedical science are required to determine which new biomedical discoveries will be clinically predictive. We should be concerned by the lack of reproducibility in biomedical science because it is a major contributing factor in the low rate of clinical trial success (only 9.6% of phase 1 studies reach approval stage). Our ability to judge which discoveries will have real-life effects is crucial.
What can be learned from clinical research?
A lot can be learnt from clinical research. The publication of clinical research must follow strict reporting guidelines to ensure the transparency of research and decision making is based on unbiased evidence gathered by systematic review. Systematic reviews provide a methodology to identify all research evidence relating to a given research question.
Importantly systematic reviews are unbiased (not opinion-based) and are carried out according to strict guidelines to ensure the clarity of the results and their reproducibility.
They form the basis of all decision making in clinical research. Importantly the evidence gathered by the review is judged to ascertain its quality. This allows the review to present results graded according to the ‘best evidence’. Determining the quality of the clinical evidence is judged primarily according to the study design (randomisation, blinding of participants and research personnel etc). There are many risk of bias tools available, each appropriate to a different study design. The GRADE seeks to incorporate the findings of a systematic review alongside study size, imprecision, consistency of evidence and indirectness; providing a clear summary of the strength of evidence to guide recommendations for decision making.
It makes sense for decision making in biomedical research to be judged in a similar open and unbiased manner. Astoundingly, the choice of which biomedical discovery is suitable for further investment is usually an opinion-based decision. Big steps have been made to introduce systematic review to preclinical research. There are reporting guidelines for animal studies and risk of bias tools. The quality of these studies is again based on study design.
At a basic bioscientific level one must argue that judging the quality of evidence should focus on more fundamental aspects of the research: 1) how valid is the chosen model (or how well does it recapitulate the human disease of interest), 2) how valid is the chosen marker (or how well does it identify the target?) 3) how reliable is the result. Pioneering work by Collins and Lang has introduced tools to perform such judgments. These tools aim to directly address the issues raised by John Ioannidis. They aim to highlight the strengths and gaps in a given research base.
Original blog can be found here https://md.catapult.org.uk/new-methods-to-improve-the-translation-of-biomedical-science-towards-clinical-trial-success/
Is your tumour model what you think it is?
Advancing a candidate drug from preclinical testing into clinical trials assumes that cancer models used in the laboratory have reproduced the clinical disease accurately. Our recent review shows that this assumption is not accurate. We looked at how well the authors of scientific publications validated laboratory models grown from patient tumours. More than half of the models selected for potential inclusion in our review had no validation methods at all; the authors assumed the model reflected the patient. This is a dangerous assumption; tissue samples can have contaminating cells which will outgrow the cells of interest, these can be neighbouring tissues, cells with other disease states or in the case of tumour tissues contamination can come from normal cells.
Of the studies that performed validation studies more than half did not confirm: 1) the tissue of origin 2) the absence of contaminating cells 3) or that the model looked like the tumor growing in the patient (histology). Just over half of all studies (57%) confirmed that the models were derived from tumour cells (and not normal cells). It is crucial that we clearly understand what the laboratory model is and not make assumptions. At QED Biomedical Ltd we aim to present transparent reports of how well human models are validated. This allows improved judgements of whether the model is appropriate for drug testing and likely to be predictive of the clinical patient. Make your decision on evidence and not assumptions or opinions.
Collins AT, Lang SH. 2018. A systematic review of the validity of patient derived xenograft (PDX) models: the implications for translational research and personalised medicine. PeerJ 6:e5981 https://doi.org/10.7717/peerj.5981