Figure. Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) approach for rating the certainty of evidence in a systematic review. PICO indicates Population, Intervention, Comparison, and Outcomes; RCT, randomized controlled trial.

Urologists often ask how different management approaches compare in terms of desirable and undesirable effects; randomized controlled trials (RCTs) are the study design that is likely to get us closest to the truth for this question. Ideally, important questions should be addressed by several RCTs conducted by different research teams that can then by summarized in a methodologically rigorous systematic review, such as those published in the Cochrane Library. These systematic reviews can inform individual patient decision-making and evidence-based clinical practice guidelines, as well as health policy. However, planning, funding, and executing RCTs is an arduous task, making such trials a precious and scarce good.¹ Every effort should therefore be made to assure that trials are of the highest quality.

From the perspective of a Cochrane author and editor, the trajectory of a high-quality trial starts with a registered protocol that outlines the pertinent details of study enrollment, conduct, and analyses. A well-planned peer- and patient-reviewed protocol can help safeguard against common risks of internal validity (bias) and ensure the research questions and outcomes are relevant.²

A registered protocol can do the following:

• Help reviewers and readers detect selective reporting. To date, too many urologic trials lack a registered, a priori protocol that allows readers assurance of whether what was planned and what was done closely align. In the absence of a registered protocol, there will always be concerns about selective reporting: whether the chosen analyses were data- or hypothesis-driven and whether the investigators are fully transparent about reporting all data.
• Include well-established methodological safeguards against bias. To date, too many trials forgo implementation of these measures for no good reason. A critical part of trial development includes assurances that the groups in which interventions are being compared are in fact comparable at baseline before the trial begins (randomization and allocation concealment), are treated and assessed the same way (blinding), are analyzed based on their assigned group regardless of whether they followed through with that treatment or not (intention to treat), and if most participants were included in the analysis (completeness of follow-up). For example, it should always be possible to blind the involved party determining the stone-freedom in a trial comparing extracorporeal shock wave lithotripsy vs percutaneous nephrolithotomy vs ureteroscopy, yet a recent Cochrane review only found 1 of 23 trials provided assurance that those determining treatment success at 3 months were in fact blinded.³
• Ensure both the question and outcomes being addressed are of actual importance to patients and clinicians. There is increasingly the expectation that patients be engaged as stakeholders at the planning stage. Peer and patient reviewers can help assure that outcomes are of direct patient importance rather than surrogate outcomes. For example, a trial of metastatic renal cell carcinoma may report on disease-specific survival rather than tumor response (which represents an indirect outcome).

After trial completion, robust reporting of results is as important as appropriate protocol development. While there is empirical evidence that the quality of reporting of urologic RCTs has improved, there is still much room for improvement.⁴

Some key points of RCT reporting are:

• Transparently reporting trial methods and results, according to the CONSORT (Consolidated Standards of Reporting Trials) criteria.⁵
• Reporting all results as fully transparent, full-text manuscript irrespective of whether the results are positive (supporting the research hypothesis) or negative (not supporting the research hypothesis).⁶ Doing so avoids the issues of publication bias, the phenomenon that positive trials are more likely to publish in journals with greater public visibility than negative trials for the same topic, thereby skewing any summary review toward positive findings unless all efforts are made to find all trials.
• Making trial raw data publicly available upon reasonable request. Several organizations, including the National Institutes of Health and the Open Trials Collaborative (OpenTrials.net), have mandated data sharing to promote transparency.⁷

Ultimately, Cochrane reviews and other similar high-quality systematic reviews not only summarize the entire body of evidence for a given clinical question, but also rate the certainty of evidence (CoE) for each outcome.⁸ This CoE rating, based on the GRADE (Grading of Recommendations, Assessment, Development, and Evaluation) approach, represents the most important intellectual contribution of a systematic review by qualifying how much trust we can place in each result, typically a pooled effect size estimate such as risk ratio or hazard ratio. To do so, trained methodologists assess the body of evidence in 5 categories:

1. Risk of bias (study limitations) of the included trials (as outlined above).
2. Indirectness: if the population, intervention, outcomes, or comparisons differ from those of interest.
3. Publication bias: omission of negative studies from the literature, which may result in overestimates of effect (Figure).
4. Inconsistency: whether results are substantially different across RCTs.
5. Imprecision: the width of the confidence interval and number of events related to the pooled effect size estimate (rather than individual trials).

While individual trials should ideally be adequately powered, it is one of the strengths of a systematic review that statistical power is increased when multiple small trials without a definitive result are pooled. These assessments then provide a GRADE CoE rating for each outcome of either high (no rating down), moderate (rated down once), low (rated down twice), or very low (rated down 3 times). This approach provides a much more nuanced assessment of a body of evidence than the hierarchy of evidence framework by the Oxford Center for Evidence-Based Medicine that misleadingly labels individual trials as level I evidence with little regard for their conduct and reporting.⁹

Finally, conducting a practice-changing RCT is not only a huge endeavor that can occupy the principal investigator for the span of their academic career, but also a tremendous privilege: participants volunteer their time and effort for little more in return than the promise of better evidence and higher quality of care for future generations. As such, investigators should keep this in mind and strive to meet the highest standards when it comes to designing, executing, and reporting trials.