Every year, children’s hospitals nervously await results of the U.S. News and World Report (USNWR) rankings, realizing that overall and specialty specific rankings may dictate how colleagues and patients view their institutions. Top programs are heartily congratulated, and programs who rank lower than expected feel defeated. Enormous amounts of time and resources are spent within institutions across the United States to collect USNWR data, and hospital initiatives are prioritized based on how closely they align with anticipated USNWR metrics.

Goals of USNWR Rankings

The USNWR Best Hospital rankings aim to provide publicly available information for patients with complex or high risk conditions to facilitate informed choices based on hospital quality.¹ However, the rankings are often interpreted as a directive about where patients who want quality care should go for any condition, regardless of risk or severity. In reality, the ranking systems are highly imperfect, may be irrelevant for common, lower risk conditions and may be outweighed by myriad other factors including patient travel distance and culture. Although it is implied that USNWR rankings improve health care quality, it is unclear whether ranking programs and hospitals using the present scoring systems actually promotes better care.

Pediatric Urology Ranking Methodology

As pediatric urologists we are most familiar with the USNWR methodology for our specialty. The table shows relative contributions of different hospital and program attributes to calculating the USNWR score. Close examination of the pediatric urology ranking system illustrates concerns about methodology of objective data collection. Furthermore, the reputation score is completely subjective.

As outlined in our recent editorial, “Ability to Prevent Surgical Complications” is weighted heavily in the USNWR scoring system yet relies solely on administrative data and self-report.² Pohl and colleagues also recently showed that outcomes requested by USNWR do not accurately reflect published hypospadias complication rates,³ yet hypospadias outcomes were used to calculate USNWR scores. Additional concerns about the current USNWR ranking methodology include lack of clear criteria determining which procedures should be included in outcomes assessment, a focus on surgical volume when surgery for certain conditions (eg vesicoureteral reflux) may not necessarily represent “best care,” a reliance on factors outside of the control of programs directly (eg nursing ratios) and reliance on the highly subjective reputation score. Improvements in outcomes assessment methodology are made each year, but it is not clear that USNWR scores accurately differentiate between the quality offered at different hospitals for pediatric urological procedures.

The USNWR ranking methodology has resulted in a situation where the “Top 10” programs are relatively geographically concentrated (see figure) such that most patients would incur significant travel burden to access care at one of these hospitals. Although access is far from perfect, excellent pediatric urological care is available in many areas not in the figure, and the travel burden implicated by the “Top 10” is likely not beneficial for most patients. In our own “Top 10” practice, we frequently care for patients who do not live near Chicago. Many live much closer to another high quality pediatric urology center and would have received care of identical quality without traveling. We also care for patients who went elsewhere for surgical reconstruction due to program reputation or ranking; living far from your reconstructive surgical team can create postoperative care challenges that outweigh a small technical benefit, even if present.

Figure. Geographic distribution of top 10 pediatric urology programs for 2020.

Adult Urology Ranking Methodology

As with pediatric urology, methodologic concerns related to reputation and outcomes scoring also exist for adult urology. A recent study showed that USNWR reputations scores are associated with presence of an active Twitter account.⁴ It is no secret why glossy mailers and reminders to sign up for Doximity (the voting platform for USNWR reputation scores) all come around the time of USNWR voting. Reputation scores likely reflect marketing ability and resources along with quality of care. Concerns about inappropriate attribution of poor patient outcomes to subspecialty care also exist, particularly because adult USNWR rankings rely heavily on administrative data. A 2020 study demonstrated that a minority of deaths attributed to urology and otolaryngology over a 5-year period actually occurred among patients on those services.⁵

Best Use of Limited Resources?

Through the upheaval of the COVID-19 pandemic, the year 2020 provided an opportunity for reflection and change. We have examined and redesigned behaviors and routines once assumed fixed. Financial and personnel resources have been stressed and are still unstable for many. Children’s hospitals are facing unprecedented challenges, so important questions need to be asked. Should scarce resources be devoted to collecting large amounts of USNWR data when there is no evidence that rankings truly reflect differences in care? Are we misleading patients such that they are traveling long distances to higher ranked programs when local care is equally efficacious? At a time when American hospitals should be joining forces to provide coordinated improvement in children’s health care, should we be spending health care dollars on marketing materials designed to create competition? This is not to say that competition cannot be an important driver to improve care. However, the type of competition that is created by USNWR is not one that many would consider “healthy.” Therefore, shouldn’t we be spending more time and effort on determining how institutions can work together for the collective benefit for patient care?

Final Thoughts

In their current state, the USNWR rankings do not necessarily benefit patient care. They create an environment that generally encourages competition rather than collaboration. We strongly encourage all physicians to closely examine how USNWR rankings impact patient care in their own specialty. If it is determined that current methodology is flawed, as it is in pediatric urology, then discuss potential solutions with your colleagues. Solutions could include a coordinated effort to overhaul USNWR methodology or a decision to collectively opt out of USNWR. Until complete methodologic redesign or opting out occurs, hospitals will be under tremendous pressure to use precious resources to facilitate USNWR rankings. Now is the time to examine and change the narrative about what role the USNWR should play in determining public perceptions and where patients and families seek medical care.