The U.S. News & World Report (USNWR) Best Hospitals Specialty Rankings is frequently used by hospitals to attract patients. In fact, 1 study found that USNWR rankings were used by hospitals in 61% of direct-to-consumer advertising.¹ The rankings are also used by millions of patients each year to guide where they choose to receive their care.^2,3 In the era of evidence-based care, it is important that these widely utilized rankings truly reflect the quality of care delivered by hospitals and specialty services. The USNWR has been criticized for its reliance on reputation to rank hospitals and has consequently changed its methodology to include objective metrics.^1,4

Metrics used in ranking the 12 data-driven specialties include outcomes (37.5%), patient experience (5%), care-related factors (30%) and expert opinion (27.5%).³ Expert opinion is based on physician survey responses, and care-related factors include but are not limited to nurse staffing ratios and the variety of patient services/technologies offered by hospitals. Patient experience data are obtained from the HCAHPS (Hospital Consumer Assessment of Healthcare Providers and Systems) survey. Thirty percent of the outcomes data are dependent upon in-hospital mortality, with the remainder related to the number of patients discharged to home after admission.³

While it seems appropriate that an evidence-based measure of specialty performance is, at least in part, based on patient mortality while under their care, there is evidence to suggest that there are flaws in the underlying methodology used to attribute patient deaths to each specialty. This, in turn, impacts the accuracy of the rankings. For example, the death of a patient may be assigned to urology, yet the specialty caring for the patient was cardiology. This is because the USNWR data are based on administrative insurance data using Medicare Severity Diagnosis Related Groups (MS-DRGs). USNWR uses an independently developed mapping schema to assign MS-DRGs to each specialty using closed insurance claims; therefore, a death assigned to a particular specialty through an MS-DRG counts as a death for that specialty regardless of whether that specialty was ever involved in the patient’s care.^1,2

Given the potential misattribution of care that may be reflected in the rankings, we performed a retrospective analysis at our institution examining the rate of discordance between assigned care and actual specialty care received for urological patients. All inpatient deaths at our institution that were mapped to urology using the U.S. News schema from 2013 to 2017 were reviewed. We identified the cause of death via chart review and confirmed if the patient was cared for on the primary urology service or seen as a consultation during their admission. Author consensus was then used to determine if the mortality was related to a urological surgery or primary diagnosis.²

Table. Summary of patient deaths attributed to urology

CC, complication or comorbidity. MCC, major complication or comorbidity.

Nineteen patients were mapped to urology. Only 3 of these 19 patients (16%) were cared for primarily by urology. The remaining 16 patients (84%) were on a medical service. Of the 16 patients on a medical service, urology was only consulted on 2 patients (13%). Of the 19 patients mapped to urology, 10 (53%) were already on palliative care where death was the expected outcome. Overall, only 2 of the 19 (11%) deaths were potentially associated with urological care. The Table provides a detailed summary of the deaths and MS-DRGs attributed to urological care.²

We found that there is a significant discordance in the 30-day inpatient mortalities mapped to urology versus the actual care received at our urban, tertiary-care institution. Only 16% of the deaths assigned to urology were actually cared for by a urologist, and this number does not change when including patients seen as consultation (11%). While our data are limited given the small sample size and data from only a single institution, the sensitive nature of hospital mortality data largely precludes a multi-institutional analysis. Furthermore, to validate our findings, an identical analysis was also performed for otorhinolaryngology, which is another surgical specialty ranked by USNWR. This revealed nearly identical findings of a high discordance between assigned care and actual specialty care, suggesting this issue is not isolated to a single specialty.

In-hospital mortality is one of many factors included in the USNWR specialty rankings. Nevertheless, it is the largest modifiable factor that is in the control of the individual specialty, rather than fixed at the hospital level, like nursing ratios, or dependent upon external expert opinions. Our data highlight several important points when considering the use of USNWR rankings. First, the systemic inaccuracies we found in attribution of care diminish the validity of the USNWR rankings to reflect the care actually delivered. As a result, specialists should not rely on these rankings to benchmark or measure their outcomes against their peers, and similarly, patients should not assume that a high-ranking specialist will provide high-quality care simply because they are ranked high. An ideal ranking system will require higher fidelity source data other than closed Medicare claims, which likely will require practice-level reporting via a clinical data registry. This is still relatively new in urology but expanding with AUA initiatives like AQUA (the AUA Quality Registry).⁵