Discriminating between top urology residency applicants is challenging but is vital to the continued success of individual programs and the field of urology. The resident selection process is complex and varies among the programs. Programs generally use a variety of methods to choose candidates for interviews and establish the rank list. While factors such as clerkship grades, United States Medical Licensing Examination (USMLE) scores, class rank, research experience, and visa status continue to play an important role in the resident selection process, most urology program directors agree that letters of recommendation are the most important factor.¹

In the field of urology, letter writers tend to use narrative letters of recommendation (NLORs) which do not have a uniform structure. Previous studies have found the commonly used NLORs to be highly flattering and ambiguous, contain gender bias, and have very low interobserver reliability in their interpretations.^2-6 In order to overcome the shortcomings of NLORs, standardized letters of recommendation (SLORs) were introduced. Such letters are easier to write and interpret compared to the NLORs. The SLOR was initially started by emergency medicine in 1997, and followed by other specialties such as otolaryngology in 2012, plastic surgery in 2012, dermatology in 2014, and orthopedic surgery in 2017.^3,7-10

The Current State of SLORs in Urology

SLORs in urology were first launched for the 2020-2021 residency match cycle. We have previously investigated the use of SLORs in the Urology Residency Match.¹¹ We found 2 main formats of SLOR among 2020-2021 Urology Match applications. Format 1 was originally proposed by Dr David Penson, and format 2 was distributed by the Society of Academic Urologists (SAU; see Figure). In that study, we observed some meaningful correlations between domain ratings and application metrics such as USMLE Step 1 and Step 2 Clinical Knowledge scores, and percentage of Honors in core clinical clerkships. A recent study on linguistic analysis of urology NLOR demonstrated that letters written for match-successful applicants had more power words, which was also the case for the NLORs written for male urology applicants compared to female urology applicants.⁶ The authors found an implicit gender bias in urology NLORs. However, in our study of urology SLORs, no statistically significant differences were found between female and male applicants in terms of domain ratings.¹¹ This is of utmost importance especially in a field like urology where gender imbalance is a real concern.

Figure. Domains of the 2 standardized letters of recommendation (A and B) used in urology residency applications. Reprinted with permission from Nabavizadeh B et al, Urology. 2021;158:18-25.¹¹

We also looked for other potential biases in the report of current formats of SLOR. No significant differences were found in domain ratings with regard to applicants’ race, depth of interaction between letter writers and applicants, and gender of letter writers. However, we found a marked ceiling effect where most applicants were rated among top tiers in both formats of urology SLORs, which may have several underpinnings such as using Likert-like scales in the current SLOR formats and limited number of domains, which may make it difficult to capture the distinguishing features of highly competitive urology applicants. In another study, we evaluated the differences in applicant characteristics and SLOR domain ratings based on the match results. We found matched applicants were more likely to have higher domain ratings compared to unmatched applicants.¹² Furthermore, several key domains of format 1 SLOR (ie, “potential as a urology resident,” “potential as an academic urology attending,” “performance as a subintern,” and “likely rank position”) were associated with successful match into urology.

Improving the Urology SLORs

The current urology SLORs are not without flaws and mainly suffer from grade inflation as discussed above. After their introduction, urology SLORs have not undergone revisions, and no formal evaluation of such letters has been performed to gauge satisfaction with their use among urologists. Urology still remains one of the most competitive specialties to match into.¹¹ Additionally, given the transition of USMLE Step 1 score reporting to pass/fail, there is an emerging need for a reliable assessment tool that can help distinguish between highly qualified urology applicants. In an effort to improve the current formats and create an optimal urology-specific SLOR, we are conducting a study supported by the SAU, where we have designed and validated a survey tool to seek recommendations from urology faculty across the country. We will analyze their feedback and implement changes to SLOR format based on their feedback. In addition, in collaboration with SAU and the AUA, we will launch a secure website that urology faculties can utilize to generate electronic SLORs (eSLORs). After verification, any practicing urologist would be able to set up an account to enter content electronically and generate an eSLOR. An individual barcode will automatically be assigned to each eSLOR which can be used for authentication purposes. The final eSLOR can be uploaded to the Electronic Residency Application Service website. The website will also be capable of storing data that can be tracked over time and used for future research. Before widespread launch, a β version of the website will be activated for a small number of urology faculties to test the website function and provide feedback.