Introduction

From virtual reality (VR) video games to computer-generated images (CGI) and animation in blockbuster movies, 3D visualization technology that has flourished in entertainment is now beginning to revolutionize medicine. In urology, 3D technology will impact surgical training, surgical preparation, patient education and research. Particularly, new advances in software now allow for 3D-printed anatomical models, virtual and augmented reality models, and computer-generated 3D computerized tomography (CT) with cinematic quality. Many of these revolutionary technologies are in their developmental stages, but improvements in rendering software and artificial intelligence may soon automate the process, opening possibilities for broad clinical, commercial and research applications. Below are some cutting-edge applications of 3D technology with urology at the forefront of medical progress.

3D Printing

In recent years, research regarding the feasibility and accuracy of 3D-printed anatomical models for surgical rehearsal, surgeon training and patient education has expanded exponentially. Ghazi et al’s 2020 review of the role of 3D printing in robotic urological procedures demonstrates the feasibility and value of 3D printing technology in partial nephrectomy and nerve sparing prostatectomy.¹ Specifically, there are reduced ischemia times, fewer positive margins, less blood loss, shorter hospital stays and fewer postoperative complications for robotic partial nephrectomies that were first rehearsed on patient-specific 3D-printed models.² Although the sample size for this study was small and hence not powered to draw clinical conclusions, it nonetheless highlights the utility of rehearsal in surgical planning and preparation. A further validation study by Witthaus and colleagues focuses on the role of anatomic 3D models in surgical training and evaluation.³ The study showed a significant difference in specific task times, forces exerted on nerves, surgical margins and anastomotic integrity between experts (>500 cases) and novices (<50 cases). The above is a wonderful example of how 3D-printed simulation can be a reliable proving ground to evaluate surgical performance and provide real-time constructive feedback. These 3D-printed, patient-specific surgical models range from $21 to $680 depending on materials, detail and complexity.¹ In high-stakes cases and in those with truly unique anatomy, the cost may be well worth the opportunity to rehearse before the opening incision; however, in more routine cases, 3D-printed models may not be a cost-effective or efficient use of time. While cost and production time are challenges for 3D printing, the technology is advancing rapidly and may become a viable standard. However, considering advances in virtual technologies, machine learning, and the announcements of technology giants like Facebook with their recent “metaverse” initiatives, the authors place their bet on a future centered on virtual and augmented reality (AR) platforms.

Virtual and Augmented Reality

Figure 1. A, user viewing a 3D model in a virtual environment using the Oculus virtual reality headset. B, annotated CT with the segmentation showing how a scan is converted into a 3D model. C, posterior view of a semitranslucent kidney model demonstrating a staghorn calculus (yellow) and a ureteral stent (blue) in the collecting system. D, anterior view of semitranslucent kidney model demonstrating a staghorn calculus (yellow) and a ureteral stent (blue) in the collecting system. The veinous (blue) and arterial (red) vessels are visible as well.

Figure 2. This QR code links to a YouTube video of a partial nephrectomy case in which virtual reality 3D models were used to review patient anatomy during surgical preparation. The video compares the 3D model to the actual tumor during surgery.

VR refers to apparatuses that interact with the human senses to recreate real-seeming visuals and feelings. AR is a form of VR in which the user sees the real world and the device adds virtual components to enhance or augment the user’s perceptions. Visual headsets, 3D movies and surround sound systems have become mainstays in the entertainment industry, and these technologies present some fascinating and cost-effective solutions in surgical preparation and patient education. In a 2020 study, Tapiero and colleagues evaluated the ability of urologists to 3-dimensionally comprehend traditional CT images and found that when asked to place a tumor on a 3D recreation of the CT, experienced surgeons had only a 33% accurate overlap with the true tumor location.⁴ However, this improved to 47% when the surgeons were first allowed to interact with a 3D virtual model prior to placing the tumor. This demonstrates the innate inability of humans to reliably understand complex 3D anatomy from 2D images (standard axial imaging presentation) and further makes a strong case more intuitive imaging. The advantages of VR “spoon-feeding” surgical brains for superior anatomic understanding was demonstrated by the same group exhibiting reduced fluoroscopy time, diminished blood loss and higher stone-free rates after percutaneous nephrolithotomy in surgeons who used VR headsets to review the anatomy in comparison to those who only had access to standard CT imaging (fig. 1).⁵ We also routinely apply this technology for complex partial nephrectomy as can be seen, albeit only in 2D, in the attached example (fig. 2). Ordinarily, VR models are viewed with a headset; however, recent developments allow them to be displayed to scale cast over a patient in such a way that the user can view a patient’s internal anatomy. Akand et al conducted a feasibility study showing that a percutaneous needle could be placed using mathematical calculation software and AR to visualize renal stones embedded in 2 different in vitro models.⁶ With advancing technology, it is reasonable to imagine a future in which CT-based AR models could dramatically reduce intraoperative fluoroscopy to establish access to large stones. 3D printing, VR and AR all share one crucial rate limiting step: segmentation. Both modalities require a skilled individual to “segment” or annotate traditional imaging slice-by-slice for a computer to render the 3D models. This process is time consuming and tedious. However, further progress in rendering technology and artificial intelligence offer the potential for automation, and this work is currently underway.

Cinematic Rendering

Animation studios have a plethora of high-powered computer-generated imaging tools that allow them to create photorealistic 3D images with simulated natural lighting. This software is now being applied to medical imaging to produce stunning new imaging avenues. Using CT images and 3D lighting effects, Rowe and colleagues produced breathtaking 3D depictions that make reviewing patient anatomy wonderfully vivid and intuitive (fig. 3).⁷ With evidence that 2D CT images are not readily understood even by trained surgeons in their true 3D anatomic nature,⁴ designing comprehensive yet palatable VR and AR models allows for more accurate and impactful surgical planning.

Figure 3. Image from the study on cinematic rendering (CR) by Rowe SP et al.⁷ A, coronal, delayed phase, contrast-enhanced CT image of a 50-year-old woman with xanthogranulomatous pyelonephritis, including a central staghorn calculus (red arrowhead), significant hydronephrosis and loss of the renal parenchyma (red arrow). B, a cinematically rendered axial excretory/urographic phase CR image. C, coronal excretory/urographic phase CR image. D, coronal arterial phase CR image of the right kidney.

Conclusion

We are in the “gold rush” years of 3D technology, and research and development is ripe for revolutionary and groundbreaking developments that will dramatically change the way we train, diagnose, treat and conduct research as urologists. The same technology bringing us amazing cinema and video games is coming to clinics and operating rooms near you.