Introduction

Currently, intraoperative dentification of surgical structures heavily depends on individual surgeon judgment, surgeon experience, time pressure, fatigue, and confirmation bias. Surgical video annotation is fundamental to constructing supervised computer vision (CV) models,¹ whereas CV models could visualize critical structures embedded in opaque undissected surgical target with authenticity and the output could be accessed by interface devices. CV is a kind of machine learning which exploits the capabilities of machines to achieve a human level of understanding of the surgical target. It uses deep learning algorithms and mathematical techniques to analyze quantifiable features in image such as color, texture, and position at the pixel level.² The facial recognition technology used routinely for security and authentication purposes is an example of CV model application.

The evolution of real-time surgical decision support has underscored the need for high-performance CV models during kidney surgeries. Such models, adept at detecting anatomical landmarks, can superimpose patient-specific 3D models onto the surgical scene. This capability is pivotal for surgeons, offering a dynamic view of the surgical scene and facilitating precise dissections, thereby minimizing intraoperative accidents.¹ Our high-performance model achieved mean intersection over unions (IoUs) of 0.49, 0.72, and 0.74 for the renal artery, renal vein, and spleen, respectively, showcasing its accuracy (Figure 1). Consensus building among 3 annotators improved model performance significantly.

Data Collection and Curation

We built a supervised deep learning model with the surgical videos of laparoscopic live donor nephrectomy (LLDN) as a baseline model of normal anatomy of the kidney and commonly encountered structures during kidney surgery. Three experienced urologists and kidney transplant surgeons remotely worked in close collaboration with CV scientists and annotated the renal artery, renal vein, and spleen in the images. A periodic consensus mechanism was evolved to standardize identification and labeling of the structures. We curated 4291 images from 6 surgical videos of LLDN for annotation using our custom-built platform, RediMinds Ground Truth Factory (Figure 2). This platform facilitated remote site collaboration among stakeholders and served as a one-stop solution for data collection, curation, annotation, and postprocessing.

We handled a large amount of data, which could be assessed by the fact that 1 minute-long, high-definition surgical video contains 25 times the data found in a high-resolution computed tomography image.³ Through meticulous data curation, we maximized image variability, ensuring a diverse data set. An automated curation tool built on our baseline model further streamlined the process, helping identify images requiring additional model training. Deep learning algorithms have achieved human level of performance in object classification.²

Annotation and Data Cleaning

The RediMinds Ground Truth Factory platform facilitated the conversion of curated images into accessible jobs for annotators. Initial annotations were undertaken by a urologist, with 2 more joining due to observed inconsistencies. In the second phase, we semiautomated the process and used the model trained for phase 1 to select images from the remaining raw videos. By our semiautomation in the second phase, image selection jumped from 55.7% to 70%. Data cleaning was essential postannotation. We finalized an annotated data set of 2164 images for model training and 324 images for testing. This data set was rich, containing multiple instances of the renal artery, renal vein, and spleen.

Consensus Building

A continuous feedback loop was maintained between the technical team and annotators. Discrepancies in annotations were diligently addressed, with consensus discussions playing a pivotal role. Rules for annotation were established, emphasizing the importance of annotating only visible structures and excluding any obscured or overlapped structures. Consensus building among annotators improved the performance of the model (Figure 3). In consensus building, the annotators’ choices and model’s training process provided the basis for consensus approach.^4-6

Model Optimization and Testing

One type of CV model architecture is the convolutional neural network. It consists of multiple neural layers including convolutional layers to extract features. Convolutional neural networks are designed to process visual data like images, as in this case.

Four model architectures were selected for use: (1) UNet, a model designed for medical applications, (2) DeeplabV3+, a model widely used in surgical image segmentation for liver, kidney, and small intestine, (3) Detectron 2, a model used by Facebook artificial intelligence applications, and (4) Mask Region–Based Convolution Neural Network, a model that provides both segmentation and detection, including bounding boxes and masks as well as confidence scores, for prediction.

Each model underwent rigorous testing with hyperparameters like batch sizes, data augmentation techniques, number of epochs, and learning rates being optimized. In addition to IoU, we also calculated pixel-level metrics precision, accuracy, and F1 scores. Recall is the proportion of total positives (TPs) out of all instances of a class (false positive [FP] and false negative [FN] or TP/[TP+FN]). The precision is equivalent to specificity. Performance metrics, including IoU, precision, accuracy, and F1 score, were employed to gauge model efficacy (Table). Precision recall curves were constructed for standout models. F1 scores provide an overall metric incorporating precision and recall.⁷

Table. A Detailed Presentation of Performance Metrics—Intersection Over Union, Precision (Specificity), Recall (Sensitivity), and F1 Score—for Each Anatomical Structure (Renal Artery, Renal Vein, and Spleen)

Comments

The endeavor to develop CV models for anatomical landmark detection is a significant stride toward our overarching goal: creating intraoperative guidance tools for kidney surgery. The past decade has witnessed a renaissance in CV models, with their integration into medical devices becoming increasingly prevalent. One of the primary challenges we faced was the dearth of large, labeled data sets. Precise annotation is a labor-intensive process, necessitating collaboration between urologists and computer scientists.

Our choice of LLDN was strategic, offering a model closely aligned with the standard surgical anatomy of the kidney. This foundational model can be expanded to accommodate kidney pathologies, enhancing its applicability to a broader range of kidney surgeries. The focus on larger anatomical structures, like the renal artery, renal vein, and spleen, was deliberate, allowing us to refine our model-building pipeline and foster consensus among annotators.

The models we optimized in this study are emblematic of the cutting-edge advancements in CV as applied to medical imaging. As we look ahead, the inclusion of additional classes will be crucial, enabling comprehensive surgical scene recognition. Structures like the left gonadal vein, ureter, and adrenal gland, which are susceptible to surgical injury, will require extensive data sets to account for their variability.

Acknowledgements

This material is based upon work supported by the National Science Foundation under Small Business Technology Transfer Grant No. 1953822. RediMinds, Inc has provided further funding for this project. Dave Meinhard, Vattikuti Foundation, provided support for the video.