This PDF file contains the front matter associated with SPIE Proceedings Volume 12927, including the Title Page, Copyright information, Table of Contents, and Conference Committee information.

We would like deep learning systems to aid radiologists with difficult decisions instead of replacing them with inscrutable black boxes. "Explaining" the black boxes with XAI tools is problematic, particularly in medical imaging where the explanations from XAI tools are inconsistent and unreliable. Instead of explaining the black boxes, we can replace them with interpretable deep learning models that explain their reasoning processes in ways that people can understand. One popular interpretable deep learning approach uses case-based reasoning, where an algorithm compares a new test case to similar cases from the past ("this looks like that"), and a decision is made based on the comparisons. Radiologists often use this kind of reasoning process themselves when evaluating a new challenging test case. In this talk, I will demonstrate interpretable machine learning techniques through applications to mammography and EEG analysis.

Artificial intelligence and machine learning (AI/ML) are powerful tools for building computer vision systems that support the work of clinicians, leading to high interest and explosive growth in the use of these methods to analyze clinical images. These promising AI techniques create computer vision systems that perform some image interpretation tasks at the level of expert radiologists. In radiology, deep learning methods have been developed for image reconstruction, imaging quality assurance, imaging triage, computer-aided detection, computer-aided classification, and radiology documentation. The resulting computer vision systems are being implemented now and have the potential to provide real-time assistance, thereby reducing diagnostic errors, improving patient outcomes, and reducing costs. We will show examples of real-world AI applications that indicate how AI will change the practice of medicine and illustrate the breakthroughs, setbacks, and lessons learned that are relevant to medical imaging.

Accurate training labels are a key component for multi-class medical image segmentation. Their annotation is costly and time-consuming because it requires domain expertise. In our previous work, a dual-branch network was developed to segment single-class edematous adipose tissue. Its inputs include a few strong labels from manual annotation and many inaccurate weak labels from existing segmentation methods. The dual-branch network consists of a shared encoder and two decoders to process weak and strong labels. Self-supervision iteratively updates weak labels during the training process. This work aims to follow this strategy and automatically improve training labels for multi-class image segmentation. Instead of using weak and strong labels to only train the network once in the previous work, transfer learning is used to train the network and improve weak labels sequentially. The dual-branch network is first trained by weak labels alone to initialize model parameters. After the network is stabilized, the shared encoder is frozen, and strong and weak decoders are fine-tuned by strong and weak labels together. The accuracy of weak labels is iteratively improved in the fine-tuning process. The proposed method was applied to a three-class segmentation of muscle, subcutaneous and visceral adipose tissue on abdominal CT scans. Validation results on 11 patients showed that the accuracy of training labels was statistically significantly improved, with the Dice similarity coefficient of muscle, subcutaneous and visceral adipose tissue increased from 74.2% to 91.5%, 91.2% to 95.6%, and 77.6% to 88.5%, respectively (p<0.05). In comparison with our earlier method, the label accuracy was also significantly improved (p<0.05). These experimental results suggested that the combination of the dual-branch network and transfer learning is an efficient means to improve training labels for multiclass segmentation.

Radiomic ultrasound-based artificial intelligence (AI) tools may improve adnexal mass evaluations by introducing more quantitative assessments. Detailed segmentation of the lesions is the first step in a radiomics AI classification pipeline. However, accurate outlining is a difficult task, prone to error, and time-consuming. We aimed to develop an automatic method to reduce variability and improve clinical workflow. To evaluate the robustness of using retrospective data, we investigated whether images with sonographic measurement markups interfere with automatic segmentations. A retrospective dataset of 296 images from 106 adnexal lesions (53 benign/53 malignant) was separated by patient into training (19 benign/17 malignant; 89 images) and independent test (34 benign/36 malignant; 207 images) sets. The U-Net was trained twice using images with and without markups. Images were cropped to 20 pixels surrounding the outline and resized to 256x256 pixels. The training set was augmented using flips and rotations. U-Net segmentation performance was compared to expert outlines using the Dice Similarity Coefficient (DSC) and the average Hausdorff distance (HD) and compared using the median and 95% confidence interval (CI) of the difference, with statistical significance indicated if the 95% CI did not cross zero. The median DSC and HD on the independent test set when markups were included in training were 0.92 and 14.4, respectively, and 0.91 and 17.1, respectively, without markups in training. The differences were 0.008 [95% CI: -0.033, 0.056] for DSC and -1.23 [95% CI: -13.5, 7.33] for HD, indicating no evidence for statistically significant difference in performance. Using a U-net algorithm to automatically outline adnexal lesions had excellent agreement with expert outlines, independent of measurement markups presence,supporting AI pipeline development to differentiate between benign and malignant adnexal masses.

Nasal obstruction (NO), which affects one-third of the adult population, is characterized by a blockage in the nasal cavity. Rhinologists commonly employ nasal endoscopy (NE) in the differential diagnosis of NO, along with a focused history and other examinations such as skin prick tests and CT scans. This study aims to establish NE as a reliable standalone diagnostic tool, eliminating the necessity for CT scans and skin prick tests in the diagnosis of NO. However, currently, there is a lack of objective methods to quantify the severity of NO. To address this problem, we used deep learning to identify the anatomical structures of the anterior nasal cavity, which will then be graded by an objective grading system. In this paper, we evaluated the performance of various deep learning methods (DeepLabv3+, MaskFormer, and Mask2Former) with different pre-trained backbones (ResNet-101 - CNN-based, and Swin-Tiny - transformer-based), for semantic segmentation of the anterior nasal cavity. Sixty-two participants were examined with NE before and after using a nasal decongestant. For model training and validation, 608 images from 46 participants were utilized, and 171 images from 16 participants were reserved for testing. The fine-tuned Mask2Former with low-light image enhancement achieved a mean intersection-over-union of 81.7% and 61.2% on the validation and testing sets, respectively. These findings represent the first successful semantic segmentation of key anatomical structures within the anterior nasal cavity. These segmented structures will serve as the basis for classifying the severity of NO and diagnosing NO conditions, enabling AIbased consultations in primary care settings such as general practices and remote locations, where access to ENT expertise may be limited.

Precise segmentation of rectal cancer tumors on routine MRI is critical for accurate clinical staging and downstream computational analyses. While deep learning-based segmentation algorithms have shown much promise in automating the otherwise tedious, subjective, and costly process of manual segmentation, they require significant amounts of manually annotated data for training. To address these limitations of deep learning-based segmentation models, we present a novel deep learning framework that incorporates human-in-the-loop (HITL) refinement to automatically delineate rectal tumors on multi-plane pre-treatment MR imaging. When evaluated on multiple holdout validation cohorts including a clinical trial dataset, the post-HITL segmentation model significantly outperformed the pre-HITL model with median dice similarity coefficient of 0.763 and Hausdorff distance of 28.4mm in comparison to 0.601 and 31.8mm, respectively. HITL refinement learning also significantly accelerated the manual annotation process by 20 minutes. HITL learning represents a feasible, effective, and efficient solution to semi-automated tumor segmentation on routine rectal cancer MRI scans.

This paper addresses the problem of lung lobe partitioning in ultra-short time echo (UTE) MRI acquisitions, which are recently used for lung ventilation assessment with MRI spirometry. Because of the low image contrast, which does not enable the lung fissures display, the developed approach relies only on the vascular structures which still can be segmented from these images. The vascular network is segmented in lobes in order to generate reference clusters used for lung space partitioning. A point cloud representing the unstructured points of the vascular medial axis is partitioned in five lobes exploiting the PointNet++ framework. The PointNet++ model is trained on data extracted from CT acquisitions and labeled using the airway and vascular trees connectivity. The airway tree lobes will define the lung lobar regions, which are propagated on the vessel structure to achieve the complete vascular labeling. A separate model is trained for the right and left lungs in order to alleviate for limited input point cloud size imposed by the model architecture and reach a high precision in classification. The trained model is applied to UTE-MRI data to generate, for a given subject, a point cloud reference that will be used for vascular lobes clustering, which will be then exploited for lung space partitioning in lobes. The approach was quantitatively evaluated on 10 CT volumes from LUNA16 dataset and qualitatively tested on additional 25 CT and 15 UTE-MRI datasets. The analysis of CT data results shows pertinent lung partitioning with respect to the lung fissures, even if a precise fissure localization is not achieved. Such result is however expected, since no information related to the lung fissure is exploited in our method because this would not be applicable to UTE-MRI data. Nevertheless, the proposed partitioning respects the vascular lobes and, to the best of our knowledge, is novel for lung MRI sequences making it possible the regional investigation of ventilation parameters in MRI spirometry. The method can be further on extended for lung fissure matching in CT data by integrating new constraints related to fissure detection.

This study presents a semi-supervised learning strategy to predict Modified Radiographic Assessment of Lung Edema (mRALE) scores from chest radiographs (CXR) of COVID-19 patients. Using a combination of EfficientNet models and large volumes of unlabeled data, we developed a robust deep-learning model that exhibits outstanding performance, securing 4th place in the MIDRC mRALE Mastermind Challenge. Our strategy harnessed both the labeled data provided in the challenge and a significant amount of unlabeled data from the Medical Imaging and Data Resource Center (MIDRC) and CheXpert. We used labeled data to train an EfficientNet-B4 model, which we used to infer pseudo labels on our unlabeled data. We then used Chexpert to pre-train an EfficientNet-B7 model from scratch, allowing it to extract valuable information from a broader scope of CXR images. The final step involved fine-tuning the pre-trained EfficientNet-B7 model using additional CXR images from MIDRC. This approach effectively adapted the model to our specific domain of interest: portable CXR images from COVID-19 patients. The success of this semi-supervised learning approach highlights its potential to enhance diagnostic capabilities in medical imaging, particularly in situations where labeled data is limited. It demonstrates the feasibility of leveraging massive volumes of unlabeled data, even when the specific disease differs, provided that radiological characteristics are similar. This study paves the way for advancements in deep learning applications in healthcare, especially in emergency scenarios that demand swift and accurate diagnostics, such as the COVID-19 pandemic.

We investigated whether geometric features of pulmonary arteries are associated with early progression or progression-free survival at twelve months following initiation of combined immunotherapy and chemotherapy treatment in patients with extensive stage small cell lung cancer (SCLC). We utilized pre-treatment Computerized Tomography (CT) scans of 64 patients’ lungs to create skeletonized networks of patients’ pulmonary arteries, and a machine learning model was used to determine associations of geometric and clinical features with treatment response and survival. Geometric analyses achieved a ROC-AUC of 0.63 ± 0.23 for the ipsilateral geometric model and 0.61 ± 0.13 for the geometric model incorporating geometric features from both lungs, both higher than that of the clinical model alone, 0.58 ± 0.08. Additionally, the combined clinical and geometric model yielded a ROC-AUC of 0.64 ± 0.25 for the ipsilateral model and 0.67 ± 0.10 for the model incorporating features from both lungs, both higher than that of the clinical model alone.

Parametric response mapping (PRM) is a voxel-based quantitative CT imaging biomarker that measures the severity of chronic obstructive pulmonary disease (COPD) by analyzing both inspiratory and expiratory CT scans. Although PRM-derived measurements have been shown to predict disease severity and phenotyping, their quantitative accuracy is impacted by the variability of scanner settings and patient conditions. The aim of this study was to evaluate the variability of PRM-based measurements due to the changes in the scanner types and configurations. We developed 10 human chest models with emphysema and air-trapping at endinspiration and end-expiration states. These models were virtually imaged using a scanner-specific CT simulator (DukeSim) to create CT images at different acquisition settings for energy-integrating and photoncounting CT systems. The CT images were used to estimate PRM maps. The quantified measurements were compared with ground truth values to evaluate the deviations in the measurements. Results showed that PRM measurements varied with scanner type and configurations. The emphysema volume was overestimated by 3 ± 9.5 % (mean ± standard deviation) of the lung volume, and the functional small airway disease (fSAD) volume was underestimated by 7.519 % of the lung volume. PRM measurements were more accurate and precise when the acquired settings were photon-counting CT, higher dose, smoother kernel, and larger pixel size. This study demonstrates the development and utility of virtual imaging tools for systematic assessment of a quantitative biomarker accuracy.

To enhance reproducibility and robustness in mammographic density assessment, various deep learning models have been proposed to automatically classify mammographic images into BI-RADS density categories. Our study aims to compare the performances of different deep learning models in making breast density classifications from full-field digital mammography (FFDM) versus synthetic mammography (SM), the newer 2D mammographic image format acquired with digital breast tomosynthesis (DBT). We retrospectively analyzed negative (BI-RADS 1 or 2) routine mammographic screening exams (Selenia or Selenia Dimensions; Hologic) acquired at sites within the Barnes-Jewish/Christian (BJC) Healthcare network in St. Louis, MO from 2015 to 2018. BI-RADS breast density assessments of radiologists were obtained from BJC’s mammography reporting software (Magview 7.1). For each mammographic imaging modality, a balanced dataset of 4,000 women was selected so there were equal numbers of women in each of the four BI-RADS density categories, and each woman had at least one mediolateral oblique (MLO) and one craniocaudal (CC) view per breast in that mammographic imaging modality. Previously validated pre-processing steps were applied to all FFDM and SM images to standardize image orientation and intensity. Images were then split into training, validation, and test sets at ratios of 80%, 10%, and 10%, respectively, while maintaining the distribution of breast density categories and ensuring that all images of the same woman appear only in one set. ResNet-50 and EfficientNet-B0 architectures were optimized, trained, and evaluated separately for different imaging modalities. Overall, the models had comparable performance, though ResNet-50 performed slightly better in most cases. Furthermore, FFDM images had better classification accuracies than SM images. Our preliminary findings suggest that further deep learning developments and optimizations may be needed as we develop breast density deep learning models for the newer mammographic imaging modality, DBT.

Breast density is an important factor in assessing individual breast cancer risk. We aim to identify women at increased risk of developing breast cancer before they enter routine screening, using mammography in combination with known risk factors. This will enable targeting of preventive therapies and personalised screening. To reduce radiation risk, this paper examines whether density measurements in one breast or mammographic view could be used to accurately reflect individual risk. We analysed breast cancer risk using breast density in a 1:3 case-control dataset of mammograms from the Predicting Risk of Cancer at Screening Study (PROCAS). Breast density was measured using pVAS, an AI-based approach. Cancer risk in low and high breast density groups was compared using conditional logistic regression. High breast density was independently associated with increased breast cancer risk. Women in the highest breast density quintile averaged across all views had an Odds Ratio (OR) of 4.16 (95% CI 2.90-5.97) compared to those in the lowest. A similar OR was found in both the left 3.77 (95% CI 2.68-5.31) and right 4.52 (95% CI 3.12-6.55) breasts individually. ORs were also significant for each individual view: right mediolateral oblique (MLO) 4.19 (2.92–6.00), right craniocaudal (CC) 4.40 (3.09–6.27), left MLO 3.27 (2.34–4.56) and left CC 3.65 (2.60–5.11). The ability to predict breast cancer risk due to increased breast density was achieved using one breast and even one mammographic view. This provides the possibility of a pre-screening risk assessment using fewer images and therefore less radiation.

Devices enabled by artificial intelligence (AI) and machine learning (ML) are being introduced for clinical use at an accelerating pace. In a dynamic clinical environment, these devices may encounter conditions different from those they were developed for. The statistical data mismatch between training/initial testing and production is often referred to as data drift. Detecting and quantifying data drift is significant for ensuring that AI model performs as expected in clinical environments. A drift detector signals when a corrective action is needed if the performance changes. In this study, we investigate how a change in the performance of an AI model due to data drift can be detected and quantified using a cumulative sum (CUSUM) control chart. To study the properties of CUSUM, we first simulate different scenarios that change the performance of an AI model. We simulate a sudden change in the mean of the performance metric at a change-point (change day) in time. The task is to quickly detect the change while providing few false-alarms before the change-point, which may be caused by the statistical variation of the performance metric over time. Subsequently, we simulate data drift by denoising the Emory Breast Imaging Dataset (EMBED) after a pre-defined change-point. We detect the change-point by studying the pre- and post-change specificity of a mammographic CAD algorithm. Our results indicate that with the appropriate choice of parameters, CUSUM is able to quickly detect relatively small drifts with a small number of false-positive alarms.

Contrast-enhanced spectral mammography (CESM) has proven to be a beneficial tool for additional management and diagnosis of breast cancer in patients. Deep learning-based automated tools for analyzing CESM images and detecting abnormalities can be crucial in reducing the workload of radiologists. Even though self-supervised learning (SSL) can significantly influence the quality of learned representations resulting in increased classification performances, it requires large amounts of data often unavailable in the CESM domain. In this work, we propose a novel loss function named Difficulty-Weighted Neighborhood Representation (DWNR) utilizing the SSL-based representations designed explicitly for the downstream classification of the region of interest (ROI) in CESM images in a small dataset setting. We propose utilizing the neighborhood representations extracted from an SSL backbone to guide the loss function in learning robust representations of the dataset’s normal, benign, and malignant ROIs. We formulate the DWNR loss using a combination of a neighborhood representation-guided triplet (NR-triplet) loss and a difficulty-weighted cross-entropy (DWCE) loss. The NR-triplet loss leverages the nearest and farthest neighbor feature representations, extracted from a self-supervised backbone, as positive and negative samples. Meanwhile, the DWCE loss exploits the class information from the nearest neighbors to perform per-sample difficulty weighting. The experimental results on the publicly available CDD-CESM dataset demonstrate that the proposed DWNR loss outperforms traditional loss functions such as Cross-Entropy (CE) and Focal loss by a significant gain of more than 2% absolute mean F1-scores for both SSL-based and ImageNet-SSL-based transfer learning. In addition, we demonstrate the efficacy of employing a DWNR-based ROI classifier for comprehensive CESM whole image analysis in BI-RADS scoring and cancer diagnosis. The results reveal a gain in mean F1-scores of more than 1% in both tasks when using the DWNR loss models for CNN initialization, indicating the significance of the proposed method to whole image analysis. Based on our analysis, incorporating DWNR loss during transfer learning could positively impact automated breast cancer diagnosis using the CESM imaging modality. This novel loss function opens new possibilities for leveraging deep learning in CESM, particularly in scenarios with limited labeled data availability.

In this study, we adapted a transformer-based method to localize lesions in digital breast tomosynthesis (DBT) images. Compared with convolutional neural network-based object detection methods, the transformer-based method does not require non-maximum suppression postprocessing. Integrated deformable convolution detection transformers can better capture small-size lesions. We added transfer learning to tackle the issue of the lack of annotated data from DBT. To validate the superiority of the transformer-based detection method, we compared the results with deep-learning object detection methods. The experimental results demonstrated that the proposed method performs better than all comparison methods.

The clinical management of breast cancer depends on an accurate understanding of the tumor and its anatomical context to adjacent tissues and landmark structures. This context may be provided by semantic segmentation methods; however, previous works have been largely limited to a singular focus on the tumor alone and rarely other tissue types. In contrast, we present a method that exploits tissue-tissue interactions to accurately segment every major tissue type in the breast including: chest wall, skin, adipose tissue, fibroglandular tissue, vasculature and tumor via standard-of-care Dynamic Contrast Enhanced MRI. Comparing our method to prior state-of-theart, we achieved a superior Dice score on tumor segmentation while maintaining competitive performance on other studied tissues across multiple institutions. Briefly, our method proceeds by localizing the tumor using 2D object detectors, then segmenting the tumor and surrounding tissues independently using two 3D U-nets, and finally integrating these results while mitigating false positives by checking for anatomically plausible tissuetissue contacts. The object detection models were pre-trained on ImageNet and COCO, and operated on MIP (maximum intensity projection) images in the axial and sagittal planes, establishing a 3D tumor bounding box. By integrating multiple relevant peri-tumoral tissues, our work enables clinical applications in breast cancer staging, prognosis and surgical planning.

To improve the model’s robustness and generalization performance, we investigate effective test-time augmentation-based ensemble prediction methods and evaluate the effectiveness of various ensemble prediction techniques in combination. In the training phase, we generate an optimized predictive model using a multi-modality regression network. The prediction is then determined through ensemble average voting with augmented test images generated by diverse data augmentation methods, including affine transformation, Mixup, Cutout, CutMix, and their combinations. Our experimentation reveal that all ensemble prediction methods demonstrated the ability to address issues through regularization, such as averaging errors on images subjected to random modifications. Notably, the use of Affine significantly improves over the baseline, with a 18.3% increase in accuracy and a 12.2% increase in AUC. The adoption of CutMix, maintains stability in both sensitivity and specificity, resulting in a higher balanced accuracy than Mixup and Cutout.

Recent advancements in neuroimaging techniques have sparked a growing interest in understanding the complex interactions between anatomical regions of interest (ROIs), forming into brain networks that play a crucial role in various clinical tasks, such as neural pattern discovery and disorder diagnosis. In recent years, graph neural networks (GNNs) have emerged as powerful tools for analyzing network data. However, due to the complexity of data acquisition and regulatory restrictions, brain network studies remain limited in scale and are often confined to local institutions. These limitations greatly challenge GNN models to capture useful neural circuitry patterns and deliver robust downstream performance. As a distributed machine learning paradigm, federated learning (FL) provides a promising solution in addressing resource limitation and privacy concerns, by enabling collaborative learning across local institutions (i.e., clients) without data sharing. While the data heterogeneity issues have been extensively studied in recent FL literature, cross-institutional brain network analysis presents unique data heterogeneity challenges, that is, the inconsistent ROI parcellation systems and varying predictive neural circuitry patterns across local neuroimaging studies. To this end, we propose FedBrain, a GNN-based personalized FL framework that takes into account the unique properties of brain network data. Specifically, we present a federated atlas mapping mechanism to overcome the feature and structure heterogeneity of brain networks arising from different ROI atlas systems, and a clustering approach guided by clinical prior knowledge to address varying predictive neural circuitry patterns regarding different patient groups, neuroimaging modalities and clinical outcomes. Comparing to existing FL strategies, our approach demonstrates superior and more consistent performance, showcasing its strong potential and generalizability in cross-institutional connectome-based brain imaging analysis.

Multi-parametric MRI of the body is routinely acquired for the identification of abnormalities and diagnosis of diseases. However, a standard naming convention for the MRI protocols and associated sequences does not exist due to wide variations in imaging practice at institutions and myriad MRI scanners from various manufacturers being used for imaging. The intensity distributions of MRI sequences differ widely as a result, and there also exists information conflicts related to the sequence type in the DICOM headers. At present, clinician oversight is necessary to ensure that the correct sequence is being read and used for diagnosis. This poses a challenge when specific series need to be considered for building a cohort for a large clinical study or for developing AI algorithms. In order to reduce clinician oversight and ensure the validity of the DICOM headers, we propose an automated method to classify the 3D MRI sequence acquired at the levels of the chest, abdomen, and pelvis. In our pilot work, our 3D DenseNet-121 model achieved an F1 score of 99.5% at differentiating 5 common MRI sequences obtained by three Siemens scanners (Aera, Verio, Biograph mMR). To the best of our knowledge, we are the first to develop an automated method for the 3D classification of MRI sequences in the chest, abdomen, and pelvis, and our work has outperformed the previous state-of-the-art MRI series classifiers.

Previous studies suggest that respiratory events are the leading cause of airway management-related deaths. Successful prediction of difficult-to-intubate (DI) patients can help clinicians precisely allocate the respiratory resources for good airway management. However, the current clinical bedside tests are highly biased with respect to anesthesiologists’ experience and yield moderate accuracy with low sensitivity. Therefore, a diagnostic tool with high accuracy, especially sensitivity, is in critical need. For these reasons, we propose an AI-based method to predict DI patients based on their facial images from frontal and profile views. Our ensemble model based on frontal and profile views of the face achieve an AUC of 0.713 and a sensitivity of 68.5%. In addition, our ensemble model increases sensitivity of thyromental distance from 26.9% to 79.2% while maintaining an acceptable trade-off in specificity. Overall, our model can meaningfully augment the accuracy of current clinical tests for DI. We envision that our model may be embedded on smartphones to serve as a bedside test for DI and that our study provides a basis for future studies for AI-based methods on facial images for DI prediction.

We propose to develop a prediction model that uses the quantitative lung fissure integrity score from chest CT scans that can identify emphysema patients that successfully respond to endobronchial valve (EBV) treatment. It is hypothesized that patients with high fissure integrity are more likely to respond to EBV treatment and achieve volume reduction of the emphysematous lobe. This study retrospectively collected 89 anonymized pre-treatment chest CT exams from patients with moderate to severe emphysema and who underwent EBV treatment. Previous work used a deep learning approach that segmented lung fissure and quantified a fissure integrity score (FIS) for the right horizontal fissure (RHF), right oblique fissure (ROF), and left oblique fissure (LOF). A FIS is defined as the percentage of total fissure voxels present along the interlobar region. Fissures were categorized as complete with a FIS of ≥90%; otherwise, it was considered incomplete. The response to EBV treatment was recorded as the amount of targeted lobe volume reduction (TLVR) compared to target lobe volume prior to treatment. EBV placement was considered successful with a TLVR of ≥350 ml. Statistical analyses were performed separately for each targeted lobe and a logistic regression model was trained using the extracted FIS. From the test set, 8 subjects achieved TLVR with a mean(±SD) FIS of 0.943(±0.052). 23 targeted lobes did not achieve the desired TLVR, with a mean(±SD) FIS of 0.751(±0.201). The EBV prediction model using the FIS achieved an AUC of 0.842. A model using the quantified FIS shows potential as a predictive biomarker for whether a targeted lobe will achieve successful volume reduction from EBV treatment.

In glioblastoma, an aggressive brain tumor with a low survival rate, accurately predicting patient survival is crucial for effective treatment planning. For a deep learning model developed using MRI images for glioblastoma prognosis, it’s essential to identify which MRI sequence – T1, T2, T1-contrast enhanced, or FLAIR – yields the most insightful prognostic features. This study utilizes a specialized autoencoder architecture, specifically a Unet model with an attention mechanism equipped with a custom-designed loss function for reconstructing tumor subregions from multimodal MRI scans. Once the autoencoder is trained, it extracts unique imaging features. Subsequently, these extracted features are analyzed using a Cox regression neural network, which is then applied to predict the survival probabilities of glioblastoma patients. The method was applied based on 5-fold cross-validation. The survival prediction performance of the model was assessed using various MRI sequences. The concordance index (C-index) values obtained were 0.62 for T1, 0.63 for T1CE, 0.63 for T2, and 0.66 for FLAIR. Additionally, the area under the curve (AUC) values recorded were 0.76 for T1, 0.74 for T1CE, 0.76 for T2, and 0.82 for FLAIR, indicating a variation in predictive accuracy across different sequences. The findings from our study underscore a discernible variation in the effectiveness of different MRI sequences for glioblastoma survival prognosis. Notably, the FLAIR sequence emerged as the most informative, providing the highest level of prognostic detail in comparison to T1, T1CE, and T2 sequences. This highlights the significance of sequence selection in enhancing the accuracy of survival predictions in glioblastoma cases.

Substantia nigra (SN) has been reported as significantly related to the progression of Parkinson’s Disease (PD). Fully automated segmentation of SN is an important step for developing an interpretable computer-aided diagnosis system for PD. Based on the deep learning techniques, this paper proposes a novel distance-reweighted loss function and combines it with the test-time normalization (TTN) to boost the fully automated SN segmentation accuracy from low contrast T2 weighted MRI. The proposed loss encourages the model to focus on the suspicious regions with vague boundaries, and the involved TTN narrows the gap between an input MRI volume and the reference MRI volumes in test-time. The results showed that both the proposed loss and TTN could help improve the segmentation accuracy. By combining the proposed loss and TTN, the averaged Dice coefficient achieved 70.90% from T2 weighted MRI, compared to 68.17% by the baseline method.

The quantification of coronary artery calcium (CAC) score is a crucial metric for evaluating atherosclerotic plaque accumulation within coronary arteries, providing valuable predictive insights into coronary artery disease (CAD) events. Automated methods for CAC assessment using cardiac computed tomography (CT) have been extensively researched. However, a key aspect of these techniques is the accurate localization and segmentation of the heart within CT scans. This study presents a model-driven investigation into the efficacy of mediastinum segmentation, emphasizing its potential impact on CAC quantification in cardiac images. Utilizing a U-Resnet architecture, our methodology focuses on achieving a balance between model compactness and predictive capacity. The research involves two distinct datasets: one from a local medical institution and another serving as an external benchmark. The proposed model has demonstrated superior generalization in medical image segmentation exhibiting high accuracy in identifying true mask pixels within both its training and unseen datasets. Performance metrics showcase its success, with an average Dice Similarity Coefficient (DS) of 0.78, Precision (Pr) of 0.81, and Sensitivity (Se) of 0.81, surpassing the classical U-net model. Accurate localization and segmentation of the heart within CT images hold significant potential for facilitating the quantification of calcium deposits, contributing to the evaluation of coronary artery disease. Moreover, our research emphasizes the importance of recognizing potential labeling disparities in public datasets, attributing them to diverse labeling practices. In conclusion, this study highlights the model's potential in robust mediastinum segmentation, with direct implications for enhancing accuracy in CAC quantification within cardiac images. This contribution advances the field of accurate cardiac image analysis, ultimately benefiting patient care and enabling a more precise assessment of cardiovascular disease.

The skeletal region is one of the common sites of metastatic spread of cancer in the breast and prostate. CT is routinely used to measure the size of lesions in the bones. However, they can be difficult to spot due to the wide variations in their sizes, shapes, and appearances. Precise localization of such lesions would enable reliable tracking of interval changes (growth, shrinkage, or unchanged status). To that end, an automated technique to detect bone lesions is highly desirable. In this pilot work, we developed a pipeline to detect bone lesions (lytic, blastic, and mixed) in CT volumes via a proxy segmentation task. First, we used the bone lesions that were prospectively marked by radiologists in a few 2D slices of CT volumes and converted them into weak 3D segmentation masks. Then, we trained a 3D full-resolution nnUNet model using these weak 3D annotations to segment the lesions and thereby detected them. Our automated method detected bone lesions in CT with a precision of 96.7% and recall of 47.3% despite the use of incomplete and partial training data. To the best of our knowledge, we are the first to attempt the direct detection of bone lesions in CT via a proxy segmentation task.

Non-invasive measurement of knee implant loosening is important to provide a diagnostic tool for patients with recurrent complaints after a total knee arthroplasty (TKA). Displacement measurements are currently estimated between tibial implant and bone using a loading device, CT imaging and an advanced 3D image analysis workflow. However, user interaction is required within each step of this workflow, especially in the segmentation of implant and bone, increasing the complexity of this task and affecting its reproducibility. A deep learning-based segmentation model can alleviate the workload by increasing automation and reducing the variability of manual segmentation. In this work, we propose a segmentation algorithm for the tibial implant and tibial bone cortex. The automatically obtained segmentations are then introduced in the displacement calculation workflow and four displacement measurements are calculated, namely mean target registration error (mTRE), maximum total point motion (MTPM), magnitude of translation and rotation. Results show that the parameter distributions are similar to the manual approach, with intra-class correlation values ranging from 0.96 to 0.99 for the different displacement measurements. Moreover, the methodological error has a smaller or comparable distribution, showing the feasibility to increase automation in knee implant displacement assessment.

Pheochromocytomas and Paragangliomas (PPGLs) are rare adrenal and extra-adrenal tumors which have the potential to metastasize. For the management of patients with PPGLs, CT is the preferred modality of choice for precise localization and estimation of their progression. However, due to the myriad variations in size, morphology, and appearance of the tumors in different anatomical regions, radiologists are posed with the challenge of accurate detection of PPGLs. Since clinicians also need to routinely measure their size and track their changes over time across patient visits, manual demarcation of PPGLs is quite a time-consuming and cumbersome process. To ameliorate the manual effort spent for this task, we propose an automated method to detect PPGLs in CT studies via a proxy segmentation task. As only weak annotations for PPGLs in the form of prospectively marked 2D bounding boxes on an axial slice were available, we extended these 2D boxes into weak 3D annotations and trained a 3D full-resolution nnUNet model to directly segment PPGLs. We evaluated our approach on a dataset consisting of chest-abdomen-pelvis CTs of 255 patients with confirmed PPGLs. We obtained a precision of 70% and sensitivity of 64.1% with our proposed approach when tested on 53 CT studies. Our findings highlight the promising nature of detecting PPGLs via segmentation, and furthers the state-of-the-art in this exciting yet challenging area of rare cancer management.

Federated learning (FL) has emerged as a promising strategy for collaboratively training complicated machine learning models from different medical centers without the need of data sharing. However, the traditional FL relies on a central server to orchestrate the global model training among clients. This makes it vulnerable to the failure of the model server. Meanwhile, the model trained based on the global data property may not yield the best performance on the local data of a particular site due to the variations of data characteristics among them. To address these limitations, we proposed Gossip Mutual Learning(GML), a decentralized collaborative learning framework that employs Gossip Protocol for direct peerto-peer communication and encourages each site to optimize its local model by leveraging useful information from peers through mutual learning. On the task of tumor segmentation on PET/CT images using HECKTOR21 dataset with 223 cases from five clinical sites, we demonstrated GML could improve tumor segmentation performance in terms of Dice Similarity Coefficient (DSC) by 3.2%, 4.6% and 10.4% on site-specific testing cases as compared to three baseline methods: pooled training, FedAvg and individual training, respectively. We also showed GML has comparable generalization performance as pooled training and FedAvg when applying them on 78 cases from two out-of-sample sites where no case was used for model training. In our experimental setup, GML showcased a sixfold decrease in communication overhead compared to FedAvg, requiring only 16.67% of the total communication overhead.

Precise deformable image registration of multi-parametric MRI sequences is necessary for radiologists in order to identify abnormalities and diagnose diseases, such as prostate cancer and lymphoma. Despite recent advances in unsupervised learning-based registration, volumetric medical image registration that requires considering the variety of data distributions is still challenging. To address the problem of multi-parametric MRI sequence data registration, we propose an unsupervised domain-transported registration method, called OTMorph by employing neural optimal transport that learns an optimal transport plan to map different data distributions. We have designed a novel framework composed of a transport module and a registration module: the former transports data distribution from the moving source domain to the fixed target domain, and the latter takes the transported data and provides the deformed moving volume that is aligned with the fixed volume. Through endto-end learning, our proposed method can effectively learn deformable registration for the volumes in different distributions. Experimental results with abdominal multi-parametric MRI sequence data show that our method has superior performance over around 67-85% in deforming the MRI volumes compared to the existing learningbased methods. Our method is generic in nature and can be used to register inter-/intra-modality images by mapping the different data distributions in network training.

Colorectal cancer, ranked as the third deadliest disease globally, can be effectively prevented through the timely detection and removal of colorectal polyps. Precise diagnosis necessitates the accurate segmentation of these polyps, a task where existing deep learning solutions exhibit limitations. Specifically, many CNN-based models emphasize local information, hence their receptive field is constrained by the kernel size, impairing their effectiveness on larger polyps. Conversely, vision transformer-based models replace the CNN-based encoder with transformers to avail more potent global contextual representations, however, the segmentation still hinges on a CNN-centric decoder. Bridging this gap, we introduce the W-shaped Multi-Attention Assisted Transformer (WMAFormer) for polyp segmentation, which employs transformer modules in lieu of conventional convolutional blocks within the decoder. Structurally, our encoder harnesses the pyramid vision transformer’s capabilities, while our decoder amalgamates three pivotal modules: Reference Feature Extractor (RFE), Semantic Feature Enhancement (SFE), and Reverse Attention Decoder (RAD). Notably, the SFE module employs mutual and dual attention mechanisms to augment shared information across varying scales and channels of feature maps. This enhancement necessitates a robust reference map, a responsibility vested in the RFE. Subsequent to this refinement, the improved feature map is channeled to the RAD, which employs reverse attention operations to yield the final prediction. Throughout this architecture, attention mechanisms remain paramount, safeguarding the preservation of global information. Our comprehensive evaluation spanning five prominent datasets showcases the model’s prowess, with both quantitative and numerical results that commendably outpace several contemporary state-of-the-art semantic segmentation methods.

Breast cancer detection at an early stage significantly increases the chances of recovery for patients. Mammography (MG) is one of the most popular non-invasive and high-resolution imaging allowing radiologists to depict early signs of the disease. Microcalcifications (MCs) often occupy less than 1mm in size and can represent a high risk of suspicion depending on the spatial distribution, morphology, and their evolution over time. Their detection is challenging both the clinicians and computer-aided detection tools. In this work, we propose a novel annotation-free framework designed specifically for the MCs detection and trained in a self-supervised manner thanks to the generation of synthetic MCs. Inspired by the UNet3+ architecture, we reduced its number of parameters to make it applicable in practice and added multi-scale features to enrich fine-grained details with more global context information. Both multi-channel segmentation and multi-class classification tasks are implemented in a multi-scale output approach to catch MC of various sizes. We perform a comparison with several state-of-the-art methods, including different flavors of ResNet-22, ConvNeXt, and UNet3+. An analysis of classification and segmentation performances has been done, using the Gradient-weighted Class Activation Mapping method to make classifiers visually explainable. In this study, we used two public datasets, INBreast and Breast MicroCalcifications Dataset for validation and test purposes. We achieved an AUC score of 0.93 in the characterization of malignant MCs while having a semantic segmentation precision of 0.70. To the best of our knowledge, we are the first study claiming segmentation performances on the BMCD dataset.

Recent Foundation Models have begun to yield remarkable successes across various downstream medical imaging applications. Yet, their potential within the context of multi-view medical image analysis remains largely unexplored. This research aims to investigate the feasibility of leveraging foundation models for predicting breast cancer from multiview mammograms through parameter-efficient transfer learning (PETL). PETL was implemented by inserting lightweight adapter modules into existing pre-trained transformer models. During model training, the parameters of the adapters were updated while the pre-trained weights of the foundation model remained fixed. To assess the model's performance, we retrospectively assembled a dataset of 949 patients, with 470 malignant cases and 479 normal or benign cases. Each patient has four mammograms obtained from two views (CC/MLO) of both the right and left breasts. The large foundation model with 328 million (M) parameters, finetuned with adapters comprising only 3.2M tunable parameters (about 1% of the total model parameters), achieved a classification accuracy of 78.9% ± 1.7%. This performance was competitive but slightly inferior to a smaller model with 36M parameters, finetuned using traditional methods, which attained an accuracy of 80.4% ± 0.9%. The results suggest that while foundation models possess considerable potential, their efficacy in medium-sized datasets and in transitioning from single-view to multi-view image analysis, particularly where reasoning feature relationships across different mammographic views is crucial, can be challenging. This underscores the need for innovative transfer learning approaches to better adapt and generalize foundation models for the complex requirements of multi-view medical image analysis.

In the study, we first introduce a novel AI-based system (MOM-ClaSeg) for multiple abnormality/disease detection and diagnostic report generation on PA/AP CXR images, which was recently developed by applying augmented Mask RCNN deep learning and Decision Fusion Networks. We then evaluate performance of MOM-ClaSeg system in assisting radiologists in image interpretation and diagnostic report generation through a multi-reader-multi-case (MRMC) study. A total of 33,439 PA/AP CXR images were retrospectively collected from 15 hospitals, which were divided into an experimental group of 25,840 images and a control group of 7,599 images with and without processed by MOM-ClaSeg system, respectively. In this MRMC study, 6 junior radiologists (5~10yr experience) first read these images and generated initial diagnostic reports with/without viewing MOM-ClaSeg-generated results. Next, the initial reports were reviewed by 2 senior radiologists (>15yr experience) to generate final reports. Additionally, 3 consensus expert radiologists (>25yr experience) reconciled the potential difference between initial and final reports. Comparison results showed that usingMOM-ClaSeg, diagnostic sensitivity of junior radiologists increased significantly by 18.67% (from 70.76% to 89.43%, P<0.001), while specificity decreased by 3.36% (from 99.49% to 96.13%, P<0.001). Average reading/diagnostic time in experimental group with MOM-ClaSeg reduced by 27.07% (P<0.001), with a particularly significant reduction of 66.48% (P<0.001) on abnormal images, indicating that MOM-ClaSeg system has potential for fast lung abnormality/disease triaging. This study demonstrates feasibility of applying the first AI-based system to assist radiologists in image interpretation and diagnostic report generation, which is a promising step toward improved diagnostic performance and productivity in future clinical practice.

PURPOSE: Accurate preoperative planning is crucial for liver resection surgery due to the complex anatomical structures and variations among patients. The need of virtual resections utilizing deformable surfaces presents a promising approach for effective liver surgery planning. However, the range of available surface definitions poses the question of which definition is most appropriate. METHODS: The study compares the use of NURBS and B´ezier surfaces for the definition of virtual resections through a usability study, where 25 participants (19 biomedical researchers and 6 liver surgeons) completed tasks using varying numbers of control points driving surface deformations and different surface types. Specifically, participants aim to perform virtual liver resections using 16 and 9 control points for NURBS and B´ezier surfaces. The goal is to assess whether they can attain an optimal resection plan, effectively balancing complete tumor removal with the preservation of enough healthy liver tissue and function to prevent postoperative liver dysfunction, despite working with fewer control points and different surface properties. Accuracy was assessed using Hausdorff distance and average surface distance. A survey based on the NASA Task Load Index measured user performance and preferences. RESULTS: NURBS surfaces exhibit improved accuracy and consistency over B´ezier surfaces, with lower average surface distance and variability of results. The 95th percentile Hausdorff Distance indicates the robustness of NURBS surfaces for the task. Task completion time was influenced by control point dimensions, favoring NURBS 3x3 (vs. 4x4) surfaces for a balanced accuracy-efficiency trade-off. Finally, the survey results indicated participants preferred NURBS surfaces over B´ezier, emphasizing the improved performance, surface manipulation, and reduced effort. CONCLUSION: The integration of NURBS surfaces into liver resection planning offers a promising advancement. This study demonstrates their superiority in accuracy, efficiency, and user preference compared to B´ezier surfaces. The findings underscore the potential of NURBS-based preoperative planning tools to enhance surgical outcomes in liver resection procedures.

Coronary artery calcification (CAC) is a strong and independent predictor of cardiovascular disease (CVD). However, manual assessment of CAC often requires radiological expertise, time, and invasive imaging techniques. The purpose of this multicenter study is to validate an automated cardiac plaque detection model using a 3D multiclass nnU-Net for gated and non-gated non-contrast chest CT volumes. CT scans were performed at three tertiary care hospitals and collected as three datasets, respectively. Heart, aorta, and lung segmentations were determined using TotalSegmentator, while plaques in the coronary arteries and heart valves were manually labeled for 801 volumes. In this work we demonstrate how the nnU-Net semantic segmentation pipeline may be adapted to detect plaques in the coronary arteries and valves. With a linear correction, nnU-Net deep learning methods may also accurately estimate Agatston scores on chest non-contrast CT scans. Compared to manual Agatson scoring, automated Agatston scoring indicated a slope of the linear regression of 0.841 with an intercept of +16 HU (R² = 0.97). These results are an improvement over previous work assessing automated Agatston score computation in non-gated CT scans.

Purpose: To rule out hemorrhage, non-contrast CT (NCCT) scans are used for early evaluation of patients with suspected stroke. Recently, artificial intelligence tools have been developed to assist with determining eligibility for reperfusion therapies by automating measurement of the Alberta Stroke Program Early CT Score (ASPECTS), a 10-point scale with > 7 or ≤ 7 being a threshold for change in functional outcome prediction and higher chance of symptomatic hemorrhage, and hypodense volume. The purpose of this work was to investigate the effects of CT reconstruction kernel and slice thickness on ASPECTS and hypodense volume. Methods: The NCCT series image data of 87 patients imaged with a CT stroke protocol at our institution were reconstructed with 3 kernels (H10s-smooth, H40s-medium, H70h-sharp) and 2 slice thicknesses (1.5mm and 5mm) to create a reference condition (H40s/5mm) and 5 non-reference conditions. Each reconstruction for each patient was analyzed with the Brainomix e-Stroke software (Brainomix, Oxford, England) which yields an ASPECTS value and measure of total hypodense volume (mL). Results: An ASPECTS value was returned for 74 of 87 cases in the reference condition (13 failures). ASPECTS in non-reference conditions changed from that measured in the reference condition for 59 cases, 7 of which changed above or below the clinical threshold of 7 for 3 non-reference conditions. ANOVA tests were performed to compare the differences in protocols, Dunnett’s post-hoc tests were performed after ANOVA, and a significance level of p < 0.05 was defined. There was no significant effect of kernel (p = 0.91), a significant effect of slice thickness (p < 0.01) and no significant interaction between these factors (p = 0.91). Post-hoc tests indicated no significant difference between ASPECTS estimated in the reference and any non-reference conditions. There was a significant effect of kernel (p < 0.01) and slice thickness (p < 0.01) on hypodense volume, however there was no significant interaction between these factors (p = 0.79). Post-hoc tests indicated significantly different hypodense volume measurements for H10s/1.5mm (p = 0.03), H40s/1.5mm (p < 0.01), H70h/5mm (p < 0.01). No significant difference was found in hypodense volume measured in the H10s/5mm condition (p = 0.96). Conclusion: Automated ASPECTS and hypodense volume measurements can be significantly impacted by reconstruction kernel and slice thickness.

Glioblastoma (GB) tumors are highly aggressive brain tumors with a dismal prognosis of 12-15 months. A significant challenge in the treatment of GB tumors is the differentiation of true tumor progression (TP) from pseudoprogression (PsP), a radiation-induced treatment effect that mimics the appearance of tumor recurrence on clinical MRI scans. While a few approaches have focused on capturing textural differences in image intensities (i.e. radiomics) on MRI to distinguish TP from PsP, in this work, we present a novel approach that seeks to capture Spatial Interactions in image intensities via Graph-based Learning (SIGL) from within and around the lesion to distinguish PsP from TP. Given that the tumor micro-environments across PsP (a benign lesion) and TP (an aggressive phenotype) are fundamentally different, we hypothesize that the spatial interactions in intensity composition within the lesion and its surrounding area, as captured through connected graph features, will be different and thus can distinguish between TP and PsP. Our retrospective analysis was conducted on a total of n = 105 GB patients curated from the Cleveland Clinic (n = 59) and Dana-Farber Cancer Center (n=46). For the purposes of model development and validation, out of the 105 patients, 72 (TP; n = 48, PsP; n = 24) were utilized for training, while the remaining 33 (TP; n = 21, PsP; n = 12) were allocated for testing. Pre-processing, including tumor segmentation and intensity normalization, was conducted, followed by the construction of a weighted tumor graph. A total of 10 graph-based metrics were initially extracted, and a feature selection method was employed to refine this dataset. The selected features were then fed into a Random Forest (RF) classifier to distinguish between True Progression (TP) and Pseudo Progression (PsP). Statistical analyses showed significant differences in 8 of the 10 SIGL features, including size, diameter, radius, average shortest path length, density, small-worldness, number of connected components, and clustering coefficient, between TP and PsP patients. The RF classifier demonstrated an 80% 10-fold cross-validation accuracy and 78% accuracy on the test set using the SIGL features. Preliminary results suggest that the graph-based features may be able to capture the inherent differences in lesion composition between TP and PsP, in GB tumors.

Alzheimer’s Disease (AD) is one of the most important neurodegenerative diseases, both difficult to diagnose and not curable. Based on multimodal data from the Alzheimer’s Disease Neuroimaging Initiative, we implemented a longitudinal continuous model able to characterize patient trajectories and we use it to predict AD development for the next 3 years based on 6 observations. In our experiments, the proposed approach reached a 0.881 prediction accuracy, illustrating the potential of the proposed approach for diagnosis and prognosis.

Alzheimer's disease (AD), a prevalent neurodegenerative disorder, is influenced by an intricate mix of risk factors including age, genetics, and environmental variables. In our study, we employed mouse models with human APOE alleles and nitric oxide synthase 2, along with environmental factors like diet, to simulate controlled genetic risk and immune response of AD. We utilized a Feature Attention Graph Neural Network (FAGNN), integrating brain structural connectomes, genetic traits, environmental factors, and behavioral data, to estimate brain age. Our method demonstrated improved accuracy in age prediction over other methods and highlighted age-associated brain connections. The most impactful connections included the cingulum, striatum, corpus callosum, and hippocampus. We further investigated these findings through fractional anisotropy in different age groups of mice which underlined the significance of white matter degradation in the aging process. Our research underscores the effectiveness of integrative graph neural networks in predicting brain age and delineating important neural connectivity in brain aging.

The detection of anatomical structures in medical imaging data plays a crucial role as a preprocessing step for various downstream tasks. It, however, poses a significant challenge due to highly variable appearances and intensity values within medical imaging data. In addition, there is a scarcity of annotated datasets in medical imaging data, due to high costs and the requirement for specialized knowledge. These limitations motivate researchers to develop automated and accurate few-shot object detection approaches. While there are generalpurpose deep learning models available for detecting objects in natural images, the applicability of these models for medical imaging data remains uncertain and needs to be validated. To address this, we carry out an unbiased evaluation of the state-of-the-art few-shot object detection methods for detecting head and neck anatomy in CT images. In particular, we choose Query Adaptive Few-Shot Object Detection (QA-FewDet), Meta Faster R-CNN, and Few-Shot Object Detection with Fully Cross-Transformer (FCT) methods and apply each model to detect various anatomical structures using novel datasets containing only a few images, ranging from 1- to 30-shot, during the fine-tuning stage. Our experimental results, carried out under the same setting, demonstrate that few-shot object detection methods can accurately detect anatomical structures, showing promising potential for integration into the clinical workflow.

Large-scale population studies have examined the detection of sinus opacities in cranial MRIs. Deep learning methods, specifically 3D convolutional neural networks (CNNs), have been used to classify these anomalies. However, CNNs have limitations in capturing long-range dependencies across the low and high level features, potentially reducing performance. To address this, we propose an end-to-end pipeline using a novel deep learning network called ConTra-Net. ConTra-Net combines the strengths of CNNs and self-attention mechanisms of transformers to classify paranasal anomalies in the maxillary sinuses. Our approach outperforms 3D CNNs and 3D Vision Transformer (ViT), with relative improvements in F1 score of 11.68% and 53.5%, respectively. Our pipeline with ConTra-Net could serve as an alternative to reduce misdiagnosis rates in classifying paranasal anomalies.

Xerostomia is a common toxicity for patients with head and neck cancer (HNC) treated with radiation therapy (RT) with significant potential to impact patient quality of life. In this study, we propose a deep learning model that predicts whether the patient will experience xerostomia 3-6 months after RT. A 3D residual network is designed to predict xerostomia at 3- 6 months post-RT from RT planning CT data. We hypothesize that self-supervised contrastive pre-training that forces the network to learn invariant features from data samples after different data augmentations, as well as samples with the same labels, can effectively overcome the issue of scarce data to improve xerostomia prediction. Furthermore, during training, a side branch that outputs latent embeddings is optimized to cluster samples with the same label. The cluster centers are updated using the exponential moving average method to better fit the samples. The Euclidean distances between the latent embeddings and the cluster centers are added to the classification logits to guide the classification for stronger supervision and generalization. The xerostomia prediction model was trained and tested on 500 HNC patient data, and achieved mean±SD AUC, sensitivity, specificity, negative predictive value, precision and accuracy of 0.80±0.03, 0.67±0.13, 0.71±0.10, 0.75±0.05, 0.65±0.07, 0.70±0.03, respectively. Our results suggest that the developed model is a promising approach for predicting the occurrence of xerostomia 3-6 months post-RT. Furthermore, the supervised contrastive learning as well as the proposed cluster-guided loss are powerful tools for improving the model’s generalizability in predicting xerostomia.

Medical segmentation of optical coherence tomography (OCT) images using deep neural networks (DNNs) has been intensively studied in recent years, but generalization across datasets from different OCT devices is still a considerable challenge. In this work, we focus on the novel self-examination low-cost full-field (SELFF)- OCT, a handheld imaging device for home-monitoring of retinopathies, and the clinically used Spectralis-OCT. Images from both devices exhibit different characteristics, leading to different representations within DNNs and consequently to a reduced segmentation quality when switching between devices. To robustly segment OCT images from an OCT-scanner unseen during training, we alter the appearance of the images using manipulation methods ranging from traditional data augmentation to noise-based methods to learning-based style transfer methods. We evaluate the effect of the manipulation methods with respect to segmentation quality and changes in the feature space of the DNN. Reducing the domain shift with style transfer methods results in a significantly better segmentation of pigment epithelial detachment (PED). Investigations of the feature space show that the segmentation quality of PED is negatively correlated with the distance between training and test distributions. Our methods and results help researchers to choose and evaluate image manipulation methods for developing OCT segmentation models which are robust against domain shifts.

Purpose: This work aims to automatically identify the fovea on 2-dimensional fundus-autofluorescence (FAFs) images in patients with age-related-macular-degeneration (AMD) using the definitions from 3-dimensional spectral-domain-optical-coherence-tomography (SD-OCT) imaging. Segmenting the fovea, a highly specialized area of the retina, in vicinity of hypo-autofluorescence in FAF images will aid in objective evaluation of AMD based structural disease features with respect to distance from fovea. Methods: A semi-automated software was used to create fovea-annotations in volumetric SD-OCT images. Acquired FAF images for the same SD-OCT visits were registered to the enface SD-OCT projections, to create a pixel-to-pixel overlap between registered FAFs and SD-OCTs. A U-Net based segmentation network, trained using OCT-registered-FAFs and corresponding foveal-annotations from SD-OCTs, was used to automatically segment foveas from the registered 2D FAF images. Results: The dataset consisted of multimodal-images from AMD patients with 900 (80%) images used for training and 222 (20%) images used in the test-set. The mean euclidean-distance-error for the test-set w.r.t the OCT-determined-ground-truth was found to be 103.5±81.4 µm, and which improved to 83.4±57.9 µm with data-augmentation-based-training. Fovea-identification in FAF images with advanced-AMD disease consisting of geographic-atrophy (GA) test subset were compared between 3 sources and the OCTdetermined-ground-truth: (1) the U-Net algorithm (using the GA test subset (111.7±46.7 μm)), (2) readers at the Wisconsin-reading-center (165±77.5 μm) and a (3) retina-physician (169.9±109.4 μm). Conclusion: Our work demonstrates the potential of using 2D FAF images to predict foveal-locations, especially in visuallychallenging-scenarios where hypo-autofluorescent fovea is surrounded with advanced-disease that alters the normal autofluorescence patterns. The results demonstrate that the developed algorithm has clinically useful performance in segmenting the fovea in FAF images which will enable critical correlation with visual-acuity and the basis for referencing the standardized measures of features relative to the fovea – such as monitoring and tracking the growth of GA and other retina-disease related changes.

This study is an initial investigation into methods to harmonize quantitative imaging (QI) feature values across CT scanners based on image quality metrics. To assess the impact of harmonization on QI features, we: (1) scanned an image quality assessment phantom on three scanners over a wide range of acquisition and reconstruction conditions; (2) from those scans, assessed image quality for each scanner at each acquisition and reconstruction condition; (3) from these assessments, identified a set of parameters for each scanner that yielded similar image quality values (“harmonized condition”); (4) scanned a second phantom with texture (i.e., local variations in attenuation) under the same set of conditions; and (5) extracted QI features and compared values between non-harmonized and harmonized image quality conditions. Quantitative image quality assessments provided contrast to noise ratio (CNR) and modulation transfer function frequency at 50% (MTF f50) values for each scanner and each condition used. A set of harmonized conditions was identified across three CT scanners based on the similarity of CNR and MTF f₅₀. To provide a comparison, several non-harmonized condition sets were identified. From the texture phantom, the standard deviation of the QI feature values (intensity mean and variance, GLCM autocorrelation and cluster tendency, GLDM high and low gray level emphasis) across the three CT systems decreased between 72.8% and 81.1% between the unharmonized and harmonized groups (with exception of intensity mean which showed little difference across scanners). These initial results suggest that selecting protocols that produce similar quantitative image quality metric values across different CT systems can reduce the variance of QI feature values across those systems.

Radiomics involves the quantitative analysis of medical images to provide useful information for a range of clinical applications including disease diagnosis, treatment assessment, etc. However, the generalizability of radiomics model is often challenged by undesirable variability in radiomics feature values introduced by different scanners and imaging conditions. To address this issue, we developed a novel dual-domain deep learning algorithm to recover ground truth feature values given known blur and noise in the image. The network consists of two U-Nets connected by a differentiable GLCM estimator. The first U-Net restores the image, and the second restores the GLCM. We evaluated the performance of the network on lung CT image patches in terms of both closeness of recovered feature values to the ground truth and accuracy of classification between normal and COVID lungs. Performance was compared with an image restoration-only method and an analytical method developed in previous work. The proposed network outperforms both methods, achieving GLCM with the lowest mean-absolute-error from ground truth. Recovered GLCM feature values from the proposed method, on average, is within 2.19% error to the ground truth. Classification performance using recovered features from the network closely matches the “best case” performance achieved using ground truth feature values. The deep learning method has been shown to be a promising tool for radiomics standardization, paving the way for more reliable and repeatable radiomics models.

Acute stroke is a systemic disease that leads to rapid cellular damage and functional impairments throughout brain regions, even those remote from the site of injury. It triggers widespread effects through various mechanisms, such as secondary degeneration of neurons, leading to diverse neurological deficits. Advances in machine learning approaches have shown promise towards predicting individualized outcomes at an early stage, through identifying imaging biomarkers of cognitive deficits. This ultimately allows for providing personalized treatment plans and early interventions to slow cognitive decline. In this study, we attempt to predict associations between the verbal fluency scores (VFS) and the brain regions with sub-clinical phenotypes of patients with acute stroke using a novel radiomic approach. Specifically, we extract radiomic features from T1-weighted (T1w) images that capture the structural alterations occurring in the brain regions due to stroke lesions. Our work explores the hypothesis that the degeneration occurring in brain regions, hence leading to cognitive impairment, can be quantified on routine imaging via radiomic features that capture the structural deformations resulting from stroke lesions. A retrospective analysis was conducted on the MRI scans (T1-weighted (T1w), FLAIR, Diffusion Weighted Imaging, and Apparent Diffusion Coefficient maps) of n = 31 subjects. Acute lesions ( < 7 days) were manually segmented, followed by registering the T1w scan of each subject to an MNI atlas via deformable registration. Structural deformations magnitudes were then obtained via the inverse-warp transformations of the diffeomorphic registration. These deformation magnitudes were used to extract descriptive statistics (mean, skewness, and kurtosis) for 117 brain regions parcellated using an MNI-based anatomical atlas. The statistics were correlated with the measured normalized VFS of each subject using Pearson’s correlation coefficient. Further, since the cerebral regions of the left hemisphere are known to play a key role in speech, the group of subjects with only left hemispheric lesions (LHL) (n = 15) was analyzed separately. Our analysis demonstrated that key bilateral verbal regions in the brains of subjects with LHL are significantly correlated with the deformation features, such as Right Frontal Inferior Opercularis (skewness r = -0.67, p = 0.007), Right Temporal Inferior (mean r = 0.64, p = 0.009) and Superior (kurtosis r = -0.69, p = 0.004), and Cingulum Posterior Left and Right (mean r = 0.72, p = 0.003 and r = 0.73, p = 0.002). These results suggest that radiomic features capturing structural alterations occurring due to lesions can identify the brain regions affected at the early stages of stroke.

Diabetes is a disease that affects hundreds of millions of people worldwide. While many people are diagnosed via a blood test during a routine visit, millions of people go undiagnosed due to inability to access primary care. Opportunistic screening of CT imaging offers an opportunity to predict and diagnose diabetes on imaging, opening an avenue to decrease the undiagnosed rates. Current literature in utilizing imaging for the prediction of diabetes from CT scans has shown that markers of adiposity are strong predictors of diabetes; however, we aim to determine if the pancreas alone is sufficient to diagnose diabetes. We trained a neural network to predict and diagnose diabetes from abdominal CT scans using just a segmentation of the pancreas. We found that we were able to match the performance of state of the art algorithms using less information and we could predict diabetes status in patient populations without comorbid markers of increased fat. This study opens the opportunity for further analysis of deep learning derived imaging biomarkers in the assessment of diabetes status and disease course.

Glioblastoma (GBM) is the most common and aggressive brain tumor with short overall survival (OS) of about 15 months. Understanding the causal factors affecting the patient survival is crucial for disease prognosis and treatment planning. Although previous efforts on survival prediction using multi-omics data has yielded useful predictive models, the causation of the correlated genetic risk factors has not been addressed. Recent advances in causal deep learning models enable the study of causality from complex dataset. In this paper, we leverage the recently proposed structural causal model (SCM) with normalizing flows parameterized by deep networks to perform the counterfactual query to investigate the causal relationship between gene mutation and OS with the presence of other confounders including sex, age and radiomics features. The query amounts to the question that what the survival days will be if the gene mutation status has been changed, i.e., from mutant to non-mutant and vice versa. The trained causal model will infer the counterfactual outcome given the intervention on specific gene mutation. We apply multivariate Cox-PH model to find the genes associated with survival, and investigate the causal genetic effect by comparing the original and counterfactual survival days in a bi-directional fashion. Particularly, the following two scenarios are considered: (1) intervention on a specific gene with non-mutant status to generate the counterfactual survival days as if the gene is mutant, with which the original survival days of the subjects with that mutant gene will be compared; (2) intervention on the gene with mutant status and perform the comparison with survival days of subjects with that non-mutant gene. Our experimental results show that no causation of two correlated genes (NF1, RB1) was revealed in the cohort (n=181), while their genetic effects on OS in terms of prolonging or shortening are generally in accordance with clinical findings.

Intravenous contrast phase classification can improve the data curation of CT scans for medical AI applications. In our early work, a five-phase classification model based on ResNet was developed to serve this purpose. It can accurately predict CT phases if the testing data follows the data distribution of the training data, while its performance drops substantially on the out-of-distribution data. To address this issue, this work aims to incorporate the inherent domain knowledge related to different CT phases for their classification. We explore the intensity distributions of a few key organs in different CT phases. TotalSegmentator was used to segment these organs including pulmonary artery, heart atrium, heart ventricle, heart myocardium, aorta, portal vein, splenic vein, liver, kidney, inferior vena cava, iliac artery, iliac veins, and urinary bladder. The intensity information was then extracted as image features to train a random forest classification model. The classification models were trained on an in-house dataset of 252 CT scans. They were validated on a testing dataset with 213 outof-distribution CT scans. The proposed method achieved better accuracy of 62.44%, while it was 53.02% using ResNet. These results showed that embedding domain knowledge could improve CT phase classification to be robust to the out-of-distribution dataset.

Proper assessment of the ability of an artificial intelligence (AI)-enabled medical device to generalize to new patient populations is necessary to determine the safety and effectiveness of the device. Assessing AI generalizability relies on performance assessment on a data set which represents the device’s intended population, which can be challenging to obtain. An understanding of the AI model’s decision space can indicate how the device is likely to perform on patients not represented in the available data. Our tool for decision region analysis for generalizability (DRAGen) assessment estimates the composition of the region of the decision space surrounding the available data. This provides an indication of how the model is likely to perform on samples which are similar to, but not represented by, the available finite data set. DRAGen can be applied to any binary classification model and requires no knowledge of the model’s training process. In a case study, we demonstrated DRAGen on a COVID classification model and showed that the decision region composition can identify differences in correct classification rates between the positive and negative classes, even with comparable performance on the original test set. Performance evaluation using a data set which was not represented during model development nor within the original test set shows a disparity in the performance between COVID-positive and COVID-negative patients, as indicated by DRAGen. By releasing this tool, we encourage future AI developers to use our tool to improve understanding of generalizability.

Chest X-rays (CXRs) play a pivotal role in cost-effective clinical assessment of various heart and lung related conditions. The urgency of COVID-19 diagnosis prompted their use in identifying conditions like lung opacity, pneumonia, and acute respiratory distress syndrome in pediatric patients. We propose an AI-driven solution for binary COVID-19 versus non-COVID-19 classification in pediatric CXRs. We present a Federated Self-Supervised Learning (FSSL) framework to enhance Vision Transformer (ViT) performance for COVID-19 detection in pediatric CXRs. ViT's prowess in vision-related binary classification tasks, combined with self-supervised pre-training on adult CXR data, forms the basis of the FSSL approach. We implement our strategy on the Rhino Health Federated Computing Platform (FCP), which ensures privacy and scalability for distributed data. The chest X-ray analysis using the federated SSL (CAFES) model, utilizes the FSSL-pre-trained ViT weights and demonstrated gains in accurately detecting COVID-19 when compared with a fully supervised model. Our FSSL-pre-trained ViT showed an area under the precision-recall curve (AUPR) of 0.952, which is 0.231 points higher than the fully supervised model for COVID-19 diagnosis using pediatric data. Our contributions include leveraging vision transformers for effective COVID-19 diagnosis from pediatric CXRs, employing distributed federated learning-based self-supervised pre-training on adult data, and improving pediatric COVID-19 diagnosis performance. This privacy-conscious approach aligns with HIPAA guidelines, paving the way for broader medical imaging applications.

The increased use of artificial intelligence (AI)-enabled medical devices in clinical practice has driven the need for better understanding of bias in AI applications. While there are many bias mitigation methods, determining which method is appropriate in each situation is not trivial due to a lack of objective bias mitigation assessment approaches. This study presents an approach to manipulate sources of bias in order to facilitate the evaluation of bias mitigation methods during AI development. This approach amplifies sources of bias in the data by varying disease prevalence between subgroups to promote associations between patient subgroups and specific disease classes. The approach can be adjusted to affect the degree to which bias is amplified. In this study, bias amplification was used to vary the bias of COVID-19 classification models using chest X-ray data between patient subgroups defined by sex (female or male) or race (black or white). Analysis of subgroup sensitivity shows that the proposed bias amplification approach results in the sensitivity of one subgroup increasing while the sensitivity of the other subgroup is decreasing, despite a consistent difference in subgroup area under the receiver operating characteristic curve. For example, the amplification of sex-related bias increased the sensitivity for COVID-19 classification of the female subgroup from 0.65±0.04 to 0.90±0.02 while decreasing the sensitivity of the male subgroup from 0.66±0.03 to 0.18±0.02. This shows that the bias amplification approaches promote bias by causing the model to enhance existing correlations between a patient’s subgroup and their COVID-19 status, which results in biased performance in a clinical setting. The approach presented in this study was found to systematically amplify sources of bias for our dataset and AI model, and can thus be used to evaluate the effectiveness of potential bias mitigation methods.

The assessment of lymph node metastases is critical for accurate cancer staging and consequently the decision for treatment options. Lymph node staging is a challenging, time-consuming task due to the fact that lymph nodes have ill-defined borders as well as varying sizes and morphological characteristics. The purpose of this study is to evaluate the effects of using different anatomical priors with the aim of guiding network attention within the application of segmentation of pathological lymph nodes in the mediastinum. The first presented prior, a distance map, displays the distance to a commonly defined point across all patients and, thus, provides an orientation of where a patch is extracted from. The second prior option, a probabilistic lymph node atlas, provides a map of areas where healthy and pathological lymph nodes are located, but also highlights lymph node stations that are more likely to become malignant. The distance map as well as the probabilistic lymph node atlas are results of an upstream atlas-to-patient registration approach. The third prior is a combination of segmentation masks of anatomical structures generated by the TotalSegmentator algorithm. A paired t-test on 5-fold cross validated results shows no significant differences in Dice score between models trained with the distance map or/and the probabilistic lymph node atlas compared to models trained with CT only. Counterintuitively, the models trained with segmentation masks of selected anatomical structures show significantly decreased segmentation accuracy. However, using the probabilistic lymph node atlas reduces the number of false negatives and consistently elevates the effect of post-processing.

Medical imaging-based machine learning (ML) for in vivo lesion diagnosis consists of two basic components, or modules, of (i) feature extraction from non-invasively acquired medical images and (ii) feature learning and classification for prediction of malignancy of lesions detected or localized in the medical images. This study investigates their individual performances for diagnosis of low-dose computed tomography screening-detected lesions of pulmonary nodules and colorectal polyps. Three feature extraction methods were investigated. One uses the gray-level co-occurrence image texture measure to extract the well-cited Haralick image texture features (HFs). One uses the convolutional neural network (CNN) to extract deep learning (DL) image characteristic features (DFs). The third one uses the interactions between tissues and X-ray energy to extract tissue-energy specific characteristic features (TFs). All three categories of features were classified by the well-cited Random Forest (RF) classifier with comparison to the baseline of DL-CNN, which reads the images, extracts DFs and classifies DFs for the prediction in an end-to-end manner. The lesion diagnosis or prediction of lesion malignancy was measured by the well-established area under the receiver operating characteristic curve (AUC). Three lesion image datasets were used. The lesions’ tissue pathological reports were used as machine learning labels. The experiments on the three datasets produced AUC values of 0.724 to 0.878 for the HFs, 0.652 to 0.965 for the DFs, and 0.985 to 0.996 for the TFs, compared to the baseline of DL-CNN with AUC from 0.694 to 0.964. These experimental outcomes indicate that the RF classifier performed comparably to the DL-CNN classification module. The extraction of tissue-energy specific characteristic features is more important than the extraction of the image characteristic features.

Deep learning models have shown potential in medical image analysis tasks. However, training a generalized deep learning model requires huge amounts of patient data that is usually gathered from multiple institutions which may raise privacy concerns. Federated learning (FL) provides an alternative to sharing data across institutions. Nonetheless, FL is susceptible to a few challenges including inversion attacks on model weights, heterogenous data distributions, and bias. This study addresses heterogeneity and bias issues for multi-institution patient data by proposing domain adaptive FL modeling using several radiomics (volume, fractal, texture) features for O6-methylguanine-DNA methyltransferase (MGMT) classification across multiple institutions. The proposed domain adaptive FL MGMT classification inherently offers differential privacy (DP) for the patient data. For domain adaptation two techniques e.g., mixture of experts (ME) with a gating network and adversarial alignment are used for comparison. The proposed method is evaluated using publicly available multi-institution (UPENN-GBM, UCSF-PDGM, RSNA-ASNR-MICCAI BraTS-2021) data set with a total of 1007 patients. Our experiments with 5-fold cross validation suggest that domain adaptive FL offers improved performance with a mean accuracy of 69.93% ± 4.8 % and area under curve of 0.655 ± 0.055 across multiple institutions. In addition, further analysis of probability density of gating network for domain adaptive FL identifies the institution that may bias the global model prediction due to increased heterogeneity for a given input. Our comparison analysis shows that the proposed method with bias identification offers the best predictive performance when compared to different commonly employed FL and baseline methods in the literature.

Skin cancers, notably melanoma, pose a significant health risk, with rising incidence rates and mortality rates. Early detection through screening is crucial, and the Department of Veterans Affairs has prioritized improved melanoma screening. Teledermatology, with digital dermatoscopy, offers a promising avenue for initial lesion assessment, prompting the exploration of algorithmic screening using deep learning. This paper investigates the efficacy of incorporating melanin index (MI) and erythema index (EI) as additional features, along with the conversion of images to the HSV color space, for use with deep learning models in melanoma classification. Considering the distinct clustering of human skin color in various color spaces, our study aims to explore the advantages of alternative color representations. Building on prior work and organized into two phases, our experiments utilize a diverse set of model architectures trained on images of varying sizes. Phase 1 aligns with the 2020 SIIM-ISIC Melanoma Classification Challenge, while Phase 2 involves an expanded dataset and more robust metrics. Despite optimistic outcomes in the earlier phase, our findings reveal no significant performance improvement when incorporating MI or EI in deep learning models for melanoma detection. This study contributes valuable insights for refining deep learning approaches in dermatoscopy, offering a cautionary note on the efficacy of specific features and of color space transformation.

When image data changes due to routine imaging machine updates, the performance of previously-trained deep learning (DL)-algorithms can be degraded. To mitigate the potential for performance degradation, we introduced an image-domain transfer approach by using a conditional Generative Adversarial Network (CGAN) that can transform the images acquired after the update to match previous images. Some studies have proposed domain adaptation (DA) method for DL algorithms from the old system to the new system. However, we sought to investigate the suitability of the DA method for transferring images from the current system to that of the previous system to adapt existing DL-algorithm in the current system. We validated our domain transfer approach using 1,000 DBT patch volumes (500 lesions; 500 normal) with two distinct image qualities under a virtual clinical trial framework. We curated two DBT image sets of breast patches with two distinct image qualities using different image reconstruction settings (simulating two different systems; e.g., previous and current). We then divided our data into training, validation, and testing set with a ratio of 0.8:0.1:0.1. Using the training set, we developed two CGANs (normal and lesion) for image-domain transformation from current to previous systems. We fine-tuned a DenseNet121 network as a reference classifier for classifying lesions vs. normal DBT patch volumes using the training set from the previous system. We evaluated our domain-transfer method by testing the reference model on the test sets in three qualities: a) previous (S_P), b) current (S_C), and c) domain-transferred images (DT_C2P). The performance of the reference model with an AUC of 1.0 on the previous images, was degraded to an AUC of 0.88 (S_C vs S_P: p < 0.005) on the current images, but restored its performance to an AUC of 0.97 on the DT-images (S_C vs DT_C2P: p < 0.005). This result demonstrated that our domain transfer approach effectively restores the reference model’s original performance on images with current quality.

Atherosclerosis of the carotid artery is a major risk factor for stroke. Current studies analyze cross-sections of 3D MR black-blood images to assess the vessel wall of carotid arteries. To increase the reproducibility of quantitative biomarkers such as vessel wall thickness and radiomic features, a reliable automatic segmentation of the vessel wall in these cross-sections is essential. CNN-based segmentation is well established and has been successfully applied for 2D vessel wall and plaque segmentation. We trained a residual U-Net on sparsely sampled cross-sections that are perpendicular to the vessel’s centerline, making our method invariant to the image plane orientation. Due to the well curated training data and the usage of the vessel’s centerline as anatomical prior we are able to achieve a high mean Dice coefficient of 0.946/0.864 for the vessel’s lumen/wall and low mean average contour distance of 0.100/0.116 mm. To prove the model’s flexibility, we show that it is able to segment regions of the carotid artery that are not incorporated in the training data, achieving a similar Dice coefficient, average contour distance and Hausdorff distance. This validates the potential of the method in accurately automating carotid artery wall segmentation for any vessel cross-section. The model is also evaluated on young, healthy subjects and the 2021 Carotid Artery Vessel Wall Segmentation Challenge test set, proving its versatility.

In 2020, prostate cancer (PCa) caused about 6.8% of male cancer-related deaths. Thus, early detection of PCa and improving clinical procedures for suspected cases are crucial. Fortunately, using biparametric magnetic resonance imaging (bpMRI) along with advanced computer-aided diagnosis (CAD) systems that involve deep learning (DL) techniques leads to PCa detection rates comparable to radiologists. Achieving this requires access to a large amount of patient data; however, sharing patient-specific information raises privacy concerns. To address this, we present a Flower federated learning (FL) framework that utilizes local DL models based on nnU-Net for PCa detection. We investigated the impact of various epoch-round combinations and server-side aggregation strategies on the performance of the FL framework and compared the results with centralized learning (CL). We performed empirical experiments using the Prostate Imaging Cancer AI (PI-CAI) dataset comprising 1500 patients from three different institutions for training/validation/testing and further evaluated the results on an independent test set of 199 patients from a fourth institution. The trained models were evaluated based on the PI-CAI score, which combines patient-level diagnostic performance (area under the receiver operating characteristic) and lesion-level detection performance (average precision). We found that FL can replicate the performance of CL and substantially improve the detection of PCa compared to local training approaches. We optimized the combination of epochs and rounds, along with a server-side aggregation strategy, for prostate lesion detection on the validation set. This configuration (5 epoch, 200 rounds, FedAdagrad aggregation) led to an enhanced PI-CAI score of 0.74 on the test set, compared to 0.73 and 0.72 for the CL and FL baselines, respectively.

Near-infrared hyperspectral imaging (NIR-HSI) is well known that it enables chemical composition analysis with high bio-transparency and high spatial resolution. Thus, hyperspectral imaging is potential in noninvasive and label-free diagnosis of deep lesion by machine learning. In this study, detection of deep lesions such as Gastrointestinal Stromal Tumor (GIST) and Gastric Cancer (GC) including unexposed areas was investigated using NIR-HSI. As the result, although GIST specimens had a normal mucosal layer covering the lesion, NIR-HSI analysis by machine learning showed an average prediction accuracy of 86.1%. In case of GC specimens, average prediction accuracy of GC regions in all area, exposed area and unexposed area were 79.9%, 80.9% and 77.8%, respectively.

Analyzing the anatomy of the aorta and left ventricular outflow tract (LVOT) is crucial for risk assessment and planning of transcatheter aortic valve implantation (TAVI). We propose 2D cross-sectional annotation and point cloud-based surface reconstruction to train a fully automatic 3D segmentation network for the aortic root and the LVOT. Our sparse annotation scheme enables easy and fast training data generation for tubular structures like the aortic root. Based on this annotation concept, we trained a 3D segmentation model that achieves a Dice similarity coefficient (DSC) of 0.9 and an average surface distance (ASD) of 0.96 mm. In addition, we show that our fully automatic segmentation approach facilitates reproducible and quantifiable measurements for TAVI planning. Our approach achieves an aortic annulus maximum diameter difference between prediction and annotation of 0.45 mm (inter-observer variance: 0.25 mm).

Prostate multiparametric magnetic resonance imaging (mpMRI) has demonstrated promising results in prostate cancer (PCa) detection using deep learning models using convolutional neural networks (CNNs). Recently, transformers have achieved competitive performance compared to CNNs in computer vision. Large-scale transformers benefit from training with large-scale annotated data, which are expensive and labor-intensive to obtain in medical imaging. Self-supervised learning can effectively leverage unlabeled data to extract useful semantic representations with no additional annotation cost. This can improve model performance on downstream tasks with limited labelled data and increase model robustness to external data. We present a novel end-to-end cross-shaped transformer model CSwin UNet to detect clinically significant prostate cancer (csPCa) in prostate bi-parametric MR imaging (bpMRI). Using a large prostate bpMRI dataset with 1500 patients, our Cswin UNet achieves 0.880±0.013 AUC and 0.790 ±0.033 pAUC, significantly outperforming state-of-the-art CNN and transformer models.

Automated detection and aggressiveness classification of prostate cancer on Magnetic Resonance Imaging (MRI) can help standardize radiologist interpretations, and guide MRI-Ultrasound fusion biopsies. Existing automated methods rely on MRI features alone, disregarding histopathology image information. Histopathology images contain definitive information about the presence, extent, and aggressiveness of cancer. We present a two-step radiology-pathology fusion model, ArtHiFy, Artificial Histopathology-style Features for improving MRI-based prostate cancer detection, that leverages generative models in a multimodal co-learning strategy, enabling learning from resource-rich histopathology, but prediction using resource-poor MRI alone. In the first step, ArtHiFy generates artificial low-resolution histopathology-style features from MRI using a modified Geometry-consistent Generative Adversarial Network (GcGAN). The generated low-resolution histopathology-style features emphasize cancer regions as having less texture variations, mimicking densely packed nuclei in real histopathology images. In the second step, ArtHiFy uses these generated artificial histopathology-style features in addition to MR images in a convolutional neural network architecture to detect and localize aggressive and indolent prostate cancer on MRI. ArtHiFy does not require spatial alignment between MRI and histopathology images during training, and it does not require histopathology images at all during inference, making it clinically relevant for MRI-based prostate cancer diagnosis in new patients. We trained ArtHiFy using prostate cancer patients who underwent radical prostatectomy, and evaluated it on patients with and without prostate cancer. Our experiments showed that ArtHiFy improved prostate cancer detection performance over existing top performing prostate cancer detection models, with statistically significant differences.

We are developing five-year survival prediction models for bladder cancer patients who underwent neoadjuvant chemotherapy and radical cystectomy. This study investigated the feasibility of using large language models (Vicuna and Dolly) to extract clinical descriptors from reports for survival prediction with a nomogram model, and with or without further combining with radiomics and deep-learning descriptors from CTU images using BPNNs. The models were developed and validated using data of 163 patients collected with IRB approval. The developed models included C (based on clinical descriptors and nomogram), R (radiomics descriptors), D (deep-learning descriptor), CR (clinical and radiomics descriptors), CD (clinical and deep-learning descriptors), and CRD (clinical, radiomics, and deep-learning descriptors). The developed models achieved the following AUCs on test set: 0.82±0.06 (C: manually labeled reference), 0.73±0.07 (R), and 0.71±0.07 (D), 0.80±0.06 (C: User1 Vicuna-C2 labeled), 0.83±0.05 (C: User1 Dolly labeled), 0.78±0.06 (C: User2 Vicuna-C2 labeled), and 0.85±0.05 (C: User2 Dolly-C2 labeled). For the combined models, the AUCs were (1) manually labeled reference: 0.86±0.05 (CR), 0.86±0.05 (CD), and 0.87±0.05 (CRD), (2) CRD performance on Vicuna-C2 labeled: 0.86±0.05 (User1) and 0.84±0.05 (User2); (3) CRD performance on Dolly-C2 labeled: 0.88±0.05 (User1) and 0.89±0.04 (User2). The results showed that the LLMs extracted three clinical descriptors with accuracy ranging from 77% to 100% relative to manual extraction, and the LLMs run by two users had similar performance. The combined models outperformed individual models, and using LLM-extracted clinical descriptors achieved similar performance as manually extracted descriptors.

Our goal was to create a deep network-based lesion detection algorithm for low dose dynamic contrast-enhanced MRI (DCE-MRI) breast images, using Radon Cumulative Distribution Transform (RCDT) to highlight subtle enhancement. We had a dataset of 11 enhancing lesions in eight women with suspected fibroadenomas who underwent a dual-dose DCEMRI protocol on a 3T Philips scanner. To overcome the data limitation, we used a domain-transfer approach, training the YOLOv5 detection model on a publicly available Duke DCE-MRI dataset of 922 biopsy-confirmed invasive breast cancer cases acquired using Siemens or GE scanners. The training data included 23,426 pre-contrast slices with corresponding post-contrast slices and biopsy-proven lesions. The dataset was split into a training set (830 women) and a validation set (92 women). We resized all slices to 400 x 400 pixels and applied RCDT on pre- and post-contrast pairs to highlight lesion enhancement. By combining RCDT images with pre- and post-contrast images, we created RGB images as input for our algorithm. The results were promising, with the algorithm successfully detecting a total of 6 lesions in both regular and low-dose slices, 3 lesions only in regular dose, and 1 lesion only in low dose. However, it missed 1 lesion in both regular and low-dose images. Our study demonstrated the feasibility of a domain-transferred and RCDT-assisted lesion detection algorithm for low-dose MRI, even when data was acquired from scanners made by three different vendors.

The prevention and early detection of breast cancer hinges on precise prediction of individual breast cancer risk. Whilst well-established clinical risk factors can be used to stratify the population into risk groups, the addition of genetic information and breast density has been shown to improve prediction. Deep learning based approach have been shown to automatically extract complex information from images. However, this is a challenging area of research, partly due to the lack of data within the field, therefore there is scope for novel approaches. Our method uses Multiple Instance Learning in tandem with attention in order to make accurate, short-term risk predictions from full-sized mammograms taken prior to the detection of cancer. This approach ensures small features like calcifications are not lost in a downsizing process and the whole mammogram is analysed effectively. An attention pooling mechanism is designed to highlight patches of increased importance and improve performance. We also use transfer learning in order to utilise a rich source of screen-detected cancers and evaluate whether a model trained to detect cancers in mammograms allows us also to predict risk in priors. Our model achieves an AUC of 0.620 (0.585,0.657) in cancer-free screening mammograms of women who went on to a screen-detected or interval cancer between 5 and 55 months later, including for common breast cancer risk factors. Additionally, our model is able to discriminate interval cancers at an AUC of 0.638 (0.572, 0.703) and highlights the potential for such a model to be used alongside national screening programmes.

Estimating model uncertainty of artificial intelligence (AI)-based breast cancer detection algorithms could help guide the reading strategy in breast cancer screening. For example, the recall decision can be made solely by AI when it exhibits high certainty, while cases where the certainty is low should be read by radiologists. This study aims to evaluate two metrics to predict model uncertainty of a lesion characterization network: 1) the variance of a set of outputs generated with stochastic layer depth, and 2) the entropy of the average output. To test these approaches, 367 mammography exams with cancer (333 screen-detected, and 34 interval) and 367 cancer-negative exams from the Dutch Breast Cancer Screening Program were included. Using a commercial lesion detection algorithm operating at high sensitivity, 6,477 suspicious regions were included (14.1% labeled malignant). By varying the uncertainty threshold, the predictions were classified as certain or uncertain by a specified proportion. Radiologists double reading had a sensitivity of 90.9% (95% CI 89.0% – 92.7%) and a specificity of 93.8% (95% CI 93.2% – 96.2%) for all regions. At equal specificity, the network had a sensitivity of 92.1% (95% CI 89.9% – 94.0%) for all regions. The sensitivity of the network was higher for regions with low uncertainty for both approaches; for the top 50% most certain regions the sensitivity was 96.9% (95% CI 94.7% – 98.4%) and 97.1% (95% CI 94.9% – 98.8%) at equal specificity to radiologists. In conclusion, AI-based lesion classification uncertainty of breast regions can be estimated by applying stochastic layer depth during prediction.

Digital breast tomosynthesis (DBT), synthetic mammography, and full-field digital mammography (FFDM) are commonly used medical imaging modalities for breast cancer screening. Due to the data complexity, most CAD research applies to only one modality, which under-utilizes the complementary information in these 2D and 3D modalities. In this study, we propose a Residual-Attention Multimodal Fusion network (ResAMF-Net) that integrates lesion features across these modalities. We evaluated network performance on a large private dataset, which contains 769 cancer cases and 1375 noncancer cases (including 362 benign and 1013 normal cases) for a total of 2144 cases. At 90% case sensitivity, ResAMF-Net increases specificity by 8%, which can substantially improve radiologist workflow because almost all screening cases are true negatives.

Introduction: Breast cancer is the most common female cancer worldwide; however ethnic differences have been observed in both prevalence and prognosis, with Black women often having less favorable outcomes. Increased breast density is an independent risk factor for breast cancer and reduces the efficacy of mammographic screening. We investigate how it relates to ethnicity, to facilitate the provision of appropriate screening and advice to all women. Method: We use data from the UK Predicting Risk of Cancer at Screening (PROCAS) study. This involved completion of a questionnaire to obtain personal risk factor information during routine breast screening. Mammographic density was assessed using Visual Analogue Scales (VAS), and these scores were used to train an AI-based density measure, pVAS, which we applied to raw mammographic data from 41,241 women in PROCAS. Analysis of covariance was used to assess the relationship between ethnicity and breast density after adjusting for age, body mass index (BMI), menopausal status, hormone replacement therapy (HRT) use, parity, alcohol consumption, and family history of breast cancer. Pairwise comparisons for each ethnic group were performed using a Bonferroni correction. Results: 91.0% of the study population were white, 1.6% Asian, 1.1% Black and 1.0% Jewish. Jewish women had higher breast density than all other ethnic groups studied (p<0.001), with a mean pVAS of 34.8% (95% CI 33.6-36.1). Asian women had a mean density of 31.4% (95% CI 30.4-32.4) and significantly denser breasts than White women who had a mean pVAS density of 28.6% (95% CI 28.4-28.7). Conclusion: Previous research has reported mixed results. The relationship between risk factors for breast cancer are complex, and data not always complete, making this a challenging area of research. Our results support published evidence that some groups have increased density, and this relationship should be considered to ensure equity in screening and diagnosis.

In this study, we explored the significance of texture information for lesion detection using counterfactuals, a framework that can explain causal situation by exploring different actions or conditions and its associated results. Specifically, we investigated counterfactuals on mammogram texture; one example could be “what would happen on classifiers if there is no texture (or grayed-out) inside/outside the lesion.” We used the dataset of 10,415 2D screening mammograms, including 4,942 recalled lesions (BI-RADS 0) with lesion masks and 5,473 normal cases (BI-RADS 1). For identifying foreground and background textures, we applied lesion masks directly on lesion cases, while artificially imposed selected lesion masks on normal controls. Using this, four counterfactual cases were examined: 1) replacing lesion foreground (LF) with its mean intensity (MI) vs. normal (N), 2) replacing normal foreground (NF) with its MI vs. lesion (L), 3) replacing lesion background (LB) with its MI vs. N, and 4) replacing normal background (NB) with its MI vs. L. We trained three convolutional neural networks, which were ResNet50v2, ResNet50, and MobileNet, to classify lesion and normal cases (non-counterfactual, baseline). The classifiers on the first three counterfactual cases (LF vs. N, L vs. NF, and LB vs. N) performed similar to the baseline, while their performances on the cases of NB vs. L was significant dropped (p-value < 0.0001). Our findings indicate that lesion shape plays a key role for lesion image classification, while background texture is important for classifying normal cases.

Early-adulthood adiposity measures, such as weight and body-mass index, are associated with breast cancer risk. Moreover, studies have shown that breast parenchymal tissue patterns as reflected in digital mammograms (DMs) are also associated with breast cancer risk. We retrospectively analyzed DMs and early-adulthood adiposity data of 326 premenopausal women with the aim to assess the relationship of early-adulthood adiposity and breast parenchymal tissue patterns later in life. Radiomic features were extracted from DMs using a well-validated computational imaging pipeline and fused into woman-specific radiomic feature vectors via principal component analysis. Unsupervised hierarchical clustering was then applied to radiomic feature vectors to identify distinct phenotypes (clusters) of breast parenchymal complexity. For each early-adulthood adiposity measure, its associations with the identified breast parenchymal complexity clusters were assessed via logistic regression, adjusted for age at screening, race, family history of breast cancer, and parity. Two statistically significant clusters of breast parenchymal complexity (p-value < 0.001), “cluster 0” and “cluster 1”, were identified on the basis of principal components. Compared to women of high breast parenchymal complexity (“cluster 0”), women assigned to the cluster of lower breast parenchymal complexity (“cluster 1”) were associated with higher early-adulthood adiposity and larger adiposity changes over time. Among all early-adulthood adiposity measures, strongest associations with breast parenchymal complexity clusters were found for annual weight change from age 18 to age at DM screening (odds ratio [OR] = 2.46, 95% CI: [1.52, 3.99]) and annual weight change from age 30 to age at DM screening (OR = 1.73, 95% CI: [1.25, 2.39]). Our preliminary data suggest that adiposity in early adulthood, as well as weight gain from early adulthood to attained age are inversely associated with breast parenchymal complexity among premenopausal women and may have a lifelong impact on breast parenchymal tissue patterns.

Periodic breast cancer screening with mammography is considered effective in decreasing breast cancer mortality. When cancer is found, the best treatment method is selected considering the cancer subtypes. In this study, we investigated a method to distinguish breast cancers with poor prognosis from those with relatively good prognosis to assist diagnosis and treatment planning. In our previous study, all regions of interest including cancer lesions were resized to the same matrix size, which had caused loss of size and local characteristic information of the lesions. In this study, local patches with the original pixel size were automatically selected during the training in each epoch. The patch sampling could also reduce the effect of class imbalance. The proposed model was tested using 264 cases by a 4-fold cross validation. The result indicates the potential usefulness of the proposed method. The computerized subtype classification may support a prompt treatment planning and proper patient care.

In recent years, deep learning has showcased substantial promise in breast cancer diagnosis via mammograms. However, the integration of longitudinal changes between consecutive mammograms, which clinicians frequently consider for diagnosis, remains under-explored. In this study, we introduce an novel method that leverages crossattention mechanisms to capture longitudinal information between consecutive mammograms taken at various time intervals. Our method’s efficacy was assessed using a case-control internal dataset consisting of 590 cases. Preliminary results underscore its superiority over models relying solely on a single ”current” mammogram exam and those that merely combine features extracted from ”current” and ”prior” mammograms. By harnessing the power of longitudinal data, our model achieved enhanced diagnostic performance.

Aim: This study proposes a method to bypass the requirement of large amounts of original training data to develop a 1- to 4-year breast cancer risk prediction model using transfer learning from a breast cancer detection model with digital mammography images as input. Methods: The study utilizes a labelled dataset of 423 low risk cases and 423 high risk cases, which is considered a small amount of data in terms of AI development, but from the viewpoint of a regional screening organization this represents a large number of high risk cases, given the rarity of such cases compared to the large number of low risk cases available. A breast cancer detection model was used to obtain a latent representation of features extracted from ‘FOR PRESENTATION’ screening mammography images from three systems from a single vendor (Siemens). Dimensionality reduction was performed on the latent space using an Autoencoder architecture. The reduced latent space was then mapped to 1- to 4-year breast cancer risk with a fully-connected model. Results: The resulting model achieved an AUC of 0.77 for differentiating high and low risk cases, outperforming the Tyrer-Cuzick model and achieving state-of-the-art performance. Conclusions: The use of transfer learning from breast cancer detection models can produce image-based breast cancer risk prediction models that are comparable to the state-of-the-art, while requiring only moderate amounts of data.

Coronary Artery Calcium Scoring (CACS) is used for cardiac risk assessment caused by atherosclerotic plaque or other coronary artery diseases. Images from Non-Contrast (NC) cardiac Computed Tomography (CT) scans acquired at 120kVp are used in computing Agatston scoring for CACS. These scans, if done at lower peak voltage can reduce X-Ray radiation exposure. This, however, changes CT attenuation values for all tissues, as well as calcification compared to 120kVp scan, thus making it unusable for Agatston scoring. We propose a learning-based method to translate a CT image acquired at lower kVp to a 120kVp equivalent image, such that the same calcium scoring protocol can be used on these scans. We establish that the proposed method enables appropriate translation of CT values in calcification regions, thereby allowing similar calcium score (error < 6%) for a patient at reduced dose. Our proposed learning-based approach shows robust performance across datasets.

Artificial intelligence (AI) systems for diagnostic assistance in medical imaging are being researched and developed for main diagnostic imaging areas, such as magnetic resonance imaging and computed tomography imaging. These fields have a large number of examinations, and sufficient training data can be obtained for AI training. However, it may be challenging to collect sufficient training data to train AI in some minor areas, such as cystoscopy. In such cases, pre-trained AI models, which are pre-trained on a large, general-purpose image database based on real images, are often used. However, such large image databases containing real images are subject to copyright issues because the images are collected from the Internet, and mislabeling issues arise because annotation is performed manually. When building AI for medical images, the transparency of the training data is more important, and such problems are undesirable for the pre-training data used to build AI for diagnostic support. Therefore, this study proposed a new pre-training method based on automatically generated image databases to train the AI as a pre-training method that does not rely on real images in developing a diagnostic AI system for cystoscopic images. The objective was to build a diagnostic support system with a classification performance equivalent to that of expert urologists. Proposed method mixed two types of formula-driven image databases with the texture and contour features inspired by cystoscopic images. In the conducted experiments, the effectiveness of proposed method was verified for the classification of cystoscopic images.

Reflectance Confocal Microscopy (RCM) is a non-invasive imaging technique used in biomedical research and clinical dermatology. It provides virtual high-resolution images of the skin and superficial tissues, reducing the need for physical biopsies. RCM employs a laser light source to illuminate the tissue, capturing the reflected light to generate detailed images of microscopic structures at various depths. Recent studies explored AI and machine learning, particularly CNNs, for analyzing RCM images. Our study proposes a segmentation strategy based on textural features to identify clinically significant regions, empowering dermatologists in effective image interpretation and boosting diagnostic confidence. This approach promises to advance dermatological diagnosis and treatment.

Many artificial intelligence algorithms are currently deployed in medicine to help decision making and treatment planning. As new data become available, these algorithms can be updated with the goal to increase performance and improve patient outcomes. One method for updating algorithms is incrementally training on newly available data in multiple steps following sequential training protocols. For segmentation algorithms, changing the class labels of background pixels is known as semantic distribution shift, which can decrease algorithm knowledge retention. In this work, we explore the effects of semantic distribution shift on the knowledge retention of sequentially trained tumor segmentation algorithms by systematically altering the reference standards to simulate annotations by different annotators. Algorithms were trained in two sequential steps. The first step used an unmodified reference standard and the second step training data had a modified reference standard. The modified reference standards, which simulate annotator over- and under-segmentation, were created by systematically dilating or eroding the reference standard with different kernel sizes. A baseline algorithm is trained with unmodified data in both steps. Two types of variability are explored. The first is homogeneous variability, where the modified reference standards are changed in a consistent manner; for example, all reference standard segmentations were dilated or eroded with the same dilation or erosion structural element. The second is heterogeneous variability, in which the modified training data contains a mixture of dilated and eroded reference standards. Algorithm performance is evaluated using the Dice Coefficient, 95% Hausdorff Distance, and Volume Distance Coefficient, an adaptation of the Volume Similarity Coefficient. For the gadolinium-enhancing tumor subregion, algorithms trained with homogeneous variability had Dice values ranging from 0.38±0.01 to 0.76±0.02, while algorithms trained with heterogeneous variability had Dice values ranging from 0.76±0.02 to 0.79±0.01, much closer to the baseline algorithm’s value of 0.81±0.03. The results show that semantic distribution shift due to homogeneous variability decreases algorithm knowledge retention more than the shift due to heterogeneous variability. In practice, this suggests that the potential decrease in knowledge retention resulting from the addition of new data can be minimized by ensuring that the new data comes from a diverse variety of sources.

Recent research has shown that Generative Adversarial Networks (GANs) can generate highly realistic breast images through synthesis. Nevertheless, most of these studies assessed image quality solely through visual appraisal or reader studies, lacking quantitative analysis for specific clinical applications. This study aimed to quantitatively assess the quality of GAN-generated breast MRI images in predicting breast cancer recurrence risk. To achieve this, we developed a GAN model to synthesize the first post-contrast sequences from precontrast MRI sequences, utilizing an in-house dataset comprising 200 patients with confirmed breast cancer and available breast Dynamic Contrast-Enhanced MRI (DCE-MRI) staging data. In our study, we conducted a statistical analysis of radiomic features, revealing that among the 98 features assessed, 83 showed no significant differences (with p-values greater than 0.05) when comparing synthesized images with real images. Additionally, we employed a Lasso-Regression model to predict the Oncotype DX recurrence risk score. This analysis indicated that the predictive results for recurrence risk, derived from both real and synthesized images, did not exhibit significant differences, underscoring the comparability of synthesized images in this context.

Transfer learning represents a recent paradigm shift in the way we build artificial intelligence (AI) systems. In contrast to training task-specific models, transfer learning involves pre-training deep learning models on a large corpus of data and minimally fine-tuning them for adaptation to specific tasks. Even so, for 3D medical imaging tasks, we do not know if it is best to pre-train models on natural images, medical images, or even synthetically generated MRI scans or video data. To evaluate these alternatives, here we benchmarked vision transformers (ViTs) and convolutional neural networks (CNNs), initialized with varied upstream pre-training approaches. These methods were then adapted to three unique downstream neuroimaging tasks with a range of difficulty: Alzheimer's disease (AD) and Parkinson’s disease (PD) classification, “brain age” prediction. Experimental tests led to the following key observations: 1. Pre-training improved performance across all tasks including a boost of 7.5% for AD classification and 4.5% for PD classification for the ViT and 19.1% for PD classification and reduction in brain age prediction error by 1.26 years for CNNs, 2. Pre-training on large-scale video or synthetic MRI data boosted performance of ViTs, 3. CNNs were robust in limited-data settings, and in-domain pretraining enhanced their performances, 4. Pre-training improved generalization to out-of-distribution datasets and sites. Overall, we benchmarked different vision architectures, revealing the impact of pre-training them with emerging datasets for model initialization. The resulting pre-trained models can be adapted to a range of downstream neuroimaging tasks, especially when training data for a domain-specific target task is limited.

This paper presents a gradient-based analytical method for improving medical image classification. The automated classification of diseases is important in computer-aided diagnosis. In addition to accurate classification, its explainability is an essential facto. A gradient-based visual explanation provides the explainability of a model of an convolutional neural network (CNN). Most studies use this explanation to assess CNN's validity in a qualitative manner. Our motivation is to utilize the visual-explanation methods to enhance the classification accuracy of a CNN model. We propose a weight-analysis-based method to improve the classification accuracy of a trained-CNN model. The proposed method selects important patterns based on a gradient-based weight analysis of a middle layer in a trained model and suppresses irrelevant patterns in the extracted features for the classification. We applied our analytical method to a convolutional and a global-average-pooling layers in a CNN, which classifies a chest CT volume into COVID-19 typical and non-typical cases. As shown in classification results on 302 testing cases, our method improved the accuracy of the COVID-19 classification.

With a prevalence of 1-2% Celiac Disease (CD) is one of the most commonly known genetic and autoimmune diseases, which is induced by the intake of gluten in genetically predisposed persons. Diagnosing CD involves the analysis of duodenum biopsies to determine the small intestine condition. In this study, we propose a singlescale pipeline and the combination of two single-scale pipelines, forming a multi-scale approach, to accurately classify CD signs in histopathology whole slide images with automatically generated labels. The automatic classification of CD signs in histopathological images of these biopsies has not been extensively studied, resulting in the absence of a standardized guidelines or best-practices for this purpose. To fill this gap, we evaluated different magnifications and architectures, including a pre-trained MoCov2 model, for both single- and multiscale approaches. Furthermore, for the multi-scale approach, methods for aggregating feature vectors from several magnifications are explored. For the single-scale pipeline we achieved an AUC of 0.9975 and a weighted F1-score of 0.9680, while for the multiscale Pipeline an AUC of 0.9966 and a weighted F1-score of 0.9250 was achieved. On large datasets, no significant differences were observed; however, with only 10% of the dataset, the multi-scale framework outperforms the single-scale framework significantly. Moreover, the multi-scale approach requires only half of the dataset and half of the time compared to the best single-scale result to identify the optimal model. In conclusion, the multi-scale framework emerges as an exceptionally efficient solution, capable of delivering superior results with minimal data and resource demands.

Risk stratification of kidney cancers based on survival at diagnosis could enable more informed treatment decisions toward improved patient survival. Given the limited prognostic ability of clinical evaluation in segregating long-term vs short-term survivors, we sought to develop prognostic models which could exploit diagnostic CT scans for overall survival prediction in kidney cancers. This requires overcoming challenges related to model interpretability (such that the model best utilizes the most relevant locations within or around the kidney) as well as model generalizability (to ensure optimal model performance despite limited cohort sizes). In this work, we present the Spatial Attention Wavelon Network (SpAWN), which leverages a novel pre-training spatial attention operation to guide localization of convolutional responses together with wavelon activation functions to overcome known issues with vanishing/exploding gradients that occur in limited cohorts with class imbalance. SpAWN was evaluated for prognosticating survival on two large-scale publicly available cohorts of over 400 CT scans from kidney cancer patients, with comprehensive ablation studies to confirm the utility of attention maps as well as wavelon activation functions. A kidney exterior focused SpAWN model (with wavelon activations) demonstrated the best overall validation performance in segregating low- and high-risk patients in a hold-out cohort of N=223 CT scans (c-index=0.58, p=0.03), and was significantly improved compared to any alternative strategy. Integrating spatial attention with wavelon activations represents a novel interpretable and robust approach for prognosticating overall survival in kidney cancers via CT scans.

Population-based analysis of medical images plays an essential role in identification and development of imaging biomarkers. Most commonly the focus lies on a single structure or image region in order to identify variations to discriminate between patient groups. Such approaches require high segmentation accuracy in specific image regions while the accuracy in the remaining image area is of less importance. We propose an efficient ROI-based approach for unsupervised learning of deformable atlas-to-image registration to facilitate structure-specific analysis. Our hierarchical model improves registration accuracy in relevant image regions while reducing computational cost in terms of memory consumption, computation time and consequently energy consumption. The proposed method was evaluated for predicting cognitive impairment from morphological changes of the hippocampal region in brain MRI images showing that next to the efficient processing of 3D data, our method delivers accurate results comparable to state-of-the-art tools.

In recent years, there has been significant interest in evaluating perivascular spaces (PVS) due to their potential to characterize multiple neurological conditions. In this study, we demonstrated the potential to improve PVS evaluation at scale by introducing an AI algorithm to review identified PVS candidates and remove false positives on T2-weighted MRI. For this task, we were able to achieve an AUC of 0.93 +/- 0.02 while identifying optimal model characteristics and exploring areas of future improvement and investigation, thus demonstrating the potential for AI to replace human review in PVS quantification at scale.

In this work, we aim to predict the survival time (ST) of glioblastoma (GBM) patients undergoing different treatments based on preoperative magnetic resonance (MR) scans. The personalized and precise treatment planning can be achieved by comparing the ST of different treatments. It is well established that both the current status of the patient (as represented by the MR scans) and the choice of treatment are the cause of ST. While previous related MR-based glioblastoma ST studies have focused only on the direct mapping of MR scans to ST, they have not included the underlying causal relationship between treatments and ST. To address this limitation, we propose a treatment-conditioned regression model for glioblastoma ST that incorporates treatment information in addition to MR scans. Our approach allows us to effectively utilize the data from all of the treatments in a unified manner, rather than having to train separate models for each of the treatments. Furthermore, treatment can be effectively injected into each convolutional layer through the adaptive instance normalization we employ. We evaluate our framework on the BraTS20 ST prediction task. Three treatment options are considered: Gross Total Resection (GTR), Subtotal Resection (STR), and no resection. The evaluation results demonstrate the effectiveness of injecting the treatment for estimating GBM survival.

Intracranial aneurysms (IAs) are a common vascular pathology and are associated with a risk of rupture, an event that is often fatal. Human evaluation of IA rupture is rather subjective, therefore we aimed to develop a deep learning based rupture event prediction model from aneurysm shape information. We used 386 CTA scans with 500 IAs that were either unruptured or ruptured (250/250). The IA rupture status was computed using bottleneck layer feature vectors sourced from two deep learning models trained for the respective auxiliary tasks of cerebral vessel labeling and aneurysm isolation from its parent vessels. The two extracted feature vectors were concatenated with 20 established features (patient sex, age, aneurysm location and morphological parameters) and used to predict aneurysm rupture status using eight different machine learning models. We achieved the best AUC of 0.851 using random forest with feature selection based on Spearman’s rank correlation thresholding. The rather good performance of IA rupture status classification renders the proposed approach as a promising tool for management of rupture risk in the ”no treatment” paradigm of patient follow-up imaging.

Epileptic seizure detection using deep learning (DL) are often focused on data recorded using scalp electroencephalography (EEG), cases with limited data such as intracranial EEG are indeed unexplored. Self-supervised learning (SSL) can serve as a pretext task to learn useful features before performing classification task using relatively smaller datasets. We herein introduce an SSL strategy for vision transformers utilizing group masked image modeling applied to EEG spectrograms for classifying seizure and non-seizure activity. Particularly, we leverage EEG spectrogram representation, which is robust to noise, and incorporates time and frequency information. By inducing corruption into each input spectrogram, our model recovers the lost information and, in the process, learns features, which enables the acquisition of significant characteristics essential for downstream classification task. Our experimental results show that longer spectrogram window sizes yielded higher accuracy in detecting seizure activity. Specifically, average classification using fully supervised model across four channels demonstrated optimal performance on 90s EEG segments with an accuracy of 88.65%, followed by 45s (88.49%) and 30s (84.54%) spectrograms. We further compared seizure classification performance using vision transformers with pretraining using spectrograms (SSL-ECoG), without pretraining, and with a pretraining strategy using non-EEG data (ImageNet-1K). Notably, we show that the classification task following SSL-ECoG pre-training attained highest accuracy of 94.32%, AUC-PR of 0.966, 96.47 % F-1 score, and AUROC of 0.977. These results highlight the potential of using SSL and vision transformers with pertinent data can achieve highly accurate classification, even with limited data sizes.

This study is to evaluate the performance of a CNN-based network with multi-scale dual attention module and a hybrid Transformer-CNN network, aiming to identify the best practices for segmenting lung tumors with significant variability. Moreover, we evaluate and compare the use of size-invariant patches, which maintain consistent tumor-tobackground ratios regardless of tumor size, against the use of the original patches. To assess the performance of two different networks, we evaluate MSDA-Net and TransUNet, using original and size-invariant patches, respectively. The results reveal practical insights into the segmentation of both small-sized and large-sized tumors. In small-sized tumor segmentation, employing size-invariant patches helps achieve better segmentation results by addressing the variability in tumor sizes and ensuing a consistent scale for analysis. For large-sized tumor segmentation, using original patch helps preserve more context information, enabling better segmentation results. Additionally, our results indicate that the CNN-based network with multi-scale dual attention modules outperformed the hybrid Transformer-CNN network in providing more robust segmentation results for both small-sized and large-sized tumors.

Magnetic Resonance Imaging (MRI) can be useful in preclinical modeling of cancer initiation and response to novel therapeutics. To measure longitudinal changes in disease progress, accurate segmentation and measurement of cancerous areas is a critical step. In the setting of preclinical lung cancer models, common semi-automated techniques for segmentation include intensity thresholding, of which manual adjustments are time-consuming and prone to inter-reader variation. To address these concerns, we evaluated several deep-learning models for automated lung lesion segmentation. MR images from six different experimental models were collected as part of preclinical research studies, with a total of 183 scans processed with tumor burden segmentations from semi-automated thresholding with manual adjustment by expert researcher. These images included two coronal acquisition protocols, one which images a single mouse per scan (0.057x0.057mm in-plane resolution, 0.5mm slice thickness, n=161 scans) and one which images three mice in same scanning session (0.179x0.179mm in-plane resolution, 0.5mm slice thickness, n=12 scans). Scans were stratified to training (n=143) and validation (n=40). Fifteen newly acquired scans were used as independent experimental testing set, all imaged with multi-mouse acquisition. Several deep learning models were developed under varying conditions including: architecture (UNet, UNETR, and SegResNet), resolution (0.057x0.057x0.5mm vs 0.179x0.179x0.5mm), and window size regions-of-interest [ROIs] (224x224x16 input vs 448x448x16 input). At inference, sliding window overlap was varied between 0.2-0.6. All models were trained to 800-1000 epochs using learning rates ranging 0.001-0.0001. The UNET model with 224x224x16 ROI had the best performance, with average image-level DICE score of 0.6446 in validation and 0.4493 in the test set compared to 0.3544 – 0.6481 in validation and 0.0026 – 0.4816 in the test set for the remaining models. Manual editing of the AI segmentation from the UNET model took a median time of 12.5 minutes (range: 5-38) compared to 27.5 minutes (range: 5-127) for fully manual segmentation.

Traditional deep learning models have been extensively utilized in lung cancer computer-aided diagnosis (CAD) studies. These models typically treat false positive and false negative cases equally during training. However, in the specific context of lung nodule malignancy CAD studies, our objective is to improve sensitivity without significantly compromising overall accuracy. To address this, our study proposes the incorporation of cost values into the sigmoid activation function for deep learning-based CAD systems used in lung nodule malignancy classification. Through empirical analysis, we observed a significant 4% increase in sensitivity while effectively maintaining the overall accuracy. The results obtained from our study provide compelling evidence that incorporating cost values into the training scheme can significantly enhance sensitivity in the classification of lung nodule malignancy. Furthermore, we emphasize the importance of considering the cost values as hyperparameters in future CAD systems. By appropriately tuning these cost values, we can further optimize the performance and efficacy of lung nodule malignancy CAD systems.

This study proposes different deep learning approaches to automatically classify pneumoconiosis based on the International Labour Office (“ILO”) classification system. Through collaboration with the National Institute for Occupational Safety and Health (NIOSH), this study curated a custom dataset of chest radiographs with (N=520) and without (N=149) pneumoconiosis. The four-point major category scale of profusion (concentration) of small opacities (0, 1, 2, or 3) were considered in this study. Several deep learning models were evaluated for classifying different levels of profusion grades including: 1) Transfer learning with models pre-trained with public chest radiograph repositories, 2) Hand-crafted radiomic feature extraction, and 3) Hybrid architecture that integrates hand-crafted radiomic features with transfer-learned features using a loss function that incorporates the inherent ordinality within the profusion grade scale. For profusion grade 0 vs. grade 3, both the transfer learning and radiomic feature extraction methods obtained test-set accuracies of greater than 91%. The highest prediction accuracy for normal (profusion grade 0) vs. abnormal (profusion grade 1, 2 and 3) was 83% using the transfer learning method. Under the setting of multi-class identification of four profusion grades, ResNet model adapted to a ordinal multi-task loss function notably outperforms traditional models reliant on cross-entropy loss (with accuracy 58%) and achieves an accuracy of approximately 61%. The amalgamation of radiomic and ResNetderived features, coupled with the application of multi-task loss, culminates in the highest recorded accuracy of approximately 62%. This demonstrates an example case where the integration of hand-crafted and deep-learned features, along with modeling the ordinality of the classes, improves classification performance of chest radiographs.

The coronavirus disease 2019 (COVID-19) emerged in December 2019 and rapidly evolved into a global pandemic affecting more than 750 million people and causing more than 6,8 million deaths worldwide. The most important and often deadly manifestation of the disease is pneumonia. Among other imaging modalities, lung ultrasound (LUS) represents an efficient tool for management of COVID-19 patients, as it can be used to classify and monitor lung involvement, as well as to assess disease’s severity. However, LUS requires a skilled sonographer to interpret findings. For this reason, we developed a Deep Learning (DL)–powered system, for multi-class automatic assessment of LUS videos in COVID-19 patients. We trained and tested (70,4% accuracy) our system with LUS images acquired from patients of the so-called first and second waves (2020). However, lung disease induced by COVID-19 is changed with time due to virus behavior, presence of vaccines and availability of therapies. Accordingly, we tested again performances of our system with LUS images of COVID-19 patients enrolled in 2022. Surprisingly, the performances of the system dramatically drop to an unsatisfiable accuracy of 48,3%. We speculate that this fact is primarily due to the differences in clinical features of patients hospitalized in 2020 and 2022. However, these findings also highlight a critical point on the management of DL systems designed for prognosis and diagnosis of rapidly-changing diseases. Accordingly, a continuous training update should be considered as the only viable strategy to preserve DL systems reliability and to improve the credibility of DL systems from the medical community.

The manual assessment of chest radiographs by radiologists is a time-consuming and error-prone process that relies on the availability of trained professionals. Deep learning methods have the potential to alleviate the workload of radiologists in pathology detection and diagnosis. However, one major drawback of deep learning methods is their lack of explainable decision-making, which is crucial in computer-aided diagnosis. To address this issue, activation maps of the underlying convolutional neural networks (CNN) are frequently used to indicate the regions of focus for the network during predictions. However, often, an evaluation of these activation maps concerning the actual predicted pathology is missing. In this study, we quantitatively evaluate the usage of activation maps for segmenting pulmonary nodules in chest radiographs. We compare transformer-based, CNN-based, and hybrid architectures using different visualization methods. Our results show that although high performance can be achieved in the classification task across all models, the activation masks show little correlation with the actual position of the nodules.

In this work, we utilize a Transformer-based network for precise anatomical landmark detection in chest X-ray images. By combining the strengths of Transformers and UNet architecture, our proposed model achieves robust landmark localization by effectively capturing global context and spatial dependencies. Notably, our method surpasses the current state-of-the-art approaches, exhibiting a significant reduction in Mean Radial Error and a notable improvement in the rate of accurate landmark detection. Each of the landmark points in the labels is presented as a Gaussian heatmap for training the network, using a hybrid loss function, incorporating binary cross-entropy and Dice loss functions, allowing for pixel-wise classification of the heatmaps and segmentation-based training to accurately localize the landmark heatmaps. The promising results obtained highlight the underexplored potential of Transformers in anatomical landmark detection and offer a compelling solution for accurate anatomical landmark detection in chest X-rays. Our work demonstrates the viability of Transformer-based models in addressing the challenges of landmark detection in medical imaging.

In this study we investigate the novel approach of using radiomic phenotypes from the lung parenchyma and tumor region of PET/CT images in non-small cell lung cancer (NSCLC) patients to predict overall survival (OS) and progression free survival (PFS) after tumor resection. We used 144 publicly available fluorodeoxyglucose ([18]F-FDG) PET/CT images from The Cancer Imaging Archive (TCIA) NSCLC Radiogenomics dataset. We extract features using the cancer phenomics imaging toolkit (CaPTk) to extract radiomic features from each of four image source regions: PET imaging of the tumor, PET imaging of the non-tumor lung parenchyma, CT imaging of the tumor, and CT of the parenchyma. Using each of the four sets of features, we independently clustered patients into phenotypes using unsupervised hierarchical clustering. The four phenotyping schemes individually, together, and in combination with clinical variables were assessed for association with time to OS and PFS via Cox proportional-hazards modeling, assessing covariate association via the log-rank p-value and model predictive performance via the C statistic. The clinical variables divided high from low hazard groups with p ≤ 0.05 for OS (p = 0.002, C = 0.62) but not for PFS (p = 0.098, C = 0.58). For PFS, radiomic phenotype derived from PET lung parenchyma performed better than the clinical variables both alone (p = 0.014, C = 0.59) and in conjunction with the clinical variables (p = 0.014, C = 0.62). Radiomic phenotypes from the lung parenchyma of PET/CT images can improve outcomes prediction for PFS after tumor resection in patients with NSCLC. Radiomic phenotypes from the non-cancerous parenchyma may derive prognostic value by detecting differences in the tissue linked to the biology of recurrence.

Synthetic datasets hold the potential to serve as cost-effective alternatives to clinical data, potentially aiding in mitigating the biases in clinical data. This paper presents a novel method that utilizes such datasets to train a computer-aided detection (CADe) algorithm. Our proposed approach uses images of a physical anthropomorphic phantom into which manufactured objects representing simplified lesions were inserted, followed by a set of randomized and parametrized augmentations of the data to increase the variability in these datasets. By incorporating these augmentations into the training phase, our proposed method aims to add variability within training datasets of limited size to improve model performance. We apply our proposed method to the false positive reduction stage of a lung nodule CADe system on computed tomography (CT) scans. Our experimental results demonstrate the effectiveness of the proposed method, where the network performance in terms of the Competitive Performance Metric (CPM) increased by 6% when a training set consisting of 50 clinical CT scans was augmented by scans obtained from a physical phantom database.

Purpose: To develop a deep learning model to estimate lung age based on low-dose chest computed tomography (CT) images and investigate the feasibility of its application in health management. Materials and Methods: Deep learning models (InceptionV3, DenseNet, VGG16, and ResNet50) were trained on a set of low-dose chest CT images of 4,646 non-smokers. They were adjusted for an equal number of participants in each age group to estimate the chronological age and functional lung age. They were further tested on a dataset of 5,841 nonsmokers and 16,709 smokers. The estimated ages were evaluated based on the mean absolute error, correlation coefficients with the actual ages, and Bland-Altman analysis. In addition, we tested whether the estimated age was higher in the group with a longer the smoking history. Results: There was a correlation between the estimated and actual ages in all models (chronological/functional lung ages): r=0.74/0.54, r=0.75/0.53, r=0.68/0.50, and r=0.67/0.34 for InceptionV3, DenseNet, VGG16, and ResNet50, respectively. The InceptionV3 model showed the best performance with an estimation error of ±4.02 and ±11.18 years for chronological and functional lung ages, respectively. Furthermore, among smokers, the longer the smoking history, the higher the estimated chronological age, suggesting that the trained model can predict lung changes other than aging, such as the effects of smoking and signs of disease onset. Conclusion: We developed an age estimation model based on low-dose chest CT images that can quantify lung damage due to smoking as an increase in predicted age. The feasibility of CT image-based lung age estimation for health care was demonstrated.

Lung cancer is the second most prevalent and deadliest cancer in the United States, primarily due to its elusive early symptoms that hinder timely diagnosis. Lung nodules, minute anomalous areas, hold potential significance in lung cancer occurrence. Swift identification of these nodules can significantly enhance patient survival rates. Thoracic thin-sliced Computed Tomography (CT) scanning has emerged as a widely adopted approach for radiologists' diagnosing and prognosticating lung abnormalities. However, human factors can be prone to errors stemming from the multifarious causes underlying nodule formation, including factors like pollutants and infections. The domain of deep learning algorithms has recently showcased remarkable prowess in classifying and segmenting medical images. This study is geared towards the creation of a comprehensive framework that seamlessly integrates explainable AI techniques to achieve precise pulmonary nodule detection for computer-aided detection and diagnosis (CAD) systems. The framework's execution is underpinned by an explanation supervision approach, employing radiologists' nodule contours as guiding signals to imbue the model with an understanding of nodule morphologies. The nodule detection is achieved by augmentation of the model's learning capabilities, particularly on smaller datasets. Furthermore, two imputation techniques are deployed to refine the nodule region annotations, thereby attenuating noise inherent in human annotations. These techniques empower the model with steadfast attributions that align with human expectations. Our approach consistently elevates ResNet18's classification performance and explanation quality regarding accuracy and specificity by 22% and 23% using training datasets with 100 samples. For small dataset learning using 50 samples, the proposed framework still outperforms ResNet18 by 16% regarding accuracy and specificity. This framework seamlessly integrates a robust explanation supervision technique, guaranteeing nodule classification and morphology assessment accuracy. Reducing the burden on radiologists allows them to focus on diagnosing and prognosticating potentially cancerous pulmonary nodules at an early stage, consequently augmenting survival rates in lung cancer.

Rotator Cuff Tears (RCTs) are primarily age-related musculoskeletal disorders affecting the shoulder region. In this paper, we present a research effort that aims to construct a Computer-Aided Diagnosis (CAD) model for RCTs. The model utilizes three-plane MRI slides coupled with diagnostic outcomes. To enhance model interpretability, we employed MRNet, conducting training on each anatomical plane, and subsequently merged the results using logistic regression. At the individual plane level and the fusion level, we applied Grad-CAM and SHapley Additive exPlanations (SHAP), respectively. Additionally, we introduce CAMscore, a method using Grad-CAM, designed to quantitatively determine the diagnostic relevance of individual MRI slides. When fusing the three planes, we achieved a maximum F1-score of 0.9508. Additionally, we observed a notably higher diagnostic efficacy in the sagittal plane compared to the axial and coronal planes. However, our study has certain limitations, including the need for greater dataset diversity and the necessity for verification by medical professionals. Nevertheless, our study advances the field of CAD by improving the understanding of the decision-making processes of models utilizing three-plane fusion.

As deep learning (DL) gains prominence in medical image analysis, its applications in orthopedics and broader medical contexts have expanded significantly. While machine learning algorithms show potential in using X-ray images for predicting outcomes, biases may arise due to insufficient preprocessing. One such source of bias is algorithmic shortcutting, a phenomenon wherein DL models inadvertently train on patterns within training data unrelated to the intended diagnostic content. Leveraging the Osteoarthritis Initiative (OAI) dataset and ResNet18, our convolutional neural network (CNN) could predict the clinical center from which X-rays originated with over 95% accuracy. Moreover, our investigation unveiled significant variations in patient demographics and outcomes between clinical centers, implying that discerning the originating clinical center could be a potential shortcut in predicting patient outcomes from X-rays. These findings underscore the importance of careful X-ray image preparation and analysis. More broadly, as deep learning models increasingly influence clinical decision-making, this work underscores the importance of addressing model shortcutting to enhance the transparency, interpretability, and efficacy of deep learning research across orthopedics and other medical domains.

In spinal patients with spondylolisthesis, timely diagnosis and early surgical intervention can enhance the overall treatment course and attenuate possible complications. The diagnosis of spondylolisthesis cases typically involves the assessment of MRI scans by a trained radiologist. With the escalating volume of medical images, automating radiology report reading becomes indispensable in identifying spondylolisthesis cases effectively and rapidly creating ground truth for further research studies. Therefore, we present a novel framework leveraging a large language model (LLM), i.e., ChatGPT (GPT4-based), for automated assessment of the presence of spinal spondylolisthesis by parsing spinal MRI radiology reports. Through rigorous evaluation on a cohort of 166 subjects undergoing spinal fusion surgery, comprising 83 cases of spondylolisthesis and 83 non-spondylolisthesis cases, our proposed framework demonstrated exceptional performance metrics with 94% sensitivity, 98.8% specificity, 97.5% precision, and 96.4% accuracy. The implementation of this algorithm effectively recognizes spondylolisthesis diagnoses by automatically reading radiology reports.

Uterine fibroids (UFs) are benign tumors that vary in clinical presentation from asymptomatic to causing debilitating symptoms. UF management is limited by our inability to predict UF growth rate and future morbidity. This study aimed to develop a predictive model to identify UFs with increased growth rate and possible resultant morbidity. We retrospectively analyzed 44 expertly-outlined UFs from 21 patients who underwent two multi-parametric MR imaging exams as part of a prospective study over an average of 16 months. We identified 100 initial features by extracting quantitative MRI, morphological and textural radiomics features from DCE, T2, and ADC sequences. Principal component analysis reduced dimensionality, with the smallest number of components explaining over 97.5% of variance selected. Employing a leave-one-fibroid-out scheme, a linear discriminant analysis classifier utilized these components to output a growth risk score. The classifier incorporated the first six principal components and achieved an area under the ROC curve of 0.80 (95% CI [0.69; 0.91]), effectively distinguishing UFs growing faster than the median growth rate of 0.93 cm3/year/fibroid from slower-growing ones within the cohort. Time-to-event analysis, dividing the cohort based on the median growth risk score, yielded a hazard ratio of 0.33 [0.15; 0.76], demonstrating potential clinical utility. In conclusion, this pilot study developed a promising predictive model utilizing quantitative MRI features and principal component analysis to identify UFs with increased growth rate. Furthermore, the model's discrimination ability supports its potential clinical utility in developing tailored patient and fibroid specific management once validated on a larger cohort.

Radiomics can be used to generate a large magnitude of quantitative features from medical images that can be applied to various predictive tasks and treatment decisions. To ensure the generalizability of such methods, radiomic features need to be robust to variations in patient positioning and segmentation of regions of interest. Feature robustness is often determined through test-retest imaging, whereby the agreement of feature values from images acquired over a brief time interval is quantified to measure robustness. However, the scarcity of test-retest data is a significant limitation of such approaches, and there is a lack of consensus for alternative methods to determine feature robustness with single scans. Hence, this study evaluates the effectiveness of assessing feature robustness using various metrics to quantify the agreement of feature values before and after image perturbation. 1002 features were extracted from thoracic computed tomography scans of patients with pleural mesothelioma before and after perturbations, including rotation, erosion, dilation, and contour randomization, and five distinct metrics were used to assess feature agreement. Feature robustness was highly variable as quantified by various combinations of perturbations and metrics of agreement. The greatest stability in subsequent steps of the predictive pipeline including feature selection and classification was achieved using the concordance and intraclass correlation coefficients with chained perturbations of image rotation, contour erosion or dilation, and contour randomization. These findings suggest that the choice of image perturbations and metrics of agreement have non-negligible consequences on feature robustness estimates and the success of downstream predictive tasks, warranting careful consideration in experimental design.

Magnetic resonance imaging (MRI) is well suited for Solid renal masses (SRMs) characterization (e.g., benign vs. malignant) due to its superior soft tissue contrast. Though renal mass detection and characterization using deep-learning (DL) methods have been extensively studied for computed tomography (CT) images, those same tasks are yet to be investigated on MRI images. SRMs need active surveillance as they consist of biologically diverse heterogeneous groups of benign or malignant masses. Among them, malignant clear cell renal carcinoma (ccRCC) is frequently aggressive. There are inter-observer and intra-observer differences in the assessment of SRMs by expert clinicians because of their experience and expertise. Therefore, it is essential to develop a machine learningbased noninvasive imaging diagnosis to distinguish SRMs as benign and malignant. Our retrospective study consisted of malignant (renal cell carcinoma- clear cell, papillary, and chromophobe) and benign (fat-poor angiomyolipoma-fpAML, oncocytomas) SRMs. We extracted first and second-order radiomics features from SRMs on T2W and T1W-CM MRI to train different machine learning (ML) models using the 5-fold cross-validation for benign vs malignant classification. The support vector machine (SVM) algorithm generated benign vs malignant classification accuracy of 90.00% with ROC-AUC of 76.19% on T2W MRI and the custom-designed multilayer perceptron model (MLP) model produced accuracy of 80.00% with ROC-AUC of 75.47% on T1W-CM MRI. Thus, ML-based radiomics features classification of SRMs extracted on MRI may be an alternative to biopsy using a non-invasive assessment of SRMs.

Radiomic features have been shown to add predictive power to risk-assessment models for future kidney decline in patients with autosomal dominant polycystic kidney disease (ADPKD), and these previous studies utilized only one imaging timepoint. Delta radiomics incorporates image features from multiple imaging timepoints and the change in features across these timepoints. There is a need to investigate utilizing delta radiomics in ADPKD and the benefit of incorporating delta-features in risk-assessment models, taking advantage of imaging that is clinically indicated for these patients. A cohort of 152 patients and their respective T2-weighted fat saturated magnetic resonance imaging coronal images were used to predict progression to chronic kidney disease (CKD) stage 3A, 3B, and >30% reduction in estimated glomerular filtration rate (eGFR) at 60-months follow up using radiomic features at (1) baseline imaging, (2) 24-months follow up, and (3) 24-months delta-features. Prediction models utilizing delta radiomics alone yielded area under the receiver operating characteristic curve (AUC) values that ranged from 0.52-0.55, versus using radiomic features from single timepoints and combined timepoint AUC values 0.67-0.76. Trends of increasing AUC values were observed when combining clinical and radiomics features for predicting CKD stage 3A and >30% reduction in eGFR.

The linking of image analysis, in particular radiomics, with genetic profiles, known as “imaging genomics,” has been successfully applied to many diseases and anatomic regions. The application of imaging genomics in mesothelioma, however, remains limited. Pleural mesothelioma (PM) is a devastating cancer associated with asbestos exposure and has a poor prognosis. The BRCA1-associated protein 1 (BAP1 ) gene is of considerable interest because somatic BAP1 mutations are the most common alteration associated with PM. Further, germline mutation of the BAP1 gene has been linked to PM. This study aims to explore the potential of radiomics to identify somatic BAP1 mutations and assess the feasibility of radiomics in future research in identifying germline mutations. A cohort of 149 patients with PM and known somatic BAP1 mutation status was collected, and a previously published deep learning model was used to automatically segment tumor on three representative sections from each patient’s computed tomography scans. Preprocessing, including gray-level discretization and resampling, was performed, and intensity-based and 2D texture features were extracted from the segmented tumor regions. Synthetic Minority Over-sampling Technique (SMOTE) combined with Tomek links was employed to address data imbalance. The top features were selected and used to train 18 separate machine learning models. The performance of the models in distinguishing between BAP1 -mutated (BAP1+) versus BAP1 wild-type (BAP1 -) tumors were evaluated using area under the receiver operating characteristic curve (ROC AUC) as the figure of merit. This study achieved an AUC value of 0.69 (95% confidence interval: 0.60, 0.77) using a decision tree classifier. Overall, this novel, proof-of-concept work demonstrates the potential of radiomics in differentiating between BAP1+/- in patients with PM. Future work will extend these methods to the assessment of germline BAP1 mutation status through image analysis for improved patient prognostication.

Tumor expression of PDL1 is a suboptimal biomarker to predict response to anti-PD1/PDL1 therapy, including pembrolizumab (PEMBRO). We hypothesize that radiomic features, high-throughput descriptors of tumor heterogeneity, can improve performance of models built using established clinical biomarkers for prediction of overall survival and presence of relevant genetic mutations KRAS and STK11 (significant in patients selected for PEMBRO-based therapy). Our study includes baseline CT scans from 308 patients with stage 4 NSCLC, treated with first line PEMBRO, from a single-center retrospective study conducted at Hospital of the University of Pennsylvania (November 2016-December 2020). Region of interest (ROI) segmentation on 3D tumor CT volumes was performed using ITK-SNAP. Feature extraction on ROIs was performed using CaPTk. Scan parameter heterogeneity (presence/absence of contrast enhancement, reconstruction kernel and voxel spacing) was mitigated using a nested ComBat harmonization technique, while protecting the variable BMI to prevent removal of heterogeneity arising from biological variation during harmonization, noting BMI had statistically significant correlation with first radiomic principal component (PC). Dimensionality of resultant set was reduced using an unsupervised hierarchical clustering technique. These identified radiomic phenotypes, when combined with clinical variables (longest tumor diameter at baseline, tumor PDL1 expression, age, sex, race, smoking status, BMI), increased the five-fold cross-validated c-score in the prediction of overall survival from 0.60 [0.54,0.63] to 0.64 [0.57,0.66]. For a subset of 287 patients (further sub-divided based on histology: squamous vs. non-squamous), the radiomic phenotypes were used to identify patients with mutations in KRAS and STK11, using linear SVM classifier: KRAS (squamous: 65% (AUC-0.61), non-squamous: 74% (AUC-0.63)), STK11 (squamous: 76% (AUC-0.62), non-squamous: 82% (AUC-0.68)). The radiomic phenotypes improve the prognostic value of clinical biomarkers in the prediction of overall survival and can identify patients with gene mutations that are significant in patient selection for PEMBRO-based therapy. Our future work involves building a multi-omic (radiomic, genomic, clinical) predictor of progression-free survival, to study patient response to PEMBRO-based therapy.

Accurate characterization of lesion boundaries is a critical task in diagnosing cancer. General large vision models (LVMs) such as the Segment Anything Model (SAM) have demonstrated promising performance in the semantic segmentation of natural images, but their test-time performance on medical images has been suboptimal, particularly in images with noise or poor contrast. To address this issue, we propose a test-time domain adaptation strategy that combines LVMs trained on large-scale datasets with the level-set active contours model, which can be tuned on a small subset of cases from the target dataset to yield more robust medical image segmentation performance at test-time. We showed the performance of our strategy on two datasets ISIC 2018 and CHASED DB1, which yielded a 0.3% and 0.11% improvement in segmentation accuracy, respectively, measured by Intersection over Union. We conclude that using active contours on top of LVMs for test-time domain adaptation can improve segmentation performance.

Automatic segmentation of colon polyps can significantly reduce the misdiagnosis of colon cancer and improve physician annotation efficiency. While many methods have been proposed for polyp segmentation, training large-scale segmentation networks with limited colonoscopy data remains a challenge. Recently, the Segment Anything Model (SAM) has recently gained much attention in both natural image and medical image segmentation. SAM demonstrates superior performance in several vision benchmarks and shows great potential for medical image segmentation. In this study, we propose Poly-SAM, a finetuned SAM model for polyp segmentation, and compare its performance to several state-of-the-art polyp segmentation models. We also compare two transfer learning strategies of SAM with and without finetuning its encoders. Evaluated on five public datasets, our Polyp-SAM achieves state-of-the-art performance on two datasets and impressive performance on three datasets, with dice scores all above 88%. This study demonstrates the great potential of adapting SAM to medical image segmentation tasks.

Hearing loss as a significant global health concern, encompassing costs across society through healthcare, education, and productivity impacts. Traditional otoscopic diagnostic methods pose challenges, prompting the development of computer-aided diagnosis (CAD) systems. Tympanic membrane (TM) segmentation is a crucial and vital task for early diagnosis and intervention in middle ear diseases. Automatic TM segmentation in CAD systems improves diagnostic accuracy. This study presents a method for the automatic segmentation of the TM from video-otoscopic frames based on Segment Anything Model Adapter (SAM-Adapter). To the best of our knowledge, this research is the first application of a SAM-Adapter segmentation model for segmenting TM areas from otoscopic frames. 765 video frames from 36 otoscopic videos were used to train and test the model. The experimental results show that the SAM-Adapter achieves high segmentation performance with a Dice similarity coefficient of 0.9486 without any pre-processing and postprocessing steps. Empirical results showed that the SAM-Adapter model is better than the U-Net-based models in our dataset.

The segmentation of chest X-ray (CXR) anatomy is a crucial yet challenging task, often subject to human errors and demanding extensive time investments. While deep learning has shown promise in medical image segmentation, its efficacy is constrained by the necessity for large annotated datasets, which are difficult and costly to generate in the medical domain. Addressing the growing need for models capable of learning from limited data, this study explores the application of Self-Supervised Learning (SSL), specifically adapting the DINO self-supervised learning framework, to leverage unannotated CXR images. Our approach, which does not rely on supervised fine-tuning, is particularly relevant for scenarios where only a small amount of annotated data is available, as is often the case at the onset of continuous learning initiatives. Employing a ConvNeXt-based architecture, our method demonstrates the potential of SSL in enhancing CXR segmentation by utilizing unannotated data, thereby reducing dependency on extensive annotated datasets. In the multilabel segmentation task, we observed increases in the average Dice Similarity Coefficient (DSC) from 0.24±0.19 to 0.56±0.22 for 3 annotated training cases. This research contributes to the evolving landscape of medical imaging analysis, offering insights into the efficient application of SSL for improving segmentation tasks in CXR images and potentially other medical imaging modalities.

Volumetric assessment of edema due to anasarca can help monitor the progression of diseases such as kidney, liver or heart failure. The ability to measure edema non-invasively by automatic segmentation from abdominal CT scans may be of clinical importance. The current state-of-the-art method for edema segmentation using intensity priors is susceptible to false positives or under-segmentation errors. The application of modern supervised deep learning methods for 3D edema segmentation is limited due to challenges in manual annotation of edema. In the absence of accurate 3D annotations of edema, we propose a weakly supervised learning method that uses edema segmentations produced by intensity priors as pseudo-labels, along with pseudo-labels of muscle, subcutaneous and visceral adipose tissues for context, to produce more refined segmentations with demonstrably lower segmentation errors. The proposed method employs nnU-Nets in multiple stages to produce the final edema segmentation. The results demonstrate the potential of weakly supervised learning using edema and tissue pseudo-labels in improved quantification of edema for clinical applications.

Radiomic features extracted from dynamic contrast-enhanced (DCE) magnetic resonance (MR) images of breast cancers have previously demonstrated merit in the prediction of response to treatment. Radiomic features can be used to support clinical trial studies, as is being conducted in I-SPY2. Our goal was to compare the quantification of features used to predict pathological complete response in the I-SPY2 clinical trial obtained through our automated methods to those obtained via reported semi-manual methods. We used automated methods based on identification of a single seed-point to segment tumors and extract features of tumors from 972 patients. On an overlapped set of patients (381 patients) between the entire set and a previously-published I-SPY2 study, the Pearson correlation coefficient was used to compare features calculated automatically to published features calculated semi-manually, focusing on longest diameter (semi-manual) versus maximum diameter (automated), sphericity (semi-manual versus automated), and functional tumor volume (semimanual) versus most enhancing tumor volume (automated), extracted from pre-treatment DCE-MR image series. The Pearson correlation coefficients were considered significant if p <0.017 (Bonferroni-corrected for three comparisons.) Results showed significant correlation (p < 0.01) between automated and semi-manual extraction for all three features, but differences were observed in the actual extracted feature values. It is interesting to note that with the use of our fully automated methods the tumors from almost all the I-SPY2 patients were able to be analyzed, thus, potentially contributing to more efficient studies of radiomic features in clinical trial assessment.

Development of kidney segmentation models has largely focused on using contrast-enhanced CT exams. The KiTS segmentation challenge, in particular, has provided a benchmark using 300 annotated arterial phase CT scans. Review of the best performing models identifies 3D Unet models with residual connections as the best performing models for kidney segmentation. Li et al. found a U-net architecture with residual connections to provide the best performance for their segmentation task. Their work focused on segmenting kidney parenchyma alongside kidney stones using a dataset of 257 studies that was recently made available. Yu et al. investigated the ability to train a multi-organ nn-Unet model using simultaneous contrast and non-contrast images. The authors found the model to achieve high dice scores for kidney segmentation with average dice scores of 0.96.4 Inspection of output segmentation found the model to underperform for non-contrast images as measured by smaller quality assessment scores. Tang et al. took a similar approach where early and late arterial phase scans were used to train a patch-based network to segment renal structures. The group found the model to perform adequately on test data, with no distinction between late and early arterial phase performance. Lee et al. attempted to reduce the dependency of labeling multiple phases by using paired samples where only the contrast-enhanced volume was annotated. They were able to outperform existing models but were limited by the need for correct anatomical correspondence between scans. Ananda et al. removed the dependency for paired samples by training a dual discriminator-based network where the model is trained in three phases. One phase ensures the consistency of segmentation of contrast phase images; the following two phases ensure the image encoding and output maps are not significantly different from contrast and non-contrast images. Dinsdale et al. implemented a similar multi-step approach to improve segmentation quality by improving resiliency to agerelated physiological changes in Brain MRI, where cross entropy(CE) loss is utilized for training a discriminator on the patient’s age using features from the bottleneck and segmentation map. In contrast, the final phase uses a confusion loss, penalizing the model for having greater confidence for a particular age.8 Despite all the modeling effort, the application of evaluating patients with impaired kidney function is challenging since variation occurs not only due to phase of contrast but also the level of renal function, and thus the consistency of imaging appearance is not guaranteed. Establishing the need for techniques that would improve the robustness of kidney segmentation models. Within the scope of this work, we proposed various techniques to generate models resilient to different contrast phases and externally validated the models.

During medical interventions, cone-beam computer tomography (CBCT) imaging is a very powerful tool for guidance and assessment of the intervention’s success. In many applications, such as heart imaging or lung interventions, an automatic segmentation could improve the user’s interaction with the system. For automatic segmentation using deep learning-based methods a lot of labeled data is required, which makes these methods challenging to use on CBCT data as there is typically no data publicly available. In this paper, we make use of publicly available databases of computer tomography (CT) data set, to perform a domain adaption from CT to CBCT data by forward projection and reconstruction. Via this geometric domain adaptation, artificial CBCT volumes are produced with the great advantage that the segmentation of the original CT data can be re-used. We train a neural network, based on the U-Net, on this data to evaluate the impact of the domain adaptation on the quality of the segmentation of the lungs. The results of our experiments show a great improvement in the dice score of the predicted segmentation on real CBCT volumes using the artificial CBCT volumes as training data from 0.88 to 0.95, compared to using the original CT data. The presented method can be extended to model further artifacts which are typical for CBCT data, such as metal and motion artifacts.

Quantitative analysis of the regional motion of the left hemi-diaphragm (LHD) and right hemi-diaphragm (RHD) can provide information regarding the distribution and severity of abnormalities in individual patients with conditions that affect respiration such as thoracic insufficiency syndrome (TIS). Such motion can be captured effectively from dynamic magnetic resonance imaging (dMRI) which does not involve ionizing radiation and can be obtained under free-breathing conditions. The analysis of motion can be performed on the diaphragm using 4D images constructed from dMRI, which in turn requires diaphragm segmentation in the 4D images. In this paper, we present our methodology for segmentation of the left and right diaphragms, which has been implemented in three steps: recognition of diaphragm, delineation of diaphragm, and splitting of diaphragm along the mid-sagittal plane into LHD and RHD. The challenges involved in dMRI images are low resolution, motion blur, suboptimal contrast resolution, inconsistent meaning of gray-level intensities for the same object across multiple scans, and low signal-to-noise ratio. Utilizing 200 and 100 3D images for training and testing, respectively, an average location error of one and a half voxels is achieved for the recognition step. For the delineation step, an average mean-HD of one and a half pixels is achieved. The mid-sagittal plane is identified within a quarter of a voxel. These results are promising, showing that our system can cope with the aforesaid challenges.

Histopathology analysis of thyroid nodule is the current gold standard for the differential diagnosis of thyroid tumors. Deep learning methods have been extensively used for the diagnosis of histopathology images. We look into the possibility of the differential diagnosis of thyroid tumors by analysing histopathology images of thyroid nodule capsules using different deep learning methods. Residual Network (ResNet), Densely Connected Network (DenseNet) and Vision Transformer (ViT). Our study shows the superiority of the histopathology images of thyroid nodule capsules for the differential diagnosis of thyroid tumors compared to histopathology images of thyroid nodules.

The three-dimensional reconstruction of bronchopulmonary segments based on computed tomography (CT) is very critical in lesion, lung cancer localization and surgical resection. However, there is currently no fast and accurate method for three-dimensional reconstruction of pulmonary segments, and the process of labeling pulmonary segments needs to rely on other information such as bronchi and blood vessels, which will greatly consume the time and mental cost of doctors. In this paper, based on the principle of pulmonary segments division, we propose a two-stage fast pulmonary segments division method based on segmental bronchi. Specifically, for a CT image, we employ two well-trained nnUNet models in the first stage to accurately segment 5 lobes and 18 segmental bronchi, respectively. This is because each pulmonary segment should encompass its corresponding segmental bronchi, while lung lobe boundaries exhibit greater distinctiveness compared to those of pulmonary segments. In the second stage, we consider the distance from each pixel point to the segmental bronchi of various pulmonary segments in each lobe, and further divide each lobe to obtain the final 18 types of segments. Finally, we visually validated the rationality of the results by employing the principle of using pulmonary veins as demarcations for pulmonary segments.

The tongue is an important organ in the oral cavity. It can provide information about the oral cavity and physical conditions, and it is also one of the references for traditional Chinese medicine diagnosis. The segmentation of the tongue is a crucial stage in computer-assisted tongue diagnostic systems. Existing methods for segmenting images of the tongue are based on standard dataset, cannot be generalized without a large number of training data from different sources, making it difficult to adapt to mobile devices. A new method for automatically segmenting tongue images by combining traditional image processing and small sample deep learning is proposed. In a complicated context, the Yolo-V5 target detection module is employed to acquire the tongue area. A unified Gaussian distribution is utilized to adjust the color of this region to minimize the negative impact of varied colors on segmentation. Then, for precise segmentation, an enhanced Unet with RFB and attention mechanism is input. The potential noise is then eliminated using a morphological combining process. This technique enhances the segmentation performance of non-standard tongue photos taken by mobile devices by 8% to 10% compared to a single segmentation network, and the average DSC and IoU on non-standard dataset are 95.62% and 91.70%, respectively. It is anticipated that the suggested technique would be applied in stationary and mobile computer-assisted tongue diagnostic equipment due to its improved multi-environment robustness.

Ophthalmic examinations, including fundus and optical coherence tomography (OCT) imaging, are integral in diagnosing systemic diseases affecting the eye, such as diabetic retinopathy. Recent studies have highlighted the related pathological features between retinal and cerebral small vessels, suggesting that retinal microvascular changes could reflect the status of cerebral small vessel disease (CSVD). In this research, we assessed patient’s fundus and OCT images, with a focus on CSVD severity. By incorporating patient demographic data, we developed a sparse Bayesian-based model using retinal and fundus vascular imaging for CSVD diagnosis, achieving an accuracy (ACC) of 71.71% and an area under the curve (AUC) of 73.35%. Our findings indicate that ophthalmic examinations can be a cost-effective screening method for CSVD. Implementing this methodology in optician and eye clinics may substantially lessen societal and familial impacts relative to conventional magnetic resonance imaging (MRI) diagnostics. Additionally, this study identified crucial early warning signs of severe CSVD, such as global venous width, and provided valuable insights into the nature of the disease.