2.1

CADL-VIF: A Context-Aware Local DL-VIF Design

VIF provides an estimation of the similarity between a test image (motion-corrupted in our case) and a reference image (motion-free) by quantifying the information preserved after a distortion process (motion, in this work), weighted by a convolution channel, designed as a surrogate of the human visual system response. VIF acts on the integral of the information across the complete image, providing a scalar output. In this work, we extend the definition of VIF to incorporate localized 3D estimation of preserved information.

The resulting VIF_L, was based on the approach in Ref [7] and adapted to 3D volumes. The key difference between VIF_L and original VIF is that when calculating the information contained in motion-corrupted (I_MC) and motion-free (I_MF) images, VIF_L preserves spatial information by omitting integration across the image dimensions, summing only across different filter channels, as shown in Eqs. 1 and 2.

Where the is the variance in motion-free image after convolution with kernels designed to mimic the frequency response of the human visual system (HVS), quantifies the variance introduced by motion artifacts, and represents the noise in the HVS channel. The term g provides an estimate of the degradation in image information due to patient motion, which depends on the covariance between the motion-corrupted and motion-free images. All terms in Eq. 1 and Eq. 2 were set and calculated as described in Ref [3].

The new VIF_L was reproduced with a novel deep CNN based on our previous DL-VIF design and illustrated in Fig. 1. The network acts on small regions of interest (ROIs) of size 64x64x64 voxels (1 mm isotropic voxel size). Context information is incorporated via a second branch acting on the entire motion-corrupted volume reconstructed at a very coarse voxel size (4 mm isotropically), resulting in a multi-resolution, context aware architecture, inspired by previous work on CT to MR image synthesis [8]. The two branches featured identical layer configurations, with an input 7x7x7 convolution layer and a leaky-ReLU activation, followed by 3 ResBlock layers [3]. Thus, both branches contained an equal number of learnable parameters, yet independently learned. Both branches output 32-channel feature maps, one incorporating local, high-frequency features, and the other providing contextual feature information.

Figure 1.

The network architecture of CADL-VIF Map and its integration into the deformable motion compensation framework.

The high-resolution branch incorporated an input batch normalization layer, while the low-resolution branch featured instance normalization. The different normalization responds to different variability presented by the high and low-resolution data. The high-resolution ROIs featured variable soft-tissue and bone regions that present a much larger variability and benefit from batch normalization, compared to the relatively consistent appearance of the low resolution, full abdomen, context.

The context feature map from low-resolution branch was then up-sampled twice and cropped accordingly to match the position and size of the high-resolution ROI. The local and context feature maps were concatenated and input to cascade of 1x1x1 convolution and leaky-ReLU layers to generate the output of high-resolution branch. To facilitate contextual feature learning of the network during training, another series of 1x1x1 convolution with leaky ReLU layers were added to the low-resolution branch after the feature maps, generating low resolution output for the entire volume.

2.2

Deformable Motion Compensation Framework

CADL-VIF was incorporated as the autofocus metric in a multi-region-based motion compensation framework (see Fig. 1). The time-varying motion vector field (MVF) was estimated with a 4D spline model, integrated into the backprojector algorithm. Deformable motion was estimated as the set of 4D spline coefficients P minimizing the multi-ROI autofocus function:

Where S is the autofocus metric for high-resolution ROI, i.e., the high-resolution CADL-VIF Map, calculated from μ_LR(P), the low-resolution image reconstructed with P, and μ_LR(P, r), the high-resolution image reconstructed with P at ROI position r. The negative natural logarithm served as a basic conditioning of the values for optimization. The final autofocus metric was integrated across ROI voxels and across all ROIs. The cost-function was minimized with the Covariance Matrix Adaptation Evolutionary Strategy (CMA-ES) [9].

2.3

Data Generation and Training

The CADL-VIF network was trained and validated on simulated data generated using 75 cases from the TCIA lymph node abdomen multi-detector CT (MDCT) database. 61 cases were used for training, 7 cases reserved for validation, and the rest 7 cases for testing. For each simulation instance, a MDCT volume was randomly selected and a 260 mm long subvolume at a random longitudinal position was extracted from the original volume. The sub-volume was then forward projected using a high-fidelity CBCT projector with geometry pertinent to interventional robotic C-arm systems (source-to-detector distance of 1200 mm, and source-to-axis distance of 785mm). The detector was modeled as a flat-panel with 864 x 660 pixels and 0.64 mm isotropic pixel size. Deformable motion was induced during forward projection, using a MVF with random maximum amplitude ranging from 10 mm to 30 mm at random directions. The MVF featured maximum amplitude at a randomly placed position and decayed smoothly following an elliptical Gaussian kernel with lateral and antero-posterior width randomly chosen from 200 mm to 300 mm and 100 mm to 150 mm, respectively. The MVF followed a cosine temporal pattern with random phase and random frequency ranging from 0.75-1.25 cycles per scan. An additional motion-free scan was simulated to obtain a reference for ground truth VIF_L. Both the motion-corrupted and motion-free images were reconstructed on a 256x256x256 voxels volume with 1x1x1 mm³ voxel size.

Ground truth VIF_L maps of motion-corrupted images for training were computed using the motion-free images as reference. To emphasize motion artifacts in soft tissue regions, a [0.01, 0.025] mm^-1 window, followed by CLAHE contrast-enhancement were applied to the motion-corrupted and motion-free volumes before calculation of VIF_L. A total of 610 motion instances were generated for training, 70 for validation, and 70 for testing.

During training, the input data was normalized to [0,1], and data augmentation was achieved via addition of zero mean Gaussian noise with σ = 0.01. For each training instance pair, a randomly placed 64x64x64 voxels sub-volume was extracted from the motion corrupted image for input to the high-resolution branch, while the full volume, downsampled by a 4x factor, provided the low-resolution contextual input. Training was achieved with a loss function based on the mean square error between the network output and the ground truth VIF_L. The network was trained with the Adam optimizer (5x10^-4 learning rate), with a batch size of 30, and for 2000 epochs, on 3 Nvidia Quadro RTX A6000 GPUs.

2.4

Validation Experiments

To test the generalizability of CADL-VIF, a separate dataset with 256 cases was created from the test TCIA volumes, using a larger range of motion frequency: 0.5 to 3 periods per scan. CADL-VIF was estimated on contiguous ROIs of 64 x 64 x 64 voxels, covering the complete 256 mm x 256 mm x 256 mm volume. The sum of CADL-VIF within each ROI was then compared with the sum of ground truth VIF_L in the same region. Results were aggregated from all 64 ROIs in each of 256 cases.

Motion compensation with CADL-VIF was evaluated on 7 simulated cases created analogously to the test dataset, with motion amplitude ranging from 8 to 15 mm, and frequency ranging from 0.8 to 1.2 periods per scan. Motion compensation was performed with a total of 4 local ROIs of 64 mm x 64 mm x 64 mm size, with a common contextual reference, and using a 9x9x9x5 4-dimensional b-spline grid. Results were assessed with SSIM and blurriness estimated using the formula proposed in Ref [10], adapted to 3D, which is calculated as follows: i) each voxel in motion-free and motion-corrupted images is compared with all its neighboring 26 voxels, and the maximum intensity variation is stored in two new volumes, V_MF and V_MC for the motion-free and motion-corrupted images, respectively; ii) The average value of V_MF and V_MC is calculated (Z_MF and Z_MC, respectively); iii) blurriness is defined as .

SIM was computed on 6mm x 6mm x 6mm ROIs, that underwent rigid registration to avoid phase mismatches between the motion-corrupted volume, compensated volume, and the static reference. Two ROIs were used per volume, one placed at the center of the motion field and a second one at a quasi-static region.

3.1

DL-VIF Map and Deformable Motion Severity

Figure 2 (A) shows the variation in VIFL as a function of motion amplitude and motion frequency, to validate is capability to accurately quantify the effects of motion on image patches. VIF_L values show decreasing trend with increased amplitude (larger motion) and motion frequency (faster motion), making it a suitable metric to quantify motion induced image quality degradation. Validation of the capability of VIF_L to quantify motion is accompanied by assessment of the agreement between reference-based VIF_L maps and CADL-VIF inferences, illustrated in Fig. 2 (B). CADL-VIF showed good agreement with reference VIF_L across volumes throughout the entire extended dataset, achieving a slope of 0.941 and a linearity of 0.843. Combining these two results: i) VIF_L can accurately reflect the severity of local motion-induced image quality degradation for an extended range of deformable MVF; ii) the combination of contextual and local features learned with CADL-VIF are representative of motion artifacts decreasing the local value of VIF_L, making CADL-VIF a suitable metric for reference-free motion quantification.

Figure 2.

(A) Sum of VIF_L across the entire volume as a function of nominal motion amplitude, for a set of motion frequency values. (B) Agreement between the reference-free CADL-VIF inferences and conventional, reference-based VIF_L, both integrated over 64x64x64-voxel

3.2

Motion Compensation with CADL-VIF Map

Figure 3 quantitatively illustrate the performance of deformable autofocus based on CADL-VIF for compensation of motion using a multi-ROI approach. CADL-VIF autofocus resulted net improvement in SSIM for all 7 motion cases, yielding an average increase in SSIM of 0.129 for regions with severe motion and 0.113 for regions with mild motion. It is worth noting that the seemingly low SSIM is likely due to the low contrast in soft tissue and the presence of noise, and it is the increase in SSIM value that reflects improved image quality, as illustrated in Figure 4, which shows an example compensation of severely motion-distorted anatomy. Figure 3 (B) show the reduction of image blurriness after motion compensation with CADL-VIF autofocus, with an average reduction of 0.032 in severe motion regions and 0.052 in mild motion regions, which is expected in successful motion compensation.

Figure 3.

SSIM values (A) and blurriness (B) before and after motion compensation for regions with severe (black triangles) and mild motion (grey stars). The dashed lines mark the unity line, in which motion compensated and motion-corrupted images are equivalent. Values above the unity line in (A) indicate a net increase in SSIM after motion compensation. Values below the unity line in (B) are associated to sharper images.

Figure 4.

Example of an instance of motion compensation using the CADL-VIF, on one of the test anatomies and motion trajectories.

Consistent improvement in SSIM and reduction in blurriness can be qualitatively appreciated in Figure 4. Motion artifacts severely distorted anatomical structures, with noticeable impact at the center of the volume (region of larger motion amplitude). The distortion and blurriness resulted in a sizable reduction of SSIM, to a value of 0.157. Autofocus motion compensation with CADL-VIF successfully restored the appearance of anatomical structures, mitigated shape distortion, and reduced image blurring, yielding a 3-fold increase in SSIM to 0.469.