2.1.

End-to-End Image Reconstruction Model

We opted for a U-Net network with an attention mechanism²⁸ as the core of our DL modules. Created for cell-related semantic segmentation tasks,²⁹ U-Net is known for its excellent performance and straightforward design characteristics. This has earned it widespread use in the biomedical realm,³⁰ allowing it to address complex optical challenges^15,27 effectively. Unlike most tasks related to natural or biomedical images, the FPM reconstruction task usually requires multiple images as input. As a result, we designed an extra head module to process these images. We merge the collected LR images along their channel dimension and resize them using a $1\times 1$ convolution layer with a pixel shuffle layer to prepare them for processing in the first step, as shown in Fig. 2(a).

It is well known that the quality of the training data set determines the performance of the neural network. However, when it comes to rare samples, collecting enough high-quality data is a significant challenge. For example, the resolution target is a typical representative of rare samples. The most informative part of it is limited and comprises just a small fraction of the entire sample. This often results in networks overlooking these crucial areas during training. Therefore, it is particularly important to collect large amounts of data for training. However, capturing data by taking multiple imaging processes of the same resolution target is labor-intensive and consumes substantial resources. In contrast, producing vast quantities of high-quality data through simulations that accurately model the imaging process proves to be a far superior approach. Nevertheless, blindly augmenting data without considering sample characteristics may not aid in training and could even be detrimental. It could also reduce the model’s interpretability. To address this, we adopted a data augmentation method that considers sample characteristics. We start by separating the samples into the background and region of interest (ROI), as shown in Figs. 3(a1), 3(a2), and 3(a3). Then, we mix these ROI parts back into the background in various rotated and flipped ways and at different spots. The corresponding LR images are obtained by performing a forward propagation process of the physics model. We categorize the samples based on how many detailed parts they have, labeling them as either simple or complex, as shown in Figs. 3(b) and 3(c). Under the training process, our model learns to rebuild images from simple data sets first and then gets better at picking up details with more complex ones. Notably, with the framework we proposed, even if the neural network is only trained on simulation data, it can still accurately reconstruct HR images from actual, real-life experiments.

Fig. 3

The overview of our proposed data augmentation methods. (a1) The ground truth of the resolution target. (a2) The background we extract. (a3) The ROI of the resolution target. (b) An example of simple samples. (c) An example of complex samples.

2.2.

Physics Model

In the framework of FPM, the sample under observation is assumed to be a two-dimensional flat thin layer. Its optical properties can be expressed by the complex transmission function $o(r)=\exp [i\phi (r)-\mu (r)]$, where $\phi (r)$ represents the phase modulation distribution, and $\mu (r)$ indicates the absorption distribution. For a typical FPM light source, usually a programable LED array, illumination from each LED unit located at coordinates ($x,y$) is treated as a spatially coherent local plane wave. The outgoing wave passing through the sample can be described as $u(r)=o(r)\exp (i{k}_m·r)$. In this equation, $k_m=(\sin {\theta }_{xm}/\lambda ,\sin {\theta }_{ym}/\lambda )$ indicates the frequency shift relative to the sample’s spectrum center, where “$\lambda$” denotes the wavelength, and (${\theta }_{xm}$, ${\theta }_{ym}$) denotes the angle of incident wave illumination, respectively. Compared with the conventional FPM framework, in which one LED is lit up once, LED illumination multiplexing is proved to be an efficient way to increase the imaging speed of programmable illumination computational microscopy.^9,12,13 When multiple LEDs at different positions are lit up simultaneously, the corresponding single-shot low-resolution image captured by the camera can be described as the sum of the intensities of illumination from each individual LED, as demonstrated in Eq. (1). In this equation, “$I$” stands for the total intensity, $o(k-k_{mi})$ is used to describe the spectrum shift with different illumination angles, $P(k)$ represents the pupil function, $F$ represents the two-dimensional Fourier transformation, and “$n$” signifies the number of LEDs illuminated simultaneously. Obviously, Eq. (1) is a simplification, and we followed previous work without considering many other influencing factors.^11,13 For example, the residual interference between the LEDs is not discussed. However, this level of simplification is sufficient for our framework. We adopt the neural network model introduced by Yang et al.¹⁵ to model the image process and reconstruct the HR image, as illustrated in Fig. 2(c),

Since the reconstruction process of FPM involves a nonlinear phase-retrieval algorithm, it is impractical to explicitly obtain the optimal multiplexed illumination model in each observation. Therefore, in our previous work, we designed an unsupervised adaptive illumination model generation method to get an optimal illumination multiplexing strategy to some extent.⁹ Using LR images taken when the central light is on as a prior, we generated an illumination model for multiplexed FPM. This model was then applied to all the image acquisition parts in our work. As shown in Fig. 4, the illumination model contains a total of 10 illumination patterns, which is much less than the sequential illumination model in a conventional FPM framework.

Fig. 4

The illumination model we generated. The gray circles represent the corresponding sample spectrum range when different LED lights are on.

2.3.

DL Fusion Model

In the last step, we concatenate the outputs from both the DL and physics models as input for the fusion model. We also included a residual connection that merges the output from the physics model with the fusion model’s results before final output to boost the framework’s ability to generalize, as shown in Fig. 2(d). In the training phase of the DL model and the iteration of the physics model, mean absolute error is employed as the loss function for both, as specified in Eq. (2). However, although their optimization goals are the same, they often produce different results, as shown in Fig. 2(b). The reason for these varied outcomes lies in the fundamental structural variances between the neural network and the physics model, which lead them on distinct trajectories during the optimization process. Here, we can describe the collection process of the LR image as $\mathrm{LR}=\mathrm{fp}(\mathrm{HR})$, and the reconstruction processes of the DL model and the physics model as Eqs. (3) and (4), where $\mathrm{fp}(\mathrm{LR})$ represents the forward propagation process of FPM, $\mathrm{LR}$ stands for the collected low-resolution image, and HR represents the true complex amplitude of the sample,

Equations (3) and (4) demonstrate how each reconstruction method transforms the original complex amplitude of the sample into two distinct feature spaces, thereby generating two different modalities. Therefore, any specific area of the sample can be described by the vector $({\mathrm{SR}}_{\mathrm{dl}},{\mathrm{SR}}_{\mathrm{pm}})$, which represents the paired features contained in the two modalities (dl, deep-learning; pm, physical model). It is important to note that extracting information from paired features presents a challenge when working with single-modality data.³¹

3.1.

Experiments on USAF Chart

Following the method proposed in Sec. 2.3, we first created a complex data set and a simple data set for the training of the first DL module. Each data set contains 768 paired data, divided into training, validation, and test sets in a 4:1:1 ratio. Each image pair included an HR image ($1024\times 1024$) and 10 corresponding LR images ($512\times 512$). Due to memory constraints and the inherent workings of convolutional neural networks, we capped the network’s output at a $256\times 256$ size. Therefore, the obtained data were evenly divided into 16 equally sized small blocks, resulting in two data sets, each containing $768\times 16$ groups of data. We first trained a randomly initialized network for 200 epochs on the simple data set and then continued training for 200 epochs on the complex data set. At this point, we obtained the end-to-end deep model for image reconstruction. Subsequently, we used the model trained on the simple data set and the physics model to reconstruct the simple training data set, thereby obtaining the data set for training the fusion model. We avoided using the more advanced model trained on the complex data set for this stage because its high performance could make the fusion model overly dependent on its results, which would impede the model’s ability to learn and use paired features effectively.

To demonstrate that our proposed framework effectively combines the powerful modeling capabilities of DL models with the strong generalization ability of physical models, we used models trained on simulated data for actual experimental data reconstruction. Specifically, we kept the experimental parameters consistent with the simulation process, placed the USAF chart 97 mm from the LED array, and used 470 nm wavelength light with a 20 nm bandwidth for illumination. We observed with a lens with a $4\times$ magnification and NA of 0.1NA and recorded LR images with a camera that boasts a dynamic range of 71.89 dB and a pixel size of $2.4\mu \mathrm{m}$. As shown in Figs. 5(b) and 5(c), the data augmentation method we used prompted the network to pay more attention to the ROI, achieving better reconstruction performance. Figures 5(d) and 5(f) illustrate that with the DL model added semantic insights, the physics model achieved superior recovery results. In contrast with previous work of multiplexed FPM, our framework achieved high-quality Fourier ptychographic microscopy imaging with only 10 collected LR images, as shown in Fig. 5(g). In addition, our method offers better imaging quality for detailed features such as digits 8, 9, and 6, which are challenging for deep learning methods to generalize, as shown in Figs. 5(i) and 5(j).

Fig. 5

Experiments results on the USAF chart. (a) The captured image illuminated by the middle LED. (b) The low-resolution area without reconstruction. (c) The reconstruction result of the DL model trained on a simple data set. (d) The reconstruction result of the DL model trained on the complex data set. (e) The reconstruction result of the physical model (PM) initialized by the image middle LED illuminated. (f) The reconstruction result of the PM initialized by the output of the DL model trained on the complex data set. (g) The final reconstruction result of our framework. (h) The ground truth (GT) reconstructed from 121 LR images captured sequentially. (i)–(m) Detailed outputs from different methods.

3.2.

Experiments on Biological Samples

We conducted tests with biological samples to showcase the real-world applicability and impressive generalization capabilities of our framework. For illumination, we used light with a wavelength of 470 nm and a bandwidth of 20 nm, and we captured LR images using a camera with a $6.5\mu \mathrm{m}$ pixel size. Our training data set for the reconstruction network included various plant sections, such as lotus stem, bamboo stem, and mint stem, while we used broad bean stem sections for the fusion network training, and privet leaf sections to evaluate the framework’s performance. It is worth mentioning that we used completely different biological samples for training and testing, closely approximating real-world application scenarios. The ground-truth images for the training set were created by reconstructing 121 LR images taken under various illumination angles with a physics model. The results, as shown in Fig. 6, illustrate that even with only 10 captured images, our framework could reconstruct the privet leaf section with remarkable clarity, underlining its robust generalization and imaging prowess.

Fig. 6

(a) The captured image illuminated by the middle LED. (b) The LR image without reconstruction. (c) The reconstruction result of the first DL model. (d) The reconstruction result of the physical model (PM). (e) The final reconstruction result of our framework.

3.3.

Ablation Experiments

Next, we conducted comprehensive ablation experiments to confirm the effectiveness of our fusion model in leveraging paired features from two modalities and enhancing final image quality. We experimented with the broad bean slice data mentioned in Sec. 3.2 by changing the input combination of the fusion model during testing and dividing the whole image into 441 smaller segments, allocating them in a 7:3 ratio between training and testing sets (we skipped creating a validation set as it did not require extra adjustments of hyperparameters). Furthermore, we excluded blank areas by sorting each cut image based on its contrast level to avoid extra interference.

The input combinations for model testing were set as (DL_output, DL_output), (PM_output, PM_output), and (DL_output, PM_output), with the results detailed in Table 1. When only the reconstruction results of the DL model were used as input, the final output of the fusion module was nearly consistent with the performance of the physics model. This occurs because without extra information, the model finds it challenging to accurately reconstruct the finer details, and as a result, it tends to adhere closely to the physical model’s output. This adherence is due to the residual connections, as explained in Sec. 2.2. When only the reconstruction results of the physics model were used as input, the results improved. This is because, while learning pairwise features, the single-modal features are also learned by the network. The best results came when we combined inputs from both the DL and physics models. This clearly showed that our framework could identify and utilize the paired features from the results of different methods, leading to a more precise result.

Table 1

Comparison of results from different methods in ablation experiments.

When using DL for recovering blank regions in samples, we observed the appearance of concentric rectangular noise in the recovered areas. This pattern of noise is distinctly different from the global, uniform noise typically seen in results from the physics model, as shown in Fig. 7. The concentric rectangular noise is due to the convolutional neural network’s dependency on its depth to gradually expand the receptive field’s boundaries, leading to noise accumulation at these edges. In contrast, the physics model operates in the frequency domain, providing a consistently global field of view across the image. As a result, the depth model and the physics model can be viewed as analyzing and interpreting the acquired data from different perspectives. The distinct noise patterns in the results from the two reconstruction methods align with the theory we discussed in Sec. 2.2. This observation from our experiments validates the presence of paired features, offering practical evidence of their existence. Through leveraging paired features, our proposed architecture effectively preserves detailed information while filtering out the noise observed in the results mentioned above as shown in Fig. 7(d).

Fig. 7

The comparison of the outputs between the DL and physics model (PM). (a) The LR image without reconstruction. (b) Reconstruction results of sample blank areas using the physics model. (c) Reconstruction results of sample blank areas using the data-driven DL model. (d) The final reconstruction result of our framework.