1.1.

Related Work

To develop a practical system with high classification accuracy, modularization structure is often preferred over a deep learning approach that simultaneously performs detection and segmentation because modularized components allow for flexibility and adaptability as shown in Fig. 3 and Refs. 1213.–14. We consider segmentation and classification to be two separate modularized components or subsystems. Additionally, direct classification from the input videos is extremely challenging because these are dynamic images evolving over time.

Fig. 3

Automated segmentation and classification system overview.

In this paper, we focus on the classification component. There has been very limited work on building an automated classification system for stem cells in video with both labeled and unlabeled datasets.⁸ Niioka et al.¹⁵ used convolutional neural network (CNN) to study cellular differentiation from myoblasts to myotubes. Their classification model was built upon the concept that cellular morphology changes during differentiation, and this feature was easily captured in stained fluorescent images. In addition, Xie et al.¹⁶ worked on fluorescent images with CNN for cell counting. Although they have a successful experiment, their classification problem was simple since their images contained only circular dots. Chang et al.¹⁷ also used CNN for human induced pluripotent stem cell regions classification.¹⁷ Their study focused on classifying cell cluster patterns. The dataset used in the works by Niioka et al.,¹⁵ Xie et al.,¹⁶ and Chang et al.¹⁷ came from experiments that use staining techniques; staining is a very intrusive technique to be used on cells for contrast enhancement. However, our hESC experiments were done without staining.

Similar work on stem cell classification with phase contrast images was proposed by Theagarajan et al.^18,19 They suggested using a generative method to train the network and classify real data. However, they did not consider realistic unlabeled data, which can be efficiently generated for training; typical generative methods have huge computational cost for synthetic dataset generation as well as training with a large set of synthetic data. Therefore, this paper proposes using the unlabeled data (without the use of generative methods) for model training and fine-tuning the model with labeled data.

1.2.

Contributions of this Paper

In this paper, we focus on the classification component. From Fig. 2, we can infer that there are four major challenges in hESC classification. First, when attached cells spread thin in the substrate, the cells are fused with the background. Second, dynamically blebbing cells and apoptotically blebbing cells are similar in intensity. Third, when a large attached cell goes through the apoptotic process, it appears as a cell cluster of apoptotically blebbing cells. Fourth, image data are obtained under both 10× and $20\times$ objectives, which adds challenges in discerning individual blebbing cells from cell clusters. In light of the state of the art, the contributions of this paper are as follows.

Section 2 presents the materials and methods in detail. Section 3 provides experimental results, and Sec. 4 provides a discussion on the proposed and compared methods. Finally, Sec. 5 presents the conclusions of the paper.

2.1.

Materials

All time lapse videos were obtained with the phase contrast microscope in BioStation IM.^7,11 The videos were acquired using either a $10\times$ or $20\times$ objective with $600\times 800\text{pixel}$ resolution. A total of 27,603 unlabeled gray scale images and 3559 labeled gray scale images were obtained from six $10\times$ videos and eight $20\times$ videos. Both unlabeled and labeled images were obtained automatically by the method described in Guan et al.^3,20,21 The labeled dataset had the following number of gray scale images for each class: (1) 636 cell cluster images, (2) 773 debris images, (3) 519 unattached cell images, (4) 704 attached cell images, (5) 413 dynamically blebbing cell images, and (6) 514 apoptotically blebbing cell images. The ground-truth for the datasets were generated manually by stem cell experts. We used 75% of the dataset for training and the remaining 25% of the dataset for out-of-sample testing for each class. To generalize the classifier, five-fold cross validation was done during model learning. Model learning is performed with training data only.

2.2.

Methods

In this section, we first present the motivation for our proposed approach. This is followed by a method for automated cell region detection, which is the segmentation component. We then describe RandNe and elaborate on the autoencoded feature extractor as well as the pre-trained subnetworks for the classification component. The classification component is part of the modularized system as shown in Fig. 3. A pseudocode for building the RandNet model is also provided.

2.2.2.

Segmentation component

Guan et al.³ proposed a model based method for automatically segmenting hESCs. This automated cell region detection is an essential algorithm in developing automated frame component decomposition in hESC phase contrast videos. They considered the foreground and background intensity distribution to be a mixture of two Gaussians. The objective of their algorithm is to find an optimal threshold that optimizes a criterion derived from the intensity distribution of foreground and background. The optimal segmentation is achieved at the highest criterion value. Since the segmentation method yields a binary image for each frame, we were able to extract a pool of individual components from each frame. Figure 4 shows the detected components of frames under $10\times$ and $20\times$ objectives. These detected components are then ready to be classified into one of the six aforementioned classes.

Fig. 4

Detected components on each frame:³ (a) image under $10\times$ objective and (b) image under $20\times$ objective.

2.2.3.

Classification system overview

The proposed classification system is built with both labeled and unlabeled data, and it consists of many random pre-trained subnetworks. The proposed method utilizes unlabeled data to build the encoder component in the pre-trained subnetworks and labeled data to fine-tune the RandNet. The RandNet structure also incorporates ensemble methodology to constrain overfitting. Figure 5 shows a graphical depiction of how RandNet is built with pre-trained subnetworks and the ensemble concept.

Fig. 5

(a) Autoencoder network; (b) pre-trained subnetwork; (c) RandNet with autoencoded feature extractor.

2.2.4.

Random network

RandNet utilizes the concept of bagging in deep learning by creating subnetworks. Bagging or bootstrap aggregation is a machine learning concept used to reduce variance and avoid overfitting.^22–25 RandNet, developed in this paper, is a method that contains many subnetworks that have a common pre-trained model and are fine-tuned with random samples. RandNet uses all of the results from each subnetwork and passes it to a stacking network in which the final decision is made. The detail of the stacking network is shown in Fig. 6. The stacking network is designed to be simple and has only two main dense layers.

Fig. 6

Stacking network. (Note: Dimensions without parentheses are kernel dimensions of the current box, and dimensions with brackets are output dimensions of the current box.)

2.2.5.

Autoencoded feature extractor

The autoencoder network is an efficient unsupervised learning method that learns the representation of a set of data. The autoencoder network contains two major components: encoder and decoder.^26–28 In this paper, we used a structure similar to AlexNet as the basis of an encoder, and then we designed a decoder network from it. Although the VGG architecture²⁹ slightly outperforms AlexNet³⁰ as shown in Sec. 3.3, this difference is not significant, and since the AlexNet architecture requires reduced computational resources, we chose it for its simple implementation. As shown in Fig. 5(a), the encoder generates a set of latent representations for the unlabeled data. The details of both encoder and decoder structures are shown in Fig. 7. The autoencoder network used the Adadelta optimizer³¹ and the pixel-wise binary cross-entropy loss function. Since the final layer in the autoencoder network was chosen to be a sigmoid activation layer, pixel-wise binary cross entropy is an applicable loss measure. The loss function equation is given as follows:

Eq. (1)

$$Los{s}_{AE}=-\sum _{i=1}^{N_S}\sum _{r=1}^{N_R}\sum _{c=1}^{N_C}{I}^{(i)}(r,c)\log (K^{(i)}(r,c))+(1-I^{(i)}(r,c))\log (1-K^{(i)}(r,c)),$$

where $Los{s}_{AE}$ is the total pixel-wise loss in the autoencoder network, $N_S$ is the total number of sample images in a batch, and $N_R$ and $N_C$ are the total number of rows and columns, respectively. $I^{(i)}(r,c)$ and $K^{(i)}(r,c)$ are the ground-truth and predicted label values, respectively, in the $r$’th row and $c$’th column for the $i$’th sample. Both $I^{(i)}(r,c)$ and $K^{(i)}(r,c)\in [\mathrm{0,1}]$.

Fig. 7

Architecture of the autoencoder network. (Note: Dimensions without parentheses are kernel dimensions of the current box, and dimensions with brackets are output dimensions of the current box.)

2.2.6.

Pre-trained subnetwork

The subnetwork used the encoder structure derived from the autoencoder network [in Step 2, Fig. 5(b)] as the basis for building a subclassifier. Each pre-trained subnetwork is fine-tuned with random samples and has a topper structure. The layers of the topper structure are shown in Fig. 8.

Fig. 8

Topper structure. (Note: Dimensions without brackets are kernel dimensions of the current box, and dimensions with parentheses are output dimensions of the current box.)

Since the encoder structure was unfrozen in each subnetwork, the fine-tuning with random samples affects the weights in the encoder structure. Therefore, we were able to emulate bagging for the proposed method. For this subnetwork, we use categorical cross entropy as our loss function, which is given as

Eq. (2)

$$Los{s}_{\mathrm{CCE}}=-\sum _{i=1}^{N_S}\sum _{j=1}^M{y}_{(i,j)}\log (p_{(i,j)}),$$

where $Los{s}_{\mathrm{CCE}}$ is the total categorical cross entropy in the pre-trained subnetwork. $N_S$ and $M$ are the total number of samples images and classes in a batch, respectively. $y_{(i,j)}$ and $p_{(i,j)}$ are the ground-truth and predicted values, respectively, for $i$’th sample and $j$’th class, where, $y_{(i,j)}$ and $p_{(i,j)}\in \{0,1\}$. Table 1 shows the pseudocode for building the classifier model.

Table 1

Pseudocode for building the classifier model.

Algorithm 1: Random Network with Autoencoded Feature Extractor
Input: $D_{\text{unlabeled}}$ is a set of unlabeled data
$D_{\text{labeled}}$ is a set of labeled data
$n$ is the total number of subnetworks
Output: Final learned model
1. Procedure RandNet($D_{\text{unlabeled}}$, $D_{\text{labeled}}$, n)
2. Normalize $D_{\text{unlabeled}}$ and $D_{\text{labeled}}$ by dividing by 255
3. Train Autoencoder network with unlabeled data [Step 1, Fig. 5(a)]
4. Extract encoder structures from autoencoder network for subnetwork training [Step 3, Fig. 5(b)]
5. Create n subset of labeled data with stratified bootstrap.32 These subsets are used to obtain n subnetworks
6. Fine-tune n subnetworks with the above datasets
7. Connect the output from n subnetworks with stacking technique [Step 3, Fig. 5(c)]33
8. Train the final network with all of the training data
9. Obtain the final model
10. End procedure

3.1.

Parameters and Optimization

In our approach, all cropped images after the detection module were resized to $224\times 224$ with bicubic interpolation, and the image intensities were normalized by dividing them by 255. No additional data augmentation was performed. For the autoencoder network, each subnetwork was trained independently, and the latent representation of the subnetwork was used to train the topper network. There are two fixed parameters for each subnetwork: epochs and batch size, which are set to be 10 and 128, respectively. The default Adadelta optimizer is used for the autoencoder network.³¹ For RandNet, there are five parameters: epochs, batch size, number of subnetworks, learning rate, and decay rate. We used 25 epochs with early stopping, a batch size of 50, and a total of 33 subnetworks. We also used a default Adam optimizer³⁴ with the learning rate of 0.001. All parameters are fixed except the number of subnetworks, which has a search range from 1 to 37 with a step size of 2. Figure 9 shows that, when the number of subnetworks equals 33, it has the highest average validation accuracy as well as the lowest average validation loss. It should also be noted that the processing speed for our approach using all 33 subnetworks during inference is 6.25 frames per second (FPS) compared with 4.16 FPS using the approach proposed by Theagarajan et al.¹⁹

Fig. 9

Five-fold cross-validation results. (a) Mean accuracy vs. number of subnetworks curve; (b) mean loss vs. number of subnetworks curve.

Using an ensemble of classifiers is similar to using dropout during training, but they are not the same.³⁵ Ensemble training focuses on training each network with a different subset of data while dropout reduces feature spaces randomly. Although both ensemble method and dropout can generalize the network, the former influences the model with data and the latter manipulates the extracted features. The proposed method uses a simple subnetwork, and each subnetwork was trained independently; therefore, dropout was not considered in each subnetwork. Most importantly, data-driven model preserves all essential features for reconstructing the input image in a simple autoencoder network. Figure 10 shows the comparison of the reconstructed images with and without dropout. It can be seen that when we use dropout the reconstructed images are blurrier due to missing feature information.

Fig. 10

Visual comparison of images reconstructed using our approach of ensemble training versus 25% and 50% dropout rate.

3.2.

Performance Measures

For performance analysis and comparison, we used the confusion matrix for evaluation.³⁶ The following equations show the calculations for the overall and individual classification accuracy from the confusion matrix. The average classification rate and individual true positive rate (TPR) are given by the following equations:

There are three different categories of accuracies in evaluating the performance of a model: (1) training accuracies, (2) validation accuracies, and (3) out-of-sample testing accuracy. Training and validation accuracies refer to cross validation accuracies for training and validating sets, respectively. The out-of-sample testing accuracy is slightly different than the validation scheme. Once the best model parameters are learned from the model selection process, the final model is obtained with the entire training dataset and the best parameters. This final model is then used to evaluate the performances of the testing dataset, and it produces the out-of-sample accuracy. Typically, training and validation accuracies show us the estimated bias and variance in the final model while out-of-sample testing accuracy shows the true variance in the final model.

3.3.

Experimental Results

The proposed RandNet is compared with the state-of-the-art methods as reported in Table 2. The top two performers are the proposed RandNet and the fused CNN triplet.¹⁹ The proposed RandNet has 97.23% mean accuracy in a five-fold cross validation and a seemingly low standard deviation in its validation results. The reason that both RandNet and fused CNN triplet outperformed other methods is that additional data are being used. Both aforementioned methods were trained with data other than the given labeled data. The RandNet used unlabeled data to pre-train its subnetworks and then fine-tuned it with the labeled data. On the other hand, fused CNN triplet¹⁹ used both synthetic data and real labeled data in training. ResNets,³⁷ VGGs,²⁹ and AlexNet³⁰ were trained with only labeled data. Consequently, they seem to have higher variance in their performances. They also perform worst in out-of-sample testing, as shown in Table 3.

Table 2

Five-fold cross-validation results.

Table 3

Testing data results.

4.1.

Misclassification Samples

The proposed method had at least 93% TPR/recall for each individual class, as shown in Table 5. It performed better in identifying attached cells, with a total of 98.30% recall. However, it performed worst for unattached cells. Unattached cells are generally easy to identify as shown in Fig. 2(c).

Table 5

Individual recall for RandNet.

From the typical misclassified images in out-of-sample testing as shown in Fig. 11, we conclude that the blurring effects in the autoencoder network might be the cause for misclassifications. As shown in Figs. 11(b) and 11(c), two unattached cells were blurred out after passing through the autoencoder network. Therefore, these cells looked similar to the attached cells visually. Moreover, this blurring effect might be more significant on the hidden representation generated by the encoder that was used to build the subnetworks.

Fig. 11

Typical misclassified images in out-of-sample testing: (a) cluster predicted as apoptotic cell; (b) unattached cell predicted as attached cell; (c) unattached cell predicted as attached cell; (d) attached cell predicted as cluster; (e) dynamic blebbing cell predicted as attached cell; (f) dynamic blebbing cell predicted as cluster; (g) debris predicted as apoptotic cell; (h) debris predicted as dynamic blebbing cell. (Note: Recovered images are obtained from the autoencoder network.)

4.2.

Additional Experiments

We compared our approach with Mask RCNN;³⁸ our approach achieved a Dice coefficient of 0.86, while Mask RCNN achieved 0.92. To train the Mask RCNN, we used 50% of the data for training. A significant difference between the two approaches is that our approach has only four learnable parameters as described in Ref. 3, while Mask RCNN has 43.9 million learnable parameters. Moreover, the approach proposed by Guan et al. can run on a single Intel i7 CPU³ while a Nvidia 1080Ti GPU is required to train the Mask RCNN model. Additionally, our detection algorithm is completely unsupervised, whereas Mask RCNN is supervised and requires annotated training data.

Further, we replaced the segmentation component proposed by Guan et al.³ in our approach with Mask RCNN³⁸ and passed the segmented images as input to our classification component. The classification results and recall for each cell types are shown in Tables 6 and 7, respectively.

Table 6

Confusion matrix for RandNet using Mask RCNN as the segmentation component.

Table 7

Recall of each cell type for RandNet using Mask RCNN as the segmentation component.

As shown in Table 7, the recall for each cell type was above 89%, and the proposed classification component had an accuracy of 93.79% on the Mask RCNN segmented images. Since the proposed classification component was not trained with samples from Mask RCNN, a small accuracy degradation was expected. The proposed classification component still showed good performance reliability on data samples that were not generated by the proposed segmentation method.