2.1

Introduction of the E-ASPP Attention Multi-Branch Expansion Module

DeeplabV3+ model can obtain multi-level information through ASPP (Atrous Spatial Pyramid Pooling), but only through simple stitching of various dimensions, it cannot obtain rich contextual information, which is not enough for the extraction of large targets such as tile surface image positioning. Therefore, the E-ASPP [11]attention multi-branch cascade module is added to the encoder structure to improve the ability of context connection and enhance information extraction. The E-ASPP attention multi-branch expansion module is shown in Fig2.

Figure2.

E-ASPP multi-branch receptive field cascade basic module

The E-ASPP multi-branch attention cascade module is divided into two parts: ECA attention module and ASPP multi-branch receptive field cascade fusion structure. The Attention Mechanism ECA module is an improved version of SENet[12](Squeeze excitation). Its structure is shown in Figure 3. ECA module proposes a local cross-channel interaction strategy without dimensionality reduction, which takes into account each channel after Global average pooling (GAP) and its k neighbors to capture local cross-channel interaction information. ECA module first performs spatial feature compression on input feature map to obtain 1×1×C feature map.

Figure3.

ECA attention mechanism unit module

Then the channel feature learning is carried out, and the cross-channel interaction information is captured by one-dimensional convolution of dynamic convolution kernel, and the coverage of local cross-channel interaction is determined by adaptive selection of convolution kernel adaptive function formula. Finally, the output feature channel weight vector generated by the activation function is multiplied with the original input feature graph by channel, and the feature graph with channel attention is output, thus improving the feature extraction. The ECA module avoids channel dimension reduction and allows the model to learn more efficient channel attention. The ECA module has fewer parameters, determined only by the size of its convolution kernel k. The formula for calculating the adaptive convolution kernel k is:

In the formula: k is the convolution kernel size, C is the number of input channels, use γ, b to change the ratio between C and k, generally γ=2, b=1. In ASPP module, the semantic information of images can be extracted at different scales through cavity convolution and pyramid pooling with different sampling rates. However, the parallel branches in ASPP module lack the sharing of the extracted weed feature information, and can not make full use of the spatial features of the data. In order to overcome the limitations of ASPP module, a cascade fusion structure of multi-branch receptive fields of feature sets is proposed.

As shown on the right side of Figure 2, the output of each branch is adjusted to 320 channels by 1×1 convolution, which is used as additional features of the next branch. Through the backbone feature extraction network, a rough feature representation is obtained for each branch. Then, these rough features are fused with the additional features of the previous branch to solve the problem of “void” in ASPP module, so as to ensure the complete transmission of information and reduce data loss. The five feature layers obtained from the output of each branch after the feature hierarchy fusion are spliced, that is, the output after four convolutions is stacked with the output of the pooling layer, so that the advanced semantic feature information with richer semantic information can be extracted by making full use of the feature layers of different scales. At the same time, the common convolution operation is replaced by 3×3 deep separable convolution block in ASPP, that is, deep separable cavity convolution, which makes the network more efficient.

2.2

Backbone Network Replace the lightweight MobileNetV2 model

The on-site production speed of ceramic tiles is fast, and 60 ceramic tiles with length of 800mm×800mm are detected every minute. The DeeplabV3+ backbone network before improvement uses ResNet. Although it has high segmentation accuracy for multiple types of image extraction features, its network complexity is high and the operation time is long. For ceramic tile surface defect images with diverse information and features, with the deepening of model training, the number of network parameters will gradually increase, and the computing speed will be greatly reduced. In order to improve the speed of feature extraction, it cannot meet the needs of ceramic tile on-site production. For this reason, MobileNetV2 is used in this paper instead of ResNet as the backbone network. The inverse residual structure and the linear bottleneck layer in MobileNetV2 network together constitute the linear inverse residual structure, as shown in Figure 4. In the inverse residual structure, point-by-point convolution and deep convolution are adopted. In the convolution process, the channel is first convolved by 1×1 point-by-point convolution Expansion layer, followed by 3×3 deep convolution for each channel, and finally, it is reduced by 1×1 point-by-point convolution dimension (Projection layer). Using shortcut, add the two together and output. The Relu activation function avoids losing the feature information at the tile border after dimensionality reduction. On the last 1×1 point-by-point convolution of the inverted residual structure, the linear bottleneck layer is replaced by the previous Relu activation function for dimensionality reduction, which greatly reduces the loss of low latitude feature information and increases the information dimension, which is of great significance for improving the segmentation accuracy. The number of parameters and computational complexity are shown in Table 1.

Figure4.

Inverse residual structure in MobileNetV2 network

Figure. 5.

Tile image and background segmentation mask

Table1.

Comparison of model parameters and calculation amount

2.3

Improvement of decoder

Since DeeplabV3+ ’s complex CNN structure has 47 convolutional layers, the use of standard convolution will inevitably lead to a large number of cases of overfitting of parameters and networks. To solve this problem, DeeplabV3+ networks use deep separable convolution in the decoder part[12,13] instead of the traditional standard convolution. Formula (2) results in the ratio of depth-separable convolution to standard convolution computation, which indicates that depth-separable convolution has high applicability in solving the number of trainable parameters or overfitting[14.15].The deep separable convolution is used to split the common convolution, reduce the redundant parameters of the model, and accelerate the convergence speed and deduction speed of the model training. At the same time, the reduction of model complexity reduces the risk of overfitting, and can better meet the needs of real-time segmentation of tile surface images.

In formula (2), the multiple of the computation depends on the number of input channels X, Y represents the number of output channels, and the size of the convolution kernel is C_i · C_i.the size of the feature graph is C_e · C_e.

3.1

Collection of data sets

The data set used in the experiment was taken from a tile production factory with 800 images taken from the tile production line by an industrial linear array camera. Among them, 200 images with a length of 700×1500 (mm size) images, 200 images with 800×800 pixels, 200 images with 400×800 pixels, and 200 images with 600×1200 pixels, divided the objects contained in each image into two categories: tile image and background, as shown in Figure 4. Label pixels other than the tile surface image as background classes. The image annotation tool LabelImg was used to generate the annotation mask of the tile, obtain the json file and convert it into an 8-bit grayscale image as a label, and produce the VOC format data set. At the same time, the overfitting problem was avoided and the generalization ability of the model was enhanced. In this paper, random rotation, flipping, Gaussian noise, random brightness and other data were enhanced for the samples, and the size of the dataset was expanded to 3 times of the original, a total of 2400 images and corresponding labels were obtained. The model is trained and verified by dividing training set and verification set according to ratio 8:2. There are 1920 training sets and 480 verification sets.

3.2

Experimental environment configuration and parameter Settings

The processing platform of this experiment is Intel(R) Xeon(R) Gold 5218R CPU, 2.10GHz, 64GB memory, and NVIDIA GeForce RTX 3090 GPU. The software environment is built by Cuda11.5+conda 4.9.2+Python 3.8.8, and the operating system is Ubuntu 18.04. The transfer learning strategy was adopted during model training, the weight parameters of the pre-trained Pascal VOC dataset were used, the initial learning rate was set to 0.009, and the Batch Size was set to 4. The random gradient descent method SGD was used as the optimizer to update the weight parameter, the momentum parameter was set to 0.9, the cross entropy loss function was adopted, and the training was attenuated once every 200 iterations, the number of iterations was set to 100, the learning rate was set to 0.001, and the learning rate gradually attenuated with the increase of the number of iterations.

3.3

Evaluation criteria

In order to objectively evaluate the segmentation effect of the network model on the tile surface image, three evaluation standard indicators were selected in this paper, and the Mean intersection over union (MIou) was used for quantitative evaluation. MIou comprehensively considered the Intersection over Union (IoU), Mean Pixel Accuracy (MPA) and F1 values of model extraction targets and non-targets. If the dataset contains k+1 categories, p_ij represents the number of pixels whose truth value is Class i that are predicted to be class j pixels.The number of pixels p_ij represented as true Yang (TP), p_ij The number (FP). of P_jj pixels for true shade (TN). p_ij is the number of pixels of false negative (FN), p_ij is the number of pixels of falsssse positive (FP).

MIoU is the ratio of the intersection of the real label and the predicted label to its union, and the IoU of each class is calculated and then averaged, that is

MPA is to calculate the proportion of pixels that are correctly classified in each category, and then average, that is

F1 is an index to measure the accuracy of the binary classification model, taking into account the accuracy and recall rate of the classification model. The maximum value is 1 and the minimum value is 0. The larger the value, the higher the accuracy of the model

The feasibility of the model in practical application is evaluated and verified by the speed of operation, the number of model parameters and the accuracy of operation.

3.4

Analysis of experimental results

Experimental analysis of segmentation performance of different backbone networks

The average crossover ratio function curve and accuracy function curve in the improved model training iteration process are shown in Figure 6, and the accuracy value and crossover ratio tend to be stable as the number of iterations increases.

Figure6.

Accuracy and average crossover ratio function curves of the model training process in this paper

The improved DeeplabV3+ network compares the MIou, MPA, F1 values, operation time and accuracy of different models respectively, and the test results in the tile data set are shown in Table 2.

Table2.

Compares with other split networks

As can be seen from Table 2, the effect of the improved DeeplabV3+ algorithm has been significantly improved. Compared with U-Net network, PSPNet network and DeeplabV3+ before improvement, MIou is increased by 4.44%, 12.36% and 5.07% respectively, MPA is increased by 4.7%, 3.55% and 4.44% respectively, and the operation time is increased by 139ms compared with DeeplabV3+ before improvement. Accuracy was improved by 5.71%, 9.53% and 1.02%, respectively. It also shows that the improved algorithm has significantly improved the segmentation accuracy and detection speed. In addition, the improved algorithm reduces the operation time and ensures efficient deployment and application under tight pipeline. Ablation results.

A comparison ablation experiment between DeeplabV3+ model and DeeplabV3 + model was designed to verify the effectiveness of the improvement. The basic model before the improvement was denoted as A. Different modification methods for the model were shown in Table 3.The same data set and validation set are used for model training and testing. In order to ensure the reliability of the experiment, each network is trained twice, and the results are compared with the average value. The experimental results are shown in Table 4.

Table 3.

Different processing methods

Table 4.

Influences of different methods on the model

As can be seen from Table 4, the improved DeeplabV3+ has been greatly improved in terms of performance. Each module proposed not only improves the model reasoning speed, but also effectively improves the segmentation accuracy. When lightweight backbone network MobileNetV2 is used as the backbone network, the model depth is reduced and more underlying features are retained. The model not only improves the segmentation accuracy, but also reduces the operation time. After ECA attention mechanism module is added to the model, the model’s attention ability and model accuracy are improved, but the introduction of attention mechanism increases the model complexity and processing time. After adding the cascade fusion of multi-branch receptive fields in ASPP module, MIou and MPA increased by 0.75 and 0.31 percentage points respectively due to the fusion of multi-level receptive fields. After replacing the convolutional layer of the decoder with a depth separable convolution, the internal structure of the decoder is optimized, and the overall performance of the model is improved again, MIou and MPA are improved by 0.27 and 0.18 percentage points, respectively. At the same time, the model processing time is shortened with the reduction of the number of model parameters. The improved model has stronger ability to extract the image of the tile surface, and better performance in real-time, which can better complete the task of tile image segmentation in industrial environment.