2.1

Original data collection

The pest images are collected automatically with the device from Jiaduo Company. The data collection device is distributed in the field environment of Anhui Province and consists of the pest light-trap device, camera, white background board, and pest cleaning device, as shown in Figure 1. Crop pests are attracted and stunned by the light-trap device. Then, pests fall to the white background board with grid lines. The pest camera collects images and the cleaning device cleans the white background board every 30 seconds. We invite agriculture experts to annotate data using Lableme software. After data cleaning and annotation, we constructed our light-trap pest dataset (LPD) includes 18585 JPEG images with 26 classes. The resolution of our LPD is 2592×1944 and the annotation files format is XML like PASCAL VOC. We divide the dataset into a trainset with 14868 images and a test set with 3717 images.

Figure 1.

light-trap pest collection device. (1) Pest light-trap device; (2) interior structure; (3) camera; (4) pest cleaning device; (5) white background board; (6) original light-trap dataset.

2.2

Joint balance-distribution oriented composition

The deep learning-based detector shows poor ability in recognizing pests with small-size and few-shot characters. We use the Faster R-CNN to explore the relation of Average Precision (AP) with object size and number in our LPD, as shown in Figure 2. Due to the fact that the AP has a positive correlation with the object size and number, we design a Joint Balance-Distribution Oriented Composition (JBDOC) with the synthetic dataset to solve the imbalance of category and size distribution.

Figure 2.

The performance of category-size distribution.

Since the light-trap device cleans the pests on the background board every 30 seconds, we can easily select a background board image without pests from the original data. We use the white background board and the bounding box in the original train set to expand the data. Specifically, we hope to expand the data as much as possible for the small-size and few-shot categories. Therefore, we use equation (1) to determine the object number of the synthetic dataset for each category:

where E_C represents the number of bounding boxes to be expanded for category C, ⎣·⎦ represents the rounding down operation, N_C represents the number of existing bounding boxes of class C, Max (N_DC) represents the category with the most number in the train set, mSize_D represents the average size of class C, mSize_D represents the mean object size of LPD, ∂ is a scale factor, and in this paper ∂ = 0.5.

Although the white background board of our dataset is conducive to reconstruct data, the extracted object bounding boxes exist black grid lines. We use the Grabcut algorithm¹³ to remove excess grid lines. To calculate the mask, the Grabcut algorithm needs to artificially present the area B with the foreground object and the whole image Bin. To solve the defects of the Grabcut, we designed an Automatic Grabcut Algorithm (AGA). Specifically, ⱯB (x, y, w, h)∈T, we have B_іп = B(x, y, 2w, 2h). Where B(x, y, w, h) indicates the bounding box coordinates that have been labeled, T represents the train set from the original dataset, B_in represents the input image of the Grabcut algorithm. For B_in, we choose only the bounding box without other objects in the range of ⱯB(x, y, w, h)-B(x, y, 2w, 2h) to avoid overlapped objects sneaking into AGA. We take B_in as the whole image and B as the object area parameter, then using the Grabcut algorithm to calculate the mask of the pest object.

In addition, overlap may occur when the pest bounding box images are inserted one by one into the blank background board images, so we design a 9-point method to avoid overlapping insertion, as shown in figure 3. For the 9 points in the pest image to be inserted (right side of Figure 3), if one of them falls into the existing bounding box, it will be judged to be overlapping and not inserted.

Figure 3.

The 9-point method for overlap judge.

Due to the irregular appearance of the pest, some nodes of the 9-point method are not on the pest pixel, as shown in the dotted box on the right of Figure 3. To solve the problem that the 9-point method is too strict in judgment, we propose a mask-9-point method to refine the object boundary points, which combines the pest mask and the 9-point method. Firstly, we default the bounding box center point to be inside the pest object. Secondly, we use the midpoint from the center point to the boundary point as the judgment point. Thirdly, if the judgment points fall in the pest mask, the boundary point is added to the 9 points, otherwise, the midpoint of the line is added to 9 points, as shown in Figure 4.

Figure 4.

Mask-9-point overlap judgment.

In addition, Figure 5 shows the overall structure of the JBDOC and we list the specific steps below:

Figure 5.

The overall framework of JBDOC.

Step 1: we select 1309 background board images without boxes as expanded data.

Step 2: we construct the expanded bounding box set from the training set using equation (1) and our AGA to get the pest mask.

Step 3: flipping the images in the expanded bounding box set horizontally and vertically to get more generality.

Step 4: the pest image in Step 3 is randomly selected and inserted into the image in Step 1 using the Poisson Fusion. Determine whether there is an overlapping bounding box. If there is, discard the insertion and re-execute Step 4. Finish Step 4 until all the pests bounding box images are inserted into the expanded dataset.

3.1

Experiment settings

We combine our method with Faster R-CNN, Cascade R-CNN, FCOS, and Retinanet respectively to show performance improvements on LPD datasets. Firstly, we use JBDOC to construct a synthetic dataset. Secondly, we use the synthetic dataset to preliminary train the detector. Thirdly, we use the original trainset to train the network and use the test set to obtain the result. All the processes of training use backpropagation and Stochastic Gradient Descent (SGD) to optimize network parameters. For Faster R-CNN and Cascade R-CNN, the RPN is requested to generate 1000 candidate boxes. We use the same training parameter for preliminary training with 12 epochs and the Momentum is set to 0.9. The learning rate follows the principle of linear scaling, in which the first 8 epochs are 0.0025 and the last 4 epochs are 0.00025. The device of servers using an NVIDIA Titan RTX GPU and Mmdetection2.0.0 framework in the environment of Python 3.8.

3.2

Experiment results

The experiment results show in Table 1. Where the JBDOC represents the method of Joint Balance-Distribution Oriented Composition, AP50 represents the AP (average accuracy) with the IoU threshold of 0.5, mRecall and mAP represents the mean AP and mean Recall under the IoU threshold of 0.5, 0.75, 0.95. To be fair, we use ResNet50 as the backbone network for all methods. Our JBDOC achieves state-of-the-art performance with Cascade R-CNN. In addition, with the help of our JBDOC, the performance of all methods is improved.

Table 1.

The performance of our JBDOC.

Further, the class-by-class performance compares as shown in Figure 6. The pest number and pest average size are displayed using normalized values (the absolute values divided by the sum values). The AP₅₀ is improved obviously by our JBDOC at few-shot small-size categories (24, 25, 26, 28, and 45). This shows that our JBDOC effectively solves the imbalance of class number and size distribution.

Figure 6.

The performance improvement of JBDOC compared with Faster R-CNN.

3.3

Qualitative results

To visually observe the accuracy, we visualized the detection results of Faster R-CNN, Retinanet, Faster R- CNN+JBDOC (ours), Retinanet+JBDOC (ours), as shown in Figure 7. Among them, the first row shows the performance of the minimum-sized category, the third row shows the test result with the interference of other categories. Our JBDOC contributes to detecting the small-size pest as shown in the first row of Figure 7. With our JBDOC, the detection bounding boxes become more precise as shown in the second line of Figure 7. Even if the images are mixed with other categories and noise, our JBDOC still serves the purpose of networks to discover pests, as shown in the third row in Figure 7.

Figure 7.

Visualization results.