3.1

Instrument dial detection

In recent years, deep learning has been widely used in the field of computer vision, and has achieved a lot of results in image segmentation, image classification, 3D reconstruction and object detection^[11-13]. ResNet^[14] proposed the idea of residual connections, which showed good results in feature extraction, and the network was widely used in image classification, segmentation, and object detection. Using ResNet-50 as the backbone network of the dashboard, the Centernet^[15] algorithm is used for the detection of the dashboard.

3.2

Instrument key point detection

The traditional processing method for instrument key point detection is not satisfactory. The traditional detection algorithm mainly consists of three stages. First, a sliding window is used to scan the entire image to generate a target proposal frame. This method requires a large amount of calculation, is inefficient, and performs many repeated calculations; Then extract the features of the proposed frame by manual design. Due to the diversity of instrument panel shapes and different lighting conditions, it is difficult to design a feature with strong robustness, and the quality of the extracted features directly affects the classification accuracy; Finally, use the classifier to complete the classification. Compared with the traditional detection algorithm, the Centernet algorithm has high detection efficiency for instrument key points, less computation, strong robustness, and good recognition effect. Therefore, this paper proposes to use the Centernet-based algorithm to detect the key points of the instrument, and select two pairs of scale numbers symmetrical about the center as the key points.

The predecessor of Centernet is Cornernet proposed by Law et al.^[16] in 2018. This algorithm introduces the idea of key point estimation into target detection. Cornernet is a single-stage target detection method that treats a detection target as a pair of key points, that is, the boundary. The upper left and lower right corners of the box. The prediction frame is obtained by detecting the two key points of the upper left corner and the lower right corner of the target frame. The anchor is not used in this method, which solves the limitation of the size of the object using the anchor. The recognition accuracy of Corternet surpasses that of Retinanet^[17], but the speed of this algorithm has no advantage due to the large backbone network and complex post-processing methods.

Centernet is mainly composed of two parts, the first part is the feature extraction network, using ResNet-50, the second part is heat map prediction, center point prediction, width and height prediction, so as to obtain the detected target frame. The size of the input image is 512×512×3, and the features of the image are extracted by ResNet-50, and then the high-resolution image output is obtained through three upsampling operations. After each deconvolution operation is completed, the height of the feature layer is high. The sum width will become twice the original size, and after the three deconvolutions are completed, a high-resolution feature map of 128×128×64 is obtained. Perform three convolutions on this feature map to obtain a heat map of C categories, a center point prediction map with a channel number of 2, and a width and height prediction map with a channel number of 2. The schematic diagram of the structure of this algorithm is shown in Figure 2.

Figure 2

Centernet network structure diagram

The loss function of the heat map is improved on the basis of Focal loss, where N represents the number of key points in the image, α and β are hyperparameters, α, β are set to 2 and 4, respectively, Y_xyc representing the true value under the sample Gound truth, representing The predicted value of the sample. The heatmap loss function is shown below.

The key points on the heat map are represented by a two-dimensional Gaussian kernel, and the value closer to the key point is closer to 1, and vice versa. When it Y_xyc is 1, the value of the easy-to-classify instrument sample is close to 1. At this time, the calculated value is small, and the calculated loss function value is small, which can correct the key points of the instrument; for the difficult-to-classify The instrument sample of, the predicted value is close to 0, the calculated value at this time is larger, and the corresponding training weight is increased. When it Y_xyc is not 1, in order to prevent the predicted value from approaching 1, it is used as a penalty item. However, (1 − Y_xyc)^β the closer the parameter is to the center point of the meter, the smaller the obtained value, which further reduces the corresponding penalty.

The target box size and center point paranoia use the L1 loss function. N is expressed as the number of key points of the s_i instrument, the real size of the instrument frame, and the predicted target instrument size. The target box size loss function is shown below.

The total loss function of the Centernet network is to sum the three partial losses corresponding to the heat map, the target box size and the center point offset. Due to the large loss of the target box size calculation, multiply it by a hyperparameter to reduce its loss value. λ_size The value is 0.1 and the λ_off value is 1. The loss function of the Centernet network is shown below.

3.3

Instrument image correction

When the instrument is installed for the first time, most of the instruments are asymmetrical about the center symmetry line due to the installation interface and inspection shooting, and the dial is in a rotating state. On this basis, the inspection robot is used for image acquisition. Since the camera and the instrument are not on the same horizontal line, the collected instrument image is inclined. Therefore, there are tilt and rotation issues for the acquired instrument image. This paper proposes to select two points of initial scale and maximum scale as key points to perform rotation correction using affine transformation. After the rotation correction is completed, two pairs of scale key point position information of the new instrument image are obtained by calculation, and the new key point information is used for perspective transformation. Complete tilt correction.

3.3.1

Instrument rotation correction

After obtaining the position information of a pair of key points about the initial scale and the maximum scale symmetrical to the central symmetry line through detection, connect the two points to calculate the angle between the line connecting the two points and the horizontal direction, and after obtaining the rotation angle, select The center point of the image is used as the rotation center, and the rotation correction of the instrument is performed. The instrument rotation angle is shown in Figure 3 below.

Figure 3

Instrument rotation angle

The principle of affine transformation is to linearly transform a picture from a two-dimensional plane to a new two-dimensional plane, and the relative positional relationship between its graphics does not change. The matrix corresponding to its transformation is shown below. θ is the required rotation angle, x and y is the abscissa and ordinate values corresponding to the key points obtained by the target detection.

Expand the formula to calculate the sum corresponding to the corresponding rotated picture, x₁ and y₁ the expansion formula is as follows.

3.3.2

Instrument tilt correction

After the affine transformation, the instrument picture is still tilted, so we use perspective transformation to correct the tilt of the instrument. Compared with affine transformation, perspective transformation can transform an elliptical dial into a circular dial. The principle of perspective transformation is to project a two-dimensional picture onto a three-dimensional plane, and then transform it into a new two-dimensional plane. During transformation, we need to obtain the position information of the four key points before the transformation and the position information of the four key points after the transformation in advance. The position information of the four key points before the transformation is obtained by rotation correction; the position information of the four key points after the transformation is obtained by collecting the standard photos of the instrument in advance, and obtaining the horizontal and vertical coordinates corresponding to the four key points of the standard photos, so as to obtain the position information of the four key points. Transformation matrix. The general expression of the perspective matrix is shown below.

x₁ and y₁ are the key point position information before perspective transformation, x₂, y₂, z are the coordinate information projected to the three-dimensional plane. A₃₃ value is set to 1.

x₃ and y₃ are the abscissa and ordinate coordinates of the key points of the corresponding picture after the perspective transformation. Substitute the four key points into the corresponding equations, and then the corresponding perspective matrix can be obtained. Among them, A₁₃ and A₂₃ these two elements will transform the picture in perspective.

The image of the instrument before and after tilt correction using perspective transformation is shown in Figure 4 below. Figure (a) is the instrument image before perspective transformation, Figure (b) is the instrument image after perspective transformation, and Figure (c) is the standard instrument image for template matching.

Figure 4

Instrument tilt correction

3.4

Instrument reading recognition

For the instrument image obtained after target detection, key point detection and image correction, image preprocessing, disc detection, pointer segmentation, pointer detection and angle calculation are performed to complete the meter reading. The preprocessing adopts mean filtering, retains the frame part, and averages the pixels within a certain range to erode the area with a small area. In order to improve the operation speed, the color image is converted into a grayscale image.

After the grayscale image is obtained, the redundant background information is removed by using the Hough transform circle detection and the position of the center of the meter circle is obtained. The principle is that all non-zero pixels in the image are considered to be a point on a potential circle. Using voting calculation, a cumulative two-dimensional coordinate plane is generated, and then the detected dial image is obtained according to the set threshold.

The extraction pointer is segmented using Otsu’s algorithm. Using the Otsu algorithm only once cannot separate the meter pointer very well, so it is necessary to use the Otsu algorithm for multiple times to separate the pointer. The Otsu algorithm is called the maximum inter-class variance method. The principle is to use the grayscale image to calculate the total number of pixels in the histogram, divide the image into two parts, the foreground and the background, and use statistical methods to select the threshold to make the foreground and the background. The background variance is the largest, and the image segmentation is completed. W_L Represents the proportion of low pixels in the image, W_H represents the proportion of high pixels in the image, E_L represents the average of low pixels, E_H represents the average of high pixels, and Erepresents the overall average of the picture. T represents the segmentation boundary between the low pixel point and the high pixel point of the segmentation, traverse its segmentation points from 0 to 255, calculate ρthe maximum value, and finally determine the required boundary T. Its calculation formula is as follows.

The detection of the gauge needle is done by straight line detection using the Hough transform. The method used in Hough transform line detection is to represent the line in the Cartesian coordinate system with polar coordinates. A point in the polar coordinate system represents a straight line in its rectangular coordinate system. Find a point in the straight line of the rectangular coordinate system. All straight lines passing through this point represent a curve in polar coordinates, where the intersection point in polar coordinates Beyond the set threshold is our desired straight line. After obtaining the pointer straight line, establish a Cartesian coordinate system with the center point of the instrument as the center of the circle, calculate the angle formed by the pointer and the initial scale α, the maximum scale and initial scale of the dial are 270°, and then according to the dial scale, the instrument representation can be obtained. Its relational expression is shown below. μ for its calculated meter reading, γ_max representing the maximum scale of the meter.

4.1

Automatic inspection of the experimental environment

The head of the inspection robot is equipped with a high-definition camera to collect instrument images. The pan/tilt viewing angle can achieve 180° left and right, 130° up and down, the maximum elevation angle is 40°, and the minimum top view angle is 90°. The bottom tire of the robot can drive the camera for mobile shooting. The inspection movement speed is 3km/h. By simulating the automatic inspection environment in the laboratory corridor, 10 meters of different heights, different light intensities and different distances are placed for inspection and collection, and the collected images of the meters are transmitted to the host for subsequent detection, correction and reading processing. The automatic inspection robot is shown in Figure 5.

Figure 5

Automatic inspection robot

The experimental training and test environments use NVIDIA GeForce RTX 2060 (8GB), Windows 10 operating system, 16GB memory, and the deep learning framework is pytorch1.2.0.

4.2

Instrument key point detection experiment

Since the instrument does not have a large data set publicly available, this experiment made the required data set. By simulating the images of the instrument in different environments, the inspection robot was used to capture the instrument images with different light intensities, different tilt angles and different rotation angles. A total of 8 types of instrument data were collected this time, and labelImg was used to label the key points of the collected 2778 photos to make xml files. The data set was divided into a 9:1 ratio, with 2221 photos in the training set, 247 photos in the validation set, and 275 photos in the test set. a photograph. During network training, the PASCAL-VOC2012 dataset is used for pre-training weights. The training settings Batch_size is 8, the learning rate is 0.001, the decay coefficient is 0.0005, and the optimizer is Adam. When the Epoch is less than 100, the backbone network is frozen and only fine-tuning is performed on the network. When the Epoch is greater than 100, the backbone network is thawed and the learning rate is reduced to 0.0001 to further reduce the loss function. When the training Epoch is 300, the network reaches convergence.

The instrument images with blur, angle deflection, and uneven illumination are collected to extract key points to test the robustness of the training model. The results are shown in Figure 6 below. The experimental results show that the network trained in this experiment has strong robustness and can achieve good key point extraction results in different situations.

Figure 6

Image key point extraction

4.3

Instrument image correction experiment

Once the key point detection is complete, a rotation correction is performed first, followed by a tilt correction. The more the inspection robot is on a level with the instrument, the better the corrected picture of the instrument using the affine and perspective transforms will be after the target detection is completed.

Experiments were carried out using equation (12) to calculate the meter reading and error rate before and after calibration, and the average error rate of the meter was calculated as shown below.

N is expressed as the number of images of the test meter, μ_real expressed as the real meter reading, μ_test expressed as the reading obtained through the experimental test, and expressed as the average error rate.

In this experiment, by simulating the inspection environment, the inspection robot is used to collect 10 sets of instrument photos with different illumination, different deflection angles and blur degrees for testing. readings and error rates. The images before and after calibration of the meter are shown in Figure 7 below, and the measurement results are shown in Table 1 below. Figure 7(a) is the image before rotation correction, Figure 7(b) is the instrument image after rotation using affine transformation, and Figure 7(c) is the instrument image completed using perspective transformation. Extract the key point location information of the template image, and obtain high-quality meter images for subsequent readings through two meter corrections.

Figure 7

Before and after instrument calibration images

Table 1

Test results of readings before and after meter calibration

The experimental results show a reduction in the average error rate of 2.98% after calibration, which is 10.19% lower than the average error rate of the uncalibrated images and an average accuracy of 97.02% for the meter readings.