3.1.

Graph Construction

The 3D skeleton data for human action recognition can be acquired by motion capture devices such as the Kinect body sensing device or human pose estimation algorithms, such as OpenPose and BlazePose.^35,36 These data are a sequence of frames, and each frame has a set of joint coordinates. Based on the sequence of human joints in the form of given 2D or 3D coordinates, a spatiotemporal skeleton undirected graph G ($V,E$) with joints as graph nodes and human node structure and natural connectivity of time as graph edges is constructed, as shown in Fig. 3.

Fig. 3

Illustration of the spatiotemporal graph.

The spatiotemporal graph of the skeleton sequence is constructed in two steps. First, within each frame, a spatial skeleton graph is constructed according to the natural skeleton connection relationship of the human body. Second, the edges between frames represent the temporal relationships of the corresponding nodes. In the graph, the node matrix $\mathbf{V}=\{v_{ti}|t=\mathrm{1,2},\dots ,T,i=\mathrm{1,2},\dots ,N\}$ includes all the nodes on the skeleton sequence. Here, $T$ is the number of frames, $N$ is the number of joints, and $v_{ti}$ denotes the $i$’th joint in the $t$’th frame. The set $\mathbf{E}$ of edges consists of two subsets. One is the connection ${\mathbf{E}}_S=\{v_{ti}{v}_{tj}(i,j)\in \mathbf{H}\}$ of nodes within each frame, $\mathbf{H}$ denotes the set of human skeleton joints. The other represents the connection ${\mathbf{E}}_F=\{v_{ti}{v}_{(t+1)i}\}$ between different frames.

3.1.1.

Spatial graph convolution

After generating the graph defined above, we perform multilayer spatiotemporal convolution operation on the graph to extract high-level features. The GAP and softmax classifier are used and finally the action category prediction results are obtained.

The mapping strategy is shown in Fig. 4. The topology of the node-space graph within a single frame of the baseline model is represented by the unit matrix $\mathbf{I}$ and the adjacency matrix $\mathbf{A}$. Implementing graph convolution in the spatial dimension is not easy. Specifically, the feature map of the network is a $C\times T\times N$ tensor, where $N$ denotes the number of vertices, $T$ denotes the length of time, and $C$ denotes the number of channels. Assuming that the input features at the $n$’th layer are ${\mathbf{f}}^n$ and the output features are ${\mathbf{f}}^{n+1}$, the expression of the graph convolution is as follows:

Eq. (2)

$${\mathbf{f}}^{n+1}=\sum _k^{K_{\max }}{\tilde{\mathbf{A}}}_k{\mathbf{f}}^n{\mathbf{W}}_k•{\mathbf{M}}_k,$$

where ${\mathbf{W}}_k$ is the weight vector of the $1\times 1$ convolution operation, $K_{\max }$ denotes the number of subsets of the adjacency matrix, and ${\mathbf{M}}_k$ is a learnable matrix that represents the strength of node connectivity:

Eq. (3)

$${\tilde{\mathbf{A}}}_k={\mathbf{D}}_k^{-1/2}{\mathbf{A}}_k{\mathbf{D}}_k^{-1/2},$$

Eq. (4)

$$\mathbf{A}+\mathbf{I}=\sum _k{\mathbf{A}}_k,$$

where $\mathbf{A}$ denotes the normalized form of the adjacency matrix ${\mathbf{A}}_k$. The normalized diagonal matrices are that ${\mathbf{D}}_k^{ii}=\sum _j{\mathbf{A}}_k^{ij}+l$. Here, ${\mathbf{A}}_0=\mathbf{I}$, ${\mathbf{A}}_1+{\mathbf{A}}_2=\mathbf{A}$, and $l$ is set to 0.001 to avoid empty rows.

Fig. 4

Illustration of the mapping strategy. Based on the ST-GCN model, the human skeleton is divided into three subsets: the vertex subset (denoted by red nodes), the centripetal subset containing neighboring vertices closer to the center of gravity (denoted by green nodes), and the centrifugal subset containing neighboring vertices farther from the center of gravity (denoted by blue nodes). Here, the symbol × denotes the gravity center of the human body.¹⁴

3.2.

Multiscale Temporal Convolutional Network

In terms of the time dimension, it is easy to implement graph convolution operation similar to classical convolution, since the number of neighbors per vertex is fixed to two (corresponding joint in two consecutive frames). Limited by the ability of the convolutional layer in extracting temporal features by the perceptual field, the single-scale temporal convolutional network of the baseline model is difficult to extract the node information with long duration in human action data. In view of this, we use a multiscale temporal convolutional network for the multiscale feature extraction of temporal information, as shown in Fig. 5. The network increases the width of the overall network and enhances its adaptability to the scale on one hand; on the other hand, different branches in the network have different perceptual fields and the scale of time-domain information extraction is different. As training proceeds, the network continuously learns the temporal node features, and the action information expressed by temporal information at different scales can make the network focus more on those salient regions when extracting features.

Fig. 5

Structure of multiscale temporal convolutional network.

In Fig. 5, this network performs a channel downscaling on the input data $\mathbf{F}\in {\mathbf{R}}^{(C\times T\times N)}$ and gets $\mathbf{F}\in {\mathbf{R}}^{(C/d\times T\times N)}$. Then, the temporal features at different scales can be extracted from the downscaled data. After that, the network dimensionally splices the extracted features and then sums them with those extracted from the residual network. Finally, these obtained features are used as the output of this network ${\mathbf{F}}_d\in {\mathbf{R}}^{(C\times T\times N)}$.

The temporal convolutional network in the baseline model performs convolutional operations in time order, in other words, the convolutional operation at moment $t$ occurs only on the data of adjacent frames. The convolutional kernel size taking three, ${\mathbf{f}}_K$ is a $3\times 1$ convolutional kernel and the input sequence is that $\mathbf{F}=({\mathbf{x}}_1,{\mathbf{x}}_2,\dots ,{\mathbf{x}}_T)$. The temporal convolution operation at ${\mathbf{x}}_{\mathrm{T}}$ can be expressed as

To enable the temporal network to learn more time-domain features, the model should be able to utilize more frames of the action data. In this paper, the multiscale temporal feature information is extracted by a dilated convolutional method, and the residual connectivity is also utilized to facilitate training, expressed as follows:

The multiscale temporal convolutional network keeps the resolution of the output feature map constant by utilizing the dilation convolution. The dilation rate $\alpha$ of the dilation convolution takes the values $[\mathrm{1,2},\dots ,d]$. When $\alpha =1$, the perceptual field of the convolution kernel is three and the parameter number is three. When $\alpha =2$, the perceptual field of the convolution kernel is five, but the parameter number remains unchanged, as shown in Fig. 6.

Fig. 6

Convolution kernels with different dilation rates. (a) Convolution kernel 3*1 $\alpha =1$ and (b) convolution kernel 3*1 $\alpha =2$.

Compared with the baseline model, the multiscale temporal convolutional network proposed in this paper has the ability of perceiving multiple scale features in time-domain and achieves better recognition results with fewer parameters.

3.3.

Spatiotemporal Excitation Network

Complex human actions are not only related to space but also often time-dependent. In view of this, we need to enhance the joint spatiotemporal feature extraction ability to improve the action recognition effect. The baseline model cannot sufficiently extract the spatiotemporal feature information of the skeleton nodes when recognizing human actions, however. Given this, we extend the spatiotemporal excitation network based on video data to the human skeleton data and propose a spatiotemporal excitation network that can effectively model dynamic skeleton data. Since the method proposed in Ref. 3 is for action recognition of video data, the data are two-dimensional (2D) and contain the adjacency matrix without the interframe depth of the actions in the video. Unlike the method in Ref. 3 which uses 3D convolution to get the spatiotemporal information in action videos, the spatiotemporal feature map in this paper is compressed into a single channel after channel pooling. Therefore, the spatiotemporal feature map is transformed into the 2D data, and then 2D convolution is used to excite the important spatiotemporal information. The network focuses on the dynamic feature information of spatiotemporal domain between local nodes of the human body, which effectively enhances the role of key nodes in similar actions and facilitates the recognition of similar actions. The structure of the spatiotemporal excitation network is given in the STE network block in Fig. 7.

Fig. 7

Illustration of the STE block.

Specifically, given $\mathbf{F}\in {\mathbf{R}}^{(C\times T\times N)}$, the global spatiotemporal tensor ${\mathbf{F}}_1\in {\mathbf{R}}^{(1\times T\times N)}$ is first obtained by channel pooling of the input data $\mathbf{F}$. The 2D convolution is performed on the pooled data ${\mathbf{F}}_1$ with a $3\times 3$ convolution kernel ${\mathbf{K}}_1$, formulated as follows:

By convolving the obtained features ${\mathbf{F}}_r$ we obtain a spatiotemporal feature mask $\mathbf{M}$, which is equivalent to the spatiotemporal attentional feature map:

After dot product operation between $\mathbf{M}$ and initial input tensor $\mathbf{F}$, the result is added with $\mathbf{F}$ to get ${\mathbf{F}}_M$. Here, by dot product operation, the excitation of key spatiotemporal features from the input features $\mathbf{F}$ with the template $\mathbf{M}$ can be realized:

The spatiotemporal excitation network is computationally more efficient than the conventional 2D convolutional operation, for the order of the parameter matrix and the number of parameters are effectively reduced through channel pooling. Each channel of $\mathbf{F}$ can perceive important spatiotemporal information through the activation function $\delta ({\mathbf{F}}_r)$ without increasing the number of parameters. In the overall network structure, the spatiotemporal excitation network replaces the residual block in the baseline model, which not only increases the network depth but also makes up for the lack of spatiotemporal feature extraction ability of the baseline model. Moreover, it enhances the role of key nodes, improving the recognition effect on similar actions consequently.

4.1.

Datasets

NTU-RGB+D60 is a publicly available 3D human skeleton action dataset containing 60 action categories with a total of 56,880 video samples, performed by 40 volunteers in the age range of 10 to 35. The action videos of each category were captured by three cameras, and all of them were filmed at the same chosen height but at different horizontal angles, that is, $-45\mathrm{deg}$, 0 deg, and 45 deg, respectively. The dataset uses the Kinect depth sensor to detect a sequence of 3D skeletal points for each frame of the human body. Each subject has 25 joints in the skeleton sequences and no more than two persons in each video. Two partitioning methods, X-Sub and X-View, are provided in Ref. 18 for dataset partitioning. X-Sub divides the dataset into a training set (40,320 videos) and a validation set (16,560 videos) by the participants in the captured videos; X-View uses 37,920 videos captured by cameras 1 and 3 as the training set, while 18,960 videos from camera 2 as the validation set.

NTU-RGB+D120 is an extension of the NTU-RGB+D60 dataset. About 106 participants completed 120 action categories, yielding 113,945 action samples with 32 different camera devices and 155 viewpoints. Different from NTU-RGB+D60, NTU-RGB+D120 is divided by X-Sub and X-Set in Ref. 19. By X-Sub, the dataset is divided into a training set (63,026 videos) and a validation set (50,919 videos) according to the participants in the captured videos. By X-Set, the dataset is divided according to the video numbers, the videos with even numbers as the training set (54,468 videos) and the videos with odd numbers as the test set (59,477 videos).

Kinetics 400 is a large-scale and high-quality dataset of YouTube video websites. It contains over 300,000 video clips in 400 categories. The skeleton dataset is extracted by the OpenPose toolbox, and each skeleton map contains 18 human joints, representing each joint in 2D spatial coordinates and predicted confidence scores.²⁰

Pytorch is employed as a deep learning framework for all the tests in this paper.³⁷ The weight decay rate is set to 0.0005, the epoch is set to 70, the initial learning rate is set to 0.05, and batchsize is 32.

Inspired by the two-stream method in baseline model 2s-AGCN, we used the same two-stream fusion strategy. Two streams, namely joint and bone, are used. The joint stream uses the original skeleton coordinates as input, and the bone stream uses the differential of spatial coordinates as input. Each stream was separately input into the network to get the softmax scores. Finally, the maximum scores of the two streams were added to obtain the fused score.

4.2.

Ablation Tests

To verify the proposed method, we designed several ablation comparison tests, and 2s-AGCN was selected as the baseline model. Three groups of tests were done and performance comparisons were conducted among several widely applied models from different aspects.

To verify the effectiveness of multiscale time-domain convolutional networks in capturing complex time-domain features, the tests of group 1 were designed and conducted to evaluate the impact of the key parameter $d$. The parameter $d$ can be used to characterize the strength of multiscale temporal feature extraction, representing the range of values of the expansion rate. In this paper, some tests were conducted to get the optimal value of the parameter $d$ under X-Sub division. Here, $d$ was taken in [1,2,3,4] sequentially. The test results are listed in Table 1. It can be seen from Table 1 that the best result is obtained at $d=2$ with the accuracy of 89.6%, followed by $d=3$, while the worst result is obtained when $d=1$. As can be seen, the appropriate dilation rate is very important, and the excessive dilation rate may lose the continuity of information and affect the recognition results. The number of parameters was also compared in this test. The number of parameters at $d=2$ is 1.6M, about 1.9M lower compared with that of the baseline model, which demonstrates its advantage in calculation amount.

Table 1

Comparison of accuracy and parameter quantities at different scales.

The effectiveness of the MTCN and STE network was evaluated on the NTU-RGB+D60 dataset, and the results are listed in Table 2. Compared with the baseline model, for X-Sub division and X-View division, MTCN achieved the improvement of 1.1% and 0.3%, and STE network achieved the improvement of 1.2% and 0.6%, respectively. This fully demonstrates the superiority of these two methods.

Table 2

Test results of MTCN network and STE network on NTU-RGB+D60 dataset.

4.3.

Visualization Analysis

As shown in Fig. 8, the left side is the adjacency matrix of the layer 8 learned from the baseline network, and the right side corresponds to the matrix learned from the network proposed in this paper. The grayscale of each element in the matrix indicates the strength of the connection. As can be seen, the values of the adjacency matrix learned by the baseline model tend to average out, making it difficult to distinguish the connection characteristics of nodes with similar actions. Whereas the method in this paper reduces the connection strength of nodes that are not related to the action (the red boxes in Fig 8), such as nodes 14, 15, 16, and 18 that are not related to the reading action. In addition, it effectively enhances the connection strength of important related nodes in the action (the yellow boxes in Fig 8), such as nodes 21, 22, and 24, which are related to the reading action.

Fig. 8

Example of the learned adjacency matrix.

A reading action sample sequence is visualized in a feature map, as shown in Fig. 9. The vertical coordinates of the feature map indicate the node number and the horizontal coordinates indicate the frame sequence number. Figures 9(a) and 9(b) show the feature map in the layer 8 learned by the baseline network and the proposed network, respectively. And Fig. 10 shows the feature visualization with key points for a selected frame of the above feature map. It can be found by comparison that the proposed method can significantly enhance the connection information of key nodes in the action in the time-domain, which fully demonstrates the superiority of the method in this paper.

Fig. 9

Visualization of feature maps for a reading action sample. (a) Feature map of the baseline model and (b) feature map of our method.

Fig. 10

Visualization of output features. Each dot represents one joint, and its size represents the degree of the attention to the node. The green lines represent the physical connections of human body. (a) Output features of the baseline model and (b) output features of our method.

In Fig. 11, examples of skeleton-based action samples are presented on the NTU-RGB +D60. For each action, five frames are selected in the chronological order to represent the whole action. The one on the left is the action frame for reading, and the right is writing. It can be seen that spatial features of the action of single frames are very similar, but the temporal features of “writing” versus “reading” vary more significantly. In this way, our view that the extraction of temporal difference features will increase the recognition effect of similar actions is thus verified.

Fig. 11

Visualization of skeleton-based action samples. (a) Reading and (b) writing.

4.4.

Results and Comparative Analysis

To explore the effectiveness of STMGCNs for human action recognition, we performed corresponding tests on the public datasets NTU-RGB+D60 and NTU-RGB+D120, and the test results are listed in Tables 3 and 4, respectively. In terms of recognition accuracy, the performance effect of our method is better than that of most models.

Table 3

Comparison of test results on NTU-RGB+D60 dataset.

Table 4

Comparison of test results on NTU-RGB+D120 dataset.

Compared with the baseline model, the accuracy of the proposed method was improved by 1.7% and 0.4% under X-Sub and X-View division on the NTU-RGB+D60 dataset. The accuracy increment of the proposed method was 8.7% and 1.0% under X-Sub, and 7.2% and 0.5% under X-View, compared with that of the typical methods ST-GCN²³ and AGC-LSTM.³³ On the NTU-RGB+D120 dataset, the accuracy of the proposed method was improved by 3.2% and 2.1% under X-Sub and X-Set, respectively, compared with the baseline model. It is obvious that the proposed method outperforms most methods. Compared with the latest MS-G3D model, although the recognition accuracy of the model is slightly lower, the total number of parameters of the proposed model is only half that of the MS-G3D model.

In the Kinetics-Skeleton dataset, our method was compared with several of the most classical methods in Table 5. Following the evaluation method in 2s-AGCN, we trained the model in the training set and provided the accuracy of top-1 and top-5 on the validation set. We see that our method improves of 0.9% for top-1 and 1.4% for top-5 compared with 2s-AGCN, demonstrating the effectiveness of our method.

Table 5

Comparison of test results on Kinetics-Skeleton dataset.

On the NTU-RGB+D60 dataset, we compared the proposed method with the baseline model on each category adopting X-Sub, and the results are shown in Fig. 12. As can be seen, the proposed method outperforms the baseline in most cases and even improves the accuracy by more than 5% on categories 9, 10, 11, 12, 15, 29, 30, 32, and 52. It is owing to the better feature extraction ability of our proposed method for local subtle movement in human action. More useful information in the temporal feature extraction process can be retained owing to the usage of the multiscale temporal convolutional network which extracts features of time-domain information at different scales, and cascades and fuses the time-domain features at different scales.

Fig. 12

Accuracy comparison between the proposed method and 2s-AGCN based on X-Sub evaluation. The horizontal coordinate is the serial number of the action category, and the vertical coordinate is the recognition accuracy. The specific action names are referred to the official website of the public dataset NTU-RGB+D.

Compared with the latest models of STA-GCN and MS-G3D, the recognition accuracy of our method is slightly lower, but our method has shown superior results in the vast majority of similar actions, as shown in Fig. 13. For the similar actions, such as clapping, reading, writing, tearing up paper, putting on a shoe, and taking off a shoe, they differ by small changes in arm movements, thus creating challenges for skeleton-based action recognition. For these similar actions, the accuracy of our method has improved 7.2% compared with that of the baseline model, 3.5% compared with that of the MS-G3D, and 4% compared with that of the STA-GCN on the NTU-RGB+D60 dataset X-Sub. This demonstrates that our method significantly increases the recognition of similar actions by increasing the temporal feature extraction capability and the importance of important nodes.

Fig. 13

Comparison of the performance on similar classes.