Multi-Modal Recognition of Worker Activity for Human-Centered
  Intelligent Manufacturing
  In a human-centered intelligent manufacturing system, sensing and
understanding of the worker's activity are the primary tasks. In this paper, we
propose a novel multi-modal approach for worker activity recognition by
leveraging information from different sensors and in different modalities.
Specifically, a smart armband and a visual camera are applied to capture
Inertial Measurement Unit (IMU) signals and videos, respectively. For the IMU
signals, we design two novel feature transform mechanisms, in both frequency
and spatial domains, to assemble the captured IMU signals as images, which
allow using convolutional neural networks to learn the most discriminative
features. Along with the above two modalities, we propose two other modalities
for the video data, at the video frame and video clip levels, respectively.
Each of the four modalities returns a probability distribution on activity
prediction. Then, these probability distributions are fused to output the
worker activity classification result. A worker activity dataset of 6
activities is established, which at present contains 6 common activities in
assembly tasks, i.e., grab a tool/part, hammer a nail, use a power-screwdriver,
rest arms, turn a screwdriver, and use a wrench. The developed multi-modal
approach is evaluated on this dataset and achieves recognition accuracies as
high as 97% and 100% in the leave-one-out and half-half experiments,
respectively.
