Clustering and Recognition of Spatiotemporal Features through
  Interpretable Embedding of Sequence to Sequence Recurrent Neural Networks
  Encoder-decoder recurrent neural network models (RNN Seq2Seq) have achieved
great success in ubiquitous areas of computation and applications. It was shown
to be successful in modeling data with both temporal and spatial dependencies
for translation or prediction tasks. In this study, we propose an embedding
approach to visualize and interpret the representation of data by these models.
Furthermore, we show that the embedding is an effective method for unsupervised
learning and can be utilized to estimate the optimality of model training. In
particular, we demonstrate that embedding space projections of the decoder
states of RNN Seq2Seq model trained on sequences prediction are organized in
clusters capturing similarities and differences in the dynamics of these
sequences. Such performance corresponds to an unsupervised clustering of any
spatio-temporal features and can be employed for time-dependent problems such
as temporal segmentation, clustering of dynamic activity, self-supervised
classification, action recognition, failure prediction, etc. We test and
demonstrate the application of the embedding methodology to time-sequences of
3D human body poses. We show that the methodology provides a high-quality
unsupervised categorization of movements.
