Unsupervised Learning of Word-Sequence Representations from Scratch via
  Convolutional Tensor Decomposition
  Unsupervised text embeddings extraction is crucial for text understanding in
machine learning. Word2Vec and its variants have received substantial success
in mapping words with similar syntactic or semantic meaning to vectors close to
each other. However, extracting context-aware word-sequence embedding remains a
challenging task. Training over large corpus is difficult as labels are
difficult to get. More importantly, it is challenging for pre-trained models to
obtain word-sequence embeddings that are universally good for all downstream
tasks or for any new datasets. We propose a two-phased ConvDic+DeconvDec
framework to solve the problem by combining a word-sequence dictionary learning
model with a word-sequence embedding decode model. We propose a convolutional
tensor decomposition mechanism to learn good word-sequence phrase dictionary in
the learning phase. It is proved to be more accurate and much more efficient
than the popular alternating minimization method. In the decode phase, we
introduce a deconvolution framework that is immune to the problem of varying
sentence lengths. The word-sequence embeddings we extracted using
ConvDic+DeconvDec are universally good for a few downstream tasks we test on.
The framework requires neither pre-training nor prior/outside information.
