A Scalable Framework for Multilevel Streaming Data Analytics using Deep
  Learning
  The rapid growth of data in velocity, volume, value, variety, and veracity
has enabled exciting new opportunities and presented big challenges for
businesses of all types. Recently, there has been considerable interest in
developing systems for processing continuous data streams with the increasing
need for real-time analytics for decision support in the business, healthcare,
manufacturing, and security. The analytics of streaming data usually relies on
the output of offline analytics on static or archived data. However, businesses
and organizations like our industry partner Gnowit, strive to provide their
customers with real time market information and continuously look for a unified
analytics framework that can integrate both streaming and offline analytics in
a seamless fashion to extract knowledge from large volumes of hybrid streaming
data. We present our study on designing a multilevel streaming text data
analytics framework by comparing leading edge scalable open-source,
distributed, and in-memory technologies. We demonstrate the functionality of
the framework for a use case of multilevel text analytics using deep learning
for language understanding and sentiment analysis including data indexing and
query processing. Our framework combines Spark streaming for real time text
processing, the Long Short Term Memory (LSTM) deep learning model for higher
level sentiment analysis, and other tools for SQL-based analytical processing
to provide a scalable solution for multilevel streaming text analytics.
