A Big Data Lake for Multilevel Streaming Analytics
  Large organizations are seeking to create new architectures and scalable
platforms to effectively handle data management challenges due to the explosive
nature of data rarely seen in the past. These data management challenges are
largely posed by the availability of streaming data at high velocity from
various sources in multiple formats. The changes in data paradigm have led to
the emergence of new data analytics and management architecture. This paper
focuses on storing high volume, velocity and variety data in the raw formats in
a data storage architecture called a data lake. First, we present our study on
the limitations of traditional data warehouses in handling recent changes in
data paradigms. We discuss and compare different open source and commercial
platforms that can be used to develop a data lake. We then describe our
end-to-end data lake design and implementation approach using the Hadoop
Distributed File System (HDFS) on the Hadoop Data Platform (HDP). Finally, we
present a real-world data lake development use case for data stream ingestion,
staging, and multilevel streaming analytics which combines structured and
unstructured data. This study can serve as a guide for individuals or
organizations planning to implement a data lake solution for their use cases.
