Analyzing Big Data with Dynamic Quantum Clustering
  How does one search for a needle in a multi-dimensional haystack without
knowing what a needle is and without knowing if there is one in the haystack?
This kind of problem requires a paradigm shift - away from hypothesis driven
searches of the data - towards a methodology that lets the data speak for
itself. Dynamic Quantum Clustering (DQC) is such a methodology. DQC is a
powerful visual method that works with big, high-dimensional data. It exploits
variations of the density of the data (in feature space) and unearths subsets
of the data that exhibit correlations among all the measured variables. The
outcome of a DQC analysis is a movie that shows how and why sets of data-points
are eventually classified as members of simple clusters or as members of - what
we call - extended structures. This allows DQC to be successfully used in a
non-conventional exploratory mode where one searches data for unexpected
information without the need to model the data. We show how this works for big,
complex, real-world datasets that come from five distinct fields: i.e., x-ray
nano-chemistry, condensed matter, biology, seismology and finance. These
studies show how DQC excels at uncovering unexpected, small - but meaningful -
subsets of the data that contain important information. We also establish an
important new result: namely, that big, complex datasets often contain
interesting structures that will be missed by many conventional clustering
techniques. Experience shows that these structures appear frequently enough
that it is crucial to know they can exist, and that when they do, they encode
important hidden information. In short, we not only demonstrate that DQC can be
flexibly applied to datasets that present significantly different challenges,
we also show how a simple analysis can be used to look for the needle in the
haystack, determine what it is, and find what this means.
