Auto-Tag: Tagging-Data-By-Example in Data Lakes
  As data lakes become increasingly popular in large enterprises today, there
is a growing need to tag or classify data assets (e.g., files and databases) in
data lakes with additional metadata (e.g., semantic column-types), as the
inferred metadata can enable a range of downstream applications like data
governance (e.g., GDPR compliance), and dataset search. Given the sheer size of
today's enterprise data lakes with petabytes of data and millions of data
assets, it is imperative that data assets can be ``auto-tagged'', using
lightweight inference algorithms and minimal user input. In this work, we
develop Auto-Tag, a corpus-driven approach that automates data-tagging of
\textit{custom} data types in enterprise data lakes. Using Auto-Tag, users only
need to provide \textit{one} example column to demonstrate the desired
data-type to tag. Leveraging an index structure built offline using a
lightweight scan of the data lake, which is analogous to pre-training in
machine learning, Auto-Tag can infer suitable data patterns to best
``describe'' the underlying ``domain'' of the given column at an interactive
speed, which can then be used to tag additional data of the same ``type'' in
data lakes. The Auto-Tag approach can adapt to custom data-types, and is shown
to be both accurate and efficient. Part of Auto-Tag ships as a
``custom-classification'' feature in a cloud-based data governance and catalog
solution \textit{Azure Purview}.
