{'fraction_non_alphanumeric': 0.01178101178101178, 'fraction_numerical': 0.002772002772002772, 'mean_word_length': 5.14468085106383, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Optimizing Organizations for Navigating Data Lakes

We consider the problem of creating a navigation structure that allows a user to most eﬀectively navigate a data lake. We deﬁne an organization as a graph that contains nodes representing sets of attributes within a data lake and edges indicating subset relationships among nodes. We present a new probabilistic model of how users interact with an organization and deﬁne the likelihood of a user ﬁnding a table using the organization. We propose the data lake organization problem as the problem of ﬁnding an organization that maximizes the expected probability of discovering tables by navigating an organization. We propose an approximate algorithm for the data lake organization problem. We show the eﬀectiveness of the algorithm on both real data lakes containing data from open data portals and on benchmarks that emulate the observed characteristics of real data lakes. Through a formal user study, we show that navigation can help users discover relevant tables that cannot be found by keyword search. In addition, in our study, 42% of users preferred the use of navigation and 58% preferred keyword search, suggesting these are complementary and both useful modalities for data discovery in data lakes. Our experiments show that data lake organizations take into ac-count the data lake distribution and outperform an existing hand-curated taxonomy and a common baseline organization.