{'fraction_non_alphanumeric': 0.029996842437638144, 'fraction_numerical': 0.005683612251341964, 'mean_word_length': 4.791590493601462, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
EDA IN THE AGE OF VERY LARGE DATABASES

of n=50 observations (x1,y1),...,(xn,yn). We may have wanted to understand the relationship between x and y, and our tools would have been a pencil, pad of graph paper, and a slide rule or adding machine. In this framework, Tukey developed his principles of EDA (exploratory data analysis). Starting from basic but novel principles, Tukey developed simple techniques for understanding data using the tools available at the time. Perhaps more important than his methods was his philosophy, which included the following ideas [1]: • Look at data to see what it tells us, not just to confirm or dispute what we think is true. • In other words, the concern is with appearance, not confirmation. • Simpler is better. • Looking below the surface improves our understanding. • The greatest value of a picture is when it forces us to notice what we never expected to see. These ideas represented a departure from the conventional thinking of the day. While Tukey’s specific methods were novel and inventive, EDA represented a procedural change in that it changed the way people thought about data. Today, data might consist of a file containing n=5,000 records (observations) like the data shown in Table 1. The observations are enterprise voice switches, where the variable Alarms is the number of alarms the switch generated, and the variable Tickets is the number of trouble tickets opened against the switch. We want to understand the relationship between log(Alarms+1) and log(Tickets+1). Our tool is a statistical package like R or S-Plus, running on a PC, workstation, or server. Over the last 40 years, data storage and retrieval technology has evolved considerably. However, data analysis methods and software have not fully reflected these changes. Statistical data analysis software has made tremendous gains over the last 40 years, both in computing power and techniques available. However, statistical packages still treat the underlying data in basically the same way as they did 40 years ago. It is ironic that in an age when there are many algorithms to detect obscure, hidden patterns and relationships in large, high dimensional data sets, analysts and users still have trouble obtaining a basic understanding from large databases. This paper explores some ways in which data analysis needs to change to take advantage of very large databases (VLDBs). Starting from basic but novel principles, Tukey developed simple techniques for understanding data using the tools available at the time. We need to revisit his approach, but take into account new technologies and ideas. In fact, Tukey’s principles above apply more today than ever before. We now have the ability to directly observe and measure many things that were previously unknown, and in greater detail than was previously imagined. Often, we need EDA type techniques just to be able to comprehend what the data has to offer. We use the term EDA instead of visualization, since the term visualization typically is used to describe methods for understanding detailed data. As we will see, methods based on detailed data tend not to scale well to handle VLDBs.