{'fraction_non_alphanumeric': 0.014179608372721135, 'fraction_numerical': 0.0, 'mean_word_length': 5.8930232558139535, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
The GraphX Graph Processing System

The growing scale and importance of graph data has driven the development of specialized graph computation engines capable of inferring complex recursive properties of graph-structured data. However, these systems are unable to express many of the inherently data-parallel stages in a typical graph-analytics pipeline. As a consequence, existing graph analytics pipelines resort to multiple stages of data-parallel and graph-parallel systems composed through external storage systems. Instead, the GraphX abstraction unifies graph-parallel and data-parallel computation in a single system and is capable of succinctly expressing the entire graph analytics pipeline. However, implemented naively with relational operators it is impractically slow. We describe an implementation of the GraphX abstraction that encodes graphs as collections of edges and vertices along with simple auxiliary index structures, and represents graph computations as a sequence of relational joins and aggregations. We incorporate techniques such as incremental view maintenance and index scans in databases and adapt these techniques to exploit common characteristics of graph computation workloads. We evaluate these techniques individually as well as our overall GraphX implementation and find that it achieves performance comparable to contemporary graph-parallel systems in graph computation while retaining the expressiveness of contemporary dataparallel systems.