{'fraction_non_alphanumeric': 0.017996870109546165, 'fraction_numerical': 0.001564945226917058, 'mean_word_length': 5.696335078534031, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Benchmarking Infrastructure for Big Data

This paper discusses the applicability of 20 years of experience benchmarking transactional systems (TPC) and storage system benchmarking (SPC) to the design of systems suitable for use in comparing different analytic systems. One of the most challenging problems is finding tools to make meaningful comparisons between substantially different architectures working on solving similar problems. For example, architects have tackled problems of poor CPU utilization in large Hadoop clusters by using different network topologies, adding high-performance storage, or using widely varying system architectures. Examining past benchmark experience highlights strategic choices: should the basis of the benchmark be a defined problem to be solved (TPC) or a defined executable to run (SPC)? How can we have enough realism in the benchmark definition so as to prevent the results from collapsing into a single dimension (TPC-C)? Do benchmarks that mandate publishing pricing encourage more realistic hardware configurations? This white paper lays out the questions that any successful system benchmark must address and makes a set of proposals for the structure of benchmarks suitable for use in comparing various implementations of Big Data.