{'fraction_non_alphanumeric': 0.021965317919075144, 'fraction_numerical': 0.010404624277456647, 'mean_word_length': 5.185714285714286, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Parameter Optimization of Spark in Heterogeneous Environment Based on Hyperband

There are many default parameters in Spark, which have a great influence on job performance. Different jobs require different parameter combos, resulting in the need to tune parameters with experience which causes low efficiency. To improve the performance of Spark in the heterogeneous environment, the Hyperband algorithm is used to optimize the Spark parameters to improve the efficiency of the Spark platform. To emulate the actual production environment and improve operation efficiency as possible, this study uses Spark on yarn mode to test. Experiments are carried out on WordCount and Sort job types. The results show that, taking 5GB dataset as an example, compared with the default parameters., this method reduces the job execution time by 12.94% and 10.41%, respectively.