{'fraction_non_alphanumeric': 0.01570048309178744, 'fraction_numerical': 0.0, 'mean_word_length': 5.549407114624506, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Communication Model for Parallel Iterative Stream Processing

Stream processing systems are used for a plethora of low-latency applications that deal with high volumes and varieties of data. In general, streaming applications are formulated as a fixed number of inter-connected operators, where the operator graph indicates the sequence of computations that apply to the streaming data in motion. Typically an iterative operation in a streaming application is done by embedding the iteration in a single operator (which excludes multiple operator instances and hence not scalable) or by using a sequence of operators, one for each iteration (which requires knowledge of the number of iterations a priori). However, the number of iterations of some applications such as k-means clustering cannot be determined at the creation of the application as it depends on the properties of the data received at runtime and the convergence criterion. Hence only a limited number of iterative computations can be executed using this approach. In this paper, we propose a communication model to support intra-operator communication so that iterative computations, including those with an arbitrary number of iterations, can be efficiently executed in streaming applications. We show that the proposed model can support different iterative algorithms that have complex communication patterns. Finally, through evaluating a number of parallel iterative algorithms using large-scale datasets, we demonstrate the scalability and performance of our proposed communication model and compare it to the existing approaches used for constructing iterative streaming applications.