{'fraction_non_alphanumeric': 0.019457956914523976, 'fraction_numerical': 0.0, 'mean_word_length': 5.180257510729613, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Towards scalable divide-and-conquer methods for computing concepts and implications

Formal concept analysis (FCA) studies the partially ordered structure induced by the Galois connection of a binary relation between two sets (usually called objects and attributes), which is known as the concept lattice or the Galois lattice. Lattices and FCA constitute an appropriate framework for data mining, in particular for association rule mining, as many studies have practically shown. However, the task of constructing the lattice, a key step in FCA, is known to be computationally expensive, due to the inherent complexity of the structure. As a possible remedy to the higher cost of manipulating lattices, recent work has laid the foundation of a divide-and-conquer approach to lattice construction whereby the key step is a merge of factor lattices drawn from data fragments. In this paper, we propose a novel approach for lattice assembly that brings in the implication rules and canonical bases. To that end, we devised a procedure that interweaves implication and concept constructions. The core of our method is the efficient discarding of invalid elements of the direct product of factor lattices and a set of heuristics has been designed for that. The method applies invariably to both complete lattices and iceberg lattices. In its most efficient realization, the approach largely outperforms the classical FCA algorithm NEXTCLOSURE.