{'fraction_non_alphanumeric': 0.025358324145534728, 'fraction_numerical': 0.0011025358324145535, 'mean_word_length': 6.2063492063492065, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Speformer: An Efficient Hardware-Software Cooperative Solution for Sparse Spectral Transformer

Nowadays, many transformer’s model optimization schemes are based on clustering algorithms, for example, Reformer utilizes locally sensitive hashing, Routing Transformer utilizes k-means clustering, etc. All these clustering methods are based on densities. However, the clustering algorithms based on consistency perform poorly for non-Gaussian distributed data sets. Unfortunately, the connectivity-based clustering algorithms tend to have higher time complexity than the previous clustering algorithms. To address the above problems, we propose an efficient sparse transformer software and hardware cooperative solution, named Speformer, based on spectral clustering. We applied the model to the enwiki8 document generation task, and the experimental results proved that our solution outperforms other models.