{'fraction_non_alphanumeric': 0.052904564315352696, 'fraction_numerical': 0.005186721991701245, 'mean_word_length': 5.069182389937107, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Efficient Approximation with Neural Networks: A Comparison of Gate Functions

We compare different gate functions in terms of the approximation power of their circuits. Evaluation criteria are circuit size s, circuit depth d and the approximation error e(s, d). Informally, gate functions γ1 and γ2 are called equivalent if {γ1}-circuits of size s and depth d can be approximated by {γ2}-circuits (and vice versa) of size poly(s), depth O(d) with approximation error e(s, d) = 2−s. Our goal is to determine those gate functions that are equivalent to splines relative to this error model. The class of equivalent gate functions contains, among others, the exponential function, the natural logarithm, (non-polynomial) rational functions and (non-polynomial) roots. Newman’s result, i.e., approximating | x | by rational functions, is obtained as a corollary of this equivalence result. Provably not equivalent are polynomials, the sine-function and linear splines.