{'fraction_non_alphanumeric': 0.012790697674418604, 'fraction_numerical': 0.009302325581395349, 'mean_word_length': 5.572519083969466, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Reparameterized Construction in L2-Norm Weight Decay for Rectifier Neural Networks

In this paper, we investigate regularization method to construct better rectifier neural networks for the classification tasks from two aspects. First we propose a novel regularization method for effective training of rectifier neural networks by reparameterizing each connection weight in convolution layers into a product of two vectors and using the L2-norm penalty to constrain those two vectors rather than the connection weight. Second, we derive a robust initialization for our reparameterization method by taking the rectifier nonlinearities into account to help learn well regularized networks. This new method has been evaluated on the CIFAR-10 and CIFAR-100 datasets and demonstrated improved performance relative to the traditional L2-norm weight decay approach.