{'fraction_non_alphanumeric': 0.024597116200169637, 'fraction_numerical': 0.0, 'mean_word_length': 5.483516483516484, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Epoch-Wise Double Descent Triggered by Learning a Single Sample

Recently, it has been empirically shown that if neural networks are trained on a single training image to reconstruct the input image, fully-connected neural networks (FCNs) learn to output the training image regardless of the test image (memorization), whereas fully-convolutional networks learn to output the given test images (generalization). We further investigated FCNs and empirically found that optimizers play an important role in the memorization process by splitting the model into the bias of the output layer (the bias component) and the rest (the weight component). Specifically, neural networks memorize the training image by the weight component at the early stage of training irrespective of the optimizer. Conversely, as the epoch increases, some optimizers force the weight component to forget the training image and the bias component to memorize the training image. By assuming that FCNs converge to the constant function during this shift, and measuring the generalization error, we observed epoch-wise double descent, which explains why early-stopping contributes to better generalizability.