Spectral Roll-off Points Variations: Exploring Useful Information in
  Feature Maps by Its Variations
  Useful information (UI) is an elusive concept in neural networks. A
quantitative measurement of UI is absent, despite the variations of UI can be
recognized by prior knowledge. The communication bandwidth of feature maps
decreases after downscaling operations, but UI flows smoothly after training
due to lower Nyquist frequency. Inspired by the low-Nyqusit-frequency nature of
UI, we propose the use of spectral roll-off points (SROPs) to estimate UI on
variations. The computation of an SROP is extended from a 1-D signal to a 2-D
image by the required rotation invariance in image classification tasks. SROP
statistics across feature maps are implemented as layer-wise useful information
estimates. We design sanity checks to explore SROP variations when UI
variations are produced by variations in model input, model architecture and
training stages. The variations of SROP is synchronizes with UI variations in
various randomized and sufficiently trained model structures. Therefore, SROP
variations is an accurate and convenient sign of UI variations, which promotes
the explainability of data representations with respect to frequency-domain
knowledge.
