{'fraction_non_alphanumeric': 0.048052650558961414, 'fraction_numerical': 0.011494771006130545, 'mean_word_length': 4.855106888361045, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Learning An Invariant Speech Representation

Recognition of speech, and in particular the ability to generalize and learn from small sets of labelled examples like humans do, depends on an appropriate representation of the acoustic input. We formulate the problem of finding robust speech features for supervised learning with small sample complexity as a problem of learning representations of the signal that are maximally invariant to intraclass transformations and deformations. We propose an extension of a theory for unsupervised learning of invariant visual representations to the auditory domain and empirically evaluate its validity for voiced speech sound classification. Our version of the theory requires the memory-based, unsupervised storage of acoustic templates -- such as specific phones or words -- together with all the transformations of each that normally occur. A quasi-invariant representation for a speech segment can be obtained by projecting it to each template orbit, i.e., the set of transformed signals, and computing the associated one-dimensional empirical probability distributions. The computations can be performed by modules of filtering and pooling, and extended to hierarchical architectures. In this paper, we apply a single-layer, multicomponent representation for phonemes and demonstrate improved accuracy and decreased sample complexity for vowel classification compared to standard spectral, cepstral and perceptual features.


Introduction
The recognition of speech sounds and units (phonemes, words) from the acoustic input, and the resilience of humans to the range and diversity of speech variations, such as rate, mode and style of speech, speaker state, identity, or pronunciation [1,2], might be related to the representation in the auditory cortex and ventral auditory pathway [3]. Given a representation that is invariant to changes in the signal that do not affect the perceived class, the task of recognition can be reduced to learning from a few examples [4]. Hierarchical, cortex-inspired models for selective and invariant visual representations, have shown to be predictive of human performance and competitive to computer vision systems [5,6]. Their main principle, of re-using modules of fitering and pooling in multilayered hierarchies, is also the essence behind deep convolutional learning networks, recently applied for speech and sound recognition [7,8,9,10]. The empirical successes motivate different theoretical formalisms on what constitutes an effective data representation and how can one be learned in brains and machines [11,12,4].
In the context of learning from data, an effective representation is related to the sample complexity of a recognition problem, i.e., the number of training examples required for a given generalization error [13]. In that respect, an appropriate representation should be invariant to intraclass transformations, such as affine time-frequency maps or pitch changes, discriminative for different classes of speech 'objects', and stable to signal deformations (nonlinear warping). Invariance is a key concept towards machines that learn from few labeled examples.
In this paper we propose a family of invariant representations for speech, based on a theory for unsupervised learning of invariances [14], where a segment is represented by the empirical distributions of the projections on sets of transformed template signals. The representation is quasi-invariant and discriminative for C classes, when using a finite number of templates K that depends on C, and reduces the sample complexity of a supervised classification stage [4]. Assuming access to templates and their transformations, through unsupervised observation, and storage in the form of "neuron" synapses or convolutional network weights, invariance can be learned in an unsupervised way. The implied computations (dot-products and cumulative nonlinearities) are biologically plausible, consistent with the pooling of complex over simple cells and may suggest a computational goal for modules of the auditory cortex.

Related Work
Features for speech recognition are conventionally based on frame-level spectral representations, passed through one or multiple layers transformations including: frequency warping (mel or bark), compression (log, power-low), predictive coding, cosine transforms (cepstrum), normalization (e.g., loudness) [15,16]. Paths in this processing chain account for the mel-frequency cepstral (MFC) and perceptual linear predictive (PLP) coefficients. Robustness has been sought through representations inspired by properties of the early and central auditory cortical processing [17]. Examples include perceptual critical bands [18], long-term temporal analysis through the modulation spectrum [19] or temporal envelopes of spectral bands [20], the auditory spectrogram [21] and spectrotemporal receptive fields [22] for multiresolution (frequency/rate/scale) analysis, localized time-frequency modulation tuning [23,24], and hierarchical organization [25]. The representational effect of the transformed feature maps is to ignore irrelevant and noisy components of the speech signal, impose invariance by smoothing out nonphonetic variations and integrating information across time/frequency selective regions.
Speech recognition witnessed performance gains by replacing feature extraction and acoustic modeling with multilayered, deep networks [26,10]. Convolutional networks [9,27], which interchange convolution and pooling layers for local time [7] or frequency [26] invariance, result in distributed representations that are tolerant to speech variability (speaker and noise) in a data driven way [28]. Their performance though depends on abundant labeled examples, for learning the filters and dependencies in a supervised way. A scattering transform representation [12] builds shift-invariance and warp-stability over long temporal windows through cascades of analytic wavelet convolutions and modulus operators [8]. Our proposed invariant representation is learned in a feed-forward, unsupervised mode, using stored filter-templates, does not explicitly model the signal and is feature-agnostic, i.e. it can be derived from any base speech representation.

Group-Invariant Speech Representations
Transformations, Invariance and Learning: Let Φ(s) be a representation of a one-dimensional, time-varying signal s ∈ X , with X ⊂ R d , or L 2 (R), the d-dimensional representation space indexed by time, frequency or (any) coding coefficient samples. For a signal processed or mapped from the raw sound waveform to some feature space Φ0(s), X can be, for example the space of spectral coefficients or spectro-temporal patches. The dimension d depends on the length of the represented speech unit (frame, phone, sub-word, word).
Signal instances s belong to some ground-truth class y ∈ Y = {1, . . . , C}, depending on the task (e.g., phonetic, lexical, speaker class). Given a (training) set of n pairs , sampled from an unknown distribution, the supervised learning problem, posed here as a multiclass classification task, is to estimate a function f : X → Y which predicts or assigns label y ∈ Y in a sample s ∈ X . A representation or map Φ(s) : X → R d , is said to be be discriminative (or unique with respect to individual class members), when Φ(s) = Φ(s ) ⇔ f (s) = f (s ), invariant to intraclass variations caused by class-preserving transformations s = gs when Φ(s) = Φ(s ) ⇔ f (s ) = f (s), and stable, i.e., Lipschitz continuous when d(Φ(s), Φ(s )) ≤ L s − s 2, for some metric d(·) and L ≤ 1. An invariant representation Φ(s) on X reduces the number n of labeled examples required for learning f with a given generalization error [4,13].
A large family of typical speech transformations and deformations can be described by s (t) = v(t) * s(φ(t)) + n(t), where * denotes convolution/filtering by v(t), n(t) is noise, and φ(t) a linear or nonlinear function that applies mapping t → φ(t) on signal s(t). These include affine maps (translation and scaling φa,τ (t) = at − τ , nonlinear time-warping, and more complex, domain-dependent changes like pronunciation, dialect, context, emotion etc. They can occur or mapped in time s(t), frequency S(ω) or joint spectro-temporal domains.

Modules For Invariant Representations
Transformations of a signal s can be described through a group G (e.g. the affine group in R), as the result of the action of an element g ∈ G on the signal gs(t) = s(g −1 t) [12]. A representation Φ(s) is said to be invariant to group G, if the action of any g ∈ G does not have an effect on it: The set of transformed signals gs, ∀g ∈ G, generated under the action of G on s, is the group orbit with an equivalence relation defined by G for signals that belong to the same orbit: s ∼ s ⇔ ∃g ∈ G: s = gs. The orbit is then an invariant representation of s, since Os = Ogs, ∀g ∈ G. An empirical measure of Os is the high-dimensional probability distribution Ps, ∀gs ∈ Os, which is invariant and unique. It can thus be used for comparing orbits, and by equivalence, the mapped signals: s ∼ s ⇔ Os ∼ O s ⇔ Ps ∼ P s . A metric on d-dimensional distributions Ps can be approximated, following Cramér-Wold Theorem and concentration of measures [4], through a finite number of one-dimensional distributions {P s,t k } K k=1 induced from the projections of gs on K template signals {t k } K k=1 , where ·, · is a dot-product defined in signal space X . Estimation of the distribution of projections on a single t k requires access to all signal transformations gs under G. For unitary groups, and normalized dot-products, i.e.
s, t k / s t k , the property allows for obtaining the one-dimensional distribution for t k through all projections of s on the transformations of t k under G. The main implication is that an invariant representation can be learned in a memory based, unsupervised way from stored, transformed versions of a large but finite number of templates. An estimate of the empirical distribution for a finite group G is obtained by the cumulative summation through a set of N nonlinear functions {ηn(·)} N n=1 : Depending on the form of η(·), the above estimate is the cumulative distribution function (i.e., smooth sigmoid functions for N bins) or any moment (mean, second-order energy, approx. max etc.). Note that, the set of all moments determines a distribution of bounded-support as in P s,t k . Equation (3) describes a generic pooling mechanism over dot-products with functions gt k , that is invariant to the actions of the group as a group average. The final signature is the collection of all estimates, for all By extension, partial invariance to any complex, but smooth transformation G, can be achieved for sufficiently localized dotproducts (partially-observable groups), through the local linearization of the transformation [4].

Obtaining Templates and Orbit Samples
The template signals t k can in principle be random and even unrelated to the input speech segment (for the case of affine transformations), e.g., coming from different phonetic categories. The number of templates required for discriminating C classes is K ≥ (2/c 2 ) log(C/δ), for an -approximation error of the true distributions by the K empirical estimates with confidence 1 − δ 2 . The theory requires stored templates and their transformations, which involves no supervision and is equivalent to setting the weights of the representation architecture. The transformations for a given template t k , or equivalently, the samples from the transformation orbit O k t , can be acquired, through unsupervised observation via the assumption of temporal adjacency: sounds close together in time correspond with high probability to the same source undergoing a transformation (i.e., speech changing in time, the phonetic variations of a speaker, or the examples provided to children by caregivers during development and language learning [29]). This is the biological equivalent of sampling the class-specific subspace of X and storing samples from the orbits O k t of template t k in an unsupervised way, i.e., the label of t k is not important.
For learning speech representations for machine-based recognition, the templates and transformations can also be stored (like in a pre-trained network), or obtained by transforming a set of randomly selected templates [14]. This belongs to a family of techniques for augmenting training datasets for machine learning [30,31]. Given unlabeled but structured data (i.e., belonging to different sound categories, speaker or dialect classes), clustering or spectral clustering can be used for forming partitions that correspond to smoothly varying signals, approximating samples from smooth transformation groups. Given a few examples (templates) t k , online learning, and bootstrapping estimates of the empirical distributions, can be used for progressively adding examples to the orbit sets {gjt k }. Observations that are not added can form new orbit sets. Templates can be chosen from a set different from the learning set, using categorical grouping, for example picking samples from different phones, speakers, dialects, etc. In this paper (Sec. 5.2) we explore both using phonetic category groups (templates are different vowels) and clustering.

Computing Segment Representations
Re-using pooling-filtering modules with templates of smaller support and adding layers on top of representation (5), can increase the range of invariance and selectivity to larger signal parts [4,11]. For this paper, we consider a single-layer representation at the level of segmented phones (50-200ms), that has multiple components (one per template). It assumes that templates and input are in the same space and have the same support. A fixed-length base representation, that handles varying lengths, is obtained by local averaging of features across frames (Sec. 5.1). An interpretation, in terms of our model, is that waveforms are mapped on a base (0-th or 1-st) feature layer through (adaptive) time and frequency pooling (filterbanks).
Consider an input set X = {xi} n i=1 ∈ X ⊂ R d , where xi = Φ0(si) the d-dimensional base representation of the segment. For each x ∈ X, t ∈ X , the dot-product x, t k is given by x T t, as X ⊂ R d . Assume access to a different set T ∈ X , denoted as the template set, to be used as a pool of data (observations) for obtaining templates t k and samples from their orbit sets, i.e., transformations t k j = {gjt k } for arbitrary and unknown {gj}. The template dataset T is partitioned into K non-overlapping subsets T k = {t k j } |T k | j=1 , T = K k=1 T k , using categorical grouping or distance-based/spectral clustering. Under such a meaningful partition, that preserves some structure in the data, we approximate Eq. (4) by µ k For each input instance xi, the representation is obtained in three steps: 1) Filtering: estimate the normalized dotproduct between xi and template t k j , for each template in T k and each of the K sets, 2) Pooling: apply nonlinear functions (histograms) on the projection values within each T k , 3) Signature: concatenate components from all subsets. If matrix X = (x1, . . . , xn) T holds the row-wise arrangement of samples, T k = (t k 1 , . . . , t k |T k | ) T ,X,T the zero-mean/unit-norm normalized matrices, row-and column-wise respectively, and I k the |T |×1 indicator vector for subset k, the computations can be written:XT T k (projection on the set of one template k),XT T (projection on all K template sets), η k (X[T T 1 , . . . ,T T K ]I k ) (pooling over the elements of a set k).

Vowel Classification
To demonstrate the potential of the proposed speech representation, we present comparisons with standard spectral and cepstral features for an acoustic classification task on TIMIT dataset. We focus on vowels, including dipthongs, the largest and most challenging phonetic subcategory on the set [33]. For the task of classifying pre-segmented phone instances in 20 vowel classes, we use supervised linear classifiers without post-mapping the vowel categories prior to scoring. We train multiclass Regularized Least Squares (RLS) with linear kernels [34], using 1/6 of the training set for validating the regularization parameter. The use of a kernel-based, nonlinear RLS is expected to further improve the classification accuracy [35,36].
We used the standard Train (462 speakers, 45572 vowel tokens) and Test (168 speakers, 16580 vowel tokens) partitions and evaluate performance through phone error rates on the Core set (24 speakers, 2341 vowel tokens). For templates and orbit sets, we use either the standard development subset (50 speakers, 4938 vowel tokens), which we denote as "Dev" or the remaining Test excluding Core (118 speakers, 14239 vowel tokens), which we denote as "Res". Note that, the distribution of tokens across the 20 vowel classes is similar on all sets.

Baseline Features and Base Representations
We used four base representations, extracted using HTK [37] at frame level from the segmented phones expanded by 30ms. An analysis window of 25ms at 16kHz, with 10ms shifts was used for frame-level encoding. Features were enhanced by first and second-order derivatives (∆−∆∆), to incorporate temporal context. Frames were aggregated across the phoneme duration by averaging on three segments (3-4-3 ratio) [36,38]

Template Sets and Signature Derivation
As a case study, we consider templates from the same phonetic categories (/vowels/) and samples of "transformed templates"  Table 1: Vowel classification error (ER) and balanced error rate (bER) using different base representations Φ0(s), the proposed Φ(s) = InvR(Φ0) and linear Regularized Least Squares classifiers (OVA: One-vs-All).
obtained from a template set not used for learning, in our case either the "Dev" or the "Res" TIMIT sets (the latter roughly three times larger). All instances in the set are assigned to a group, either based on categorical metadata (e.g., phone, word, phrase, dialect, speaker) or by clustering on the base feature space. Even though a form of labeling is used in the former, this serves only to simulate the unsupervised observation of the orbit samples (e.g., "/aa/ instances of one dialect region"). We used N -bin histograms for the empirical distributions {µ k n } N n=1 which outperformed in this task other moment sets. All reported results refer to N = 20 bins. The final KN -dimensional signature Φ(s) in (5) formed by concatenating K histograms, each derived from the projections to all elements of a single set. All projections, as in Sec. 4, are normalized dot-products and feature matrices are standardized (using the training set) before both the projections and the classification. Table 1 shows classification results for the 20-class task on Core Set, listing both average and balanced error rates. Each block corresponds to a different base representation Φ0(s) and the proposed invariant representation built on top of it, i.e., using dot-products on this space, is denoted by InR(·). Results with three potential partitions of the template set T on subsets are shown: phoneme category (phn), phoneme and dialect category (phn-dr) and kmeans. The number of groups depend on T for the first two (K = 20 and 135 resp.), while for the latter is a free parameter (here K = 120 and K = 200). The invariant representation, using the richer "Res" set, systematically improves accuracy with One-Vs-All (OVA) classifiers, with a decrease in error rates of 8.12% (MFC-∆ ) and 6.7% (PLP-∆ ). Using the smaller "Dev" set improves accuracy in the case of the decorrelated features by 4.1% for MFC-∆ and 3.25% for PLP-∆ . Thus, the number of orbit samples (or transformed templates) is related to more accurate group averages, which is aligned with the theoretic requirements access to the entire orbit. For forming the orbit sets, the best results are obtained by using the phn-dr category, that assumes a template for one phonetic category over one dialect, giving 135 template orbit sets for "Res". In this case, the set labels the 20 categories of the supervised task; the scheme provides the overall best accuracy (37.80% error for MFC-∆ ). This performance is comparable to previously reported segmental-based classification results [24], without using additional cues (vowel logduration [35,36,38,2]) and One-Vs-One (OVO) schemes. Results with the cosine-distance kmeans clustering of the template set are better than baselines but lacking compared to the phn-dr, that seemingly provides better orbit approximations. On sam- Figure 2: Classification error dependency on the size of training set, decreasing TIMIT Train Set from 45572 to 46 (20-class, TIMIT Core Test). The plots correspond to the best baseline with PLP-∆ , with and without the invariant representation, averaged over 500 random partitions at each fraction level. ple complexity, the proposed representation attains performance comparable to the 37.55% PLP-∆ performance on OVO (190 binary classifiers). Fig. 2 shows how the proposed representation is consistently better than standard PLP while decreasing the training set size down to three orders.

Conclusions
A novel representation for acoustic signals was proposed on the basis of a theory for learning representations, with formal assertions for invariance to simple and complex, but localized, transformations. The theory explains the properties of hierarchical architectures, specifies how to build representations of improved supervised learning capacity and conjectures a mode for their unsupervised learning in the auditory cortex. We empirically demonstrated that a single-layer, phone-level representation, inspired by these principles and extracted from base speech features, improves segment classification accuracy and decreases the number of training examples. We plan to evaluate multilayer, hierarchical extensions for larger-scale speech representations (e.g., words) on zero-resource matching [15] and the applicability for acoustic modeling in ASR. We are also working towards group-structure-aware unsupervised and online learning of the template orbit sets.