{'fraction_non_alphanumeric': 0.03460810860936978, 'fraction_numerical': 0.012519010349048555, 'mean_word_length': 4.789027270775177, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Spanish and English Phoneme Recognition by Training on Simulated Classroom Audio Recordings of Collaborative Learning Environments

Audio recordings of collaborative learning environments contain a constant presence of cross-talk and background noise. Dynamic speech recognition between Spanish and English is required in these environments. To eliminate the standard requirement of large-scale ground truth, the thesis develops a simulated dataset by transforming audio transcriptions into phonemes and using 3D speaker geometry and data augmentation to generate an acoustic simulation of Spanish and English speech. The thesis develops a low-complexity neural network for recognizing Spanish and English phonemes (available at github.com/muelitas/keywordRec). When trained on 41 English phonemes, 0.099 PER is achieved on Speech Commands. When trained on 36 Spanish phonemes and tested on real recordings of collaborative learning environments, a 0.7208 LER is achieved. Slightly better than Google's Speech-to-text 0.7272 LER, which used anywhere from 15 to 1,635 times more parameters and trained on 300 to 27,500 hours of real data as opposed to 13 hours of simulated audios.


Automatic Speech Recognition
ASR has been a topic of interest for decades [2] [3] [4]. Throughout time, researchers have had to implement different techniques to tackle the large list of areas in ASR.
Among the most popular techniques, we have Hidden Markov Models, Gaussian Mixture Models, Deep Neural Networks, Convolutional Neural Networks, Recurrent Neural Networks, Transformers, and AutoEncoders. Additionally, one can also find implementations where the aforementioned techniques are used jointly. For example, [16] implements a transformer along with some convolutional layers. The network's architectures of [15] and [37] are composed of multiple CNNs and RNNs. Graves et al. [17] use a deep recurrent neural network.
These techniques have achieved breakthroughs in different areas of ASR. In keyword spotting, where one tries to only detect a set of keywords, [8] and [13] have accomplished SOTA results on the Google Speech Commands v2 Dataset [26]. In End-to-End Speech Recognition (recognition of all labels or words), [9], [37] and [15] have shown great results on datasets like LibriSpeech [27], TIMIT [28] and WSJ [29].
Aside from this, multilingualism is yet another area of ASR. One involves multiple languages to be recognized, even though some of them might have limited resources (a small number of available recordings for example). On this end, Müller et al. [18] [19] have shown great results on the Euronews Corpus [24] by training a network capable of recognizing four different languages [19].
In addition to the aforementioned areas of ASR, there are two more problems worth mentioning, denoising and data augmentation. Removing background noise and increasing the size of limited-resources datasets are essential tasks in ASR.

Environments
Despite the vast amount of research performed in ASR, we claim that speech recognition in bilingual collaborative learning environments remains challenging. The AOLME's after-school program has offered multiple extracurricular activities to underrepresented middle school students in which STEM topics like python and image processing are taught. Figure 1.1 shows an example of a classroom setup in which these activities take place.
Each classroom includes six to eight tables and each table is composed of four to six students and one facilitator. The program's objective is not only to teach the content, but to study translanguaging as well; which is the successful incorporation of student's linguistic repertoire into the teaching [5].
Therefore, given that most students in these environments speak English and Spanish, large datasets must be gathered to train the network since dynamic recognition of both languages is required. Additionally, one's system must be clever enough to address cross-talking and background noise (loud noises, room echoes, etc.) and to tackle the unfavorable microphone-to-speakers ratio as it ranges from 1-to-4 to 1-to-6. The one on the left pertains to the TIMIT dataset, where the phrase "she had your dark suit in greasy wash water all year" was read from a book in a quiet environment.
On the other hand, the one from the right pertains to the phrase "the computer understands these numbers". It was extracted from a conversation that took place in an AOLME classroom.
These two audio plots show that the smoothness between signals varies greatly. On the left, we have a signal of a clear recording from TIMIT. On the right, we have a noisy recording taken from an AOLME conversation. The image on the left pertains to the phrase "she had your dark suit in greasy wash water all year", while the image on the right belongs to the phrase "the computer understands these numbers".
The one on the left is less crowded than the one on the right. Additionally, to further portray the differences between noisy and clear recordings, we show the resulting spectrograms of the aforementioned signals in Figure 1.3. Dark blue is associated with silence, while yellow is associated with noise. Notice that the dark blue color is not as abundant and dark on the spectrogram from the right as it is on the spectrogram from the left. Similarly, one can even see some vertical dark blue lines on the spectrogram from the left, representing pauses between sounds. Such pauses are rarely seen in AOLME's recordings. The image on the left pertains to the phrase "she had your dark suit in greasy wash water all year" from TIMIT's dataset, and the image on the right belongs to the phrase "the computer understands these numbers" from an AOLME conversation.

Thesis Statement
My thesis is that a phoneme-based method can be effectively used to recognize Spanish and English speech in collaborative learning environments, without the use of language models (LMs) and post-processing techniques. Furthermore, for collaborative learning environments, my thesis is that I can train a low-parameter method that can outperform SOTA methods that utilize larger datasets that do not reflect the complexities of this kind of environment.

Contributions
This thesis implements a low complexity model capable of training in Spanish and English, individually and jointly. The proposed method is based on DeepSpeech2's [15] architecture and trains on Convolutional and Recurrent Neural Networks. Additionally, it uses the CTC loss function to perform labels' predictions of unsegmented sequences.
The thesis uses a simulation framework developed by Antonio Gomez in his dissertation. The simulation framework uses text transcriptions to produce realistic datasets and allows data augmentation after using multiple random speaker geometries. Based on Pyroomacoustics [50], this framework lets us expand our datasets by a factor of four.
Furthermore, this thesis generated two different datasets. These were extracted directly from AOLME using ELAN's [45] and Praat's [44] software. One is Englishbased while the other is Spanish-based. The final number of recordings in each dataset is 2,036 and 516 respectively. Recordings ranged from a duration of 0.5 to 5 seconds and each was provided with a respective transcription.
Lastly, the resulting work of this thesis has been submitted and accepted in [68].

Outline
In the following chapters of this thesis, we discuss SOTA research performed on different areas of Speech Recognition (chapter 2). In chapter 3, we present the list of datasets used in this work. We include the ones that were available online and the ones produced in this thesis. Later on (chapter 4), we discuss our network architecture and methods used during training. Results are shown in chapter 5 and conclusions, along with future work, are discussed in chapter 6.

Chapter 2 Background
At first, this thesis's goal was to recognize English keywords in recordings from collaborative learning environments (AOLME). As research progressed, however, so did the angle. Instead of focusing on one language only, the goal became to recognize two languages (English and Spanish). Additionally, the aim switched from recognizing a set of keywords, to recognizing all phonemes present in the language(s).
In this section, we discuss previous research performed on KWS, E2E speech recognition, and multilingualism. We include descriptions of the datasets used by these systems. We also discuss the research behind E2E speech recognition commercial systems and previous work performed on AOLME by members of the Image and Video Processing and Communications Lab (ivPCL).

Online Datasets
Datasets that are available online and have been used by different SOTA systems are discussed here. We describe their contents and provide a table (Table 2.1) with brief descriptions and access URLs. For this thesis, we used Speech Commands and CSS10 to perform initial assessments of our neural network and used TIMIT for our baseline model.

EuroNews
The EuroNews Corpus data was gathered from different news of the Euronews portal.
Such corpus is split into ten since data was retrieved for ten different languages.

Keyword Spotting (KWS)
The objective in KWS, is to recognize a set of words (usually a short set) in an utterance from those that are not present in the set. Vygon and Mikhaylovskiy [8] have obtained good results in clean audio. Their approach is triplet loss-based.
They use three different inputs, the current training sample, a "positive" sample (that shares the same label as the training sample), and a "negative" sample (that pertains to a different label). The function's task is to minimize the distance between the training sample and the positive sample and maximize the distance between the training sample and the negative sample. This approach, combined with a k-nearest neighbor classifier (using only 109,000 parameters), achieved a 96.4% accuracy on the 35-word task of Speech Commands [26].
On the other hand, Berg et al. [13] targeted the same task using multi-head attention and multi-layer perceptron blocks in a transformer encoder. In multi-head attention, a module is run several times in parallel through an attention mechanism.
The resulting independent attention calculations are then combined to produce a final score. With this approach, they achieved a 97.69% accuracy.

End-to-end (E2E) Speech Recognition
Unlike KWS, E2E speech recognition is a task that aims to recognize all given characters, phonemes, or words of a given sequence. More specifically, from a set of acoustic features, the objective is to map such features to a label or set of labels.
DeepSpeech2 implementation is closest to our approach since its architecture is the backbone of the model proposed in this thesis. It is composed of multiple CNN layers connected to one or more RNN layers. While training, the network uses the CTC loss function to label unsegmented sequences. With this configuration, DeepSpeech2 achieved great results on the LibriSpeech and CHiME datasets. Their results obtained on both datasets are shown in Table 2.3.
DeepSpeech2 differs from this work's implementation, however, since they apply a convolution after the RNN layer(s) to filter the total of future context needed.
Additionally, this thesis focuses on recognizing IPA phonemes instead of ASCII characters. Furthermore, the augmentation technique used in this work is more complex than theirs. Instead of using a signal-to-noise ratio, we implement an augmentation framework using Pyroomacoustics [50].
Similar to DeepSpeech2, and our work, wav2vec 2.0 starts with multiple layers of CNNs. In contrast, however, they connect these CNNs to a transformer instead of an RNN. Additionally, instead of using CTC loss, they use contrastive loss, a function that considers a pair of samples and reduces loss by minimizing the distance between similar pairs and maximizing it between dissimilar pairs. Furthermore, wav2vec 2.0 trains with raw waveforms instead of spectrograms and implements LMs for fine-tuning. Their method achieved SOTA results in datasets like LibriSpeech and TIMIT. We provide their results in Table 2.3.
Lastly, Pytorch-Kaldi presents a compilation of results on different datasets. They train a vast variety of systems using one or multiple acoustic features. The system that resonates the most with ours is the one they trained with GRUs and tested on the CHiME corpus. The reason being is that they show how GRUs produce better results than vanilla RNNs and LSTMs when it comes to noisy environments.
Such results can be seen in Table 2.3. Notice that we also show their testing results obtained on other datasets like TIMIT and LibriSpeech.

Multilingualism
Yet another area of ASR that is related to this work is Multilingualism; the recognition of multiple languages even when some might have limited resources (a small number of available recordings for example). In this end, Müller et al. [18], [19] have shown great results on the Euronews Corpus [24] by training networks capable of recognizing two [18] and four different languages [19].
Similar to our work, the network architecture of [18] and [19] is based on Deep-Speech2 as their CNNs connect to RNNs and use the CTC loss function to predict sequences of labels. Additionally, their approach is also phoneme-based as they utilize MaryTTS to translate words into phoneme pronunciations. They also train with mel-based spectrograms, decode with an argmax operation and compute phoneme error rates.
Unlike us, however, they use BottleNeck Features as input to the network and append Language Feature Vectors during training. This so that they can compensate for language-dependent singularities. Additionally, the recordings used to assess their models cannot be considered as noisy as the ones present in AOLME. The reason being is that, even though background music and spontaneous change of language might be present in Euronews, the great quality of the audio and lack of crosstalk makes AOLME a more challenging dataset. Furthermore, while they do not use augmentation, this work uses a simulation and augmentation framework that produces AOLME-like recordings (with cross-talk and background noise).

Commercial Systems
In addition to SOTA open source systems, we explored multiple commercial systems to compare this work's performance on AOLME's recordings against theirs.
Using the Spanish version of AOLME-Sentences (section 3. In this section, we provide an insight into the network architectures behind each commercial system. We also discuss similarities and differences to this thesis, the number of network parameters, utilized resources, and training data.

Microsoft Azure: Speech-to-text
Although it is not explicitly written, we expect that the network architecture behind Microsoft Azure Speech-to-text is the one presented in Xiong et al. [31], where they discuss Microsoft's conversational speech recognition system in detail.
So-called CNN-BLSTM, this new architecture is closely related to the one we propose, since it also connects features extracted from a CNN to one or more layers of bi-directional RNNs. The differences are that this thesis uses GRUs and less than 200 thousand parameters instead of LSTMs and 38 million parameters.

Google Cloud: Speech-to-text
Two of the network architectures mostly used by Google are "RNN-Transducer" (RNN-T) [32], and "Listen, Attend and Spell" (LAS) [35]. Even though it is not written, we expect one of these two is the backbone of Google Cloud's Speech-totext service.
The first one, RNN-T, is mainly used for streaming ASR [33] [34] and is composed of three networks: an encoder, a prediction, and a joint network. Both, the encoder and prediction networks, are trained independently and are composed of multiple LSTM layers. Their outputs are then combined through the joint network (usually involving a forward-backward algorithm with a softmax function) that in return predicts a distribution of the next label.
On the other hand, the LAS architecture is composed of multiple pyramidal bidirectional LSTMs that connect to an attention-based LSTM transducer. SpecAugment [37] uses LAS architectures to experiment with techniques like time warping, frequency masking, and time masking, for data augmentation. In [36], the authors optimize the LAS architecture by incorporating multi-head attention and hypothesize that this adjustment results in better performance when dealing with noisy data.

IBM Watson: Speech-to-text
Multiple documents in [40] state that IBM's Speech-to-text service is based on bidi- Later on, Audhkhasi et al. [42] re-evaluated the aforementioned system and obtained better performance on the A2W model. They sorted their data in ascending order, added delta and delta-delta coefficients, incorporated Nesterov's momentumbased stochastic gradient descent, used projection layers on the outputs, and increased the size of their phone BLSTM system. However, even though better results were obtained, the out-of-vocabulary (OOV) issues remained; the system was not able to recognize words that were not in the vocabulary. To solve this, they trained a BLSTM network using characters and words jointly. Such a tactic ameliorated the performance of OOV words.

Amazon Web Services: Amazon Transcribe
Similar to the previous commercial systems, to our best knowledge, the network architecture behind Amazon Transcribe (Amazon's Speech-to-text service) is that of an RNN-T. A system composed of three different networks (as discussed in 2.3.2), where the first two are trained independently and come together through the third one to predict the next output label. Amazon has recently researched this architecture in [38] and [39].
On the first one, the minimum word error rate (MWER) method is appended during training to the RNN-T. By doing so, the network "smartly groups alignments of the same hypothesis together" [38]. On the second one, subword regularization is studied using an RNN-T to analyze and improve the performance of words unseen by the network. [39]. Also in the area of detection, Darsey [56] studied hand movement detection using histograms of optical flow, patch-color classification, and space-time patches of video.

Prior Work on AOLME
Shi et al. [64], [65] and Tapia et al. [66] focused their research on head and face detection, respectively. The first one not only focused on head detection, but also the detection of the subject's attention. By looking at the subjects' face direction and using AM-FM models, both tasks were achieved. The latter used low complex-ity CNNs to study face detection and determine the advantages of using AM-FM representations instead of raw images.
Furthermore, the studies of Jacoby et al. [59], [60] and Eilar et al. [57], [58] examined the topic of human activity. Jacoby  human activity in these kind of environments but focused only on talking detection [69] and person detection [72].
Lastly, in terms of detection, Jatla et al. [61], [62], implemented a SOTA automated segmentation technique in image processing models that automate the detection of coronal holes while Tran et al. [70] studied facial recognition.
Other fields aside from detection have also been studied. For example, Ulloa et al. [55] proposed a 3D CNN system that assists cardiologists, as it is capable of predicting one-year all-cause mortality by analyzing echocardiographic videos. Carranza et al. [54] were able to maximize throughput with optimized 2D-FFT libraries and vectorbased memory I/O. Kent et al. [63] incremented speed on 2x2 and 3x3 max-pooling with the help of FPGAs.

Chapter 3 Datasets
Datasets that were developed in this research are discussed here, as well as details on their usage, contributions, and development. We also provide additional information on Speech Commands and the way their data is split into training, validation and testing sets.

Speech Commands Details
As mentioned in [26], this dataset is split into training, validation, and testing using a hashing function. Such a function allows them to append additional files to future versions of the dataset and maintain independence between sets.
In [26], they provide two documents, validation_list.txt and testing_list.txt. The number of recordings listed in each file is 9,981 and 11,005, respectively. Therefore, about 10% of the recordings are kept for validation, another 10% for testing, and the remaining 80% for training.
Lastly, although it is not mentioned, the distribution of words between sets seems proportional. For example, for the word "right", the validation set contains 363 samples, the testing set 396 and the training set the remaining 3,019 samples.

Revision of CSS10 (Spanish)
Before utilizing the CSS10 dataset, multiple adjustments were performed to the text transcripts. For instance, any digits or uppercase letters present were translated to written numbers and lowercase letters, respectively. Roman numerals were translated as well; i.e., "Carlos VII" was translated to "Carlos séptimo", where séptimo stands for seventh in English. Any special character (' ?', ':', '.', ',', '-', '¡', among others) was removed from the transcripts. In the end, the resulting transcripts were composed only of characters in the Spanish alphabet (including letters with accents) and the white space character.
Additionally, for this work, only a subset of the full dataset was used. If a recording included one of the words "Cero", "Uno", "Dos", "Tres", "Cinco", "Número" or "Números", the recording was kept; otherwise, it was discarded. The reason being is that initially, this work had a keyword spotting angle. Nonetheless, the resulting dataset of 577 recordings was not modified. Each recording was simulated and augmented, as described in section 3.4.2, to mimic real recordings of AOLME's classrooms.
After simulation and augmentation, the resulting dataset was composed of 2,308 recordings. Such were split into training and testing to assess the performance of the low-complexity model proposed in this document. Results of this assessment can be found in section 5.3.

AOLME Datasets
AOLME's after-school program has been hosted in six different terms. Each term is composed of approximately ten different groups. On average, groups have ten sessions each term and each session lasts around one and one and half hours. With these numbers, one can say that AOLME is composed of approximately 900 hours of recordings from Collaborative Learning Environments.
In this research, three different datasets were developed using the aforementioned corpus. As described below, the first one is composed not only of recordings extracted from AOLME but also of recordings from Speech Commands and TIMIT. The second is composed of spoken sentences uniquely extracted from AOLME. The third one was developed using AOLME's transcripts rather than its recordings.

AOLME-Numbers
Three different datasets were used to develop this subset. Speech Commands was used for training while TIMIT and AOLME were used for testing. Three different categories were considered: "One", "Zero" and "Others". Each category was composed of single-word recordings. If the spoken word was not "zero" or "one", it was added to the "others" category.
The training set was composed of 1,500 recordings. Five hundred were obtained from the "zero" folder of Speech Commands. Other 500 were obtained from the "one" folder and the remaining 500, for "others", were randomly chosen from the remaining 33 folders. The two testing datasets were obtained by extracting spoken words from AOLME and TIMIT. A Python script was utilized to perform such a task. The script would play a selected recording and ask the user for the start and end times of the word of interest (a sub-sample of the recording). It would then use these times to crop and play the sub-sample. Then, the user would have the option of saving or denying the sub-sample (if denied, the script would ask for the times once again).
This would happen recursively until a satisfying sub-sample was obtained. Table   3.1 shows the number of utterances (sub-samples) used from each dataset for each category. Table 3.1: Number of utterances used from each dataset for the categories "One", "Zero" and "Others".

AOLME-Sentences
AOLME-Sentences was developed by extracting spoken sentences from different AOLME recordings. Two versions were considered in this work, one for Spanish and one for English. Sessions from six different groups were used to maintain diversity in gender and the number of speakers; three groups for Spanish and four for English, where one overlapped between the two of them.
Initially, the same script described in 3.3.1 was used to extract these utterances.
However, its implementation was tedious and lengthy because it required its entries to have a three-digit precision. Thankfully, a tool was available to help alleviate this process, ELAN Software [45]. Its GUI is shown in    Example of Praat's GUI using a recording from AOLME. The phrase "the computer understands these numbers" pertains to this recording.
Praat simplifies the task of annotating recordings because one can focus on multiple features when picking time-frames. If the wave signal is noisy and does not give a clear idea of where a person has stopped talking, one can use the spectrogram, intensity, pitch, pulse, or formants to make a more precise guess.
In the end, a total of 516 utterances were extracted for Spanish and 2,036 for English. Recording's duration varied from 0.5 seconds to 5 seconds. Table 3.2 shows the number of recordings extracted from each group for Spanish and English. It also provides the utterances' duration of each group in minutes. Notice that a letter has been given to each group, so that anonymity is kept; and that group 'B' is the one that overlaps between both languages. dataset is yet to be used, as it is mentioned in chapter 6.

AOLME-Transcripts
In addition to recordings of each session, AOLME also possesses transcripts for some of these recordings. Examples of these transcripts are shown in the first column of Another Python script was used to perform such a task. The results of this script are shown in column two of the table. Notice that in row one, the speaker's pseudonym and action were removed from the original transcript. Similarly, in row two, the pseudonyms of the speaker and the "inaudible" comment was removed.
Lastly, in row three, the pseudonyms of the speaker and the translation to English was removed.
Once this additional information was removed, a second correction took place.
Similar to CSS10 (section 3.2), original transcripts contained uppercase letters, digitbased numbers and special characters. Therefore, the same rules were applied to remove special characters, and replace uppercase letters and digit-based numbers.
Column three shows the resulting transcripts after applying the aforementioned rules.
In the given examples, notice that uppercase letters were replaced, and that question marks and dots were fully removed from the sentences. Afterwards, these sentences were run through a speech synthesizer, thereby generating recordings with AOLME's classroom context. Furthermore, these recordings were simulated, so that they would mimic AOLME's real recordings, including crosstalk and background noise. Lastly, data augmentation was performed in order to increase the size of the AOLME-Transcripts dataset. Details on how this dataset was synthesized, simulated and augmented are provided in sections 3.4.1 and 3.4.2.
Although both versions, Spanish and English, were synthesized, only the Spanish version was simulated and augmented. As listed in chapter 6, the English version will be simulated and augmented in the future. The resulting number of recordings for the Spanish AOLME-Transcripts dataset was 37,264. Such recordings were used to train and test the low complexity network proposed in this work. Validation and testing results are provided in section 5.4.
It is important to mention that the transcripts used to populate AOLME Transcripts were chosen carefully. They do not overlap with transcripts that pertained to the sessions' recordings of AOLME-Sentences. This way, independence between datasets is guaranteed.

Data Generation
This section explains the procedures taken to synthesize, simulate or augment certain datasets used in this thesis.

Speech Synthesis
To generate audio from text, a text-to-speech service was utilized. The rest of the English voices will be used in the future.
To synthesize an audio, one has to manually type the text in the website, select the language, choose the voice, and click download. Performing such task manually would have been tedious and time-consuming. Furthermore, if one wanted access to the API in hope of automatizing this task, one would have had to spend $99.00 for a one-year subscription. Therefore, to save time and money, a different method was used.
Such a method was performed with the aid of Selenium WebDriver [47], a framework that allows users to automatically perform quality assurance assessments in their web applications. For instance, it can automate a website's commands like clicking, selecting, typing, and inserting text. Such commands and functionalities were used in this research to automate the speech synthesis task mentioned above.
Sample commands of Selenium's Python API are shown in Figure 3 In other words, lines 10 and 11 mimic a user clicking inside the username field and inputting the username. Lines 12 and 13 repeat the aforementioned process, but for the password. Line 15 clicks the login button by using its "xpath" atrribute, while line 17 closes the browser. In summary, the script shows how one can open a browser, access a link, type the username and password, click the login button, and close the browser. A complete rundown of the commands used to automate this task can be found in Appendix A.
Given that the Spanish version of AOLME-Transcripts was composed of 2,857 sentences, after running them through TTSMP3, using each Spanish voice mentioned above, a total of 11,428 recordings were produced. These were checked thoroughly, since some recordings did not sound human-like. They either sounded "robotic" or a mispronunciation of a word was present. After this quality assurance step, 9,316 recordings were kept.
Although all 1,096 English sentences were also run through TTSMP3 (using three different voices), a thorough check has not yet been performed. This will be performed in the future as stated in chapter 6.

Data Augmentation and Simulation
The next step in creating a dataset that would resemble original recordings of AOLME was to use the resulting recordings from TTSMP3 and run them through a simulator. This work used Pyroomacoustics [50] (Pyroom) to achieve this. On Pyroom, each source (blue dots) reproduces a different utterance, so that cross-talking is mimicked. Similarly, each room noise (green dots) is provided with noisy recordings to resemble loud noises in the classroom. The objective is to carefully configure pitch and gain attributes of each source and each noise so that a certain source is captured by the microphone. In other words, once a source is "chosen" by the user, the microphone's task is to recognize this source's recording. Therefore, the recordings that were obtained from TTSMP3 are the ones that we provided to this source. All other sources were provided with random recordings. Therefore, not only did Pyroom permit simulations to closely resemble recordings from AOLME, but it also allowed each TTSMP3 recording to be augmented. However, to generate realistic simulations that were not too similar, only four different Pyroom geometries were configured. As a result, the Spanish dataset of AOLME Transcripts was expanded by four and resulted in a total of 37,264 recordings.

Network Architecture
The low complexity model proposed in this research was built on PyTorch. A complete overview of our network architecture is provided on Python in Appendix C.
Additionally, a sample code demonstrating how to run the network is provided in D.
A complete implementation of our code is provided in [52]. Our model is motivated by the architecture of DeepSpeech2, and it was built on top of Michael Nguyen's [51] implementation. We provide a diagram of the proposed architecture in Figure 4.1.
First, mel-spectrograms of each recording were calculated and used as input.
Before calculating the spectrograms, however, each utterance was zero-padded if it Most datasets' recordings were mono (one-channel) in the WAV format using a LINEAR16 encoding and a 16,000 Hz sample rate. If recordings were not configured in this format, they were carefully converted using ffmpeg. The command used for such a task is provided in Appendix B.
After spectrograms were calculated, they were passed through a 1-layer 2D CNN and a fully connected layer right after. For the CNN, stride value and kernel size never changed. They remained 2-by-2 and 3-by-3, respectively, in all runs. At the end, two fully connected layers were implemented with a GELU activation function and a dropout in between. The outputs of the network were then run through a logarithmic softmax function.
The Connectionist Temporal Classification (CTC) loss function was used to transform these outputs into a conditional probability distribution of labels. To obtain a resulting sequence of phonemes, an argmax decoder was utilized. Additionally, from the resulting sequences, blank labels and repeated labels were removed from such sequences. To illustrate this step, we provide Table 4.1. On the left, we show the words that are being predicted (ground truth). In the middle, we provide possible outputs of the aforementioned decoding method. Notice that some phonemes might be repeated and some blank labels (represented as underscores) are present in the prediction. Therefore, after removing them, predictions like the ones in the third column are obtained. Original words are provided in the first column. Predicted sequences of phonemes are provided in the second column, and corrected predictions are provided in the third column. Notice that the blank label is represented by an underscore.

Word to be Predicted Prediction in Phonemes
Prediction After Corrections audio O: d _ I oU oU oU O: d I oU network n E _ _ t w _ 3: k k n E t w 3: k architecture A:ô A:ô k I _ t E k tS _ Ä A:ô k I t E k tS Ä simulation s I m j U l eI eI eI S @ @ n _ s I m j U l eI S @ n work _ _ w 3: 3: k w 3: k Once the blank and repeated labels were removed, the Levenshtein distance operation was performed to obtain Phoneme and Word error rates. This operation determines how similar one string is to another by keeping track of the number of insertions, deletions, and substitutions. An insertion occurs when a label is added to a prediction. For example, if the word to predict is "shine" and the prediction is "shrine", the 'r' character is considered an insertion. On the other hand, a deletion occurs when a character is omitted in the prediction. The prediction "sin" from the word "sing" would be a deletion since the letter 'g' is omitted. Lastly, an addition happens when a label is added to the prediction. If the word "tree" is the word to predict and "three" is the prediction, the character 'h' is the addition. For further details, refer to [7].
It is important to mention that the GELU activation function was used through-

Phonemization
Given that the goal is to recognize two different languages simultaneously (Spanish and English), this work proposes the usage of an extended character set based on the International Phonetic Alphabet (IPA). By doing so, phonemes from both languages that sound the same can be identified with the same phoneme (label). have been bolded. For example, row one shows that the phoneme 'b' sounds the same for the Spanish word "babear" and the first 'b' of the English word "barbecue".
Similarly, row two shows how phonemes 'm' and 's' sound the same in the words "museo" and "mouse", for Spanish and English respectively.
These translations to IPA were performed using bootphon's Phonemizer library [48]. Appendix E shows a sample Python script of its usage. Briefly, the user provides the code with a language and backend to use. Then, the user determines which word or phrase (in the language of interest) will be phonemized. Then, one runs the regarding this backend and its phonemes' transcriptions can be found here [49].
As initially described in section 3.3.3, AOLME-Transcripts had to be split in two, Spanish and English. The reason being is that if a word in one language is phonemized using a different language (in the phonemize), the produced set of phonemes is erroneous. For example, take the Spanish word "carro". Phonemizing this word with the English language would output "k A:ô ô oU", which is nothing similar to its translation in Table 4.2 (which was performed with the Spanish language). Instead, it seems as if the Spanish word "carro" rhymes with the English word "sparrow", which translates to "s p ae ô oU", which clearly, should not be the case.
In other words, translating a Spanish word to phonemes, using the English language would do exactly that; it will phonemize the word with an English accent.
The same would happen if one tries to phonemize an English word using the Spanish language. This is the main reason both languages had to be separated from each other, and why sentences including both languages were removed from the dataset.
In this thesis, Phonemizer was applied to three datasets: Speech Commands, Lastly, since CSS10 is based on Spanish from Spain, a different language was used when phonemizing. Therefore, in addition to the resulting labels of AOLME-Transcripts, two more were added to the mix: 'L' and 'T', where 'L' comes from words like "b a t a L a" (batalla) and "r o ð i L a" (rodilla); and 'T' comes from words like "g R a T j a s" (gracias) and "f w E R T a" (fuerza). In the order in which they were written, the respective translations to English are: battle, knee, thank you, and strength.
We would like to add that typing phoneme characters in LaTeX was not an easy task. To help others in the future, we provide the packages and commands used to achieve so (refer to Appendix F). Although it might not be an optimal solution, we hope it brings ease of pain to others.

Learning Rate Scheduler
The learning rate scheduler used in this thesis consists of a "limited linear decay".
The idea was obtained from [1] and consists of diminishing the learning rate until τ epochs have passed. The equation used to update the learning rate after epoch k was the following: .
Here α = k/τ , e 0 represents the initial learning rate, and e τ the final learning rate. In all of our runs, we calculated e τ by multiplying e 0 by 0.01. The variable k represents the current epoch. Therefore, e k represents the learning rate at epoch k.
Lastly, the variable τ represents the number of epochs to pass before the learning rate remains constant (at a value of e τ ).

Baseline Keyword Classification Model
In 2014, DeepSpeech [14] claimed to perform better than other systems in noisy environments (including commercial systems). Therefore, we decided to perform a study and determine whether or not DeepSpeech was able to recognize AOLME's noisy recordings. The study consisted of comparing DeepSpeech's performance against a simple SVM-PCA Classifier. The details and results of such an experiment are discussed below.
The first step was to develop a concise dataset suitable for the study. Its development is discussed in section 3.3.1. In brief, it was composed of 1,500 utterances from Speech Commands, 30 from TIMIT, and 36 from AOLME. The next step was to install and configure DeepSpeech; for which we used the 0.5.1 release. Transfer learning was then performed on such using the 1,500 utterances from Speech Commands. After this, the resulting model was used to run inferences on AOLME and TIMIT.
In regards to the SVM-PCA Classifier, one can see its structure in Figure 4.3.
Notice that first thirteen Mel-Frequency Cepstrum Coefficients (MFFCs) are calculated from each utterance. Where only twelve are kept (from the second to the thirteenth). Then, Principal Component Analysis (PCA) is performed on these coefficients to diminish their dimensionality, but still keep most of their information. Next, the data is run through a OneVsRestClassifier using an SVC estimator (the SVM block in the diagram). To optimize the hyper-parameters of the system, a nested cross-validation procedure was utilized. During training, the system would output a score for each fold. During testing on the other hand, with the predict method, the system would output labels' predictions of each tested utterance. The overall results of this experiment are shown in section 5.1.

Chapter 5 Results
This section presents the results obtained by our low complexity network. We also discuss the parameters used for each model, including early stopping techniques.
The datasets used to train and test the network were Speech Commands, CSS10, AOLME-Transcripts, and AOLME-Sentences. At the end of this section, we provide a comparison against commercial systems on AOLME-Sentences.

Baseline with DeepSpeech and SVM-PCA Classifier
Results obtained from DeepSpeech and SVC-PCA are provided in Table 5.1. Accuracy for DeepSpeech was calculated using Acc = 100 * (1 − W ER), where WER ranged from 0 to 1 given the similarity of the prediction to the target (0 being the same, 1 being complete opposites). On the other hand, accuracy for the SVM-PCA system was calculated using Acc = CorrectP redictions N umberOf Samples . One can see in the table that, even with a stricter criterion, the SVM-PCA performed better than DeepSpeech when dealing with noisy recordings. In the case that this was not the outcome, we considered replacing PCA with Independent Component Analysis (ICA) since it is known to successfully remove statistical dependence between features [67]. Nonetheless, the results of Table 5.1 demonstrated the need to perform further research of speech recognition in collaborative learning environments. Table 5.1: Speech recognition accuracy obtained from DeepSpeech and SVM-PCA Classifier on TIMIT and AOLME.

Performance on Speech Commands
The first performance assessment of our low-complexity model was in the English language, using the Speech Commands v2 dataset. The test was performed on the 35word task. Therefore, the data was split in accordance to the files validation_list.txt and testing_list.txt, as described in section 3.1. In the end, the training, validation, and testing datasets were composed approximately of 24, 2.5, and 3 hours, respectively.
As discussed in section 4, the main structure of the model consisted of CNNs and GRUs. Using the python code shown in Appendix C, for this assessment, the specifications of the network were the following. Regarding the CNN, four filters were used with 3-by-3 kernels and a stride of 2-by-2. Regarding the GRU, two bidirectional layers were used with 64 input units and 64 hidden units. In the end, the network used 155,353 parameters.
We used 41 labels (phonemes), as discussed in section 4.2. We also used the learning rate scheduler described in section 4.3, where e 0 was set to 5e-4 and τ to 40. Additionally, SpecAugment's frequency and time masking techniques were implemented. In Figure 5.1 we show an example of a masked spectrogram. It pertains to the word "backward" of the Speech Commands dataset.
The mask in the frequency domain (rows) was given by the boundary 'FM'. In this particular example, the length of the frequency domain was 128 and FM was set to 14. Therefore, no more than 14 rows could have been masked in the spectrogram.
On the other hand, a variable 'TM' was used to assign a boundary to the time domain. A decimal value was assigned to this variable which was then multiplied by the length of the time domain. In this example, the length of the time domain was 81 and the value of TM was set to 0.0625. Therefore, the maximum number of columns to be masked was 5 (0.0625 x 81). In terms of performance, two different metrics were considered, PER and WER.
Early stopping was applied by checking the PER. If it did not decrease (improve) by a factor of 1/1000 in six consecutive epochs, training was stopped. From now on, we will refer to this early stopping technique as the common technique.  In Table 5.2, we compare our results with those of SOTA systems on the 35-word task of Speech Commands v2. In their works, [8] and [13] used each word as a label, so accuracy is determined by the correct predictions divided by the total number of targets. In our work, accuracy is determined by subtracting the Word Error Rate from 1 and multiplying the result by 100, to provide the percentage of correctly classified words. We also show the number of labels and the number of parameters used by each system.
Although our accuracy was not as high as theirs, it shows that our approach has potential and is worth further analyzing. In particular, we use a similar amount of parameters as res15 (and much less than kwt3) and more labels than both of them.

Training with CSS10
The second assessment performed on our low-complexity model was on the Spanish language. The model was trained on the 2,308 simulated recordings of CSS10 (approximately equivalent to 5 hrs.), discussed in section 3.2. The model was tested on the Spanish version of AOLME-Sentences and it targeted 40 different labels (including 'L' and 'T' as defined in section 4.2).
With an amount of 88,828 parameters, the best results were obtained using one 2D CNN (with 16 filters, 3x3 kernels, and a stride of 2x2) and three bidirectional GRU layers (with 32 input units and 32 hidden units). The initial learning rate (e 0 ) was set to 4e-4 and τ was set to 30. In terms of data augmentation, SpecAugment was not used, since recordings had been simulated and augmented with Pyroomacoustics.
Early stopping was applied by keeping track of the validation PER, with the common technique. If such did not decrease (improve) by a factor of 1/1000 in four consecutive epochs, training was stopped. Using the same GPU as before, the model ran into early stopping after 34 epochs (2 hours and 20 minutes).

Training and Testing with AOLME
The third and last assessment was performed using the Spanish recordings of AOLME-Transcripts (section 3.3.3 describes this dataset in detail). We discuss the exhaustive number of runs performed by the model, present results obtained on AOLME-Sentences, and compare performance against commercial systems.

Observations on a New Metric, Ratio PER
At first, we hypothesized that training jointly on CSS10 and AOLME Transcripts would produce better results. However, after multiple runs were performed and transfer learning combinations were implemented, it was clear that CSS10 was not improving performance. It was affecting it instead. We claim that this happened due to the Spaniard accent and the two additional phonemes. Therefore, hereafter, we continued to train the model purely on AOLME-Transcripts.
solely the validation PER was not providing the best generalization results. Therefore, we decided to implement a criterion that would save a second checkpoint. Such a criterion is described below.
The early stopping common technique that we used would save a checkpoint once the metric did not improve by a certain percentage after some epochs. Although functional, we noticed that in some runs the validation PER would continue to improve even though "convergence" was already achieved. We define convergence as the point at which the validation and training metrics meet. We provide an example of convergence in Figure 5.4 where the model was trained on the Spanish version of AOLME-Transcripts. It shows the validation and training PERs per epoch.  Therefore, from here on, we performed early stopping with Ratio PER.

Training and Testing
For the last set of runs, we used and compared two different augmentation techniques, Transcripts. The last column shows the results obtained on the Spanish recordings of AOLME-Sentences. One can see that Pyroomacoustics performed better than SpecAugment when validating and testing the model on AOLME-Transcripts. Additionally, Pyroomacoustics also showed better performance when testing on AOLME-Sentences (record-ings directly extracted from AOLME students' interactions). This is important to us because it shows that AOLME's room simulations decrease the gap between synthesized and real audios by 17%, in terms of Phoneme Error Rate.

Comparison Against Commercial Systems
We decided to compare our results with the speech-to-text service of different commercial systems. In this research, we consider Microsoft, Google, IBM, and Amazon which systems were discussed in section 2.3. Each of these systems was assessed on the Spanish version of AOLME-Sentences. The predicted transcripts were compared against the original counterpart and character error rates (CERs) were calculated (given that their systems outputted characters).
Since some predictions included uppercase letters and special characters, we replaced the first ones with lower cases letters and removed the special characters. We performed this step so that their predictions were aligned with the original transcripts and their performance was not affected. In Table 2.3, we show the results obtained from each system. We also include the results obtained from our Pyroomacoustics model (mentioned in section 5.4.2). The accuracy presented in the table was calculated by subtracting the CER from 1 and multiplying by 100 (PER was subtracted from 1 in our case).
Notice that Amazon's number of parameters is blank since we could not gather a concrete number. However, we expect their models to train on the millions since their network architectures like [38] and [39], used a total of seven LSTM layers with 1,024 hidden units each.
In comparison to Google's Speech-to-text service (whether it is based on an RNN-T or a LAS architecture), we claim that when targeting collaborative learning environments, our approach is competitive. Not only is a better accuracy obtained, but the low-complexity aspect of our network reduces the number of parameters, utterances and GPUs needed. For instance, our network uses less than 180k parameters while theirs use more than 2.5M. We use less than 13 hrs. for training while they use 300 or more. In terms of GPUs, systems like SpecAugment [37] were trained on 32 Google Cloud TPU chips for seven days, while we trained our model in a single 8GB GeForce GTX 1050 Mobile for three hrs. and 45 minutes.
In regards to IBM's speech-to-text service, we observe that a slightly higher accuracy was obtained in comparison to us. We claim, however, that we can improve our results by properly optimizing the network and simulation framework (as discussed in chapter 6).
Due to the large gap between IBM's and Google's results from Amazon's, we argue that a language model or post-processing technique must be taking place in their service and works well for small vocabularies like the one of AOLME-Sentences (composed only of 368 different words). Therefore, we claim that our model is not directly comparable to theirs since we have not implemented language models or post-processing techniques.
Similarly, we are certain that the system behind Microsoft implements LMs (like LSTM-LM and ngram-LM) and extensive post-processing techniques (like LSTM rescoring, ngram rescoring, and backchannel penalty) as it is described in [31]. Therefore, we cannot be compared to their system as well since we have not implemented any of these techniques.

Conclusions and Future Work
In this work, we have presented a phoneme-based low-complexity network capable of training in one or two languages. Using less than 180k parameters and 13 hrs. of data, it can compete against speech-to-text systems like Google's and IBM's. We have also described a simulation and augmentation framework. It is capable of mimicking a noisy classroom and expanding a dataset up to a factor of four. Additionally, we show that it can outperform SOTA data augmentation techniques (like SpecAugment) in collaborative learning environments. Furthermore, we have introduced an audio dataset obtained from live interactions of middle school students in classrooms. We also discuss the obstacles we faced when gathering this dataset, particularly the ones related to choosing the starting and ending boundaries. Although we present two different programs that facilitate this procedure, we reiterate that this task remains tedious and time-consuming.
For future work, we plan to finalize the English version of AOLME-Transcripts by thoroughly checking the produced recordings from TTSMP3. Then, after running the recordings through our simulation and augmentation framework, we will train our model on such and assess performance on the English version of AOLME-Sentences.

Chapter 6. Conclusions and Future Work
Consequently, this will allow us to train and test our model in both languages at the same time.
Additionally, we intend to optimize our model by implementing cross-validation and studying language models and post-processing techniques. We also plan to optimize our simulation framework by determining more accurate speaker geometries.
Lastly, we intend to use autoencoders for self-supervised learning. Given that most of AOLME's recordings are unlabeled, this approach will impact our work since there will not be a need for text transcriptions. Therefore, once the autoencoder learns from the unlabeled data, we can use the resulting weights to train on the limited amount of labeled data.