3 Schlippe et al. [13] also collected Twitter text for aug- menting the language model training data and the vocabulary for a French news transcription task, which led to a small (1.5% relative) reduction in word error rate. The tweets are collected using topic words from the RSS feeds for a 5- day period before the show, and text normalization uses language-specific knowledge (dictionary spell checking and abbreviation expansion). Key differences in our use of Twitter text are the focus on general conversational speech rather than recent news topics, simple text normalization methods that are minimally language-dependent, and the introduction of a new priortization strategy for rapid collection. A standard approach to incorporating web text in language modeling is via n-gram mixture models, where component n- grams are trained on different data sources. Class language models [14] have been used to reduce the number of param- eters in the model for combining data from different domains [15], but also to provide more degrees of freedom in mixture model combinations of data from different sources by allowing history-dependent mixture weights using classes [6]. Because of the limited resources, the work here emphasizes reducing the number of parameters, but the novel contribution is using the out-of-domain data to learn the word classes, which turn out to be more powerful than those learned on the sparse in- domain data. A small set of in-domain data forms the basis from which the data collection is bootstrapped. We want to query Twitter using phrases from the in-domain data to find additional phrases from a similar distribution. The queries are composed of bigrams from the in-domain data taken in descending order of frequency. Not all bigrams from the in-domain data are suitable queries. For example, due to code switching the III. TWITTER TEXT FOR LANGUAGE MODELING A. Twitter Data Collection randomly chosen query may not actually be in the target language. If the chosen query happens to be in English but the target language is Bengali, then keeping this query could pose a significant problem. If the query matches an English bigram then an overwhelming amount of irrelevant results will be returned. This can be avoided by checking that the query results overlap in vocabulary with the in-domain data and discarding queries that fail to meet a minimum threshold. In our experiments, we required an in-vocabulary hit rate of at least 10%. Instead of using the query results directly, we take the set of users who matched our query and then collect the entire history (up to 2,000 tweets per user) of those users’ posts. By taking the users’ histories, instead of using the query results directly, we gather sentences that have no overlap with our query words and avoid over-representing the queries in the collected data. Like previous researchers [13], we find it necessary to do significant cleaning of the Twitter text before building any language models. In the cleaning process, tokens such as URLs and hashtags are removed but other out-of-vocabulary (OOV) tokens such as mentions (“@username”) are left in place to provide appropriate context breaks when building the language model. Capitalization is standardized when appropriate for that language. We perform duplicate sentence removal to deal with spam and retweets. We found a consistent but small gain from exempting very short tweets (those consisting of one or two words) from the de-duplication. Presumably, this is because many short phrases are naturally repeated many times in conversation but the same is not true for longer phrases. After the text is cleaned, much of the collected data is still not useful for training the language model due to topic and style mismatch. We would like to select, from all of the collected text, the sentences that are the best match to our target domain.