2 propose to use only Twitter for collecting out-of-domain data for ASR language models for conversational speech. We have several reasons to prefer searching Twitter for data instead of the whole Internet. First, Twitter has a semi- structured format that makes it easy to select the user generated content and ignore ads, boilerplate text and other unwanted text. Second, the writing style is somewhat conversational which is a better match for the conversational speech domain than other types of writing often found on the Internet. Third, since Twitter is used all over the world, we will likely be able to find good data for most of the languages that we care about. Lastly, Twitter provides a friendly API that makes it easy to access a lot of data quickly. When working under a time constraint, we were able to collect as much as 6 million sentences from low resource languages in under 24 hours. Of course, Twitter text is notoriously noisy in that tweets often contain abbreviated or misspelled forms of words, as well as URLs, hashtags, usernames, etc. In this work, we describe simple language-independent methods for handling these that require filtering out a large amount of data but lead to surprisingly useful text. In fact, we find that Twitter text is more useful for learning word classes than the original in- domain text. The word classes can be used to achieve large reductions in perplexity for low-resource language models. Further, we introduce a prioritizing scheme for downloading text from “useful” users that results in a significant improve- ment in vocabulary coverage over random selection. In the remainder of the paper, we begin by summarizing prior work on leveraging text harvested from the web in language modeling in Section II, to provide the context for our work. Next, in Section III, we describe our initial process for collecting data from Twitter and two methods for using the data in language modeling. Section IV contains details of experiments demonstrating the effectiveness of the collection and modeling methods. Finally, in Section V we describe a method for prioritizing the crawling queue and how it can be applied to improve the utility of additional collected data. This work brings together two strategies for improving language models for conversational speech in limited training data scenarios – web text collection and class language models – but changes key elements in the particular approach for both. As mentioned previously, there have been several efforts aimed at using text gathered from the web. Early work used the number of search results returned by the search engine as a proxy for the n-gram probability [2], but this method does not capture the domain characteristics of the target task. The penalty for domain-mismatch is quite severe. When working with conversational speech, a small set of conversation transcripts makes for better training data than 100 times as much newswire text [3]. Other studies find that even speech transcripts can be of little utility if there is a formality mismatch, e.g. using broadcast news as out-of-domain data for a conversational task. A series of papers used frequent in-domain n-grams as queries to Google for collecting text [4], [5], [6] and perplexity filtering for improving the match to the target domain. Good results were obtained on a variety of conversational speech recognition tasks by several sites. Variations on this approach changing the query generation and filtering stages have also been explored [7], [8], [9] or by using cross-entropy rather than perplexity as a data selection criterion [10], [11]. Recent work has shown reduced perplexity and word error rate for a task of transcribing university YouTube videos [12]. Unfortunately, this method was not effective for the low-resource languages we are working with, perhaps because of the specific choice of languages. Other recent alternatives have used targeted text collections, e.g. broadcast news transcripts [1] and RSS feeds [13]. II. RELATED WORK