4 When the in-domain data is very limited, in-vocabulary hit rate with respect to the target domain provides a useful proxy for stylistic and topical similarity. In prior work [6], we found that a two-stage process of vocabulary hit rate followed by perplexity filtering was useful. Here, efforts to use the estimated n-gram probabilities to rank the Twitter sentences were not successful. There are two possible reasons for this. First, the in-domain LM training data is small and perplexity estimates on individual sentences have high variance. Second, for the high vocabulary growth rate languages explored here, many of the words are not in-vocabulary, so n-gram context is often broken and unigrams dominate the score. We put a threshold on the in-vocabulary (IV) hit rate to decide which sentences we will keep and choose the threshold that minimizes perplexity on a held-out set. Obviously, there is a trade-off between the amount of data used to build the language model and the degree to which the data matches the target domain. The optimal threshold depends upon the amount of data we have to begin with. As the size of the unfiltered data increases, the optimal filtering threshold becomes more aggressive. This is most easily seen in Figure 1 when the unfiltered data size ranges from 14 million to 27 million tokens. Even though the 27 million token data is almost twice as big as the 14 million token one, the more aggressive filtering threshold on the bigger source causes them both to have about the same number of tokens after filtering. In our experiments the optimal vocabulary hit rate threshold tends to fall between 50% and 70% . Typically, only a small percentage of sentences have IV hit rates above the chosen threshold. After filtering, the Twitter LM is built from the remaining sentences concatenated with the in-domain data. The LM is a trigram model with modified Kneser-Ney smoothing and is built using the SRILM toolkit [16]. This is interpolated with another LM built only from the target domain data. Here, the vocabulary is restricted to the words found in the in-domain data. Concatenating the Twitter data with the in-domain data ensures that the full vocabulary from the in-domain data is represented in the language model training data. Thus, the mixture model combines two components: in-domain only and the in-domain plus Twitter combination. The size of the in- domain data, in our case, is typically less than 1% of the size of the Twitter data. Except for rare words that only appear in the in-domain data, the n-gram counts of the in-domain plus Twitter combination are dominated by the Twitter data. As noted earlier, prior work has shown that class-based models can be useful for combining data from multiple do- mains, though much of the work used part-of-speech (POS) classes. There is no POS tagger readily available for the languages we are working with. Instead, we need to learn the word class assignments from the data. Specifically, we use the unsupervised hierarchical clustering approach from Brown et al. [14]. The clustering algorithm attempts to find the assignment of words to clusters that maximizes the mutual B. Language Modeling Approaches Fig. 1. Optimal IV Threshold for Filtering Bengali Twitter Data