{'fraction_non_alphanumeric': 0.03446359088382435, 'fraction_numerical': 0.008337965536409116, 'mean_word_length': 5.185567010309279, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Cross-language text classification

Our goal in cross-language text classification (CLTC) is to use English training data to classify Czech documents (although the concepts presented here are applicable to any language pair). CLTC is an off-line problem, and the authors are unaware of any previous work in this area. CLTC is motivated by both the non-availability of Czech training data (the case, presently, in our dataset) and the possibility of leveraging different topic distributions in different languages to improve overall classification for information retrieval. Consider, for example, that English speakers tend to contribute more to some topics than their Czech counterparts (e.g., to discuss London more than Prague), so that, having only documents in English, we may expect to do poorly at identifying topics like Prague. Czech speakers, on the other hand, often talk about Prague, so that by leveraging Czech data, we might expect to improve on detecting the topic Prague in English speakers; and Prague in English speakers is exactly the sort of thesaurus label which information seekers are most interested in—because it is rare. Accordingly, while a lack of Czech training data presently necessitates CLTC, we would have no reason to warrant the method’s abandonment if such data were to suddenly become available. Our dataset is a collection of manually transcribed, spontaneous, conversational speech in English and Czech. English transcripts have human assigned labels from a hierarchical thesaurus of approximately 40,000 labels. Presently, labeled Czech data is not available for classifier training. The hierarchy may be divided into two principle branches, containing 1) concept labels (e.g., education) and 2) precoordinated place-date labels (e.g., Germany, 1914 – 1918).