{'fraction_non_alphanumeric': 0.02391904323827047, 'fraction_numerical': 0.012879484820607176, 'mean_word_length': 5.112359550561798, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Towards Automatic Generation of Evaluation Plans for Context-based MT Evaluation

This report extends the FEMTI guidelines for context-based MT evaluation with new functionalities aimed at evaluators and experts. The proposed interface to FEMTI generates an outline evaluation plan depending on the characteristics of the context in which an MT system will be used, entered by the evaluators. We first summarize the principle of context-based MT evaluation and the initial FEMTI proposal. Then, we introduce a vector-based representation of the context and of the quality characteristics, which underlies the process of evaluation design. We then show how this process is simplified by the proposed interfaces to FEMTI, and how expertise can be input into the system by using more advanced interfaces. A unified account of expert vs. evaluator use of FEMTI is finally proposed. The proposals outlined here represent work-in-progress accomplished in the project: " Quality models and resources for the evaluation of machine translation " (SNF division II, grant number 103318, 2004-2006).