{'fraction_non_alphanumeric': 0.01349527665317139, 'fraction_numerical': 0.0, 'mean_word_length': 5.566371681415929, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Levels of Evaluation

The formative-summative paradigm is useful in that it draws attention to the importance of process and outcome measures in evaluation. The paradigm is not helpful in practical evaluation planning, however, because it unnecessarily separates qualitative and quantitative methods, and because it does not differentiate among various types of formative evaluation. The article presents a strategy for organizing the evaluation effort into three levels. Each level combines aspects of formative and summative evaluations and includes a variety of methods for addressing the evaluation questions. Resources and skills needed by evaluators at each level are recommended, and a plan for implementing the strategy is suggested.