{'fraction_non_alphanumeric': 0.013930348258706468, 'fraction_numerical': 0.007960199004975124, 'mean_word_length': 5.407643312101911, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Problems with Running a Successful Multimedia Retrieval Benchmark ∗

Content-based image retrieval (CBIR) and multimedia retrieval are at the point where they are ready to leave the pure research status and become integrated into commercial prototypes and products. This requires techniques not only to be interesting as theoretical approaches but also to be comparable with respect to performance obtained. Similar to the text retrieval domain many years ago, several evaluation events or benchmarks have started these past years to compare multimedia retrieval techniques with a varying focus. TRECVID focuses on video, INEX Multimedia on structured data, ImageEval on visual retrieval and classification, and ImageCLEF on multimodal and multilingual data access. Running such a benchmark poses many problems and difficulties. This article summarises some of the major problems encountered in the organisation of ImageCLEF from 2003 to 2007 and tries to find solutions for at least part of the problems.