{'fraction_non_alphanumeric': 0.02237926972909305, 'fraction_numerical': 0.0, 'mean_word_length': 5.159420289855072, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A DBpedia-based Benchmark for Ontology-mediated Query Answering

Ontology-mediated query answering (OMQA) is a framework for querying data with a background ontology. Detailed evaluation of OMQA systems remains a challenge due to limitations in existing benchmarks. In this paper, we propose a new benchmark for OMQA based on natural language questions over DBpedia. In particular, the data are sampled from DBpedia with adjustable volumes and can easily reach a scale that is difficult for existing OMQA systems to handle. Logical rules are automatically extracted from DBpedia using a rule learner, and the queries come from real-life natural language questions over DBpedia. We evaluated two state-of-the-art systems under various settings, to demonstrate the potential of our benchmark in benchmarking and analyzing the behavior of OMQA systems.