{'fraction_non_alphanumeric': 0.017501508750754374, 'fraction_numerical': 0.002715751357875679, 'mean_word_length': 4.204081632653061, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Towards Distributed Information Retrieval based on Economic Models

Introduction 
With the ever increasing amount of 
data on the Internet, there is an increasing 
need to search this information 
in new and more efficient ways. 
A part of the data on the Internet are 
not accessible to traditional search engines, 
as these data can only be accessed 
by a form for example. With 
distributed information retrieval systems 
however, these types of data can 
be accessed. In these systems there is 
a central broker with multiple servers, 
and the broker redirects queries to the 
servers. Each server fetches results 
from its own database and returns this 
to the broker. 
We are interested if this architecture 
can be built using an economic 
model, in which servers need to pay for 
the right to return results. We have 
seen from previous research that the 
use of an economic model might yield 
good results, as a successful spam filter 
based on an economic model has 
already been built. 
The aim of this research is to build 
a successful distributed information retrieval 
system based on an economic 
model, allowing servers to open up 
their part of the deep web. 
Methodology 
This research consists of three parts: 
1) selecting suitable economic models, 
2) simulating these models, and 3) performing 
a real-world test. 
We selected the economic models 
starting with a review of the current 
literature on economic models. 
With the obtained information we 
performed a multi-criteria analysis, a 
model checking phase, and a test on 
economic properties to select suitable 
models. 
The remaining models were simulated 
in custom-built simulation software, 
in which multiple variables were 
modified in different runs in order to 
observe their effects. 
The most suitable economic model 
was implemented in a real-world test, 
in which users valued the results of the 
system based on an economic model as 
well as a traditional search engine. 
Results 
We found the models of Vickrey auction 
and bond redistribution to be 
the most suitable ones. These models 
behaved well in our simulation 
and both outperformed a naive comparison 
model. The Vickrey auction 
model performed best in a scenario 
that mostly resembles the Internet. On 
average 69% of all models with a strong 
correlation between the economic outcomes 
and the performance of information 
retrieval (Kendall’s-τ > 0.6) is 
a Vickrey auction model. In the realworld 
test we show that users appreciate both the use and administration of 
an information retrieval system based 
on an economic model. Furthermore, if 
we apply a perfect categorization, the 
economic model outperforms the comparison 
engine with a 66% increase in 
performance. 
Discussion 
We conclude that it is possible to build 
a distributed information retrieval system 
based on an economic model. It 
performs better than a naive system 
and also in a real-world test it outperforms 
a traditional engine. However, 
non-human categorization of the 
queries negatively influenced the performance 
of the models, which shows 
the need for better categorization algorithm. 
Exposing the deep web with 
the use of an economic model is feasible 
and might even introduce new business 
models for servers and brokers by 
earning money with search results.