Attending industry conferences for information systems (e.g. medical, news,
intellectual property domains), it is easy to observe, in the last 2-3 years, a surge in
semantic search systems, using artificial intelligence to produce the best results for
a variety of work tasks that have an underlying search application. End -users of such
systems have no means of assessing the value of these systems, but have to trust
the companies offering them. At the same time, companies developing these search
based applications have no reliable tools to integrate effectiveness evaluation as part
of their testing procedures.
The challenge here lies in the fact that while there are numerous benchmarks
available in the academic community, there is no quantification of the differences
between them. Such a benchmark is typically constituted of a set of documents to be
indexed by the search engine (the document collection), a set of queries that
simulate user information needs (the query set), and a set of relevance judgements
(the qrel set). Changes in any of these, in order for a search system to maintain
optimal performance, need to be reflected in changes in the system parameters. But
while changes in effectiveness and changes in system parameters are typically easy
to observe or measure, changes in the benchmark are currently difficult, if not
impossible to measure.
Building on the state-of-the-art in representation learning, KoDicare investigates
methods to understand changes in the benchmarks beyond the simple term
statistics. Significant changes in the document collection or query set need to be
quantified at a semantic level. Using such a quantification, which we denote as the
Knowledge Delta, we will be able to run ablation studies tests where we change, in
a controlled environment, units of Knowledge and observe differences in
performance of the search system. The ability to do so has a significant impact both
on academic research (providing the means for more controlled experiments in
information retrieval) as well as on industry (providing the means to update the
search engine if and only if the environment has significantly changed).
KoDicare brings together the Research Studios Austria Forschungsgesellschaft, the
Laboratoire dInformatique de Grenoble, and Qwant SAS to develop the fundamental
theory needed to integrate effectiveness evaluation into future ( semantic) search
systems.