Infrastructure
An innovative living lab infrastructure for information retrieval evaluations
STELLA (InfraSTrucutrEs for Living LAbs) offers an Evaluation-as-a-Service platform for living lab experiments with ranking and recommender systems. By using STELLA, researchers can evaluate their experimental systems based on user feedback which stands in contrast to the Cranfield-style approaches with test collections in offline evaluations. STELLA facilitates conventional AB tests but also more data-efficient interleaving experiments in which results lists of two ranking or recommender functions are mixed. A fundamental component of STELLA is the integration of experimental systems as micro-services. While previous living labs restricted the system results to the most popular top-k queries, we allow more comprehensive evaluations by integrating micro-services with entire retrieval and recommender systems. The CLEF lab “Living Labs for Academic Search (LiLAS)” made use of the STELLA infrastructure and served as the first test-bed to evaluate the feasibility of our new infrastructure design. We welcome contributions and look for collaborations with researchers and sites alike.
More information can be found on the following sites: