2nd Workshop on Offline and Online Evaluation of Interactive Systems
2019 workshop materials. Dates, affiliations and biographies reflect the original event.
Workshop Day
Date: August 5, 2019 (a half day workshop, from 8am till 12pm)
Location: Summit 13- Ground Level, Egan
Workshop Objectives
While there has been a long-standing focus of empirical evaluation in building Interactive systems, new challenges have been faced by traditional evaluation methodologies to truly reflect the systems' actual online performance. This workshop aims to connect academic researchers and industrial practitioners who are working on, or interested in, online and offline evaluation of Interactive systems. The goal is to provide a forum so that industrial practitioners can expose real-world challenges and share practical experiences; academic researchers can popularize state-of-art research; and collaboration between the two can be fostered.
Topics and Themes
Topics of the workshop include, but are not limited to, the following:
* Classic offline evaluation methodologies of systems, especially those based on standard metrics such as RMSE (Root mean square error) and NDCG (Normalized Discounted Cumulative Gain).
* Online controlled experiments such as A/B testing, etc.
* Online interleaving experiments.
* Experiments in wake of network effects.
* Online adaptive sequential experiments.
* Offline evaluation of direct metrics such as revenue and click-through rate gains.
* Efficacy of Causal inference and counterfactual analysis when experiments are expensive.
* Practical applications and lessons related to evaluation of online services.
Examples of open questions we would like to discuss include (and are not limited to):
* What are the best ways we can measure the performance of a machine learning algorithm offline? When are traditional machine learning criteria such as precision, recall, and AUC good enough for reflecting actual quality of the system?
* What are the more efficient ways of doing online experiments, other than the vanilla version of randomized controlled experiments? Can adaptive sequential experimentation techniques be helpful? Can variants such as interleaving be useful for applications other than information retrieval?
* What effective offline evaluation techniques do we have for measuring conversions/revenue gain, and what are their limitations?
* How can we measure the confidence in the estimates, either in the online or offline cases? Should we use t-tests or Bayesian statistics?