2nd Workshop on Offline and Online Evaluation of Interactive Systems
Submissions for the 2019 workshop are closed.
2019 workshop materials. Dates, affiliations and biographies reflect the original event.
Workshop Day
Date: August 5, 2019 (a half day workshop, from 8am till 12pm)
Location: Summit 13- Ground Level, Egan
Call For Papers
As Machine Learning and Data Science have become more ubiquitous and are being applied to so many different domains such as urban computing, customer relationship management, self-driving cars, fintech and not just limited to online advertising and recommendation systems (as was the case few years ago), it is imperative that we come together as a community and understand how to best evaluate these complex and multi-layered systems offline as well as online. For example, in offline mode, scientists may be optimizing for proxy metrics coming from supervised methods like precision, recall, AUC, and online the business would like focus on innovation and impact on business metrics. So, can we bridge the gap between these two worlds? Moreover, the interactive and dynamic nature of these systems pose unique challenges for evaluating them offline. Recent research makes the connection of offline evaluation to counterfactual analysis and causal inference, offering new ways to evaluate these systems offline. We believe it is thus timely to organize a workshop to foster discussion on these topics between the Machine Learning and Data Science communities. In this workshop we would like to discuss not only the evaluation metrics but broadly cover evaluation techniques and systems such as A/B testing, causal discovery, synthetic control experiments, causal inference,etc.
Submission Instructions
Submissions must follow the ACM KDD 2019 format, and should be in PDF format. Formatting instructions and materials can be found at KDD 2019 website - paper submission requirements. Each submission should not exceed 6 pages, including references and appendices. Please make a submission through https://cmt3.research.microsoft.com/OOEW2019. The review process is single-blinded and the authors do not need to remove their names from the submissions. All accepted submissions will be presented at the workshop, in the form of posters or talks. At least one author of each accepted paper will have to register for the workshop.
Important Dates
Submission deadline: May 19th May 5, 2019 (23:59 Hawaii Standard Time)
Acceptance notification: June 1, 2019
Camera-ready deadline: June 14, 2019
Submission website : https://cmt3.research.microsoft.com/OOEW2019
Topics and Themes
Topics of the workshop include, but are not limited to, the following:
* Classic offline evaluation methodologies of systems, especially those based on standard metrics such as RMSE (Root mean square error) and NDCG (Normalized Discounted Cumulative Gain).
* Online controlled experiments such as A/B testing, etc.
* Online interleaving experiments.
* Experiments in wake of network effects.
* Online adaptive sequential experiments.
* Offline evaluation of direct metrics such as revenue and click-through rate gains.
* Efficacy of Causal inference and counterfactual analysis when experiments are expensive.
* Practical applications and lessons related to evaluation of online services.
Examples of open questions we would like to discuss include (and are not limited to):
* What are the best ways we can measure the performance of a machine learning algorithm offline? When are traditional machine learning criteria such as precision, recall, and AUC good enough for reflecting actual quality of the system?
* What are the more efficient ways of doing online experiments, other than the vanilla version of randomized controlled experiments? Can adaptive sequential experimentation techniques be helpful? Can variants such as interleaving be useful for applications other than information retrieval?
* What effective offline evaluation techniques do we have for measuring conversions/revenue gain, and what are their limitations?
* How can we measure the confidence in the estimates, either in the online or offline cases? Should we use t-tests or Bayesian statistics?