2nd Workshop on Offline and Online Evaluation of Interactive Systems
2019 workshop materials. Dates, affiliations and biographies reflect the original event.
Workshop Day
Date: August 5, 2019 (a half day workshop, from 8am till 12pm)
Location: Summit 13- Ground Level, Egan
Schedule
| 8:00~8:45:am |
Unbiased Evaluation for Unbiased Rankings
by Thorsten Joachims (Cornell University)
Ranking systems like search engines, recommender systems, and e-commerce sites are intelligent agents that intervene in the world through the rankings they present. This creates potential for misleading and unfair bias among the ranked items in at least two ways. First, the choices made by the current ranking function influence which items are likely to receive implicit feedback (e.g. clicks), thus biasing the data that is collected for evaluation. To overcome this selection bias, this talk explores how to use causal-inference techniques to design unbiased estimators of ranking quality despite biased data. This leads to debiased training objectives for Learning-to-Rank (LTR) through inverse-propensity-score (IPS) weighting. To estimate these propensities, I will propose a new propensity estimator that does not require disruptive randomized interventions. Second, even if selection biases are properly accounted for, there are still several other biases that could unfairly influence the results. For example, the rankings could be biased between groups of items (e.g. by political orientation) in how they allocate exposure (i.e. where to rank an item) based on merit (i.e. relevance). To address this second type of bias, I will challenge the long-standing foundation of ranking evaluation, namely the Probability Ranking Principle (PRP), as the ideal that our evaluation criteria should approximate. In particular, I argue that ranking systems based on the PRP can be unfair to the items that are being ranked, and I will propose alternate ranking metrics. |
||
| 8:45~9:00am |
Statistical Network Models: How to Leverage Network Structure to Improve Estimation & Prediction by Nasreen Ahmed (Intel Research)
Networks are a natural representation of complex systems across domains, from social, to biological, to technological. Modeling the dynamics and structure of networks is central to the understanding of these systems and have a huge economic impact. However, the characteristics of network data captured from online complex systems present a number of challenges to the design and evaluation of machine learning methods. This talk will outline some of the important algorithmic and evaluation challenges that arise due to the massive size, streaming nature and heterogenous structure of complex networks, and discuss statistical online methods, unbiased and shrinkage estimators, as well as learning techniques to address some of these challenges. |
||
| 9:00~9:30am |
The Reasonable Uneffectiveness of Big Data: And where it still helps by Tim Althoff (University of Washington)
The rise of Big Data and Data Science promise unparalleled opportunities to develop new types of scientific approaches. However, there are fundamental challenges of causal inference that are not addressed through simply larger datasets. In this talk, I will highlight key challenges and argue why more data will not be a solution. Instead, gaining valuable insights from data requires new computational approaches that turn observational, scientifically “weak” data into strong scientific results. I will discuss two key benefits of large-scale datasets, natural experiments and unforeseen proxies, that can enable actionable and impactful insights. Challenges and approaches will be illustrated through examples from health applications, including physical activity, sleep, and cognitive performance. |
||
| 9:30~9:45am | Coffee Break | ||
| 9:45~10:00am |
Safely and Quickly Deploying New Features with a Staged Rollout Framework Using Sequential Test and Adaptive Experimental Design by Zhenyu Zhao (Uber) paper link
During the rapid development cycle for Internet products (websites and mobile apps), new features are developed and rolled out to users constantly. Features with code defects or design flaws can cause outages and significant degradation of user experience. The traditional method of code review and change management can be time-consuming and error-prone. In order to make the feature rollout process safe and fast, this paper proposes a methodology for rolling out features in an automated way using an adaptive experimental design. Under this framework, a feature is gradually ramped up from a small proportion of users to a larger population based on real-time evaluation of the performance of important metrics. If there are any regression detected during the ramp-up step, the ramp-up process stops and the feature developer is alerted. There are two main algorithm components powering this framework: 1) a continuous monitoring algorithm - using a variant of the sequential probability ratio test (SPRT) to monitor the feature performance metrics and alert feature developers when a metric degradation is detected, 2) an automated ramp-up algorithm - deciding when and how to ramp up to the next stage with larger sample size. This paper presents one monitoring algorithm and three ramping up algorithms including time-based, power-based, and risk-based (a Bayesian approach) schedules. These algorithms are evaluated and compared on both simulated data and real data. There are three benefits provided by this framework for feature rollout: 1) for defective features, it can detect the regression early and reduce negative effect, 2) for healthy features, it rolls out the feature quickly, 3) it reduces the need for manual intervention via the automation of the feature rollout process. |
||
| 10:00~10:30am |
Evaluation Challenges in Network Analysis by Danai Koutra (University of Michigan)
Networks naturally capture a host of real-world interactions, from social interactions and email communication to web browsing to brain activity. In this talk, I will discuss our recent work on representation learning and graph summarization, as well as their applications to entity resolution, entity linking across data sources, knowledge graph tasks and neuroscience. Throughout the talk I will emphasize the key evaluation challenges that arise in these settings, ways to overcome them, and related open problems. |
||
| 10:30~11:00am |
Interactive Tradeoffs Between Competing Offline Metrics with Bayesian Optimization by Michael McCourt (SigOpt)
Many real world applications (ML models, simulators, etc.) have multiple competing metrics that define performance; these require practitioners to carefully consider potential tradeoffs. However, assessing and ranking this tradeoff is nontrivial, especially when the number of metrics is more than two. Often times, practitioners scalarize the metrics into a single objective, e.g., using a weighted sum. In this talk, we pose this problem as a constrained multi-objective optimization problem. By setting and updating the constraints, we can efficiently explore only the region of the Pareto efficient frontier of the model/system of most interest. We motivate this problem with the application of an experimental design setting, where we are trying to fabricate high performance glass substrate for solar cell panels. Quantifying the Impact of a Recommendation by Arden Dertat (Netflix) Showing relevant content to members is crucial both on-site with homepage personalization and off-site using ads. Traditional machine learning approaches relying on correlational data perform poorly, especially in paid marketing. We will discuss challenges faced and the solutions to improve recommendations and business metrics. Evaluating the Performance and Applicability of User Behavior Model by Sungchul Kim (Adobe) paper link The rapid growth of mobile devices has resulted in the generation of a large number of user behavior logs that contain latent intentions and user interests. However, modeling such data in real-world applications is still difficult for service providers due to the complexities of user behavior over a sheer number of possible actions that can vary according to time. To address this, we proposed a time-aware RNN model for predictive analysis from user behavior data and proved the superiority of our model over the baselines. In this talk we present how we evaluated the performance of user behavior model including its by-product; user embedding, which indicates the applicability of the model. |
||
| 11:00~11:30am |
Solving Experiment Interference, Practical Solutions @LinkedIn by Ya Xu (Linkedin)
Interference, also called spillover or leakage, can happen when there is a medium connecting treatment and control groups, either through direct connections as in social networks, or through latent variables such as shared resources (e.g. budget). With over 100 experiments ramped every day at LinkedIn, we have observed and measured interference in many experiments. In this talk, we will share some learning and practical solutions we have used at LinkedIn to detect and measure interference. |
||
| 11:30~12:00am | Off-policy optimization and missing not at random by Ed Chi (Google) |