Fernando Diaz – The Harsh Reality of Production Information Access Systems (Karen Spärck Jones Award)

Tuesday, 27th of March at 9:30am

The past twenty years have seen the widespread deployment of large scale information access systems.  While academic research provided the foundation for everyday technologies such as web search, music recommendation, and social media filtering, the scale and nuance of a production environment required revisiting core assumptions and developing new methods.  More recently, the widespread adoption of these systems has forced scientists and engineers to confront the harsh reality that designing a production information access system requires a deep understanding of individuals, groups, and society.  In this presentation, I will reflect on lessons learned and discuss open problems for the information retrieval community.

Fernando Diaz is a Director of Research at Spotify.  Prior to joining Spotify, Fernando was a senior researcher and founding member of Microsoft Research New York. Before that, Fernando was a senior scientist at Yahoo Research.  His primary research area is information retrieval, focusing on formal models for query understanding, core ranking, and evaluation.  Fernando received a B.Sc. in Computer Science and a B.A. in Political Science, both from the University of Michigan, and a Ph.D. from the University of Massachusetts Amherst.   His work has been recognized at SIGIR, WSDM, ICML, ISCRAM, and ECIR.

Gabriella Kazai – Challenges in Building IR Evaluation Pipelines

Wednesday, 28th of March at 9:00am

Evaluation is key in the development of information retrieval (IR) systems. Offline evaluation relies on human judgments and metrics based on the collected judgments, while online evaluation relies on observations recorded about the users’ interactions with the system and metrics that rely on these signals. The building of an offline evaluation pipeline includes the design of mechanisms to obtain representative samples of data for a given IR task, e.g., document corpus, query sets, user profiles, etc., the development or selection of suitable metrics and the design of labelling systems to collect high-quality human judgments. Online experimentation typically requires an AB testing framework and the instrumentation of the UI. The challenge of building evaluation pipelines from which reliable conclusions can be drawn is no trivial matter. For example, biases in the collected labels can invalidate the evaluation and lead to problems when used in the training of machine learning systems. In this talk I will reflect on some of the challenges and lessons learnt from building IR evaluation pipelines in three different contexts: 1) Constructing a test collection and developing novel metrics for the evaluation of a new, so called, focused search paradigm at INEX, 2) Addressing the challenges of incorporating crowdsourcing into the offline metrics pipeline of the Bing search engine, and 3) Offline vs online evaluations for the Lumi News app, a recommender system for crowd-curated Web content.

I am a Senior Applied Scientist at Microsoft, tackling a range of IR evaluation challenges as part of the Bing Web and AI Sciences Group. In the 3 years prior to that, I worked at two startup companies. As Lead Data Scientist at Mudano Ltd, I worked on building an AI-driven project management system to deliver large-scale IT change projects in the financial sector; and as head of Data Science at Semion Ltd, I led a small team of data scientists implementing the AI behind the Lumi News app, providing out-of-the-box personalised media content to users based on interests discovered from their Twitter streams. Before that, I worked for 7 years as a research consultant at Microsoft Bing and as a postdoc at Microsoft Research. I obtained my PhD from the University of London under the supervision of Prof Mounia Lalmas. My background is in computer science with over 10 years of research experience in IR. My research interests include offline/online IR evaluation, IR metrics, crowdsourcing, gamification, recommender systems, data mining, applied ML, social IR, information seeking behaviour, and activity based PIM.

Radim Řehůřek – Anatomy of an idea: mixing open source, research and business (Industry day Keynote)

Thursday, 29th of March at 9:00am

Radim Řehůřek is the founder and director of RARE Technologies, a leading R&D company focused on machine learning and natural language processing. Radim has been building practical solutions for businesses for over a decade. He is the creator of Gensim, a popular open source Python library for topic modeling and information retrieval.