YINS - Dengyong Zhou, Microsoft Research
“Incentives in Human Computation”
Abstract: Crowdsourcing (or human computation) is becoming a major player in collecting training examples for building intelligent systems from image recognition to machine translation. Many companies have developed their own crowdsourcing platforms, including Facebook, Google and Microsoft. There are also many public crowdsourcing web services such as Amazon Mechanical Turk. The most remarkable advantage of crowdsourcing is that a large amount of labels can be quickly collected from worldwide workers at a low cost. However, due to the lack of task-specific expertise and maligned incentive mechanisms, crowdsourcing is typically plagued by the problem of low quality.
To address this fundamental issue, we designed a novel while extremely simple reward mechanism (called “double-or-nothing”) which incentivizes workers to only answer the questions that they are sure and skip the rest that they are not sure. In preliminary experiments involving over 900 worker-tasks in Amazon Mechanical Turk, we observed up to a three-fold drop in the error rates under this unique incentive mechanism. Theoretically, our mathematical proofs surprisingly show that our “double-or-nothing” mechanism is the only possible incentive-compatible mechanism that satisfies a natural and desirable ‘no-free-lunch’ requirement (which says that, if all attempted answers from a work are wrong, he will receive zero payment). In this talk, I will focus on this new mechanism and also discuss its extensions to several other crowdsourcing scenarios.
It is joint work with Nihar Shah and Yuval Peres.
Bio: Denny Zhou is a Senior Researcher in Microsoft Research (MSR) Redmond. His research interests are centered on statistical machine learning and its applications. In particular, he extensively works on algorithmic foundations for developing large-scale human-machine systems that combine the intelligence of human and the computing power of machine to address the problems that are difficult to solve by either human or machine alone. He received Gold Star Award from Microsoft Research in 2010 for his exceptional contributions to Microsoft products. Before joining MSR in 2006, he worked in NEC Labs America in Princeton, NJ, and Max Planck Institute for Intelligent Systems, Tuebingen, Germany. He obtained his PhD in computer science from Chinese Academy of Sciences in 2002 together with Presidential Award for his research excellence.