In a recent paper, presented at the ACM Conference on Computer Supported Cooperative Work (CSCW), John Horton, Daniel Chen and I used a large-scale experiment to test the effect of different incentive schemes on the quality of crowdsourcing work.

The results surprised us. They suggest that workers perform most accurately when the task design credibly links payoffs to a worker’s ability to think about the answers that their peers are likely to provide.

a horserace experiment! (photo cc-by-sa by iyoupapa)

The idea for this study came out of our sense that, as social scientists, we had something unique to offer the existing research on human computation. Early and influential crowdsourcing research has focused on how to filter the judgments of the crowd to find the best answers. We wanted to know whether simple task-design changes could improve the quality of data coming into a crowdsourcing system in the first place.

To test this idea, we chose 14 different incentive schemes and framing techniques developed and validated across the social sciences and set up a horse race experiment to see which schemes/techniques would work best.

Consistent with our personal biases (John and Daniel are both economists, and I’m a sociologist), some of the schemes were financially oriented, some were social or psychological, and some were hybrids combining social and financial incentives. The details of all the schemes are included in the paper (it’s a long list, and some of them are kind of involved), but it’s worth giving some examples.

On the financial end of the incentives spectrum, we had one condition we called “reward-accuracy,” which was pretty much what you’d expect: we told workers, “we’ll pay you a bonus if you get the answers right.” We also had one called “punishment-accuracy,” the gist of which you can deduce. On the purely social-psychological side, we had one we called “trust,” in which we told workers, “we’ll pay you for this job no matter how bad your performance, we trust that you’ll still make your best effort.”

One of the weirdest schemes turns out to be important, so I need to explain that one. Called “Bayesian Truth Serum” (BTS), it incorporates a design from the work of Drazen Prelec, a behavioral economist at MIT, who realized that research subjects could probably provide useful information regarding the expected distribution for subjective, qualitative questions (nb, the mechanics of how he does this are arcane in a way that is almost sure to delight the geeks among you, so I encourage you to read his paper). Few of the details of real BTS are important, except that we incorporated the piece about asking workers to answer the questions themselves and predict the distribution of other workers’ responses. We also told them we’d give them a bonus if their predictions were correct.

We then created a task that asked workers to answer five questions. In this case, the questions were drawn from another study examining participatory features of websites, for which we already possessed validated data collected by research assistants.

All workers answered the same five questions about the same website (www.kiva.org) while being exposed to one and only one of the 14 incentive schemes (or a control condition of no scheme). Roughly 2,000 individuals participated in the study, resulting in over 100 subjects in each of the experimental conditions. (The statistics and science nerds out there will be pleased to know that both the drop-out rate and demographic covariates were distributed evenly across conditions.)

To measure worker performance, we used the research assistant responses as correct answers to the questions and then calculated the total number of matching answers (out of five) provided by each worker. The results (aggregated across all treatments) are plotted in a histogram below and show that the average worker answered just over two questions out of five correctly.

Then, in order to see how the treatments compared against each other relative to the control group, we calculated the mean correct response rate for each condition and conducted difference of means tests to see which of these means were significantly greater than the control group. The results of this comparison appear below (in a new plot that doesn’t even appear in the paper!):

The orange dots show the value of the mean in each condition, and the blue bars illustrate the 95% confidence interval around that mean. The treatments are sorted by the size of the difference in means from the control. (More hard-core nerd stuff: the means are adjusted using Intent-To-Treat estimators).

From these results, we concluded that our horse race had two clear front-runners: the “Bayesian Truth Serum” (BTS) and “Punishment – disagreement” conditions, each of which improved average worker performance by almost half of a correct answer above the 2.08 correct answers in the control group. A few of the other financial and hybrid incentives had fairly large point estimates, but were not significantly different from control once we adjusted the test statistics and corresponding p-values to account for the fact that we were making so many comparisons at once (apologies if this doesn’t make sense — it’s yet another precautionary measure to avoid upsetting the stats nerds among you). In a tough turn for the sociologists and psychologists, none of the purely social/psychological treatments had any signficant effects at all.

Why do BTS and punishing workers for disagreement succeed in improving performance significantly where so many of the other incentive schemes failed? The answer hinges on the fact that both conditions tied workers’ payoffs to their ability to think about their peers’ likely responses. (We elaborate on the argument in more detail in the paper.)

Does this mean that we should give up on simple financial or social-psychological incentives? Probably not. The fact that we conducted the experiment on MTurk means that the deck may have been stacked against incentives like the “trust” condition I described earlier. Because requesters on MTurk have little oversight, workers are more likely to respond to financial incentives than stated promises. In this sense, the marketplace has structured the interaction between workers and requesters in a way that may limit the opportunities to harness motivations that are not linked to money in some explicit way.

You can download the full paper to read more.

This is a guest post written by my friend Dana Chandler on how the context of a task motivates the person working on it.  He has a longer academic paper on the topic you can find at the bottom of this post.  It once again shows how traditional economic incentives can’t fully explain workers’ behaviors on Mechanical Turk.

Imagine for a moment that you were a turker from either the US or India, looking at the above image. You are given the task of clicking on the blue circular objects with red borders. What you see is only a fraction of the full image. Each image has 90 blue objects to identify. If you’re as good as the average worker, you’ll complete your first image in a little over five minutes and you’ll earn 10 cents (for an hourly wage of $1.20).

After your first image, you can either quit and take your 10 cents, or identify points on another image. Over the next four hours, you’ll have the chance to label as many images you want. But there’s a catch—you’ll only be paid 9 cents for the second image, 8 cents for the third, and so on, all the way down to 2 cents. This will lower the hourly wage even more.

Before you even qualify for the task, you’ll have to spend five minutes watching a training video and passing a quiz. During the video, half of you will be given only basic work instructions on how to identify “objects of interest.” The other half will be given both instructions and cues of meaning: recognition for your contribution and an explanation of your task’s purpose¹. The reason given here? To help researchers identify cancerous tumor cells.

We posted these HITs on MTurk in January, 2010. Almost 300 people from the U.S. and India accepted the task, becoming unknowing participants in our experiment examining MTurk worker motivation. It is commonly believed (and other researchers have verified with demographic surveys) that Indian workers are more motivated by pecuniary concerns and that US turkers are primarily doing tasks for leisure or other non-pecuniary motives. Is this true?

In both countries, half of the turkers in the experiment were randomly assigned to label nondescript “objects of interest” without being given any context or greater purpose — they were our zero-context group. The other half, our meaningful group, were told they were helping researchers identify cancerous tumor cells. Which group of turkers do you think worked harder? You might be surprised.

Therefore, our experiment compared two groups with and without a clear wage motivation, to see if workers behave differently responded to meaningfulness in their tasks.

Results

We measured three metrics: “showing up”, the quantity of work, and the quality of that work. The first two metrics are straightforward. Showing up meant that you sat through our training video, passed our qualification test and helped label at least one image. Quantity of work was simply the number of images labeled.

We repeatedly told both groups of turkers that they needed to click on all points and as closely as possible to each point. Work quality was determined by the fraction of cells that a person clicked on (the recall) and the average distance between the “true center” of each cell and where the user clicked (the centrality).

Our most interesting finding was the extent to which a meaningful task (and giving recognition) motivated US workers, but not Indian workers, to complete a task. As any requester knows, attrition on MTurk is a real problem. We found that adding cues of meaning could motivate turkers to undergo training and label at least one image. In the US, adding cues-of-meaning raised the fraction of turkers who completed our task from, 92% of people who sat through our training video, took our quiz, and labeled an image showed up. This figure compares to only 83% of zero-context group (see figure which also has standard errors). In India, there was no difference between the groups and both groups had a 66% completion rate (attrition being higher due to possible language barriers, slow connection speeds, hardware issues, etc.).

However, once a person did some work, both treatment and control groups did a similar quantity of work: The cues-of-meaning group labeled 6.0 images and the zero-context group labeled 5.7 images. This difference was not statistically significant, so it suggests that once you get turkers to work on a task, they are motivated to label just as many images irrespective of the task’s meaningfulness. Notably, of the people who worked, Indians worked longer and labeled an average of 7.3 images vs. 5.2 in the US.

Surprisingly, all workers did an equally good job identifying points whether they had zero-context or whether they thought they were identifying tumor cells. The quality as measured by the fraction of points identified (the recall) or the average pixel distance (the centrality) was statistically insignificant irrespective of the task’s meaningfulness.

This finding has important implications for those who employ labor in crowdsourcing markets. Companies and intermediaries should develop an understanding of what motivates the people who work on tasks. Employers must think beyond monetary incentives and consider how they can reward workers through non-monetary incentives such as by changing how workers perceive their task. Alienated workers are less likely to do work if they don’t know the context of the work they are doing and employers may find they can get more work done for the same wages simply by telling turkers why they are working.

For more details of this study, please see our full academic paper at: http://danachandler.com/index.php/research. We welcome any comments and feedback.

About the authors:
Dana Chandler is a researcher at the University of Chicago’s Becker Center where he works with Steven Levitt, author of Freakonomics. He previously worked as a management consultant at the Boston Consulting Group and at Aureos Capitol, a Colombian private equity company. He will begin his Ph.D. at MIT in the Fall.  Dana’s research interests include digital labor markets, development economics, and randomized experiments in companies. email: dchandler {at} uchicago {dot} edu

Adam Kapelner is currently earning his Ph.D. in Statistics at Wharton. Adam is the founder of  dictionarysquared.com and the inventor of its vocabulary-learning  technology. While working as an undergraduate researcher at Stanford University, he helped engineer the open-source software, www.gemIdent.com, that enables researchers worldwide to locate cells in microscopic images. GemIdent was recently extended to make use of MTurk for outsourcing of medical image identification. The extension, called www.distributeeyes.com, was  adapted to serve as the platform for this experiment. email: kapelner  {at} wharton {dot} upenn {dot} edu

Acknowledgments: We thank Professor Susan Holmes of Stanford University for allowing us to adopt DistributeEyes (funded under NIH grant #R01GM086884-02) for use in this study. We would also like to thank Panos Ipeirotis for kindly providing us with demographic and market data that we cite in our study. Lawrence Brown, Patrick DeJarnette, John Horton, Emir Kamenica, Steven Levitt, Susanne Neckermann, Jesse Shapiro, Jorg Spenkuch, Jan Stoop, Chad Syverson, Mike Thomas, Abraham Wyner, and seminar participants at the University of Chicago provided especially helpful comments.

Online labor markets dramatically lower the cost and hassle of conducting experiments. On Amazon’s Mechanical Turk, it is easy to run multiple experiments per week. Figuring out how to run experiments isn’t that hard, as there are already some nice tutorials available.

However, what I felt was missing from the field was a discussion of why, precisely, we can trust results from online experiments. This was the motivation for a new paper, jointly written with Dave Rand (who wrote up part of this study here on the Dolores Labs blog) and Richard Zeckhauser.

You can download the paper here.

While we make the practical and theoretical case for online experimentation, we believe that acceptance of online results as “valid” will come after people start seeing how easy and reliably one can replicate previous studies. This is why blogs like Experimental Turk and Deneme—both of which report results from AMT experiments—are so helpful. In our paper, we continue this process by replicating three results that are fairly well established.

In one experiment for the economists, we show—contra the usual intuition—that at least some Turkers are financially motivated, despite the very low stakes. After performing an initial text transcription task, workers were offered some randomly chosen amount of money to do an additional transcription. Results show the counts of people who agreed (“Yes”) and the counts of people who did not agree (“No”), by amount offered.

Nothing too surprising—offer to pay more and more workers will accept—but at this stage in the development of online experiments as a methodology, “surprising” would probably be bad news.

Anyway, the full paper is here. We’d love to get comments and feedback—it’s not too late to earn a place in our coveted “thanks” footnote!

“Who are these people?” and “Why do they participate?” are two perennial questions about AMT. Askers are generally incredulous that AMT workers are willing to do rather tedious tasks for small amounts of money.

To investigate this question of motivation, NYU Prof. Panos Ipeirotis asked a bunch of workers their reasons and tabulated the responses here. His two posts are actually on the syllabus for a course at Stanford (incidentally the course is taught by one of the creators of Protovis, which is very cool and is on my list of things to learn). There is also this amusing investigation.

For a joint project with Dave Rand and Richard Zeckhauser, we asked ~ 400 AMT workers both (a) where they are from and (b) the primary reason they participate on AMT. Because economic opportunities differ by country, we might expect that motivation and behavior should also differ by country. The cross tabulation plot is below (reasons are in the “rows”, countries in the “columns”–the size of each rectangle is proportional to the number of responses in that cell):

Two things to note:
1) Money is a big motivation for everyone
2) Money aside, people from India are there to learn; people from the US are there to have fun

Although the India/US differences are consistent with the different-countries/different-motivations hypothesis, the most relevant fact is the unconditional importance of money.

While these findings seem reasonable, I feel compelled to make the standard reliability critique of self-reported data. Our learning/fun AMT workers might also be there for the money, but feel sheepish about saying so. Though this could go the other way as well I suppose: if, for example, a worker has an intrinsic love of image captioning but finds this passion shameful, they might report that they are in it for the money. But this seems less likely than the other scenario of downplaying financial motivations.