In a recent paper, presented at the ACM Conference on Computer Supported Cooperative Work (CSCW), John Horton, Daniel Chen and I used a large-scale experiment to test the effect of different incentive schemes on the quality of crowdsourcing work.

The results surprised us. They suggest that workers perform most accurately when the task design credibly links payoffs to a worker’s ability to think about the answers that their peers are likely to provide.

a horserace experiment! (photo cc-by-sa by iyoupapa)

The idea for this study came out of our sense that, as social scientists, we had something unique to offer the existing research on human computation. Early and influential crowdsourcing research has focused on how to filter the judgments of the crowd to find the best answers. We wanted to know whether simple task-design changes could improve the quality of data coming into a crowdsourcing system in the first place.

To test this idea, we chose 14 different incentive schemes and framing techniques developed and validated across the social sciences and set up a horse race experiment to see which schemes/techniques would work best.

Consistent with our personal biases (John and Daniel are both economists, and I’m a sociologist), some of the schemes were financially oriented, some were social or psychological, and some were hybrids combining social and financial incentives. The details of all the schemes are included in the paper (it’s a long list, and some of them are kind of involved), but it’s worth giving some examples.

On the financial end of the incentives spectrum, we had one condition we called “reward-accuracy,” which was pretty much what you’d expect: we told workers, “we’ll pay you a bonus if you get the answers right.” We also had one called “punishment-accuracy,” the gist of which you can deduce. On the purely social-psychological side, we had one we called “trust,” in which we told workers, “we’ll pay you for this job no matter how bad your performance, we trust that you’ll still make your best effort.”

One of the weirdest schemes turns out to be important, so I need to explain that one. Called “Bayesian Truth Serum” (BTS), it incorporates a design from the work of Drazen Prelec, a behavioral economist at MIT, who realized that research subjects could probably provide useful information regarding the expected distribution for subjective, qualitative questions (nb, the mechanics of how he does this are arcane in a way that is almost sure to delight the geeks among you, so I encourage you to read his paper). Few of the details of real BTS are important, except that we incorporated the piece about asking workers to answer the questions themselves and predict the distribution of other workers’ responses. We also told them we’d give them a bonus if their predictions were correct.

We then created a task that asked workers to answer five questions. In this case, the questions were drawn from another study examining participatory features of websites, for which we already possessed validated data collected by research assistants.

All workers answered the same five questions about the same website (www.kiva.org) while being exposed to one and only one of the 14 incentive schemes (or a control condition of no scheme). Roughly 2,000 individuals participated in the study, resulting in over 100 subjects in each of the experimental conditions. (The statistics and science nerds out there will be pleased to know that both the drop-out rate and demographic covariates were distributed evenly across conditions.)

To measure worker performance, we used the research assistant responses as correct answers to the questions and then calculated the total number of matching answers (out of five) provided by each worker. The results (aggregated across all treatments) are plotted in a histogram below and show that the average worker answered just over two questions out of five correctly.

Then, in order to see how the treatments compared against each other relative to the control group, we calculated the mean correct response rate for each condition and conducted difference of means tests to see which of these means were significantly greater than the control group. The results of this comparison appear below (in a new plot that doesn’t even appear in the paper!):

The orange dots show the value of the mean in each condition, and the blue bars illustrate the 95% confidence interval around that mean. The treatments are sorted by the size of the difference in means from the control. (More hard-core nerd stuff: the means are adjusted using Intent-To-Treat estimators).

From these results, we concluded that our horse race had two clear front-runners: the “Bayesian Truth Serum” (BTS) and “Punishment – disagreement” conditions, each of which improved average worker performance by almost half of a correct answer above the 2.08 correct answers in the control group. A few of the other financial and hybrid incentives had fairly large point estimates, but were not significantly different from control once we adjusted the test statistics and corresponding p-values to account for the fact that we were making so many comparisons at once (apologies if this doesn’t make sense — it’s yet another precautionary measure to avoid upsetting the stats nerds among you). In a tough turn for the sociologists and psychologists, none of the purely social/psychological treatments had any signficant effects at all.

Why do BTS and punishing workers for disagreement succeed in improving performance significantly where so many of the other incentive schemes failed? The answer hinges on the fact that both conditions tied workers’ payoffs to their ability to think about their peers’ likely responses. (We elaborate on the argument in more detail in the paper.)

Does this mean that we should give up on simple financial or social-psychological incentives? Probably not. The fact that we conducted the experiment on MTurk means that the deck may have been stacked against incentives like the “trust” condition I described earlier. Because requesters on MTurk have little oversight, workers are more likely to respond to financial incentives than stated promises. In this sense, the marketplace has structured the interaction between workers and requesters in a way that may limit the opportunities to harness motivations that are not linked to money in some explicit way.

You can download the full paper to read more.

“Who are these people?” and “Why do they participate?” are two perennial questions about AMT. Askers are generally incredulous that AMT workers are willing to do rather tedious tasks for small amounts of money.

To investigate this question of motivation, NYU Prof. Panos Ipeirotis asked a bunch of workers their reasons and tabulated the responses here. His two posts are actually on the syllabus for a course at Stanford (incidentally the course is taught by one of the creators of Protovis, which is very cool and is on my list of things to learn). There is also this amusing investigation.

For a joint project with Dave Rand and Richard Zeckhauser, we asked ~ 400 AMT workers both (a) where they are from and (b) the primary reason they participate on AMT. Because economic opportunities differ by country, we might expect that motivation and behavior should also differ by country. The cross tabulation plot is below (reasons are in the “rows”, countries in the “columns”–the size of each rectangle is proportional to the number of responses in that cell):

Two things to note:
1) Money is a big motivation for everyone
2) Money aside, people from India are there to learn; people from the US are there to have fun

Although the India/US differences are consistent with the different-countries/different-motivations hypothesis, the most relevant fact is the unconditional importance of money.

While these findings seem reasonable, I feel compelled to make the standard reliability critique of self-reported data. Our learning/fun AMT workers might also be there for the money, but feel sheepish about saying so. Though this could go the other way as well I suppose: if, for example, a worker has an intrinsic love of image captioning but finds this passion shameful, they might report that they are in it for the money. But this seems less likely than the other scenario of downplaying financial motivations.

Many workers on Amazon Mechanical Turk are willing to help others at a cost to themselves, just like participants in laboratory experiments.

While traditional economic models assume that people are entirely selfish, a central theme in behavioral economics is the existence of ‘social preferences’, or caring for others. Countless laboratory experiments have demonstrated that many people are willing to help others, even at a cost to themselves. This behavior is clearly inconsistent with being motivated only by your own monetary payoff – if you are entirely selfish, you would never pay money to help someone else in the totally anonymous conditions of the lab. In this post I describe an experiment I conducted together with John Horton, and with invaluable technical assistance from Xiaoqi Zhu, that replicates the existence of social preferences on Amazon Mechanical Turk (AMT), showing that many Turkers behave altruistically.

We also demonstrate the principle of priming, another focus of great interest in experimental economics. In priming studies, stimuli unrelated to the decision task (and which do not affect the monetary outcomes) can nonetheless significantly alter subjects’ behavior.

To assess altruistic behavior on AMT, 194 subjects played an incentivized Prisoner’s Dilemma (PD), the canonical game for studying altruistic cooperation. Subjects were informed that they had been randomly assigned to interact with another Turker, and that they would each have a choice between two options, A or B. In addition to a 20 cent “show-up fee”, they were informed of the following payoff structure: if both subjects chose A, they receive each earn a 120 cent bonus; if both chose B, they would each receive an 80 cent bonus; if one chose A while the other chose B, the A player would receive 40 cents while the B player would receive 160 cents. The resulting payoff matrix is as follows (in each cell I first show the row player’s payoff, and then the column player’s payoff):

Thus A represents cooperation, and B represents defection. If both people chose A, they both do better than if both choose B. However, regardless of the other’s action, you earn more by choosing B (hence the ‘dilemma’). Rational self-interested players should therefore always select B, and it is altruistic to choose A (helping the other at a cost to you). Given previous evidence from experiments in the laboratory, however, we predicted that AMT subjects would demonstrate a level of cooperation significantly greater than 0 in a one-shot PD.

To explore the effects of priming on AMT subjects, we built on a previous study demonstrating that exposure to religious words and phrases increases altruistic behavior, particularly among those who believe in god (Shariff & Norenzayan 2007). Among the 194 subjects in our experiment, the prime group (N=89) read a Christian religious passage about the importance of charity (Mark 12:21-22) before playing the PD, whereas the no-prime group (N=105) did not. Following the PD, subjects completed a demographic questionnaire reporting age, gender, and education, and indicated whether they had ever had an experience which convinced them of the existence of god.Based on the results of Sheriff & Norenzayan, we hypothesized that the religious prime would increase cooperation, and further hypothesized that the effect would be driven by subjects that believe in god.

Consistent with our first prediction, we observe a level of cooperation significantly greater than 0 in both the no-prime (54% C: sign-rank test, p<0.001) and prime (71% C: sign-rank test, p<0.001) conditions. Consistent with our second prediction, we observe significantly more cooperation in the prime condition compared to the no-prime condition (Chi² test, p=0.018). Consistent with our third prediction, the prime only increases cooperation among subjects who believe in god (Chi² test, non-believers: p=0.82, believers: p=0.004). The results are visualized in Figure 1. Using logistic regression with robust standard errors, we also find that these results are robust to controlling for age, gender, country of residence (US vs non-US), religion (Christian vs non-Christian) and education.

Figure 1. Reading a religious passage significantly increases Prisoner’s Dilemma cooperation among those who believe in god, but not among non-believers.


To summarize, we have demonstrated two aspects of Turker behavior:

1. A majority of Turkers chose the altruistic option of cooperating in a Prisoner’s Dilemma. Thus even in the entirely anonymous and profit-motivated online labor market of AMT, many people still choose to help each other. This sort of altruistic cooperation is a fundamental part of the natural world, and is the building block of human societies. For more, see (Nowak 2006).

2. Reading a religious passage about the important of charity makes religious Turkers more altruistic, but has no effect on Turkers who do not believe in god. This shows that Turkers respond in basically the same way as “normal” lab subjects, and is fairly intuitive. Those who believe in god are receptive to calls for generosity phrased in religious language, while non-believers aren’t. Secular primes have been shown to work for both religious and non-religious subjects (Shariff & Norenzayan 2007).

Although AMT workers are certainly not a generally representative sample, this study demonstrates that they show several of the same basic behavioral features observed in behavioral laboratory experiments. Furthermore, AMT allowed this study to be run extremely quickly and inexpensively. The 200 subjects were recruited in less than 2 days, at a total cost of $253. As a behavioral researcher, this is amazingly exciting! I usually spend months and thousands of dollars per study. AMT opens the possibility of exploring countless interesting ideas that otherwise we would have had neither the time nor money to pursue.

For other studies about cooperation, reward and punishment that I’ve conducted at Harvard, see the pdfs on my webpage: www.DavidGertlerRand.com.