In a recent paper, presented at the ACM Conference on Computer Supported Cooperative Work (CSCW), John Horton, Daniel Chen and I used a large-scale experiment to test the effect of different incentive schemes on the quality of crowdsourcing work.

The results surprised us. They suggest that workers perform most accurately when the task design credibly links payoffs to a worker’s ability to think about the answers that their peers are likely to provide.

a horserace experiment! (photo cc-by-sa by iyoupapa)

The idea for this study came out of our sense that, as social scientists, we had something unique to offer the existing research on human computation. Early and influential crowdsourcing research has focused on how to filter the judgments of the crowd to find the best answers. We wanted to know whether simple task-design changes could improve the quality of data coming into a crowdsourcing system in the first place.

To test this idea, we chose 14 different incentive schemes and framing techniques developed and validated across the social sciences and set up a horse race experiment to see which schemes/techniques would work best.

Consistent with our personal biases (John and Daniel are both economists, and I’m a sociologist), some of the schemes were financially oriented, some were social or psychological, and some were hybrids combining social and financial incentives. The details of all the schemes are included in the paper (it’s a long list, and some of them are kind of involved), but it’s worth giving some examples.

On the financial end of the incentives spectrum, we had one condition we called “reward-accuracy,” which was pretty much what you’d expect: we told workers, “we’ll pay you a bonus if you get the answers right.” We also had one called “punishment-accuracy,” the gist of which you can deduce. On the purely social-psychological side, we had one we called “trust,” in which we told workers, “we’ll pay you for this job no matter how bad your performance, we trust that you’ll still make your best effort.”

One of the weirdest schemes turns out to be important, so I need to explain that one. Called “Bayesian Truth Serum” (BTS), it incorporates a design from the work of Drazen Prelec, a behavioral economist at MIT, who realized that research subjects could probably provide useful information regarding the expected distribution for subjective, qualitative questions (nb, the mechanics of how he does this are arcane in a way that is almost sure to delight the geeks among you, so I encourage you to read his paper). Few of the details of real BTS are important, except that we incorporated the piece about asking workers to answer the questions themselves and predict the distribution of other workers’ responses. We also told them we’d give them a bonus if their predictions were correct.

We then created a task that asked workers to answer five questions. In this case, the questions were drawn from another study examining participatory features of websites, for which we already possessed validated data collected by research assistants.

All workers answered the same five questions about the same website (www.kiva.org) while being exposed to one and only one of the 14 incentive schemes (or a control condition of no scheme). Roughly 2,000 individuals participated in the study, resulting in over 100 subjects in each of the experimental conditions. (The statistics and science nerds out there will be pleased to know that both the drop-out rate and demographic covariates were distributed evenly across conditions.)

To measure worker performance, we used the research assistant responses as correct answers to the questions and then calculated the total number of matching answers (out of five) provided by each worker. The results (aggregated across all treatments) are plotted in a histogram below and show that the average worker answered just over two questions out of five correctly.

Then, in order to see how the treatments compared against each other relative to the control group, we calculated the mean correct response rate for each condition and conducted difference of means tests to see which of these means were significantly greater than the control group. The results of this comparison appear below (in a new plot that doesn’t even appear in the paper!):

The orange dots show the value of the mean in each condition, and the blue bars illustrate the 95% confidence interval around that mean. The treatments are sorted by the size of the difference in means from the control. (More hard-core nerd stuff: the means are adjusted using Intent-To-Treat estimators).

From these results, we concluded that our horse race had two clear front-runners: the “Bayesian Truth Serum” (BTS) and “Punishment – disagreement” conditions, each of which improved average worker performance by almost half of a correct answer above the 2.08 correct answers in the control group. A few of the other financial and hybrid incentives had fairly large point estimates, but were not significantly different from control once we adjusted the test statistics and corresponding p-values to account for the fact that we were making so many comparisons at once (apologies if this doesn’t make sense — it’s yet another precautionary measure to avoid upsetting the stats nerds among you). In a tough turn for the sociologists and psychologists, none of the purely social/psychological treatments had any signficant effects at all.

Why do BTS and punishing workers for disagreement succeed in improving performance significantly where so many of the other incentive schemes failed? The answer hinges on the fact that both conditions tied workers’ payoffs to their ability to think about their peers’ likely responses. (We elaborate on the argument in more detail in the paper.)

Does this mean that we should give up on simple financial or social-psychological incentives? Probably not. The fact that we conducted the experiment on MTurk means that the deck may have been stacked against incentives like the “trust” condition I described earlier. Because requesters on MTurk have little oversight, workers are more likely to respond to financial incentives than stated promises. In this sense, the marketplace has structured the interaction between workers and requesters in a way that may limit the opportunities to harness motivations that are not linked to money in some explicit way.

You can download the full paper to read more.

Motivation in the workplace. Created by user: pescatello on flickr and licensed cc-by 2.0

A few weeks ago, I was talking about the motivations of crowdsourcing workers with Judd, who has already done a ton of great work looking at motivations for participation across a wide range of online environments. He is a recent Ph.D. from the UC Berkeley School of Information and just joined Yahoo! Research as a social psychologist and research scientist in the Internet Experiences Group, so it was no surprise that he had a great idea about how to design an experiment to better understand crowdsourcing.

The most straightforward way to ask crowdsourcing workers why they do what they do is with a survey (e.g., Panos Ipeirotis’ fascinating recent informal survey of MTurk workers.) However, you also might recall from one or two of my previous posts that I tend not to take survey results at face value.

Judd’s “list experiment” presents the subjects of a study with a list of several motivations and asks them to provide a count of the number of items in the list they agree with (rather than posing yes/no questions or checkboxes).

Here’s what that looked like once Judd had it set up in Crowdflower:

A screenshot from one version of our list experiment

We presented experimental treatment groups with four other permutations of the same list — each one missing one of the items — and aggregated the results across every group. This allowed us to estimate the proportion of respondents choosing each item in the list.

The advantage of the list experiment over the traditional survey format is that it doesn’t require anybody to explicitly say, “I crowdsource because it gives me a sense of purpose.” Indeed, it perfectly preserves the anonymity of individual user preferences, since the results that we generate are estimates based on summaries of behavior across the different treatment groups. The questions are less obtrusive and there’s no pressure to hide your true sentiments or conform to the expectations of others. List experiments are thus amazing tools to examine preferences that may be controversial or otherwise influenced by social pressures in some way.

Judd and I designed a pilot experiment with the list above and administered it to MTurk workers through Crowdflower. For the sake of comparison, we also included a control condition that asked Turkers the same questions in traditional, agreement-style survey form. To simplify things, we limited the responses to US workers only.

Comparing the results from the survey condition and the list experiment revealed some mind-blowing differences:

Note the discrepancy between some of the paired bars. Whereas 97% of the Turkers in the control group agreed with the statement “I am motivated to do HITs on Mechanical Turk to make extra money,” just 60% of the Turkers in the list experiment condition expressed the same preference.

Similarly, check out the difference between the agreement-style questions and list experiment results in the “for fun” category. Again, agreement statements elicit over-reporting when compared with the list experiment (although this time to a less extreme degree).

Our preliminary conclusions from this pilot study? The ideas of crowdsourcing for money and crowdsourcing for fun sound better than they actually are.

Another, slightly more science-y way to put this is that the workers in our study over-report the extent to which they are motivated by money and fun in response to agreement statements versus a list experiment, suggesting that they perceive these two factors to be socially desirable.

Understanding the cause of this social desirability bias as well as its implications for crowdsourcing across different environments will require further research. In other contexts, social desirability bias (a.k.a. “the Broadus effect”, if you read the amazing Nate Silver) has played a role in everything from elections to educational attainment. There’s no reason to believe it doesn’t affect the way people work and participate in various online environments as well.

Perhaps most interesting of all, our findings here further complicate the growing debate over how paid crowdsourcing ought to be understood and (potentially) regulated. If a substantial proportion of workers aren’t actually on MTurk for the money, does that support the claim that we should regulate crowdsourcing along the same lines that we regulate other post-industrial sectors?

These are big questions that we should continue to probe through future studies and discussion. In the meantime, Judd and I re-ran our list experiment with a few minor adjustments and a much bigger sample. We’re in the process of writing up this larger version of the study for a conference submission and will post the full paper here as soon as we can.

On one of those quiet, rainy afternoons that we call “Winter” here in San Francisco, I found myself in the Crowdflower offices with the legendary Mike Love.

Well, okay, maybe not that Mike Love. The other legendary Mike Love.

In any event, Mike and I were chatting about statistics and blogging (he’s exceptionally skilled at both) when he raised an interesting question: could you Crowdsource Benford’s Law?

Doe-eyed, mathematical neophyte that I am, neither Benford nor his law rang any bells, so I high-tailed it over to Wolfram Alpha MathWorld. There, in an article by Eric W. Weisstein, I encountered the following histogram:

Turns out that this simple, elegant, and purple plot describes the probability distribution of the first digit for many naturally occurring distributions. In other words, about 30% of the leading digits of the numbers in a large variety of tables or listings wil tend to be 1; 17.6% will tend to be 2; and so on down to around 4.6% of the numbers beginning with 9.

The phenomenon has an interesting history (at least as far as I could tell by going a few clicks past the Wikipedia article). It was originally described in an 1881 paper (subscription required) by Simon Newcomb, an astronomer who noticed that the first few pages in books of logarithm tables were worn much more heavily than later pages. The eponymous physicist Frank Benford then re-discovered the distribution in a 1938 paper where he tested it on a number of different data sources. More recently, the mathematician Ted Hill has provided a more sophisticated proof of the origins of the phenomenon and pioneered its use as a diagnostic to discover irregularities in data such as tax and elections returns (Tax season pro-tip: don’t fake a bunch of receipts that start with the number 9).

In any case, at Mike’s suggestion, I set out to see whether I could re-re-discover Benford’s Law with Crowdflower. The task I designed consisted of one question: “pick any number greater than zero.” In a little over an hour, I had about 500 valid responses and was off to the races to see what the resulting distribution of first digits looked like.

The results were all over the place. Here’s what they look like in a scatter plot with a log-scaled y-axis:

And here I plotted all of the numbers sorted by magnitude and leading digit:

The height of each tower of numbers captures the frequency of that leading digit. The size of the numerals corresponds to the magnitude of the number and the color corresponds roughly to its order of magnitude (red = big). Within each digit, the numbers are sorted along the Y-axis.

Even though it’s too small to really be legible, you can probably pick out a few interesting things from that graphic, such as the extraordinary frequency of the number 1 (it occurred, 67 times) or the size of the biggest number (2.345 x 10^13). The mean of all submissions was 4.72 x 10^10 while the median was 7. In other words, this little experiment resulted in a seriously skewed distribution.

But what about Benford’s Law? The height of the towers of numbers suggests that this data corresponded pretty well to the histogram at the beginning of my post. To supplement the visual, here’s a table showing the raw frequency and percentages by first digits (note, the first digits are at the top and are written out as words).

It’s still hard to tell how well that does or does not match up with the distribution in the histogram above. There are some nice statistical tests that could help us here, but for the sake of a decent blog post, I went in search of a better way to compare the distributions visually. With that in mind, another plot:

Here I superimposed the percentage values in table 1 (as brown dots) on a density curve of the original Benford’s Law values (shaded in blue). Then I also added a brown (loess) smoothed curve along with some gray confidence interval shading (thank you, Hadley Wickham) to capture the overall trend of the Crowdflower data. As you can see, the fit looks pretty good – I think Benford would be proud. Or at least maybe Mike Love. Maybe.

Admittedly, I may have cheated a bit in creating this second plot. Since the variable presented along the X axis (leading digit) consists of ordered categories, the smoothed lines are a bit of a representational stretch. Nevertheless, I’m satisfied with the way the resulting graphic visualizes the relationship between the probability function underlying Benford’s Law and the distribution of the Crowdflower responses. If anybody out there is inclined to pursue more interesting visualizations and/or rigorous tests to verify the fit, here’s a copy of the Crowdflower data I used.

I do all of my analysis and graphs in R using ggplot2. As always, feel free to send requests for code to: aaron [at] crowdflower [dot] com and be sure to tip your server.

What kinds of questions produce the best results in crowdsourcing tasks and surveys? To answer that question, I bring you another geeked-out blog post in which I pit the multiple choice (or forced choice) question against its bitter arch-rival, the check-all-that-apply (or checkbox) question.Both kinds of formatting can be useful when you want people to identify or categorize something(s) in a list. Check-all-that-apply seems to offer the added bonus of easily fitting an entire list into a single question, and thereby requiring less mental effort from respondents and (presumably) reducing response times. How do the two kinds of questions compare on answer quality, though?In my first post a few weeks ago, I talked about some of the reasons why response scales matter when you’re designing multiple choice questions for a survey or data collection task. In the comments, michael raised an interesting point:

Why use a scale at all? I would make those types of questions always open ended. Anyone who takes the survey has to think about how many hours they spend online anyway. That’s the first step. The second is fitting their estimate in one of the categories. Seems like unnecessary work for the participants.

You can check out the rest of the thread to see michael’s idea in context as well as how other people (including me) replied.The discussion got me thinking more about multiple choice questions and some of the costs and benefits that they entail in comparison to other types of questions. As luck would have it, a few of the other questions that I included in my original experiment can provide additional grist for the mill.

Question Format Smackdown!

In order to test how each format affects responses, I asked workers in the Crowdlabor pools one (and only one) version of the following:

As you can see, the forced choice version is a little clunky because I had to separate each item as a separate question. Nevertheless, there’s no substantive difference between the two versions other than the answer choice format, which makes it possible to compare the results.I should explain why I included several extremely popular websites (Google) among the answer options as well as some slightly less well-traveled, but still popular sites (Times of India, New York Times). Basically, this was in order to avoid too many people having visited all the sites or none of the sites. If a lot of responses fell into either extreme, it would have been impossible to estimate the extent to which the two formats affected the outcomes.As with my response scale example, the groups that saw the two versions of the question did not vary widely on potentially confounding demographic covariates such as gender or country of residence.

Here’s a table showing the number of positive responses per format per site:

And a plot to visualize the variations per site as a percentage of total responses per question format:

With one exception (Orkut), forced choice formatting resulted in more people saying they had visited every single site in the list.

Estimating the Effect

In order to get a precise measurement of the effect of forced choice format vs. checkbox format, I reshape the data into cumulative counts and compare the distributions of total number of sites visited among people who saw the checkbox and forced choice versions respectively. Here’s the resulting table:

A pair of density plots represents the same information in graphical form:

On each plot, I’ve highlighted the minimum, maximum, and mean (sparklines-style). The heavy left-leaning skew of the checkbox curve contrasts nicely with slightly right-leaning shape of the forced choice curve.From both the table and the density plots, it’s easy to see that the two question formats appear to have caused a substantial difference. The difference in means between the two distributions suggests that a respondent who saw the forced or multiple choice format identified (on average) one additional site they had visited which their peers who saw checkboxes did not identify.

Should I hate Checkboxes?

The demographic profile of the two groups was pretty similar, so the disparity in the results was almost certainly due to the question format. But why does the question format have such a powerful effect?Given the opportunity, checkbox respondents either failed to notice or ignored answer choices when they were not forced to provide a response to each one. As with numerical response scales, this is yet another example of how mental shortcuts can compromise data quality.This time around, a solution is pretty simple. All things being equal, you’re better off using forced choice formatting when you care about precise results. That said, things are never really equal and there will always be some reason you might want to consider making life faster/simpler for the people answering the questions. For example, if you’re asking people to choose tags or labels for something, the precision of each response might not matter very much and checkboxes would work just fine.

I used R for all the analysis and plots. I created the first plot using Hadley Wickham’ ggplot2 package. Contact me with requests for data or code at aaron [at] doloreslabs [dot] com and leave your questions, complaints, or suggestions below.

Jonathan Zittrain, Professor of Law and Faculty Co-director (and co-founder) of the Berkman Center for Internet and Society at Harvard University, gave a presentation at the Computer History Museum in Mountain View about a month ago that ought to be required viewing for anyone interested in Cloudlabor and Crowdsourcing.

Drawing examples from all over the Internet – including a certain iPhone app that you may have heard of – Zittrain raises some serious (and some seriously entertaining) questions about ethical and legal aspects of distributed human computing.

Straight from the Berkman Center YouTube channel, here’s the full video (which is also available for download under a Creative Commons Attribution 3.0 license from the President and Fellows of Harvard College:

I enjoyed his analysis and the discussion afterwards, although I suspect that some of the conversation with the audience might get lost in the video. As with Zittrain’s most recent book, The Future of the Internet and How to Stop It, this is some of the best thinking about life online that you’ll find anywhere.

Zittrain has also published an abbreviated portion of his argument in Newsweek under the slightly more extreme title “Work the New Digital Sweatshops.”

I find a lot of what Zittrain has to say compelling; however, I do wonder if the efforts of ReCaptcha-spammers and sock-puppeteers to exploit Crowdsourcing markets will ultimately prove successful. I also wonder whether the imposition of labor regulations in these contexts makes sense or would prove effective. Should my decision to kill time or make a few extra bucks by filtering images be subject to labor law? What about the ability of other people to offer money for distasteful and perhaps unethical (but usually not illegal) micro-tasks?

It may be a few years before anyone really understands if Crowdsourcing lends itself to unique types of market failure along these lines, but Zittrain and others such as Lily Irani and Aaron Koblin are doing us all a favor by asking some of the most important questions early in the game.

Full disclosure: the author of this post is affiliated with Harvard and the Berkman Center for Internet and Society, where he was a fellow during 2008-2009. While he doesn’t think that his affiliation influences his opinions about Zittrain’s work, it does mean that he’s very pleased not to be spending another winter in Cambridge this year.

What makes a bad survey question and why does it matter? I thought I’d use my first blog posts as Dolores Labs’s friendly neighborhood social scientist to talk a little bit about question design since it’s a relevant, but often overlooked, area of Crowdsourcing work.

You can ask “the crowd” all kinds of questions, but if you don’t stop to think about the best way to ask your question, you’re likely to get unexpected and unreliable results. You might call it the GIGO theory of research design.

To demonstrate the point, I decided to recreate some classic survey design experiments and distribute them to the workers in Crowdflower’s labor pools. For the experiments, every worker saw only one version of the questions and the tasks were posted using exactly the same title, description, and pricing. One hundred workers did each version of each question and I threw out the data from a handful of workers who failed a simple attention test question. The results are actual answers from actual people.

An Example: Response Scales

The rest of this post focuses on one example question that involved a response scale and a test to see how altering the scale would affect people’s answers. Here are two versions of the same question that I posted to Crowdflower:

Low Scale Version:


About how many hours do you spend online per day?

(a) 0 – 1 hour
(b) 1 – 2 hours
(c) 2 – 3 hours
(d) More than 3 hours

High Scale Version:


About how many hours do you spend online per day?

(a) 0 – 3 hours
(b) 3 - 6 hours
(c) 6 – 9 hours
(d) More than 9 hours

Notice that both versions can accommodate any answer and that the only difference is in the range of the scale items. You can give an accurate response to either question and neither version explicitly pushes you to give any answer over another.

So what did people say? Here’s a pair of histograms breaking the responses up by the two versions of the question:

I didn’t label the height of the bars because the results are almost useless in this form. The only conclusion we can draw is that a lot of people in the Crowdflower worker pool tend to spend more than three hours per day online (whoa, no way…).

At the same time, it seems like the workers might have given low answers more frequently in response the low scale (check out how big the first three blue bars are compared to just the first orange bar).

To look at that comparison more closely, let’s break the answers into two categories for each scale: (1) the percentage of responses that were less than three hours, or (2) the percentage of responses that were more than 3 hours.

The difference between the height of the orange points (high scale) is much bigger than the corresponding difference between the height of the blue points (low scale). In other words, people who saw the high scale were much more likely to say they spent more than 3 hours online. In case you’re a stats nerd, the Chi-square test showed that this variation was significant with a p-value < 0.001, so the difference was almost certainly not due to chance.

But maybe collapsing the responses like this is a little too coarse and you'd still like to see how the variation worked across the scale as a whole. With that in mind, Lukas suggested another way to look at the effects – a comparison of the cumulative percentage of responses – and the differences are even more clear.

That gap between the blue and the orange line at “Less than 3 hours” – the one level that was measured explicitly on both scales – is huge!

Explaining the Gap

If you’re thinking that the differences between the scales alone can’t explain why all of these results are so skewed, that’s a good thought. However, the fact that this was a randomized experiment on a relatively homogeneous group of people makes it very unlikely that anything else explains the difference. Just to be sure, I did some other tests and found no significant differences between the sets of respondents that saw the low and high scales in terms of gender, country of origin, and the amount of time they took to complete the survey. So it seems like the scale is indeed the most likely culprit.

But what explains why scale questions can bias people’s responses so heavily? Survey researchers call this kind of behavior satisficing – it happens when people taking a survey use cognitive shortcuts to answer questions. In the case of questions about personal behaviors that we’re not used to quantifying (like the time we spend online), we tend to shape our responses based on what we perceive as “normal.” If you don’t know what normal is in advance, you define it based on the midpoint of the answer range. Since respondents didn’t really differentiate between the answer options, they were more likely to have their responses shaped by the scale itself.

These results illustrate a sticky problem: it’s possible that a survey question that is distributed, understood, and analyzed perfectly could give you completely inaccurate results if the scale is poorly designed.

Okay, it’s Broken. Now How Do I fix It?

So what are you supposed to do in order to figure out which scale is more accurate? One of the best ways to mitigate the problem is to do some open-ended research on your respondent population so that you can get a good sense of a reasonable range of responses. Then you can re-center your response scale around that distribution.

To try this out, I ran the survey yet again with the same question, except that this time I left the “hours online” question open-ended, allowing Crowdflower workers to type in their responses. Here’s a density plot of those responses with the minimum, maximum, and mean responses highlighted (sparklines style):

While the distribution is skewed and has something of a long-ish tail, the mean (6.53 hours per day), median (6 hours per day), and mode (5 hours per day) are all close to the midpoint of the high scale in my original questions. Therefore, the responses from the high scale were probably a more accurate reflection of the worker’s judgments.

Keep in mind, this technique provides no guarantee that the workers have accurate knowledge of how many hours they spend online – it’s turtles all the way down. I’d be willing to bet that their best guesses are pretty good, but if a big policy decision was riding on this question, I’d try to supplement my little survey with some other data sources. No matter what, there’s no perfect solution.

So what?

The point of all this has not been to undermine survey research, but to illustrate some of the problems that can happen if you’re not careful with things like scale design, as well as to present some strategies for solving those problems. As crowdsourcing becomes a mainstream tool in a range of academic and commercial fields, survey and questionnaire design techniques are also becoming more widely applicable. Nevertheless, people don’t usually encounter this kind of stuff outside of research methodology textbooks and the polling season of an election year.

I have a few more examples from these same experiments that I hope to follow up with in more posts soon. Meanwhile, leave a comment or email me at aaron [at] doloreslabs [dot] com with questions, comments, corrections and requests for data/code. All of these plots were created using R.