In a recent paper, presented at the ACM Conference on Computer Supported Cooperative Work (CSCW), John Horton, Daniel Chen and I used a large-scale experiment to test the effect of different incentive schemes on the quality of crowdsourcing work.

The results surprised us. They suggest that workers perform most accurately when the task design credibly links payoffs to a worker’s ability to think about the answers that their peers are likely to provide.

a horserace experiment! (photo cc-by-sa by iyoupapa)

The idea for this study came out of our sense that, as social scientists, we had something unique to offer the existing research on human computation. Early and influential crowdsourcing research has focused on how to filter the judgments of the crowd to find the best answers. We wanted to know whether simple task-design changes could improve the quality of data coming into a crowdsourcing system in the first place.

To test this idea, we chose 14 different incentive schemes and framing techniques developed and validated across the social sciences and set up a horse race experiment to see which schemes/techniques would work best.

Consistent with our personal biases (John and Daniel are both economists, and I’m a sociologist), some of the schemes were financially oriented, some were social or psychological, and some were hybrids combining social and financial incentives. The details of all the schemes are included in the paper (it’s a long list, and some of them are kind of involved), but it’s worth giving some examples.

On the financial end of the incentives spectrum, we had one condition we called “reward-accuracy,” which was pretty much what you’d expect: we told workers, “we’ll pay you a bonus if you get the answers right.” We also had one called “punishment-accuracy,” the gist of which you can deduce. On the purely social-psychological side, we had one we called “trust,” in which we told workers, “we’ll pay you for this job no matter how bad your performance, we trust that you’ll still make your best effort.”

One of the weirdest schemes turns out to be important, so I need to explain that one. Called “Bayesian Truth Serum” (BTS), it incorporates a design from the work of Drazen Prelec, a behavioral economist at MIT, who realized that research subjects could probably provide useful information regarding the expected distribution for subjective, qualitative questions (nb, the mechanics of how he does this are arcane in a way that is almost sure to delight the geeks among you, so I encourage you to read his paper). Few of the details of real BTS are important, except that we incorporated the piece about asking workers to answer the questions themselves and predict the distribution of other workers’ responses. We also told them we’d give them a bonus if their predictions were correct.

We then created a task that asked workers to answer five questions. In this case, the questions were drawn from another study examining participatory features of websites, for which we already possessed validated data collected by research assistants.

All workers answered the same five questions about the same website (www.kiva.org) while being exposed to one and only one of the 14 incentive schemes (or a control condition of no scheme). Roughly 2,000 individuals participated in the study, resulting in over 100 subjects in each of the experimental conditions. (The statistics and science nerds out there will be pleased to know that both the drop-out rate and demographic covariates were distributed evenly across conditions.)

To measure worker performance, we used the research assistant responses as correct answers to the questions and then calculated the total number of matching answers (out of five) provided by each worker. The results (aggregated across all treatments) are plotted in a histogram below and show that the average worker answered just over two questions out of five correctly.

Then, in order to see how the treatments compared against each other relative to the control group, we calculated the mean correct response rate for each condition and conducted difference of means tests to see which of these means were significantly greater than the control group. The results of this comparison appear below (in a new plot that doesn’t even appear in the paper!):

The orange dots show the value of the mean in each condition, and the blue bars illustrate the 95% confidence interval around that mean. The treatments are sorted by the size of the difference in means from the control. (More hard-core nerd stuff: the means are adjusted using Intent-To-Treat estimators).

From these results, we concluded that our horse race had two clear front-runners: the “Bayesian Truth Serum” (BTS) and “Punishment – disagreement” conditions, each of which improved average worker performance by almost half of a correct answer above the 2.08 correct answers in the control group. A few of the other financial and hybrid incentives had fairly large point estimates, but were not significantly different from control once we adjusted the test statistics and corresponding p-values to account for the fact that we were making so many comparisons at once (apologies if this doesn’t make sense — it’s yet another precautionary measure to avoid upsetting the stats nerds among you). In a tough turn for the sociologists and psychologists, none of the purely social/psychological treatments had any signficant effects at all.

Why do BTS and punishing workers for disagreement succeed in improving performance significantly where so many of the other incentive schemes failed? The answer hinges on the fact that both conditions tied workers’ payoffs to their ability to think about their peers’ likely responses. (We elaborate on the argument in more detail in the paper.)

Does this mean that we should give up on simple financial or social-psychological incentives? Probably not. The fact that we conducted the experiment on MTurk means that the deck may have been stacked against incentives like the “trust” condition I described earlier. Because requesters on MTurk have little oversight, workers are more likely to respond to financial incentives than stated promises. In this sense, the marketplace has structured the interaction between workers and requesters in a way that may limit the opportunities to harness motivations that are not linked to money in some explicit way.

You can download the full paper to read more.

Writing is easy. Just sit in front of a typewriter, open up a vein and bleed it out drop by drop.
– Red Smith

Underwood Five Typewriter

When I was in college, a professor I respected said that one of the best ways to demystify writing is to write like people you admire. Specifically, to find passages that you love, and try to revise them in your own words. This exercise proved invaluable. It allowed me to walk in their literary footsteps, shedding light on why they chose — or avoided — certain words, punctuation, and syntax.

With this in mind, I recently wondered whether you can crowdsource writing, specifically, revising.

I posted a task through CrowdFlower that asked the crowd to rewrite four famous quotations, pithily, while preserving their meaning.

In one evening, I was able to get 20 revisions of each quotation from people across the country. I won’t summarize them all here, but I will pull a few highlights.

Original Quotation 1 (from Macbeth, William Shakespeare):

To-morrow, and to-morrow, and to-morrow,/ Creeps in this petty pace from day to day,/ To the last syllable of recorded time;/ And all our yesterdays have lighted fools/ The way to dusty death. Out, out, brief candle!/ Life’s but a walking shadow, a poor player/ That struts and frets his hour upon the stage/ And then is heard no more. It is a tale/ Told by an idiot, full of sound and fury/ Signifying nothing.

Revision 1:

Time marches on, and everyone dies; life is meaningless.
– Hatboro, PA

Life creeps along and ends suddenly like the end of a bad play. The play is dramatic and had poor acting, and has no point or moral in the end.
–Salt Lake City, UT

Original Quotation 2 (from “The Leviathan,” Thomas Hobbes):

No arts, no letters, no society, and which is worst of all, continual fear and danger of violent death, and the life of man solitary, poor, nasty, brutish, and short.

Revision 2:

The life of a man on his own is barbaric and degrading.
– East Aurora, NY

No art, letters or society. Worst of all, living in fear of being alone, poor and short.
– Arlington, TX

Life is all but a mere scam.
–Overland Park, KS

Original Quotation 3 (“First Inaugural Address,” Abraham Lincoln):

We are not enemies, but friends. We must not be enemies. Though passion may have strained it must not break our bonds of affection. The mystic chords of memory, stretching from every battlefield and patriot grave to every living heart and hearthstone all over this broad land, will yet swell the chorus of the Union, when again touched, as surely they will be, by the better angels of our nature.

Revision 3:

We’re fools for fighting each other. We should co-operate, and let our example bring everyone together.
– Plattsburgh, NY

We must be friends, we should now forget each other. Our memories will always stay through our rough times and good times.
– Tallahassee, FL

Original Quotation 4 (from “Chicago,” Carl Sandburg):

“They tell me you are wicked and I believe them, for I have seen your painted women under the gas lamps luring the farm boys.”

Revision 4:

Your reputation follows you and it is a bad one, makeup and lust.
– Iola, WI

This city isn’t a nice place, it’s full of hookers.
– Milledgeville, GA

The full data is available here.

What’s your opinion? Can the revision process be crowdsourced?

I’d love to hear your thoughts.

As the NFL season loomed on the horizon back in August, there was a lot of fantasy football talk around the CrowdFlower office. Naturally, we decided to crowdsource a killer fantasy football team.

Fantasy football tackle. Photo Credit: dennis

We asked CrowdFlower workers to help us build a ranked list of the Top 75 players to guide our fantasy football draft. We used pair-wise comparisons to determine rankings. Specifically, we presented workers with two players, asking them to pick which player they thought would be more valuable.

We used the Top 75 players from ESPN’s 2010 Fantasy Football Draft Kit¹, matching each player with his 74 counterparts, giving us a total of 2,775 player pairs. We also included each player’s position and team, as well as a link to more detailed statistics.

Screenshot of fantasy football task.

After the job finished, we ordered the players according to number of head-to-head victories.

Crowdsourced list of the Top 25 most valuable fantasy football players.

We found that workers choose the player on the left 53 percent of the time, even though each match-up appears twice, once with Player A on the left and again with Player A on the right. With 5,500 data points, this is significant bias toward the player on the left. However, the final ranking doesn’t change even after accounting for this bias.

We settled on two likely explanations for the bias:

1. The anchoring effect of putting something on the left (and as the first response option) may have caused more workers to select the first player.
2. Our Gold was slightly biased toward the player on the left, which we have previously seen² to have an effect on the overall distribution of the answers.

As a first comparison of our crowd of football fans with ESPN’s Fantasy Football brain trust, we plotted each player by the difference between his two rankings.

Players on the left were ranked higher by ESPN, while players on the right were ranked higher by AMT.

Will Chad Ochocinco and Matt Ryan vindicate the crowd? Stay tuned to find out.

Marcel Proust

A few months ago, I started a blog written by Mechanical Turk workers. Eight times a day, on the hour, a script posts a personal question on MTurk. The questions are randomly selected from a subset of the Proust Questionnaire. Example questions include:

1. What is your favorite food and drink?
2. What is your idea of misery?
3. Which natural talent would you most like to be gifted with?

The first worker to submit his or her answer automatically has their response anonymously posted on the blog, which I call “Mechanical Proust.” Workers are told that their responses will made public, but remain anonymous. I make no attempt to check responses before they are posted, nor do I check whether a worker has submitted in the past. (I enabled Google Ads to see if I could make the blog financially self-sustaining. No luck here so far.)

What have I learned? Most of the responses are unsurprising but they are occasionally poignant or insightful. People are proud of their children. They regret dropping out of school. They want to live in Paris. They fear dying and being alone. They like chicken dishes, etc. Lots of workers are funny (see screenshot at left).

Technical Notes:

A Python script posts the questions (using boto), and cron schedules them. The questions are presented as external HITs, with the “submit” button launching another Python script that posts the text response to the blog, using Google’s API. If you want the code or want help making something similar, let me know.

Next steps (taken by someone else):

I’d like to see someone make a new version of this blog where blog readers could submit questions, which would go into a queue, with newer questions going at the bottom. Each new question would have up/down Reddit-type buttons, which could move questions up and down the stack. Questions could then be posted on MTurk LIFO-style.

When you were a kid, did you ever want to be in a famous band? Now you can be part of the next best thing … and you can even tell your younger self about it.

Arcade Fire has collaborated with Google and Chris Milk to set up a crowdsourced art project that anyone with a computer can contribute to and even curate, from anywhere in the world.

Here’s how it works:

By visiting “The Wilderness Downtown” (best viewed in Google Chrome) and entering your childhood address, you can travel down memory lane through the streets you grew up in, accompanied by Arcade Fire’s “We Used To Wait.”

You can then send your childhood self a note, a drawing, or other art, which gets incorporated into your personalized art project. If you like what you created, you can submit it to Arcade Fire, where it may end up as part of their concert tour.

Every submission to this crowdsourced art project will be curated in real time by CrowdFlower, whose workforce sifts through the messages and identifies stellar — or offensive — content.

Prior to crowdsourcing, a global art project like this (not to mention real-time curation and content moderation by human beings) would have been unthinkable. It’s yet another glimpse of the future you can send to the childhood you.

Please visit http://thewildernessdowntown.com/ to view and contribute to the project. And if you want to help curate, visit http://crowdflower.com/judgments/mob/20094.

Motivation in the workplace. Created by user: pescatello on flickr and licensed cc-by 2.0

A few weeks ago, I was talking about the motivations of crowdsourcing workers with Judd, who has already done a ton of great work looking at motivations for participation across a wide range of online environments. He is a recent Ph.D. from the UC Berkeley School of Information and just joined Yahoo! Research as a social psychologist and research scientist in the Internet Experiences Group, so it was no surprise that he had a great idea about how to design an experiment to better understand crowdsourcing.

The most straightforward way to ask crowdsourcing workers why they do what they do is with a survey (e.g., Panos Ipeirotis’ fascinating recent informal survey of MTurk workers.) However, you also might recall from one or two of my previous posts that I tend not to take survey results at face value.

Judd’s “list experiment” presents the subjects of a study with a list of several motivations and asks them to provide a count of the number of items in the list they agree with (rather than posing yes/no questions or checkboxes).

Here’s what that looked like once Judd had it set up in Crowdflower:

A screenshot from one version of our list experiment

We presented experimental treatment groups with four other permutations of the same list — each one missing one of the items — and aggregated the results across every group. This allowed us to estimate the proportion of respondents choosing each item in the list.

The advantage of the list experiment over the traditional survey format is that it doesn’t require anybody to explicitly say, “I crowdsource because it gives me a sense of purpose.” Indeed, it perfectly preserves the anonymity of individual user preferences, since the results that we generate are estimates based on summaries of behavior across the different treatment groups. The questions are less obtrusive and there’s no pressure to hide your true sentiments or conform to the expectations of others. List experiments are thus amazing tools to examine preferences that may be controversial or otherwise influenced by social pressures in some way.

Judd and I designed a pilot experiment with the list above and administered it to MTurk workers through Crowdflower. For the sake of comparison, we also included a control condition that asked Turkers the same questions in traditional, agreement-style survey form. To simplify things, we limited the responses to US workers only.

Comparing the results from the survey condition and the list experiment revealed some mind-blowing differences:

Note the discrepancy between some of the paired bars. Whereas 97% of the Turkers in the control group agreed with the statement “I am motivated to do HITs on Mechanical Turk to make extra money,” just 60% of the Turkers in the list experiment condition expressed the same preference.

Similarly, check out the difference between the agreement-style questions and list experiment results in the “for fun” category. Again, agreement statements elicit over-reporting when compared with the list experiment (although this time to a less extreme degree).

Our preliminary conclusions from this pilot study? The ideas of crowdsourcing for money and crowdsourcing for fun sound better than they actually are.

Another, slightly more science-y way to put this is that the workers in our study over-report the extent to which they are motivated by money and fun in response to agreement statements versus a list experiment, suggesting that they perceive these two factors to be socially desirable.

Understanding the cause of this social desirability bias as well as its implications for crowdsourcing across different environments will require further research. In other contexts, social desirability bias (a.k.a. “the Broadus effect”, if you read the amazing Nate Silver) has played a role in everything from elections to educational attainment. There’s no reason to believe it doesn’t affect the way people work and participate in various online environments as well.

Perhaps most interesting of all, our findings here further complicate the growing debate over how paid crowdsourcing ought to be understood and (potentially) regulated. If a substantial proportion of workers aren’t actually on MTurk for the money, does that support the claim that we should regulate crowdsourcing along the same lines that we regulate other post-industrial sectors?

These are big questions that we should continue to probe through future studies and discussion. In the meantime, Judd and I re-ran our list experiment with a few minor adjustments and a much bigger sample. We’re in the process of writing up this larger version of the study for a conference submission and will post the full paper here as soon as we can.

This is a guest post written by my friend Dana Chandler on how the context of a task motivates the person working on it.  He has a longer academic paper on the topic you can find at the bottom of this post.  It once again shows how traditional economic incentives can’t fully explain workers’ behaviors on Mechanical Turk.

Imagine for a moment that you were a turker from either the US or India, looking at the above image. You are given the task of clicking on the blue circular objects with red borders. What you see is only a fraction of the full image. Each image has 90 blue objects to identify. If you’re as good as the average worker, you’ll complete your first image in a little over five minutes and you’ll earn 10 cents (for an hourly wage of $1.20).

After your first image, you can either quit and take your 10 cents, or identify points on another image. Over the next four hours, you’ll have the chance to label as many images you want. But there’s a catch—you’ll only be paid 9 cents for the second image, 8 cents for the third, and so on, all the way down to 2 cents. This will lower the hourly wage even more.

Before you even qualify for the task, you’ll have to spend five minutes watching a training video and passing a quiz. During the video, half of you will be given only basic work instructions on how to identify “objects of interest.” The other half will be given both instructions and cues of meaning: recognition for your contribution and an explanation of your task’s purpose¹. The reason given here? To help researchers identify cancerous tumor cells.

We posted these HITs on MTurk in January, 2010. Almost 300 people from the U.S. and India accepted the task, becoming unknowing participants in our experiment examining MTurk worker motivation. It is commonly believed (and other researchers have verified with demographic surveys) that Indian workers are more motivated by pecuniary concerns and that US turkers are primarily doing tasks for leisure or other non-pecuniary motives. Is this true?

In both countries, half of the turkers in the experiment were randomly assigned to label nondescript “objects of interest” without being given any context or greater purpose — they were our zero-context group. The other half, our meaningful group, were told they were helping researchers identify cancerous tumor cells. Which group of turkers do you think worked harder? You might be surprised.

Therefore, our experiment compared two groups with and without a clear wage motivation, to see if workers behave differently responded to meaningfulness in their tasks.

Results

We measured three metrics: “showing up”, the quantity of work, and the quality of that work. The first two metrics are straightforward. Showing up meant that you sat through our training video, passed our qualification test and helped label at least one image. Quantity of work was simply the number of images labeled.

We repeatedly told both groups of turkers that they needed to click on all points and as closely as possible to each point. Work quality was determined by the fraction of cells that a person clicked on (the recall) and the average distance between the “true center” of each cell and where the user clicked (the centrality).

Our most interesting finding was the extent to which a meaningful task (and giving recognition) motivated US workers, but not Indian workers, to complete a task. As any requester knows, attrition on MTurk is a real problem. We found that adding cues of meaning could motivate turkers to undergo training and label at least one image. In the US, adding cues-of-meaning raised the fraction of turkers who completed our task from, 92% of people who sat through our training video, took our quiz, and labeled an image showed up. This figure compares to only 83% of zero-context group (see figure which also has standard errors). In India, there was no difference between the groups and both groups had a 66% completion rate (attrition being higher due to possible language barriers, slow connection speeds, hardware issues, etc.).

However, once a person did some work, both treatment and control groups did a similar quantity of work: The cues-of-meaning group labeled 6.0 images and the zero-context group labeled 5.7 images. This difference was not statistically significant, so it suggests that once you get turkers to work on a task, they are motivated to label just as many images irrespective of the task’s meaningfulness. Notably, of the people who worked, Indians worked longer and labeled an average of 7.3 images vs. 5.2 in the US.

Surprisingly, all workers did an equally good job identifying points whether they had zero-context or whether they thought they were identifying tumor cells. The quality as measured by the fraction of points identified (the recall) or the average pixel distance (the centrality) was statistically insignificant irrespective of the task’s meaningfulness.

This finding has important implications for those who employ labor in crowdsourcing markets. Companies and intermediaries should develop an understanding of what motivates the people who work on tasks. Employers must think beyond monetary incentives and consider how they can reward workers through non-monetary incentives such as by changing how workers perceive their task. Alienated workers are less likely to do work if they don’t know the context of the work they are doing and employers may find they can get more work done for the same wages simply by telling turkers why they are working.

For more details of this study, please see our full academic paper at: http://danachandler.com/index.php/research. We welcome any comments and feedback.

About the authors:
Dana Chandler is a researcher at the University of Chicago’s Becker Center where he works with Steven Levitt, author of Freakonomics. He previously worked as a management consultant at the Boston Consulting Group and at Aureos Capitol, a Colombian private equity company. He will begin his Ph.D. at MIT in the Fall.  Dana’s research interests include digital labor markets, development economics, and randomized experiments in companies. email: dchandler {at} uchicago {dot} edu

Adam Kapelner is currently earning his Ph.D. in Statistics at Wharton. Adam is the founder of  dictionarysquared.com and the inventor of its vocabulary-learning  technology. While working as an undergraduate researcher at Stanford University, he helped engineer the open-source software, www.gemIdent.com, that enables researchers worldwide to locate cells in microscopic images. GemIdent was recently extended to make use of MTurk for outsourcing of medical image identification. The extension, called www.distributeeyes.com, was  adapted to serve as the platform for this experiment. email: kapelner  {at} wharton {dot} upenn {dot} edu

Acknowledgments: We thank Professor Susan Holmes of Stanford University for allowing us to adopt DistributeEyes (funded under NIH grant #R01GM086884-02) for use in this study. We would also like to thank Panos Ipeirotis for kindly providing us with demographic and market data that we cite in our study. Lawrence Brown, Patrick DeJarnette, John Horton, Emir Kamenica, Steven Levitt, Susanne Neckermann, Jesse Shapiro, Jorg Spenkuch, Jan Stoop, Chad Syverson, Mike Thomas, Abraham Wyner, and seminar participants at the University of Chicago provided especially helpful comments.

The distributed distributed work meetup was this past Monday, and it would be an injustice to not have a blog post on the workers who make distributed work possible. Over the weekend we decided we would rerun and reexamine Panos Ipeirotis’s survey of turkers. Panos, by the way, has a great blog on crowdsourcing, Amazon Mechanical Turk, and other interesting topics. I highly recommend reading it for many Turk related experiments and studies.

We used mostly the same questions as Panos’s survey, which asked for:

• the Turker’s age (year of birth)
• gender
• educational level
• income level
• marital status
• questions about their engagement on Turk
• how often they Turk
• income from Turk
• why they Turk

In contrast to Panos’s original survey which was run over a 3 week period, we ran this survey over the weekend in a 24 hour period. Due to this abbreviated weekend running, there will most likely be greater confounding factors as well as stronger selection bias, i.e. groups who work on weekends as opposed to during the week. To help mitigate the timing of the experiment and provide surveys close to uniformly over the 24 hours, we setup a script to release only 50 surveys an hour. Responses generally came at a steady pace and Turkers available at each hour were represented.

Since we’ve examined Turker motivation before, albeit hardly rigorously, I will mostly focus on the rise of India as well other location specific considerations in this post.

In comparison to Panos’s survey the greatest difference in the results was in the distribution of respondents’ location. We found that India made up 46.85% of our respondents, while the US made up 42.7%. In contrast, Panos’s survey (run over a 3 week period in February) had 46.8% of respondents from the US, and 34.0% were from India. To determine if this difference was due to self reporting error, we checked self reported location vs geocoded IP location and found that they matched almost exactly. There were 31 differences out of 1016 survey responses, but these differences could potentially be attributed to ambiguity in the question (“Where are you from?”). Ultimately these differences between self report location and our geocoded IP location were not statistically significant. This is an encouraging result, suggesting Turkers are overwhelmingly honest when answering survey questions about where they are from.

Viewing the responses over time suggests that Turkers work during non-sleeping hours. The graph below shows this pattern. If we were to repeat this experiment we’d want to run this task over a week or two instead of over 24 hours. Because of this abbreviated time period the pattern is not as evident towards the end of the job where work rates declined across the board.

To better test the hypothesis that Turkers generally work during non-sleeping hours of their respective countries, below is a graph showing the distribution of work done for CrowdFlower on Mechanical Turk each hour of a day by continent (specifically Asia, Europe, and the “Americas”) over the course of 6 months. In this 6 months we collected approximately 9 million judgments. In the graph this means if Asia has a point at 6%, 4 AM GMT then 6% of judgments made in Asia came around 4AM GMT.

For each continent the peaks roughly correspond to daytime while the valleys to nighttime, which is what we’d expect. Asia’s peak seems to be more of a plateau, and this is likely due to in part to the number of timezones Asia encompasses. In my post about task localization, we saw (what is intuitively obvious) that workers’ locales are an important factor in assessing quality, especially for language specific tasks. On Mechanical Turk, to hit the right workforce for a language specific task, it is advisable to restrict available hits to certain times to limit the number of responses from countries whose native languages are not applicable. We have to use this round about method for Mechanical Turk because you cannot restrict the workforce to a set of multiple countries, either restrict work to one country or restrict work to all but one country.

We’ve already noted that India represents a sizable and rapidly growing portion of the Turk workforce. We are particularly interested in this rate of growth and the future trends in worker locales. The next graph compares the US to India in terms of monthly volume of judgments completed on our jobs posted to Mechanical Turk in 2010. The location information comes from geocoded IP adresses.

The above shows that on Mechanical Turk we’ve seen an increase in the proportion of our workers who are Indian since December. This sample was collected over a relatively short period of time, and is definitely something we’ll want to monitor in the future. Lastly, I want to emphasize that though this experiment is hardly rigorous and there are many more factors to analyze, Mechanical Turk as well as other vendors of work (Gambit, Samasource, LiveOps, etc.) are continuing to evolve and grow extremely rapidly, and consequently so grows the potential and possibilites for distributed work. Next we’ll examine a survey of Gambit, then Samasource.

John

What kinds of questions produce the best results in crowdsourcing tasks and surveys? To answer that question, I bring you another geeked-out blog post in which I pit the multiple choice (or forced choice) question against its bitter arch-rival, the check-all-that-apply (or checkbox) question.Both kinds of formatting can be useful when you want people to identify or categorize something(s) in a list. Check-all-that-apply seems to offer the added bonus of easily fitting an entire list into a single question, and thereby requiring less mental effort from respondents and (presumably) reducing response times. How do the two kinds of questions compare on answer quality, though?In my first post a few weeks ago, I talked about some of the reasons why response scales matter when you’re designing multiple choice questions for a survey or data collection task. In the comments, michael raised an interesting point:

Why use a scale at all? I would make those types of questions always open ended. Anyone who takes the survey has to think about how many hours they spend online anyway. That’s the first step. The second is fitting their estimate in one of the categories. Seems like unnecessary work for the participants.

You can check out the rest of the thread to see michael’s idea in context as well as how other people (including me) replied.The discussion got me thinking more about multiple choice questions and some of the costs and benefits that they entail in comparison to other types of questions. As luck would have it, a few of the other questions that I included in my original experiment can provide additional grist for the mill.

Question Format Smackdown!

In order to test how each format affects responses, I asked workers in the Crowdlabor pools one (and only one) version of the following:

As you can see, the forced choice version is a little clunky because I had to separate each item as a separate question. Nevertheless, there’s no substantive difference between the two versions other than the answer choice format, which makes it possible to compare the results.I should explain why I included several extremely popular websites (Google) among the answer options as well as some slightly less well-traveled, but still popular sites (Times of India, New York Times). Basically, this was in order to avoid too many people having visited all the sites or none of the sites. If a lot of responses fell into either extreme, it would have been impossible to estimate the extent to which the two formats affected the outcomes.As with my response scale example, the groups that saw the two versions of the question did not vary widely on potentially confounding demographic covariates such as gender or country of residence.

Here’s a table showing the number of positive responses per format per site:

And a plot to visualize the variations per site as a percentage of total responses per question format:

With one exception (Orkut), forced choice formatting resulted in more people saying they had visited every single site in the list.

Estimating the Effect

In order to get a precise measurement of the effect of forced choice format vs. checkbox format, I reshape the data into cumulative counts and compare the distributions of total number of sites visited among people who saw the checkbox and forced choice versions respectively. Here’s the resulting table:

A pair of density plots represents the same information in graphical form:

On each plot, I’ve highlighted the minimum, maximum, and mean (sparklines-style). The heavy left-leaning skew of the checkbox curve contrasts nicely with slightly right-leaning shape of the forced choice curve.From both the table and the density plots, it’s easy to see that the two question formats appear to have caused a substantial difference. The difference in means between the two distributions suggests that a respondent who saw the forced or multiple choice format identified (on average) one additional site they had visited which their peers who saw checkboxes did not identify.

Should I hate Checkboxes?

The demographic profile of the two groups was pretty similar, so the disparity in the results was almost certainly due to the question format. But why does the question format have such a powerful effect?Given the opportunity, checkbox respondents either failed to notice or ignored answer choices when they were not forced to provide a response to each one. As with numerical response scales, this is yet another example of how mental shortcuts can compromise data quality.This time around, a solution is pretty simple. All things being equal, you’re better off using forced choice formatting when you care about precise results. That said, things are never really equal and there will always be some reason you might want to consider making life faster/simpler for the people answering the questions. For example, if you’re asking people to choose tags or labels for something, the precision of each response might not matter very much and checkboxes would work just fine.

I used R for all the analysis and plots. I created the first plot using Hadley Wickham’ ggplot2 package. Contact me with requests for data or code at aaron [at] doloreslabs [dot] com and leave your questions, complaints, or suggestions below.