Motivation in the workplace. Created by user: pescatello on flickr and licensed cc-by 2.0
A few weeks ago, I was talking about the motivations of crowdsourcing workers with Judd, who has already done a ton of great work looking at motivations for participation across a wide range of online environments. He is a recent Ph.D. from the UC Berkeley School of Information and just joined Yahoo! Research as a social psychologist and research scientist in the Internet Experiences Group, so it was no surprise that he had a great idea about how to design an experiment to better understand crowdsourcing.
The most straightforward way to ask crowdsourcing workers why they do what they do is with a survey (e.g., Panos Ipeirotis’ fascinating recent informal survey of MTurk workers.) However, you also might recall from one or two of my previous posts that I tend not to take survey results at face value.
Judd’s “list experiment” presents the subjects of a study with a list of several motivations and asks them to provide a count of the number of items in the list they agree with (rather than posing yes/no questions or checkboxes).
Here’s what that looked like once Judd had it set up in Crowdflower:
A screenshot from one version of our list experiment
We presented experimental treatment groups with four other permutations of the same list — each one missing one of the items — and aggregated the results across every group. This allowed us to estimate the proportion of respondents choosing each item in the list.
The advantage of the list experiment over the traditional survey format is that it doesn’t require anybody to explicitly say, “I crowdsource because it gives me a sense of purpose.” Indeed, it perfectly preserves the anonymity of individual user preferences, since the results that we generate are estimates based on summaries of behavior across the different treatment groups. The questions are less obtrusive and there’s no pressure to hide your true sentiments or conform to the expectations of others. List experiments are thus amazing tools to examine preferences that may be controversial or otherwise influenced by social pressures in some way.
Judd and I designed a pilot experiment with the list above and administered it to MTurk workers through Crowdflower. For the sake of comparison, we also included a control condition that asked Turkers the same questions in traditional, agreement-style survey form. To simplify things, we limited the responses to US workers only.
Comparing the results from the survey condition and the list experiment revealed some mind-blowing differences:
Note the discrepancy between some of the paired bars. Whereas 97% of the Turkers in the control group agreed with the statement “I am motivated to do HITs on Mechanical Turk to make extra money,” just 60% of the Turkers in the list experiment condition expressed the same preference.
Similarly, check out the difference between the agreement-style questions and list experiment results in the “for fun” category. Again, agreement statements elicit over-reporting when compared with the list experiment (although this time to a less extreme degree).
Our preliminary conclusions from this pilot study? The ideas of crowdsourcing for money and crowdsourcing for fun sound better than they actually are.
Another, slightly more science-y way to put this is that the workers in our study over-report the extent to which they are motivated by money and fun in response to agreement statements versus a list experiment, suggesting that they perceive these two factors to be socially desirable.
Understanding the cause of this social desirability bias as well as its implications for crowdsourcing across different environments will require further research. In other contexts, social desirability bias (a.k.a. “the Broadus effect”, if you read the amazing Nate Silver) has played a role in everything from elections to educational attainment. There’s no reason to believe it doesn’t affect the way people work and participate in various online environments as well.
Perhaps most interesting of all, our findings here further complicate the growing debate over how paid crowdsourcing ought to be understood and (potentially) regulated. If a substantial proportion of workers aren’t actually on MTurk for the money, does that support the claim that we should regulate crowdsourcing along the same lines that we regulate other post-industrial sectors?
These are big questions that we should continue to probe through future studies and discussion. In the meantime, Judd and I re-ran our list experiment with a few minor adjustments and a much bigger sample. We’re in the process of writing up this larger version of the study for a conference submission and will post the full paper here as soon as we can.
mark
08/10/10
GREAT post and couldn’t agree more. While compensation is critical. it’s amazing and really nice to see that competition, purpose, validation and exposure are as much a part of people’s motivation as money.
Thought Dan Pink did an excellent job of explaining this in DRIVE.
Alek Felstiner
08/15/10
Terrific experiment, and well explained. I would add, though your results sort of imply as much, that compensation is only an imprecise an incomplete measure of the actual employment bargain. People seek much more than that when they agree to work, and if they’re lucky they receive more.
Compensation is the easiest (and most obvious) component to regulate, and thus the most commonly regulated. I think there’s a decent case to be made for compensation as a stand-in for other intangible elements. As there is a decent case to be made for the proposition that regulating tangibles has an indirect affect on intangibles (e.g. when an employer is prohibited from discriminating, or must respect organizing rights, that indirectly protects and furthers dignitary interests).
What I’m cautious of is the idea that we should be hesitant to regulate conduct if people don’t engage in that conduct primarily for the money. If you extend that argument some more, certain activities cease to be “work” if they’re not sufficiently motivated by compensation.
Imagine if the same attorneys fees rules didn’t apply to us public interest lawyers, based on the logic that we’re not as motivated by money and thus won’t abuse the system to our benefit. Or imagine if non-profits didn’t have to actually file and behave as such, but were simply tax-exempted by virtue of a legislative assignation of pure motives.
Work is work, regardless of why we do it. We can tailor regulations to promote fairness, to increase productivity, to avoid exploitation, or whatever other reason, but I don’t like the idea that we might, for example, decide against a regulation that seeks to prevent financial exploitation because our surveys indicate that the regulated group doesn’t care much about being financially exploited. Strikes me as bad policy, and has some troubling historical resonance as well.
Think about why domestic and agricultural labor was exempted from minimum wage, for example. Why graduate student teachers can’t join unions. Etc.
There’s nothing wrong with looking at motivations in order to understand problems that workers (or any potentially regulated group) will face. But the question should be “Do those problems cry out for requlation, and if so, what kind?” It shouldn’t be “Do the motivations of these workers make them better or worse candidates for regulation?”
sunny
09/01/10
None of the people you have asked has ever worked for you at mturk.
Your motivational group was motivated to get the poor penny and click just something.
Doing your jobs would be a nice thing, if they would work. Most of the Hits your compnay sets up at mturk are not working…Yes, they miss in quality of your installation.
People with high result rates ( I am talking about quality work and not a big amount) are leaving their fingers of your work, because they see it as a waste of time.
Take a closer look at one job you offer:
Find exeptional Images
-checking images for 0.05 would be a fun, if:
-the amount of images was smaller;
-these images would all load;
-your result system would work correct.
So checking 18 images, which take nearly 2 minutes to load is not fun, it is a waste of time. Of course after 2 minutes the images are not checked- they are only available to zoom. So it take another 2 minutes to check them.
4 minutes for 0.05 -if you are really quick or just runing through it.
This means $ 0.75 for an hour.
this is not fun- this is hard earned money for people who depend on every dime and these people never had the chance to get your survey.
You want a poll with a real reaction to you work?
Set it up at mturk —made it available for all countries —and let people tell you in their own works about your hits…
That result would not be shown on this blog- it would belong to facts, you really would not like..
Joseph Childress
09/02/10
@sunny
Thank you for your post and for this feedback. We have many high quality workers who actively participate in our jobs, and we strive to keep them happy. We have run thousands of jobs on the platform, and not all jobs are suited for everyone. If you feel that a job is not compensating enough, or is unfair in any way, please inform us. We have many clients, self service and enterprise, that run jobs on CrowdFlower. Naturally, mistakes happen. Your notifications help us to identify errors and weak points in the product and correct them. This is mutually beneficial because this enables us to provide you with higher quality jobs to work on, and also allows us to run more efficient jobs for our clients. The job you mentioned was never at 3 cents per assignment, but we did adjust the payment upward. It sometimes takes a while to respond to your messages, but please know that we are trying to keep your needs in mind. We will look into your issue regarding load times, as we are seeing the average submission rate for this job much lower than this. Also, thanks for your suggestion regarding an open poll to workers.
Kind Regards,
Joseph
Account Manager
CrowdFlower
Rochelle
09/13/10
Joseph, workers know Dolores Labs are among the worst, if not THE worst on Amazon’s Job board.
The most beneficial thing will happen when your mis-classification of workers is halted through class action.
Anne
09/28/10
I enjoy my “real” job a lot (mostly). Does this mean that I should earn less than someone doing the same job who does enjoy it?
You could make an argument that people who enjoy their job – or the mTurk HIT – are likely to do a better job, put more effort into it, be innovative and honest. And thus are worth more to their employer or the HIT submitter
Anne
09/28/10
ooops. Doesn’t …
“Does this mean that I should earn less than someone doing the same job who doesN’T enjoy it?”
Sharron Clemons
12/21/10
Joseph, workers know Dolores Labs are among the worst, if not THE worst on Amazon’s Job board. The most beneficial thing will happen when your mis-classification of workers is halted through class action.
Silvia Merrill
12/23/10
ooops. Doesn’t … “Does this mean that I should earn less than someone doing the same job who doesN’T enjoy it?”