In a recent paper, presented at the ACM Conference on Computer Supported Cooperative Work (CSCW), John Horton, Daniel Chen and I used a large-scale experiment to test the effect of different incentive schemes on the quality of crowdsourcing work.

The results surprised us. They suggest that workers perform most accurately when the task design credibly links payoffs to a worker’s ability to think about the answers that their peers are likely to provide.

a horserace experiment! (photo cc-by-sa by iyoupapa)

The idea for this study came out of our sense that, as social scientists, we had something unique to offer the existing research on human computation. Early and influential crowdsourcing research has focused on how to filter the judgments of the crowd to find the best answers. We wanted to know whether simple task-design changes could improve the quality of data coming into a crowdsourcing system in the first place.

To test this idea, we chose 14 different incentive schemes and framing techniques developed and validated across the social sciences and set up a horse race experiment to see which schemes/techniques would work best.

Consistent with our personal biases (John and Daniel are both economists, and I’m a sociologist), some of the schemes were financially oriented, some were social or psychological, and some were hybrids combining social and financial incentives. The details of all the schemes are included in the paper (it’s a long list, and some of them are kind of involved), but it’s worth giving some examples.

On the financial end of the incentives spectrum, we had one condition we called “reward-accuracy,” which was pretty much what you’d expect: we told workers, “we’ll pay you a bonus if you get the answers right.” We also had one called “punishment-accuracy,” the gist of which you can deduce. On the purely social-psychological side, we had one we called “trust,” in which we told workers, “we’ll pay you for this job no matter how bad your performance, we trust that you’ll still make your best effort.”

One of the weirdest schemes turns out to be important, so I need to explain that one. Called “Bayesian Truth Serum” (BTS), it incorporates a design from the work of Drazen Prelec, a behavioral economist at MIT, who realized that research subjects could probably provide useful information regarding the expected distribution for subjective, qualitative questions (nb, the mechanics of how he does this are arcane in a way that is almost sure to delight the geeks among you, so I encourage you to read his paper). Few of the details of real BTS are important, except that we incorporated the piece about asking workers to answer the questions themselves and predict the distribution of other workers’ responses. We also told them we’d give them a bonus if their predictions were correct.

We then created a task that asked workers to answer five questions. In this case, the questions were drawn from another study examining participatory features of websites, for which we already possessed validated data collected by research assistants.

All workers answered the same five questions about the same website (www.kiva.org) while being exposed to one and only one of the 14 incentive schemes (or a control condition of no scheme). Roughly 2,000 individuals participated in the study, resulting in over 100 subjects in each of the experimental conditions. (The statistics and science nerds out there will be pleased to know that both the drop-out rate and demographic covariates were distributed evenly across conditions.)

To measure worker performance, we used the research assistant responses as correct answers to the questions and then calculated the total number of matching answers (out of five) provided by each worker. The results (aggregated across all treatments) are plotted in a histogram below and show that the average worker answered just over two questions out of five correctly.

Then, in order to see how the treatments compared against each other relative to the control group, we calculated the mean correct response rate for each condition and conducted difference of means tests to see which of these means were significantly greater than the control group. The results of this comparison appear below (in a new plot that doesn’t even appear in the paper!):

The orange dots show the value of the mean in each condition, and the blue bars illustrate the 95% confidence interval around that mean. The treatments are sorted by the size of the difference in means from the control. (More hard-core nerd stuff: the means are adjusted using Intent-To-Treat estimators).

From these results, we concluded that our horse race had two clear front-runners: the “Bayesian Truth Serum” (BTS) and “Punishment – disagreement” conditions, each of which improved average worker performance by almost half of a correct answer above the 2.08 correct answers in the control group. A few of the other financial and hybrid incentives had fairly large point estimates, but were not significantly different from control once we adjusted the test statistics and corresponding p-values to account for the fact that we were making so many comparisons at once (apologies if this doesn’t make sense — it’s yet another precautionary measure to avoid upsetting the stats nerds among you). In a tough turn for the sociologists and psychologists, none of the purely social/psychological treatments had any signficant effects at all.

Why do BTS and punishing workers for disagreement succeed in improving performance significantly where so many of the other incentive schemes failed? The answer hinges on the fact that both conditions tied workers’ payoffs to their ability to think about their peers’ likely responses. (We elaborate on the argument in more detail in the paper.)

Does this mean that we should give up on simple financial or social-psychological incentives? Probably not. The fact that we conducted the experiment on MTurk means that the deck may have been stacked against incentives like the “trust” condition I described earlier. Because requesters on MTurk have little oversight, workers are more likely to respond to financial incentives than stated promises. In this sense, the marketplace has structured the interaction between workers and requesters in a way that may limit the opportunities to harness motivations that are not linked to money in some explicit way.

You can download the full paper to read more.

Motivation in the workplace. Created by user: pescatello on flickr and licensed cc-by 2.0

A few weeks ago, I was talking about the motivations of crowdsourcing workers with Judd, who has already done a ton of great work looking at motivations for participation across a wide range of online environments. He is a recent Ph.D. from the UC Berkeley School of Information and just joined Yahoo! Research as a social psychologist and research scientist in the Internet Experiences Group, so it was no surprise that he had a great idea about how to design an experiment to better understand crowdsourcing.

The most straightforward way to ask crowdsourcing workers why they do what they do is with a survey (e.g., Panos Ipeirotis’ fascinating recent informal survey of MTurk workers.) However, you also might recall from one or two of my previous posts that I tend not to take survey results at face value.

Judd’s “list experiment” presents the subjects of a study with a list of several motivations and asks them to provide a count of the number of items in the list they agree with (rather than posing yes/no questions or checkboxes).

Here’s what that looked like once Judd had it set up in Crowdflower:

A screenshot from one version of our list experiment

We presented experimental treatment groups with four other permutations of the same list — each one missing one of the items — and aggregated the results across every group. This allowed us to estimate the proportion of respondents choosing each item in the list.

The advantage of the list experiment over the traditional survey format is that it doesn’t require anybody to explicitly say, “I crowdsource because it gives me a sense of purpose.” Indeed, it perfectly preserves the anonymity of individual user preferences, since the results that we generate are estimates based on summaries of behavior across the different treatment groups. The questions are less obtrusive and there’s no pressure to hide your true sentiments or conform to the expectations of others. List experiments are thus amazing tools to examine preferences that may be controversial or otherwise influenced by social pressures in some way.

Judd and I designed a pilot experiment with the list above and administered it to MTurk workers through Crowdflower. For the sake of comparison, we also included a control condition that asked Turkers the same questions in traditional, agreement-style survey form. To simplify things, we limited the responses to US workers only.

Comparing the results from the survey condition and the list experiment revealed some mind-blowing differences:

Note the discrepancy between some of the paired bars. Whereas 97% of the Turkers in the control group agreed with the statement “I am motivated to do HITs on Mechanical Turk to make extra money,” just 60% of the Turkers in the list experiment condition expressed the same preference.

Similarly, check out the difference between the agreement-style questions and list experiment results in the “for fun” category. Again, agreement statements elicit over-reporting when compared with the list experiment (although this time to a less extreme degree).

Our preliminary conclusions from this pilot study? The ideas of crowdsourcing for money and crowdsourcing for fun sound better than they actually are.

Another, slightly more science-y way to put this is that the workers in our study over-report the extent to which they are motivated by money and fun in response to agreement statements versus a list experiment, suggesting that they perceive these two factors to be socially desirable.

Understanding the cause of this social desirability bias as well as its implications for crowdsourcing across different environments will require further research. In other contexts, social desirability bias (a.k.a. “the Broadus effect”, if you read the amazing Nate Silver) has played a role in everything from elections to educational attainment. There’s no reason to believe it doesn’t affect the way people work and participate in various online environments as well.

Perhaps most interesting of all, our findings here further complicate the growing debate over how paid crowdsourcing ought to be understood and (potentially) regulated. If a substantial proportion of workers aren’t actually on MTurk for the money, does that support the claim that we should regulate crowdsourcing along the same lines that we regulate other post-industrial sectors?

These are big questions that we should continue to probe through future studies and discussion. In the meantime, Judd and I re-ran our list experiment with a few minor adjustments and a much bigger sample. We’re in the process of writing up this larger version of the study for a conference submission and will post the full paper here as soon as we can.

In previous posts, I discussed the nature of employment law as it relates to crowd work, and the problems involved in trying to classify crowd workers according to existing categories and in transferring rights of free assembly and collective action into virtual space. Now comes the controversial part: explaining why I think it’d be a good idea for the law to jump into the middle of this complicated mess and start telling people what to do.

For some lawyers and lawmakers, “because we can” is a good enough reason. Others might press for regulation because advising clients in a regulation-free market generates fewer billable hours. But for a moment, let’s at least pretend that we as a society ought to engage in some kind of critical inquiry before intervening in an as-yet unregulated industry. And, while we’re pretending, let’s presume that such an inquiry would be shaped not by political dynamics but by the best information we have regarding how the law works and how regulation affects economic and social activity.

I’m not an economist, so I won’t be discussing the potential influence of economic theory on regulatory policy in this area. Instead I’ll focus on how the law deals with scenarios, like this one, in which existing doctrine appears woefully ill-equipped. The first question should always be: Does a problem actually exist? (Contrary to what you may believe, many lawyers and judges are perfectly willing to leave well enough alone. We’re not all “activists,” and in some cases, the most activist thing one can do is to permit the unfettered private ordering of employment relationships.)

So does a problem exist? When I’ve presented my argument that Mechanical Turk Providers should be classified as statutory employees, and that Amazon should function as a joint employer, I’ve gotten a variety of responses from classmates and colleagues. A few agree right off the bat, perhaps out of ideological sympathy (or pity). Others reject the argument, deciding that to the extent the parties are legally connected at all, they are governed by private contracts. And some go a step further. They conclude that no one in this situation is really a performing the kind of “work” that any of our laws — employment, labor, or contract — ought to regulate. In other words, they’re saying that there isn’t a problem. At least not one the law can address.

As you might imagine, I vigorously dispute that view. My argument for statutory coverage may stretch a little thin in places, but just because crowd workers don’t fit the “statutory employee” definition does not mean they fall easily into another. And they are being paid for their work. Many of them (perhaps as high as 18%) rely on it to make ends meet. It seems self-evident to me that their work should fit somewhere on the employment law spectrum, and if there’s no space right now, we should make room.

Something Doesn’t Look Right
It is true that many crowd workers perform their tasks in spare time, while doing something else, or for recreational/entertainment purposes. And often, that kind of work ends up outside the scope of employment law. But that’s not because it gets performed in spare time, or while watching TV, or simply for fun. It’s because when we think of idle college students, retirees, and stay-at-home moms, we think of them filling their time with entertainment, volunteerism, or education-focused internships — none of which are covered by employment statutes.

The key thing to recognize here is that for the most part the work itself determines statutory coverage. Or, at least, that’s the way it should be (agricultural and domestic workers absolutely deserve protection, in my view, but were excluded from minimum wage and collective bargaining laws for political and cultural reasons). Regardless of who they are, or why exactly they perform these tasks, crowd workers don’t fit the picture of the type of workers legislatures, courts, and administrative agencies have traditionally chosen to exempt from statutory coverage. They can bargain independently on only certain crowdsourcing platforms, and rarely have an opportunity to maximize profits through business organization and initiative. In short, though they may think of themselves as entrepreneurs, they aren’t really the type of entrepreneurs that employment law tends to leave alone. Turkers and similar crowd workers would more accurately be described as fungible particles in an on-demand labor pool. In that sense, they resemble day laborers, migrant farmworkers, and urban domestic workers. Most of them deserve coverage, they just don’t have it yet.

The Law Abhors a Vacuum
I’ll reiterate at this point that I have no particular economic expertise. My amateurish assessment leads me to believe that crowd labor presents at least some potential for market failure (information asymmetry, deception, problems with competition and global supply, etc.). I readily concede that it’s probably too soon to give any weight to those conjectures. Luckily for me, legal scholars don’t really require an impending market failure to justify regulatory intervention. Impending legal failure will suffice.

If we have an unstable, growing industry, with no reliable law and an unclear picture of who may owe what duties to whom, we can end up with problems. Stakeholders can’t adequately assess and manage risk. Lawyers give bad, conflicting advice, or, worse, there’s no way to tell whether any advice is good or bad. Practices develop, and expectations settle, without any consideration of how they might fit or contradict our existing legal principles and public policies. The law abhors a vacuum. Absence of regulation may be a major boon to industry pioneers (such as the one that has been generous enough to grant me space on its blog), but regulatory vacuums can really wreak havoc on the rest of us.

“Wait and See” Created This Problem
I have heard some in the industry and in the cyberlaw field suggest that it may be too early to address legal problems presented by online work. They argue that we don’t know exactly how it will play out, and that premature regulation could unintentionally suppress the healthy development of online democracy, commerce, and information exchange. I agree that we don’t know how it will develop, and that in regulating now we run some risk of stifling valuable development. But this argument really underestimates both the flexibility of the law and our own capacity to identify and articulate our priorities. Regulation does not necessarily imply blanket prohibitions and severe criminal penalties. There are creative legislators, lawyers, and judges out there. For that matter, there is no reason crowdsourcing stakeholders couldn’t participate in crafting a flexible and somewhat open-ended or discretionary approach to regulating crowd work. And we ought to be able to figure out our objectives without knowing exactly how the technology and industry will develop. For all its faults, that is the function of the legislative process, and if we trust it at all, we can trust it in this context.

What we shouldn’t do is “wait and see.” “Wait and see,” or rather, “wait and ignore,” is what got us here in the first place. It may be that in order to craft an effective regulatory approach to virtual property, lawmakers require a more fully developed picture of VP transactions. But such procrastination has not helped American workers in the slightest. Our laws were out of touch before the Internet. Permatemps, day laborers, and other contingent workers are already falling outside the reach of laws that should protect them. We cannot afford to exacerbate the problem.

Moreover, now is actually a good time to undertake some kind of regulatory intervention. Once expectations have settled, and the industry has begun to function in a certain way, and accumulate its own political clout, legislators and judges will find it more and more difficult to set rules. Customary practices will become norms, and eventually transform into sanctified industrial principles that cannot be disturbed. I’m sure that suits companies like Amazon just fine, since they already play such a prominent role in the industry and will likely continue. But my inner organizer and my inner corporate reformer don’t want to see settled expectations become law simply by virtue of the fact that things happen to have turned out that way. Even if the emerging structure of crowd labor perfectly reflected pure economic principles, and could thus function happily and indefinitely without any correction, I still wouldn’t want to see that structure automatically become law. Neither should you, if (like me) you believe in the potential of crowd work to transform economies and provide unprecedented opportunity. We have a chance to do better by workers (and employers) this time around, and we should take it.

**Note: This is the opinion of the author, and is not necessarily shared by CrowdFlower or, say, its CEO.

In the last post, I looked briefly at the roots of our existing employment and labor law, and why employment classification presents such a tricky issue in crowd work. For individual workers, being classified a statutory employee is the main hurdle. Once you’re “covered” by a particular law, you’re entitled to its protections unless the law gives employers a way out (e.g. if a business is too small to meet the size requirement).

But employment law does more than entitle individuals to particular benefits – it also protects them when they act together. The individual right to join together with others is crucial to the ability of individuals to function effectively in groups. Especially where the group is large and, perhaps by virtue of the size, or for some other reason, no individual member has sufficient power to alter his/her situation.

The first and most basic of these rights are the right of free assembly, memorialized in the First Amendment, and the right of free association, which has been read by the courts as a necessary outgrowth of the freedom of speech. Free assembly traditionally refers to peaceful protests or meetings, whereas free association involves joining organizations or associations.

The internet provides a terrific, possibly unmatched platform for free assembly. It’s designed to connect people. Whether it actually lives up to its potential is matter of heated debate among legal scholars, social theorists, and all sorts of other people. But the design, at least, enables forms of free assembly heretofore unimagined. Quite understandably, this represents a threat to governments, large employers, public companies, trade associations—basically any entity whose power relies on the isolation and relative inertia of its constituents. Networked groups of people attempting to bring their collective power to bear often encounter aggressive and dramatic opposition. Most often, we see the drama play out in the IP and free speech arenas, not the online workplace. But that’s not because online work is somehow antithetical to collective action; it’s just that online work hasn’t been around very long.

When workers (online or not) take collective action, they enjoy the some of the same free assembly and associational rights that everyone else has, plus a few other protections and restrictions unique to the employment context. For example, the nature of the employment relationship (as it is legally understood) pretty much precludes any absolute right to free assembly. Employers generally have the authority to decide where a worker must be during worktime, and to prohibit any unauthorized meetings or gatherings. This employer authority flows from the employer’s role as “master” in the traditional “master-servant” construct,* which we Americans inherited from English law and have embraced wholeheartedly. If, for some reason, the master-servant structure does not give employers control over when, where, and how employees can assemble, employers can usually still invoke their property rights regain that control. After all, most employment takes place on the employer’s property.

Of course property rights are notoriously difficult to define in cyberspace. Like employment laws, property laws (apart from those dealing with intangible property) evolved in reference to physical objects and spaces. For twenty years courts have struggled with the proper application of these principles to virtual space, and even so the law remains somewhat murky.

Say a group of workers want to enlist the rest of their colleagues in some cause. If they do so by going onto employer property together, holding up signs, distributing literature, asking for signatures, etc., the employer would have the right to prohibit such solicitation or exclude them on grounds of trespass (except in certain situations protected by the National Labor Relations Act). But what happens if they instead use the company email system, message board, or chat room to accomplish the same task?

In Intel v. Hamidi, an employee (Hamidi) used company email to broadcast his complaints about the employer (Intel). Intel claimed, among other things, that Hamidi’s emails constituted ordinary trespass and trespass to chattel (which is essentially trespass on personal possessions, as opposed to real estate). Intel argued that it owned the email system and the servers through which email traffic would pass, and that use of that system and those servers represented a violation of its property rights. The California Supreme Court agreed, to an extent. It recognized that an email could amount to a trespass to chattel if the owner could show actual or threatened damage to computer hardware or software, or interference with its ordinary and intended operation. But the Court explicitly rejected the idea that software or servers could be considered real property, subject to traditional trespass law. According to Hamidi, there is no such thing as trespass on virtual space.

Fortunately for workers, and unions, the National Labor Relations Act picks up where the First Amendment falls away-at the boundary between public and private property. The NLRA protects workers when they engage in “concerted activity for mutual aid and protection,” basically anywhere. “Concerted activity” basically means activity involving two or more employees, or one employee acting on behalf of others or seeking to enlist the support of others. “Mutual aid and protection” ensures that the purpose of the activity is to to improve wages, working conditions, or some other aspect of employment that affects the group. Usually, what we’re talking about is communications (mass or one-on-one), meetings, petitions, protests, and—in extreme cases—picketing, strikes, or boycotts.

Whether or not the conduct is actually protected by the NLRA will depend on whether crowd workers are “employees” for the purposes of the Act (nobody wants to revisit that thorny question from the last post). And even if they are protected, the traditional concept of unionization might not fit particularly well with crowd labor. For one thing, seniority and job security, two of the central tenets of collective bargaining, won’t have much meaning in a high turnover, “open call” labor market. For another, collective bargaining requires a group of workers sufficiently similar to share a “community of interest,” and the National Labor Relations Board (which administers the NLRA) has, like the rest of the court system, proven fairly hostile to the idea of a legally cognizable community in cyberspace.

In the absence of NLRA protection, I predict that we will still see some familiar forms of collective action emerge, as well as some unique to the crowdsourcing milieu. Turkopticon, which allows AMT Providers to rate and comment on Requesters, offers an excellent example of the kind of collective action that can only really exist in online crowd work. Message boards and social networks are further examples of virtual assembly. I expect that as the market for crowd labor grows, independent gathering places for crowd workers will grow in size and organizing capacity.

Crowd workers may follow the example of writers, graphic artists, and other groups generally neglected by organized labor, by establishing a membership-based organization to promote their interests. A crowd worker association could lobby on behalf of crowd workers, attempt to establish group benefits, handle disputes, inform crowd workers of their legal rights, and serve as a clearinghouse for campaign activities. Such organizations tend not to follow the traditional union model, instead tailoring themselves to the specific dynamics of their industry. Crowdsourcing would seem ideally suited for an industry-specific worker association.

Some crowd workers will probably go one step further, taking direct action to raise their compensation and change their employment terms. We know how much networked communities depend on the presence and contribution of the crowd. Crowd workers could withhold labor from untrustworthy employers, adopt voluntary restrictions (such as a blanket refusal to perform tasks below a certain wage level), or create worker associations to provide training and “accredit” responsible employers. I also anticipate some version of the “work to rule” strategy, in which employees attempt to hobble an otherwise smoothly functioning system by ignoring the way things actually run and instead adhering rigidly to the rules.

When faced with an organized group that already communicates primarily online, crowdsourcing vendors might prove more responsive than we might expect.

*More on masters and servants here.

On May 10^th I presented at the Crowdflower Distributed Distributed work Meetup, talking about employment law and regulation. As it turned out, my 15-minute presentation turned into an hour and a half of exploring the legal issues involved in crowd work. The group of clients, workers, and crowdsourcing vendors displayed a real hunger for guidance on how the courts might apply existing laws to crowd work. Our long and involved discussion illustrated how little is actually out there to help people understand these issues.

By way of a recap for those with the good fortune of not having had to endure my fleeting law professor fantasy, I’ve broken up the presentation into a few parts, which I’ll post here. This segment looks at the basic reasons why applying our current law is so difficult, and the threshold question of how to properly classify crowd workers. The next one will deal with civil liberties and group rights in virtual work, and the final one (if I get that far) will present my best case for immediate regulatory intervention. If you don’t want to wait, you can find a thorough (and probably boring) treatment of this topic here.

The Basic Problem: Old, Stale Law

Most of our employment laws – the ones dealing with minimum wage, union organizing, health and safety, etc. – were written in middle of the 20^th Century, in response to a period of rapid industrialization. The people who wrote these laws had a particular concept of employment in mind, one characterized by physical worksites, long-term employment relationships, and one-to-many connections between firms and workers. By the 1990s (if not earlier), the rise of contingent and temporary labor, complex subcontracting structures, and remote work had substantially eroded the principles relied upon by the original drafters of the law. But nobody did much of anything to update that law, and as a result, huge swaths of people whose work would have been covered under the employment arrangements of the previous generation just fell into a regulatory vacuum. (For a dense but compelling description of this process, and its legal implications, check out Katherine Stone’s From Widgets to Digits).

Distributed work, microwork, crowd work, whatever you want to call it, threatens to basically nullify existing employment laws, and the 20^th Century conception of employment upon which they were based. That’s why, when you ask questions like “Should Turkers be entitled to minimum wage?,” no one seems to have a very good answer. Most people confronted with that question revert to considering the Turkers themselves, and trying to decide whether a college student or a stay-at-home mom “deserves” the legal protections afforded to more traditional employees, rather than trying to assess whether the work itself should fall within the scope of statutory employment.

But someone is going to have to answer the hard legal questions eventually, because the industry is growing too quickly to be ignored. It might be the courts, it might be the legislatures, it might be the IRS or the Department of Labor, but one way or another our regulatory systems will be forced to take a position.

Twenty Thousand Independent Contracts?

The first big threshold question is whether crowd workers are independent contractors or statutory employees. To lawyers, “employee” doesn’t just mean “someone who works for me.” “Employees” are a creature of statute, with a special meaning. They are a particular class of workers who, because of their lack of power in relation to their employers and their various other vulnerabilities, have received specific protection from Congress and the states in the form of minimum wage, organizing rights, etc. By contrast, independent contractors (at least in theory) are specialist entrepreneurs who bargain for their services and execute contracts to perform those services. They should have no relationship with the employer apart from the contract (hence, “independent”), and any disputes that come up should be decided by interpreting the contract language, under contract law.

This distinction between independent contractors and statutory employees really matters. Independent contractors are not covered by federal or state minimum wage/overtime laws, nor are they covered by OSHA (health and safety), Title VII (employment discrimination), FMLA (family leave), or the NLRA (union organizing). And they don’t receive workers compensation or unemployment unless they pay into the state funds themselves.

Now let’s look at a typical crowdsourcing platform to see where the workers fall. Here’s simplified structural diagram.

We know what governs the relationship between workers and vendors, and what governs the relationship between firms and vendors – the terms of use that you have to sign in order to gain access to the platform (aka the participation agreement). But we don’t know much about the relationship between workers and firms, and it’s the nature of that relationship that determines whether a worker is an employee or an independent contractor. Unsurprisingly, the vendors try to fill in the content of the worker-firm relationship by inserting a bunch of language into the only legal document that does exist: the participation agreement. One of the most important “inserts,” and the vast majority of crowdsourcing vendors use it, is a requirement that crowd workers provide services as independent contractors. The typical language goes something like this (from oDesk)

“Provider’s relationship with Buyer will be that of an independent contractor, and nothing in this Agreement should be construed to create a partnership, joint venture, or employer-employee relationship.”

For a more comprehensive list, check out this slideshow. Fortunately for crowd workers, the labels put on the relationship by the parties, or by some third party, aren’t the end of the story. Courts go further, trying to understand whether the relationship is really an independent contract or an employer-employee scenario. They consider a variety of factors, depending on what law the parties are trying to apply. For example, the Department of Labor suggests a seven-factor test for application of the Fair Labor Standards Act, the federal law that establishes minimum wage and overtime rules (Alexander Sorokin performed a nice summary of the factors in an earlier post on this blog). But the test changes with the law to be applied, and also with the jurisdiction, the industry/type of work, and the particular situation presented to the court. Some tests focus on the employer’s right to control. Others look at whether the worker has an opportunity to function as an entrepreneur or is dependent on the employer. I won’t go through all the tests here – suffice it to say that there’s a lot of variation, and right now crowd workers fall squarely into the gray area between employees and independent contractors.

Personally, I find the notion of independent contractors quite fitting in some contexts, such as eLance or Guru, and essentially ridiculous in others – such as Mechanical Turk and LiveOps. The concept of a rapid succession of independent contracts, perhaps hundreds in a row for a particular worker, tens of thousands per day on a platform like Mechanical Turk, just runs contrary to the underlying rationale for distinguishing independent contractual agreements from statutory employment. Does clicking an “Accept HIT” button and agreeing to tag ten minutes worth of video truly create a separate, independently bargained contract? I’m not sure such a proposition would pass the laugh test, even for a fairly conservative judge.

The problem is that there aren’t any other categories into which we can place this kind of work. A conservative judge may find ludicrous the idea of a ten-minute long, unbargained, anonymous “independent contract,” but he’s not going to just award a bunch of clickworkers minimum wage when hundreds of thousands of contingent workers and permatemps don’t enjoy the same protection. There simply isn’t a good fit, because employment and labor law has not kept pace with the emergence of our information economy.

Which brings me back to my original point. Our existing law is archaic and broken. We’re all waiting for the first group of Turkers to file a wage and hour class action, so we can find out what happens next.

This is a guest post written by my friend Dana Chandler on how the context of a task motivates the person working on it.  He has a longer academic paper on the topic you can find at the bottom of this post.  It once again shows how traditional economic incentives can’t fully explain workers’ behaviors on Mechanical Turk.

Imagine for a moment that you were a turker from either the US or India, looking at the above image. You are given the task of clicking on the blue circular objects with red borders. What you see is only a fraction of the full image. Each image has 90 blue objects to identify. If you’re as good as the average worker, you’ll complete your first image in a little over five minutes and you’ll earn 10 cents (for an hourly wage of $1.20).

After your first image, you can either quit and take your 10 cents, or identify points on another image. Over the next four hours, you’ll have the chance to label as many images you want. But there’s a catch—you’ll only be paid 9 cents for the second image, 8 cents for the third, and so on, all the way down to 2 cents. This will lower the hourly wage even more.

Before you even qualify for the task, you’ll have to spend five minutes watching a training video and passing a quiz. During the video, half of you will be given only basic work instructions on how to identify “objects of interest.” The other half will be given both instructions and cues of meaning: recognition for your contribution and an explanation of your task’s purpose¹. The reason given here? To help researchers identify cancerous tumor cells.

We posted these HITs on MTurk in January, 2010. Almost 300 people from the U.S. and India accepted the task, becoming unknowing participants in our experiment examining MTurk worker motivation. It is commonly believed (and other researchers have verified with demographic surveys) that Indian workers are more motivated by pecuniary concerns and that US turkers are primarily doing tasks for leisure or other non-pecuniary motives. Is this true?

In both countries, half of the turkers in the experiment were randomly assigned to label nondescript “objects of interest” without being given any context or greater purpose — they were our zero-context group. The other half, our meaningful group, were told they were helping researchers identify cancerous tumor cells. Which group of turkers do you think worked harder? You might be surprised.

Therefore, our experiment compared two groups with and without a clear wage motivation, to see if workers behave differently responded to meaningfulness in their tasks.

Results

We measured three metrics: “showing up”, the quantity of work, and the quality of that work. The first two metrics are straightforward. Showing up meant that you sat through our training video, passed our qualification test and helped label at least one image. Quantity of work was simply the number of images labeled.

We repeatedly told both groups of turkers that they needed to click on all points and as closely as possible to each point. Work quality was determined by the fraction of cells that a person clicked on (the recall) and the average distance between the “true center” of each cell and where the user clicked (the centrality).

Our most interesting finding was the extent to which a meaningful task (and giving recognition) motivated US workers, but not Indian workers, to complete a task. As any requester knows, attrition on MTurk is a real problem. We found that adding cues of meaning could motivate turkers to undergo training and label at least one image. In the US, adding cues-of-meaning raised the fraction of turkers who completed our task from, 92% of people who sat through our training video, took our quiz, and labeled an image showed up. This figure compares to only 83% of zero-context group (see figure which also has standard errors). In India, there was no difference between the groups and both groups had a 66% completion rate (attrition being higher due to possible language barriers, slow connection speeds, hardware issues, etc.).

However, once a person did some work, both treatment and control groups did a similar quantity of work: The cues-of-meaning group labeled 6.0 images and the zero-context group labeled 5.7 images. This difference was not statistically significant, so it suggests that once you get turkers to work on a task, they are motivated to label just as many images irrespective of the task’s meaningfulness. Notably, of the people who worked, Indians worked longer and labeled an average of 7.3 images vs. 5.2 in the US.

Surprisingly, all workers did an equally good job identifying points whether they had zero-context or whether they thought they were identifying tumor cells. The quality as measured by the fraction of points identified (the recall) or the average pixel distance (the centrality) was statistically insignificant irrespective of the task’s meaningfulness.

This finding has important implications for those who employ labor in crowdsourcing markets. Companies and intermediaries should develop an understanding of what motivates the people who work on tasks. Employers must think beyond monetary incentives and consider how they can reward workers through non-monetary incentives such as by changing how workers perceive their task. Alienated workers are less likely to do work if they don’t know the context of the work they are doing and employers may find they can get more work done for the same wages simply by telling turkers why they are working.

For more details of this study, please see our full academic paper at: http://danachandler.com/index.php/research. We welcome any comments and feedback.

About the authors:
Dana Chandler is a researcher at the University of Chicago’s Becker Center where he works with Steven Levitt, author of Freakonomics. He previously worked as a management consultant at the Boston Consulting Group and at Aureos Capitol, a Colombian private equity company. He will begin his Ph.D. at MIT in the Fall.  Dana’s research interests include digital labor markets, development economics, and randomized experiments in companies. email: dchandler {at} uchicago {dot} edu

Adam Kapelner is currently earning his Ph.D. in Statistics at Wharton. Adam is the founder of  dictionarysquared.com and the inventor of its vocabulary-learning  technology. While working as an undergraduate researcher at Stanford University, he helped engineer the open-source software, www.gemIdent.com, that enables researchers worldwide to locate cells in microscopic images. GemIdent was recently extended to make use of MTurk for outsourcing of medical image identification. The extension, called www.distributeeyes.com, was  adapted to serve as the platform for this experiment. email: kapelner  {at} wharton {dot} upenn {dot} edu

Acknowledgments: We thank Professor Susan Holmes of Stanford University for allowing us to adopt DistributeEyes (funded under NIH grant #R01GM086884-02) for use in this study. We would also like to thank Panos Ipeirotis for kindly providing us with demographic and market data that we cite in our study. Lawrence Brown, Patrick DeJarnette, John Horton, Emir Kamenica, Steven Levitt, Susanne Neckermann, Jesse Shapiro, Jorg Spenkuch, Jan Stoop, Chad Syverson, Mike Thomas, Abraham Wyner, and seminar participants at the University of Chicago provided especially helpful comments.

Online labor markets dramatically lower the cost and hassle of conducting experiments. On Amazon’s Mechanical Turk, it is easy to run multiple experiments per week. Figuring out how to run experiments isn’t that hard, as there are already some nice tutorials available.

However, what I felt was missing from the field was a discussion of why, precisely, we can trust results from online experiments. This was the motivation for a new paper, jointly written with Dave Rand (who wrote up part of this study here on the Dolores Labs blog) and Richard Zeckhauser.

You can download the paper here.

While we make the practical and theoretical case for online experimentation, we believe that acceptance of online results as “valid” will come after people start seeing how easy and reliably one can replicate previous studies. This is why blogs like Experimental Turk and Deneme—both of which report results from AMT experiments—are so helpful. In our paper, we continue this process by replicating three results that are fairly well established.

In one experiment for the economists, we show—contra the usual intuition—that at least some Turkers are financially motivated, despite the very low stakes. After performing an initial text transcription task, workers were offered some randomly chosen amount of money to do an additional transcription. Results show the counts of people who agreed (“Yes”) and the counts of people who did not agree (“No”), by amount offered.

Nothing too surprising—offer to pay more and more workers will accept—but at this stage in the development of online experiments as a methodology, “surprising” would probably be bad news.

Anyway, the full paper is here. We’d love to get comments and feedback—it’s not too late to earn a place in our coveted “thanks” footnote!

“Who are these people?” and “Why do they participate?” are two perennial questions about AMT. Askers are generally incredulous that AMT workers are willing to do rather tedious tasks for small amounts of money.

To investigate this question of motivation, NYU Prof. Panos Ipeirotis asked a bunch of workers their reasons and tabulated the responses here. His two posts are actually on the syllabus for a course at Stanford (incidentally the course is taught by one of the creators of Protovis, which is very cool and is on my list of things to learn). There is also this amusing investigation.

For a joint project with Dave Rand and Richard Zeckhauser, we asked ~ 400 AMT workers both (a) where they are from and (b) the primary reason they participate on AMT. Because economic opportunities differ by country, we might expect that motivation and behavior should also differ by country. The cross tabulation plot is below (reasons are in the “rows”, countries in the “columns”–the size of each rectangle is proportional to the number of responses in that cell):

Two things to note:
1) Money is a big motivation for everyone
2) Money aside, people from India are there to learn; people from the US are there to have fun

Although the India/US differences are consistent with the different-countries/different-motivations hypothesis, the most relevant fact is the unconditional importance of money.

While these findings seem reasonable, I feel compelled to make the standard reliability critique of self-reported data. Our learning/fun AMT workers might also be there for the money, but feel sheepish about saying so. Though this could go the other way as well I suppose: if, for example, a worker has an intrinsic love of image captioning but finds this passion shameful, they might report that they are in it for the money. But this seems less likely than the other scenario of downplaying financial motivations.

Many workers on Amazon Mechanical Turk are willing to help others at a cost to themselves, just like participants in laboratory experiments.

While traditional economic models assume that people are entirely selfish, a central theme in behavioral economics is the existence of ‘social preferences’, or caring for others. Countless laboratory experiments have demonstrated that many people are willing to help others, even at a cost to themselves. This behavior is clearly inconsistent with being motivated only by your own monetary payoff – if you are entirely selfish, you would never pay money to help someone else in the totally anonymous conditions of the lab. In this post I describe an experiment I conducted together with John Horton, and with invaluable technical assistance from Xiaoqi Zhu, that replicates the existence of social preferences on Amazon Mechanical Turk (AMT), showing that many Turkers behave altruistically.

We also demonstrate the principle of priming, another focus of great interest in experimental economics. In priming studies, stimuli unrelated to the decision task (and which do not affect the monetary outcomes) can nonetheless significantly alter subjects’ behavior.

To assess altruistic behavior on AMT, 194 subjects played an incentivized Prisoner’s Dilemma (PD), the canonical game for studying altruistic cooperation. Subjects were informed that they had been randomly assigned to interact with another Turker, and that they would each have a choice between two options, A or B. In addition to a 20 cent “show-up fee”, they were informed of the following payoff structure: if both subjects chose A, they receive each earn a 120 cent bonus; if both chose B, they would each receive an 80 cent bonus; if one chose A while the other chose B, the A player would receive 40 cents while the B player would receive 160 cents. The resulting payoff matrix is as follows (in each cell I first show the row player’s payoff, and then the column player’s payoff):

Thus A represents cooperation, and B represents defection. If both people chose A, they both do better than if both choose B. However, regardless of the other’s action, you earn more by choosing B (hence the ‘dilemma’). Rational self-interested players should therefore always select B, and it is altruistic to choose A (helping the other at a cost to you). Given previous evidence from experiments in the laboratory, however, we predicted that AMT subjects would demonstrate a level of cooperation significantly greater than 0 in a one-shot PD.

To explore the effects of priming on AMT subjects, we built on a previous study demonstrating that exposure to religious words and phrases increases altruistic behavior, particularly among those who believe in god (Shariff & Norenzayan 2007). Among the 194 subjects in our experiment, the prime group (N=89) read a Christian religious passage about the importance of charity (Mark 12:21-22) before playing the PD, whereas the no-prime group (N=105) did not. Following the PD, subjects completed a demographic questionnaire reporting age, gender, and education, and indicated whether they had ever had an experience which convinced them of the existence of god.Based on the results of Sheriff & Norenzayan, we hypothesized that the religious prime would increase cooperation, and further hypothesized that the effect would be driven by subjects that believe in god.

Consistent with our first prediction, we observe a level of cooperation significantly greater than 0 in both the no-prime (54% C: sign-rank test, p<0.001) and prime (71% C: sign-rank test, p<0.001) conditions. Consistent with our second prediction, we observe significantly more cooperation in the prime condition compared to the no-prime condition (Chi² test, p=0.018). Consistent with our third prediction, the prime only increases cooperation among subjects who believe in god (Chi² test, non-believers: p=0.82, believers: p=0.004). The results are visualized in Figure 1. Using logistic regression with robust standard errors, we also find that these results are robust to controlling for age, gender, country of residence (US vs non-US), religion (Christian vs non-Christian) and education.

Figure 1. Reading a religious passage significantly increases Prisoner’s Dilemma cooperation among those who believe in god, but not among non-believers.


To summarize, we have demonstrated two aspects of Turker behavior:

1. A majority of Turkers chose the altruistic option of cooperating in a Prisoner’s Dilemma. Thus even in the entirely anonymous and profit-motivated online labor market of AMT, many people still choose to help each other. This sort of altruistic cooperation is a fundamental part of the natural world, and is the building block of human societies. For more, see (Nowak 2006).

2. Reading a religious passage about the important of charity makes religious Turkers more altruistic, but has no effect on Turkers who do not believe in god. This shows that Turkers respond in basically the same way as “normal” lab subjects, and is fairly intuitive. Those who believe in god are receptive to calls for generosity phrased in religious language, while non-believers aren’t. Secular primes have been shown to work for both religious and non-religious subjects (Shariff & Norenzayan 2007).

Although AMT workers are certainly not a generally representative sample, this study demonstrates that they show several of the same basic behavioral features observed in behavioral laboratory experiments. Furthermore, AMT allowed this study to be run extremely quickly and inexpensively. The 200 subjects were recruited in less than 2 days, at a total cost of $253. As a behavioral researcher, this is amazingly exciting! I usually spend months and thousands of dollars per study. AMT opens the possibility of exploring countless interesting ideas that otherwise we would have had neither the time nor money to pursue.

For other studies about cooperation, reward and punishment that I’ve conducted at Harvard, see the pdfs on my webpage: www.DavidGertlerRand.com.