Arcade Fire has collaborated with Google and Chris Milk to set up a crowdsourced art project that anyone with a computer can contribute to and even curate, from anywhere in the world.

Here’s how it works:

By visiting “The Wilderness Downtown” (best viewed in Google Chrome) and entering your childhood address, you can travel down memory lane through the streets you grew up in, accompanied by Arcade Fire’s “We Used To Wait.”

You can then send your childhood self a note, a drawing, or other art, which gets incorporated into your personalized art project. If you like what you created, you can submit it to Arcade Fire, where it may end up as part of their concert tour.

Every submission to this crowdsourced art project will be curated in real time by CrowdFlower, whose workforce sifts through the messages and identifies stellar — or offensive — content.

Prior to crowdsourcing, a global art project like this (not to mention real-time curation and content moderation by human beings) would have been unthinkable. It’s yet another glimpse of the future you can send to the childhood you.

Please visit http://thewildernessdowntown.com/ to view and contribute to the project. And if you want to help curate, visit http://crowdflower.com/judgments/mob/20094.

The power of crowdsourcing has transformed the nature of disaster-relief efforts. In a recent Red Cross survey, more than half the interviewees said that they would report emergencies via social media like Facebook, Twitter, or a personal blog, and more than half said that they would report emergencies by text-message if the service was available.

Globally, we in the humanitarian community are already seeing massive waves of unstructured information coming out of crisis-affected regions. One of the biggest challenges is that the crisis-response community is already stretched in human resources at precisely the time that this information is streaming in.

A team of dedicated people at PakReport are mapping real-time data from Pakistan by collecting information from aid agencies, the media, and direct reports by email and SMS.

CrowdFlower is helping PakReport to geolocate and translate reports from the ground in response to the floods in Pakistan. For the time-intensive task of translating, categorizing, and geolocating these messages, volunteers from anywhere in the world can come online to help process each report simply by reading the message and filling out a form.

CrowdFlower’s internal workings allow us to automatically cross-check this work among multiple volunteers so that the information is not susceptible to the potential errors of any one volunteer. This ensures data-quality for the aid agencies using the data and means that the volunteers can help without fear of accidently introducing bad information.

This is the second time that we have been able to utilize the CrowdFlower platform for crisis response, after we designed a similar process for translating, geolocating, and categorizing emergency text messages in the wake of the January 12 earthquake in Haiti.

Here, we have integrated CrowdFlower as a plugin for the Ushahidi mapping platform, enabling us to leverage the goodwill and hard work of hundreds of volunteers globally. We hope to see this successful partnership and information processing strategy continue to support more crisis-affected communities in the future, wherever in the world they may be.

Again, if you have even a few minutes to spare, you can volunteer now at PakReport.org.

Robert Munro is the Chief Information Officer at Energy for Opportunity and a Graduate Fellow in computational linguistics at Stanford where he specializes in methods for processing large volumes of information in less-resourced languages.

Intro with Niel Robertson

Jim Giles

Jeffrey Heer

Patrick McKenna

Motivation in the workplace. Created by user: pescatello on flickr and licensed cc-by 2.0

A few weeks ago, I was talking about the motivations of crowdsourcing workers with Judd, who has already done a ton of great work looking at motivations for participation across a wide range of online environments. He is a recent Ph.D. from the UC Berkeley School of Information and just joined Yahoo! Research as a social psychologist and research scientist in the Internet Experiences Group, so it was no surprise that he had a great idea about how to design an experiment to better understand crowdsourcing.

The most straightforward way to ask crowdsourcing workers why they do what they do is with a survey (e.g., Panos Ipeirotis’ fascinating recent informal survey of MTurk workers.) However, you also might recall from one or two of my previous posts that I tend not to take survey results at face value.

Judd’s “list experiment” presents the subjects of a study with a list of several motivations and asks them to provide a count of the number of items in the list they agree with (rather than posing yes/no questions or checkboxes).

Here’s what that looked like once Judd had it set up in Crowdflower:

A screenshot from one version of our list experiment

We presented experimental treatment groups with four other permutations of the same list — each one missing one of the items — and aggregated the results across every group. This allowed us to estimate the proportion of respondents choosing each item in the list.

The advantage of the list experiment over the traditional survey format is that it doesn’t require anybody to explicitly say, “I crowdsource because it gives me a sense of purpose.” Indeed, it perfectly preserves the anonymity of individual user preferences, since the results that we generate are estimates based on summaries of behavior across the different treatment groups. The questions are less obtrusive and there’s no pressure to hide your true sentiments or conform to the expectations of others. List experiments are thus amazing tools to examine preferences that may be controversial or otherwise influenced by social pressures in some way.

Judd and I designed a pilot experiment with the list above and administered it to MTurk workers through Crowdflower. For the sake of comparison, we also included a control condition that asked Turkers the same questions in traditional, agreement-style survey form. To simplify things, we limited the responses to US workers only.

Comparing the results from the survey condition and the list experiment revealed some mind-blowing differences:

Note the discrepancy between some of the paired bars. Whereas 97% of the Turkers in the control group agreed with the statement “I am motivated to do HITs on Mechanical Turk to make extra money,” just 60% of the Turkers in the list experiment condition expressed the same preference.

Similarly, check out the difference between the agreement-style questions and list experiment results in the “for fun” category. Again, agreement statements elicit over-reporting when compared with the list experiment (although this time to a less extreme degree).

Our preliminary conclusions from this pilot study? The ideas of crowdsourcing for money and crowdsourcing for fun sound better than they actually are.

Another, slightly more science-y way to put this is that the workers in our study over-report the extent to which they are motivated by money and fun in response to agreement statements versus a list experiment, suggesting that they perceive these two factors to be socially desirable.

Understanding the cause of this social desirability bias as well as its implications for crowdsourcing across different environments will require further research. In other contexts, social desirability bias (a.k.a. “the Broadus effect”, if you read the amazing Nate Silver) has played a role in everything from elections to educational attainment. There’s no reason to believe it doesn’t affect the way people work and participate in various online environments as well.

Perhaps most interesting of all, our findings here further complicate the growing debate over how paid crowdsourcing ought to be understood and (potentially) regulated. If a substantial proportion of workers aren’t actually on MTurk for the money, does that support the claim that we should regulate crowdsourcing along the same lines that we regulate other post-industrial sectors?

These are big questions that we should continue to probe through future studies and discussion. In the meantime, Judd and I re-ran our list experiment with a few minor adjustments and a much bigger sample. We’re in the process of writing up this larger version of the study for a conference submission and will post the full paper here as soon as we can.

We’ve already noted how crowdsourcing could aid federal investigators digging into Goldman Sachs transactions (and even offered to categorize and tag the first 100,000 documents for free).

Now comes The New York Times story of companies skirting the House of Representatives’ recent ban on earmarks to for-profits by creating strikingly similar nonprofits that, of course, the ban exempts.

Some lawmakers are clamoring to take advantage of this loophole — already racking up $150 million in earmark requests for these newly minted nonprofits.

So who’s left to sort out the genuine nonprofits from the other guys?

As the Times notes:

In ignoring the spirit of the ban, some lawmakers are leaving it up to Congressional committees to block them, a prospect that both Democrats and Republicans on Capitol Hill concede will be near impossible.

“No matter what they tell you, there is just no way they can police all that,” [Representative Jeff Flake (R-AZ)] said. “They just don’t have the time or resources.”

So why not level the playing field? The committees may not be able to check each proposal for evidence of foul play, but a relative novice could flag the nonprofits that have strong ties to for-profit entities.

The Internet has already boosted transparency on this front. As of last year, all members of Congress are required to post their earmark requests online.

It’s time for people to take the next logical step. Rather than just reading about requested earmarks, the public can use crowdsourcing to help keep them honest.

CrowdFlower Office

On July 19, the following speakers will come to CrowdFlower to discuss developments in modern crowdsourcing work:

Jim Giles

is a freelance journalist, who focuses on the relationships between science, technology, and society. He has written for numerous publications, including The Atlantic, the New York Times, New Scientist, and The Guardian.

Jeffrey Heer

is an Assistant Professor in the Computer Science Department at Stanford University, where he works on human-computer interaction, visualization, and social computing.

Patrick McKenna

is founder and former vice president of LiveOps, an on-demand, crowdsourced calling center. He is also founder and CEO of Keniks.

Please RSVP at Meetup.com and Plancast.

Where:
CrowdFlower Office
455 Valencia Street
San Francisco, CA 94103

When:
July 19 from 5:00 p.m. – 8:00 p.m.

For some lawyers and lawmakers, “because we can” is a good enough reason. Others might press for regulation because advising clients in a regulation-free market generates fewer billable hours. But for a moment, let’s at least pretend that we as a society ought to engage in some kind of critical inquiry before intervening in an as-yet unregulated industry. And, while we’re pretending, let’s presume that such an inquiry would be shaped not by political dynamics but by the best information we have regarding how the law works and how regulation affects economic and social activity.

I’m not an economist, so I won’t be discussing the potential influence of economic theory on regulatory policy in this area. Instead I’ll focus on how the law deals with scenarios, like this one, in which existing doctrine appears woefully ill-equipped. The first question should always be: Does a problem actually exist? (Contrary to what you may believe, many lawyers and judges are perfectly willing to leave well enough alone. We’re not all “activists,” and in some cases, the most activist thing one can do is to permit the unfettered private ordering of employment relationships.)

So does a problem exist? When I’ve presented my argument that Mechanical Turk Providers should be classified as statutory employees, and that Amazon should function as a joint employer, I’ve gotten a variety of responses from classmates and colleagues. A few agree right off the bat, perhaps out of ideological sympathy (or pity). Others reject the argument, deciding that to the extent the parties are legally connected at all, they are governed by private contracts. And some go a step further. They conclude that no one in this situation is really a performing the kind of “work” that any of our laws — employment, labor, or contract — ought to regulate. In other words, they’re saying that there isn’t a problem. At least not one the law can address.

As you might imagine, I vigorously dispute that view. My argument for statutory coverage may stretch a little thin in places, but just because crowd workers don’t fit the “statutory employee” definition does not mean they fall easily into another. And they are being paid for their work. Many of them (perhaps as high as 18%) rely on it to make ends meet. It seems self-evident to me that their work should fit somewhere on the employment law spectrum, and if there’s no space right now, we should make room.

Something Doesn’t Look Right
It is true that many crowd workers perform their tasks in spare time, while doing something else, or for recreational/entertainment purposes. And often, that kind of work ends up outside the scope of employment law. But that’s not because it gets performed in spare time, or while watching TV, or simply for fun. It’s because when we think of idle college students, retirees, and stay-at-home moms, we think of them filling their time with entertainment, volunteerism, or education-focused internships — none of which are covered by employment statutes.

The key thing to recognize here is that for the most part the work itself determines statutory coverage. Or, at least, that’s the way it should be (agricultural and domestic workers absolutely deserve protection, in my view, but were excluded from minimum wage and collective bargaining laws for political and cultural reasons). Regardless of who they are, or why exactly they perform these tasks, crowd workers don’t fit the picture of the type of workers legislatures, courts, and administrative agencies have traditionally chosen to exempt from statutory coverage. They can bargain independently on only certain crowdsourcing platforms, and rarely have an opportunity to maximize profits through business organization and initiative. In short, though they may think of themselves as entrepreneurs, they aren’t really the type of entrepreneurs that employment law tends to leave alone. Turkers and similar crowd workers would more accurately be described as fungible particles in an on-demand labor pool. In that sense, they resemble day laborers, migrant farmworkers, and urban domestic workers. Most of them deserve coverage, they just don’t have it yet.

The Law Abhors a Vacuum
I’ll reiterate at this point that I have no particular economic expertise. My amateurish assessment leads me to believe that crowd labor presents at least some potential for market failure (information asymmetry, deception, problems with competition and global supply, etc.). I readily concede that it’s probably too soon to give any weight to those conjectures. Luckily for me, legal scholars don’t really require an impending market failure to justify regulatory intervention. Impending legal failure will suffice.

If we have an unstable, growing industry, with no reliable law and an unclear picture of who may owe what duties to whom, we can end up with problems. Stakeholders can’t adequately assess and manage risk. Lawyers give bad, conflicting advice, or, worse, there’s no way to tell whether any advice is good or bad. Practices develop, and expectations settle, without any consideration of how they might fit or contradict our existing legal principles and public policies. The law abhors a vacuum. Absence of regulation may be a major boon to industry pioneers (such as the one that has been generous enough to grant me space on its blog), but regulatory vacuums can really wreak havoc on the rest of us.

“Wait and See” Created This Problem
I have heard some in the industry and in the cyberlaw field suggest that it may be too early to address legal problems presented by online work. They argue that we don’t know exactly how it will play out, and that premature regulation could unintentionally suppress the healthy development of online democracy, commerce, and information exchange. I agree that we don’t know how it will develop, and that in regulating now we run some risk of stifling valuable development. But this argument really underestimates both the flexibility of the law and our own capacity to identify and articulate our priorities. Regulation does not necessarily imply blanket prohibitions and severe criminal penalties. There are creative legislators, lawyers, and judges out there. For that matter, there is no reason crowdsourcing stakeholders couldn’t participate in crafting a flexible and somewhat open-ended or discretionary approach to regulating crowd work. And we ought to be able to figure out our objectives without knowing exactly how the technology and industry will develop. For all its faults, that is the function of the legislative process, and if we trust it at all, we can trust it in this context.

What we shouldn’t do is “wait and see.” “Wait and see,” or rather, “wait and ignore,” is what got us here in the first place. It may be that in order to craft an effective regulatory approach to virtual property, lawmakers require a more fully developed picture of VP transactions. But such procrastination has not helped American workers in the slightest. Our laws were out of touch before the Internet. Permatemps, day laborers, and other contingent workers are already falling outside the reach of laws that should protect them. We cannot afford to exacerbate the problem.

Moreover, now is actually a good time to undertake some kind of regulatory intervention. Once expectations have settled, and the industry has begun to function in a certain way, and accumulate its own political clout, legislators and judges will find it more and more difficult to set rules. Customary practices will become norms, and eventually transform into sanctified industrial principles that cannot be disturbed. I’m sure that suits companies like Amazon just fine, since they already play such a prominent role in the industry and will likely continue. But my inner organizer and my inner corporate reformer don’t want to see settled expectations become law simply by virtue of the fact that things happen to have turned out that way. Even if the emerging structure of crowd labor perfectly reflected pure economic principles, and could thus function happily and indefinitely without any correction, I still wouldn’t want to see that structure automatically become law. Neither should you, if (like me) you believe in the potential of crowd work to transform economies and provide unprecedented opportunity. We have a chance to do better by workers (and employers) this time around, and we should take it.

**Note: This is the opinion of the author, and is not necessarily shared by CrowdFlower or, say, its CEO.

John Carney, senior editor at CNBC.com, had an idea for sifting through the data — crowdsource it.

We agree. In fact, CrowdFlower will categorize and tag the first 100,000 documents at no cost to the government.

If you’re just tuning in, the federal Financial Crisis Inquiry Commission (FCIC) subpoenaed Goldman for its AIG transactions, following accusations that Goldman cooked up a mortgage investment scheme that was rigged to fail.

FCIC has around 50 employees, an $8 million budget, and roughly six months to pore over the five terabytes of data. (Can you say, “Too small to succeed”?)

Clearly, technology presents a double-edged sword for investigators and other regulators.

On the one hand, companies under investigation can use technology to more efficiently bury investigators in terabytes of data (paging Goldman Sachs). On the other hand, technology provides tools for deftly sifting through the data (enter crowdsourcing).

Crowdsourcing public documents may be relatively new, but it’s not unprecedented. In fact, the British Parliament is under way with a project that uses crowdsourcing to investigate MPs’ expenses.

We’ll keep you posted on whether the government takes up our offer.

– Additional contributions by Anisha Sekar.

Computer vision researchers work hard to replicate human ability to understand images. It makes sense they are excited about crowdsourcing. Now they can ask hundreds of thousands of people to interpret a few millions images. It becomes possible to scale datasets, collect new types of data and design novel algorithms that rely on people in the loop.

To go over practical details of crowdsourcing for computer vision, Alex Sorokin (CrowdFlower) and Fei-Fei Li (Stanford, Image Net) will present a tutorial on the use of Mechanical Turk and CrowdFlower for Computer Vision at CVPR 2010. The tutorial will walk through basic concepts of crowdsourcing, common issues, and practical examples. The tutorial assumes no prior experience in crowdsourcing or with Mechanical Turk.

Time: 8:30 – 12:30 am , June 18,
Location: Hyatt Regency San Francisco
5 Embarcadero Center
San Francisco, CA 94111

A closely related workshop Advancing Computer Vision with Human in the Loop will happen on Jun 14. Check it out!

But employment law does more than entitle individuals to particular benefits – it also protects them when they act together. The individual right to join together with others is crucial to the ability of individuals to function effectively in groups. Especially where the group is large and, perhaps by virtue of the size, or for some other reason, no individual member has sufficient power to alter his/her situation.

The first and most basic of these rights are the right of free assembly, memorialized in the First Amendment, and the right of free association, which has been read by the courts as a necessary outgrowth of the freedom of speech. Free assembly traditionally refers to peaceful protests or meetings, whereas free association involves joining organizations or associations.

The internet provides a terrific, possibly unmatched platform for free assembly. It’s designed to connect people. Whether it actually lives up to its potential is matter of heated debate among legal scholars, social theorists, and all sorts of other people. But the design, at least, enables forms of free assembly heretofore unimagined. Quite understandably, this represents a threat to governments, large employers, public companies, trade associations—basically any entity whose power relies on the isolation and relative inertia of its constituents. Networked groups of people attempting to bring their collective power to bear often encounter aggressive and dramatic opposition. Most often, we see the drama play out in the IP and free speech arenas, not the online workplace. But that’s not because online work is somehow antithetical to collective action; it’s just that online work hasn’t been around very long.

When workers (online or not) take collective action, they enjoy the some of the same free assembly and associational rights that everyone else has, plus a few other protections and restrictions unique to the employment context. For example, the nature of the employment relationship (as it is legally understood) pretty much precludes any absolute right to free assembly. Employers generally have the authority to decide where a worker must be during worktime, and to prohibit any unauthorized meetings or gatherings. This employer authority flows from the employer’s role as “master” in the traditional “master-servant” construct,* which we Americans inherited from English law and have embraced wholeheartedly. If, for some reason, the master-servant structure does not give employers control over when, where, and how employees can assemble, employers can usually still invoke their property rights regain that control. After all, most employment takes place on the employer’s property.

Of course property rights are notoriously difficult to define in cyberspace. Like employment laws, property laws (apart from those dealing with intangible property) evolved in reference to physical objects and spaces. For twenty years courts have struggled with the proper application of these principles to virtual space, and even so the law remains somewhat murky.

Say a group of workers want to enlist the rest of their colleagues in some cause. If they do so by going onto employer property together, holding up signs, distributing literature, asking for signatures, etc., the employer would have the right to prohibit such solicitation or exclude them on grounds of trespass (except in certain situations protected by the National Labor Relations Act). But what happens if they instead use the company email system, message board, or chat room to accomplish the same task?

In Intel v. Hamidi, an employee (Hamidi) used company email to broadcast his complaints about the employer (Intel). Intel claimed, among other things, that Hamidi’s emails constituted ordinary trespass and trespass to chattel (which is essentially trespass on personal possessions, as opposed to real estate). Intel argued that it owned the email system and the servers through which email traffic would pass, and that use of that system and those servers represented a violation of its property rights. The California Supreme Court agreed, to an extent. It recognized that an email could amount to a trespass to chattel if the owner could show actual or threatened damage to computer hardware or software, or interference with its ordinary and intended operation. But the Court explicitly rejected the idea that software or servers could be considered real property, subject to traditional trespass law. According to Hamidi, there is no such thing as trespass on virtual space.

Fortunately for workers, and unions, the National Labor Relations Act picks up where the First Amendment falls away-at the boundary between public and private property. The NLRA protects workers when they engage in “concerted activity for mutual aid and protection,” basically anywhere. “Concerted activity” basically means activity involving two or more employees, or one employee acting on behalf of others or seeking to enlist the support of others. “Mutual aid and protection” ensures that the purpose of the activity is to to improve wages, working conditions, or some other aspect of employment that affects the group. Usually, what we’re talking about is communications (mass or one-on-one), meetings, petitions, protests, and—in extreme cases—picketing, strikes, or boycotts.

Whether or not the conduct is actually protected by the NLRA will depend on whether crowd workers are “employees” for the purposes of the Act (nobody wants to revisit that thorny question from the last post). And even if they are protected, the traditional concept of unionization might not fit particularly well with crowd labor. For one thing, seniority and job security, two of the central tenets of collective bargaining, won’t have much meaning in a high turnover, “open call” labor market. For another, collective bargaining requires a group of workers sufficiently similar to share a “community of interest,” and the National Labor Relations Board (which administers the NLRA) has, like the rest of the court system, proven fairly hostile to the idea of a legally cognizable community in cyberspace.

In the absence of NLRA protection, I predict that we will still see some familiar forms of collective action emerge, as well as some unique to the crowdsourcing milieu. Turkopticon, which allows AMT Providers to rate and comment on Requesters, offers an excellent example of the kind of collective action that can only really exist in online crowd work. Message boards and social networks are further examples of virtual assembly. I expect that as the market for crowd labor grows, independent gathering places for crowd workers will grow in size and organizing capacity.

Crowd workers may follow the example of writers, graphic artists, and other groups generally neglected by organized labor, by establishing a membership-based organization to promote their interests. A crowd worker association could lobby on behalf of crowd workers, attempt to establish group benefits, handle disputes, inform crowd workers of their legal rights, and serve as a clearinghouse for campaign activities. Such organizations tend not to follow the traditional union model, instead tailoring themselves to the specific dynamics of their industry. Crowdsourcing would seem ideally suited for an industry-specific worker association.

Some crowd workers will probably go one step further, taking direct action to raise their compensation and change their employment terms. We know how much networked communities depend on the presence and contribution of the crowd. Crowd workers could withhold labor from untrustworthy employers, adopt voluntary restrictions (such as a blanket refusal to perform tasks below a certain wage level), or create worker associations to provide training and “accredit” responsible employers. I also anticipate some version of the “work to rule” strategy, in which employees attempt to hobble an otherwise smoothly functioning system by ignoring the way things actually run and instead adhering rigidly to the rules.

When faced with an organized group that already communicates primarily online, crowdsourcing vendors might prove more responsive than we might expect.

*More on masters and servants here.

Another interesting thing about the Gambit workforce is how similar its composition is to the Turk workforce of two years ago, in which 80 percent of workers were from North America (the United States and Canada). (The exact results and methods for determining this is discussed below). This similarity points to how labor workforces are globalizing very quickly with Turk (one of the first marketplaces for online work) at the forefront of this growth. Gambit could possibly be moving in this direction as well. For now we’ll leave this discussion to a later blog post, as first we must see the results to this survey.

Survey Methodology and Results

Though we ran the same survey, asking the same questions, we reworded some questions for the Gambit workforce, specifically:

• Age (year of birth)
• Gender
• Educational level
• Income level
• Marital status
• Questions about their engagement on Gambit
• How often they do tasks
• Income from tasks
• Why they do tasks/other comments

Again, we ran the survey over a 24 hour period and used the same script to make surveys available uniformly. This worked well—responses came in at about the same rate, 50 responses an hour, as in our Turk survey.  As we did in our Turk survey, we must note possible confounding factors and selection bias, i.e. groups who work on weekends as opposed to during the week, workers who do not participate in surveys. In fact, because Gambit provides offers through online games, a strong selection bias against surveys exists, as opposed to other tasks, because they may be more closely associated with product spam, which is universally abhorred.

As we did with our Turk survey, we will focus on where Gambit workers come from, but because scarcely anything is written on who Gambit workers are, here are some graphs.

A very large majority of Gambit workers self-reported that they were from the US, 83.27 percent, while another 6.57 percent said they were from North/South America. When geocoding the users’ IP addresses we found that 85.23 percent of responses came from the US and 8.53 percent from Canada. The other countries represented in this geocoding process were Australia, England, Indonesia, Singapore, Malaysia, and the Philippines. In contrast, on Turk we saw a much larger set of countries represented in our survey. It is interesting that the Gambit distribution resembles that of Turk from two years ago, where 80 percent of workers were from the US, UK, and Canada.

We might expect a skew towards English speakers because they are one of the largest target audiences for virtual currency offers. However, this survey is in English and, hence, self-selects people from large English speaking countries. We have run non-English language tasks before on Gambit and have noticed a majority of those workers came from regions which spoke the language, particularly France. This confounding factor exists for our Turk survey as well, as we have noticed that our German Turk tasks almost exclusively consists of workers from Germany. These tasks are run without restrictions so anybody can do the work, but we only considered workers who met our quality standards and were not rejected.

The best way to find out worker locales’ may be to ask Gambit and Amazon to track and share this information directly. Barring this, running a “free money” task is our best bet to attract as wide an audience as possible and then geocode IP addresses to determine workforce distribution across countries.

Here is the gender breakdown:

As we can see Gambit workers are mostly female. This may be due to where these tasks presented i.e. next to advertisements and offers in online games. However, we cannot say for sure that this is the reason.

These next few graphs show distribution of age and education level.

Lastly, we also asked the workers why they do our tasks and their responses fell into a number of categories:

• Boredom, to kill time
• To earn extra money for hobbies, fun days, etc.
• Earn online currencies (e.g. Swag Bucks)
• More productive use of time as opposed to say TV, video games
• They are fun to do

Gambit workers shared the following comments about being bored and wanting to feel productive with their time:

• “My husband and I work at the same place. I just answer phones and babysit kids when their parents are buying a piano from my husband. I do these to keep from being bored to death.”
• “I figure that if Im just sitting around the house watching TV, I might as well try to do something productive”
• “I find this a fun way to kill time – there interesting and i love earning swagbucks”
• “i just like to keep bussy all the time!”

Users really love their Swag Bucks, too:

• “I get bored and I have nothing else to do, or I would like to earn some more Swagbucks.”
• “These tasks help me to add swagbucks to my account and I actually get credited for them whereas the special offers never seem to acknowledge that I’ve completed an offer and will not give me my earned bucks.”
• “Swagbucks offers some nice magazine subscriptions and I love summer reading for free.”
• “I like to see my swagbucks account growing”
• “I do tasks as a way to try and earn extra Swag Bucks.”
• “I do these tasks for Swag Bucks. So, I receive no actual compensation.”
• “i do them to earn swagbucks. sometimes i try to get as many SBs in one day as i can. tasks make a great filler. i never do the grouping tasks. i think they need better instructions, or maybe it’s just one of those things that other people are better at than me.”

Earning Swag Bucks is a huge motivating factor which we cannot underestimate. In fact when we compare the quality of work from Gambit to work from Turk on HITs where we know the answer, we get about the same ratio of correct answers to wrong answers (about four to one in both cases). We use mistakes to help train workers by telling them why they got something wrong so they can avoid making such mistakes on future HITs. For Swag Bucks, people do correct work so they do not get kicked out of our system. As Labor-on-Demand continues to grow we need to understand how to best incentivize workers, possibly with Swag Bucks, so we can get more work done for meaningful tasks.

Lastly, we noticed how similar Gambit’s workforce was to Turk’s workforce from a two years ago (based on Panos’s previous survey). These trends in the evolution of workforces point to the rapid growth and globalization of the industry. We’ll have to watch these trends closely to see how crowdsourcing/Labor-on-Demand/Labor-as-a-Service continues to evolve/develop.

In a later blog post, we’ll also take a look at Samasource‘s workforce.

-John

By way of a recap for those with the good fortune of not having had to endure my fleeting law professor fantasy, I’ve broken up the presentation into a few parts, which I’ll post here. This segment looks at the basic reasons why applying our current law is so difficult, and the threshold question of how to properly classify crowd workers. The next one will deal with civil liberties and group rights in virtual work, and the final one (if I get that far) will present my best case for immediate regulatory intervention. If you don’t want to wait, you can find a thorough (and probably boring) treatment of this topic here.

The Basic Problem: Old, Stale Law

Most of our employment laws – the ones dealing with minimum wage, union organizing, health and safety, etc. – were written in middle of the 20^th Century, in response to a period of rapid industrialization. The people who wrote these laws had a particular concept of employment in mind, one characterized by physical worksites, long-term employment relationships, and one-to-many connections between firms and workers. By the 1990s (if not earlier), the rise of contingent and temporary labor, complex subcontracting structures, and remote work had substantially eroded the principles relied upon by the original drafters of the law. But nobody did much of anything to update that law, and as a result, huge swaths of people whose work would have been covered under the employment arrangements of the previous generation just fell into a regulatory vacuum. (For a dense but compelling description of this process, and its legal implications, check out Katherine Stone’s From Widgets to Digits).

Distributed work, microwork, crowd work, whatever you want to call it, threatens to basically nullify existing employment laws, and the 20^th Century conception of employment upon which they were based. That’s why, when you ask questions like “Should Turkers be entitled to minimum wage?,” no one seems to have a very good answer. Most people confronted with that question revert to considering the Turkers themselves, and trying to decide whether a college student or a stay-at-home mom “deserves” the legal protections afforded to more traditional employees, rather than trying to assess whether the work itself should fall within the scope of statutory employment.

But someone is going to have to answer the hard legal questions eventually, because the industry is growing too quickly to be ignored. It might be the courts, it might be the legislatures, it might be the IRS or the Department of Labor, but one way or another our regulatory systems will be forced to take a position.

Twenty Thousand Independent Contracts?

The first big threshold question is whether crowd workers are independent contractors or statutory employees. To lawyers, “employee” doesn’t just mean “someone who works for me.” “Employees” are a creature of statute, with a special meaning. They are a particular class of workers who, because of their lack of power in relation to their employers and their various other vulnerabilities, have received specific protection from Congress and the states in the form of minimum wage, organizing rights, etc. By contrast, independent contractors (at least in theory) are specialist entrepreneurs who bargain for their services and execute contracts to perform those services. They should have no relationship with the employer apart from the contract (hence, “independent”), and any disputes that come up should be decided by interpreting the contract language, under contract law.

This distinction between independent contractors and statutory employees really matters. Independent contractors are not covered by federal or state minimum wage/overtime laws, nor are they covered by OSHA (health and safety), Title VII (employment discrimination), FMLA (family leave), or the NLRA (union organizing). And they don’t receive workers compensation or unemployment unless they pay into the state funds themselves.

Now let’s look at a typical crowdsourcing platform to see where the workers fall. Here’s simplified structural diagram.

We know what governs the relationship between workers and vendors, and what governs the relationship between firms and vendors – the terms of use that you have to sign in order to gain access to the platform (aka the participation agreement). But we don’t know much about the relationship between workers and firms, and it’s the nature of that relationship that determines whether a worker is an employee or an independent contractor. Unsurprisingly, the vendors try to fill in the content of the worker-firm relationship by inserting a bunch of language into the only legal document that does exist: the participation agreement. One of the most important “inserts,” and the vast majority of crowdsourcing vendors use it, is a requirement that crowd workers provide services as independent contractors. The typical language goes something like this (from oDesk)

“Provider’s relationship with Buyer will be that of an independent contractor, and nothing in this Agreement should be construed to create a partnership, joint venture, or employer-employee relationship.”

For a more comprehensive list, check out this slideshow. Fortunately for crowd workers, the labels put on the relationship by the parties, or by some third party, aren’t the end of the story. Courts go further, trying to understand whether the relationship is really an independent contract or an employer-employee scenario. They consider a variety of factors, depending on what law the parties are trying to apply. For example, the Department of Labor suggests a seven-factor test for application of the Fair Labor Standards Act, the federal law that establishes minimum wage and overtime rules (Alexander Sorokin performed a nice summary of the factors in an earlier post on this blog). But the test changes with the law to be applied, and also with the jurisdiction, the industry/type of work, and the particular situation presented to the court. Some tests focus on the employer’s right to control. Others look at whether the worker has an opportunity to function as an entrepreneur or is dependent on the employer. I won’t go through all the tests here – suffice it to say that there’s a lot of variation, and right now crowd workers fall squarely into the gray area between employees and independent contractors.

Personally, I find the notion of independent contractors quite fitting in some contexts, such as eLance or Guru, and essentially ridiculous in others – such as Mechanical Turk and LiveOps. The concept of a rapid succession of independent contracts, perhaps hundreds in a row for a particular worker, tens of thousands per day on a platform like Mechanical Turk, just runs contrary to the underlying rationale for distinguishing independent contractual agreements from statutory employment. Does clicking an “Accept HIT” button and agreeing to tag ten minutes worth of video truly create a separate, independently bargained contract? I’m not sure such a proposition would pass the laugh test, even for a fairly conservative judge.

The problem is that there aren’t any other categories into which we can place this kind of work. A conservative judge may find ludicrous the idea of a ten-minute long, unbargained, anonymous “independent contract,” but he’s not going to just award a bunch of clickworkers minimum wage when hundreds of thousands of contingent workers and permatemps don’t enjoy the same protection. There simply isn’t a good fit, because employment and labor law has not kept pace with the emergence of our information economy.

Which brings me back to my original point. Our existing law is archaic and broken. We’re all waiting for the first group of Turkers to file a wage and hour class action, so we can find out what happens next.

Imagine for a moment that you were a turker from either the US or India, looking at the above image. You are given the task of clicking on the blue circular objects with red borders. What you see is only a fraction of the full image. Each image has 90 blue objects to identify. If you’re as good as the average worker, you’ll complete your first image in a little over five minutes and you’ll earn 10 cents (for an hourly wage of $1.20).

After your first image, you can either quit and take your 10 cents, or identify points on another image. Over the next four hours, you’ll have the chance to label as many images you want. But there’s a catch—you’ll only be paid 9 cents for the second image, 8 cents for the third, and so on, all the way down to 2 cents. This will lower the hourly wage even more.

Before you even qualify for the task, you’ll have to spend five minutes watching a training video and passing a quiz. During the video, half of you will be given only basic work instructions on how to identify “objects of interest.” The other half will be given both instructions and cues of meaning: recognition for your contribution and an explanation of your task’s purpose¹. The reason given here? To help researchers identify cancerous tumor cells.

We posted these HITs on MTurk in January, 2010. Almost 300 people from the U.S. and India accepted the task, becoming unknowing participants in our experiment examining MTurk worker motivation. It is commonly believed (and other researchers have verified with demographic surveys) that Indian workers are more motivated by pecuniary concerns and that US turkers are primarily doing tasks for leisure or other non-pecuniary motives. Is this true?

In both countries, half of the turkers in the experiment were randomly assigned to label nondescript “objects of interest” without being given any context or greater purpose — they were our zero-context group. The other half, our meaningful group, were told they were helping researchers identify cancerous tumor cells. Which group of turkers do you think worked harder? You might be surprised.

Therefore, our experiment compared two groups with and without a clear wage motivation, to see if workers behave differently responded to meaningfulness in their tasks.

Results

We measured three metrics: “showing up”, the quantity of work, and the quality of that work. The first two metrics are straightforward. Showing up meant that you sat through our training video, passed our qualification test and helped label at least one image. Quantity of work was simply the number of images labeled.

We repeatedly told both groups of turkers that they needed to click on all points and as closely as possible to each point. Work quality was determined by the fraction of cells that a person clicked on (the recall) and the average distance between the “true center” of each cell and where the user clicked (the centrality).

Our most interesting finding was the extent to which a meaningful task (and giving recognition) motivated US workers, but not Indian workers, to complete a task. As any requester knows, attrition on MTurk is a real problem. We found that adding cues of meaning could motivate turkers to undergo training and label at least one image. In the US, adding cues-of-meaning raised the fraction of turkers who completed our task from, 92% of people who sat through our training video, took our quiz, and labeled an image showed up. This figure compares to only 83% of zero-context group (see figure which also has standard errors). In India, there was no difference between the groups and both groups had a 66% completion rate (attrition being higher due to possible language barriers, slow connection speeds, hardware issues, etc.).

However, once a person did some work, both treatment and control groups did a similar quantity of work: The cues-of-meaning group labeled 6.0 images and the zero-context group labeled 5.7 images. This difference was not statistically significant, so it suggests that once you get turkers to work on a task, they are motivated to label just as many images irrespective of the task’s meaningfulness. Notably, of the people who worked, Indians worked longer and labeled an average of 7.3 images vs. 5.2 in the US.

Surprisingly, all workers did an equally good job identifying points whether they had zero-context or whether they thought they were identifying tumor cells. The quality as measured by the fraction of points identified (the recall) or the average pixel distance (the centrality) was statistically insignificant irrespective of the task’s meaningfulness.

This finding has important implications for those who employ labor in crowdsourcing markets. Companies and intermediaries should develop an understanding of what motivates the people who work on tasks. Employers must think beyond monetary incentives and consider how they can reward workers through non-monetary incentives such as by changing how workers perceive their task. Alienated workers are less likely to do work if they don’t know the context of the work they are doing and employers may find they can get more work done for the same wages simply by telling turkers why they are working.

For more details of this study, please see our full academic paper at: http://danachandler.com/index.php/research. We welcome any comments and feedback.

About the authors:
Dana Chandler is a researcher at the University of Chicago’s Becker Center where he works with Steven Levitt, author of Freakonomics. He previously worked as a management consultant at the Boston Consulting Group and at Aureos Capitol, a Colombian private equity company. He will begin his Ph.D. at MIT in the Fall.  Dana’s research interests include digital labor markets, development economics, and randomized experiments in companies. email: dchandler {at} uchicago {dot} edu

Adam Kapelner is currently earning his Ph.D. in Statistics at Wharton. Adam is the founder of  dictionarysquared.com and the inventor of its vocabulary-learning  technology. While working as an undergraduate researcher at Stanford University, he helped engineer the open-source software, www.gemIdent.com, that enables researchers worldwide to locate cells in microscopic images. GemIdent was recently extended to make use of MTurk for outsourcing of medical image identification. The extension, called www.distributeeyes.com, was  adapted to serve as the platform for this experiment. email: kapelner  {at} wharton {dot} upenn {dot} edu

Acknowledgments: We thank Professor Susan Holmes of Stanford University for allowing us to adopt DistributeEyes (funded under NIH grant #R01GM086884-02) for use in this study. We would also like to thank Panos Ipeirotis for kindly providing us with demographic and market data that we cite in our study. Lawrence Brown, Patrick DeJarnette, John Horton, Emir Kamenica, Steven Levitt, Susanne Neckermann, Jesse Shapiro, Jorg Spenkuch, Jan Stoop, Chad Syverson, Mike Thomas, Abraham Wyner, and seminar participants at the University of Chicago provided especially helpful comments.

Here are a few of the colors we asked about.

Lukas gave a presentation today at TEDxDU.  It was a great event with more than 1,000 attendees.

At the beginning of his presentation he told the audience members if they were bored they could work on a CrowdFlower task.  It turns out a lot of people were bored!  The task was describing colors, but with a twist—describe the color as precisely as possible.  You can see the task here.

So far, the most common color reported was a tie between “blue” and “Blue” both were reported 8 times. “purple” came up 7 times, “lavendar” 5, and “Purple” 5.

There are lots of approaches for naming colors – but the award has to go to the respondent who answered in hexidecimal “#F36F36″.  Nice.

Some of the more interesting and long-winded colors descriptions are; “soft peach sunrise through smog, haze or volcanic ash”, “that deep green that everyone tries to describe as a forest, even though no forest is really that deep and vibrant”, and “tanned orange, the color of skin that many high school girls have when they hit the tanning booth too often.”

I’m pretty sure this answerer is from a photographer, “Dark grey or light black. Take true black and lghten it up about 3 or 4 stops.”

Then there is the poet’s response, “It is the smell of spring, the feel of a pulse and taste of a berry.”

We saw responses that were helped along with food or wildflowers; Pumpkin orange, Goldenrod, creamsicle orange. Creamsicles are food, yes?

We’ve created a little graph where the labels are colored by the value of the image we showed. Each label is the aggregate response for that color, and is sized by how many people chose that label when they saw the color.

Thank you for participating and please continue adding to a very important list. May your answers live alongside the esteemed answers; “Brown kind of yukkie yellowish brown”, “Mix of light beige and light green to make a puke brown”, and “Vibrant Utah Blue Sky against frozen red earth sand dunes”.

We used mostly the same questions as Panos’s survey, which asked for:

• the Turker’s age (year of birth)
• gender
• educational level
• income level
• marital status
• questions about their engagement on Turk
• how often they Turk
• income from Turk
• why they Turk

In contrast to Panos’s original survey which was run over a 3 week period, we ran this survey over the weekend in a 24 hour period. Due to this abbreviated weekend running, there will most likely be greater confounding factors as well as stronger selection bias, i.e. groups who work on weekends as opposed to during the week. To help mitigate the timing of the experiment and provide surveys close to uniformly over the 24 hours, we setup a script to release only 50 surveys an hour. Responses generally came at a steady pace and Turkers available at each hour were represented.

Since we’ve examined Turker motivation before, albeit hardly rigorously, I will mostly focus on the rise of India as well other location specific considerations in this post.

In comparison to Panos’s survey the greatest difference in the results was in the distribution of respondents’ location. We found that India made up 46.85% of our respondents, while the US made up 42.7%. In contrast, Panos’s survey (run over a 3 week period in February) had 46.8% of respondents from the US, and 34.0% were from India. To determine if this difference was due to self reporting error, we checked self reported location vs geocoded IP location and found that they matched almost exactly. There were 31 differences out of 1016 survey responses, but these differences could potentially be attributed to ambiguity in the question (“Where are you from?”). Ultimately these differences between self report location and our geocoded IP location were not statistically significant. This is an encouraging result, suggesting Turkers are overwhelmingly honest when answering survey questions about where they are from.

Viewing the responses over time suggests that Turkers work during non-sleeping hours. The graph below shows this pattern. If we were to repeat this experiment we’d want to run this task over a week or two instead of over 24 hours. Because of this abbreviated time period the pattern is not as evident towards the end of the job where work rates declined across the board.

To better test the hypothesis that Turkers generally work during non-sleeping hours of their respective countries, below is a graph showing the distribution of work done for CrowdFlower on Mechanical Turk each hour of a day by continent (specifically Asia, Europe, and the “Americas”) over the course of 6 months. In this 6 months we collected approximately 9 million judgments. In the graph this means if Asia has a point at 6%, 4 AM GMT then 6% of judgments made in Asia came around 4AM GMT.

For each continent the peaks roughly correspond to daytime while the valleys to nighttime, which is what we’d expect. Asia’s peak seems to be more of a plateau, and this is likely due to in part to the number of timezones Asia encompasses. In my post about task localization, we saw (what is intuitively obvious) that workers’ locales are an important factor in assessing quality, especially for language specific tasks. On Mechanical Turk, to hit the right workforce for a language specific task, it is advisable to restrict available hits to certain times to limit the number of responses from countries whose native languages are not applicable. We have to use this round about method for Mechanical Turk because you cannot restrict the workforce to a set of multiple countries, either restrict work to one country or restrict work to all but one country.

We’ve already noted that India represents a sizable and rapidly growing portion of the Turk workforce. We are particularly interested in this rate of growth and the future trends in worker locales. The next graph compares the US to India in terms of monthly volume of judgments completed on our jobs posted to Mechanical Turk in 2010. The location information comes from geocoded IP adresses.

The above shows that on Mechanical Turk we’ve seen an increase in the proportion of our workers who are Indian since December. This sample was collected over a relatively short period of time, and is definitely something we’ll want to monitor in the future. Lastly, I want to emphasize that though this experiment is hardly rigorous and there are many more factors to analyze, Mechanical Turk as well as other vendors of work (Gambit, Samasource, LiveOps, etc.) are continuing to evolve and grow extremely rapidly, and consequently so grows the potential and possibilites for distributed work. Next we’ll examine a survey of Gambit, then Samasource.

John