The votes are in from our Oscar crowdsourcing experiment, and the crowd successfully picked the winners of 14 of the academy awards. For reference, Roger Ebert got 15 predictions correct so we’d have to conclude that the crowd performed reasonably well at predicting the winners of this glorified popularity contest.

Predicted and Actual Winners of the 2011 Academy Awards

One fascinating thing about aggregating responses is that the crowd as a whole will often outperform the average worker. In this case, among the 500 people we polled, the majority of respondents picked fewer than 10 awards correctly (mean of 9.6 and median of 9). And yet, by aggregating all the responses, such that the nominee with the most “votes” is predicted to win, the crowd as a whole correctly picked 14 awards. While the “wisdom of crowds” doesn’t come as much of a surprise, it’s always reassuring to see it confirmed in new applications.

As we noted in our earlier post, though, the more interesting question was whether workers who indicated higher confidence in their responses would outperform workers with lower confidence. Looking at the results, however, we saw no significant correlation between a worker’s predicted accuracy and  actual performance.

"Squint all you want, but there's no pattern"

While it’s certainly possible that we didn’t offer enough of an incentive for workers to estimate their own accuracy, the more likely explanation is that predicting the winners of the Oscars is not something that a person can do with any degree of certainty. Confident or not, the people we polled did not see the “Inside Job” coming.

As a final exercise, we ran a regression on every explanatory variable we could find, including what state workers came from, what day they made their predictions, whether they made their predictions during the day or at night, how long they spent making their predictions and even their historical accuracy on other CrowdFlower tasks. The only variable with any significance turned out to be how long they spent on making their predictions, and while it was significant (at p=0.001), no model we could come up with explained more than 5 percent of the total variation in accuracy.

While the wisdom of crowds seems to extend to picking Oscar winners, the more interesting experiment of having workers self-select as trustworthy is ongoing. In the future, it would be worthwhile to repeat this experiment with questions that can be answered objectively and without uncertainty (solving algebra problems seems like a good candidate), to see if any correlation emerges between predicted and actual accuracy.

Can the quality of crowdsourced work be linked to geography?

In short, the answer is no. Here’s why.

Our team looked at whether including a workforce from a specific region helped or hurt the completion rate and quality of a recent website categorization project. Overall, the project showed a common trend, where the quality of work improved over time, regardless of geography.

The figure above shows the overall volume of judgments (blue) plotted against trusted judgments (green). In addition to the large amount of untrusted work over the first two days, this graph shows spikes in volume that correspond with time of day. Two possible explanations for the change in quality are:

1. Only good workers were able to continue working after the first few days.
2. A flood of bad work from specific geographies, related to the spikes in throughput at certain hours.

“Just a Taste” Workers

One of the ways that CrowdFlower maintains quality is by incorporating questions with known answers, and tracking worker performance on these units as a proxy for overall accuracy. If workers don’t maintain a minimum level of accuracy, they are prohibited from continuing work on any given task.

In this job, it appears that many workers were unable to meet our quality standards. This was a somewhat subtle categorization job, characterizing the content of websites according to handful of criteria, so it’s not surprising that many workers had trouble.

As a result, many workers did a relatively small amount of work, but they could not continue because they didn’t meet our quality standards. The figure below shows the number of workers completing a specified number of judgments.

This graph demonstrates a known behavior in online tasks, where many workers attempt only a small amount of work before abandoning a given task. Our quality-control mechanism requires workers to demonstrate accuracy before it accepts their work.

While it is interesting that our quality control identified certain high accuracy workers and allowed them to continue, this doesn’t answer the question of why there were such dramatic changes in throughput during the day.

Non-US Workers

Especially over the first two days, the relative amount of untrusted work was much greater between 10 p.m. and 10 a.m. (Pacific Time), which is when we see a preponderance of work coming from workers outside of the United States. Indeed, after filtering workers by IP address, we saw that 73 percent of all workers in this job came from India. On the other hand, the region’s workers accounted for only 46 percent of trusted workers.

However, while these workers account for a relatively small proportion of trusted workers, they did much better in terms of trusted work submitted.

This workforce accounts for fully two-thirds of the total trusted work on this job. While that is somewhat less than their overall representation in the labor pool, this suggests that we can’t dismiss these workers as low-quality.

Pareto Me This

All of this raises a very interesting observation. As you may have seen elsewhere¹, a relatively small minority of people often account for the vast majority of observed effects. This is common in crowdsourcing, just as it was in terms of land ownership in 19th century Italy.

In this example, the top three percent of most prolific workers provided over 40 percent of trusted work. The top 20 percent of workers provided over 80 percent of trusted work.

Keep the Bums Out

Focusing on the top 20 percent of most prolific workers, we see that one-half came from India while another one-third came from the U.S. Other countries provided the remaining workers. While it is true that the vast majority of untrusted workers in this job, who collectively provided relatively few judgments, came from India, it is also true that the country’s workers make up half of the most prolific workers and provided two-thirds of all trusted work.

The solution to improving the efficiency of this job, then, is not the crude choice of excluding workers from certain geographies. Rather, we can discourage bad workers by increasing the burden of entry, so that only workers with an interest in completing more than a few judgments will bother with the job.

Writing is easy. Just sit in front of a typewriter, open up a vein and bleed it out drop by drop.
– Red Smith

Underwood Five Typewriter

When I was in college, a professor I respected said that one of the best ways to demystify writing is to write like people you admire. Specifically, to find passages that you love, and try to revise them in your own words. This exercise proved invaluable. It allowed me to walk in their literary footsteps, shedding light on why they chose — or avoided — certain words, punctuation, and syntax.

With this in mind, I recently wondered whether you can crowdsource writing, specifically, revising.

I posted a task through CrowdFlower that asked the crowd to rewrite four famous quotations, pithily, while preserving their meaning.

In one evening, I was able to get 20 revisions of each quotation from people across the country. I won’t summarize them all here, but I will pull a few highlights.

Original Quotation 1 (from Macbeth, William Shakespeare):

To-morrow, and to-morrow, and to-morrow,/ Creeps in this petty pace from day to day,/ To the last syllable of recorded time;/ And all our yesterdays have lighted fools/ The way to dusty death. Out, out, brief candle!/ Life’s but a walking shadow, a poor player/ That struts and frets his hour upon the stage/ And then is heard no more. It is a tale/ Told by an idiot, full of sound and fury/ Signifying nothing.

Revision 1:

Time marches on, and everyone dies; life is meaningless.
– Hatboro, PA

Life creeps along and ends suddenly like the end of a bad play. The play is dramatic and had poor acting, and has no point or moral in the end.
–Salt Lake City, UT

Original Quotation 2 (from “The Leviathan,” Thomas Hobbes):

No arts, no letters, no society, and which is worst of all, continual fear and danger of violent death, and the life of man solitary, poor, nasty, brutish, and short.

Revision 2:

The life of a man on his own is barbaric and degrading.
– East Aurora, NY

No art, letters or society. Worst of all, living in fear of being alone, poor and short.
– Arlington, TX

Life is all but a mere scam.
–Overland Park, KS

Original Quotation 3 (“First Inaugural Address,” Abraham Lincoln):

We are not enemies, but friends. We must not be enemies. Though passion may have strained it must not break our bonds of affection. The mystic chords of memory, stretching from every battlefield and patriot grave to every living heart and hearthstone all over this broad land, will yet swell the chorus of the Union, when again touched, as surely they will be, by the better angels of our nature.

Revision 3:

We’re fools for fighting each other. We should co-operate, and let our example bring everyone together.
– Plattsburgh, NY

We must be friends, we should now forget each other. Our memories will always stay through our rough times and good times.
– Tallahassee, FL

Original Quotation 4 (from “Chicago,” Carl Sandburg):

“They tell me you are wicked and I believe them, for I have seen your painted women under the gas lamps luring the farm boys.”

Revision 4:

Your reputation follows you and it is a bad one, makeup and lust.
– Iola, WI

This city isn’t a nice place, it’s full of hookers.
– Milledgeville, GA

The full data is available here.

What’s your opinion? Can the revision process be crowdsourced?

I’d love to hear your thoughts.

“I know it when I see it.” — Justice Potter Stewart

We have been running Crowdsifter, our content moderation product backed by Amazon’s Mechanical Turk for a while and we wanted to share some quality metrics and some stats on how our system aggregates redundant results to improve those metrics.

Controlling for Worker Quality, Bias, and Item Difficulty

In the graph above we picked the the best error rate for raw AMT with 1-11 workers and the best error rate that Crowdsifter provided on a porn judgment task with 2491 images (1006 porn, 1485 non-porn). The error rate is the rate at which wrong decisions are made. A wrong decision is whenever we label porn as not porn or non-porn as porn. The above experiment includes images which were labeled as ambiguous, which is the reason the error rates shown seem so high.

Using Crowdsifter with an average of 3.93 workers per image we achieve the same possible minimum error rate as majority voting in raw AMT with 9 workers per image. We do this by controlling worker quality by keeping track of their judgments. And if we have a “expert” evaluated gold standard of what is pornographic, then we can keep track of which workers are doing a good job or a bad job. On non-gold standard images we weight workers’ judgments based on how well we trust their judgment to reflect our standard of porn. Without these controls, majority voting in raw AMT is vulnerable to the many scammers that lurk there.

For images where obscenity is particularly ambiguous, we can allocate more workers. This results in a better sampling of whether an image is obscene. Some images don’t need many judges to accurately determine if they are pornographic. We can determine which images are easily classifiable as porn by sampling a group of workers and checking whether they all agree. Using too many judges per image can become prohibitively costly. It is important to have this scheme so we can dynamically allocate workers. Raw AMT is both wasteful and inefficient, applying many judgments to easy items, while not using enough judgments for hard items.

Better Measures

The raw error rate includes both images incorrectly labeled as porn, and incorrectly labeled as non-porn. In content moderation we want to minimize our porn miss rate (also known as false negative rate) because we don’t want to let any porn onto our site. The graph for the porn miss rate corresponding with the above graph is shown below.

The most important part is the porn miss rate, and our rate is close to the rates of 9 to 11 workers per image on AMT, even though we are using less than half that number of workers, meaning we significantly cut our costs.

Adjusting Thresholds
We can adjust our certain thresholds to lower the porn miss rate, but we do this at the risk of labeling all our images as porn, so nothing would make it onto our site. Adjusting the threshold to meet the needs of minimizing the porn miss rate, while maintaining an acceptable non-porn miss rate, is a task Crowdsifter can readily handle.

We’ll save what we can do with threshold adjustment for a later blog post.
-John

Rion Snow presented the paper, “Cheap and Fast – But is it Good?” at EMNLP last week.

Here are the slides from the talk:

Update 9/19: Final PDF version has been uploaded. See also the comments below for updates — our released data is already being used by others!

We rigorously tested the quality of AMT responses for several classic human language problems, and found that the quality was the same or better than the expert data that most researchers use. We wrote a paper, “Cheap and Fast — But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks,” that will be presented in an upcoming conference, EMNLP-2008.

Our findings:

1. Turker-generated data is good. AMT makes it easy to ask many people for judgments, so for several tasks, we looked at accuracy rates for how well the averaged Turker judgments correlate to the expert gold standard. With more judgments per example, accuracy increases. For comparison, on each graph the horizontal dotted line indicates the rate at which a single expert agrees with their gold standard. Enough non-experts can match or often beat experts’ reliability.

2. Turker-generated data is cheap and fast. We can collect thousands of labels per dollar and per hour.

3. Expert data enhances individual Turker data. First off, individual workers have differing accuracy rates:

So we implemented a statistical technique where we test their accuracy on a portion of the experts’ gold standard data, then reweight votes by worker reliability. This yields higher aggregated accuracy. (Also see our related threshold calibration post.)

4. Turker data enhances NLP systems. For one of the tasks, predicting the emotions elicited by a newspaper headline, we wrote a simple machine-learned classifier and trained it on the Turker data. It easily outperforms one trained on expert data. (There’s a subtle effect here; see the paper for details.)

We’ll update this blog post with a link to the final version of the paper in the coming weeks. Many thanks to our friend Rion, who spearheaded this collaboration. The current version of the paper is here:

• Rion Snow, Brendan O’Connor, Daniel Jurafsky, Andrew Y. Ng. “Cheap and Fast — But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks.” EMNLP-2008.

[ This article is part of a series, Wisdom of Small Crowds, on crowdsourcing methodology. ]

This is a follow-up to the previous post on individual workloads and rates. Here are the submission times and durations for every worker on the same graph. Each worker is one horizontal line. An assignment is started at a dot, and its duration is for the line segment extending to the right.

The particular data set isn’t the same as in the previous post, but was for a similar task and exhibits a similar structure. Worker rates substantially differ. Some workers do a few HIT’s, but others work on as many as are available. Some work rapidly with breaks (19, 36). Some assignment durations are as long as 5-10 minutes (13, 37). Some work very intermittently (29).

This view makes the parallelism of AMT apparent. At any vertical timeslice you can see how many workers are active at that time. The entire job ends on the right side when the available HIT’s run out.

[ This article is part of a series, Wisdom of Small Crowds, on crowdsourcing methodology. ]

[ Update: see also another visualization of this. ]

AMT’s great new interface makes it easy to download completion times for individual worker assignments. Therefore, it’s easy to visualize :) For a recent small job we did (250 HIT’s, 5 workers per HIT), here’s a graph of completion times per worker, over the entire 15 minute duration of the job. Each assignment is a single point, graphed by when it was done versus how long it took.

Most workers come in, do a string of HITs, then leave. Some do all of the HITs available. There seem to be two distinct work modes. Most people do lots of HITs in rapid succession. But several of them work slowly (e.g. workers 8, 13, and 18, with more horizontal space between points), either spending more time on each assignment, or perhaps leaving then coming back.

This graph also illustrates a common trend we see: lots of the work gets done by “tail” workers; that is, people who do only a small amount of work. This is where crowdsourcing really shines — it’s OK if individuals give you a small number of judgments, because you can aggregate across many of them. The total “prolificness” of each worker was lightly skewed on this task — 50% of the work was done by 8 out of 37 workers. Usually, we see a split more like 50% of the work being done by the top 10% of workers; this one had a more even distribution probably because it was small, so enthusiastic workers didn’t have an opportunity to do a very large number of HITs.

Another phenomenon: some workers have a downward trend in work time. This could be learning to do the task faster, or it could be increased carelessness. A quality analysis (along the lines of part 1) can flesh this out.

The task was a fairly subjective image classification problem where positives are rare; purple points are “YES” responses. Responding “YES” takes more time (presumably, more cognitive load) — average work times for YES vs NO responses are 22 vs 12 seconds, significant at t-test p<.001.

-Brendan

p.s. The graph is due to the R’s awesome lattice package. It’s incredibly easy to use: not much more than xyplot(WorkTimeInSeconds ~ SubmitTime | WorkerId).

[ This article is part of a series, Wisdom of Small Crowds, which focuses on crowdsourcing methodology for Amazon Mechanical Turk-like systems. ]

[ This article is part of a series, Wisdom of Small Crowds, which focuses on crowdsourcing methodology for Amazon Mechanical Turk-like systems. ]

We use Turkers to classify all sorts of data, by having several workers render judgments on each item. But what should we do when they disagree? Like any other human behavior, Turker judgments are noisy: sometimes there are mistakes, and sometimes the task is genuinely difficult or subjective, and there is no “right” answer. Once we have a bunch of Turker judgments, we need to aggregate them — that is, use some sort of voting mechanism — to give as accurate a classification as possible. It turns out that one simple trick, threshold calibration, can substantially improve accuracy, and can be tuned to the specifics of the problem.

Here’s an example. A recent client of ours had a de-duping task: given a pair of similar articles, the task was to decide if they were “about the same topic” or “about different topics”. This is just a binary classification problem; call these labels “YES” and “NO”. To figure out how well Turkers could perform the task, we had our client provide us with a gold standard data set. That is, for 135 examples, their experts did the task themselves and provided “gold” ground truth labels.

We used a very high number of workers per example (about 20). For all 135 examples in the gold standard, the following graph plots them vertically by their “Turker confidence in YES” — that’s just the percentage of votes for “YES” among the 20 or so judgments for that particular example. I’ve also colored each example with the experts’ gold label. You can see that this simple Turker data provides some statistical separation between the classes.

This graph also shows how to create a classifier from Turker votes. We have to choose a confidence threshold for our classifier’s decision: above the threshold, say “YES”, and below say “NO”. Unfortunately, Turkers aren’t perfect at modeling the experts: anywhere we place the threshold, errors occur. However, some thresholds are better than others. The threshold with the best accuracy is at 73% confidence — that is, a 73% super-majority voting rule — and it classifies instances correctly 90% of the time. Furthermore, we can tune for different types of errors. If we are particularly concerned with avoiding false positive errors, we can set a higher, more conservative threshold; or, if we want to find as many “YES” instances as possible, we can set a lower, more liberal threshold.

Here’s another chart that more carefully details the tradeoffs between true and false positives vs. true and false negatives. For a particular decision threshold, it shows how it divides up the instances into the confusion matrix’s 4 categories of correct and incorrect decisions.

A final note on why threshold calibration is important: For this task, the Turkers were considerably more liberal than the experts at deciding what a “YES” example was — experts marked only 36% of examples as “YES”, whereas a simple Turker majority voting rule marks 57% that way. This is because the experts understood the full implications of the decision, which were substantial — various entries in their database and website would be merged, and users would be confused if they were exposed to a bad merge. False positives had a very high cost. The prompt for Turkers, by contrast, was fairly vague. (In our experience, we generally find that good task design is a huge factor in getting better Turker accuracy.) However, since Turker decisions noisily correlate with the experts, moving the decision threshold can help accuracy. Here’s the threshold vs. accuracy graph:

Statistical analysis of Turker data can substantially improve accuracy performance, even with something as simple as choosing the best decision threshold. This blog post only scratched the surface; there are a few more useful things to consider. Stay tuned for Part 2 and hopefully many more!

A few more notes on Turker voting and threshold calibration:

An interesting question is the upper bound of possible performance on the task. A good experiment to try is to have two experts independently perform a task, and check their agreement rate. We should be satisfied if Turkers can match experts as reliably as experts match each other. That is, for this task, if experts agree no more than 90% of the time, then Turkers perform the task as well as experts. (We didn’t have this particular experiment done in this case, but I’d be very curious to see the results!) In general, agreement rates can help indicate the difficulty of a task. If expert agreement rates are low, it can be argued that the task is not very “real”.

The terminology of “true/false positives”, “true/false negatives”, “precision” and “recall” are all part of a statistics/machine learning mini-field of binary classifier evaluation. Any statistical classifier that outputs a confidence value or ranking among instances (Naive Bayes, logistic regression, IR ranking, etc.) can be subject to this sort of threshold analysis. A decent place to read more is the ROC Wikipedia page. ROC and Precision-Recall curves have long been used to show thresholding tradeoffs. I think the above plots make it easier to interpret the basic information, but the more traditional graphs are also useful. Here they are for this data (provided courtesy of the excellent ROCR package):

A nice overview of these topics can be found in these lecture notes from William Press.

-Brendan