Update 9/19: Final PDF version has been uploaded. See also the comments below for updates — our released data is already being used by others!

We rigorously tested the quality of AMT responses for several classic human language problems, and found that the quality was the same or better than the expert data that most researchers use. We wrote a paper, “Cheap and Fast — But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks,” that will be presented in an upcoming conference, EMNLP-2008.

Our findings:

1. Turker-generated data is good. AMT makes it easy to ask many people for judgments, so for several tasks, we looked at accuracy rates for how well the averaged Turker judgments correlate to the expert gold standard. With more judgments per example, accuracy increases. For comparison, on each graph the horizontal dotted line indicates the rate at which a single expert agrees with their gold standard. Enough non-experts can match or often beat experts’ reliability.

2. Turker-generated data is cheap and fast. We can collect thousands of labels per dollar and per hour.

3. Expert data enhances individual Turker data. First off, individual workers have differing accuracy rates:

So we implemented a statistical technique where we test their accuracy on a portion of the experts’ gold standard data, then reweight votes by worker reliability. This yields higher aggregated accuracy. (Also see our related threshold calibration post.)

4. Turker data enhances NLP systems. For one of the tasks, predicting the emotions elicited by a newspaper headline, we wrote a simple machine-learned classifier and trained it on the Turker data. It easily outperforms one trained on expert data. (There’s a subtle effect here; see the paper for details.)

We’ll update this blog post with a link to the final version of the paper in the coming weeks. Many thanks to our friend Rion, who spearheaded this collaboration. The current version of the paper is here:

• Rion Snow, Brendan O’Connor, Daniel Jurafsky, Andrew Y. Ng. “Cheap and Fast — But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks.” EMNLP-2008.

[ This article is part of a series, Wisdom of Small Crowds, on crowdsourcing methodology. ]

We all know crowds can tell us the weight of an ox, but can they help us in bed?

With artists Fernanda Viegas and Martin Wattenberg, we took pictures of the nude human body, and asked people online to rate how much they’d like to touch, or be touched at, different positions all over the body. Here is where men most like to be touched, front and back:

We got this data by posting a survey on the Amazon service, Mechanical Turk — an online marketplace for simple tasks — and offered to pay a few cents for a hundred ratings. In a few days we collected more than 30,000 responses over 700 body positions, from about 280 different people. The above image is from Fernanda and Martin’s Skin to Skin visualization, you can compare silhouette heatmaps of the data. For example, from the woman’s back, here’s a comparison of where women want to be touched, versus where men (and some women) want to touch them.

Some patterns jump out — for example, the back of the neck seems to be a neglected area. (Take note!) But otherwise, desire to touch and be touched seem pretty well matched up.

What’s also interesting about this experiment is that, despite the subjectiveness of the task and large variability in the data, patterns still emerge. To get some idea of how noisy the data is, here are rating histograms for several of the “desire to be touched” locations for women. (You can explore all body locations with the Sorting Out Desire visualization.)

We had from 20 to 50 responses for most positions. While there are clear differences between these locations, there’s enough noise such that a large amount of data is necessary to discern the signal. This is exactly the same finding we have seen with other types crowdsourcing work. If we didn’t have access to such a large and available-on-demand pool of participants, we wouldn’t have gotten any coherent results at all.

Finally, we collected demographic data from the raters. 10% reported as gay, and 19% as bisexual. Interestingly, straight men are the least likely to report wanting to be touched. (Feel free to hypothesize why!)

We also asked for the age of the rater. What ages have more desire? Is there truth to the stereotype of a lecherous old man? Our evidence is mixed. Below are average ratings when answering “How good would it feel to touch this area?” People in their twenties and thirties give higher ratings, but older men give higher responses than do women — even considering that men give higher responses overall.

Many thanks to Fernanda and Martin for approaching us with this fun idea. This has definitely been the most ludicrous task we’ve ever worked on :) Take a look at this — and other great work — at their site, Fleshmap.com.

-Brendan

On this blog we’ve used crowdsourcing techniques to study media bias, linguistics of color, social perceptions, information retrieval, etc.

But this is thinking small. The ability to quickly and easily collect data from thousands of people on the Web should allow us to study a huge swath of human behavior. Therefore, the next obvious topic is… human sexuality :-) Here are two important questions many of us would like to know the answers to:

• Where do women enjoy being touched?
• Where do men enjoy being touched?

The Internet has many opinions on these questions. But wouldn’t it be nice to have a concise statistical-visual answer?

We’re on the job! With visualization artists Fernanda Viegas and Martin Wattenberg, we’ve pinpointed thousands of locations on a photograph of a naked body and asked respondents how much they like to touch, or would like to be touched, at that spot. (Fernanda and Martin are working on an art project on this theme, to be unveiled next week).

Are there places that men want to touch but where women do not want to be touched? Vice versa? Soon the world will know.

What do you think the results will look like? We will be publishing the results next Monday, so follow this blog for updates!

-Brendan

This is a follow-up to the previous post on individual workloads and rates. Here are the submission times and durations for every worker on the same graph. Each worker is one horizontal line. An assignment is started at a dot, and its duration is for the line segment extending to the right.

The particular data set isn’t the same as in the previous post, but was for a similar task and exhibits a similar structure. Worker rates substantially differ. Some workers do a few HIT’s, but others work on as many as are available. Some work rapidly with breaks (19, 36). Some assignment durations are as long as 5-10 minutes (13, 37). Some work very intermittently (29).

This view makes the parallelism of AMT apparent. At any vertical timeslice you can see how many workers are active at that time. The entire job ends on the right side when the available HIT’s run out.

[ This article is part of a series, Wisdom of Small Crowds, on crowdsourcing methodology. ]

[ Update: see also another visualization of this. ]

AMT’s great new interface makes it easy to download completion times for individual worker assignments. Therefore, it’s easy to visualize :) For a recent small job we did (250 HIT’s, 5 workers per HIT), here’s a graph of completion times per worker, over the entire 15 minute duration of the job. Each assignment is a single point, graphed by when it was done versus how long it took.

Most workers come in, do a string of HITs, then leave. Some do all of the HITs available. There seem to be two distinct work modes. Most people do lots of HITs in rapid succession. But several of them work slowly (e.g. workers 8, 13, and 18, with more horizontal space between points), either spending more time on each assignment, or perhaps leaving then coming back.

This graph also illustrates a common trend we see: lots of the work gets done by “tail” workers; that is, people who do only a small amount of work. This is where crowdsourcing really shines — it’s OK if individuals give you a small number of judgments, because you can aggregate across many of them. The total “prolificness” of each worker was lightly skewed on this task — 50% of the work was done by 8 out of 37 workers. Usually, we see a split more like 50% of the work being done by the top 10% of workers; this one had a more even distribution probably because it was small, so enthusiastic workers didn’t have an opportunity to do a very large number of HITs.

Another phenomenon: some workers have a downward trend in work time. This could be learning to do the task faster, or it could be increased carelessness. A quality analysis (along the lines of part 1) can flesh this out.

The task was a fairly subjective image classification problem where positives are rare; purple points are “YES” responses. Responding “YES” takes more time (presumably, more cognitive load) — average work times for YES vs NO responses are 22 vs 12 seconds, significant at t-test p<.001.

-Brendan

p.s. The graph is due to the R’s awesome lattice package. It’s incredibly easy to use: not much more than xyplot(WorkTimeInSeconds ~ SubmitTime | WorkerId).

[ This article is part of a series, Wisdom of Small Crowds, which focuses on crowdsourcing methodology for Amazon Mechanical Turk-like systems. ]

[ This article is part of a series, Wisdom of Small Crowds, which focuses on crowdsourcing methodology for Amazon Mechanical Turk-like systems. ]

We use Turkers to classify all sorts of data, by having several workers render judgments on each item. But what should we do when they disagree? Like any other human behavior, Turker judgments are noisy: sometimes there are mistakes, and sometimes the task is genuinely difficult or subjective, and there is no “right” answer. Once we have a bunch of Turker judgments, we need to aggregate them — that is, use some sort of voting mechanism — to give as accurate a classification as possible. It turns out that one simple trick, threshold calibration, can substantially improve accuracy, and can be tuned to the specifics of the problem.

Here’s an example. A recent client of ours had a de-duping task: given a pair of similar articles, the task was to decide if they were “about the same topic” or “about different topics”. This is just a binary classification problem; call these labels “YES” and “NO”. To figure out how well Turkers could perform the task, we had our client provide us with a gold standard data set. That is, for 135 examples, their experts did the task themselves and provided “gold” ground truth labels.

We used a very high number of workers per example (about 20). For all 135 examples in the gold standard, the following graph plots them vertically by their “Turker confidence in YES” — that’s just the percentage of votes for “YES” among the 20 or so judgments for that particular example. I’ve also colored each example with the experts’ gold label. You can see that this simple Turker data provides some statistical separation between the classes.

This graph also shows how to create a classifier from Turker votes. We have to choose a confidence threshold for our classifier’s decision: above the threshold, say “YES”, and below say “NO”. Unfortunately, Turkers aren’t perfect at modeling the experts: anywhere we place the threshold, errors occur. However, some thresholds are better than others. The threshold with the best accuracy is at 73% confidence — that is, a 73% super-majority voting rule — and it classifies instances correctly 90% of the time. Furthermore, we can tune for different types of errors. If we are particularly concerned with avoiding false positive errors, we can set a higher, more conservative threshold; or, if we want to find as many “YES” instances as possible, we can set a lower, more liberal threshold.

Here’s another chart that more carefully details the tradeoffs between true and false positives vs. true and false negatives. For a particular decision threshold, it shows how it divides up the instances into the confusion matrix’s 4 categories of correct and incorrect decisions.

A final note on why threshold calibration is important: For this task, the Turkers were considerably more liberal than the experts at deciding what a “YES” example was — experts marked only 36% of examples as “YES”, whereas a simple Turker majority voting rule marks 57% that way. This is because the experts understood the full implications of the decision, which were substantial — various entries in their database and website would be merged, and users would be confused if they were exposed to a bad merge. False positives had a very high cost. The prompt for Turkers, by contrast, was fairly vague. (In our experience, we generally find that good task design is a huge factor in getting better Turker accuracy.) However, since Turker decisions noisily correlate with the experts, moving the decision threshold can help accuracy. Here’s the threshold vs. accuracy graph:

Statistical analysis of Turker data can substantially improve accuracy performance, even with something as simple as choosing the best decision threshold. This blog post only scratched the surface; there are a few more useful things to consider. Stay tuned for Part 2 and hopefully many more!

A few more notes on Turker voting and threshold calibration:

An interesting question is the upper bound of possible performance on the task. A good experiment to try is to have two experts independently perform a task, and check their agreement rate. We should be satisfied if Turkers can match experts as reliably as experts match each other. That is, for this task, if experts agree no more than 90% of the time, then Turkers perform the task as well as experts. (We didn’t have this particular experiment done in this case, but I’d be very curious to see the results!) In general, agreement rates can help indicate the difficulty of a task. If expert agreement rates are low, it can be argued that the task is not very “real”.

The terminology of “true/false positives”, “true/false negatives”, “precision” and “recall” are all part of a statistics/machine learning mini-field of binary classifier evaluation. Any statistical classifier that outputs a confidence value or ranking among instances (Naive Bayes, logistic regression, IR ranking, etc.) can be subject to this sort of threshold analysis. A decent place to read more is the ROC Wikipedia page. ROC and Precision-Recall curves have long been used to show thresholding tradeoffs. I think the above plots make it easier to interpret the basic information, but the more traditional graphs are also useful. Here they are for this data (provided courtesy of the excellent ROCR package):

A nice overview of these topics can be found in these lecture notes from William Press.

-Brendan

Before last weekend, our FaceStat website was chugging away with a small but loyal userbase:

But on Sunday, an insane number of people suddenly decided to flock to our site. Let’s extend the previous chart by 2 days, then a little bit of y-axis auto-scaling says it all:

Turns out the giant spike was due to our being featured via a news article on Yahoo.com’s front page!

Of course, we had to frantically rearchitect the system and scale it under this deluge of traffic. You can read the blow-by-blow account of our crazy few days on Lukas’s blog, here.

The web startup community seems pretty interested in the mad scaling issues, so I’ll respond to some of the comments on Lukas’s blog below:

Yes, we’re pretty much using Rails. We actually use an offshoot called Merb — which is a bit more efficient — on top of Thin. We find that a Rails-like platform is invaluable for rapidly prototyping a new site, especially since we started FaceStat as a pure experiment with no idea whether people would like it or not, and with a very different feature set in mind compared to what it later became. And it’s invaluable that Chris on our team is such a Ruby expert :).

However, the high-level platform really doesn’t matter compared to overall architecture: how we use the database (postgres), how much we cache (memcached/merb-cache), how we distribute load, how we deploy new systems (xen/slicehost), etc. It’s hasn’t been trivial since FaceStat is write-heavy and performs fairly complex statistical calculations, and various issues remain. But we are serving many users at nearly 100x our old load, so something must be going right — at least for now!

-Brendan

p.s. Thank you, Google Analytics, for the above charts. Some day when I grow up, I hope I am wise enough to create an equally brilliant data visualization tool.

Lots of people have been making great new visualizations of our color names data. Here are 4 more that folks have sent us.

Chris Harrison, a Ph.D. student at CMU HCI, combined our data with results from his own previous experiment, and created beautiful flower and spiral images. Unlike my and Martin’s color wheels, hue is scaled along the radius, creating a striking effect.

Next: network and cluster diagrams from David Sparks, Ph.D. student at Duke PoliSci. The layout below was computed from a similarity metric on color names. (I’m unclear whether it’s on labels or colors.) Size of node corresponds to the label’s frequency.

All of the visualizations so far have had to map three-dimensional color points into a 2D space. But Jeff Clark in Toronto went ahead and wrote a 3D explorer — you fly around a space of the color labels. He built it with the excellent Processing framework.

Finally, yet another tack: instead of creating a picture with all the labels, why not fit labels to a picture? Kristina Durivage, Chris Burg, and Scott Olson did that for an undergrad CS project at Winona State University. Their software takes any image and overlays color names. An example:

Four new visualizations in a month — whew!

To look at all our color posts, check out blog.doloreslabs.com/topics/colors.

-Brendan

Search engines control the information we see and use. Their key component is a ranking algorithm that tries to determine the most relevant web pages for your query. How good are these algorithms? Publicly, there’s a lot of hype, while privately, all the big engines run proprietary quality evaluation efforts. But there’s virtually no real data out there for the rest of us.

Using Mechanical Turk, we can evaluate engine relevance. We tried an experiment where we took five hundred queries and ran them against the top 4 English language web search engines: Ask, Google, Live, and Yahoo. The queries were a random sample from a real-world set of search queries. We had annotators rate the relevance of the top five results for each engine. Our results:

Ask clearly performed the worst. The other three engines were in a statistical tie. Their ordering was Google, Yahoo, then Live, but the differences were miniscule: the top 3 engines all answer about 80% of queries effectively.

What do these results mean?

People often talk about Google as being the most relevant search engine, with the best algorithms and the like. This study finds little evidence to support that. Sure, our methods are preliminary and could be improved in any number of ways; we can probably shrink those error bars and find more statistical differences. However, it is the case that for 500 typical queries, a rough but pretty objective measurement of search quality found that Google, Live, and Yahoo all performed about the same.

Note that these results don’t speak to the entire user experience. To be able to compare between engines, we extracted only the core web results with their titles, urls, and snippets. But a search engine also includes much more: the presentation, branding, video and image results, ads, etc. We only tested the relevance of core web search.

Many more details below.

How we’re measuring search relevance

Evaluating search engine quality is a tricky task. Here’s our first pass on the methodology.

We take a set of queries and run them against several search engines, scraping their web interfaces. We then show the query and results to human raters, asking them how relevant each result is. It’s a blind test: they don’t know which engines the results came from.

Here is an example of what a rater sees:

The raters all come from Amazon Mechanical Turk, a distributed workforce. We submit the above query/result judgment surveys to the AMT service, and pay its users – “Turkers” – to do the relevance ratings. (If you want to learn more, try looking at the Dolores Labs FAQ.)

What queries are being used? We took a random sample from the AOL query log data set; these are actual queries that real users typed in to a search engine. The AOL data set is remarkable for being pretty much the only publicly available, real-world data on web search behavior. Of course, it’s infamous for very valid privacy issues. We’re only using the part of it that doesn’t involve personal information – the raw queries, without user identification. (This NYT article on the issue is interesting.)

What’s being measured? For each engine, we count the number of queries that had at least one “Highly Relevant” result within the first five results the engine returned. This is a version of the “precision at 5″ metric from information retrieval. There are, of course, many other methods to explore. We wanted a metric that was simple and easy to interpret.

How were raters’ judgments used? We had three raters per result, and basically took a simple majority vote. We didn’t attempt to model individual annotator biases.

Are these judgments trustworthy? The ratings are certainly noisy. And sure, the workers have little training and are (somewhat) anonymous to us. However, the relevance judgment task is fairly subjective and therefore inherently noisy. Further, it’s arguably better to use untrained annotators, since this more closely mimics normal search users. And finally, we’re finding some statistically significant, systematic differences between engines on a query set, with only extremely simple analysis – so something must be working right.

What’s the statistical methodology? As dead simple as possible: 95% confidence intervals on the graph, and engine comparisons via paired t-tests. These are all on that per-query precision-at-5 metric. We think that with larger scale experiments, more fine-grained breakdowns, survey design improvements, better analyses, etc., we can flesh out more differences.

What’s the “meta-engine upper bound”? That’s just how many queries had a “Highly Relevant” result on at least one engine. So hypothetically, if you were to combine all the engines and select the best results for the top, it would perform at this upper bound. This bound is overly high for a number of reasons (e.g. it’s artificially inflated by judgment noise and it assumes smart re-ranking); but it gives some idea how much the search engines could still improve.

Anyway, we’re thinking of doing more work along these lines if people are interested. There are certainly big improvements that could be made; we’d love any feedback you have.

-Brendan

As anyone who follows political races knows, different sources can report the same event in very different ways. We took nearly six thousand recent articles over the past month about Clinton and Obama and sent them to Mechanical Turk to be classified as favorable or unfavorable for the respective candidates. We scraped the articles from Google News restricted to several sources, and threw in front page headlines from Digg.

Here is the graph for favorability scores, aggregated by source. We found that Digg was far and away the most favorable for Obama.

The next graph tracks overall news favorability by date. To provide some context, we compared it with the change in Obama stock on the Intrade prediction market.

More details after the jump:

We created our data set by doing two separate searches, one for “Barack Obama” and one for “Hillary Clinton”. This did a pretty good job ensuring that results from Google News or Digg’s search facility demonstrated how the article was about the given candidate. For each article, we showed the headline, search result snippet, and link to several Turkers. They reported whether it was positive, neutral, or negative toward the candidate.

The favorability metric was created by averaging the ratings across articles. Pro-Obama and anti-Hillary articles were both worth 1 point; anti-Obama and pro-Hillary both worth -1, and neutrals 0.

Therefore, if all articles are either positive towards Obama or negative towards Hillary, the rating is +100%; and vice-versa for -100%.

The data is very noisy. The question of favorability is extremely tricky: it includes a combination of expectations, sentiment, and the objective events a newspaper chooses to report. All of these are hard to reliably assess or even define. (And whether anything you measure constitutes “media bias” is another complicated question!)

Despite all this philosophical intractability, the data must be showing something real, because we have a statistically sound result: the difference between Digg and the others was statistically significant (t-test, p<.001). The differences within the mainstream media were not statistically significant.

-Brendan, Chris, Lukas, Mike

Martin Wattenberg at IBM Research took our color names data and made a cool new cloud view:

Instead of plotting each individual color name like in the original, he grouped together identical names, took an average position, and sized the word by frequency. That’s why the more common names like “red” and “green” are large. This really helps readability (and, I’ll admit, the black background works a bit better :))

Thanks to Martin for sending this on!

I just packaged and released the data set for our color names experiment. It has 10,000 color/label pairs.

This is the download link. Read on for more details:

I tried to generate the color patches in a way to get interesting colors. This of course is incredibly subjective. My main concern was to eliminate muddy dark grays, which are very common when uniformly sampling over standard RGB values. (Perhaps I went too far — see the big donut hole in the color wheel plots.) So the color patches were sampled from HSV with uniform sampling over hue, but saturation and value biased high (normal distribution). The exact code and parameters for this is included in the download.

The plots in the post and the explorer look like a color wheel with hue as the angle. But actually they’re from running PCA over the RGB values, using the first two principal components as x and y. This was a very arbitrary decision, but seemed to make a nice visual effect. There are many other reasonable ways to plot the data.

The data includes anonymized identity on the workers. (The Mechanical Turk service makes all workers anonymous, but we anonymized yet again for releasing the data set.) You can see that certain workers did a large number of annotations. We have no demographic information for this one, sorry.

The files are:

• data.csv, which contains the color/label pairs, also with rgb and hsv representations.
• R.R, which has some routines that were used to generate and plot the data. It has examples of how to read and use the data, if you like to use R.
• html.rb, which with write_html() creates the explorer.
• sample-hit.html, one of the web forms used for data collection. There were 1000 forms with 10 colors each. For each single form (“HIT”), exactly one annotator filled it out. Individual annotators sometimes did multiple forms if they wanted to.

Let us know if this is useful, if you have any questions, or find something wrong with the download — either email or leave a comment here. And if you do anything cool with this data, we’d really love to hear about it.

-Brendan

What would you call these colors?

We showed thousands of random colors like this to people on Mechanical Turk and asked what they would call them. Here’s what they said:

The above picture contains about 1,300 colors and the names for them that Turkers gave.  Each is printed in its color and positioned on a color wheel.  Just looking around, there sure seem to be different regions for different names.  But there are also rich sets of modifiers (“light”, “dark”, “sea”), multiword names (“army green”), and fun obscure ones (“cerulean”). To help look at all this, we also made a color label explorer, so you can search for different terms and see different parts of the space. If the link doesn’t work for you, here are a few examples:

This study is basically the same design as the famous World Color Survey, where anthropologists showed color patches to speakers of many different languages and asked for names, to test the universality of language.  Of course, we have mostly native English speakers. However, we can get much more data.  (The above picture and links use only a small percentage of all the colors and names we collected.)  There’s tons more that can be done. Want to make a better visualizer?  Statistical analysis of colors to name terms?  Let us know and we should be able to get this data set online.

UPDATE 3/18: I posted the data set.

-Brendan

Human annotators are great at providing basic information about images. We were wondering if we could find something interesting about magazine covers. Stumbling upon 2800 Sports Illustrated cover images going back to 1954, we sent them to Mechanical Turk, asking people to identify the race and gender of the person featured (if any), and what sport was depicted. There are lots of interesting things in this data; this post will touch on just a few we’ve had time to whip together some graphs for.

Here is a historical graph of the frequency of how often people of different races appear on the cover of Sports Illustrated. The story is simple and striking:

Next: which sports get featured on the cover? Here’s a chart for several sports over that same time.

It might be possible to find links between the careers of famous athletes and rises and falls their sports’ popularity; for example, boxing peaks in the 70’s (Muhammad Ali?), basketball peaks in the 90’s (Michael Jordan?) and golf bounces back in the 90’s after a long decline (Tiger Woods?).

Many other sports appear in the data, too; for this chart, we made sure to pick the three most common, and a few other particularly interesting ones. Percentages don’t add up to 100% because we didn’t plot all the other sports, including things like horse racing which used to be much more popular. If you’re really curious, here’s the full chart of all sports we asked about, including many of the smaller ones.

-Brendan