Image by Sarah Butrymowicz

Being tapped to write a blog post here at CrowdFlower is usually left to the experts. So with that, let me begin by making the disclaimer that I am neither a political analyst nor a data scientist. But I do have a personal fervor for politics and access to some impressive tools, thanks to my job here at CrowdFlower.

For those who aren’t familiar with CrowdFlower, we specialize in tapping human contributors worldwide to do massive amounts of simple, repetitive tasks (especially tasks that are hard for computers to do by themselves). Here’s a quick how-it-works animation.

I had been reading some old blog posts on the CrowdFlower blog when I came across an interesting 2008 post on election media bias.

I determined that this could be a great opportunity to revisit sentiment analysis, and specifically set out to see if automated sentiment detection tools vs. human assessments could yield any blog-worthy findings.

To see how far the automated sentiment tools have come, I began by using an enterprise-grade social media monitoring tool that provides sentiment analysis.

I ran a few quick monitoring searches of my own to see how the current Republican Primary election was tracking — it seemed a topical place that would be chock full of good commentary.

The instant access to well-organized data from blogs, news sources, and a variety of social media sources was outstanding.

However, I was surprised to find that for each search I conducted, the automated sentiment detection tool consistently returned an overwhelming proportion of “Neutral” ratings (frequently exceeding 90%). This seemed funny to me, given the typically emotive nature of politics.

It’s important to note that this particular tool uses a default value of “Neutral” for any post it cannot interpret.

A particularly interesting subset of the data was several thousand tweets about Herman Cain immediately following the news of alleged sexual harassment by Cain during his time as leader of the National Restaurant Association. Surely this would yield some sentiment-rich commentary that even machines couldn’t resist tagging.

For the posts about Herman Cain on Oct 31st, here is what the machine detected on just under 3,000 posts:

Naturally, I took to the CrowdFlower platform and decided I would run the same data through a simple sentiment analysis workflow.  With the help of our team of crowdsourcing gurus, I utilized some simple, but effective best practices to control for quality (you can get a good overview here). Here is what the CrowdFlower contributors detected:

Here are just a couple of posts marked “Neutral” by the machine and “Negative” and “Positive”, respectively, by CrowdFlower contributors:

Takeaways

A spot check of the results on the automated set confirmed that when the machine actually tagged a post as positive or negative, it was usually very accurate (good precision).

However, the large amount of data that the machine was unable to make a determination on suggests that the pervasive problem of ‘recall’ is still the big challenge with automated sentiment detection.

This graph illustrates the recall difference a bit more clearly. The need for human analysis when dealing with the subtleties of language could not be more apparent.

Automated Tool: Good precision. Poor recall.

CrowdFlower Tool: Good precision. Good recall.

Sentiment Analysis is Insightful AND Entertaining

In addition to the Herman Cain Twitter data, I looked at headlines, blogs, and a broad swath of social media commentary on all the candidates. The conclusion I can draw from my effort is that sentiment detection, is indeed, still a very challenging problem to solve through automation.

This is consistent with what I see here at CrowdFlower daily — in today’s data-wealthy world, there are countless tasks that require human attention (good to know if my blogging career never gets off the ground).

Hopefully I’ll get the chance to continue exploring the sentiment about topical news as it breaks, and will look forward to sharing future findings.

Have experience monitoring sentiment? Let us know if this is consistent with what you’ve seen. Leave a comment.

***

To find out more about how CrowdFlower technology is used for sentiment analysis and a wide range of other human powered projects, visit the CrowdFlower products page.

Millions of people have participated in our tasks over the last few years, and tens of thousands of people are active at any given moment. However, crowdsourcing is not a traditional engagement model. Tasks are elective, which means people are free to come and go as they please. It’s a fair question, then, to ask whether they keep coming back.

Do people perform tasks only fleetingly, or has crowdsourcing become more of a long-term engagement? Furthermore, just how important is contributor retention in the world of crowdsourcing?

While a majority of people fall into the “one-and-done” camp, many of the most productive contributors tend to have participated in previous jobs. Within any single job, these seasoned veteran contributors also provide far more work than their less experienced counterparts.

Over a period of two months, we ran a series of five very similar jobs, retrieving ratings information for businesses throughout North America. In total, we collected over half a million judgments from a total of 2,901 unique contributors,¹ representing multiple labor channels and 101 countries. As a first test to whether seasoned veterans are common, we looked at how many people participated in more than one job. In total, 2,389 people participated in one job only, meaning that First-Timers accounted for over 82 percent of all contributors.

But this may not be the best number to look at. We’re really interested in whether certain people recognize and seek out specific types of tasks after having worked on them before. To look for this behavior, we analyzed the most recent job, counting how many people participated in at least one prior iteration of the task.

For a recent job, a total of 906 people participated, 247 of whom had done the task previously. By this measure, Seasoned Vets constitute approximately 27% of the workforce. While the impact of returning contributors is greater under this methodology, the fact remains that these “loyal” contributors are firmly in the minority on this series of jobs.

Count of Contributors by Profile

However, as we’ve seen before, the individual impact of contributors varies widely, with a minority of people often providing the vast majority of work. With this in mind, we looked at the contributions of First-Timers and Seasoned Vets and found some striking differences.

Share of Contributions by Profile

Seasoned Vets, while constituting only 27% of the workforce, provided 47% of the total work completed. On average, each Seasoned Veteran provided 2.5 times more judgments than their less experienced counterparts. It’s also interesting to note that there was no significant difference between the quality of work provided by First-Timers and Seasoned Vets, no doubt due to our suite of quality control measures.

Given that the people who stick around tend to be far more productive, improving retention is a useful consideration (for these jobs, at least). We’re now interested in how best to attract people to return to the types of jobs they’ve already seen. That, however, is a work in progress and a story for another day.

1. Note that this analysis only considers Trusted workers.

Expanse of Cheese: Caseificio di San Silvestro's Aging Room in Castelvetro (MO)

Last May, I took a trip to Italy for two weeks. A little bit of history: my friend Jessica and I are both Italophiles, and when her mom sent us a link to a video contest where the prize was a round trip flight to Italy, we knew we had to enter. After a week of writing and editing lyrics in a Google Doc — half in Italian, half in English — the resulting music video ended up winning us a trip to the holy land of olive oil, vino, and other delectable edibles.

Apart from being a passionate eater, I’m a passionate supporter of the Slow Food movement, an organization which promotes good, clean, and fair food around the world. Each year, they publish a guidebook to restaurants in Italy that adhere to their principles. In Italy, this usually means each restaurant is handpicked to showcase the traditional food of a particular region; each restaurant supports artisanal methods and products that otherwise might go extinct (were eaters not eating them), and where the food is most likely naturally organic and local anyway.

But, a problem: it’s 2011, and I’m more apt to travel through interactive maps than with old-fashioned guidebooks. More importantly, I don’t plan, and I needed to know what edible options were around me at any moment in time on my trip. What I really needed was a version of the guidebook, in map form, that I could use on my mobile. No such option existed, of course, and I was left with two options: magically visualize restaurants around me by poring through the guidebook, or, crowdsource it.

What I needed to do (and, what much of our work at CrowdFlower comprises), was structure unstructured data. To create the map from the book, I went through the following steps:

• Obtained a PDF version of the book
• Split the book into pages, and uploaded each page as a separate PDF (thanks to pdftk)
• Created a CSV (comma separated values file) with each page’s PDF link and page number
• Created a crowdsourcing task to structure the data, using the previously uploaded individual PDF pages
• Geocoded the structured data
• Output the geocoded data in KML (Keyhole Markup Language) form
• Uploaded the KML file to a mapping site (e.g. Google Maps)

Conveniently, each page that outlined a restaurant in the PDF was formatted nearly the same. This made it easy to give instructions to workers, as seen below:

Instructions

For each area on the page, workers were asked to copy and paste specific sections into the task. Each page, then, was split up into corresponding parts (Region, City, Directions, Restaurant Name, and the two Capture Areas). This is the essential concept here: structuring the unstructured data such that I could later geocode it properly, and display it in the way that I needed.

The tonnarelli cacio e pepe at Da Felice a Testaccio in Rome (used in the Geocode example)

Once the task finished, I downloaded the resulting CSV file, and whipped out Google Refine (a.k.a. Excel on crack), which has a feature that allows you to enter a template API call that changes based on specific values in each row. Using the Google Geocoding API (any will do), I constructed the following API call, using the address value in each row as the “address” parameter for each API call:

http://maps.googleapis.com/maps/api/geocode/json?address=via+Mastro+Giorgio%2C+26+Roma+Lazio&sensor=false&region=it&language=it

After slicing and dicing the rest of the data into bits that I wanted to display on a map, I used Google Refine’s “templating” feature to export each row as a Placemark in KML format. Finally, I uploaded the resulting KML file into several different maps, each representing one region in Italy.

A portion of the Campania map - tons of places to explore around here!

Try it out! Check out the map for Tuscany, and the map for Campania.

If you want to give crowdsourcing a spin, head on over to our self-service product and click “sign up”. Or, if you’d prefer a hands-off “we-do-everything” approach, contact us to get started.

Watch the video that started it all: Io Sono Balsamico by Balsamico

Aron is a Crowdsourcing Project Manager at CrowdFlower, and is the resident agriculturalist-eater. Follow him on Twitter (@aron) for more sage bits of agricultural-eating learnings.

psychologytoday.com

Evidence in experimental psychology suggests that most people overestimate their own ability to complete objective tasks accurately. This phenomenon, often called confidence bias, refers to “a systematic error of judgment made by individuals when they assess the correctness of their responses to questions related to intellectual or perceptual problems.” ¹ But does this hold up in crowdsourcing?

We ran an experiment to test for a persistent difference between people’s perceptions of their own accuracy and their actual objective accuracy. We used a set of standardized questions, focusing on the Verbal and Math sections of a common standardized test. For the 829 individuals who answered more than 10 of these questions, we asked for the correct answer as well as an indication of how confident they were of the answer they supplied.

We didn’t use any Gold in this experiment. Instead, we incentivized performance by rewarding those finishing in the top 10%, based on objective accuracy.

Does Bias Exist?

To estimate confidence bias, we looked at the difference between the average of how confident an individual was of his/her answers and how many he/she answered correctly. If the difference is positive, the individual overestimated how well they did. Amazingly, over 75% of contributors overestimated their ability to answer multiple choice questions correctly.

Are Individuals Consistently Biased?

Because our dataset consisted of Math and Verbal questions, we looked at each individual contributor’s confidence bias for both types of questions. In aggregate, people tended to have more trouble with the Verbal questions (average accuracy of 28%, compared to 41% for Math), though the average confidence score was nearly identical (63% +/-1).

The vast majority of contributors fall into the “overconfident on both” quadrant (top right), while only a handful of contributors were overconfident for one question type and underconfident for the other (top left and bottom right quadrants). Overall, there is certainly a correlation between bias scores on the two problem types, suggesting that many individuals are consistently biased on different types of problems. However, this explains only a portion of the variation.

Does Bias Vary Across Groups?

Given that overconfidence seems to be a consistent trait, we were curious how this trait varies across the different groups making up our contributor pool. We sliced and diced our contributors into a number of different sub-groups, which are summarized below.

There are a lot of interesting things going on here. To highlight a few, accuracy increases consistently as the contributor’s education level advances from High School to College, but so does confidence, leaving the bias score nearly unchanged. There’s a similar pattern with Age, with older contributors tending to be both more accurate and more confident.

Gender and Location also have an effect on confidence bias. Taking the two countries that supplied the most people, contributors from the US were much more accurate and slightly more confident than the average, while those from India were average in terms of accuracy but much more confident. As such, the bias score for contributors from India is nearly double that of contributors from the US. With respect to gender, confidence didn’t vary much, but women were more accurate and thus less biased than men. Moving on.

Further Research

In the context of experimentation, we decided against using Gold to minimize any selection bias among contributors. However, this makes it difficult to apply these results to enterprise crowdsourcing, at least as practiced by CrowdFlower. In the future, it would be interesting to look at confidence bias among trusted workers only, and particularly among trusted workers with repeated experience in specific job types. We would expect these workers to have a better sense of whether their answers are correct, though it is possible (and perhaps likely) that confidence would increase along with accuracy.

1. Pallier, G., Wilkinson, R., Danthir, V., Kleitman, S., Knezevic, G., Stankov, L., & Roberts, R. D. (2002). The role of individual differences in the accuracy of confidence judgments. Journal of General Psychology, 129,257–299

]]> http://blog.crowdflower.com/2011/09/confidence-bias-evidence-from-crowdsourcing/feed/ 0 http://blog.crowdflower.com/2011/08/crowdsourcing-and-sem-now-with-even-more-cat-pics/ http://blog.crowdflower.com/2011/08/crowdsourcing-and-sem-now-with-even-more-cat-pics/#comments Fri, 26 Aug 2011 18:58:20 +0000 Zoe Vance http://blog.crowdflower.com/?p=3450

Every modern business wrestles with the elusive lady that is the search engine and the potential she offers to connect with customers. Google and Bing make it easy for anyone to buy keywords and drive customers to a website, but what keywords are our customers searching for? Would a sales manager frustrated with the average 70-80% accuracy of business listings bought from data providers search for “crowdsourcing,” “address checking,” or something else entirely? Since we’re a crowdsourcing company, we had to try crowdsourcing the solution …

In the last two weeks of my summer internship at CrowdFlower, the marketing team challenged me to generate the widest range of search engine seed terms that could be used in SEM keyword tools to generate “hot” search phrases. For those of you who’ve dealt with SEM, you know that thinking of seed phrases to plug into these tools can be a painfully frustrating and surprisingly difficult task. (For those of you who haven’t and don’t believe me, try right now to describe what your company does — or anything for that matter — in 10 significantly different ways.)

Keyword tools are based on your thought process, which takes care of the customers who are thinking in the same way you are, but what about all the people of a different mindset who are trying to find your solution? For example, if I were looking for pet grooming services, depending on my thought process, vocabulary range, and amount of sleep the night before, I could search anything from “pet grooming salon” to “quality feline hair cuts” to “kitty bad hair day.” The challenge was to understand the full breadth of how the crowd approaches a certain problem, essentially the perfect task for the crowd.

Looks like PetSmart forgot to buy an ad for "kitty bad hair day"

I set up the job to have the contributor imagine working for a business that could use CrowdFlower’s services (whether or not they know of them) and then have the contributors write search queries they would use to find a solution to their problem on the Internet. I created numerous versions for each scenario, varying both the industry jargon and the background we gave the contributor, so that I could fully mimic the knowledge and background of a real customer, and thus elicit as wide a range of responses as possible. I then ran a second job to rate those queries to identify quality results worth pursuing. Finally, I took those quality terms and fed them back into a keyword generator tool, making sure that we had the full range of potential search phrases optimized to what people would search for the most.

An unspoken secondary challenge was “Can you become a full-fledged crowdsourcing ninja before you leave?” This final job I ran was by far the most challenging because crowdsourcing content generation requires a seemingly daunting use of CrowdFlower’s automated workflow system, which uses gold units and an active, real-time peer review system. I successfully passed this final test, with the help of our in-house content generation team.

Sample Generated Keyword Seeds

The keyword gen job yielded really interesting results. There were some genuinely clever and alternative thought process seeds, some that we were already using, and some that were borderline ridiculous. My personal favorite keyword seed though, as the office intern, had to be “how to find an intern” — the obvious solution to all life’s problems.

In a recent paper, presented at the ACM Conference on Computer Supported Cooperative Work (CSCW), John Horton, Daniel Chen and I used a large-scale experiment to test the effect of different incentive schemes on the quality of crowdsourcing work.

The results surprised us. They suggest that workers perform most accurately when the task design credibly links payoffs to a worker’s ability to think about the answers that their peers are likely to provide.

a horserace experiment! (photo cc-by-sa by iyoupapa)

The idea for this study came out of our sense that, as social scientists, we had something unique to offer the existing research on human computation. Early and influential crowdsourcing research has focused on how to filter the judgments of the crowd to find the best answers. We wanted to know whether simple task-design changes could improve the quality of data coming into a crowdsourcing system in the first place.

To test this idea, we chose 14 different incentive schemes and framing techniques developed and validated across the social sciences and set up a horse race experiment to see which schemes/techniques would work best.

Consistent with our personal biases (John and Daniel are both economists, and I’m a sociologist), some of the schemes were financially oriented, some were social or psychological, and some were hybrids combining social and financial incentives. The details of all the schemes are included in the paper (it’s a long list, and some of them are kind of involved), but it’s worth giving some examples.

On the financial end of the incentives spectrum, we had one condition we called “reward-accuracy,” which was pretty much what you’d expect: we told workers, “we’ll pay you a bonus if you get the answers right.” We also had one called “punishment-accuracy,” the gist of which you can deduce. On the purely social-psychological side, we had one we called “trust,” in which we told workers, “we’ll pay you for this job no matter how bad your performance, we trust that you’ll still make your best effort.”

One of the weirdest schemes turns out to be important, so I need to explain that one. Called “Bayesian Truth Serum” (BTS), it incorporates a design from the work of Drazen Prelec, a behavioral economist at MIT, who realized that research subjects could probably provide useful information regarding the expected distribution for subjective, qualitative questions (nb, the mechanics of how he does this are arcane in a way that is almost sure to delight the geeks among you, so I encourage you to read his paper). Few of the details of real BTS are important, except that we incorporated the piece about asking workers to answer the questions themselves and predict the distribution of other workers’ responses. We also told them we’d give them a bonus if their predictions were correct.

We then created a task that asked workers to answer five questions. In this case, the questions were drawn from another study examining participatory features of websites, for which we already possessed validated data collected by research assistants.

All workers answered the same five questions about the same website (www.kiva.org) while being exposed to one and only one of the 14 incentive schemes (or a control condition of no scheme). Roughly 2,000 individuals participated in the study, resulting in over 100 subjects in each of the experimental conditions. (The statistics and science nerds out there will be pleased to know that both the drop-out rate and demographic covariates were distributed evenly across conditions.)

To measure worker performance, we used the research assistant responses as correct answers to the questions and then calculated the total number of matching answers (out of five) provided by each worker. The results (aggregated across all treatments) are plotted in a histogram below and show that the average worker answered just over two questions out of five correctly.

Then, in order to see how the treatments compared against each other relative to the control group, we calculated the mean correct response rate for each condition and conducted difference of means tests to see which of these means were significantly greater than the control group. The results of this comparison appear below (in a new plot that doesn’t even appear in the paper!):

The orange dots show the value of the mean in each condition, and the blue bars illustrate the 95% confidence interval around that mean. The treatments are sorted by the size of the difference in means from the control. (More hard-core nerd stuff: the means are adjusted using Intent-To-Treat estimators).

From these results, we concluded that our horse race had two clear front-runners: the “Bayesian Truth Serum” (BTS) and “Punishment – disagreement” conditions, each of which improved average worker performance by almost half of a correct answer above the 2.08 correct answers in the control group. A few of the other financial and hybrid incentives had fairly large point estimates, but were not significantly different from control once we adjusted the test statistics and corresponding p-values to account for the fact that we were making so many comparisons at once (apologies if this doesn’t make sense — it’s yet another precautionary measure to avoid upsetting the stats nerds among you). In a tough turn for the sociologists and psychologists, none of the purely social/psychological treatments had any signficant effects at all.

Why do BTS and punishing workers for disagreement succeed in improving performance significantly where so many of the other incentive schemes failed? The answer hinges on the fact that both conditions tied workers’ payoffs to their ability to think about their peers’ likely responses. (We elaborate on the argument in more detail in the paper.)

Does this mean that we should give up on simple financial or social-psychological incentives? Probably not. The fact that we conducted the experiment on MTurk means that the deck may have been stacked against incentives like the “trust” condition I described earlier. Because requesters on MTurk have little oversight, workers are more likely to respond to financial incentives than stated promises. In this sense, the marketplace has structured the interaction between workers and requesters in a way that may limit the opportunities to harness motivations that are not linked to money in some explicit way.

You can download the full paper to read more.

Once upon a time, I had a job that included looking through boxes of documents that were supposedly related to environmental litigation, but were generally (a) unrelated, (b) dusty and (c) mind-numbingly dull. Earlier this year, as I looked back on those dark days, it seemed to me that crowdsourcing would be a great tool for a first pass through documents, helping a legal team focus its efforts away from documents that are obviously not responsive to a given request.

To test this suspicion, we used a dataset of ~2,700 documents that were pre-coded with relevance assessments by a team of legal experts associated with the TREC 2010 Legal Learning task.¹ We asked multiple workers whether each document, emails made public during the course of the Enron investigation, was responsive to a request for Residential Real Estate (full instructions here):

“So a Team of Lawyers walks into a Conference Room…”

We ran two iterations of the document review task. In the first, we used a Gold ² distribution of 50% Relevant and 50% Not Relevant. For the second iteration, we increased the proportion of Relevant documents in our Gold set to 60%. Our thinking was that, at least in the context of litigation, returning an irrelevant document (false positive) was preferable to missing a Relevant document (false negative).

In the chart above, Recall is the percentage of all responsive documents that were returned; it measures how thorough the search is. Precision is the percentage of returned documents that are responsive; it measures how accurate the process is. F1 is the harmonic mean, a simple summary measure that rewards high values in both. We used a simple majority among workers to determine a document’s relevance.

Before looking at how this performance compares with manual and automated document review, it’s worth noting that changing the Gold distribution had very little effect on the overall accuracy, but it had a large effect on the distribution of errors. By increasing the frequency of Relevant Gold units, we cut the number of false negative errors by nearly 50%. In a context where one type of error is relatively more “expensive” than another, this is a useful tool to be aware of.

Without running the same dataset through crowdsourced, automated, and manual document review, it’s difficult to compare performance across methods. Nevertheless, Grossman and Cormack (2011) discuss manual and automated document review, finding that average recall for manual review of documents can be as low as 20-50%, though typically with much higher precision. For automated review on a dataset similar to the one we used, recall averaged 77%, though with average precision of 85%.³

Living in a (non)Binary World

Every document in our test was also graded with a probabilistic measure of Relevance by default. Because we asked multiple reviewers whether a given document was Relevant, we used inter-coder agreement to suggest the likelihood that a document is responsive. Further, because we tracked each individual worker’s performance on Gold, we weighted each worker’s contribution to the agreement by his/her estimated accuracy.

For this exercise, we remapped our confidence scores such that a document that was the least likely to be Relevant received a Relevance Score of 0.01, while the documents most likely to be Relevant received a Relevance Score of 0.99. ⁴ The distribution of documents by Relevance Score is included below.

Note that because most documents in this test received judgments from three different workers, there isn’t much variation in the middle of the distribution. Most units had either unanimous agreement or a 2/1 split. Nevertheless, the Relevance Score makes it possible to set a threshold on what should be considered Relevant.

By changing the threshold, we can include any document that received at least one judgment of Relevant (increased Recall) or to include only documents that did not receive any judgment of Not Relevant (increased Precision). As shown below, different thresholds dramatically influence the number and characteristics of the documents returned as Relevant.

While there is no substitute for trained legal experts, these results show that crowdsourcing is an effective complement to eDiscovery document review. The promise of putting multiple pairs of eyes on every document dramatically decreases the likelihood of missing a relevant document. And consider that in less than 24 hours, we collected over 15,000 unique relevance judgments on nearly 3,000 documents, and for much less than the billing rate of your average attorney.

1 http://plg1.uwaterloo.ca/~gvcormac/treclegal09/

2 One of the ways that we control for quality is by randomly inserting a subset of units for which we already know the answers. We refer to this data as Gold. We track worker performance on these Gold units as a proxy for overall accuracy. Additional documentation is here.

3 Maura R. Grossman & Gordon V. Cormack, Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review, XVII RICH. J.L. & TECH. 11 (2011), http://jolt.richmond.edu/v17i3/article11.pdf

4 For “Relevant” documents, P(Relevance)=0.99*(Confidence). For “Not Relevant” documents, P(Relevance)=1-0.99(Confidence)

We’re proud to announce CrowdFlower’s participation in one of the most prestigious academic conferences worldwide.

The 20th Annual Text REtrieval Conference (TREC 2011), organized by the National Institute of Standards (NIST), will take place Nov 15-18, 2011, at NIST’s campus in Gaithersburg, MD. TREC 2011 will feature a new Crowdsourcing Track that will investigate:

• How to obtain high-quality relevance judgments from individual crowd workers;
• How to effectively compute consensus judgments from individual judgments;
• Interaction between these (i.e., worker accuracy vs. subsequent consensus accuracy achieved).

Teams can choose to participate in either or both tasks: (1) collecting judgments and (2) computing consensus. We plan to evaluate using both ranking and classification metrics, giving teams the flexibility to focus on either one or both.

We welcome participation from those who have never before participated in TREC; note there is an agreement form to submit in advance; the form is available from the track website.

View the guidelines here.

Tentative Schedule

May 6, 2011: Initial deadline for participating teams to email organizers with team name and
intent to participate (will affect the partition of data distributed to all participants)

May 13: Finalized guidelines released, Stage 1 training sets distributed to participants

May 20: Stage 2 training set distributed to participants

August 1: Release of Stage 1 test data; submission system opens immediately

August 8: Stage 1 submissions due

August 15: Stage 2 test topics distributed

August 22: Stage 2 submissions due

~September 15: Preliminary results announced to participants

~October 15: Team papers due to NIST for inclusion in TREC conference
notebook

November 15-18: Text REtrieval Conference (TREC 2011) in Gaithersburg, MD

~February 1, 2012: Final team papers due to NIST for inclusion in TREC conference proceedings

For more information, please contact Track Organizers Gabriella Kazai (Microsoft Research) or Matthew Lease (University of Texas at Austin).

After hearing that our friends at Citizen Effect were sponsoring Brackets with Benefits, a cool new March Madness pool where the proceeds from all the gambling go to one of the many awesome community development projects they support, we thought it would be fun to get involved. Considering how most people** watched their brackets go up in flames by the time the Sweet Sixteen rolled around, we thought it would be fun to give everyone a second chance to play.

We teamed up with Citizen Effect to invite all Brackets with Benefits participants, as well as the general public, to pick the results of the Sweet Sixteen and Final Four, a total of 10 games. We had one clear winner, who correctly predicted the outcome of 8 games (and missed Duke/Arizona and Florida/BYU, for those keeping track at home) and will be going home with a fancy new camcorder. For what it’s worth, the aggregated response of the crowd correctly picked 7 games.

In a tournament full of upsets, there were two games that surprised our participants most. Not surprisingly, these were the games where the two #1 seeds (Duke and Ohio State) lost in the Sweet Sixteen, which just under 10 percent of people picked. There was one person from Glenolden, PA (perhaps an embittered Villanova fan?) who picked both Duke and Ohio State to lose, but ended up picking only three other games correctly.

All in all, a fun little experiment. We want to send a big thanks to our friends over at Citizen Effect for putting together the pool and inviting us to participate.

Couldn’t make the March Meetup at CrowdFlower? Check out the video highlights below.

If you like what you see above, join our Crowdsourcing Work Meetup Group for news on upcoming Meetups.

The votes are in from our Oscar crowdsourcing experiment, and the crowd successfully picked the winners of 14 of the academy awards. For reference, Roger Ebert got 15 predictions correct so we’d have to conclude that the crowd performed reasonably well at predicting the winners of this glorified popularity contest.

Predicted and Actual Winners of the 2011 Academy Awards

One fascinating thing about aggregating responses is that the crowd as a whole will often outperform the average worker. In this case, among the 500 people we polled, the majority of respondents picked fewer than 10 awards correctly (mean of 9.6 and median of 9). And yet, by aggregating all the responses, such that the nominee with the most “votes” is predicted to win, the crowd as a whole correctly picked 14 awards. While the “wisdom of crowds” doesn’t come as much of a surprise, it’s always reassuring to see it confirmed in new applications.

As we noted in our earlier post, though, the more interesting question was whether workers who indicated higher confidence in their responses would outperform workers with lower confidence. Looking at the results, however, we saw no significant correlation between a worker’s predicted accuracy and  actual performance.

"Squint all you want, but there's no pattern"

While it’s certainly possible that we didn’t offer enough of an incentive for workers to estimate their own accuracy, the more likely explanation is that predicting the winners of the Oscars is not something that a person can do with any degree of certainty. Confident or not, the people we polled did not see the “Inside Job” coming.

As a final exercise, we ran a regression on every explanatory variable we could find, including what state workers came from, what day they made their predictions, whether they made their predictions during the day or at night, how long they spent making their predictions and even their historical accuracy on other CrowdFlower tasks. The only variable with any significance turned out to be how long they spent on making their predictions, and while it was significant (at p=0.001), no model we could come up with explained more than 5 percent of the total variation in accuracy.

While the wisdom of crowds seems to extend to picking Oscar winners, the more interesting experiment of having workers self-select as trustworthy is ongoing. In the future, it would be worthwhile to repeat this experiment with questions that can be answered objectively and without uncertainty (solving algebra problems seems like a good candidate), to see if any correlation emerges between predicted and actual accuracy.

With the most glamorous award ceremony of the year just around the corner, we asked 500 people from across the United States to help predict who the big winners are going to be. Below are their predictions, sorted in descending order of agreement.

"I'd like to thank all the little people..."

Based on these early results, Pixar looks like a lock for yet another Best Animated Picture award and Natalie Portman had better start polishing her acceptance speech.

Anyone can make anonymous predictions, so we made things interesting by rewarding workers for answering correctly. We structured the job such that everyone receives 5 cents for completing the survey, plus an additional 2 cents for each correct answer.

To spice things up even more, we asked workers to guess how many predictions they would answer correctly (their “Magic Number,” with a minimum of 5), with the stipulation that we will pay bonuses only to those workers who answer at least their Magic Number of predictions correctly.

As you can see below, while quite a few workers played it safe by selecting the minimum number of correct responses (“5″), the most frequent choice was a Magic Number of 10. Also, more than a few brave souls thought they would get every prediction correct.

But the big question, apart from whether Darren Aronofsky (Black Swan) can edge out David Fincher (The Social Network) for Best Director, is whether any correlation exists between worker confidence and actual performance. When workers have a cash incentive to estimate their own accuracy correctly, do the self-labeled “experts” perform any better than the “novices” at predicting Oscar winners?

Make some popcorn, grab your Kleenex and stay tuned. After they’ve rolled up the red carpet, we’ll come back and see who did a better job of predicting the winners.

A few months ago, we created a crowdsourced ranking of NFL players according to their fantasy value. For those just tuning in, we took the Top 75 players in ESPN’s preseason preview as the experimental sample, paired them against each other, then had workers through the CrowdFlower platform select the better of the two.

Now that the regular season has finished, let’s see who did a better job of predicting the final ranking of this sample. Of the 75 players in the sample, only 28 finished in the Top 50. In other words, ESPN’s team of experts failed to predict nearly half of the most valuable players in 2010.

We brushed off this halting start by Team ESPN and looked at those players in the preseason sample who did finish in the Top 50. Did Team ESPN or Team CrowdFlower do a better job of ranking them? To find out, we looked at which of the two ranking systems contained more of the top players at a number of intervals.

For example, of the seven most valuable players, five were in the preseason sample. Team CrowdFlower correctly picked four of the five, which we’ll call a Precision Score at 7 of 0.8 (that is 4 out of 5 players, or 80 percent). Team ESPN, on the other hand, picked only two of the five players to finish in the Top 7, corresponding with a Precision Score at 7 of 0.4.

As you can see in the graph above, crowdsourced workers were at least as good as ESPN’s experts. For the Top 5 Rated players, Team CrowdFlower predicted all three of the players in the preseason sample, while Team ESPN picked zero. For the Top 20 Rated players, Team CrowdFlower predicted 9 of the 13 players in the preseason sample (Precision Score at 20 of 0.69) while Team ESPN predicted only 6 of the 13 players (Precision Score at 20 of 0.46).

Across the board, crowdsourced workers never performed worse than a professional team of experts at predicting the value of Fantasy Football players. Approximately 80 percent of the time, crowdsourced workers were better. Most interesting, the biggest difference between crowdsourced workers and experts was for the most valuable players.

Writing is easy. Just sit in front of a typewriter, open up a vein and bleed it out drop by drop.
– Red Smith

Underwood Five Typewriter

When I was in college, a professor I respected said that one of the best ways to demystify writing is to write like people you admire. Specifically, to find passages that you love, and try to revise them in your own words. This exercise proved invaluable. It allowed me to walk in their literary footsteps, shedding light on why they chose — or avoided — certain words, punctuation, and syntax.

With this in mind, I recently wondered whether you can crowdsource writing, specifically, revising.

I posted a task through CrowdFlower that asked the crowd to rewrite four famous quotations, pithily, while preserving their meaning.

In one evening, I was able to get 20 revisions of each quotation from people across the country. I won’t summarize them all here, but I will pull a few highlights.

Original Quotation 1 (from Macbeth, William Shakespeare):

To-morrow, and to-morrow, and to-morrow,/ Creeps in this petty pace from day to day,/ To the last syllable of recorded time;/ And all our yesterdays have lighted fools/ The way to dusty death. Out, out, brief candle!/ Life’s but a walking shadow, a poor player/ That struts and frets his hour upon the stage/ And then is heard no more. It is a tale/ Told by an idiot, full of sound and fury/ Signifying nothing.

Revision 1:

Time marches on, and everyone dies; life is meaningless.
– Hatboro, PA

Life creeps along and ends suddenly like the end of a bad play. The play is dramatic and had poor acting, and has no point or moral in the end.
–Salt Lake City, UT

Original Quotation 2 (from “The Leviathan,” Thomas Hobbes):

No arts, no letters, no society, and which is worst of all, continual fear and danger of violent death, and the life of man solitary, poor, nasty, brutish, and short.

Revision 2:

The life of a man on his own is barbaric and degrading.
– East Aurora, NY

No art, letters or society. Worst of all, living in fear of being alone, poor and short.
– Arlington, TX

Life is all but a mere scam.
–Overland Park, KS

Original Quotation 3 (“First Inaugural Address,” Abraham Lincoln):

We are not enemies, but friends. We must not be enemies. Though passion may have strained it must not break our bonds of affection. The mystic chords of memory, stretching from every battlefield and patriot grave to every living heart and hearthstone all over this broad land, will yet swell the chorus of the Union, when again touched, as surely they will be, by the better angels of our nature.

Revision 3:

We’re fools for fighting each other. We should co-operate, and let our example bring everyone together.
– Plattsburgh, NY

We must be friends, we should now forget each other. Our memories will always stay through our rough times and good times.
– Tallahassee, FL

Original Quotation 4 (from “Chicago,” Carl Sandburg):

“They tell me you are wicked and I believe them, for I have seen your painted women under the gas lamps luring the farm boys.”

Revision 4:

Your reputation follows you and it is a bad one, makeup and lust.
– Iola, WI

This city isn’t a nice place, it’s full of hookers.
– Milledgeville, GA

The full data is available here.

What’s your opinion? Can the revision process be crowdsourced?

I’d love to hear your thoughts.

As the NFL season loomed on the horizon back in August, there was a lot of fantasy football talk around the CrowdFlower office. Naturally, we decided to crowdsource a killer fantasy football team.

Fantasy football tackle. Photo Credit: dennis

We asked CrowdFlower workers to help us build a ranked list of the Top 75 players to guide our fantasy football draft. We used pair-wise comparisons to determine rankings. Specifically, we presented workers with two players, asking them to pick which player they thought would be more valuable.

We used the Top 75 players from ESPN’s 2010 Fantasy Football Draft Kit¹, matching each player with his 74 counterparts, giving us a total of 2,775 player pairs. We also included each player’s position and team, as well as a link to more detailed statistics.

Screenshot of fantasy football task.

After the job finished, we ordered the players according to number of head-to-head victories.

Crowdsourced list of the Top 25 most valuable fantasy football players.

We found that workers choose the player on the left 53 percent of the time, even though each match-up appears twice, once with Player A on the left and again with Player A on the right. With 5,500 data points, this is significant bias toward the player on the left. However, the final ranking doesn’t change even after accounting for this bias.

We settled on two likely explanations for the bias:

1. The anchoring effect of putting something on the left (and as the first response option) may have caused more workers to select the first player.
2. Our Gold was slightly biased toward the player on the left, which we have previously seen² to have an effect on the overall distribution of the answers.

As a first comparison of our crowd of football fans with ESPN’s Fantasy Football brain trust, we plotted each player by the difference between his two rankings.

Players on the left were ranked higher by ESPN, while players on the right were ranked higher by AMT.

Will Chad Ochocinco and Matt Ryan vindicate the crowd? Stay tuned to find out.