Image by Sarah Butrymowicz

Being tapped to write a blog post here at CrowdFlower is usually left to the experts. So with that, let me begin by making the disclaimer that I am neither a political analyst nor a data scientist. But I do have a personal fervor for politics and access to some impressive tools, thanks to my job here at CrowdFlower.

For those who aren’t familiar with CrowdFlower, we specialize in tapping human contributors worldwide to do massive amounts of simple, repetitive tasks (especially tasks that are hard for computers to do by themselves). Here’s a quick how-it-works animation.

I had been reading some old blog posts on the CrowdFlower blog when I came across an interesting 2008 post on election media bias.

I determined that this could be a great opportunity to revisit sentiment analysis, and specifically set out to see if automated sentiment detection tools vs. human assessments could yield any blog-worthy findings.

To see how far the automated sentiment tools have come, I began by using an enterprise-grade social media monitoring tool that provides sentiment analysis.

I ran a few quick monitoring searches of my own to see how the current Republican Primary election was tracking — it seemed a topical place that would be chock full of good commentary.

The instant access to well-organized data from blogs, news sources, and a variety of social media sources was outstanding.

However, I was surprised to find that for each search I conducted, the automated sentiment detection tool consistently returned an overwhelming proportion of “Neutral” ratings (frequently exceeding 90%). This seemed funny to me, given the typically emotive nature of politics.

It’s important to note that this particular tool uses a default value of “Neutral” for any post it cannot interpret.

A particularly interesting subset of the data was several thousand tweets about Herman Cain immediately following the news of alleged sexual harassment by Cain during his time as leader of the National Restaurant Association. Surely this would yield some sentiment-rich commentary that even machines couldn’t resist tagging.

For the posts about Herman Cain on Oct 31st, here is what the machine detected on just under 3,000 posts:

Naturally, I took to the CrowdFlower platform and decided I would run the same data through a simple sentiment analysis workflow.  With the help of our team of crowdsourcing gurus, I utilized some simple, but effective best practices to control for quality (you can get a good overview here). Here is what the CrowdFlower contributors detected:

Here are just a couple of posts marked “Neutral” by the machine and “Negative” and “Positive”, respectively, by CrowdFlower contributors:

Takeaways

A spot check of the results on the automated set confirmed that when the machine actually tagged a post as positive or negative, it was usually very accurate (good precision).

However, the large amount of data that the machine was unable to make a determination on suggests that the pervasive problem of ‘recall’ is still the big challenge with automated sentiment detection.

This graph illustrates the recall difference a bit more clearly. The need for human analysis when dealing with the subtleties of language could not be more apparent.

Automated Tool: Good precision. Poor recall.

CrowdFlower Tool: Good precision. Good recall.

Sentiment Analysis is Insightful AND Entertaining

In addition to the Herman Cain Twitter data, I looked at headlines, blogs, and a broad swath of social media commentary on all the candidates. The conclusion I can draw from my effort is that sentiment detection, is indeed, still a very challenging problem to solve through automation.

This is consistent with what I see here at CrowdFlower daily — in today’s data-wealthy world, there are countless tasks that require human attention (good to know if my blogging career never gets off the ground).

Hopefully I’ll get the chance to continue exploring the sentiment about topical news as it breaks, and will look forward to sharing future findings.

Have experience monitoring sentiment? Let us know if this is consistent with what you’ve seen. Leave a comment.

***

To find out more about how CrowdFlower technology is used for sentiment analysis and a wide range of other human powered projects, visit the CrowdFlower products page.

The votes are in from our Oscar crowdsourcing experiment, and the crowd successfully picked the winners of 14 of the academy awards. For reference, Roger Ebert got 15 predictions correct so we’d have to conclude that the crowd performed reasonably well at predicting the winners of this glorified popularity contest.

Predicted and Actual Winners of the 2011 Academy Awards

One fascinating thing about aggregating responses is that the crowd as a whole will often outperform the average worker. In this case, among the 500 people we polled, the majority of respondents picked fewer than 10 awards correctly (mean of 9.6 and median of 9). And yet, by aggregating all the responses, such that the nominee with the most “votes” is predicted to win, the crowd as a whole correctly picked 14 awards. While the “wisdom of crowds” doesn’t come as much of a surprise, it’s always reassuring to see it confirmed in new applications.

As we noted in our earlier post, though, the more interesting question was whether workers who indicated higher confidence in their responses would outperform workers with lower confidence. Looking at the results, however, we saw no significant correlation between a worker’s predicted accuracy and  actual performance.

"Squint all you want, but there's no pattern"

While it’s certainly possible that we didn’t offer enough of an incentive for workers to estimate their own accuracy, the more likely explanation is that predicting the winners of the Oscars is not something that a person can do with any degree of certainty. Confident or not, the people we polled did not see the “Inside Job” coming.

As a final exercise, we ran a regression on every explanatory variable we could find, including what state workers came from, what day they made their predictions, whether they made their predictions during the day or at night, how long they spent making their predictions and even their historical accuracy on other CrowdFlower tasks. The only variable with any significance turned out to be how long they spent on making their predictions, and while it was significant (at p=0.001), no model we could come up with explained more than 5 percent of the total variation in accuracy.

While the wisdom of crowds seems to extend to picking Oscar winners, the more interesting experiment of having workers self-select as trustworthy is ongoing. In the future, it would be worthwhile to repeat this experiment with questions that can be answered objectively and without uncertainty (solving algebra problems seems like a good candidate), to see if any correlation emerges between predicted and actual accuracy.

With the most glamorous award ceremony of the year just around the corner, we asked 500 people from across the United States to help predict who the big winners are going to be. Below are their predictions, sorted in descending order of agreement.

"I'd like to thank all the little people..."

Based on these early results, Pixar looks like a lock for yet another Best Animated Picture award and Natalie Portman had better start polishing her acceptance speech.

Anyone can make anonymous predictions, so we made things interesting by rewarding workers for answering correctly. We structured the job such that everyone receives 5 cents for completing the survey, plus an additional 2 cents for each correct answer.

To spice things up even more, we asked workers to guess how many predictions they would answer correctly (their “Magic Number,” with a minimum of 5), with the stipulation that we will pay bonuses only to those workers who answer at least their Magic Number of predictions correctly.

As you can see below, while quite a few workers played it safe by selecting the minimum number of correct responses (“5″), the most frequent choice was a Magic Number of 10. Also, more than a few brave souls thought they would get every prediction correct.

But the big question, apart from whether Darren Aronofsky (Black Swan) can edge out David Fincher (The Social Network) for Best Director, is whether any correlation exists between worker confidence and actual performance. When workers have a cash incentive to estimate their own accuracy correctly, do the self-labeled “experts” perform any better than the “novices” at predicting Oscar winners?

Make some popcorn, grab your Kleenex and stay tuned. After they’ve rolled up the red carpet, we’ll come back and see who did a better job of predicting the winners.

When federal investigators asked Goldman Sachs for its transactions with insurance giant AIG, Goldman turned over the information — several hundred billion pages’ worth.

John Carney, senior editor at CNBC.com, had an idea for sifting through the data — crowdsource it.

We agree. In fact, CrowdFlower will categorize and tag the first 100,000 documents at no cost to the government.

If you’re just tuning in, the federal Financial Crisis Inquiry Commission (FCIC) subpoenaed Goldman for its AIG transactions, following accusations that Goldman cooked up a mortgage investment scheme that was rigged to fail.

FCIC has around 50 employees, an $8 million budget, and roughly six months to pore over the five terabytes of data. (Can you say, “Too small to succeed”?)

Clearly, technology presents a double-edged sword for investigators and other regulators.

On the one hand, companies under investigation can use technology to more efficiently bury investigators in terabytes of data (paging Goldman Sachs). On the other hand, technology provides tools for deftly sifting through the data (enter crowdsourcing).

Crowdsourcing public documents may be relatively new, but it’s not unprecedented. In fact, the British Parliament is under way with a project that uses crowdsourcing to investigate MPs’ expenses.

We’ll keep you posted on whether the government takes up our offer.

– Additional contributions by Anisha Sekar.

As anyone who follows political races knows, different sources can report the same event in very different ways. We took nearly six thousand recent articles over the past month about Clinton and Obama and sent them to Mechanical Turk to be classified as favorable or unfavorable for the respective candidates. We scraped the articles from Google News restricted to several sources, and threw in front page headlines from Digg.

Here is the graph for favorability scores, aggregated by source. We found that Digg was far and away the most favorable for Obama.

The next graph tracks overall news favorability by date. To provide some context, we compared it with the change in Obama stock on the Intrade prediction market.

More details after the jump:

We created our data set by doing two separate searches, one for “Barack Obama” and one for “Hillary Clinton”. This did a pretty good job ensuring that results from Google News or Digg’s search facility demonstrated how the article was about the given candidate. For each article, we showed the headline, search result snippet, and link to several Turkers. They reported whether it was positive, neutral, or negative toward the candidate.

The favorability metric was created by averaging the ratings across articles. Pro-Obama and anti-Hillary articles were both worth 1 point; anti-Obama and pro-Hillary both worth -1, and neutrals 0.

Therefore, if all articles are either positive towards Obama or negative towards Hillary, the rating is +100%; and vice-versa for -100%.

The data is very noisy. The question of favorability is extremely tricky: it includes a combination of expectations, sentiment, and the objective events a newspaper chooses to report. All of these are hard to reliably assess or even define. (And whether anything you measure constitutes “media bias” is another complicated question!)

Despite all this philosophical intractability, the data must be showing something real, because we have a statistically sound result: the difference between Digg and the others was statistically significant (t-test, p<.001). The differences within the mainstream media were not statistically significant.

-Brendan, Chris, Lukas, Mike

Human annotators are great at providing basic information about images. We were wondering if we could find something interesting about magazine covers. Stumbling upon 2800 Sports Illustrated cover images going back to 1954, we sent them to Mechanical Turk, asking people to identify the race and gender of the person featured (if any), and what sport was depicted. There are lots of interesting things in this data; this post will touch on just a few we’ve had time to whip together some graphs for.

Here is a historical graph of the frequency of how often people of different races appear on the cover of Sports Illustrated. The story is simple and striking:

Next: which sports get featured on the cover? Here’s a chart for several sports over that same time.

It might be possible to find links between the careers of famous athletes and rises and falls their sports’ popularity; for example, boxing peaks in the 70’s (Muhammad Ali?), basketball peaks in the 90’s (Michael Jordan?) and golf bounces back in the 90’s after a long decline (Tiger Woods?).

Many other sports appear in the data, too; for this chart, we made sure to pick the three most common, and a few other particularly interesting ones. Percentages don’t add up to 100% because we didn’t plot all the other sports, including things like horse racing which used to be much more popular. If you’re really curious, here’s the full chart of all sports we asked about, including many of the smaller ones.

-Brendan