1. People like asking questions, people love giving answers (even if you don’t pay them).

Charlie Cheever, the founder of Quora, pulled back the curtain and talked about who the Quora user is, and why the crowd loves to answer questions—for free.

Quora’s success has come from prioritizing quality over quantity. This means one question, and oftentimes a lengthy discussion leading to a singular answer. Answers come in the form of an essay, or a research paper, due to the fact that the Quora community is largely made up of college undergrads and graduate students.

All of these users have an abundance of free time coupled with a very specific knowledge set, the perfect recipe for a good answer fueled by the right crowd.

2. You may not be able to crowdsource good AV when planning a conference, but it’s an ideal platform for finding your next best friend.

3. Moby Dick Makes a Great Emojicon Story Too!

Fred Benenson from Kickstarter used the crowd to translate the entirety of Moby Dick into emojicons. To take this to the next crowdsourcing level, the whole project was funded by 83 backers on Kickstarter.

4. Philip Rosedale crowdsourced an iPhone app for $2,485.

Using his newly released crowdsourcing platform Coffee & Power, Philip developed his entire company infrastructure and platform through a globally distributed workforce. 288 contributors in 127 locations worked together to get this startup off the ground in a whole new way.

The Coffee & Power platform was built in 1,700 commits ranging from $6 quality checks all the way up to full source-code editing. One element of this process was developing the Hudat iPhone app. In less than a month for $2,485, the Coffee & Power community got this mobile app up and running.

5. Tech journalists believe in the crowd.

Martin Giles, the US Technology Correspondent for The Economist and Rafe Needleman, CNET editor, moderated panels on “Industry Champions” and “Cloud Labor” respectively.

These panels took a closer look at the real life day-to-day applications of crowdsourcing, and how they are shaking up traditional industry models. TopCoder, Task Rabbit, GigWalk, Get Satisfaction, Mechanical Turk, LiveOps, uTest, and Trada were all on stage to talk about how the crowd is changing the way work, as we know it, is being done.

As Rafe points out in this piece from October, the crowdsourcing model does not all lie in the cloud. Many of the previously mentioned companies are using the crowd to connect people for labor being done in the real world. Skilled tasks ranging from home and auto repairs to resume writing to Christmas shopping are all being done by the crowd.

6. A crowd is a crowd, humanitarian or paid.

Much of the conversation at CrowdConf was about financially motivating the crowd. This panel, dedicated to crowdsourcing humanitarian work, was an insightful shift in perspective.

Jeannie Stamberger from Carnegie Mellon Silicon Valley moderated a panel with Rob Munro of Global Viral Forecasting Initiative, Leila Chirayath Janah of Samasource, Patrick Meier from Ushahidi and Vijay Pande, Associate Professor at Stanford University. These organizations are taking on issues such as tracking real-time disease outbreak and interactive mapping for monitoring crisis information around the world.

7. Plinko has great use outside of “The Price is Right.”

Charles Festa talked about the success threadless.com has found by tapping the design community within the crowd. Interestingly, he also went into detail about how the same company failied to get another project off the ground using the exact same platform.

That project was Naked & Angry, a pattern design concept that never really took off. For this project they used the shotgun approach to engage designers through popular social media channels. Looking back, Charles thinks a more precise trial and error process, much like the one used on the popular Price is Right game “Plinko”, would have been a much more effective way of communicating with the textile design community.

8. 12 hours of hacking and you can control the “crowd” into detecting skin cancer, predicting the weather, and cleaning up neighborhoods in India.

The inaugural CrowdConf Hackathon was a huge success. Brendan Gill and Sina Khanifar were awarded the Best Hack for their Android app that pulls in barometric data from Android phones for a real time weather map called … that’s right, WeatherMapper. For their efforts, Brendan and Sina received a massive CrowdHack Khukuri knife flown in from Nepal.

Tim Olson was awarded the Best Use of CloudFactory API for Clean Up India. Tim used CloudFactory to commission people in India to leave their computers to go out and clean up a street or park in their neighborhood. They had to show a before and after photo in order to complete the task and get paid.

The Best Use of CrowdFlower API award went to John Le and Dave Oleson for melaKNOWma. Their hack allows users to upload images of any moles or growths they have to check for malignancy. It then sends the images to the crowd, where they have workers assess the mole’s asymmetry, borders, and color.

Arran Bardige got the Best Use of Twilio API for Ringing Restaurants. Arran’s hack used the Twilio Robot to call local restaurants to see if they would be a good fit for his vegetarian girlfriend.

9. 250+ companies from more than 20 countries, ranging from single-employee startups to Fortune 50 companies came together to talk about crowdsourcing.

10. The Crowd can help make next year’s CrowdConf even better.

CrowdFlower is already planning CrowdConf 2012 and are looking forward to the same growth, enthusiasm, and creativity that have been building over the last two years. If you have ideas or thoughts for speakers, topics, sponsors, locations, beverages, or any other content-related matter, contact mollie@crowdflower.com.

CrowdConf is far from being the only event we participate in throughout the year. Visit our Events and Announcements Sign Up Page to stay in the loop about all things crowd all year long.

Image by Sarah Butrymowicz

Being tapped to write a blog post here at CrowdFlower is usually left to the experts. So with that, let me begin by making the disclaimer that I am neither a political analyst nor a data scientist. But I do have a personal fervor for politics and access to some impressive tools, thanks to my job here at CrowdFlower.

For those who aren’t familiar with CrowdFlower, we specialize in tapping human contributors worldwide to do massive amounts of simple, repetitive tasks (especially tasks that are hard for computers to do by themselves). Here’s a quick how-it-works animation.

I had been reading some old blog posts on the CrowdFlower blog when I came across an interesting 2008 post on election media bias.

I determined that this could be a great opportunity to revisit sentiment analysis, and specifically set out to see if automated sentiment detection tools vs. human assessments could yield any blog-worthy findings.

To see how far the automated sentiment tools have come, I began by using an enterprise-grade social media monitoring tool that provides sentiment analysis.

I ran a few quick monitoring searches of my own to see how the current Republican Primary election was tracking — it seemed a topical place that would be chock full of good commentary.

The instant access to well-organized data from blogs, news sources, and a variety of social media sources was outstanding.

However, I was surprised to find that for each search I conducted, the automated sentiment detection tool consistently returned an overwhelming proportion of “Neutral” ratings (frequently exceeding 90%). This seemed funny to me, given the typically emotive nature of politics.

It’s important to note that this particular tool uses a default value of “Neutral” for any post it cannot interpret.

A particularly interesting subset of the data was several thousand tweets about Herman Cain immediately following the news of alleged sexual harassment by Cain during his time as leader of the National Restaurant Association. Surely this would yield some sentiment-rich commentary that even machines couldn’t resist tagging.

For the posts about Herman Cain on Oct 31st, here is what the machine detected on just under 3,000 posts:

Naturally, I took to the CrowdFlower platform and decided I would run the same data through a simple sentiment analysis workflow.  With the help of our team of crowdsourcing gurus, I utilized some simple, but effective best practices to control for quality (you can get a good overview here). Here is what the CrowdFlower contributors detected:

Here are just a couple of posts marked “Neutral” by the machine and “Negative” and “Positive”, respectively, by CrowdFlower contributors:

Takeaways

A spot check of the results on the automated set confirmed that when the machine actually tagged a post as positive or negative, it was usually very accurate (good precision).

However, the large amount of data that the machine was unable to make a determination on suggests that the pervasive problem of ‘recall’ is still the big challenge with automated sentiment detection.

This graph illustrates the recall difference a bit more clearly. The need for human analysis when dealing with the subtleties of language could not be more apparent.

Automated Tool: Good precision. Poor recall.

CrowdFlower Tool: Good precision. Good recall.

Sentiment Analysis is Insightful AND Entertaining

In addition to the Herman Cain Twitter data, I looked at headlines, blogs, and a broad swath of social media commentary on all the candidates. The conclusion I can draw from my effort is that sentiment detection, is indeed, still a very challenging problem to solve through automation.

This is consistent with what I see here at CrowdFlower daily — in today’s data-wealthy world, there are countless tasks that require human attention (good to know if my blogging career never gets off the ground).

Hopefully I’ll get the chance to continue exploring the sentiment about topical news as it breaks, and will look forward to sharing future findings.

Have experience monitoring sentiment? Let us know if this is consistent with what you’ve seen. Leave a comment.

***

To find out more about how CrowdFlower technology is used for sentiment analysis and a wide range of other human powered projects, visit the CrowdFlower products page.

Cell counts | Flickr - Photo Sharing!

Lab scientists spend countless hours manually reviewing and annotating cells. What if we could give these hours back, and replace the tedious parts of science with a hands-off, fast, cheap, and scalable solution?

That’s exactly what we did when we used the crowd to count neurons, an activity that computer vision can’t yet solve. Building on the work we recently did with the Harvard Tuberculosis lab, we were able to take untrained people all over the world (people who might never have learned that DNA Helicase unzips genes…), turn them into image analysts with our task design and quality control, and get results comparable to those provided by trained lab workers.

Here’s how:

We took cortex slide images from mice provided by a neuroscience lab at Harvard University. We cut each image into smaller pieces, so they’d be easier for people to work on.

After a brief set of instructions, contributors were instructed to count the neurons in the slide by clicking on each individual neuron.

Fig 1: example slide

Contributors were given examples of edge cases (i.e. Is this a cell or not? Is this two or three cells?). Contributors identified and clicked on every cell, which placed a green marker on top of each cell. An automated counter kept track of the number of clicks.

We controlled quality using Gold Standard (“Gold”) units, which were images with known cell counts that we added to the task. The benefits here are threefold. First, Gold questions provide training and feedback to our contributors, so that they can get better at the task over time. Second, contributors don’t know which questions are Gold, forcing them to honestly answer all questions. Finally, if a contributor fails to answer enough Gold correctly, we remove them from the job.

Fig 2: Map of all contributors on this task. Step it up Iceland!

Visit the batchgeo.com navigable map.

After removing all of our “untrusted” contributors, we are left with our “trusted” contributors. Each image had four trusted contributors count the neurons. We took the average count less any outliers in order to get the most accurate results.

How did this experiment actually turn out? How did our results compare to those of trained lab workers? In short, we performed extremely well. As you can see in our results below, the average difference between the Crowd and professional lab counts is 2.0%, which can be chalked up to ambiguity in certain clusters of cells.

Future iterations of this will include breaking the images into smaller pieces, as well as using automated solutions to tag the high confidence units, leaving only the edge case cells that require human eyes.

We hope that results like these will encourage more scientists, labs, and biotech firms to crowdsource pieces of their research. We believe this could free up their time for more complicated work, decrease the latency of results for experiments, and quicken the pace of scientific discovery.

Yesterday it was TB Cells, today its neuron cells; tomorrow: cancer cells? We’re issuing an open call for any university lab, biotechnology, or pharmaceutical company with large image data sets: contact us to utilize the crowd to shorten the life cycle from idea to major scientific advancement.

via: FastCompany

About the author: Philip Rosedale is the creator of Second Life and a Co-Founder of LoveMachine, Inc.

My co-founder Ryan and I are having so much fun pulling together data and thoughts for my upcoming keynote at CrowdConf next week. It’s a great opportunity to try and summarize much of what we’ve learned over the last few years about whether and how crowdsourcing can be taken to the next logical (we think) level: to replace a bunch of what we’ve come to think of as the nature of “work” and “company”.

The Silicon Valley startup formula is now a well-recognized and time-honored strategy, which I think we’ve all worn into a bit of a rut: 3 or 4 very smart people (usually guys) hunker down in someone’s garage, work a bleary-eyed 80 hours a week producing a prototype, getting funding, hiring those first handful of key engineers, etc.

But what would happen if you never hired anyone at all? What about doing a ‘real’ Silicon Valley startup – meaning a new, risky, design-sensitive idea backed by venture capital – but without the garage and the crunch time and team of ‘A players’? What if you built a system that instead allowed people all over the world to be paid small amounts of money to help you prototype, build, and then release a high quality new website and software?

Well that’s what we did, and there is a lot to talk about!

-Philip

Come hear Philip and many other experts in the crowdsourcing field speak at CrowdConf 2011. It is a great opportunity to discover what is happening in the world of crowdsourcing. It starts next week, November 1-2 at the Mission Bay Conference Center in San Francisco. You can buy your tickets here. Hope to see you there!

Do people perform tasks only fleetingly, or has crowdsourcing become more of a long-term engagement? Furthermore, just how important is contributor retention in the world of crowdsourcing?

While a majority of people fall into the “one-and-done” camp, many of the most productive contributors tend to have participated in previous jobs. Within any single job, these seasoned veteran contributors also provide far more work than their less experienced counterparts.

Over a period of two months, we ran a series of five very similar jobs, retrieving ratings information for businesses throughout North America. In total, we collected over half a million judgments from a total of 2,901 unique contributors,¹ representing multiple labor channels and 101 countries. As a first test to whether seasoned veterans are common, we looked at how many people participated in more than one job. In total, 2,389 people participated in one job only, meaning that First-Timers accounted for over 82 percent of all contributors.

But this may not be the best number to look at. We’re really interested in whether certain people recognize and seek out specific types of tasks after having worked on them before. To look for this behavior, we analyzed the most recent job, counting how many people participated in at least one prior iteration of the task.

For a recent job, a total of 906 people participated, 247 of whom had done the task previously. By this measure, Seasoned Vets constitute approximately 27% of the workforce. While the impact of returning contributors is greater under this methodology, the fact remains that these “loyal” contributors are firmly in the minority on this series of jobs.

Count of Contributors by Profile

However, as we’ve seen before, the individual impact of contributors varies widely, with a minority of people often providing the vast majority of work. With this in mind, we looked at the contributions of First-Timers and Seasoned Vets and found some striking differences.

Share of Contributions by Profile

Seasoned Vets, while constituting only 27% of the workforce, provided 47% of the total work completed. On average, each Seasoned Veteran provided 2.5 times more judgments than their less experienced counterparts. It’s also interesting to note that there was no significant difference between the quality of work provided by First-Timers and Seasoned Vets, no doubt due to our suite of quality control measures.

Given that the people who stick around tend to be far more productive, improving retention is a useful consideration (for these jobs, at least). We’re now interested in how best to attract people to return to the types of jobs they’ve already seen. That, however, is a work in progress and a story for another day.

1. Note that this analysis only considers Trusted workers.

via: reddead.wikia.com/

The standard CrowdFlower model is agnostic towards the quality of any individual contributor. Typically, we let anyone attempt a task, using our technology to filter out low-quality contributors and score the responses. Without further ado, what follows is quick review of the steps we take to do that filtering.

Gold (What is it Good For?)

In almost every job, we take a subset of the data to be processed and manually score the correct response. This manually-scored set, which we refer to as Gold Standard Data, is at the core of managing quality in the context of enterprise crowdsourcing:

• Filtering: We use Gold to create an up-front test, creating a barrier to entry such that only workers who understand and successfully complete a task are allowed to participate. This allows us to prevent unsavory characters from entering jobs and contaminating results.
• On-going Training: We also use Gold to conduct on-going training, offering corrections for units that are answered incorrectly. This allows us to continually instruct and improve highly prolific contributors.
• Dynamic Trust Score: We use each contributor’s performance on Gold as a basis to determine their overall accuracy within a task. Each contributor must exceed our minimum trust thresholds to continue working on a task. If at any point a contributor falls below the trust threshold, we’ll exclude his work.

Because creating Gold is labor-intensive, we’ve created an automated process to generate Gold using units that have already been completed. This has significantly reduced the time needed for setup and ongoing job creation, without sacrificing our ability to differentiate contributors.

Of course, the amount and distribution of Gold is critical. Often, a uniform distribution of Gold across response types is ideal, though in certain situations we’ll use a skewed Gold set. For example, in an experiment on crowdsourced document review, we used a skewed Gold set to avoid missing relevant documents (reduced “false negatives,” if you prefer).

Department of Judgment Redundancy Department

If the purpose of Gold is to manage the quality of individual contributors, we use multiple judgments per unit to improve the accuracy of completed units. The basic premise is simple enough. We look for agreement among trusted workers to indicate correct responses at the unit level. For example, if we ask four people to verify a phone number for a business, the answer is more likely to be correct if all four agree. In fact, every unit processed by CrowdFlower is annotated with a response as well as a Confidence Score (based on agreement weighted by Trust, plus some secret sauce).

More generally, assume we set the trust threshold for a given job at 70 percent (meaning that anyone who doesn’t answer at least 70 percent of Gold correctly gets booted) and that contributors are uniformly distributed in terms of ability (not true, but convenient). We can easily model the effect of additional judgments on estimated accuracy, showing that the probability that the majority response is correct increases with the number of judgments collected:

Of course, while collecting 10 judgments per unit yields highly accurate results, it may not be the most efficient way to structure a job. Imagine that the first 2, 4 or even 6 contributors agree on how a unit should be classified. At some point, the marginal impact of an additional judgment is not worth the additional cost. We’ve automated a process to vary the number of judgment each unit receives based on agreement thresholds, so that we can reach accuracy targets more efficiently.

The following shows actual results from a sample job, where we set a minimum confidence threshold of 0.7:

Approximately 50 percent of units completed with just 2 judgments and 75 percent completed with 4 or fewer. In any case, each unit received only as many judgments as necessary to reach the confidence threshold. For any job, some subset of units will be ambiguous enough that they won’t reach a confidence threshold, so we use also set maximum judgments cap to “stop the bleeding.” Depending on the specific circumstances, we may reroute those ambiguous units to a parallel process with different structure, contributors, etc. for another round of judgments.

One More Trick

For complex tasks, we have developed a workflow management system to link together multiple jobs. For example, we might ask one pool of contributors to write a product description, verify the spelling and accuracy with a second pool and rank the subjective quality with a third pool. Alternatively, we might take a business listing and break out each attribute for independent collection and verification, with a separate job for name, address and phone number, or cuisine type, cash-only, types of credit cards accepted, on-site parking, or any other attribute that can be verified online. In general, peer review means that we can always give data a second pass to improve accuracy.

Expanse of Cheese: Caseificio di San Silvestro's Aging Room in Castelvetro (MO)

Last May, I took a trip to Italy for two weeks. A little bit of history: my friend Jessica and I are both Italophiles, and when her mom sent us a link to a video contest where the prize was a round trip flight to Italy, we knew we had to enter. After a week of writing and editing lyrics in a Google Doc — half in Italian, half in English — the resulting music video ended up winning us a trip to the holy land of olive oil, vino, and other delectable edibles.

Apart from being a passionate eater, I’m a passionate supporter of the Slow Food movement, an organization which promotes good, clean, and fair food around the world. Each year, they publish a guidebook to restaurants in Italy that adhere to their principles. In Italy, this usually means each restaurant is handpicked to showcase the traditional food of a particular region; each restaurant supports artisanal methods and products that otherwise might go extinct (were eaters not eating them), and where the food is most likely naturally organic and local anyway.

But, a problem: it’s 2011, and I’m more apt to travel through interactive maps than with old-fashioned guidebooks. More importantly, I don’t plan, and I needed to know what edible options were around me at any moment in time on my trip. What I really needed was a version of the guidebook, in map form, that I could use on my mobile. No such option existed, of course, and I was left with two options: magically visualize restaurants around me by poring through the guidebook, or, crowdsource it.

What I needed to do (and, what much of our work at CrowdFlower comprises), was structure unstructured data. To create the map from the book, I went through the following steps:

• Obtained a PDF version of the book
• Split the book into pages, and uploaded each page as a separate PDF (thanks to pdftk)
• Created a CSV (comma separated values file) with each page’s PDF link and page number
• Created a crowdsourcing task to structure the data, using the previously uploaded individual PDF pages
• Geocoded the structured data
• Output the geocoded data in KML (Keyhole Markup Language) form
• Uploaded the KML file to a mapping site (e.g. Google Maps)

Conveniently, each page that outlined a restaurant in the PDF was formatted nearly the same. This made it easy to give instructions to workers, as seen below:

Instructions

For each area on the page, workers were asked to copy and paste specific sections into the task. Each page, then, was split up into corresponding parts (Region, City, Directions, Restaurant Name, and the two Capture Areas). This is the essential concept here: structuring the unstructured data such that I could later geocode it properly, and display it in the way that I needed.

The tonnarelli cacio e pepe at Da Felice a Testaccio in Rome (used in the Geocode example)

Once the task finished, I downloaded the resulting CSV file, and whipped out Google Refine (a.k.a. Excel on crack), which has a feature that allows you to enter a template API call that changes based on specific values in each row. Using the Google Geocoding API (any will do), I constructed the following API call, using the address value in each row as the “address” parameter for each API call:

http://maps.googleapis.com/maps/api/geocode/json?address=via+Mastro+Giorgio%2C+26+Roma+Lazio&sensor=false&region=it&language=it

After slicing and dicing the rest of the data into bits that I wanted to display on a map, I used Google Refine’s “templating” feature to export each row as a Placemark in KML format. Finally, I uploaded the resulting KML file into several different maps, each representing one region in Italy.

A portion of the Campania map - tons of places to explore around here!

Try it out! Check out the map for Tuscany, and the map for Campania.

If you want to give crowdsourcing a spin, head on over to our self-service product and click “sign up”. Or, if you’d prefer a hands-off “we-do-everything” approach, contact us to get started.

Watch the video that started it all: Io Sono Balsamico by Balsamico

Aron is a Crowdsourcing Project Manager at CrowdFlower, and is the resident agriculturalist-eater. Follow him on Twitter (@aron) for more sage bits of agricultural-eating learnings.

psychologytoday.com

Evidence in experimental psychology suggests that most people overestimate their own ability to complete objective tasks accurately. This phenomenon, often called confidence bias, refers to “a systematic error of judgment made by individuals when they assess the correctness of their responses to questions related to intellectual or perceptual problems.” ¹ But does this hold up in crowdsourcing?

We ran an experiment to test for a persistent difference between people’s perceptions of their own accuracy and their actual objective accuracy. We used a set of standardized questions, focusing on the Verbal and Math sections of a common standardized test. For the 829 individuals who answered more than 10 of these questions, we asked for the correct answer as well as an indication of how confident they were of the answer they supplied.

We didn’t use any Gold in this experiment. Instead, we incentivized performance by rewarding those finishing in the top 10%, based on objective accuracy.

Does Bias Exist?

To estimate confidence bias, we looked at the difference between the average of how confident an individual was of his/her answers and how many he/she answered correctly. If the difference is positive, the individual overestimated how well they did. Amazingly, over 75% of contributors overestimated their ability to answer multiple choice questions correctly.

Are Individuals Consistently Biased?

Because our dataset consisted of Math and Verbal questions, we looked at each individual contributor’s confidence bias for both types of questions. In aggregate, people tended to have more trouble with the Verbal questions (average accuracy of 28%, compared to 41% for Math), though the average confidence score was nearly identical (63% +/-1).

The vast majority of contributors fall into the “overconfident on both” quadrant (top right), while only a handful of contributors were overconfident for one question type and underconfident for the other (top left and bottom right quadrants). Overall, there is certainly a correlation between bias scores on the two problem types, suggesting that many individuals are consistently biased on different types of problems. However, this explains only a portion of the variation.

Does Bias Vary Across Groups?

Given that overconfidence seems to be a consistent trait, we were curious how this trait varies across the different groups making up our contributor pool. We sliced and diced our contributors into a number of different sub-groups, which are summarized below.

There are a lot of interesting things going on here. To highlight a few, accuracy increases consistently as the contributor’s education level advances from High School to College, but so does confidence, leaving the bias score nearly unchanged. There’s a similar pattern with Age, with older contributors tending to be both more accurate and more confident.

Gender and Location also have an effect on confidence bias. Taking the two countries that supplied the most people, contributors from the US were much more accurate and slightly more confident than the average, while those from India were average in terms of accuracy but much more confident. As such, the bias score for contributors from India is nearly double that of contributors from the US. With respect to gender, confidence didn’t vary much, but women were more accurate and thus less biased than men. Moving on.

Further Research

In the context of experimentation, we decided against using Gold to minimize any selection bias among contributors. However, this makes it difficult to apply these results to enterprise crowdsourcing, at least as practiced by CrowdFlower. In the future, it would be interesting to look at confidence bias among trusted workers only, and particularly among trusted workers with repeated experience in specific job types. We would expect these workers to have a better sense of whether their answers are correct, though it is possible (and perhaps likely) that confidence would increase along with accuracy.

1. Pallier, G., Wilkinson, R., Danthir, V., Kleitman, S., Knezevic, G., Stankov, L., & Roberts, R. D. (2002). The role of individual differences in the accuracy of confidence judgments. Journal of General Psychology, 129,257–299

]]> http://blog.crowdflower.com/2011/09/confidence-bias-evidence-from-crowdsourcing/feed/ 0 http://blog.crowdflower.com/2011/08/crowdsourcing-and-sem-now-with-even-more-cat-pics/ http://blog.crowdflower.com/2011/08/crowdsourcing-and-sem-now-with-even-more-cat-pics/#comments Fri, 26 Aug 2011 18:58:20 +0000 Zoe Vance http://blog.crowdflower.com/?p=3450 Every modern business wrestles with the elusive lady that is the search engine and the potential she offers to connect with customers. Google and Bing make it easy for anyone to buy keywords and drive customers to a website, but what keywords are our customers searching for? Would a sales manager frustrated with the average 70-80% accuracy of business listings bought from data providers search for “crowdsourcing,” “address checking,” or something else entirely? Since we’re a crowdsourcing company, we had to try crowdsourcing the solution …

In the last two weeks of my summer internship at CrowdFlower, the marketing team challenged me to generate the widest range of search engine seed terms that could be used in SEM keyword tools to generate “hot” search phrases. For those of you who’ve dealt with SEM, you know that thinking of seed phrases to plug into these tools can be a painfully frustrating and surprisingly difficult task. (For those of you who haven’t and don’t believe me, try right now to describe what your company does — or anything for that matter — in 10 significantly different ways.)

Keyword tools are based on your thought process, which takes care of the customers who are thinking in the same way you are, but what about all the people of a different mindset who are trying to find your solution? For example, if I were looking for pet grooming services, depending on my thought process, vocabulary range, and amount of sleep the night before, I could search anything from “pet grooming salon” to “quality feline hair cuts” to “kitty bad hair day.” The challenge was to understand the full breadth of how the crowd approaches a certain problem, essentially the perfect task for the crowd.

Looks like PetSmart forgot to buy an ad for "kitty bad hair day"

I set up the job to have the contributor imagine working for a business that could use CrowdFlower’s services (whether or not they know of them) and then have the contributors write search queries they would use to find a solution to their problem on the Internet. I created numerous versions for each scenario, varying both the industry jargon and the background we gave the contributor, so that I could fully mimic the knowledge and background of a real customer, and thus elicit as wide a range of responses as possible. I then ran a second job to rate those queries to identify quality results worth pursuing. Finally, I took those quality terms and fed them back into a keyword generator tool, making sure that we had the full range of potential search phrases optimized to what people would search for the most.

An unspoken secondary challenge was “Can you become a full-fledged crowdsourcing ninja before you leave?” This final job I ran was by far the most challenging because crowdsourcing content generation requires a seemingly daunting use of CrowdFlower’s automated workflow system, which uses gold units and an active, real-time peer review system. I successfully passed this final test, with the help of our in-house content generation team.

Sample Generated Keyword Seeds

The keyword gen job yielded really interesting results. There were some genuinely clever and alternative thought process seeds, some that we were already using, and some that were borderline ridiculous. My personal favorite keyword seed though, as the office intern, had to be “how to find an intern” — the obvious solution to all life’s problems.

As more and more projects with CrowdFlower graduate to production, different groups (engineering, product management, quality engineering) across the organization are seeing the benefits of crowdsourcing and are keen on embracing this new paradigm.

With so many projects in-flight and so many ideas coming up, would it make sense for organizations to setup a Crowdsourcing Center of Excellence (CCE) much like the PMOs?

I personally think that there are a lot of advantages in doing this. Here are my thoughts:

• Crowdsourcing is growing fast – new tools and technologies are emerging daily. It would benefit for a centralized team to keep abreast of the developments in this space.
• Crowdsourcing is an emerging solution, so it is important that everyone in the organization understand what Crowdsourcing can and cannot do. The CCE can evangelize, educate, mentor, and consult with teams and individuals on crowdsourcing.
• The CCE can design and implement processes and methodologies for the organization. The CCE can also develop tools that can be deployed across the organization.
• The CCE can serve as the centralized office to evaluate the many vendor options, and select the right one(s).
• The CCE can design, implement, and help manage metrics and KPIs at a project and program level.

I am intentionally excluding from the charter the responsibility of running individual projects. There are pros and cons of having a centralized (or decentralized) process to implement and manage your individual projects.

I would like to hear from the crowdsourcing community, your thoughts and feedback on the pros and cons of establishing a CCE.

In particular, what do you think about these two specific questions? Thanks in advance for your feedback.

1. Do you think organizations will benefit from establishing a CCE?
2. The CCE will obviously involve cost – to establish and run. Will this investment be justified?

KPIs (via howtoworkthis.com)

This is part of a series of guest posts by Ram Rampalli, our crowdsourcing partner at eBay.
Part I – Assessment Stage
Part II – Pilot Stage
Part III – Analysis Stage
Part IV – Production Stage

About the author: Ram Rampalli created and leads the crowdsourcing program within the Selling & Catalogs team at eBay Inc. You can follow him on Twitter (@ramrampalli)

Building a successful portfolio of crowdsourcing projects – Part 4

In the first three parts of this series, we discussed the Assessment, Pilot, and Analysis & Optimization stages. Now that the task is moved to production, what steps can you take to manage this effectively?

1. Leverage the CrowdFlower API

When you move to production, the task is already designed, but you still need to provide input data. If you want the crowd to perform a product categorization tasks, you need to provide the crowd with the product and category information.

At this point, while it is not required, I recommend integrating with the CrowdFlower API. Integrating with the CorwdFlower API allows you to send and receive data seamlessly. This allows you to get your data back as it’s completed without waiting for batch results in the form of a CSV.

2. Identify the KPIs

With the success of our early projects here at eBay, we quickly added new projects and soon had seven concurrent projects running with CrowdFlower. With this type of volume, it is impossible to track every key metric for each project all the time.

I recommend ranking the top three key metrics and identifying them to the CrowdFlower team. This will give the CrowdFlower team targeted metrics for which they can optimize.

Closing Remarks

Crowdsourcing is a relatively new paradigm. Different companies have implemented this paradigm in different ways, focusing on different ways of structuring their interactions with the crowd. So just because a project did not work under one model does not necessarily mean that the project will never work.

These companies are also constantly innovating, coming up with new products and solutions. Over the last 18 months we’ve worked closely with the CrowdFlower team to develop features that are now staples of their platform.

For me, every project has been a learning experience, and as I work on newer projects, I have lots to learn. I hope you enjoyed this four-part series. I would love to hear your thoughts and comments. Happy crowdsourcing!

It is high noon in business listing verification crowdsourcing land. We are throwing down the gauntlet. We are stepping in the ring. We are mixing our metaphors.

Undramatic Intro

Yelp engineers recently described their efforts to correct business listing data using Amazon Turk. They tapped the services of 4,660 contributors; only 79 passed their quality assurance testing (1.7% of contributors were “trusted”), and the data they output was (very roughly) 80% accurate.

This smelled funny to us. Our business listing verification service routinely returns results above 97% accuracy. In fact, some of the most recognizable names in local search and business data pay for that service. (See a full report on 100,000 listings we did for a major search company to see some typical figures). Out of the last couple dozen crowdsourcing tasks we’ve run, the absolute minimum proportion of contributors who were “trusted” was 34%. But more importantly, our platform identifies these trusted contributors within minutes, meaning the best contributors get the job done quickly.

URL Precision Numbers (excerpted from an actual past report to client)

So. Why Did Yelp’s Project Struggle to Meet Enterprise Accuracy Standards?

It was not because of a lack of brains. The bios of the eight folks at Yelp who worked on this project are smattered with words like “Harvey Mudd” and “Computer Science” and “Stanford” and “PhD”.  And they work at Yelp, which is, y’know, awesome.

And it was not because of a lack of good contributors. CrowdFlower has first-hand experience with well over one million contributors (many from the Mechanical Turk platform), and, when given the right tools and feedback, we’ve found them to be very accurate.

No, Really. Why?

It was in part because the Yelp team did not have the tools developed over the years by CrowdFlower’s crack engineering team.

• Our contributors face ongoing tests as they complete work, and whenever they get an answer wrong they are given feedback as to why they were wrong in real-time; these tests are carefully calibrated to test for the most common types of contributor errors.
• Our contributor UIs are the products of dozens of A|B tests run through CrowdFlower’s custom A|B testing infrastructure.
• We use digital assembly line technology, chaining together many very simple tasks to yield a complex result. The below is a (somewhat outdated) representation of our business listing verification assembly line, where each blue box represents one discrete user task:

Digital Assembly Line

Just as important… this project probably struggled to succeed because crowdsourcing to enterprise standard quality is incredibly hard! The business listing verification team at CrowdFlower only succeeded after a full year and tens of millions of human judgments. We worked quite a few 24-hour days, and our social skills atrophied from lack of use.

In the end, we have achieved a solution that is fast, accurate, and affordable to use – and continues to be improved upon.

We did all this work so others won’t have to. If you’re contemplating going it alone, give us a call! It’s not worth it! So many people love you!

The Challenge

On behalf of our contributors, CrowdFlower, and the business listing verification team, I’d like to offer a challenge to you, friendly neighborhood Yelp engineers.

Give us 5,000 business listings. If we can raise the precision of those listings to 95%+ and beat any machine learning algorithms you can build, you give us two engineers. No, just kidding. If we can do so, you’ll write about the experience on your engineering blog.

If we lose (actually, regardless of whether we win or lose), we’ll happily sit with you to show and tell you everything we’ve ever learned about business listing verification.

For more information and actual sample data from another Business Listing Verification project, check out this customer report.

via: www.jamorama.com

This is part of a series of guest posts by Ram Rampalli, our crowdsourcing partner at eBay.
Part I – Assessment Stage
Part II – Pilot Stage
Part III – Analysis Stage
Part IV – Production Stage

About the author: Ram Rampalli created and leads the crowdsourcing program within the Selling & Catalogs team at eBay Inc. You can follow him on Twitter (@ramrampalli)

Building a successful portfolio of crowdsourcing projects – Part 3

In the first two parts of this series, we discussed the Assessment and Pilot stages. Now that the pilot task is finished and you have a copy of the results file, it’s time to analyze these results and plan for the next steps.

What can you expect to get from CrowdFlower?

CrowdFlower can provide you both the aggregated judgment file (a single “consensus” judgment per unit) and the full judgment file (all the judgments collected for that unit). Each judgment is annotated with additional data, including date/time collected, labor channel and geographic origin.

Analysis

Post pilot, we analyze the results of the data. Many new projects that we pilot with CrowdFlower are also operational through one of our outsourcing partners. Therefore, we compare the performance of CrowdFlower with that of our outsourcer on three metrics:

• Performance (Speed)
• Cost
• Quality (Accuracy)

The project sponsor marks the project metrics using traffic light icons: green (met/exceeded standard), yellow (did not meet standard but is acceptable for now), red (did not meet standard). Upon completion of this analysis, we re-engage the CrowdFlower team to review the results and determine next steps in any areas that might need improvement.

Optimization

The CrowdFlower platform is setup in such a way that it can adapt to changing business requirements. This adaptability is especially important as you gain insight into the data that you are collecting. The platform offers optimizations such as geographic segmentation, runtime quality checks, and increased levels of throughput.

Here are some examples from projects that we ran:

Geographic Segmentation

• In one test, we realized that the quality of work completed by US based contributors was significantly higher than the rest of the contributors because of contextual knowledge. Therefore, we restricted the task to a US based workforce only.

Runtime Quality Checks

• In another test, we were able to identify a couple of runtime quality checks that can be performed on the task. Implementing these changes helped us improve the quality metric – the only lagging metric for this test.

Increased Throughput

• In a third test, the speed of the work done was slower than anticipated. Therefore, we opened up the work to more contributor channels for this task and the speed significantly increased.

When we implemented one or more of these optimization techniques, we ran a few tests to ensure that these changes gave us the right results. Once that was set, the next step was to move these projects to the final stage – Production.

This is part of a series of guest posts by Ram Rampalli, our crowdsourcing partner at eBay.
Part I – Assessment Stage
Part II – Pilot Stage
Part III – Analysis Stage
Part IV – Production Stage

About the author: Ram Rampalli created and leads the crowdsourcing program within the Selling & Catalogs team at eBay Inc. You can follow him on Twitter (@ramrampalli)

A quick aside before jumping into Part 2: This series lays out a methodology for compiling a successful portfolio of high-accuracy, deterministic crowdsourcing projects done through the CrowdFlower platform. It is not an absolute methodology for all possible crowdsourcing projects.

Building a successful portfolio of crowdsourcing projects – Part 2

In the first part of this series, we discussed the Assessment stage. If your proposed project passed the initial assessment, it graduates to the next stage: Pilot.

The first recommended step in the pilot stage is the pilot requirements specification. In my experience, some ideas that passed the assessment stage were later disqualified after further review with the CrowdFlower team. This prompted us to come up with the pre-pilot document, in which we ask the project sponsors to complete pilot set-up information that also serves as a second checkpoint:

• Name
  • Choose a simple and descriptive name.
• Objective
  • Define clear and concise objectives.
  • Example: Improve the recommendation engine algorithm accuracy from 65% to 90% by end of 2011.
• Description
  • Describe the task that you expect the worker to perform. If it’s difficult to describe in a few sentences, you may need to refine the task.
  • Often projects are rejected when the sponsor cannot provide a simple job description.
• Sample Data
  • Provide sample data that is representative of the source data for the ongoing project.
  • This process often results in project sponsors uncovering bottlenecks in the data generation process.
• Gold Units
  • The project sponsor should either present sample gold units (preferable) or have a plan in place for the gold units.
• Volume
  • Define how often the test is to be run and the volume for each run.
• Success Metrics
  • Define the metrics to evaluate the success of this project.
  • If this project is currently run through outsourcing, or other means, list the current performance metrics.

If the project satisfies the pilot specifications document, we work with the CrowdFlower team to design the pilot, with thorough testing by the CrowdFlower team prior to launch.

Many projects are launched in two stages.

Stage I: Controlled Launch

In the first stage, we do a controlled launch of a subset of all units. We monitor the task closely to make sure that it is on track to achieve the success metrics outlined previously. If necessary, we calibrate and optimize the task to improve performance.

Stage II: Full Launch

Once we are sure the project is running smoothly we then open up the task at full scale. Upon completion, CrowdFlower sends a copy of the results.

To save time, be sure to agree upon the structure of the results file, so that you are getting the data in its most usable format.

This sets you up for the next phase – Optimization and Analysis.