One of the most exciting things about working at CrowdFlower is the ongoing discovery of the wide range of crowdsourcing applications. While our core focus is enterprise solutions, we’re also involved in a number of social innovation projects. At recent meetups and in recent blog posts, we’ve described CrowdFlower implementations that help create unprecedented social impact. Many of these projects involve processing data in support of crisis relief or public health research.[1] What we continue to see over time is that there are truly inspirational ripple effects emerging from these efforts.

Cell counts | Flickr - Photo Sharing!

Lab scientists spend countless hours manually reviewing and annotating cells. What if we could give these hours back, and replace the tedious parts of science with a hands-off, fast, cheap, and scalable solution?

That’s exactly what we did when we used the crowd to count neurons, an activity that computer vision can’t yet solve. Building on the work we recently did with the Harvard Tuberculosis lab, we were able to take untrained people all over the world (people who might never have learned that DNA Helicase unzips genes…), turn them into image analysts with our task design and quality control, and get results comparable to those provided by trained lab workers.

Here’s how:

We took cortex slide images from mice provided by a neuroscience lab at Harvard University. We cut each image into smaller pieces, so they’d be easier for people to work on.

After a brief set of instructions, contributors were instructed to count the neurons in the slide by clicking on each individual neuron.

Fig 1: example slide

Contributors were given examples of edge cases (i.e. Is this a cell or not? Is this two or three cells?). Contributors identified and clicked on every cell, which placed a green marker on top of each cell. An automated counter kept track of the number of clicks.

We controlled quality using Gold Standard (“Gold”) units, which were images with known cell counts that we added to the task. The benefits here are threefold. First, Gold questions provide training and feedback to our contributors, so that they can get better at the task over time. Second, contributors don’t know which questions are Gold, forcing them to honestly answer all questions. Finally, if a contributor fails to answer enough Gold correctly, we remove them from the job.

Fig 2: Map of all contributors on this task. Step it up Iceland!

Visit the batchgeo.com navigable map.

After removing all of our “untrusted” contributors, we are left with our “trusted” contributors. Each image had four trusted contributors count the neurons. We took the average count less any outliers in order to get the most accurate results.

How did this experiment actually turn out? How did our results compare to those of trained lab workers? In short, we performed extremely well. As you can see in our results below, the average difference between the Crowd and professional lab counts is 2.0%, which can be chalked up to ambiguity in certain clusters of cells.

Future iterations of this will include breaking the images into smaller pieces, as well as using automated solutions to tag the high confidence units, leaving only the edge case cells that require human eyes.

We hope that results like these will encourage more scientists, labs, and biotech firms to crowdsource pieces of their research. We believe this could free up their time for more complicated work, decrease the latency of results for experiments, and quicken the pace of scientific discovery.

Yesterday it was TB Cells, today its neuron cells; tomorrow: cancer cells? We’re issuing an open call for any university lab, biotechnology, or pharmaceutical company with large image data sets: contact us to utilize the crowd to shorten the life cycle from idea to major scientific advancement.

via: FastCompany

About the author: Philip Rosedale is the creator of Second Life and a Co-Founder of LoveMachine, Inc.

My co-founder Ryan and I are having so much fun pulling together data and thoughts for my upcoming keynote at CrowdConf next week. It’s a great opportunity to try and summarize much of what we’ve learned over the last few years about whether and how crowdsourcing can be taken to the next logical (we think) level: to replace a bunch of what we’ve come to think of as the nature of “work” and “company”.

The Silicon Valley startup formula is now a well-recognized and time-honored strategy, which I think we’ve all worn into a bit of a rut: 3 or 4 very smart people (usually guys) hunker down in someone’s garage, work a bleary-eyed 80 hours a week producing a prototype, getting funding, hiring those first handful of key engineers, etc.

But what would happen if you never hired anyone at all? What about doing a ‘real’ Silicon Valley startup – meaning a new, risky, design-sensitive idea backed by venture capital – but without the garage and the crunch time and team of ‘A players’? What if you built a system that instead allowed people all over the world to be paid small amounts of money to help you prototype, build, and then release a high quality new website and software?

Well that’s what we did, and there is a lot to talk about!

-Philip

Come hear Philip and many other experts in the crowdsourcing field speak at CrowdConf 2011. It is a great opportunity to discover what is happening in the world of crowdsourcing. It starts next week, November 1-2 at the Mission Bay Conference Center in San Francisco. You can buy your tickets here. Hope to see you there!

Millions of people have participated in our tasks over the last few years, and tens of thousands of people are active at any given moment. However, crowdsourcing is not a traditional engagement model. Tasks are elective, which means people are free to come and go as they please. It’s a fair question, then, to ask whether they keep coming back.

Do people perform tasks only fleetingly, or has crowdsourcing become more of a long-term engagement? Furthermore, just how important is contributor retention in the world of crowdsourcing?

While a majority of people fall into the “one-and-done” camp, many of the most productive contributors tend to have participated in previous jobs. Within any single job, these seasoned veteran contributors also provide far more work than their less experienced counterparts.

Over a period of two months, we ran a series of five very similar jobs, retrieving ratings information for businesses throughout North America. In total, we collected over half a million judgments from a total of 2,901 unique contributors,¹ representing multiple labor channels and 101 countries. As a first test to whether seasoned veterans are common, we looked at how many people participated in more than one job. In total, 2,389 people participated in one job only, meaning that First-Timers accounted for over 82 percent of all contributors.

But this may not be the best number to look at. We’re really interested in whether certain people recognize and seek out specific types of tasks after having worked on them before. To look for this behavior, we analyzed the most recent job, counting how many people participated in at least one prior iteration of the task.

For a recent job, a total of 906 people participated, 247 of whom had done the task previously. By this measure, Seasoned Vets constitute approximately 27% of the workforce. While the impact of returning contributors is greater under this methodology, the fact remains that these “loyal” contributors are firmly in the minority on this series of jobs.

Count of Contributors by Profile

However, as we’ve seen before, the individual impact of contributors varies widely, with a minority of people often providing the vast majority of work. With this in mind, we looked at the contributions of First-Timers and Seasoned Vets and found some striking differences.

Share of Contributions by Profile

Seasoned Vets, while constituting only 27% of the workforce, provided 47% of the total work completed. On average, each Seasoned Veteran provided 2.5 times more judgments than their less experienced counterparts. It’s also interesting to note that there was no significant difference between the quality of work provided by First-Timers and Seasoned Vets, no doubt due to our suite of quality control measures.

Given that the people who stick around tend to be far more productive, improving retention is a useful consideration (for these jobs, at least). We’re now interested in how best to attract people to return to the types of jobs they’ve already seen. That, however, is a work in progress and a story for another day.

1. Note that this analysis only considers Trusted workers.

Our recent post about confidence bias, where we showed that most contributors vastly overestimate their own ability to complete tasks correctly, raised a lot of questions about how we manage quality at CrowdFlower. You might remember these themes from such classic posts as: AMT is Fast, Cheap and Good or the Wisdom of Small Crowds series [1] [2] [3].

via: reddead.wikia.com/

The standard CrowdFlower model is agnostic towards the quality of any individual contributor. Typically, we let anyone attempt a task, using our technology to filter out low-quality contributors and score the responses. Without further ado, what follows is quick review of the steps we take to do that filtering.

Gold (What is it Good For?)

In almost every job, we take a subset of the data to be processed and manually score the correct response. This manually-scored set, which we refer to as Gold Standard Data, is at the core of managing quality in the context of enterprise crowdsourcing:

• Filtering: We use Gold to create an up-front test, creating a barrier to entry such that only workers who understand and successfully complete a task are allowed to participate. This allows us to prevent unsavory characters from entering jobs and contaminating results.
• On-going Training: We also use Gold to conduct on-going training, offering corrections for units that are answered incorrectly. This allows us to continually instruct and improve highly prolific contributors.
• Dynamic Trust Score: We use each contributor’s performance on Gold as a basis to determine their overall accuracy within a task. Each contributor must exceed our minimum trust thresholds to continue working on a task. If at any point a contributor falls below the trust threshold, we’ll exclude his work.

Because creating Gold is labor-intensive, we’ve created an automated process to generate Gold using units that have already been completed. This has significantly reduced the time needed for setup and ongoing job creation, without sacrificing our ability to differentiate contributors.

Of course, the amount and distribution of Gold is critical. Often, a uniform distribution of Gold across response types is ideal, though in certain situations we’ll use a skewed Gold set. For example, in an experiment on crowdsourced document review, we used a skewed Gold set to avoid missing relevant documents (reduced “false negatives,” if you prefer).

Department of Judgment Redundancy Department

If the purpose of Gold is to manage the quality of individual contributors, we use multiple judgments per unit to improve the accuracy of completed units. The basic premise is simple enough. We look for agreement among trusted workers to indicate correct responses at the unit level. For example, if we ask four people to verify a phone number for a business, the answer is more likely to be correct if all four agree. In fact, every unit processed by CrowdFlower is annotated with a response as well as a Confidence Score (based on agreement weighted by Trust, plus some secret sauce).

More generally, assume we set the trust threshold for a given job at 70 percent (meaning that anyone who doesn’t answer at least 70 percent of Gold correctly gets booted) and that contributors are uniformly distributed in terms of ability (not true, but convenient). We can easily model the effect of additional judgments on estimated accuracy, showing that the probability that the majority response is correct increases with the number of judgments collected:

Of course, while collecting 10 judgments per unit yields highly accurate results, it may not be the most efficient way to structure a job. Imagine that the first 2, 4 or even 6 contributors agree on how a unit should be classified. At some point, the marginal impact of an additional judgment is not worth the additional cost. We’ve automated a process to vary the number of judgment each unit receives based on agreement thresholds, so that we can reach accuracy targets more efficiently.

The following shows actual results from a sample job, where we set a minimum confidence threshold of 0.7:

Approximately 50 percent of units completed with just 2 judgments and 75 percent completed with 4 or fewer. In any case, each unit received only as many judgments as necessary to reach the confidence threshold. For any job, some subset of units will be ambiguous enough that they won’t reach a confidence threshold, so we use also set maximum judgments cap to “stop the bleeding.” Depending on the specific circumstances, we may reroute those ambiguous units to a parallel process with different structure, contributors, etc. for another round of judgments.

One More Trick

For complex tasks, we have developed a workflow management system to link together multiple jobs. For example, we might ask one pool of contributors to write a product description, verify the spelling and accuracy with a second pool and rank the subjective quality with a third pool. Alternatively, we might take a business listing and break out each attribute for independent collection and verification, with a separate job for name, address and phone number, or cuisine type, cash-only, types of credit cards accepted, on-site parking, or any other attribute that can be verified online. In general, peer review means that we can always give data a second pass to improve accuracy.

Expanse of Cheese: Caseificio di San Silvestro's Aging Room in Castelvetro (MO)

Last May, I took a trip to Italy for two weeks. A little bit of history: my friend Jessica and I are both Italophiles, and when her mom sent us a link to a video contest where the prize was a round trip flight to Italy, we knew we had to enter. After a week of writing and editing lyrics in a Google Doc — half in Italian, half in English — the resulting music video ended up winning us a trip to the holy land of olive oil, vino, and other delectable edibles.

Apart from being a passionate eater, I’m a passionate supporter of the Slow Food movement, an organization which promotes good, clean, and fair food around the world. Each year, they publish a guidebook to restaurants in Italy that adhere to their principles. In Italy, this usually means each restaurant is handpicked to showcase the traditional food of a particular region; each restaurant supports artisanal methods and products that otherwise might go extinct (were eaters not eating them), and where the food is most likely naturally organic and local anyway.

But, a problem: it’s 2011, and I’m more apt to travel through interactive maps than with old-fashioned guidebooks. More importantly, I don’t plan, and I needed to know what edible options were around me at any moment in time on my trip. What I really needed was a version of the guidebook, in map form, that I could use on my mobile. No such option existed, of course, and I was left with two options: magically visualize restaurants around me by poring through the guidebook, or, crowdsource it.

What I needed to do (and, what much of our work at CrowdFlower comprises), was structure unstructured data. To create the map from the book, I went through the following steps:

• Obtained a PDF version of the book
• Split the book into pages, and uploaded each page as a separate PDF (thanks to pdftk)
• Created a CSV (comma separated values file) with each page’s PDF link and page number
• Created a crowdsourcing task to structure the data, using the previously uploaded individual PDF pages
• Geocoded the structured data
• Output the geocoded data in KML (Keyhole Markup Language) form
• Uploaded the KML file to a mapping site (e.g. Google Maps)

Conveniently, each page that outlined a restaurant in the PDF was formatted nearly the same. This made it easy to give instructions to workers, as seen below:

Instructions

For each area on the page, workers were asked to copy and paste specific sections into the task. Each page, then, was split up into corresponding parts (Region, City, Directions, Restaurant Name, and the two Capture Areas). This is the essential concept here: structuring the unstructured data such that I could later geocode it properly, and display it in the way that I needed.

The tonnarelli cacio e pepe at Da Felice a Testaccio in Rome (used in the Geocode example)

Once the task finished, I downloaded the resulting CSV file, and whipped out Google Refine (a.k.a. Excel on crack), which has a feature that allows you to enter a template API call that changes based on specific values in each row. Using the Google Geocoding API (any will do), I constructed the following API call, using the address value in each row as the “address” parameter for each API call:

http://maps.googleapis.com/maps/api/geocode/json?address=via+Mastro+Giorgio%2C+26+Roma+Lazio&sensor=false&region=it&language=it

After slicing and dicing the rest of the data into bits that I wanted to display on a map, I used Google Refine’s “templating” feature to export each row as a Placemark in KML format. Finally, I uploaded the resulting KML file into several different maps, each representing one region in Italy.

A portion of the Campania map - tons of places to explore around here!

Try it out! Check out the map for Tuscany, and the map for Campania.

If you want to give crowdsourcing a spin, head on over to our self-service product and click “sign up”. Or, if you’d prefer a hands-off “we-do-everything” approach, contact us to get started.

Watch the video that started it all: Io Sono Balsamico by Balsamico

Aron is a Crowdsourcing Project Manager at CrowdFlower, and is the resident agriculturalist-eater. Follow him on Twitter (@aron) for more sage bits of agricultural-eating learnings.

psychologytoday.com

Evidence in experimental psychology suggests that most people overestimate their own ability to complete objective tasks accurately. This phenomenon, often called confidence bias, refers to “a systematic error of judgment made by individuals when they assess the correctness of their responses to questions related to intellectual or perceptual problems.” ¹ But does this hold up in crowdsourcing?

We ran an experiment to test for a persistent difference between people’s perceptions of their own accuracy and their actual objective accuracy. We used a set of standardized questions, focusing on the Verbal and Math sections of a common standardized test. For the 829 individuals who answered more than 10 of these questions, we asked for the correct answer as well as an indication of how confident they were of the answer they supplied.

We didn’t use any Gold in this experiment. Instead, we incentivized performance by rewarding those finishing in the top 10%, based on objective accuracy.

Does Bias Exist?

To estimate confidence bias, we looked at the difference between the average of how confident an individual was of his/her answers and how many he/she answered correctly. If the difference is positive, the individual overestimated how well they did. Amazingly, over 75% of contributors overestimated their ability to answer multiple choice questions correctly.

Are Individuals Consistently Biased?

Because our dataset consisted of Math and Verbal questions, we looked at each individual contributor’s confidence bias for both types of questions. In aggregate, people tended to have more trouble with the Verbal questions (average accuracy of 28%, compared to 41% for Math), though the average confidence score was nearly identical (63% +/-1).

The vast majority of contributors fall into the “overconfident on both” quadrant (top right), while only a handful of contributors were overconfident for one question type and underconfident for the other (top left and bottom right quadrants). Overall, there is certainly a correlation between bias scores on the two problem types, suggesting that many individuals are consistently biased on different types of problems. However, this explains only a portion of the variation.

Does Bias Vary Across Groups?

Given that overconfidence seems to be a consistent trait, we were curious how this trait varies across the different groups making up our contributor pool. We sliced and diced our contributors into a number of different sub-groups, which are summarized below.

There are a lot of interesting things going on here. To highlight a few, accuracy increases consistently as the contributor’s education level advances from High School to College, but so does confidence, leaving the bias score nearly unchanged. There’s a similar pattern with Age, with older contributors tending to be both more accurate and more confident.

Gender and Location also have an effect on confidence bias. Taking the two countries that supplied the most people, contributors from the US were much more accurate and slightly more confident than the average, while those from India were average in terms of accuracy but much more confident. As such, the bias score for contributors from India is nearly double that of contributors from the US. With respect to gender, confidence didn’t vary much, but women were more accurate and thus less biased than men. Moving on.

Further Research

In the context of experimentation, we decided against using Gold to minimize any selection bias among contributors. However, this makes it difficult to apply these results to enterprise crowdsourcing, at least as practiced by CrowdFlower. In the future, it would be interesting to look at confidence bias among trusted workers only, and particularly among trusted workers with repeated experience in specific job types. We would expect these workers to have a better sense of whether their answers are correct, though it is possible (and perhaps likely) that confidence would increase along with accuracy.

1. Pallier, G., Wilkinson, R., Danthir, V., Kleitman, S., Knezevic, G., Stankov, L., & Roberts, R. D. (2002). The role of individual differences in the accuracy of confidence judgments. Journal of General Psychology, 129,257–299

]]> http://blog.crowdflower.com/2011/09/confidence-bias-evidence-from-crowdsourcing/feed/ 0 http://blog.crowdflower.com/2011/07/from-sea-to-shining-sea/ http://blog.crowdflower.com/2011/07/from-sea-to-shining-sea/#comments Fri, 08 Jul 2011 23:59:38 +0000 Patrick Philips http://blog.crowdflower.com/?p=2698

With Independence Day just behind us, I thought it might make sense to reflect on the people who do CrowdFlower tasks throughout the Union.

Sure enough, we do have people doing CrowdFlower tasks in all 50 states. California, Texas, New York, and Florida are well represented among people working on our tasks, which isn’t especially surprising given that these are the most populous states. A more interesting question would be how the states stack up relative to their populations:

There are some cold weather states, like Vermont and South Dakota, that are punching above their weight class in terms of CrowdFlower judgments submitted, although Maine and North Dakota are lagging behind so that doesn’t really work. Connecticut and New Jersey, two of the wealthier states in terms of average income, are underperforming relative to their population, but Massachusetts and New York are doing just fine. Mississippi and West Virginia, coming from the opposite end of the spectrum in terms of average income, show up strongly, but Louisiana and Montana don’t.

Any thoughts about why some states are more represented? Leave ‘em in the comments.

In a recent paper, presented at the ACM Conference on Computer Supported Cooperative Work (CSCW), John Horton, Daniel Chen and I used a large-scale experiment to test the effect of different incentive schemes on the quality of crowdsourcing work.

The results surprised us. They suggest that workers perform most accurately when the task design credibly links payoffs to a worker’s ability to think about the answers that their peers are likely to provide.

a horserace experiment! (photo cc-by-sa by iyoupapa)

The idea for this study came out of our sense that, as social scientists, we had something unique to offer the existing research on human computation. Early and influential crowdsourcing research has focused on how to filter the judgments of the crowd to find the best answers. We wanted to know whether simple task-design changes could improve the quality of data coming into a crowdsourcing system in the first place.

To test this idea, we chose 14 different incentive schemes and framing techniques developed and validated across the social sciences and set up a horse race experiment to see which schemes/techniques would work best.

Consistent with our personal biases (John and Daniel are both economists, and I’m a sociologist), some of the schemes were financially oriented, some were social or psychological, and some were hybrids combining social and financial incentives. The details of all the schemes are included in the paper (it’s a long list, and some of them are kind of involved), but it’s worth giving some examples.

On the financial end of the incentives spectrum, we had one condition we called “reward-accuracy,” which was pretty much what you’d expect: we told workers, “we’ll pay you a bonus if you get the answers right.” We also had one called “punishment-accuracy,” the gist of which you can deduce. On the purely social-psychological side, we had one we called “trust,” in which we told workers, “we’ll pay you for this job no matter how bad your performance, we trust that you’ll still make your best effort.”

One of the weirdest schemes turns out to be important, so I need to explain that one. Called “Bayesian Truth Serum” (BTS), it incorporates a design from the work of Drazen Prelec, a behavioral economist at MIT, who realized that research subjects could probably provide useful information regarding the expected distribution for subjective, qualitative questions (nb, the mechanics of how he does this are arcane in a way that is almost sure to delight the geeks among you, so I encourage you to read his paper). Few of the details of real BTS are important, except that we incorporated the piece about asking workers to answer the questions themselves and predict the distribution of other workers’ responses. We also told them we’d give them a bonus if their predictions were correct.

We then created a task that asked workers to answer five questions. In this case, the questions were drawn from another study examining participatory features of websites, for which we already possessed validated data collected by research assistants.

All workers answered the same five questions about the same website (www.kiva.org) while being exposed to one and only one of the 14 incentive schemes (or a control condition of no scheme). Roughly 2,000 individuals participated in the study, resulting in over 100 subjects in each of the experimental conditions. (The statistics and science nerds out there will be pleased to know that both the drop-out rate and demographic covariates were distributed evenly across conditions.)

To measure worker performance, we used the research assistant responses as correct answers to the questions and then calculated the total number of matching answers (out of five) provided by each worker. The results (aggregated across all treatments) are plotted in a histogram below and show that the average worker answered just over two questions out of five correctly.

Then, in order to see how the treatments compared against each other relative to the control group, we calculated the mean correct response rate for each condition and conducted difference of means tests to see which of these means were significantly greater than the control group. The results of this comparison appear below (in a new plot that doesn’t even appear in the paper!):

The orange dots show the value of the mean in each condition, and the blue bars illustrate the 95% confidence interval around that mean. The treatments are sorted by the size of the difference in means from the control. (More hard-core nerd stuff: the means are adjusted using Intent-To-Treat estimators).

From these results, we concluded that our horse race had two clear front-runners: the “Bayesian Truth Serum” (BTS) and “Punishment – disagreement” conditions, each of which improved average worker performance by almost half of a correct answer above the 2.08 correct answers in the control group. A few of the other financial and hybrid incentives had fairly large point estimates, but were not significantly different from control once we adjusted the test statistics and corresponding p-values to account for the fact that we were making so many comparisons at once (apologies if this doesn’t make sense — it’s yet another precautionary measure to avoid upsetting the stats nerds among you). In a tough turn for the sociologists and psychologists, none of the purely social/psychological treatments had any signficant effects at all.

Why do BTS and punishing workers for disagreement succeed in improving performance significantly where so many of the other incentive schemes failed? The answer hinges on the fact that both conditions tied workers’ payoffs to their ability to think about their peers’ likely responses. (We elaborate on the argument in more detail in the paper.)

Does this mean that we should give up on simple financial or social-psychological incentives? Probably not. The fact that we conducted the experiment on MTurk means that the deck may have been stacked against incentives like the “trust” condition I described earlier. Because requesters on MTurk have little oversight, workers are more likely to respond to financial incentives than stated promises. In this sense, the marketplace has structured the interaction between workers and requesters in a way that may limit the opportunities to harness motivations that are not linked to money in some explicit way.

You can download the full paper to read more.

We’re proud to announce CrowdFlower’s participation in one of the most prestigious academic conferences worldwide.

The 20th Annual Text REtrieval Conference (TREC 2011), organized by the National Institute of Standards (NIST), will take place Nov 15-18, 2011, at NIST’s campus in Gaithersburg, MD. TREC 2011 will feature a new Crowdsourcing Track that will investigate:

• How to obtain high-quality relevance judgments from individual crowd workers;
• How to effectively compute consensus judgments from individual judgments;
• Interaction between these (i.e., worker accuracy vs. subsequent consensus accuracy achieved).

Teams can choose to participate in either or both tasks: (1) collecting judgments and (2) computing consensus. We plan to evaluate using both ranking and classification metrics, giving teams the flexibility to focus on either one or both.

We welcome participation from those who have never before participated in TREC; note there is an agreement form to submit in advance; the form is available from the track website.

View the guidelines here.

Tentative Schedule

May 6, 2011: Initial deadline for participating teams to email organizers with team name and
intent to participate (will affect the partition of data distributed to all participants)

May 13: Finalized guidelines released, Stage 1 training sets distributed to participants

May 20: Stage 2 training set distributed to participants

August 1: Release of Stage 1 test data; submission system opens immediately

August 8: Stage 1 submissions due

August 15: Stage 2 test topics distributed

August 22: Stage 2 submissions due

~September 15: Preliminary results announced to participants

~October 15: Team papers due to NIST for inclusion in TREC conference
notebook

November 15-18: Text REtrieval Conference (TREC 2011) in Gaithersburg, MD

~February 1, 2012: Final team papers due to NIST for inclusion in TREC conference proceedings

For more information, please contact Track Organizers Gabriella Kazai (Microsoft Research) or Matthew Lease (University of Texas at Austin).

Photo of crowd, courtesy of PictFactory

Join us at the Stanford Institute of Design on Tuesday, May 3, for a night of networking, interactive presentations, and thought-provoking conversations.

The following speakers will discuss developments in modern crowdsourcing work, how this impacts the world of design, and demonstrate their latest findings:

Speakers:

Ed Chi, Research Scientist, Google

Gagan Palrecha, CEO and Co-Founder, Chirply

Steven Dow, Postdoctoral Scholar, Stanford

Mira Dontcheva, Research Scientist, Adobe

Bjoern Hartmann, Professor, UC Berkeley

Matt Mireles, CEO, SpeakerText

Where:
Stanford d.school
Building 550
416, Escaondido Mall
Stanford, CA

There are no valet restrictions after 4pm, so parking will be readily available. The nearest parking structure is at the corner of Via Ortega and Panama Street.

When:
May from 6:00 — 8:30 p.m. Speakers start at 6:30 p.m.

Please RSVP to guarantee your space.

Interested in hosting the next meetup? Have suggestions for speakers? Please email me.

After hearing that our friends at Citizen Effect were sponsoring Brackets with Benefits, a cool new March Madness pool where the proceeds from all the gambling go to one of the many awesome community development projects they support, we thought it would be fun to get involved. Considering how most people** watched their brackets go up in flames by the time the Sweet Sixteen rolled around, we thought it would be fun to give everyone a second chance to play.

We teamed up with Citizen Effect to invite all Brackets with Benefits participants, as well as the general public, to pick the results of the Sweet Sixteen and Final Four, a total of 10 games. We had one clear winner, who correctly predicted the outcome of 8 games (and missed Duke/Arizona and Florida/BYU, for those keeping track at home) and will be going home with a fancy new camcorder. For what it’s worth, the aggregated response of the crowd correctly picked 7 games.

In a tournament full of upsets, there were two games that surprised our participants most. Not surprisingly, these were the games where the two #1 seeds (Duke and Ohio State) lost in the Sweet Sixteen, which just under 10 percent of people picked. There was one person from Glenolden, PA (perhaps an embittered Villanova fan?) who picked both Duke and Ohio State to lose, but ended up picking only three other games correctly.

All in all, a fun little experiment. We want to send a big thanks to our friends over at Citizen Effect for putting together the pool and inviting us to participate.

Couldn’t make the March Meetup at CrowdFlower? Check out the video highlights below.

If you like what you see above, join our Crowdsourcing Work Meetup Group for news on upcoming Meetups.

Photo of crowd, courtesy of PictFactory

It’s meetup time again here at CrowdFlower. We will have a great lineup of speakers here this Wednesday, March 23. Come by to see:

Wyatt Nordstrom, CEO at Maven Research

Jacob Colker, CEO and Co-Founder at Sparked.com

Ray Solnik, Customer Development In Action

Nathan Waterhouse, Co-Founder at OpenIDEO

Omar Alonso, Technical Lead at Bing, Microsoft

Brian McConnell, Co-Founder at Der Mundo

Where:
CrowdFlower
2111 Mission Street, Suite 302
San Francisco, CA 94110

The office is one block South of the 16th Street Mission BART Station.
For a train schedule, please see: http://www.bart.gov/schedules/bystation.aspx

When:
March 23 from 5:30–8:30 p.m.
Speakers start at 6:30 p.m.

Please RSVP to guarantee your space.

Hope you can make it,

Mollie Allick
CrowdFlower

The votes are in from our Oscar crowdsourcing experiment, and the crowd successfully picked the winners of 14 of the academy awards. For reference, Roger Ebert got 15 predictions correct so we’d have to conclude that the crowd performed reasonably well at predicting the winners of this glorified popularity contest.

Predicted and Actual Winners of the 2011 Academy Awards

One fascinating thing about aggregating responses is that the crowd as a whole will often outperform the average worker. In this case, among the 500 people we polled, the majority of respondents picked fewer than 10 awards correctly (mean of 9.6 and median of 9). And yet, by aggregating all the responses, such that the nominee with the most “votes” is predicted to win, the crowd as a whole correctly picked 14 awards. While the “wisdom of crowds” doesn’t come as much of a surprise, it’s always reassuring to see it confirmed in new applications.

As we noted in our earlier post, though, the more interesting question was whether workers who indicated higher confidence in their responses would outperform workers with lower confidence. Looking at the results, however, we saw no significant correlation between a worker’s predicted accuracy and  actual performance.

"Squint all you want, but there's no pattern"

While it’s certainly possible that we didn’t offer enough of an incentive for workers to estimate their own accuracy, the more likely explanation is that predicting the winners of the Oscars is not something that a person can do with any degree of certainty. Confident or not, the people we polled did not see the “Inside Job” coming.

As a final exercise, we ran a regression on every explanatory variable we could find, including what state workers came from, what day they made their predictions, whether they made their predictions during the day or at night, how long they spent making their predictions and even their historical accuracy on other CrowdFlower tasks. The only variable with any significance turned out to be how long they spent on making their predictions, and while it was significant (at p=0.001), no model we could come up with explained more than 5 percent of the total variation in accuracy.

While the wisdom of crowds seems to extend to picking Oscar winners, the more interesting experiment of having workers self-select as trustworthy is ongoing. In the future, it would be worthwhile to repeat this experiment with questions that can be answered objectively and without uncertainty (solving algebra problems seems like a good candidate), to see if any correlation emerges between predicted and actual accuracy.