Radiology and Crowdsourcing

Recently, I’ve witnessed a rise in the number of crowdsourcing jobs targeted at accomplishing tasks for the medical industry. Every once in a while I’ll be fortunate enough to run across an exceptionally intriguing job that makes use of crowdsourcing in a really unique way.

Antonio Foncubierta’s crowdsourcing job is one of those unique tasks I happened to stumble upon. Antonio works for the Business Informatics Institute at the University of Applied Sciences Western Switzerland (HES-SO). The job he created gives the crowd the task of categorizing medical images, such as x-rays, PET scans and CAT scans. Here’s how Antonio describes the problem:

“The problem with medical images is that they are produced in vast quantities everyday – 30% of the world storage capacity is used by medical images. Retrieval and analysis are quite challenging. In order to train our models and computer-based systems, ground truth is necessary, but it also requires a lot of manual work and time to obtain. Therefore, we thought of using crowdsourcing as a way to obtain quickly basic ground truth.”    

The job first displays a link to a page, educating the contributor by clearly distinguishing between different types of medical images as an example. The crowd contributor will then view a number of images on the task, similar to the one below.

radiology

Example images from Antonio’s radiology job. Starting from the top left and moving clockwise, this page shows Magnetic Resonance, Ultrasound, 2D Radiography, and Computer Tomography.

So what are some of Antonio’s tips on conducting a successful crowdsourced job?

“I think that crowdsourcing is a huge opportunity for researchers, when repetitive tasks need to be performed. However, it is extremely important to have good methods for assessing the quality of the judgments in order to use all the potential of crowdsourcing.”

It’s a creative approach to a pressing problem. This job provides insight into the benefits that can be derived from crowdsourcing, while also demonstrating the close and often intertwined relationship technology and medicine share. With the exponential increase in medical data being generated, crowdsourcing is poised to be the next step in overcoming some of the most crucial obstacles in the medical domain.

You can read more about Antonio Foncubierta’s research findings at the links below:

  • Antonio Foncubierta -Rodríguez and Henning Müller, Ground truth generation in medical imaging: A crowdsourcing based iterative approach, in: Workshop on Crowdsourcing for Multimedia, ACM Multimedia, Nara, Japan, 2012
  • Antonio Foncubierta -Rodríguez and Henning Müller, Crowdsourcing opportunities in medical imaging (2013), in: IEEE Communication Society letter

 

Builder Pro Training Day New York – May 22

We’re excited to let you know that CrowdFlower training is coming to New York. We held one here in San Francisco in March to rave reviews and now we’re taking it on the road. Here’s what we’ll be covering:

  • CrowdFlower Builder Pro overview and architecture
  • A walkthrough of Builder Pro features
  • Creating jobs with Builder Pro
  • Technology directions and feature roadmap

This event is ideal for data scientists, search scientists, developers, researchers, sentiment analysts and others interested in learning how to use microtasking with or in their applications. The content is technical, and we will be writing a little bit of code (in the morning, when we’re all fresh).

The class will be held in midtown at SaGE Office Suites 276 5th Avenue next Wednesday, May 22. Space is limited, so please register here if you plan on attending. The course is free, and all attendees will get platform credits to put Builder Pro through its paces.

Ember.js at CrowdFlower

Every day, I love coming to work at CrowdFlower for having the opportunity to build UIs that are changing the way work gets done around the world, conceivably for the next twenty years. We’re just beginning to scratch the surface as we empower customers and contributors alike with great interfaces.

It’s an exciting time to be working with Ember.js to build interfaces like the following as depicted for Skills Tests, our Senti dashboard, and our Contributor dashboard:

composite

three apps built with Ember.js

Though Ember.js has seen some criticism recently, I’ve been a big fan of its approach to solving the challenges of modern web development for over a year, particularly as the framework saves me time by generating much of the code I would otherwise have to write. Developing with such a bleeding-edge technology hasn’t been all rainbows and ponies though. A major hurdle we’ve faced has been dealing with changes in the framework’s API. We currently have three applications in production using three different versions of the framework (and we’re about to roll out a fourth).

The volatility of the API has presented two issues in particular. First, it’s tough to codify a set of best practices for architecture and testing to apply uniformly across applications against a moving target. The second issue has been to bring other engineers up-to-speed on those (evolving) frontend practices – we’re all solid full-stackers but not as many of us spend as much time in the presentation layer as I do.

Our approach to the use of Ember.js in specific speaks to our engineering culture in general: to minimize exposure to risk, we vet bleeding-edge technologies where one or two people are able to become localized centers of excellence. Then, as the technologies mature, those who ran point are able to pay it forward by transferring knowledge across the team through pairing, kitchen conversations, wikis, and code lunches.

If you’d like to be a part of creating great user experiences as we change how the world gets works done, join us.

Anthony (@inkredabull) is the Sr. Web Engineer at CrowdFlower. He’s presented on Ember.js, maintains the Yeoman Ember.js generator, teaches the Pro Ember.js Class at Marakana, and thanks CrowdFlower for supporting his participation in the Ember.js and OSS communities.

Recap: Builder Pro Training Day

Builder Pro Training Day

Class is in Session! // Photo Credit: Paige Buck

We just hosted our very first Builder Pro Training Day at our office in San Francisco. For all on the east coast — not to worry — we’ll be hosting a day in New York for you soon!

Forward-looking Fortune 500 companies spanning various industries have been forming Crowdsourcing Centers of Excellence and internal Crowdsourcing Programs for years. As more businesses recognize the value of bringing crowdsourcing expertise in-house, we are ready to provide access to our market-leading microtasking platform: Builder Pro. It puts the tools of the CrowdFlower Admin at the fingertips of any company.

As the crowdsourcing industry matures, we’ve begun to cross the chasm. In his book Crossing the Chasm, Geoffrey A. Moore describes the chasm as the gap in technology adoption between innovative, early adopting customers and the subsequent group dubbed the early majority. Well, we’ve been listening to our early adopters intently and in response recently launched Builder Pro — a premium (read: totally souped-up) version of our self-service microtasking platform (Builder).

To soft launch launch Builder Pro, we invited 20 customers into our office and held an exclusive, full-day training event. While having a lot of fun, we also received instant product feedback. Ah yes, and all attendees received a 1-month free trial of Builder Pro and $99 of credit to test it out :) We brought in guest speakers, had roundtable discussions, and defined immediately valuable use cases of crowdsourcing.

Here’s a snapshot of what we covered:

  • How to deconstruct a business process and create tasks for the crowd to perform

  • Best practices in task monitoring, after the crowd has begun working on a project

  • A broad survey of successful crowdsourcing projects and interesting applications

  • Our product roadmap for Builder Pro and the exciting features that will be released

  • Advanced methods in managing the quality of results for high volume projects

Builder Pro Training Day was a resounding success, so we’re taking this show on the road! Our next stop will be New York. If you have feedback, questions or you’d like an invitation to our New York training day, reach out to me or leave a comment below!

@arielklein

 

Discovering Drug Side Effects with Crowdsourcing

As a recovering biology major, one of my favorite applications of crowdsourcing is solving public health problems. So far at CrowdFlower, we’ve enlisted the crowd to kill TB cells, count neurons in mouse cortices, and track epidemics. With some of the new tools we’ve built in the last year, I’d like to add tracking drug adverse events in social media to the list.

The Experiment:

We collected all tweets (on Twitter) that contained “Claritin” for the month of October 2012. After some basic filtering for spam tweets offering to sell Viagra with allergy medication, we were left with 4,900 tweets.

Next we looked to adverseevents.com for Claritin’s top 10 most common adverse events. The top 10 adverse events are from the FDA Adverse Event Reporting System (FAERS), which collects mandatory adverse event reports from drug manufacturers, and voluntary reports from medical professionals and consumers. Remember the Vioxx and fen-phen recalls? These drugs were withdrawn due to safety concerns as a result of Serious Adverse Events (SAEs) that were reported to the FDA through FAERS.

The Results:

We used our sentiment analysis product, Senti, to have our crowd review the 4,900 tweets and classify them for relevance, sentiment, author’s gender, and any of the top 10 most common adverse events as reported to the FDA. You can explore the data by clicking on the interactive graph below.

Interactive Senti Dashboard of one month of Claritin Adverse Events on Twitter
built with d3.js and crossfilter

We found 295 instances of adverse events in the top 10 categories on Twitter – a number higher than is reported to the FDA. In the last 12 months for which data is reported (Jul2012-Jun2012, data is only available through June 2012 at time of writing), there has been an average of 8 adverse events where Claritin was the primary suspect in reports to the FDA, and 265 total adverse events per month where Claritin was mentioned to the FDA in conjunction with other drugs. Almost all of the cases we found on Twitter were primarily due to Claritin, over 30X the number of primary events that are reported to the FDA.

Adverse Event Twitter FDA – PR* FDA – TR**
Dizziness 11 0 16
Convulsions 0 1 6
Heart Palpitations 5 0 7
Shortness of Breathe 4 0 19
Headaches 7 0 16
Drug Effect Decreased 66 0 3
Allergies Worse After Taking Drug 132 0 8
Bad Interaction Between Claritin And Another Drug 40 0 5
Nausea 4 0 19
Insomnia 26 0 9
Other 0 7 157
Total 295 8 265
Twitter adverse events are for October, 2012; FDA’s are rolling 12 month average
*Primary Reports **Total Reports

Over six percent of tweets mention an adverse event. The most common complaint is Claritin’s notorious failure rate. It’s not surprising that more serious adverse events (convulsions, heart palpitations, shortness of breathe) – are underrepresented in social media while comparatively minor effects (decreased drug efficacy, allergies worse, nausea) are overrepresented on Twitter relative to FDA reports.

Serious health problems are much more likely to be reported through traditional medical channels as opposed to social media. However, many lower impact effects – drug not working, allergies worse, bad interactions between Claritin and other drugs – may be deemed too minor to be reported through traditional medical channels and show up on Twitter in greater volume.

Percentages by Gender
Source Women Men
Total Tweets that contain “Claritin” 62% 48%
Tweets w/Adverse Events on Twitter 65% 35%
Adverse Events Reported to FDA 65% 35%

The two genders seem to tweet about Claritin in roughly equal proportion to stated adverse events for FDA – with women making up the majority of adverse events sufferers by almost a 2:1 margin. Women mention more serious conditions (heart palpitations, shortness of breathe, headaches) while men do not. Its unclear if this is because women are more likely to suffer from those conditions, are more likely to tweet about them, or that the sample size is too small to glean anything meaningful.

What It Means:

There is an order of magnitude more data on adverse events available on Twitter than the FDA receives – providing a much more expansive and inclusive dataset for adverse events. With the ability to find many more adverse events than are currently found, social media could provide an early warning detection system for postmarket surveillance of drugs, providing a safer environment for both consumers and pharmaceutical companies. There’s incredibly rich data out there, if you’re willing to look for it.