I suspect these measurements are be biased, though I can’t figure out in what direction. In the real world, the number of porn images is nowhere near 40%. I don’t know what the actual number is, but I would expect less than 0.01% (you guys probably have a better idea about this than I do). Porn images are rare. I can see two potential consequences of this.

Scenario 1: workers almost always click “not porn”, and fatigue sets in. They stop paying attention to the task and their accuracy goes down.

Scenario 2: because porn is so rare and so different from the other images, it really stands out. Workers do not have to think much or pay much attention to get a very good accuracy rate.

Do you guys have any guesses as to which of these happens? I would be very interested in seeing how the accuracy changes as you vary the proportion of porn images in the training set.

My second question is: why is this task so hard? Are there really so many borderline cases that a 5% false negative is reasonable? Maybe you need Justice Potter Stewart as an annotator.

My third question is that if the task is as hard as all that, what’s your agreement among the people doing the “gold standard” annotations? Or do you just annotate easy cases in your gold standard? That’d probably work just as well for adjusting for annotator accuracy in predictions.

1) It’s not trivial to implement an alogirthm using the turk API where different images are shown with different frequencies. It’s also not trivial to hide in gold standard data that immediately calls out worker performance.

2) Finding an optimal weighting algorithms and worker quality calculation is not trivial.

3) We have historical data on every worker that has done this task, so we know which workers are higher quality and lower quality and we know who is scamming us.

The result is more than double the efficiency of aggregating turk results without these benefits. So not only do you NOT have to deal with the turk API, and build your own reporting tools, but your images will get the same quality of moderation in half the time.

My goal with this post was less to brag about the quality of our system and more because I thought our intern John had done a great job of systematically measuring our quality and comparing to a baseline. I thought it would be interesting to show blog readers how we do our quality measurements and the way think about the quality problem with multiple annotators.