With a gold standard set of true examples, get a high #annos/example, and get per-item error rates (1-sensitivity) err_i. This is to make a high recall system, so “yes” decisions must be unanimous (single dissenter causes “no” decision), meaning that all annotators must make an error to get an aggregate error. Then for k annotations per example the expected number of errors is \sum_i (err_i)^k.

This can be viewed as a model where all workers have equal capability, but items have a difficulty parameter. It gives a more pessimistic error estimate than most naive model where all items have the same difficulty, whose expected errors is n * (err)^k. I believe the estimates are different because hard items don’t get solved too well by throwing more and more annotators at them.

I have exactly the same feeling about difficulty, but I haven’t been able to fit any of the logistic models with a latent difficulty predictor. The basic model is just p(anno[i,j]) = inverseLogit(accuracy[j] - difficulty[i]), just like the items response model. I’m going to send some mail to the epidemiologists fitting these models — some of the more recent paper discuss the instability of the model fitting. Even with the true category known, I’m having trouble fitting the models.

The problem seems to be that there’s two ways to account for variability in an annotation, either that it’s difficulty or that the annotators are error-prone. If I crank prior variance on difficulty down close to zero, it fits, but there’s not much of a difficulty effect. If I let variance even approach 1, different chains don’t mix well at all and I just can’t infer the difficulties reliably. I’ve tried this with both real and simulated data.

I did manage to fit the binary mixture of easy/hard items, but that seems less relevant with more and more annotators and especially with very noisy annotators.

Since we didn’t include the exact numbers for worker modelling/correction in the paper, to make this complete, here are exact numbers from both of our experiments so far (plus a new one I just did):

Accuracy rates at Turker ensembles matching the gold standard, RTE with 10 judgments/example:

89.7% - naïve voting
92.6% - hidden labels, inferred worker prior [MAP via Gibbs sampler] [link]
92.9% - hidden labels, uniform worker prior [MAP via Gibbs sampler] [link]
92.6% - known labels (LOO), add-1 worker prior [MAP via direct inference]
92.9% - known labels (LOO), non-bayesian: drop workers with <67% accuracy then naïve vote the rest

Yes, that last one is way less principled than the others, and wasn’t in the paper, though probably should have been. (My fault; I’ll blame a deadline rush I suppose.)

Anyway, I think it’s impressive that the hidden label model does exactly as well as any system using known labels.

I’d be interested to see how things do for smaller numbers of annotators per example, since there’s more headroom down there -- e.g. at 3 anno/example, naïve accuracy is only 80.5%.

I’m also wondering if a model with per-item difficulty will start becoming useful. When I do RTE myself, I feel there’s large variance in difficulty. Of course this highlights why the BUGS approach makes progress much easier...

[[
More on these new experiments. (1) The thresholding technique: In practice it will work a little less well, because I iterated the threshold parameter and took the best one, and the space was a little bumpy. Though no lower than 92% for all reasonable thresholds; what’s key is eliminating the several noisy+prolific workers. In fact, the 67% threshold drops nearly half of all judgments (!). Though in practice with the AMT feedback cycle you wouldn’t pay for all those bad judgments, just pretest and eliminate bad workers early on, and pay good workers for more work.

(2) Unlike the paper, I used here leave-one-out instead of 20-fold cross-validation. (I have a faster implementation now; I think I'm starting to hit the pain points of R so went back to python ... http://github.com/brendano/dlanalysis/tree/master/workers.py )
]]