Comments on: Ask A Stupid Question Part 2: Forced Choice vs. Checkboxes

By: Tom

Tom — Sat, 06 Mar 2010 10:55:18 +0000

I have only recently discovered your blog, but am enjoying the exploration and insights; thank you!

I wanted to comment on the formatting… graphs exported from R are quite large, thousands of pixels on a side, scaled down to a few hundred using the IMG tag. While the compressed file size is only a few hundred kB, the size in RAM is approaching 100 MB. This slightly slows sown the page loads on my DSL connection, and, more significantly, prevents the images from loading when I’m viewing the blog on my iPhone. If you scaled the image down to a thousand pixels on the longest side, you’d reduce the RAM requirements to a few MB and make your blog fully viewable on the iPhone. Better still would be to scale the image to the desired display.

I look forward to your future posts.

By: Nick Diakopoulos

Nick Diakopoulos — Wed, 03 Mar 2010 16:50:13 +0000

So there could be some other explanations for the higher mean in the forced response condition. For instance, since the forced choice condition lists “yes” as the first option, there may be a primacy effect which would lead to a larger number of yeses being reported. Randomize the order of those choice for each response and re-run? I’ve also seen cheating instances (not infrequently) where Turkers will pick, say, the 1st response option for all questions. I also agree with Bob about the click effort difference between the conditions.

By: Bob Carpenter

Bob Carpenter — Mon, 01 Mar 2010 20:59:31 +0000

Have you considered the mechanical Turk effect of people spamming the results? Just think about how much work (in terms of clicks) are involved. They need to click once per question in the “forced choice” setup, whereas it’s extra work to click in the radio boxes. You’d expect exactly the discrepancy you saw from the spammers.

Try controlling for motivation (you can’t control for effort). For instance, offer no money for the task but a bonus for getting 5 or 10 easy true/false questions right and present them in both formats, a “check the true statements” format and a true/false format.

I also bet the answers in the “forced choice” section would be different if you reordered the responses. Or if you paraphrased them.