However, what I felt was missing from the field was a discussion of why, precisely, we can trust results from online experiments. This was the motivation for a new paper, jointly written with Dave Rand (who wrote up part of this study here on the Dolores Labs blog) and Richard Zeckhauser.

You can download the paper here.

While we make the practical and theoretical case for online experimentation, we believe that acceptance of online results as “valid” will come after people start seeing how easy and reliably one can replicate previous studies. This is why blogs like Experimental Turk and Deneme—both of which report results from AMT experiments—are so helpful. In our paper, we continue this process by replicating three results that are fairly well established.

In one experiment for the economists, we show—contra the usual intuition—that at least some Turkers are financially motivated, despite the very low stakes. After performing an initial text transcription task, workers were offered some randomly chosen amount of money to do an additional transcription. Results show the counts of people who agreed (“Yes”) and the counts of people who did not agree (“No”), by amount offered.

Nothing too surprising—offer to pay more and more workers will accept—but at this stage in the development of online experiments as a methodology, “surprising” would probably be bad news.

Anyway, the full paper is here. We’d love to get comments and feedback—it’s not too late to earn a place in our coveted “thanks” footnote!

To investigate this question of motivation, NYU Prof. Panos Ipeirotis asked a bunch of workers their reasons and tabulated the responses here. His two posts are actually on the syllabus for a course at Stanford (incidentally the course is taught by one of the creators of Protovis, which is very cool and is on my list of things to learn). There is also this amusing investigation.

For a joint project with Dave Rand and Richard Zeckhauser, we asked ~ 400 AMT workers both (a) where they are from and (b) the primary reason they participate on AMT. Because economic opportunities differ by country, we might expect that motivation and behavior should also differ by country. The cross tabulation plot is below (reasons are in the “rows”, countries in the “columns”–the size of each rectangle is proportional to the number of responses in that cell):

Two things to note:
1) Money is a big motivation for everyone
2) Money aside, people from India are there to learn; people from the US are there to have fun

Although the India/US differences are consistent with the different-countries/different-motivations hypothesis, the most relevant fact is the unconditional importance of money.

While these findings seem reasonable, I feel compelled to make the standard reliability critique of self-reported data. Our learning/fun AMT workers might also be there for the money, but feel sheepish about saying so. Though this could go the other way as well I suppose: if, for example, a worker has an intrinsic love of image captioning but finds this passion shameful, they might report that they are in it for the money. But this seems less likely than the other scenario of downplaying financial motivations.