Online labor markets dramatically lower the cost and hassle of conducting experiments. On Amazon’s Mechanical Turk, it is easy to run multiple experiments per week. Figuring out how to run experiments isn’t that hard, as there are already some nice tutorials available.

However, what I felt was missing from the field was a discussion of why, precisely, we can trust results from online experiments. This was the motivation for a new paper, jointly written with Dave Rand (who wrote up part of this study here on the Dolores Labs blog) and Richard Zeckhauser.

You can download the paper here.

While we make the practical and theoretical case for online experimentation, we believe that acceptance of online results as “valid” will come after people start seeing how easy and reliably one can replicate previous studies. This is why blogs like Experimental Turk and Deneme—both of which report results from AMT experiments—are so helpful. In our paper, we continue this process by replicating three results that are fairly well established.

In one experiment for the economists, we show—contra the usual intuition—that at least some Turkers are financially motivated, despite the very low stakes. After performing an initial text transcription task, workers were offered some randomly chosen amount of money to do an additional transcription. Results show the counts of people who agreed (“Yes”) and the counts of people who did not agree (“No”), by amount offered.

Nothing too surprising—offer to pay more and more workers will accept—but at this stage in the development of online experiments as a methodology, “surprising” would probably be bad news.

Anyway, the full paper is here. We’d love to get comments and feedback—it’s not too late to earn a place in our coveted “thanks” footnote!

Brendan and I wrote a chapter for an O’Reilly book called Beautiful Data. We took a lot of the analysis from earlier blog posts and distilled it into a longer book chapter about exploring a large data set and turning the messy data into beautiful, compelling graphs. We tried to highlight the tools and techniques that don’t make it into textbooks and are instead passed along by word-of-mouth among people in the field.

You can check out a version of our chapter, and if you like it, we recommend you buy the book which is full of authors I admire: Jeff Hammerbacher, Toby Segaran, Aaron Koblin, Nathan Yau, Mike Migurski, Peter Norvig, Andrew Gelman and many more.