Marcel Proust

A few months ago, I started a blog written by Mechanical Turk workers. Eight times a day, on the hour, a script posts a personal question on MTurk. The questions are randomly selected from a subset of the Proust Questionnaire. Example questions include:

1. What is your favorite food and drink?
2. What is your idea of misery?
3. Which natural talent would you most like to be gifted with?

The first worker to submit his or her answer automatically has their response anonymously posted on the blog, which I call “Mechanical Proust.” Workers are told that their responses will made public, but remain anonymous. I make no attempt to check responses before they are posted, nor do I check whether a worker has submitted in the past. (I enabled Google Ads to see if I could make the blog financially self-sustaining. No luck here so far.)

What have I learned? Most of the responses are unsurprising but they are occasionally poignant or insightful. People are proud of their children. They regret dropping out of school. They want to live in Paris. They fear dying and being alone. They like chicken dishes, etc. Lots of workers are funny (see screenshot at left).

Technical Notes:

A Python script posts the questions (using boto), and cron schedules them. The questions are presented as external HITs, with the “submit” button launching another Python script that posts the text response to the blog, using Google’s API. If you want the code or want help making something similar, let me know.

Next steps (taken by someone else):

I’d like to see someone make a new version of this blog where blog readers could submit questions, which would go into a queue, with newer questions going at the bottom. Each new question would have up/down Reddit-type buttons, which could move questions up and down the stack. Questions could then be posted on MTurk LIFO-style.

Online labor markets dramatically lower the cost and hassle of conducting experiments. On Amazon’s Mechanical Turk, it is easy to run multiple experiments per week. Figuring out how to run experiments isn’t that hard, as there are already some nice tutorials available.

However, what I felt was missing from the field was a discussion of why, precisely, we can trust results from online experiments. This was the motivation for a new paper, jointly written with Dave Rand (who wrote up part of this study here on the Dolores Labs blog) and Richard Zeckhauser.

You can download the paper here.

While we make the practical and theoretical case for online experimentation, we believe that acceptance of online results as “valid” will come after people start seeing how easy and reliably one can replicate previous studies. This is why blogs like Experimental Turk and Deneme—both of which report results from AMT experiments—are so helpful. In our paper, we continue this process by replicating three results that are fairly well established.

In one experiment for the economists, we show—contra the usual intuition—that at least some Turkers are financially motivated, despite the very low stakes. After performing an initial text transcription task, workers were offered some randomly chosen amount of money to do an additional transcription. Results show the counts of people who agreed (“Yes”) and the counts of people who did not agree (“No”), by amount offered.

Nothing too surprising—offer to pay more and more workers will accept—but at this stage in the development of online experiments as a methodology, “surprising” would probably be bad news.

Anyway, the full paper is here. We’d love to get comments and feedback—it’s not too late to earn a place in our coveted “thanks” footnote!

“Who are these people?” and “Why do they participate?” are two perennial questions about AMT. Askers are generally incredulous that AMT workers are willing to do rather tedious tasks for small amounts of money.

To investigate this question of motivation, NYU Prof. Panos Ipeirotis asked a bunch of workers their reasons and tabulated the responses here. His two posts are actually on the syllabus for a course at Stanford (incidentally the course is taught by one of the creators of Protovis, which is very cool and is on my list of things to learn). There is also this amusing investigation.

For a joint project with Dave Rand and Richard Zeckhauser, we asked ~ 400 AMT workers both (a) where they are from and (b) the primary reason they participate on AMT. Because economic opportunities differ by country, we might expect that motivation and behavior should also differ by country. The cross tabulation plot is below (reasons are in the “rows”, countries in the “columns”–the size of each rectangle is proportional to the number of responses in that cell):

Two things to note:
1) Money is a big motivation for everyone
2) Money aside, people from India are there to learn; people from the US are there to have fun

Although the India/US differences are consistent with the different-countries/different-motivations hypothesis, the most relevant fact is the unconditional importance of money.

While these findings seem reasonable, I feel compelled to make the standard reliability critique of self-reported data. Our learning/fun AMT workers might also be there for the money, but feel sheepish about saying so. Though this could go the other way as well I suppose: if, for example, a worker has an intrinsic love of image captioning but finds this passion shameful, they might report that they are in it for the money. But this seems less likely than the other scenario of downplaying financial motivations.

This is the first in (what we hope will be) a series of guest posts from John Horton, a Doctoral Candidate in Public Policy at the Harvard Kennedy School. John and Aaron Shaw are collaborating on some research projects and we were both introduced to Dolores Labs around the time of last year’s Mechanical Turk Meetup.

Since then, John’s been busy establishing himself as a Crowdsourcing research pioneer by designing a suite of online data collection tools as well as running numerous experimental and observational studies on several different Crowdsourcing labor markets. We really admire his work, which tends to involve well-designed methods and cut straight to big, interesting questions. In this post, John discusses a recent experiment he ran on Amazon Mechanical Turk that looks at worker motivations in the context of labor economics and theories of the “reservation wage.”

Hi – this is my first post here (though I’ve commented a bit). I work with Aaron Shaw and got to know Lukas at the last meet-up he hosted. Anyway, I’m interested in crowdsourcing and online labor more generally and Lukas was kind enough to let me write about some of my research here.

It’s pretty clear that many Amazon Mechanical Turk (AMT) workers are motivated primarily by money, which suggests economics is the best tool for understanding worker decision-making.Research by Winter Mason and Duncan Watts shows that workers behave in a way consistent with economic rationality: when they were paid more, workers produced more output. Although any sensible model predicts that workers will work more when paid more, standard labor economics models make several other predictions (some might call them assumptions): workers should make decisions based solely on the real wage offered — payment divided by time spent. They should compare this offered wage to their reservation wage for a particular task.

Because it drives decision-making, the reservation wage is the key parameter in labor supply models, but it is hard to estimate in practice; when we observe someone working, even if we know their wage we don’t get to observe their reservation wage parameter — we just know that their wage is above the reservation wage. In a new paper (joint with Lydia Chilton), we use a unique method that allows us to estimate reservation wages for AMT workers. Although we find some agreement with the predictions of the simple rational model, we also find some evidence that workers are “target earners,” meaning that the work until they reacg certain salient earnings targets (e.g., the maximum amount available). This kind of behavior has been found in other contexts, but it runs counter to the rational model.

The Task

For our task, subjects clicked back and forth between two vertical bars in a Flash game (screen shot below). A block of 10 back-and-forth clicks made up one unit of output, and subjects could decide how many blocks to complete. The amount paid per-block was constantly decreasing. This constantly decreasing rate allowed us to esimate a worker’s reservation wage, by looking at the implied wage when they “quit.” A live demo of the task is available here.

Results

Subjects were randomly assigned to either a HIGH or LOW group. The HIGH group was paid 3 times more than LOW for every task. The figure below shows output in both groups. One striking feature of the data is how bimodal output is: some workers produced lots of output and some produced very little. For this bimodality to be consistent with rationality, the distribution of reservation wages themselves would have to be very bimodal, which seems unlikely.We found that the imputed reservation wage distributions were quite different across groups. Because of randomization, the distributions should have been indistinguishable. In particular, we found that the reservation wages in LOW were too low, suggesting that workers in LOW, on average, worked more than they should have. Why?

Target earning

One possible explanation for why there is too much output in LOW is that at least some workers try to earn the maximum amount possible, regardless of the “wage” associated with this strategy. Having an earnings target may sound rational, but can lead to some perverse results. For example, workers might work longer when wages are low (because they still want to meet their target) than when they are higher (though there are other reasons this can happen, namely income effects). It is an open controversy in economics whether employees with “real” jobs are target earners (see this work by Henry Farber as well as Colin Camerer’s work), but we find several pieces of evidence for target earning in our data.

The strongest evidence we find for target earning is that some workers show a preference for earning total amounts divisible by 5. In the figure below, the earnings of workers in HIGH are plotted as a histogram, with horizontal panels for the whole cents earned. E.g., earnings amounts 29.2, 29.5 and 29.9 would all be in the same “29 cent” panel. The height of the bars show how many subjects earned that amount of money. Panels where the whole cents are divisible by 5 have black histogram bars; the others have white bars.We can see that several subjects earn the smallest amounts available (e.g., 2, 3 and 5 cents). Because these low earners quit very early, they presumably do not have a target or would not need a target. However, we see clear output spikes at 15, 20 and 25 cents. The probability of this happening by chance is about 3 in 1000 (see the paper for details).

Conclusion

We find some agreement with the rational model, as well as important anomolies consistent with some ideas from behavioral economics. While it’s probably too early to offer much practical design advice, it does seem that designers should give workers natural targets, as they seem to help at least some workers. The paper is called “The Labor Economics of Crowdsourcing” and is available here.