As the NFL season loomed on the horizon back in August, there was a lot of fantasy football talk around the CrowdFlower office. Naturally, we decided to crowdsource a killer fantasy football team.
Fantasy football tackle. Photo Credit: dennis
We asked CrowdFlower workers to help us build a ranked list of the Top 75 players to guide our fantasy football draft. We used pair-wise comparisons to determine rankings. Specifically, we presented workers with two players, asking them to pick which player they thought would be more valuable.
We used the Top 75 players from ESPN’s 2010 Fantasy Football Draft Kit1, matching each player with his 74 counterparts, giving us a total of 2,775 player pairs. We also included each player’s position and team, as well as a link to more detailed statistics.
Screenshot of fantasy football task.
After the job finished, we ordered the players according to number of head-to-head victories.
Crowdsourced list of the Top 25 most valuable fantasy football players.
(Download the full results here.)
We found that workers choose the player on the left 53 percent of the time, even though each match-up appears twice, once with Player A on the left and again with Player A on the right. With 5,500 data points, this is significant bias toward the player on the left. However, the final ranking doesn’t change even after accounting for this bias.
We settled on two likely explanations for the bias:
- The anchoring effect of putting something on the left (and as the first response option) may have caused more workers to select the first player.
- Our Gold was slightly biased toward the player on the left, which we have previously seen2 to have an effect on the overall distribution of the answers.
As a first comparison of our crowd of football fans with ESPN’s Fantasy Football brain trust, we plotted each player by the difference between his two rankings.
Players on the left were ranked higher by ESPN, while players on the right were ranked higher by AMT.
Will Chad Ochocinco and Matt Ryan vindicate the crowd? Stay tuned to find out.
1. http://games.espn.go.com/frontpage/ffldraftkit
2. http://www.ischool.utexas.edu/~cse2010/slides/le.pptx
gameswithwords
I think it’s pretty common to see a left-side bias, which is why researchers typically counterbalance side (as you did). Jennifer Arnold at UNC has some neat data I heard about recently, in which she noted that (1) people tend to look at the left side of a picture first, and (2) that affected their answers on subsequent questions. The details aren’t important; that’s just by way of saying such biases aren’t uncommon or unimportant.
Jonathan
and the results are?
Thomas Watts
Thanks for the information. But isn’t the LHS of the page got anything to do with the way our brain works … ie LHS bias in our brains itself … also being our analytically side. I know it might be a long bow to stretch but as we are analyzing figures.