Crowdsourcing Anxiety and Attention Research

As a psychologist, I spent years of my life running experiments in a lab the traditional way: emailing volunteers one by one, scheduling them for an hour in my calendar, and guiding them through the tasks I want them to complete. In a good month I’d have a dataset with 20 participants and I’d spend weeks analyzing it every possible way before running another experiment.

Now that I work at CrowdFlower, I want to find out whether we can use crowdsourcing to answer psychological questions more efficiently, without lowering data quality. If so, it has the potential to revolutionize the whole field.

The Experiment

I picked a survey-based experiment that I conducted multiple times as a researcher, which centers on the following question: what is the relationship between anxiety and attention?

Trait anxiety is how likely we are to have anxious thoughts and behaviors. It’s different than state anxiety which is how we feel at the present moment.

Attentional control is a technical term that describes our ability to pay attention.

Some psychologists have found that the more anxious you tend to be, the harder it is to pay attention (e.g., Derryberry & Reed, 2002). Stress causes the brain regions that process emotional information to “hijack” the brain regions that control attention. Highly anxious people’s brains incorrectly categorize neutral information as emotional information, which causes them to be easily distracted.

In my own research I had trouble reproducing this result, so I wanted to try the same experiment using CrowdFlower. I surveyed about 1000 people (click here to view a pdf with details on methodology and demographics of our sample).

Higher anxiety was clearly correlated with worse attention with a p-value of less than 0.00001.  As you can see from the scatterplot, the correlation is not perfect, so it was necessary to access a large group of people to clearly show a statistically significant connection.

Now, what else did we find out from this large data set?  We saw that there is no statistical difference between females and males on this correlation. The regression lines are almost exactly the same for both females and males.

We found that people aged 18 to 22 have significantly less attentional control than people 23 and above. People 40 and above have significantly higher attentional control than people 22 or younger. This makes me think that maybe the best time for higher education is when you’re 23, not 18!

We also found a large increase in anxiety when people turn 18, and then a slow drop off as age increases.

 

We found that democrats (n=372) and independents (n=304) were each significantly more anxious as a group than republicans (n=237). Although it appears that the Green party has the highest anxiety score, there were only 6 respondents in that group – not enough to draw any conclusions!

Similarly to the above findings, we also found that single people (this does not include cohabiting couples) are significantly more anxious that married people, and atheists are more anxious than Christians.

Most psychological research comes from studying a very narrow slice of the population. Typically, experimental participant pools are confined to the undergraduates at the scientist’s university, or worse, the university’s psychology majors who do experiments for class credit. How can we discover generalities about how the human mind works if we base our conclusions on, for example, the psychology majors at New York University? From my experience at NYU, these students are upper middle class, Caucasian females from Long Island between the ages of 18 and 20.  With crowdsourcing, the hope is that we can expand the participant pool to a wider range of geographical locations, ages, income levels, and political and religious affiliations. Moreover, we can collect more data in much less time, for less money.

 

- Emma

 

Resources

StatWing (https://www.statwing.com)

  • The statistical analyses you see in this post were calculated using StatWing, a new and free online tool to make sense of your data! Brought to you by one of CrowdFlower’s old colleagues. Just upload and analyze.

R (http://www.r-project.org/) and RStudio (http://rstudio.org/)

  • The pretty graphs were made in R using custom scripts. R is a statistical programming language and RStudio is a friendly graphical developing environment. Both are free to use.

 

Further Reading

  1. Berggren, N. and Derakshan, N. (2012). Attentional control deficits in trait anxiety: Why you see them and why you don’t. Biological Psychology, epub ahead of print.
  2. Bishop, S.J. (2009). Trait anxiety and impoverished prefrontal control of attention. Nature Neuroscience, 12(1):92-8.
  3. Derryberry, R., and Reed, M.A. (2002). Anxiety-related attentional biases and their regulation by attentional control. Journal of Abnormal Psychology, 111(2):225-36.
  4. Spielberger, C. D., Gorsuch, R. L., Lushene, R., Vagg, P. R., & Jacobs, G. A. (1983). Manual for the State-Trait Anxiety Inventory. Palo Alto, CA: Consulting Psychologists Press.

The gentle revolutionary fervor

Guest post by David Alan Grier, President of the IEEE Computer Society, and Author of “When Computers Were Human”.

Two years ago, a group of us awoke to a stunning realization that enough people were involved in this thing called crowdsourcing to form a community. We hoped to build our community at an annual conference where we could share common interests and learn from each other. In the heat of the moment, we also believed we were the vanguard of a new revolution in work. The hour had come. The old ways were in retreat. It was time to stand triumphant upon the ramparts and declare a new industrial revolution.

The revolution has not declined, nor have we lost the sense that we are seeing the start of a fundamental shift in the nature of work. A recent visit to an entrepreneur in Washington, DC, shed light on this point. With a couple of full-time employees and a few interns from local colleges, the startup’s operational model relied almost solely on crowdsourcing. Web services, web development, accounting, and marketing were all crowdsourced.

Even more fascinating was crowdsourcing’s effect on the firm’s hierarchy. Each full time member of the company, including the interns, was an acting manager. They were responsible for requesting services, procuring services, and assessing the results of those services. It’s an organizational structure that looks quite different from the one that operated 20 years ago, and probably only hints at the organizations we will see 20 years in the future.

The time we occupy probably has much in common with the beginnings of the software industry. In 1968, programs were unique to each computing site. Few programs were ever shared. None were shared across different brands of computers. Over the course of 12 years, the position of the programmer changed substantially. Instead of apply the labor of programming to one company or one site, the new industry amortized programming skill across hundreds or even thousands of sites. It gave industry a new common base to exchange information.

Crowdsourcing is now at the forefront of building a common base for all workers.  It will bring to industry the value of aggregated labor.  It will bring to workers the value of aggregated demand, which should include a greater choice in work and more opportunities to gain skill.

In the end, we may be too close to the daily ins and outs to fully understand the changes being wrought.  Just as software pioneers didn’t quite grasp that they were restructuring the way programming work got done, so we don’t quite see how we are restructuring labor as a whole.  So we come to CrowdConf to talk and reflect, to see what we have accomplished in the last year, to anticipate what will change in the next 20, and to share a common vision throughout.

David Alan Grier
President, 2013 IEEE Computer Society
Author When Computers Where Human, Too Soon To Tell, and The Company We Keep

Quality in the crowd

I’m going to be delving into the topic of quality in the crowd at Crowdsortium on Thursday, September 13, and I figured I’d give a preview of some of the things we’re thinking about at CrowdFlower.

Chris and I started CrowdFlower on the premise that we could use smarter statistical analysis to collect high quality data from the crowd. What we’ve observed over and over again is that while math goes a long way, it isn’t enough. Quality in crowdsourcing comes from clearly communicating the problem we are asking someone to solve, and motivating the worker to do a great job.

Over the past few years, we’ve built several critical tools to communicate with our contributors. We use training data, called “Gold”, that immediately gives contributors feedback on whether they got an answer right or wrong. We show contributors more Gold when they start a new task, and we tailor Gold around edge cases where we think someone might make a mistake. We also have a technology called CrowdFlower Markup Language that helps us and our customers build beautiful job interfaces for the individuals doing our tasks.

Our most recent developments are focused on motivating contributors. We’ve created a dashboard for each contributor that tracks progress over time on CrowdFlower jobs:

 

As you can see, the dashboard includes badges that reward strong work, and we’re using these badges to give our best contributors access to higher paying work. We’ve also built a leaderboard for our most prolific contributors (it’s something that we’ll be adding to our Teams page, a constant reminder of the individual contributors who are at the core of what we do). So far, our contributors are writing in that they love the changes, and we’re incorporating their feedback into the product.

I’m looking forward to learning about other cool ways of approaching quality in the crowd at Crowdsortium on Thursday.

-Lukas

What Color is this? in 9 languages

I’ve always wanted to re-do some of the scientific studies of the past, like the World Color Survey. While I don’t have plane tickets or time to travel the world, I do have access to CrowdFlower’s 4 million contributors to re-test hypotheses about the universality of color-naming.

Four years ago, we showed English language speakers random colors and asked for the color names. Four years later, with CrowdFlower contributors now in every country of the world, the experiment becomes much richer. The question is not only “Where does blue end, and red begin?”, but do people from different countries have different concepts of color boundaries?

The color-wheel above (thanks D3 and Dawn) contains 4,000 colors (we collected many more, but didn’t want to crash everyone’s browsers). Mouse-over the color-wheel to see the names of the colors in nine different languages, with translations into English. You can also filter by language using the search box and country flags, so you can see the differences between where Russians vs. Chinese vs. Japanese see red.

On the whole, it looks like countries have extremely similar conceptions of color. Type “blue” into the search box, click on the different countries, and you can see the overlap. There are outliers though. Some narrower colors – such as “purple” – are used much more in Japan than in Russia. The use of certain modifiers such as “light” are used pretty uniformly across the color spectrum in English, but much more prevalently in the Blue-Green region in Japanese.

What do you see in the data? You can download the raw data here. Like last time, find something interesting in the data and we’ll post it here!

And stay tuned for more blog posts on when big crowd meets old science.

The CrowdFlower task seen by contributors in the U.S., France, Germany, and China:

Announcing Senti – Richer Sentiment Analysis Through Crowdsourcing

Over the past few months, our PM Dave and our engineering team have been working day and night on a new product: Senti.

Senti makes it incredibly easy to ask the crowd rich questions about your social media data.


The Senti Dashboard (designed by our talented Mars):

 

The core idea behind Senti isn’t new: our customers have used CrowdFlower as a text analysis tool almost since the day we launched.

One of our customers looked into how people feel about different airlines, and found that smaller carriers had more positive tweets.

Another customer looked at tweets about the weather across states to figure out the best places to live. You can actually watch cold fronts and snowstorms pass over the country as blocks of people become increasingly negative about their home towns.

Four years ago, it felt exciting and innovative just to look at snippets of headlines and tweets and check if they were positive or negative. Over the years, we’ve learned a lot from our customers and the thing we heard the loudest is that they want to ask deeper questions about their social media.

This is where Senti comes in.

When we ask nuanced, complicated questions, we start to see more actionable data.  For example, basic sentiment analysis might tell me that half the tweets about a new blockbuster movie are positive.  Senti could tell you that 90% of those positive tweets are from people who haven’t seen the movie yet, while 90% of the negative tweets are from people who have suffered through the movie.  As a consumer, I could avoid seeing the movie :).  As a movie studio exec, I could decide to invest in more previews and advertisements to counteract negative word-of-mouth.

Want to see Senti at work?

We ran a sample of tweets about LuluLemon and found that tweets from people who already owned the clothes were much more positive about the brand than people who didn’t. You can play with the Senti dashboard for this data.

We ran the same kind of analysis on BestBuy and found that people who were talking about a BestBuy advertisement or promotion felt relatively positively about the company. The dashboard is here.

Over the next two weeks, we’ll be posting some fun experiments using our new Senti tool. My hope is that Senti will push the field of social media analysis forward by making it feasible to ask more meaningful questions about all of the social data we have at our fingertips.

-Lukas