Then there is the eternal authentic/strategic problem. If the survey respondent knows that his votes have an effect then he spend his tokens very differently. I had a small survey, designed to pick a winning paper, blow up on me once because one of the respondents was brilliantly strategic – turning all the dials to max for his preference and to min for the others.

The error I find most frustrating though is when the survey designer lacks a model of what the distribution of is, so his multiple choice is unable to capture a good signal for the parameters of the actual distribution. e.g. is distirbution zip or normal

How many time do you spend online per weekday?
___ hours and ___ minutes online on weekdays

And how many time do you spend online per day on weekends?
___ hours and ___ minutes online per days on weekends

I really can’t imagine that. Throw up an error if the participant enters any letters. If the value is bigger than 24 hours or 60 minutes you can throw it out right away. (You have to be a bit careful with the minutes because from my experience some like to leave the hours blank and enter something like 90 minutes or 120 minutes.) All an easy enough fix.

If those open ended questions get any more complicated you will get into trouble, that’s for sure. But as long as it’s as easy as this, I doubt there will be many problems. And from what I know – talking to other communication studies researchers (who ask these types of questions all the time) – you actually get very accurate time estimates. No even asking for minutes is a fruitless exercise. Measurements of time spent online or in front of the TV match up pretty good to the self reported data (i.e. less than one hour difference, hence minutes matter, but you could probably leave them out if you wanted to make it easier).

Still, I can see you point about complexity. But that also depends on the respondents. I’ve seen people answering this question and some think long and hard and really try to come up with an accurate estimate. Others will just pull a number out. I would guess that scales are easier for the second group. But not exactly for the first. Scales might be even harder for them: they can’t just write their estimate in but have to fit it into categories.

(I like you point that scales help quantify things. But open ended questions can help there, too, albeit in a bit more subtle and maybe a bit less helpful way. When asking how often the participants go to the cinema or opera the researcher should first get a feeling for the average in the population and then decide whether to ask for visits per week, month or year.)

Michael – your comment illustrates a great point that deserves a lot more discussion: scale response questions may not be the best way to get the information you want on a given topic. That said, there are a few reasons I can think of why you still might want to use scales in certain situations:

(1) They’re fast for respondents. Every question you put on a survey is a demand on your respondents’ time and patience. A multiple choice scale makes life a tiny bit easier and quicker for respondents, therefore making a tiny bit more likely that they will answer the question and complete the rest of your survey.

(2) They eliminate a lot of data entry errors. If you leave a text box in your survey, somebody will invariably tell you that they spend “1q0″ hours online per day. Was that supposed to be 10? Did they accidentally hit 1 instead of the tab key and therefore intend to enter a zero? If you don’t know for sure, you (the survey administrator) will need to throw out their response. Depending on the study, missing data may raise really big problems with your sample and your ability to generalize about the population you’re researching.

(3) They help respondents quantify things: the fact that people can use scales to dis-orient themselves in relation to a range of responses (as the people with the low scale did in my experiment) implies that they can also use scales to orient themselves accurately. For example, I don’t regularly quantify the amount of time I spend on public transportation each week. Assuming that a researcher has done his/her homework in designing the scale responses, the scale may actually help me provide a more accurate answer than I would have on my own.

(4) They make impolite questions or unusual responses seem safe. Your example about the impropriety of asking people about their income or age in Germany is exactly what I have in mind here. In addition, a well-designed scale can sometimes help people feel more comfortable identifying themselves as outliers on a sensitive issue. Again, this may help more people respond to a question and give accurate answers.

I think these are all great reasons why the particulars of the study, the population, and the topics of interest should drive the research design process. Every type of question has characteristics that may look like limitations in a given set of circumstances, but strengths in another.

(Well, actually I would also make it two questions. One for weekdays, another for weekends. Otherwise the participants might be forced to do averaging in their heads – something which our programs are much better at.)

Pretty much the only time where I would use a scale in a case where the participant could also just give a number is income. And that’s only because asking directly for the income is considered very impolite (in Germany). A few years ago you would have to add age to that, but that has already changed and the thing with the income will, too.