I was out of town for
several days while a telephone survey project was going on at my office. When I got back to work and started editing
the completed interviews I noticed some odd things in the data. I cannot give too many details without
violating the client’s confidentiality, but I can say that it was a problem
that I anticipated before I left town. I
briefed our interviewers on the project the night before I caught my flight to Phoenix
The survey was one that we
have conducted almost every year for the last several years. The sample for the survey is retired people
and the questions with the weird answers can be confusing to people of any
age. We ask people to calculate
percentages of their household income.
When I briefed the project last week I told interviewers to slow down,
especially when reading these questions.
I reminded the interviewers that we are interviewing people in their
70s, 80s and 90s, so they might not hear or completely understand the questions
if the interviewer reads too fast. One
of the interviewers, a man in his 70s, said in the briefing that he was
offended by this. I replied that I was
basing the instruction on my experience with the project. I knew that we would have problems if the
interviewers did not slow down.
A good interviewer will
mirror their pace to the pace at which a respondent speaks. A good interviewer will also listen for when
a respondent is not listening and re-read questions as appropriate. Our questionnaires are written with certain
words EMPHASIZED so that respondents have a clearer understanding of what we
are asking. When I brief and coach
interviewers, I usually tell them to emphasize sibilant
words. Words such as “IF” and “BOTH” do
not travel well over the telephone, so respondents sometimes do not hear
them. When this happens, respondents
often become confused by the question or think we are asking a different question.
I held a meeting with the
interviewers on my first day back at work, before we started the dialing
shift. I told them I was cutting off
their coffee until they slowed down.
very true that the pace or speed in which a survey is read has big implications in the collected data.
ReplyDeleteBesides not fully understanding the question, often times multiple responses are accepted, but a quick or hurried reading might not make that clear to the respondent. Good interviewers will understand to probe fully, but less experienced or thorough interviewers may not pick up on fact that respondent is thinking only one answer required and will not offer up additional responses unless prompted.
The real concern is that a too quick reading of the questions essentially creates multiple surveys being administered. That is, you may get some respondents answering the questions as written and others answering what they think they heard or ubderstood.
Many respondents will grow weary of a survey where they constantly have to ask for the interviewer to re-read the question. That will result in an unacceptably high breakoff rate or respondents basically flatlining their responses just to get thru the interview.
Neither result promotes the confidence in the collected data.
And, interviewers will hurry when the survey is just too long. If there are too many questions, interviewers will speed up to squeeze all the questions in. Again, if there are too many questions, interviewers will feel compelled to cut respondents off in mid-sentence.
ReplyDeleteI think there is a trend these days to add more and more probes and clarifications within one question without adding additional time. Clients, too, are on the hook for getting every last drop of information out of a survey because research budgets are down and opportunities to hear from customers or users are greatly reduced.
Every question has to have a purpose and lead to actionable results.