Tuesday, 3 July 2012

How to calculate the length of a survey

As an industry we tend to use survey length as the cornerstone for how we price surveys, but often the estimated lengths and real lengths of surveys can turn out to be wildly different. Leading as I have experience to potential conflict.

The reason is we have not established in the research industry a common and reliable way of estimating the length of a survey.  The most common method in circulation is to assume we answer surveys at 2.5 questions per minute but this technique is fatally flawed.  This is because question themselves can vary wildly in length e.g. a survey with 10 grid question with say 50 options may take 50 times longer to answer than a survey of 10 simple yes no questions.

So I have been on a bit of quest to work out some slightly more accurate ways of do this. As a result of some recent work we have been doing to examine in detail how long respondents answer survey I have come up with 3 new alternative methods I would like to put forward to more accurately calculate the length of a survey. 

I hope they may be of use to some of you.

Method 1: Survey length = (W/5 + Q*5 + (D-Q)*2 + T*15)/60


This is the most accurate way of doing it (though I recognise it take a quite a bit of work). This formula will given you the length of an English language survey in minutes.

W = word count: Do a word count of the total length of questionnaire (questions, instructions and options). An easy way to do this is to cut and paste the survey into word but don't forget to remove any coding instructions first and it will tell you the word count. Respondents read English in western markets at an average rate of 5 words per second.

Q = Number of Questions: Count how many questions the average respondent has to answer. Allow 4 seconds per question general thinking time and 1 second navigation time* (assuming 1 question per page).
*this may vary depending on survey platform if it takes longer than 1 second to load each page adjust accordingly

D = Total number of decisions respondents have to make: Count in total how many decisions the average respondent makes in total using this guide below and allow then 2 seconds per decision.

Single choice question = 1 decision
Multi-choice question = 0.5 of a decision per option
Grids = 1 decision per row

T = Open ended text questions: Count how many open ended text feedback questions a respondents has to answer and allow 15 seconds per question. (note this may vary quite dramatically based on the content of the question but on average people dedicate 15 seconds to answering and open ended question).

Method 2: Survey length = (W/5 + R*1.8)/60

If you want slightly simpler approach use this formula which is not quite so reliable but will get you close...

W= word count
R = total number of row options: Note this is just rows and not columns on a grid. This can be quite easily done by cutting and pasting your survey into excel and then in a side column mark up all the rows and then sort.


Method 3: W/150

If you don't have enough time to add up all the number of questions and row options this is another quick a dirty method (though I would not vouch for it being much more acurate than the 2.5 questions per minute approach).

This will give you a rough estimate of the length of a survey in minutes.  It is no where near as acurate as the above 2 more detailed methods but it will be someone in the correct ball park. Careful though if  you spot your a dealing with a particularly verbose questionnaire.

A wisdom of the crowd approach I would recommend would be to use both the 2.5 question and W/150 methods and compare the differences - if they produce just about similar figures well go with that, if they generate big differences it might be worth adopting method 1 to do it properly.


Where all these formula will fall over?


1. If all the respondents don't see all the questions: Skip logic can mean not everyone sees every question in a survey which means it can be hard to work out the average number of question respondents will have to answer which you will need to know to accurately work out the average survey completion time. Most errors in estimating survey length centre around this issue. There is often no easy way of doing this other than manually working it out using a spreadsheet.

2. Not properly taking into account question loops. This another issue that leads to people miscalculating the length of a survey. If for example there is a loop of question that you ask for a set of brands people often forget to include the extra time it take to answer these question and only count one loop.

3. If you are working out the length of a survey not conducted in English: or where English is not the primary language (India for example) you will need to weight for longer reading, comprehension, consideration and survey loading times in different countries. Below is a rough weighting guide if you are working from a translated version of an English survey, (sorry that I don't have time weighting data from every country):

            Length   Weighting
Japanese/Korean 0.95
Netherlands 1.00
Germany 1.05
French 1.06
Spain 1.09
Scandinavia 1.10
Italy 1.11
Chinese 1.13
India 1.34
Eastern Europe 1.35
Russia 1.37
Latin America 1.43

4. If there are a lot of images in the survey:  you will need to allow for extra loading times. Allow between 2-10 seconds per mb.

5. If you are including a lot of  non-standard question formats in the survey e.g. sorting and drop down style question take longer to answer.

6. If you are boring people to death with a long highly repetitive survey!  Respondents will start speeding when they get bored and so average decision making time can drop.

Do you have any thoughts?


Now I would love to hear from anyone who has some thoughts on this or have come up with what they think is a more effective means of doing this. My ultimate aim is to find an agreed means for the who industry to adopt to use as a more effective trading currency when pricing surveys.




6 comments:

  1. Jon

    Big caveat needed - that this is how to calculate for doing a pc-based web survey.

    If you are doing a pc-oriented survey on a mobile device - then the math would be fundamentally different to take into account all the pinching and squeezing on the screen that is currently required.

    If you had a properly mobile-centric survey, then you may have a very different experience and the time to complete may well go down.

    And what if you can actually take a survey and change the way that you are sampling such that you can accurately and systematically break up the survey - increase the sample size, but reduce the time for any given person to answer the questions.

    Cheers

    Rolfe
    www.realitymine.com

    ReplyDelete
  2. Can you please say how you came up with the "length weighting" for different countries? Is this based on empirical data?

    ReplyDelete
  3. Yes good points Rolf about a mobile surveys. These timings are specifically for online PC surveys. I suspect slightly increased loading times would be something that would also have to be factored into it.

    ReplyDelete
  4. Regarding the length weightings yes these are based upon the comparisons of a wide cross section of surveys completed in different countries. They are only intended to act as a rough guide.

    ReplyDelete
  5. This comment has been removed by a blog administrator.

    ReplyDelete
  6. I think I'm doing something wrong. With the first method I got 10, but when I do the 2.5 per question I get two hours. 48 questions with several grids, shouldn't be 10. Help! This is my first survey

    ReplyDelete