I were to offer one piece of advice to the market research industry on my death bed it would be "do more piloting!".
We have found that effective piloting really can lead to more efficiently executed research so it amazes me that more research companies have not latched on to the value of doing this.
1. Editing back questions
One of the biggest problems online surveys suffer from is verbosity! They are often too long which can lead to respondents disengaging and dropping out before the end of the survey. The main reason for this is that so often the questions being asked are "speculative" in nature, and so to ensure that you have shed enough light on an issue it is often the case that questions need to be asked about the same topic in a variety of different ways to cover all bases.
Well, anyone who has ever done a survey will, I am sure, have experienced that feeling of seeing the results from a particular question in the survey and thinking, “hm, that's not very interesting”.
We too often find ourselves throwing away the answers from questions in a survey once we see the data. Analyse how many questions in a typical survey actually make it to the report. In my experience, it can often be less than half. There is nothing inherently wrong in this - research is often an exploratory process meaning that it cannot be helped that you end up heading down some blind alleys. It is a part of the process.
The great thing is, is that piloting can really help with this. You can ask a question in a pilot and if it sheds light on an issue keep it and if not, then remove or adapt it.
2. Refine option choices
On a more detailed basis you can also use piloting to trim back redundant option choices. How often have you found yourself building a tick list of choices, guessing which options you need to include and finally ending up with a number of options that only a very few % of people actually click on? Conducting a pilot will enable you to see early on how many people click each option allowing you to remove those that are either rarely selected or that significantly overlap with other options.
If you are really clever and can afford to, do a larger scale pilot so that you can look at the statistical relationship between answers to different questions and weed out ones that significantly overlap or can be modelled from related questions.
3. Refining range options
At an even deeper level you can use pilot data to refine option ranges. Say, for example, if everyone appears to be opting for the top 2 boxes on a likert scale you may well decide to reword the range options to deliver a wider spread of responses.
4. Comparing answers to the same question asked in different ways
The way a question is framed can often have a major bearing on how it is answered. Which is something you can often find yourself getting locked into big debates with your clients and colleagues about. The beauty of piloting means you can test out different wordings and see how it affects the answers. You can play about with different approaches. You don’t have to speculate - you can test. And therefore you can present your clients and colleagues with the facts and end those frustrating debates!
5. Designing more intelligent answer code frames
The code frames used for many questions, particularly word associations, are very often created by the flights of imagination of the researcher or client or have been adapted from another survey. As a result some of the words or option choices you select may do little more than confuse respondents and make it harder for them to make their selections due to the options simply not being relevant.
An example of this was a question I saw recently about what motivates your decision to buy a soft drink. There were over 20 option choices including things such as "I research online before I make my decision" and "I would seek advice from a sales person" which completely missed the mark, having no bearing on the original question itself.
Well, you can use piloting to help you design the most beneficial code frames. You do this simply by asking it as an open ended question: "what motivates your decision to buy a soft drink?". You can then pick out the most popular responses and use them to create the optimum option list within your main study.
This is such a simple technique that we have used with successful results on several occasions and would recommend it highly as a tool to help with creating a far more efficient and engaging survey.
6. More clearly understanding sample size requirements
Without piloting, planning out what sample size to use in a survey is often a question of licking your finger and holding it up in the air. You can only really guess. The only way you can effectively estimate sample size requirements for a survey is by getting some people to do the survey and see how the answers pan out.
For example if you ask 50 people a question and 24 choose option A and 26 choose option B you will need a far larger sample to differentiate which choice is actually better. But if the answers came back whereby 10 people chose A and 40 people B, in this instance you would know that you don’t need any more sample at all; you have already answered the question.
Often the sample size requirement will vary by question, but it isn’t difficult these days to adjust on a question by question basis to determine the required number of people per question. This really can be a very smart way of optimising the length of a survey.
We recently employed this technique in one of our surveys and were able to reduce the overall length of it by 30% having discovered that only 20% of people need to answer this question, 50% that question and 100% those questions and so on.
7. Working out how to encourage the best response to an open ended question
One of the most successful uses for piloting is to experiment with ways to encourage people to give you more open ended feedback. This is often very sensitive to methods used and the gains can be enormous. We recently changed a question in a survey we were working on with Sony music from "Tell us about your interests in music" to "Imagine you were being interviewed by a magazine, how would you describe your interest in music" . As you can sense by seeing this question set out here in their differing ways, this can completely change the quality of the feedback. We found this worked so much so, that we adopted the technique throughout the rest of the questionnaire and the overall word count of the survey increased from 230 to over 400 words per respondent.
8. Working out how to effectively engage respondents
On a broader level you can use a pilot to test out different intro messages and see what effect this has on survey completion times, experiment with different images and styles of communication all these seemingly small things can have a real impact.
9. Error trapping
The last thing is that piloting is great for error trapping. It’s often so difficult to spot all the errors there may be in a survey. Particularly things like broader logic routing problems. Some surveys may have 20+ different paths making it very difficult to check. Again piloting really helps with ensuring you don't run into problems when the real survey hits the field.
Piloting techniques
OK so you want to do some piloting now. How do you go about piloting? Here are some tips and suggestions:
1. Be open minded about how many responses you need for your pilot.
Having seen the benefits of piloting I don't think I personally ever consider launching an online survey without doing a pilot first, even for the smallest of studies.
We have found that effective piloting really can lead to more efficiently executed research so it amazes me that more research companies have not latched on to the value of doing this.
The reasons for this are I think historical. In the days of pen and paper interviews it was simply not a practical option to pilot a piece of research as it would potentially add weeks to the turn-around time of a project and could end up being prohibitively costly, so it never became an established part of survey design protocols.
However, today with the speed of online research, sketching out a survey and piloting it with just 25-50 respondents is something that can be done in a matter of just a few hours and will add nothing to the overall cost of a project. In fact, it could very well save you money in the long run.
This blog post is a short guide to how you can best use piloting to improve your market research...
With the right understanding of the techniques involved, piloting can have a very useful role in shaping the content and structure of a survey; helping to define the value of questions, perfect the question wording, assessing what options should be included or omitted, and also to help work out exactly what sample sizes you need to employ. The impact of piloting can often be completely transformational.
Through a process of piloting we have been able to double, and even treble in some cases, the volume of feedback obtained, seen significantly reduced drop-out levels from surveys, enabled us to assess whether surveys have even been worth conducting in the first place and validated theories before the main wave of the research has even been carried out. Used effectively, piloting really can make your research budget go further.
So how do you go about effectively piloting a MR study and what are the main things that can be achieved through this process? Here is a guide to some of the tricks we have learnt about how to effectively do it:
With the right understanding of the techniques involved, piloting can have a very useful role in shaping the content and structure of a survey; helping to define the value of questions, perfect the question wording, assessing what options should be included or omitted, and also to help work out exactly what sample sizes you need to employ. The impact of piloting can often be completely transformational.
Through a process of piloting we have been able to double, and even treble in some cases, the volume of feedback obtained, seen significantly reduced drop-out levels from surveys, enabled us to assess whether surveys have even been worth conducting in the first place and validated theories before the main wave of the research has even been carried out. Used effectively, piloting really can make your research budget go further.
So how do you go about effectively piloting a MR study and what are the main things that can be achieved through this process? Here is a guide to some of the tricks we have learnt about how to effectively do it:
Firstly what can you use piloting to do?
1. Editing back questions
One of the biggest problems online surveys suffer from is verbosity! They are often too long which can lead to respondents disengaging and dropping out before the end of the survey. The main reason for this is that so often the questions being asked are "speculative" in nature, and so to ensure that you have shed enough light on an issue it is often the case that questions need to be asked about the same topic in a variety of different ways to cover all bases.
Well, anyone who has ever done a survey will, I am sure, have experienced that feeling of seeing the results from a particular question in the survey and thinking, “hm, that's not very interesting”.
We too often find ourselves throwing away the answers from questions in a survey once we see the data. Analyse how many questions in a typical survey actually make it to the report. In my experience, it can often be less than half. There is nothing inherently wrong in this - research is often an exploratory process meaning that it cannot be helped that you end up heading down some blind alleys. It is a part of the process.
The great thing is, is that piloting can really help with this. You can ask a question in a pilot and if it sheds light on an issue keep it and if not, then remove or adapt it.
In my experience you can easily trim back a typical survey upwards of 20% of the questions this way. In fact, I have been known to have thrown away whole surveys in the bin on the back of a pilot because the results were of no real benefit. I have also completely rewritten others in the light of the insight that the pilot survey delivered.
2. Refine option choices
On a more detailed basis you can also use piloting to trim back redundant option choices. How often have you found yourself building a tick list of choices, guessing which options you need to include and finally ending up with a number of options that only a very few % of people actually click on? Conducting a pilot will enable you to see early on how many people click each option allowing you to remove those that are either rarely selected or that significantly overlap with other options.
If you are really clever and can afford to, do a larger scale pilot so that you can look at the statistical relationship between answers to different questions and weed out ones that significantly overlap or can be modelled from related questions.
3. Refining range options
At an even deeper level you can use pilot data to refine option ranges. Say, for example, if everyone appears to be opting for the top 2 boxes on a likert scale you may well decide to reword the range options to deliver a wider spread of responses.
4. Comparing answers to the same question asked in different ways
The way a question is framed can often have a major bearing on how it is answered. Which is something you can often find yourself getting locked into big debates with your clients and colleagues about. The beauty of piloting means you can test out different wordings and see how it affects the answers. You can play about with different approaches. You don’t have to speculate - you can test. And therefore you can present your clients and colleagues with the facts and end those frustrating debates!
5. Designing more intelligent answer code frames
The code frames used for many questions, particularly word associations, are very often created by the flights of imagination of the researcher or client or have been adapted from another survey. As a result some of the words or option choices you select may do little more than confuse respondents and make it harder for them to make their selections due to the options simply not being relevant.
An example of this was a question I saw recently about what motivates your decision to buy a soft drink. There were over 20 option choices including things such as "I research online before I make my decision" and "I would seek advice from a sales person" which completely missed the mark, having no bearing on the original question itself.
Well, you can use piloting to help you design the most beneficial code frames. You do this simply by asking it as an open ended question: "what motivates your decision to buy a soft drink?". You can then pick out the most popular responses and use them to create the optimum option list within your main study.
This is such a simple technique that we have used with successful results on several occasions and would recommend it highly as a tool to help with creating a far more efficient and engaging survey.
6. More clearly understanding sample size requirements
Without piloting, planning out what sample size to use in a survey is often a question of licking your finger and holding it up in the air. You can only really guess. The only way you can effectively estimate sample size requirements for a survey is by getting some people to do the survey and see how the answers pan out.
For example if you ask 50 people a question and 24 choose option A and 26 choose option B you will need a far larger sample to differentiate which choice is actually better. But if the answers came back whereby 10 people chose A and 40 people B, in this instance you would know that you don’t need any more sample at all; you have already answered the question.
Often the sample size requirement will vary by question, but it isn’t difficult these days to adjust on a question by question basis to determine the required number of people per question. This really can be a very smart way of optimising the length of a survey.
We recently employed this technique in one of our surveys and were able to reduce the overall length of it by 30% having discovered that only 20% of people need to answer this question, 50% that question and 100% those questions and so on.
One of the most successful uses for piloting is to experiment with ways to encourage people to give you more open ended feedback. This is often very sensitive to methods used and the gains can be enormous. We recently changed a question in a survey we were working on with Sony music from "Tell us about your interests in music" to "Imagine you were being interviewed by a magazine, how would you describe your interest in music" . As you can sense by seeing this question set out here in their differing ways, this can completely change the quality of the feedback. We found this worked so much so, that we adopted the technique throughout the rest of the questionnaire and the overall word count of the survey increased from 230 to over 400 words per respondent.
8. Working out how to effectively engage respondents
On a broader level you can use a pilot to test out different intro messages and see what effect this has on survey completion times, experiment with different images and styles of communication all these seemingly small things can have a real impact.
9. Error trapping
The last thing is that piloting is great for error trapping. It’s often so difficult to spot all the errors there may be in a survey. Particularly things like broader logic routing problems. Some surveys may have 20+ different paths making it very difficult to check. Again piloting really helps with ensuring you don't run into problems when the real survey hits the field.
Piloting techniques
OK so you want to do some piloting now. How do you go about piloting? Here are some tips and suggestions:
1. Be open minded about how many responses you need for your pilot.
20 may be enough but you may need 100 if some of the answers are not clear. It’s easy for sample providers to send out small batches of invites and close off surveys. So use small batch samples aiming to get 20 to 50 completes at a time, then review the data and if more clarity is needed ask the panel provider to send out more invites.
2. Use micro test and control cells of 20 to compare question technique.
May sound small but in my experience there its enough to roughly get the picture. Particularly when you are auditing things like the volume of free text this should be enough sample to tell if one version is working more significantly better than another.
3. Don't bother testing out the whole survey
2. Use micro test and control cells of 20 to compare question technique.
May sound small but in my experience there its enough to roughly get the picture. Particularly when you are auditing things like the volume of free text this should be enough sample to tell if one version is working more significantly better than another.
3. Don't bother testing out the whole survey
Pick the parts that you think need developing, but obviously try and make it as close to the real experience as possible
4. Don't fret over things like typo's or conditionality
4. Don't fret over things like typo's or conditionality
Focus on the bigger picture it is, after all, only a sketch.
5. Don't forget to ask respondents for feedback
5. Don't forget to ask respondents for feedback
It sometimes can really be useful to ask respondents their experience of doing the survey. What they thought about answering specific questions.
6. Suggest, if you can, doing the pilot before you talk to the client.
6. Suggest, if you can, doing the pilot before you talk to the client.
Obviously depending on the client and the brief, some will offer invaluable feedback and it is useful that they are involved. But with others it is like taking 2 people to the video shop you can never make a decision. Most of the things you do in a piloting are quite technical, to decide what range choices to use for example. Once you do the pilot you can then go back to the client armed with incite as to how to effectively write the final questionnaire.
7. Build the piloting process into your established methodology and cost for it.
Yes it will add to the cost of an individual survey but in the long run, over the rolling cycle of say 10 surveys it will almost certainly pay for itself.
8. Use your company/friend/family as free panellists!
8. Use your company/friend/family as free panellists!
It depends on the project but if the demographics are not important, why not?
9. If you are working with a panel provider who is going to do the scripting of the pilot for you, brief them about this at the quote stage.
9. If you are working with a panel provider who is going to do the scripting of the pilot for you, brief them about this at the quote stage.
And don't muck them around with too much tweaking the pilot! The one issue I would say I have had with conducting pilots is clients getting wrapped up in them and not being able to differentiate the protocols of signing off a pilot versus a main study. I have had instances where I get a long list of minor wording corrections for example that have no real bearing on the results and insisting on loads of unnecessary tweaks and changes.
Why not think about piloting your next online survey you will find that...
Why not think about piloting your next online survey you will find that...
- Your survey will be shorter and more efficient
- Will deliver back better data to your clients
- You will not be wasting so much of your budget on asking unnecessary questions
- Your survey is less likely to have errors in it and you will sleep better at night while the survey is in the field
Having seen the benefits of piloting I don't think I personally ever consider launching an online survey without doing a pilot first, even for the smallest of studies.
You really have great potential and consistency!Thanks for striking the sleepy minds!
ReplyDeleteQuestionnaire Questions