We have found that effective piloting really can lead to more efficiently executed research so it amazes me that more research companies have not latched on to the value of doing this.
With the right understanding of the techniques involved, piloting can have a very useful role in shaping the content and structure of a survey; helping to define the value of questions, perfect the question wording, assessing what options should be included or omitted, and also to help work out exactly what sample sizes you need to employ. The impact of piloting can often be completely transformational.
Through a process of piloting we have been able to double, and even treble in some cases, the volume of feedback obtained, seen significantly reduced drop-out levels from surveys, enabled us to assess whether surveys have even been worth conducting in the first place and validated theories before the main wave of the research has even been carried out. Used effectively, piloting really can make your research budget go further.
So how do you go about effectively piloting a MR study and what are the main things that can be achieved through this process? Here is a guide to some of the tricks we have learnt about how to effectively do it:
1. Editing back questions
One of the biggest problems online surveys suffer from is verbosity! They are often too long which can lead to respondents disengaging and dropping out before the end of the survey. The main reason for this is that so often the questions being asked are "speculative" in nature, and so to ensure that you have shed enough light on an issue it is often the case that questions need to be asked about the same topic in a variety of different ways to cover all bases.
Well, anyone who has ever done a survey will, I am sure, have experienced that feeling of seeing the results from a particular question in the survey and thinking, “hm, that's not very interesting”.
We too often find ourselves throwing away the answers from questions in a survey once we see the data. Analyse how many questions in a typical survey actually make it to the report. In my experience, it can often be less than half. There is nothing inherently wrong in this - research is often an exploratory process meaning that it cannot be helped that you end up heading down some blind alleys. It is a part of the process.
The great thing is, is that piloting can really help with this. You can ask a question in a pilot and if it sheds light on an issue keep it and if not, then remove or adapt it.
2. Refine option choices
On a more detailed basis you can also use piloting to trim back redundant option choices. How often have you found yourself building a tick list of choices, guessing which options you need to include and finally ending up with a number of options that only a very few % of people actually click on? Conducting a pilot will enable you to see early on how many people click each option allowing you to remove those that are either rarely selected or that significantly overlap with other options.
If you are really clever and can afford to, do a larger scale pilot so that you can look at the statistical relationship between answers to different questions and weed out ones that significantly overlap or can be modelled from related questions.
3. Refining range options
At an even deeper level you can use pilot data to refine option ranges. Say, for example, if everyone appears to be opting for the top 2 boxes on a likert scale you may well decide to reword the range options to deliver a wider spread of responses.
4. Comparing answers to the same question asked in different ways
The way a question is framed can often have a major bearing on how it is answered. Which is something you can often find yourself getting locked into big debates with your clients and colleagues about. The beauty of piloting means you can test out different wordings and see how it affects the answers. You can play about with different approaches. You don’t have to speculate - you can test. And therefore you can present your clients and colleagues with the facts and end those frustrating debates!
5. Designing more intelligent answer code frames
The code frames used for many questions, particularly word associations, are very often created by the flights of imagination of the researcher or client or have been adapted from another survey. As a result some of the words or option choices you select may do little more than confuse respondents and make it harder for them to make their selections due to the options simply not being relevant.
An example of this was a question I saw recently about what motivates your decision to buy a soft drink. There were over 20 option choices including things such as "I research online before I make my decision" and "I would seek advice from a sales person" which completely missed the mark, having no bearing on the original question itself.
Well, you can use piloting to help you design the most beneficial code frames. You do this simply by asking it as an open ended question: "what motivates your decision to buy a soft drink?". You can then pick out the most popular responses and use them to create the optimum option list within your main study.
This is such a simple technique that we have used with successful results on several occasions and would recommend it highly as a tool to help with creating a far more efficient and engaging survey.
6. More clearly understanding sample size requirements
Without piloting, planning out what sample size to use in a survey is often a question of licking your finger and holding it up in the air. You can only really guess. The only way you can effectively estimate sample size requirements for a survey is by getting some people to do the survey and see how the answers pan out.
For example if you ask 50 people a question and 24 choose option A and 26 choose option B you will need a far larger sample to differentiate which choice is actually better. But if the answers came back whereby 10 people chose A and 40 people B, in this instance you would know that you don’t need any more sample at all; you have already answered the question.
Often the sample size requirement will vary by question, but it isn’t difficult these days to adjust on a question by question basis to determine the required number of people per question. This really can be a very smart way of optimising the length of a survey.
We recently employed this technique in one of our surveys and were able to reduce the overall length of it by 30% having discovered that only 20% of people need to answer this question, 50% that question and 100% those questions and so on.
One of the most successful uses for piloting is to experiment with ways to encourage people to give you more open ended feedback. This is often very sensitive to methods used and the gains can be enormous. We recently changed a question in a survey we were working on with Sony music from "Tell us about your interests in music" to "Imagine you were being interviewed by a magazine, how would you describe your interest in music" . As you can sense by seeing this question set out here in their differing ways, this can completely change the quality of the feedback. We found this worked so much so, that we adopted the technique throughout the rest of the questionnaire and the overall word count of the survey increased from 230 to over 400 words per respondent.
8. Working out how to effectively engage respondents
On a broader level you can use a pilot to test out different intro messages and see what effect this has on survey completion times, experiment with different images and styles of communication all these seemingly small things can have a real impact.
9. Error trapping
The last thing is that piloting is great for error trapping. It’s often so difficult to spot all the errors there may be in a survey. Particularly things like broader logic routing problems. Some surveys may have 20+ different paths making it very difficult to check. Again piloting really helps with ensuring you don't run into problems when the real survey hits the field.
OK so you want to do some piloting now. How do you go about piloting? Here are some tips and suggestions:
1. Be open minded about how many responses you need for your pilot.
2. Use micro test and control cells of 20 to compare question technique.
May sound small but in my experience there its enough to roughly get the picture. Particularly when you are auditing things like the volume of free text this should be enough sample to tell if one version is working more significantly better than another.
3. Don't bother testing out the whole survey
4. Don't fret over things like typo's or conditionality
5. Don't forget to ask respondents for feedback
6. Suggest, if you can, doing the pilot before you talk to the client.
8. Use your company/friend/family as free panellists!
9. If you are working with a panel provider who is going to do the scripting of the pilot for you, brief them about this at the quote stage.
Why not think about piloting your next online survey you will find that...
- Your survey will be shorter and more efficient
- Will deliver back better data to your clients
- You will not be wasting so much of your budget on asking unnecessary questions
- Your survey is less likely to have errors in it and you will sleep better at night while the survey is in the field
Having seen the benefits of piloting I don't think I personally ever consider launching an online survey without doing a pilot first, even for the smallest of studies.