As part of these experiments he showed an example of the negative impact of an over elaborate confusing flash format question can have on data quality and the willingness of respondents to take part in future surveys.
Well I wanted to contribute to this debate and so here are my thoughts...
I think this highlights an issue about the quality and standard of survey design not just about using flash per say...
We have done about 3 years of work looking at how to effectively design and deliver flash questions in surveys and have published a couple of papers on this topic and along the way have experienced very similar issues to Bernie where using a more creative question approach can have a negative effect if they are poorly designed or confusing for respondents to use.
Yes poorly designed question can damage data
When designing more creative question formats if you do not think about their ergonomic design or you try to over elaborate them, or expect them to do too much, yes things can go badly wrong - triggering drop-out and respondent dissatisfaction. We have found some drag and drop question for example can be very confusing for respondents if you make them too complicated and with slider format question where the slider can prompt the answer they can deliver back data that can be quite different to what you might expect when asking the same question in a more traditional way (see my other blog post on how best to use sliders in a survey to find out more on this How to use sliders )
But get it right more creative question approaches can be much more effective...
But what I would want highlight that if you get it right, well designed and thought through flash questions can really work and deliver exactly the same balance of data as traditional question though with less neutral scoring (which is an indirect measure of respondent boredom), radically improve respondent enjoyment levels, reduce dropout and straight lining effects. In experiments we have been able to reduce dropout from surveys by up to 75% and increase data click counts by as much as 50% - so when you get it right, the benefits really are quite significant.
Its all about simplicity, visual clarity, aesthetic appeal, and working on the flow (e.g. using techniques like animated auto-nexting of question options so you don't flood the screen with options) ....
Its all about simplicity, visual clarity, aesthetic appeal, and working on the flow (e.g. using techniques like animated auto-nexting of question options so you don't flood the screen with options) ....
Appreciating the skill involved and experience needed in the designing of surveys
Just like say creating an advert on TV there a good ones that sell products and bad ones that don't. You could liken the use of flash v normal radio questions to say using Print v TV advertising. Sure you can say that TV advertising is a lot more effective than print. But you can easily make a dreadful TV ad that performs worse than a print ad. It is a creative process and in the case of advertising it is heavily dependent on the quality of advertising agency you employ to do the task.
Just like say creating an advert on TV there a good ones that sell products and bad ones that don't. You could liken the use of flash v normal radio questions to say using Print v TV advertising. Sure you can say that TV advertising is a lot more effective than print. But you can easily make a dreadful TV ad that performs worse than a print ad. It is a creative process and in the case of advertising it is heavily dependent on the quality of advertising agency you employ to do the task.
There is a thought out there in the industry that just by dumping a couple of off the shelf wacky flash questions in your survey you are going to make the survey a whole lot more engaging. As Bernie has shown, you are just as likely to confuse people chopping between formats and highlight how boring the rest of the survey is and the net value will be negative.
When I discussed this with Bernie he also highlighted a very salient point that survey programmers are usually tasked with speed, not ‘thinking’. Survey production is a job that is often deemed something that the IT dept does under the researchers instructions and is not really seen as anything other than a production line process. So whilst a good survey programming team are probably well aware of some of the main issues that effect data quality in survey design they take their dues often from researchers who may have little or know knowledge of these issue through lack of experience.
When I discussed this with Bernie he also highlighted a very salient point that survey programmers are usually tasked with speed, not ‘thinking’. Survey production is a job that is often deemed something that the IT dept does under the researchers instructions and is not really seen as anything other than a production line process. So whilst a good survey programming team are probably well aware of some of the main issues that effect data quality in survey design they take their dues often from researchers who may have little or know knowledge of these issue through lack of experience.
Next generation survey design companies
I believe making survey is a holistic creative process that requires not just looking at the design, but the wording and structure. It should be treated with the same respect as the design of say an advert and with the understanding that it is not something you can be universally successful at implementing, or that anyone can think they are able to do themselves - it requires a lot of skills and experience in the same way as creating advertising does.
I see a future where as we seem more and more creative techniques becoming available, a new breed of specialist survey design companies will emerge into the market research industry that work along side MR firms that are treated with the same respect as design companies or advertising agencies, who are total experts in the creative execution of surveys.
I see a future where as we seem more and more creative techniques becoming available, a new breed of specialist survey design companies will emerge into the market research industry that work along side MR firms that are treated with the same respect as design companies or advertising agencies, who are total experts in the creative execution of surveys.
P.S.
I did email Bernie to say I would post this blog and I would like to thank him for his sincere openness in encouraging me to add my views to this debate and he made it clear that his presentation was put out there in the first place the goal of fostering dialogue on this topic which clearly he has managed to acheive. I totally agree with him that more industry standards need to be established as new question formats emerge into the main stream.
If anyone would like some further reading I would be happy to distribute the papers we have problished on how to engage respondents in online research and I would recommend anyone getting hold of the brilliant book "The Art of Asking Questions" by Stanley L Payne published in 1951 for the definitive words written on this topic half a century ago.
I did email Bernie to say I would post this blog and I would like to thank him for his sincere openness in encouraging me to add my views to this debate and he made it clear that his presentation was put out there in the first place the goal of fostering dialogue on this topic which clearly he has managed to acheive. I totally agree with him that more industry standards need to be established as new question formats emerge into the main stream.
If anyone would like some further reading I would be happy to distribute the papers we have problished on how to engage respondents in online research and I would recommend anyone getting hold of the brilliant book "The Art of Asking Questions" by Stanley L Payne published in 1951 for the definitive words written on this topic half a century ago.
You're being far too charitable to this guy. This is the same type of argument that small minds have always made about progress with their concern troll nonsense. Conflating the technology with its implementation is a sure sign someone can be safely ignored.
ReplyDelete