Questions on trial

Surveys are competing with a billion+ largely more fun things to do online these days and we are reaching tipping point were many people, the young age groups in particular are simply refusing to complete survey because they are too boring.

To have any chance of competing we have to change our approach and I think this starts with taking a long hard look at some of the boring questions we are asking in our surveys.

One of them being this one...

"What brands are you aware of?"

It a question asked in nearly every consumer survey I come across, usually asked both unprompted with an open ended question and then prompted with a closed question set of brand options (twice the work). Its one of those sacred cow set of questions that everyone insists on asking.

Do we actually need to keep asking this question?  Are there not better questions that could be asked in better ways, that make better use of a respondents brain that deliver more useful data?

Here is the case for the prosecution:

• Its a really dull question: From a respondents point of view these are probably the dullest most cliched question they continually have to answer in surveys. Respondents don't like answering these questions,  they trigger drop out. 
• Respondents put little thought into their answers: Less than half the respondents you ask this question name more than one or two brand when asked unprompted and well over 20% say don't know. The prompted question often gathers together a random set of clicks from respondents.  
• Little statistical value:  If the average respondent list 1 or 2 brand and assuming the number of individual brands listed adhere to a Zipf's law style distribution, the most popular brands named much more often than the least popular brands - on a typical sample of 400 respondents there will only likely to be 1 or 2 brands you ever have enough data to work with statistically. 
• Large error boundaries:  The data error boundaries on this question on most survey samples are of often so large that in wave to wave in brand tracking studies the fluctuations down to pure statistical error are often of an order of magnitude larger than the actual underlying change in brand awareness.  Resulting in a lot of "overfit"
• No stand alone value: The data it delivers back is almost always duplicated or can effectively be modeled from answers given to other questions in the same survey.
• Meaningless metric: Brand awareness information is totally useless in isolation from anything else. What it measures is intangible it's certainly not an accurate measure of purchase behavior for example - Unprompted awareness correlate at around about 0.54 with purchase behavior*, prompted awareness it drops to under 0.2*.  If you specifically want to find out what brand consumers are likely to buy there are other question that are far more effective.  It is also not a measure of how much I may like a brand....
• I have never see how this information is really used profitably:  It's always the chart that everyone skips past in a presentation. 

The case for the defense:  

Now I have challenged several prominent and respected researchers many of whom are still very wedded to asking this question in survey and asked them why they like to use it.

• Its a fundamentally important measure: The brand that is mentioned first is the brand that has the most brain neuron connections and associations between the product and the category. So many see it as fundamentally the most important question you can ever ask in a survey.

Now I get this but I am still left with a feeling of so what....if you ask me what brand of chocolate I am aware of  - I will say Cadbury's.  I have been exposed to this brand all my life seen thousands of Cadbury's ads, seen the brand in every confectionery counter I have ever visited.   But I never buy Cadbury's and so what x number of people are aware of a brand.

Now I am open to some other arguments as to why it should be kept if anyone wants to make them, but my judgement verdict is that its a question that if not completely banned, should at least be taxed, in the same way that cigarettes and alcohol are taxed to discourage their usage.

What could be asked instead?

There are a range of alternative ways of directly or indirectly measuring brand awareness that are more interesting and potentially useful.

I find one of the frustrations to answering the what brands  do you recall question is thinking why does it matter and not knowing how many to list, Simply applying a rule to the question that contains the task it in a more meaningful framework for respondents can make it far less dull to answer.

you could ask:
...what are their favorite brands
...which brands they would have in their perfect supermarket
...which brand, if they could only buy one, would they choose to buy a life long supply of... ...which brands they would take to the desert island
...which brand they would invite to a party
...which brands they would recommend to their best friends
...which brand they would invest in
...which brands they think they will still be buying in 5 years time

All these will gather top of mind awareness to some degree or other but are more interesting and purposeful for the respondents to answer and adding in these "rule" can make them more salience and relevance e.g. asking about what brands they would have in their perfect supermarket  it not just just a measures awareness but also intent to purchase.

In head to head experiments we have found we get more responses to these more conceptually fun questions too.

You could turn it into a full blown game by adding to your list of brands some fake ones and challenge respondents to pick out the real brands.  We have used this approach on several occasions, its more fun for respondents and the data you get back is almost identical to prompted brand awareness.  You could also show them a facet of the packaging of a brand and see if they can guess which brand its for, or show them a de-branded ad.

I would also advise jumping straight past the specific awareness question and asking what range of brands they have purchased on there last 10 shopping occasions. The has very high correlation with actual sales c0.8 and again cuts to the heart of the problem, if they are aware of it but not purchased it in any of their last 10 shopping occasions its not on their purchase radar!

If you want to defend the use of this question in your surveys and have got a strong argument for doing so i would love to hear your thoughts.