Assessing the nutritional value of your survey?

Is the feedback from your survey the equivalent of a double quarter pounder burger with extra cheese and a side of fries?



Yep you get the quick hit from it you want, you are filled up with facts - people strongly agreeing that your brand or is great and a whole load of nice descriptive words that people associate with you product that you can talk to people about at your next marketing meeting.

My question is....have you assessed the real nutritional value of your survey?  Are you actually getting any actionable information that you can use to improve your product or services or are you feeding your organisation with a whole load of pretty useless information that is clogging up the communication arteries?

If you interview 1,000 people and 50% of them say they are aware of your brand and they rate it highly, this might make everyone in the marketing department happy but apart  from slapping yourself on the back what are you actually able to do with this information? To know that your brand is seen as being modern, technologically advanced & innovative, is that just nice to know or is it going to help you move your brand forward?

I see so many surveys that are just plane too fat!

My primary target for this type of criticism would be customer feedback surveys which are often stuffed full on benchmark questions that tell you how much people rated various aspects of  a companies service rather than focusing on having a conversation with customers to find out how they feel and what the company could do to improve themselves.

I just came back from a trip with an airline, I think it probably diplomatic for me best not to mention their name and completed their customer satisfaction survey. It included over 100 questions covering in fastidious detail every aspect of my journey with them. All hail the companies attempt to be thorough and is concern about every detail of their service but the survey was just over weight.

I was asked 6 specific question about the staff custom service: if the staff were friendly, were they open and up front; did they make things easy for me; did they care if I had a good experience; did they treat me as an individual etc All these were asked on a x point scale and so I click the same option for all 6 questions because their service was well, fine, they were nice, I had no opinion about it! If fact i didn't really notice them I am afraid I am sorry. So  I didn't need to answer 6 questions.

 I wonder how many people have done this airline survey, I guess well over 50,000?  I wonder how much over the course of a year the rating of their staff customer service actually changes?  I bet the first 500 on average deliver pretty much the same answer as the last 500.  So why ask all 50,000 people all 6 of these questions, why not just ask one in 10 people one of these question and aggregate it out.  I bet that would be statistically good enough to get a good steer on their customer service.

What is more this extremely long survey it was also totally self obsessed.   I was not asked for example to rate how good their staff customer service was compared to other companies and whether I thought it needed to be improved or was good enough, That really was the only question I really needed to be asked in this case.

What was ironic about this was that in amongst this whole survey it did not ask me about the one thing that had actually bothered me about my flight..

Why could then not have just asked me just one question - how was the flight?

5 tips for tackling survey obesity

1. If its a tracker or customer feedback survey where you have some existing data I suggest the first thing you do is get someone to through the answers to that survey and looked at which questions predict the answers to other questions and looked at which questions are actually delivering unique incite.

2. Work out statistically how many people need to answer each question to get a statistically accurate indicator.  For some question it might be closer to 50 people than 500 people.

3. Next question has anyone sat down to find out the answers to which questions are actually being used by anyone in the organisation?  Ask the marketing department if they had to buy the data back what they would pay for the answer to each question?  Ask them what decisions will be made as a result of each question.  If they don't have an answer to this then challenge them to ask them why they want to know.

4. Has anyone thought about exactly what questions would be useful to ask each customer and thought about customising what question were asked based upon the attitudes of each customer?

5. Randomise:  I think the problem we face particularly when we are doing exploratory research is we have slipped into a habit of using a scatter gun approach of asking questions with a hope that some will deliver back some incite.  There is nothing wrong with doing this at perhaps the pilot phase of a survey.  There is nothing wrong with asking lots of question so long as I don't have to answer all of them - I would advocate where there is doubt randomise splitting samples so you ask more questions but each to less people!  (see last post on the monte carlo method)