The average respondent spend 15 seconds answering the average open ended question and you get on average of 5 words. 5 word might be enough if it were meaningful feedback but so often it is humdrum verbage which makes it a bit of a nightmare to analyse. The most common response if you were say to ask people watch and ad and write down what they thought of it would be "it was OK" and the second and third most common remarks would be, "I liked it" and "I didn't like it" and the 4th is "I don't know" i.e often not a great deal of value to the extent you may as well asked a yes no question did you like it or not.
This is a guide to show you how to write more effective open ended questions to improve the quality of feedback to encourage respondents to:
- Put more thought into their answers
- Be more creative
- Be more analytical
- Be more reductivist
- Be more free form in their thinking
Techniques for Motivating respondents to give you the type of answer you want.
The first challenge you face is motivating respondents to answer the question you pose. Often respondents have little or no reason to write something down in the first place. So they question you should ask yourself is what would make me want to answer this question?
1. Explaining the reason why you want to know the answer: It may not always be possible of relevant to do this but explaining why you want to know can help motivate people to give an answer. Instead of asking "what do you like about this product" you might ask it like this "This is a very popular product but we are not quite sure why, what do you like about it?"
2. Bring respondents into the problem: I would also advocate bringing respondents into the problem you are trying to solve. So instead of saying "why do you like this brand" you might say "this brand is a lot more popular amongst some people than other and we are trying to understand why.."
3. Emotionalizing your appeal for feedback: If you can find a way of emotionalising this appeal more the better. Below is an example of the impact this can have. we asked one group of mothers to talk about their experience of using baby wipes and another group we explained that the reason why was to try and improve baby wipes for future generations of parents and supported it with this visual. We got nearly 50% more feedback from this group.
4. Using visuals: Using a well chosen visual to re-enforce question as this example above demonstrates can have a measurable impact on improving the volume of feedback. Take this next example below which was part of an international experiment we conducted looking at the general effects of taking a more creative approach to survey design. One group we ask to imagine they were in charge of the TV coverage for the Olympics games and had to build up a list of sports they would want to cover. The second group we visually re-enforced this with a picture of the Olympic logo and a camera. We have tested this out in 15 counties and in every one the presence of the image increased the feedback by upwards of 20%.
5. Challenging them: Using language that challenges respondents to do something can be an incredibly powerful weapon to encourage more feedback. So instead of say asking "what brands come to mind" you might re-phrase that to say "how many brands can you guess". Or instead of saying "what words come to mind when you think of this brand" you might say "can you think what the most popular words people associate with this brand". This type of approach can sometimes double the level of feedback we have discovered.
6. Using reward mechanics: Have you ever thought about offering a little prize for the best answer? I can tell you it it is an extremely powerful approach, in experiments we have run we seen up to 6 fold improvements in feedback, obviously a bit complicated to administrate but is a somewhat overlooked technique in my mind.
Setting expectations on type of feedback you want
Most people usually use their own standard to asses how much time and effort they should put into answering an open ended question. We have found that by establishing some expectations without making them actually conditional can help improve the volume of feedback. There are several different ways you can do this.
7. Using word counters: Saying to people please write X number of words on this topic and using a word counter so they can see how much they have written is a great way to extend out the volume of open ended feedback. The x limit might be 3 words or 300 depending on the circumstances.
8. Setting time limits: Like wise telling them how long they might spend answering a question is another clever technique. If you say "please spend no more than 3 minutes telling us what you think about this topic" this may sound like you are asking someone to not spend too long. In fact you are doing the opposite and establishing a benchmark that if they come anywhere near to adhering to they will be spending a lot longer than the 15 seconds they normally would spend answering the question.
9.Showing examples: by showing examples of what other people have written you can subtlety socially condition people to write more. If they see that other people have written several sentences it will encourage them to write a similar amount.
Understanding the barriers for respondents to give feedback
10. Dealing with spelling and literacy concerns: One of the reasons why some people are reluctant to give open ended feedback in surveys is because they are worried about their spelling and how well they can write. There are a couple of ways of dealing with this. Firstly to make it clear that you are not worried about this. Secondly we have found that by showing a range of more informal feedback with not necessarily perfect spelling it encourage respondent not to worry so much about their personal style of delivering feedback.
11. Foreplay - now's not the right time to get intimate! People are often reluctant to answer open ended questions right at the very start of a survey, in fact they can be the cause of considerable drop-out. You need to warm people up and get them in the right frame of mind to answer open ended questions first.
12. Tackling boredom thresholds: Likewise people put far less effort into answering open ended question at the end of a survey and there are often trigger points you will notice where people tend to emotionally give up and their desire to get to the end of a survey cuts in so strongly that their responses to open ended questions dries up.
13. Being overwhelmed by choice of what do say: Sometime people are overwhelmed by the choice of things they could say in an open ended question and it leads them to say just one thing. If I ask you for example what your favourite bands are this is a highly complex question to answer as someone may like lots of different bands and music artist and with all that choice your mind implodes and you cant think of any that stand out. This issue has been observed by social scientists in a famous experiment where people were asked to name something they might find in a fridge. The average person got stuck after naming 7 items. Yet if they were asked to specifically name white things, or things with sell by dates or bottled items in a fridge this opened them up and enable them to list a lot more. The lesson to learn from this is that when wording questions you need to think of respondents like oysters you need to feed them the grains of sand to stimulate the pearls of thinking and help them focus. So you might ask what was the first band you fell in love with or what was the last?
How to stimulate more thinking
14. Break out into thinking chunks: If you are asking what do you like or dislike about something you will find if you break this out into 2 open end text boxes "likes" and "dislikes" you will get overall more responses. I often see open ended questions that try to ask several questions at once all in one question. My advice would always be to to chunk it up and separate out to unique question the separate things you want to know. You can cluster them together so it comes across as one specific topic you are asking about but you will get much better feedback if you do it this way.
15. Using Random stimulation: Giving respondents some sort of random stimulus can be extremely powerful means of unlocking ideas. Say you are wanting people to think of an idea for a new type of ice cream you could create maybe 20 or so random thought to help people consider things from different angles this could be anything e.g. "if lady gaga were creating an ice cream what would it be like", "invent an ice cream to eat on cold days", "create an ice cream you think football fans would like" etc - the more abstract the better and each respondents you present a random set of these stimuli one at a time
16. Set rules: I have talked about this quite extensively in some of my work on the gamification of research but adding a rule to a question can make it a whole lot more interesting and fun for respondents to answer and thus trigger more thoughtful feedback. The easiest way to do this is by placing word limits - e.g. review this film in 3 words.
17. Make it personal: Always remember you are one human being talking to another and most people's favourite topic is themselves so try where ever possible to frame the question in a personal context e.g. instead of saying why do you like this brand you might say "if a friend ask you what you thought of this brand what would you say"
18. Realise the power of the word "imagine": The word imagine is a magical word to use as a means of getting people to think. We did an experiment where we asked one group of people to make a list of their favorite shops and another said "imagine you could design your perfect shopping centre, what shops would you have in it" - this increase the feedback 4 fold.
19. Apply NLP techniques: I would not purport to be an expert on this but the concept of the embedded command in a question can be extremely powerful. I could say to you "tell me what you do in 7 words" or I could say "tell me what you do in EXACTLY 7 words". respondents seem to treat the word EXACTLY as more of a command and so it tend to solicits more responses.
20. Study the form (linked in discussion): It is interesting to see which discussion topics posted on linkedin trigger the most comment and involvement. By studying this you can start to get a feel of how to pitch a question that solicits a strong response.
What can you achieve if you get it all right?
In experiments the biggest uplift in feedback we have ever achieved as a result of redesigning the way we asked an open ended question has been a 6 fold increase from 17 words to 103 worlds. This used a combination of techniques listed above and might be described as my current PB. With the right thought and approach you can easily achieved double or trebble the volume of feedback by re-engineering the wording of a question. But note as I said at the start it is often not about volume of words as much as the clarity of thinking these words stimulate. We find that with increases in volume feedback is also linked to more thoughtful feedback.
So what to do with all that open ended data?
All this begs the question about what you do with all this open ended feedback, often it can be overwhelming. With the emergence of social media text analytic software and natural language computational techniques there are starting to be some clever ways you can automatically process open ended feedback and there are some very interesting things you can do with respondent self coding feedback, but this I think is a topic for a future post....
No comments:
Post a Comment