Gamification interview with Ray Poynter

Here is a link to a short interview with Ray Poynter at the Swedish Market Research Day which I recently attended discussing gamification.  http://www.youtube.com/watch?v=tpY4Tr1zA9w&feature=relmfu 

The Ginny Valentine Badge of Courage Awards

I had the honour to attend this week  the Ginny Valenine Badge of Courage awards organised by Fiona Blades and John Griffiths on behalf of the Research Liberation Front  http://researchliberationfront.com/ginny-valentine.html and I wanted to pay tribute to the organisers and to all the winners. What a wonderful moving event this was and long may it live in the future.

Very rarely in a job like market research can I say that I have been emotionally moved by something or really felt such a strong sense of pride as I did at this award ceremony which celebrated some of the less prominent hero's of market research.  It is an award ceremony set up specifically to celebrate those people in market research who had the courage to stick their necks out and do something different, to go against the flow and who battled on through adversity to reach their goals.

The highlights for me were:

Simon Lidington who nominated his own daughter Rosie, for the efforts she put into established the Big Sofa research company. For 4 years she took a sofa around the shopping centres of the country to conduct face to face interviews with the public before pulling in there first major pieces of business.  http://www.thebigsofa.com/about_businessbenefits.html

Betty Adamou's nomination from Ray Poyner for having the courage to put her money where her mouth was and set up her own research gaming company.http://www.researchthroughgaming.com/

And Alison White who was actually brave enough to nominate herself. She explained the battle she had to set up here own field research company, her first attempt was stolen off her, the second her offices were burnt down twice.  http://facefactsresearch.com/ 

But the most astonishing tale of all though was told by Finn Raben who on behalf of ESOMAR nominated an Afghan company ORCA who had 2 researchers shot dead while collecting data.  Which puts all our own battles into perspective.

There were no black ties, 3 course dinners and celebrity presenters at this event, instead home made sandwiches and pay bar but all the better for it. Ginny Valentines son gave a wonderful eulogy to his mother and it was lovely to see him handing out the awards at the end to the winners.

200 question surveys?!!!!

In the presentation I gave yesterday at the MRS conference I mentioned that we had been working on some experimental survey games where we had managed to get people to voluntarily complete a 200 frame survey.

Now firstly, this has been quoted as a 200 "question" survey which I am afraid is a bit of an exaggeration, as a lot of the frames were feedback pages and not questions, it was about 120 questions in total.  I apologise I did not make this clear during my presentation.

I am NOT, may I repeat, NOT espousing or suggesting that anyone does a 200 question survey or indeed a 120 question survey for that matter!!!!!!!

I am slightly concerned about the mixed message this is conveying  and so I thought I should clarify things a little with this blog post about the details of this experiment.

This survey in question was designed purely as an experiment to see how many question respondents were prepared to answer, when instead of doing a survey they were playing a game and getting feedback that was  of some use and interest to them. This was not a traditional survey but a shopping game we had specially designed, that stepped away from the thinking constraints of a typical survey and focusing purely on the game and feedback mechanic.

The respondents had to work their way through a series of  "levels" where they were are asked to do things like guess the most and least expensive products, the most and least popular products, the prices of products and try and predict what different celebrities and types of people would buy. See below screen grabs of what the survey looked like.

 They would get points for getting things right and at the end of each level they would find out how well they did and we also revealed to them what this told us about the type of shopper they were.  So for example the respondents found out how price concious they were compared to other people and whether they were a social shopper who buys popular products or a individualist who buys not so popular products.  Each level was voluntary, they were asked if they wanted to proceed to the next level. There were 6 levels in total and 15-20 challenges in each level and we found 94% voluntarily completed all 6 levels spending over 20 minutes on average completing it.   The survey had an enjoyment score of 9.0 out of 10. The highest audience evaluation score we have ever achieved for a survey.

We often say that surveys should not being longer than 20 minutes. That is because most if not all surveys are not entertaining enough to persuade  us to want do them for any longer. Most surveys fail to cross the entertainment divide.  20 minutes is in effect a tolerance limit for expecting anyone to do anything they find boring.  But if you start to look at a survey through the lens of being a piece of entertainment or a game then yes it does open up possibilities for surveys that are genuinely entertaining to be longer than 20 minutes.  After all we happily will watch a film for a couple of hours, read a book all day on holiday play Angry Birds in any spare waking moment we get with our mobile phone.   But the aim of this research was not to path the way for, or encourage the industry to start churning out ever longer dull surveys!

7 factors, No sorry, 78 factors influencing the authenticity of responses

Note:  Having written my post outlining the 7 factors influencing the honestly of responses, Edward Appleton rather politely pointed out that I may well have missed one or two issues! Directing me to this great list of cognitive biases listed on Wikepedia:  http://en.wikipedia.org/wiki/List_of_cognitive_biases

Enjoy reading though them. I think  Confirmation, Congurence, Hindsight Hyperbolic discounting biases are ones I need to watch out for.  I wonder if I should offer a prize to anyone who can come up with any more.

Evolution of Advertising v Survey design

Designing surveys is without doubt a creative process similar in many ways to other creative commercial art forms like advertising and presentation design.  I feel there is something we can learn from looking at the evolution of these other sister creative processes...

Quirks Research Gamification Article

Here for those who have not seen it yet is a link to an article I have written on Research Gamification for Quirk's online Marketing Research magazine.

http://www.quirks.com/articles/2012/20120126-1.aspx 

Honesty of responses: the 7 factors at play

I read this very interesting post by Edward Appleton about the authenticity of peoples online behaviour published on the Greenbook blog. http://www.greenbookblog.org/2012/02/01/how-authentic-are-we-online/

The authenticity of online respondents is a very interesting philosophical question and is an area we have been looking at recently too - albeit in perhaps in a more literal way...

We have been examining ‘honesty’ within the online survey environment, by comparing answers to questions where we hold known norm figures about behavioural activity and building up a picture of levels of honest answering to different types of questions.

Fun = Feedback

We have recently done an experiment where we asked respondents to complete a series of survey questions designed in different ways and then at the end of the survey asked them how much fun it was to answer each question.

We then analysed the results to look at the time they spent answering each question, the volume of feedback and the level of data granularity as measured by the level of straight-lining.

In total we had data like this asked across 20 different questions and each question was asked in 2 or 3 different ways.

In every single instance the most fun version of the question encouraged respondents to spend more time answering the question, generated more feedback and resulted in least straight-lining.

Now think about a survey from a consumers point of view, it is as much a piece of entertainment as anything. Most people do survey in the spare time as a bit of fun as an alternative to perhaps playing games or going on Facebook.

 Like any form of entertainment, the more entertaining it is, the more time you we dedicate to consume it.

A Year of awards that probably won't come round very often

I am very pleased to have won our third award of the year, this one, The MRS Award for Innovation in Research Methodology that we won with Deborah Sleep at Engage Research.

You wait half you life for an award and then 3 come along all at once.   Therefore I accept in pure statistical terms this may well be the very last award I win for the rest of my life, so for the moment, forgive me I while I bask in some level of glory!

Hopefully won't last too long, my girlfriend thinks 3 awards in 1 year is bordering on greedy and has already decided I need to be brought down a peg or two starting with the washing up and sorting out the leak in the hallway.

Editorial: Pop Science Books

Reg Baker in his inevitable style posted a wonderfully forthright blog piece (click here to read) bemoaning the industries taste in books, pointing out that a great majority of the books in the list of nominees for the books that had the most transformative impact on market research were Popular Science and highlighted how unscientific some of these types of books are, championing potentially totally bogus theories that have not been subjected to proper scientific scrutiny. 

Now I can total understand his emotional reaction to this. I have exactly the same response when I go into health food shops and see all homoeopathic remedies on the shelf and am horrified that they can get away with their often totally bogus healing claims that have not been subjected to any rigorous medical trials.

But I have a few points on this that I thought I would share:

1. Books are not apples:  and in most cases not in the class of homoeopathic style remedies, one or two bad ones do not rot the whole crop.  Whilst there are some very bad pop science books, there are also some really brilliant ones that help, as Tom Ewing has pointed out, eloquently digest and explain often very complex subject matter in a clear and understandable way. To draw over riding conclusions about pop science books on the bases of analysis of parts of the sample is in itself is bad science.

What is the next big thing?

I asked the panel of judges of the my research transformation awards, who were made up of industry thought leaders and innovators from across the global market research industry, to make their prediction of what they thought would be transforming market research in the future. There were some very interesting and intelligent thoughts and so I thought I would share these predictions with you:

The Winners of The Research Transformation award

These are the result of the Research Transformations awards which I was asked to organise for a special session at the NewMR Festival.   These awards were initiated to celebrate the things that have had (or are having!) the most transformative impact on market research.  The awards have been judged by an esteemed panel of 30 leading research innovators and thought leaders from across the industry.  The full list of judges is published below and I would like to thank them all for their time and active contributions to these awards.

This is the judging survey and if you would like to CAST YOUR OWN VOTES on these awards please do,  and I hope to be able to publishing the results of this open vote in the future, it may be interesting to compare the open vote with those of the judging panel:
http://qsurvey.gmisurveys.com/dc/index.html?p=jwoYlg

The big ideas that are transforming how we think about market research
The first award is for the big ideas that are transforming how we think about market research. Over the few years a number of major ideas and theme have emerged that have shaped the way we think.

1. Listening rather than asking
The emergence of social media has opened up a whole new viewpoint on how to conduct research moving away from asking questions to listening to what people are saying.  What is has spawned a whole new  industry monitoring and measuring and analysing what we are saying when we are not asked questions by market researcher.  Listening is becoming more important than asking.

2. The hidden decision making process of our brain 

We are slowly learning more and more about how our brains work and we are finding out that the way we decide things and make decisions is a lot more complicated that we think and a lot of it happens outside of our consciousness.  This thought is really transforming how many market researchers think about conducting research, no longer can we rely on simply asking questions we have look further.
3. Information can be beautiful 

We in the market research agency can be labelled by the outside world an boring numbers people but the arrival of inforgraphics onto the scene and new story telling technique have started to make our industry a whole lot more sexy!

This is the full list of nominations.... 

Wisdom of the crowds: Discovering that groups can make intelligent decisions

Information can be beautiful: The idea that data can be sexy, and there are a lot more creative ways of presenting data than a bar chart

The hidden decision making process of our brain: Leaning that our concious mind is not always in control of our decisions

Herd thinking: Understanding the power of social influence

Co-creation: We can work together to create things

Gamification: Using gaming techniques to get respondents to think more effectively

Listening rather than asking: The idea of gaining incite from the mass of communicaton happening on the internet and using observation and means of conducting research

Big data: The emerging opportunity to consolidate information from all sources of sources to undertake super analysis

Thin slicing: The idea that you can generate a lot more incite than you think from very small data sources

The communitisation of market research: We are all sharing a lot more ideas & information with things like twitter, linkedin and blogs and this is "superscalling" our industry

30+ New Buzz words and concepts emerging from ESOMAR 3D conference

Want to know what Silent Dog Analysis is?  Have you heard about Facebook's new cluster influence model? Did you know that news is becoming a social currency?  Have you done any crowd interpretation recently? Are you up to date with your Digital Etiquette? Here are some of the new buzz words and concepts that I picked up upon at the ESOMAR 3D conference that you might want to drop into your next market research conversation. It was a veritable Exoflood of ideas!

ESOMAR Congress Award for Best Methodology Paper

I am very pleased to have won the ESOMAR best methodology paper award at this years annual congress along with Deborah Sleep from Engage research for our game experiments exploring how to put the fun into surveys and the impact it can have.

http://www.esomar.org/index.php/events-congress-2011-awards.html#best-methodological-paper

A thank you to the company that pioneered gamification

Failed experiments

Everyone likes to talk about successes of their research techniques and methodologies but we often tend to brush the failures under the carpet and don't like to talk about them. This is understandable, nobody likes to be associated with failure. But this silence on things can result in other people wasting a lot of time and effort going down the same road and making the same mistake. I am afraid I witnessed a case of this at a conference I recently attended where someone was talking about conducting video based online interviews which we experimented with a few years ago with not much success and I could see they were heading in the same direction. 

I feel a bit guilty now that we did not publish the findings from this failed experimentation at the time and so thought I’d use the opportunity of having this blog to make up for this a bit by laying a few of our failed experiments on the table as a gesture towards encouraging more open collaboration on failures as well as successes in the market research industry...

The General Theory of Gaming

If you think about things in gaming terms, you can see that so much of what we do in life itself could be classified as a game of one sort or another.

Consider that a game is a bit of thinking that we do for fun, that we gain rewards from playing them, and that there are some rules.

Well, on this basis, take TV watching. TV watching is a great game; easy to learn, tremendously rewarding and highly addictive. Here are the rules: Find something to watch that you believe you would enjoy; watch it; while you are watching find things; be it in the dialogue of a drama or the visual imagery of a documentary and enjoy them.

Here is another one Gardening. Another great game that most people don’t think of as being a game but when you think about it, is. Rules: Try and grow stuff; Reward – a wonderful garden to look at, Gardening is probably one of the most addictive games that is out there for those over 50.

And another – Photography. This is probably one of my own personal favourite games. Rules: Try and take a nice picture! 

The best games are the ones with the most self defining reward mechanics. i.e. I decide on whether or not I am rewarded by what I have done.

Drag & Drop questions: a user guide

Drag & Drop style question formats open up an raft of more creative questioning techniques for researcher, but they can be open to miss-used and many researchers are unsure as to the creative possibilities of what you can do with this style of question. 

This guide covers the factors that should be considered when deciding when and how to use Drag & Drop questions in online surveys, and an outline of the range of Drag & Drop question formats that are available.

Most Thumbed Books

I afraid I have got a terrible habit of turning over the corner of pages of books when I come across something I think is interesting and scribbling notes, my girlfriends mum Alison who is a book collector would be horrified to hear about this I am sorry but, it is quite a good way of telling retrospectively how useful the book is.

This is a list of my top most thumb-nailed/annotated books and I thought this could be the start of a quest to find the most useful books for market researchers to read…

Understanding game mechanics: Which is the best game Twitter, Facebook or LinkedIn?

I have been studying game theory deeply over the last few weeks in my quest to work out how to effectively gamify surveys and I have started to see game play mechanics embedded into all sort of activities.

Just for a bit of fun, well as an invented game in-fact if you look at it this way,  I tried to identify and compare the game play mechanics embedded into Twitter, Facebook and LinkedIn to work out which was the best game.

These are some of the basic mechanics of successful games: 

1.  Strict rules
2. A reward mechanism that delivers both pleasure and emotional satisfaction
3.   A well balanced success/failure ratio - An accomplishable challenge
4.  A broad learning curve
5.   A wider competitive element
6. A balance of luck v skill
7. Levels
8. Addiction factor – is it an absorbing activity in itself
9. Team play

How they score...

Pimp my option choices!

In my mind one of the biggest turnoffs for respondents completing online surveys is the utterly uninspiring language we use to write them with.

I have talked in this blog about the impact that taking a more creative approach to how we ask questions in online surveys and how this can improve the quality of responses.

I would also like to challenge the way we think about using option ranges in surveys too. Particularly the over reliance on tired and unimaginative 5 point likert scales.

To me phrases "like how much do you agree or disagree with this statement” or”” how appealing or unappealing is this and “on a scale of 1 to 10” come across like clichés and do nothing to stimulate our imagination of respondents when they are answering questions, lack any real context, often don't fit the range requirements of the question.


These are the 5 questions you need to ask yourself if you want to design more effective range options:

Questionnaire piloting: a user guide

I were to offer one piece of advice to the market research industry on my death bed it would be "do more piloting!".

We have found that effective piloting really can lead to more efficiently executed research so it amazes me that more research companies have not latched on to the value of doing this.

The reasons for this are I think historical. In the days of pen and paper interviews it was simply not a practical option to pilot a piece of research as it would potentially add weeks to the turn-around time of a project and could end up being prohibitively costly, so it never became an established part of survey design protocols.

However, today with the speed of online research, sketching out a survey and piloting it with just 25-50 respondents is something that can be done in a matter of just a few hours and will add nothing to the overall cost of a project. In fact, it could very well save you money in the long run.

This blog post is a short guide to how you can best use piloting to improve your market research...

ARF Great Minds Quality in Research Award 2011

I am pleased to announce that my team at GMI intereactive has won the inaugural ARF 2011 Great Minds, Quality In Research Award for the work we have been doing with Mintel to re-engineer their online consumer brand tracking surveys.

These surveys were re-designed using QStudio our in house flash survey design platform, as a replacement for  existing HTML survey and we managed to dramatically improve both the consumer experience, reducing drop-out by 50% and also improve the quality of data, increasing average click count by 20%.

We will be publishing a full case study explaining the journey we went through to re-engineer these surveys soon.

But in the mean time my team is basking in the glory of winning this award!

 

Game Theory – turning surveys into games

Gamification is clearly a bit of a buzz word in the marketing industry right now and a subject of growing interest amongst market researcher as witness by the enthusiasm this topic generated at the recent NewMR festival web conference organised by Ray Poynter that I was part of.

This is an article I wrote following on from this conference for MrWeb that I am reproducing  that looks at the work we have been doing to explore the role of game play in online surveys.

The Eureka Experiments

We recently conducted a series of experiments looking at how best to stimulate creative thinking within an online survey environment.  Co-creation is a hot topic out there and moderated think tanking is a well established art.  But within the online survey environment it is a very difficult task to get people to think and act creatively, most rely on co-opting large numbers or profiling groups of creative people to come up with ideas.  We wanted to look at how you could stimulate creativity amongst small groups of regular joe respondents and try and develop some practical technique that our clients could use as part of their every day research.

This is a short report on these research experiments.

Sliders: a user guide

The slider question format is a fantastic tool for comparative decision-making, and respondents enjoy using them in surveys if you don't use them in the right way and don't over load the survey with them. The question commonly arises however, as to how the data from slider questions compare to standard range questions. In fact they have been singled out as one of those sexy question that can deliver dangerous answers.

Well a great deal of detailed research has been undertaken into this topic, and so this is a summary of the main issues and a guide to how to effectively use sliders in your online surveys.

Sexy Questions, Dangerous Results?

At the recent NewMR Festival Bernie Malinoff delivered a presentation highlighting some of this issues surrounding the use of more "sexed up" flash question formats in surveys and have noted that this has stirred up quite a bit of follow up commentary and debate in the last few weeks.

As part of these experiments he showed an example of the negative impact of an over elaborate confusing flash format question can have on data quality and the willingness of respondents to take part in future surveys.

Well I wanted to contribute to this debate and so here are my thoughts...

MR Personality test

Do you want to find out what type of market researcher you are?  Well I am developing a market researcher personality test to explore the character traits of different types of market researchers.

This is a link to a prototype version of the survey  and I would love for as many people as possible to take it to set some benchmark norms and give their feedback on what other question I could ask.

survey link

Once I have enough benchmark data I will build the norms into a feedback mechanic so you can see how you score v the rest of the industry...

The 7 power phrases to boost question responses

1. "We challenge you.." : This phrase seems to trigger peoples competitive spirit and when used in the right way you can often farm 2 or 3 times as many answers out of people.

2. "You have 1 minute..":  It causes a bit of a panic reaction amongst respondent but is a great way of encouraging spontaneous thought and whilst 1 minute does not seem much time it is fact 30 seconds longer that the average time a person naturally spend answering an open ended question in most surveys.

3. " In no more than 5 words...":  I don't know why but people like describing things in a few words like this but they do, its quite fun. Again 5 psychologically sounds like you are asking them to limit themselves, but in reality you are actually extending..if you asked people to list words they would use to describe thing you often only get 2 or 3 so 5 is good!

4. "Every little detail helps..": This phrase can help increase the volume of feedback by 10-20% depending on how you use it - ok a relatively small amount by hey - every little helps!

5.  "Because…": Why do you want to know this... because...you may not think it is important to explain what you are doing or why but respondent are often interested and if you let them into your motives you can encourage more active participation.

6." Imagine…": This word takes you all over the place, can can unlock whole new mindsets amongst respondents.  The more imaginatively use use this word the more imaginative the answers you get .  "Imagine  you had £1000 to spend online buying any stuff you liked in the next hour, what would you do..."

7. "This part is voluntary…": This is one of the most powerful phrases you can use in an online survey. It is counter intuitive to make questions voluntary but it is an extremely powerful piece of behavioural psychology that actually can encourage greater levels of active participation.  We have experimented with do this at the end of a quite boring 20 minute survey for the last section we said  "this last part is voluntary..."  Over 90% agreed to do it, mostly I think out of curiosity and what's we found the they spent 60% more time doing the task  versus a control cell who were just asked to do the task anyway.  The psychology of this is that once you have said yes it is an emotional commitment you feel you have made and we are all socially conditioned to do what we say we will do.

Using Greek rhetoric to improve the level of feedback from your surveys

Building "Ethos"  one of the 3 tenents of Greek rhetoric, is the idea of establishing a bond of trust between you and your audience. Aristotle showed out how important this is when you are trying to get people to believe what you say in a court of law or get someone to do something for you and I want to show how you can use aspects of this thinking to help increase the feedback and responsiveness of your market research.

Welcome to my new blog

This blog has been set up to focus on the science of designing surveys. Looking at the best ways to engage and communicate with survey respondents,  how to generate effective feedback from your surveys and exploring new ideas and techniques surrounding online research.

I have set it up with a few articles and obviously hope to build on the content in the coming weeks.  I do welcome any form of feedback, ideas or suggestions for future articles. I hope you find the content of some value.

The inspiration for this has been many of the fantastic market research blogs out their that I have learnt from and enjoyed.  I am a great believer in the open source dissemination of ideas that will help improve the working practices of the industry and I see this as my small contribution to this process.