England are going to win the World Cup in 2018: A Market Researchers prediction!

This is an blog post I first published after the 2014 world cup, that I am republishing to herald our expected victory next year! 

IN 2014 England got dumped out in the first round of the World Cup and everyone in our country was disappointed, an emotion we are quite used to feeling. After every world cup failure begins a round of postmortems that we all probably secretly enjoy as much as the competition itself, working out who to blame for the team’s failure.  In past World Cups this has been quite easy: for example David Beckham kicking a player and getting sent off, having a turnip head as a manager or a lack of goal line technology.  But in 2014 it was a little difficult to work out who was to blame. I read a lot of overfit analysis, none of which is particularly convincing because, well, in the scheme of things the sparky young team played quite well. It seems like we were just a bit unlucky last time round.

The role of randomness

Its quite hard to accept the role that randomness plays in the outcome of world cup matches.  Every nation when they get kicked out or fail to even qualify probably believes their teams were "unlucky" and that their teams are better than they actually are.  So what is the relative importance of  luck v skill when it come to winning the world cup?

Unlike the premiership where there are 38 games over which time the performance of the teams is largely correlated to the quality of the squads (take a read of the Number Game* by Chris Anderson and David Sally)  performance of a world cup squad cannot be calculated by the aggregated skill value of the squad there is a lot more randomness involved.  Imagine if the premiership only lasted 3 games: in two out of the last four seasons the team that won the premiership might have been relegated.

*a must read if you are a market researcher and like football!

There is another factor too, in the premiership the best players get sucked up into the best teams hence the much higher win ratios between the top and bottom performing sides compared to the world cup where the best players are distributed more randomly and is proportional to the size of each footballing nation.  This in tern makes the outcome of international matches even more random.

Who influences the outcome of a match?

If you look at who has goal scoring influence across a team you will notice that the negative effects of causing goals a pretty well distributed across a team but the positive effects of scoring goals are a lot more clustered among some individuals. See chart below showing statistics from an imaginary team based on typical performance data taken from the premier league.

 

The potential performance of a world cup team must be measured not by the overall skill value of the team but the value of a smaller network of attacking based players who can make the most game changing contributions. In the case of players like Lionel Messi a single player can carry the whole goal scoring burden of a squad.  It only takes one or two randomly allocated star players in a world cup team to elevate its performance chances (think of Pele or Maradonna).

The performance of defence is more a case of luck. You might have one or two unreliable defenders who you may not want in your premier league squad because you know over the course of a season they may cost you a match or two, but at the individual game level and a world cup is based on the outcome of  three or four key individual games, the chances are a poor defender might well run their luck.   The other two important factors defenders have to contend with are the extra stress and lack of team playing experience of a world cup team compared to a premiership squad.  Without doubt stress plays a big part, players are really hyped up and there is probably an order of magnitude increase in tension which is the root cause of many errors in world cup matches. If you look at the defensive mistakes that cost us goals in recent world cups some of the biggest mistakes were caused by effectively our most reliable players, John Terry and Steven Gerrard and Phil Neville.  There is also a lack of formation practice to contend which is particularly critical for defence. How many hours of playing together does it take for a defence to gel? Most world cup squads have days rather than months to prepare.

A team like England might well have a higher aggregated skill performance average compared to other teams, but this does not result in the same reliable performance ratios that you see in the premiership. This is because over half the value is based on their defensive skill which can be completely undermined by bad luck and we don’t have a cluster of super-skilled players to elevate the team out of bad luck matches by scoring more goals than we let in.

The influence of the Ref

To win world cup matches you are much more reliant on the manager’s structural approach, the contributions from clusters of individuals who might form good attacking combinations and one other person – the REF!  Or rather, the ref in conjunction with the crowd and the linesmen.

If you analyse a typical game you will find that the number of major goal scoring decisions that are in the hands of the referee and linesmen are actually enormous compared to any individual player. It’s difficult to put a figure on it but let’s say on average there are about 6 decisions that could have affected a goal one way or another by the referee* its instantly obvious the relative influence they have on a match.

*That is a wisdom of the crowd estimate by asking a collection of football fans how many goal-affecting decisions are made in the match by the referee and linesmen, six was the median average estimate.

Now in nine times out of ten these decisions balance themselves out but refs are only human and so it’s no wonder why there is such a big home team advantage – with 50,000 fans screaming penalty it must be extremely difficult for refs not to be influenced by the crowd.  In fact you can almost put a figure on the influence of the crowd by comparing home and away goal scoring averages the home side gains an average 0.35 of a goal per game net advantage if you examine premiership games,  which can only be really down to the net contribution of the crowd/ref decision effects.

It’s no wonder as a result that there is such a disproportionate home nation advantage.  Effectively every home nation team is starting with a 0.35 goal lead, this advantage aggregated up over the course of a tournament  has means that nearly 30% of all world cups have been won by the home nation that is 10 times higher than chance.

Am I likely to ever see England win another world cup in my lifetime?

Is probably a question most England fans ask themselves. What does it take to win a world cup – how good do you have to be to override luck?  We have taken a look at this and run some calculations.

The chart below take a little explaining but it maps out a team’s skill level v the number of times it’s likely to win a world cup over the course of someone’s average football supporter’s lifetime of 72 years = 18 world cups.  If there are 32 teams in a word cup and you are an average team and your team qualifies for every world cup final the chances are you will win 1.1 world cups over your lifetime. If in you are England and only qualify roughly 80% of the time the changes will drop to 0.96.  If your team is twice as good as average, you are likely to win roughly 2 world cups and 4 times better 6 world cups.

 England have one one world cup, Germany three and Brazil five so does that mean we are average team and Germany are three times better than us and Brazil four times better than us?

Well essentially yes, if you look at the average game-win ratios of all the teams that have played the most regularly in World Cups v the number of World Cups they have won its pretty closely correlated at 0.91.    Germany has a three times higher win ratio than us and Brazil four times higher.

Now I appreciate there is some self-selection involved here – this chart should really be based on first round matches only for a totally fair comparison, but we don’t have that data. I think it’s reasonable to say though that England has not really been done out of its fair share of World Cups.  I think we have won as many as our teams aggregated performance deserves.  You might argue that some teams have been luckier than most: Italy certainly and others unlucky, Mexico should have won it twice by
now based on their aggregated performance.

A victory every 50 years

Doing the maths, based on England average win ratio, we should win a world cup roughly once every 50 years.  Now as England last won the world cup in 1966 so in 2018 it will be 52 years since we won, I think that mean we are now officially due a victory this time round does it not?

How does a polling company find out how many followers the anti-research party has?

A conundrum for you...

Imagine I have set up a completely new political party and in my manifesto I tell my followers not to trust the polls and to slam down the phone on any polling company that tried to call and not answer any surveys.

My party is now effectively invisible to researchers!

How does a polling company work out how many followers this new party has?

Could this phenomenon go to explain the massive miss read in the US election polls?

...Trump painted polling companies as the enemy, it is no wonder some of his followers as results might have refused to engage with them and as a result the polls end up with a hole in their numbers.

This is the conundrum market researchers have to face up to if they want to get to grips with political polling in the future.

We need to find a way of measuring the opinions of those that hide away from expressing their opinions.

My mum spotted a nice solution, at her local church fete during the Brexit campagn a stall was selling 2 varieties of rhubarb, "in" and "out" they sold 26 bundles on in and 28 bundles of out! Job done. 

Here is what just happened

I am writing this on the morning after the US election so I don't think I need to explain the headline....

I am sure there will be more sophisticated explanations emerge over the following day but this is my take to understand the result of the election.

In the run up to the US election out team have been conducting an ongoing series of political polling experiments to understand what has been going on. We have not been polling in a conventional sense, more experimenting with how to measure voting sentiments and voting intention in an attempt to find a better way of predicting the outcome of tight elections.

We have been asking a lot more indirect question about the difficulties people were having in making up their mind, exploring more implicit measures of voter sentiment to see how they stack up with declared measures, getting people to play games to predict the outcome to reveal some of their hidden feelings and we have also focused a lot of attention on asking open ended questions to measure the level of passion in the arguments and look at what reasons people have been using to make their choices. 

Here is what I think has just happened from my perspective from the learnings from all this research.

Trump messaging was far stronger than Hillarys from the get go. It was more coherent. Make American great again. Close the borders. Drain the swamp, Hillary Clinton is corrupt.  We were seeing all this being echoed back over and over again to us in the explanations gave us for why people wanted to vote for him. 

On the other hand Clintons messaging was, extremely weak, in fact almost none of it seemed to stick. Less than 10% I estimate of the reasons cited for voting for Clinton were anything to do with liking her policies, it was nearly all to do with stopping Trump winning. So many people caveated their choice with an explanation that they were picking the lesser of 2 evils.

All the implicit candidate favourability measures we undertook showed us how much Clinton grated on the American public. Flashing pictures of her face solicited up to a -60% negative reaction, even worse that Trump who is known as being a pantomime villain.  Here face and perhaps more importantly her voice did not fit. The majority of American public implicitly did not warm to her. 

What Trump was up against in this campaign was not Clinton, but Trump himself and, well let's not beat about the bush here, his personal sociopathic character traits.

The Trump sexual harassment scandal embodied all this and the misgivings so many people were having about handing him power seriously pegged back Trumps latent momentum in the final month of the campaign as all this news broke out. 

But beneath all this, he was a lot of people's implicit preference,  but they could not express that in opinion polls or even to themselves to  that matter due to the outrage being voiced in the media about his behaviour, but his core message resonated.

Then comes along the FBI email investigation a week or so before the election. This was like pulling out literally a “Trump” card.  What is did was give all the people who latently liked his messaging but were suffering from cognitive dissonance over his character, a strong emotional counter argument for prefering him it gave them something that they could dress up as a lot more significant, it validated all his messaging about his opponent too. It could not have been more perfectly timed.

We  actually saw the change happening in real time as we conducted a large scale research experiment the week before the FBI press release so on the Monday after this we did a follow up piece of research to see what was happening and  Clinton's 7 point lead we had seen the week before literally evaporated overnight. 

The shy Trump supporters were released from the closet so to speak and at the same time all the people who didn't like Trump or Clinton were given a strong reason not to vote for either candidate or stay at home and not vote at all. 

The last minute FBI volte-face was far too late in the day to undo any of this.

What we have learnt from all this that the public's opinion on how they will vote is very emotional process,  similar to the way my daughter makes up her mind about what type of curry she wants to order in our local curry house. She tries on several  choices to see how she feel about them thinks she has made up her mind, changes it, changes it again and then ditches them at the last minute when the waiter is standing over her and decides with 

her gut instinct.

In this case the US gut fancied trying something new, despite some serious misgivings  because the other dish did not seem too appetizing at all they decided to roll the dice.

You can chide the pollsters if you like, but this type of emotionally charged election is almost impossible to predict even a day or so out but certainly its clear you cannot predict an election by simply asking people which way they intend to vote.  

Here are a few other closing thought about why the polls got it wrong....

1. Are the types of people that slam down the phone if a pollster calls and would never think about doing an online survey also the same type of people who might be more likely to have voted for Trump?

3. Clearly Hillary Clinton did not motivate here democratic base to vote in the same way Trump rallied his supporters and so the polls which are often weighted by past voting activity were delivering a miss read

2. Likewise did weighting polls by traditional political allegiances have any relevance in this election

4. Male blue collar workers who voted for Trump in droves are the hardest group to reach with research as they are working

5. There is some evidence in our research that Trump supporters were slower to respond to our online poll invitations and so some short turn around polls might have closed up before all the Trump supports had a chance to register their opinion.

10 things I learn in 2014

1. We convert statistics into emotions: and so the best way to fast track getting your statistics remembered is to emotionalise them!
2. Our brains are Bayesian decision making engines: by and large designed to work out what choices will make us most happy
3. A question is a problem that you ask respondents to solve:  it is easy to lose sight of this simple thought. Often we design questionnaires that skirt around the problem we are trying to solve. We ask questions so euphemistically,  we ask questions that are a Chinese whisper away from what we are trying to find out. 
4. We like to think in different ways:  Researchers like to quantify things, particularly types of people and how they think and consumer.  We have personalities that get classified and we are this type of person or that. The simple fact is that we are all sorts of different types of people depending on the time of day our mood and our circumstance. We all like to think in different ways, its boring the make the same types of decision especially when we go shopping.  the concept of "type" in research is limiting. The same person who liked to try out new types of shampoo  
5. Scale effects: you scale something up and sometimes different maths starts to apply
6. If there are infinite number of universes,  then in some universes it is certain that a god will exist (as some of us know it) ..and in others it is certain that a god as we know it will not exist:  A nice thought, that's assuming that there is such a thing as infinity, some physics question this too ... it might, but there is an infinitesimal small chance!
7. Rating something is inherently a system 2 thinking process:   compared to a binary choice process which is more system 1. The example being you come out of a film your friend asked did you like it and in a fraction of a second you can say yes or no, but likewise if the friend asked you to rate the film this takes more mental processing and anything up to 5 seconds thinking to give it a score.
8. 16 is a crowd:  Prediction as opposed to market research is not about the numbers of people you ask its about the quality of information available to the group of predictors and their effort and objectivity.  
9. Unwise crowds:  crown wisdom is a nice idea but it only work in certain rather rare circumstances. Crowd predictions are mostly corrupted by system errors & network cognitive biases. 
10. Computers using artificial intelligence can now read some of our emotions better than we can:  Artificial intelligence tools are getting so good at reading our emotions by combining various input data sources ranging from how we are typing, the language we are using and the music we are listening to that they can identify traits of depression often months before it is apparent to ourselves

Meeting Jargon in heard in 2014

These are some of the wonderful pieces of  jargon I heard and noted in various meeting I attended this year...enjoy (and forgive me, I have probably use most of these myself!)

• Boil the ocean 

• Spin our wheels 

• Capability area 

• Landing it 

• Resonate with...

• Next steps

• Gun up

• Creating confusion 

• Lense (wide angle /focused/zoom)

• Data science unicorns 

• The unsexy stuff 

• Slower pace 

• Focus in on... 

• Support of the board 

• Additional resources 

• Executions logistics 

• Treat as a priority 

• Alignment 

• Asking for permission not asking for permission 

• How we are going to scale it 

• Catching lightening in a bottle 

• CDO 

• UI

• Mathmagicians 

• Tentpoling 

• Lumascape 

• Ecosystem 

• Insights into actions 

• Data at the centre... 

• Making a brand more culturally relevant 

• Triangular across the data 

• Effecasy 

• The mandate 

• Manadated 

• Implementation 

• A hard stop 

• The pillars 

• The lead sled dog 

• Skin in the game 

• In the new world 

• Analysis paralysis 

• 90% of data created in the last 2 years

• Experience based economy 

• Product service experience 

• Ubiquitous computing

• Ambient intelligence

• Embedded

• Network 

• Internet of things 

• Context aware 

• Sensor fusion 

• Personalised behavioural profiling 

• Adaptive ai learning 

• Anticipatory predictive analysis 

• The Google car crash dilemma 

• Cut off data cut off blood 

• Hyper personalisation

If you have any nominations do tweet them #mrjargon

2014 market research book list

Coming to the end of the year, I I thought I would share a list of the best books I have read in 2014 that I think other market researchers might like to read.  Now not all of these are new books by any means so forgive me if you have yourself read half of them.

This will make you smarter

John Brockman

This book is a compendium of scientific and philosophical ideas in one of two page essays on quite a remarkable cross section of topics. There are some really exciting thought packed into this book that I think market researcher could make good use of. I think reading it really did make me a little smarter!

Expert Political Judgment: How Good Is It? How Can We Know?

Philip E. Tetlock

Philip Tetlock's thinking has had more influence on how I think about conducting market research than any one person this year. I was introduced to this book by Walker Smith from the Futures Company and I would recommend that anyone who has an interest in the science of prediction should read this book.  Learn that political experts are not quite as good as chimps tossing coins at predicting things!

The Signal and the Noise: The Art and Science of Prediction

Nate Silver

I realise this book is a few years old now, and I wish I had read it sooner. There are so many really important ideas stuffed into this book that market researcher can use in their every day research. Its both inspiring and useful.

Strategy: A History

Sir Lawrence Freedman

This small thumbnail belies a bloody thick book which I have to admit to not to have read every page of.  It looks at strategy from all sorts of angles from war through to politics and summarizes the thinking of every major strategist in history including the likes of Sun Tzu, Napoleon and Machiavelli.  There is loads of great thinking for market researchers to digest. And probably even more valuable incites for anyone running a business.   It contains a detailed look game theory and the trials and issues with trying to apply strategy in real life. There is some sage advice in this book

Decoded: The Science Behind Why We Buy

Phil Barden

This book is a really helps explain the basics of shopping decision making and is a compendium of behavioral economic theory, an important topic for nearly all market researchers to understand - I really like the way it uses visual examples to explain some of the theory making it an effortless read. This book should be on every market researchers shelf.

100 Things Every Designer Needs to Know about People

Susan Weinschenk 

This book should really be titled, 100 things market researchers designing surveys and presentations should know about people!  ...And everyone involved in either of these task encouraged to read this.   Loads and loads of really clear, sensible advice.

The Curve: Turning Followers into Superfans

Nicholas Lovell

I read this after reading a very enthusastic linkedin review by Ray Poynter, thank you!  It persuaded me to buy it. There are some nice radical ideas in here about how to market things by giving things away and at the same at the other end of the scale offering premium high price solutions for the those willing to pay for them.

The Numbers Game: Why Everything You Know About Football is Wrong

Chris Anderson (Author), David Sally (Author)

I rather immersed myself in reading sports stats books this year. The way that data is transforming sporting strategy, there are lessons to be learnt by the whole of the market research industry. As an English person with a love of football, I feel rather a bounden duty to promote the Numbers game which looks at how statistical data has changed how the game is played. I loved this book and I am afraid I bored senseless everyone I knew who had any interest in football quoting incites from it. I also read Money Ball this year too which is the classic opus on how a proper understanding of stats transformed the fortunes of a major league baseball team, it is a great story and lovely read.

Who owns the future?

Jaron Lanier

This book has an important message about the impact of the digital economy on our future I cite from the book directly as it best explains  "In the past, a revolution in production, such as the industrial revolution, generally increased the wealth and freedom of people. The digital revolution we are living through is different. Instead of leaving a greater number of us in excellent financial health, the effect of digital"  Worth a read!

The golden rules of Acting 

Andy Nyman

This is a lovely little book, you can read in one short sitting. Why though do I recommend market researchers read it?  Well not because it teaches you anything about acting more about life and humanity and dealing with failure and the right approach to challenges.  There is not much difference in my mind to going for an audition and going and doing a pitch presentation. I took some heart from reading this book.






Want to see some other book recommendations?  Try this site:

http://inspirationalshit.com/booklist#

Your 2015 recommendations?

Love to hear your recommendations for books I might read in 2015  tweet me @jonpuleston

Questions on trial

Surveys are competing with a billion+ largely more fun things to do online these days and we are reaching tipping point were many people, the young age groups in particular are simply refusing to complete survey because they are too boring.

To have any chance of competing we have to change our approach and I think this starts with taking a long hard look at some of the boring questions we are asking in our surveys.

One of them being this one...

"What brands are you aware of?"

It a question asked in nearly every consumer survey I come across, usually asked both unprompted with an open ended question and then prompted with a closed question set of brand options (twice the work). Its one of those sacred cow set of questions that everyone insists on asking.

Do we actually need to keep asking this question?  Are there not better questions that could be asked in better ways, that make better use of a respondents brain that deliver more useful data?

Here is the case for the prosecution:

• Its a really dull question: From a respondents point of view these are probably the dullest most cliched question they continually have to answer in surveys. Respondents don't like answering these questions,  they trigger drop out. 
• Respondents put little thought into their answers: Less than half the respondents you ask this question name more than one or two brand when asked unprompted and well over 20% say don't know. The prompted question often gathers together a random set of clicks from respondents.  
• Little statistical value:  If the average respondent list 1 or 2 brand and assuming the number of individual brands listed adhere to a Zipf's law style distribution, the most popular brands named much more often than the least popular brands - on a typical sample of 400 respondents there will only likely to be 1 or 2 brands you ever have enough data to work with statistically. 
• Large error boundaries:  The data error boundaries on this question on most survey samples are of often so large that in wave to wave in brand tracking studies the fluctuations down to pure statistical error are often of an order of magnitude larger than the actual underlying change in brand awareness.  Resulting in a lot of "overfit"
• No stand alone value: The data it delivers back is almost always duplicated or can effectively be modeled from answers given to other questions in the same survey.
• Meaningless metric: Brand awareness information is totally useless in isolation from anything else. What it measures is intangible it's certainly not an accurate measure of purchase behavior for example - Unprompted awareness correlate at around about 0.54 with purchase behavior*, prompted awareness it drops to under 0.2*.  If you specifically want to find out what brand consumers are likely to buy there are other question that are far more effective.  It is also not a measure of how much I may like a brand....
• I have never see how this information is really used profitably:  It's always the chart that everyone skips past in a presentation. 

The case for the defense:  

Now I have challenged several prominent and respected researchers many of whom are still very wedded to asking this question in survey and asked them why they like to use it.

• Its a fundamentally important measure: The brand that is mentioned first is the brand that has the most brain neuron connections and associations between the product and the category. So many see it as fundamentally the most important question you can ever ask in a survey.

Now I get this but I am still left with a feeling of so what....if you ask me what brand of chocolate I am aware of  - I will say Cadbury's.  I have been exposed to this brand all my life seen thousands of Cadbury's ads, seen the brand in every confectionery counter I have ever visited.   But I never buy Cadbury's and so what x number of people are aware of a brand.

Now I am open to some other arguments as to why it should be kept if anyone wants to make them, but my judgement verdict is that its a question that if not completely banned, should at least be taxed, in the same way that cigarettes and alcohol are taxed to discourage their usage.

What could be asked instead?

There are a range of alternative ways of directly or indirectly measuring brand awareness that are more interesting and potentially useful.

I find one of the frustrations to answering the what brands  do you recall question is thinking why does it matter and not knowing how many to list, Simply applying a rule to the question that contains the task it in a more meaningful framework for respondents can make it far less dull to answer.

you could ask:
...what are their favorite brands
...which brands they would have in their perfect supermarket
...which brand, if they could only buy one, would they choose to buy a life long supply of... ...which brands they would take to the desert island
...which brand they would invite to a party
...which brands they would recommend to their best friends
...which brand they would invest in
...which brands they think they will still be buying in 5 years time

All these will gather top of mind awareness to some degree or other but are more interesting and purposeful for the respondents to answer and adding in these "rule" can make them more salience and relevance e.g. asking about what brands they would have in their perfect supermarket  it not just just a measures awareness but also intent to purchase.

In head to head experiments we have found we get more responses to these more conceptually fun questions too.

You could turn it into a full blown game by adding to your list of brands some fake ones and challenge respondents to pick out the real brands.  We have used this approach on several occasions, its more fun for respondents and the data you get back is almost identical to prompted brand awareness.  You could also show them a facet of the packaging of a brand and see if they can guess which brand its for, or show them a de-branded ad.

I would also advise jumping straight past the specific awareness question and asking what range of brands they have purchased on there last 10 shopping occasions. The has very high correlation with actual sales c0.8 and again cuts to the heart of the problem, if they are aware of it but not purchased it in any of their last 10 shopping occasions its not on their purchase radar!

If you want to defend the use of this question in your surveys and have got a strong argument for doing so i would love to hear your thoughts.

How I feel is how I choose: A Gedankenexperiment

As market researchers we like to classify people and in particular we like to classify how people make decisions. However, we have a dreadful habit of thinking that there are different types of people who think and make decisions in these different ways.  We define segments like loyalists and switchers, impulse v considered shoppers.

The reality is that how we feel so often dictates how we choose; and our ever changing mood states mean we are all manner of different types of shopper all rolled into one.

We might have...our stress choice, our curious choice, our distracted choice, our ‘I've just been paid’ choice, our practical choices, our happy choices, our don't care choices etc....

The process of observing things changes things

All these different thinking process are all bundled together and co-exist in a quantum style mix - we are all or nothing of these or a random mix at any one point in time. It’s difficult to determine, and - just as Schrodinger pointed out - just the process of observing changes things.  You ask someone why they have purchased something and it immediately puts people into an analytical thinking framework, far removed from the mind state they might have been in when they made the purchase.

The biggest problem for market researchers trying to understand decision making processes is how to simulate these different moods to be able to effectively measure these different choice situations and how to take account of observational biasing effects.

Some mental states are easier to evoke than others.  The price conscious mood state and the impulse mood state are actually quite easy to simulate in say an online piece of research. You just ask people to either shop with a price conscious budget or ask them to perform the choices quickly.

Some of the others, though, are a lot harder to evoke. For example our distracted shopping mind-set is an important one, as when we go shopping it’s likely that we will often get distracted and have other things on our minds while we do it e.g. I am having a relationship crisis and as I go round the supermarket I am thinking mostly about that and making some choices on auto pilot.  What are these choices like? Are they the same as when we are, say, in a hurry? I think not. The choices would probably be the more habitual ones and could perhaps be more reward driven and may also be quite impulsive.

How you evoke these mind-set in a survey is a difficult one.  If you ask people why they bought something their rational thinking processes kick in and what you get out is in effective cognitive dissonance - we can shape our reasons for purchasing around what makes us feel good so often.  

Don't get me wrong; this is useful information in itself. But it can hide other less conscious factors that for many marketers are the factors we are most interested in.

e.g. to say “I bought it because it was cheap” could hide some resentment to the purchase. In a sense this statement is an excuse for buying it or hide some guilty feelings about buying a brand that is viewed as extravagant.

We are Bayesian decision making engines working out what would make us happy

It has been observed that when we buy things we make a Bayesian trade off prediction about what will make us most happy. On one side might be the price, how long a product will last and the risks involved in making that choice, on the other side are the benefits that the product delivers.

So to try and untangle some of this, I have been thinking about a new research technique. A form of self-observed Choice ethnography, where we get people to try to mentally map out all the different thinking protocols involved in making a decision for different products.  A sort of helicopter viewpoint on their own behaviour, the sort of thinking that takes place when you sit in front of a therapist and really try to self-observe your own thoughts and feelings.

To test this idea out, I conducted a Gedankenexperiment, a thought experiment on myself to see how easy it was to self-observe my decision making protocols.

What I found was that the more I thought about it, the easier and more interesting a process it became….

My Gedankenexperiment

I started to observe my own behaviour and traced out my decision making on a variety of different types of purchases. 

I started out by imagining my choice of beer in a pub, thinking about it. Here are some of the potential factors that I decided were influential in my choice:

What I fancy,

What choices they have available

What others are drinking

What I had last time

How much I had already drunk.

Interestingly, when I examined my decision making behaviour in a pub, the first thing I learned, was that it was never about what it costs.

Even more interestingly I realized that when I go into my two local pubs,  in each one I am a completely different type of shopper: In one, perhaps because it is more conventional with a limited choice of beer, I always buy exactly the same brand of German lager. I never vary my choice of drink ever...I thought about it and realized I am buying instant relaxation. That brand has been ingrained into the experience.  

In the other pub, which has a fuller range of craft beers, I never, ever buy the same beer twice and certainly never ever buy lager.  I take pleasure out of trying different things and so I move from one pub being beer monogamous to another where I am completely polygamous.

I then started to think about my choice of a beer in a supermarket. These are the criteria that I process in my mind:

What they cost

What they have

What l like

What I have not tried before

What looks nice

In the supermarket the look of the beer became paramount compared to the pub where it was the taste. Additionally, in the supermarket cost suddenly becomes near the top of the list.

Then I thought about my choice of wine in supermarkets...

A weird thing about my wine shopping is that I realise I decide based on trying to game the system as here there is actually too much choice... what I do is I look at all the brands on discount and try to find the ugliest bottle. My reasoning is that is probably the best tasting wine but it’s on discount because nobody wanted to buy it because of the ugly packaging.

Next, I thought about my choice of shampoo. This boils down to:

What they have

What it costs 

What I can bear to buy

I end up picking the cheapest brand that has the least offensive packaging. But it often takes several minutes to decide.

Compared to choice of deodorant & toothpaste….do they have Dove/Colgate, is it on offer, yes but my decision making time is seconds. In both these categories I have made up my mind as to what brand to buy and have stopped deciding.

This went on and I examined my thinking process across a wide range of product purchases, trying as best as possible to observe some of the less conscious factors. 

I realise, for example, that my choice of confectionary was triggered often by childhood associations.  would literally buy a chocolate bar to try and feel like I felt in a situation when I was younger.

What have I learnt about thinking about my own behaviour?

1.    How dramatically varied it is, I could be described as both a loyalist and a switcher. From a market research point of view it would be almost impossible to classify me as one thing – and I am sure the same could be said for anyone.

2.    I realise I was able - by simply observing my own thinking process - to gather a range of personal “incites”

• I buy old things in new places where I don't feel comfortable
• In new places where I feel comfortable I buy new things
• In familiar places I buy new things to break out of the routine sometimes
• I buy a limited portfolio of things if I am unsatisfied with the portfolio on offer generally but hold resentments to the products I buy and so am not loyal
• I am an extremely loyal purchaser once I have made up my mind about what is the best product
• When I have not made up my mind I vacillate
• When there are more than one product I like in a category I sleep around, so to speak, and can be very promiscuous (wine is a good example)
• When I am buying something from a category for the first time aesthetics control so much of my decision making protocols - what it looks like is key and I rely on design cue that basically say that product understands me
• When I don't like any of the packaging this is when I become an instantly disgruntled shopper

3.    The process, once I got into it, was fun, easy and quite cathartic, and I am sure with a bit of explanation anyone could use this same process easily on themselves, but it does take time and thought.

I am left with the thought that this would be a very interesting process to do on a larger scale, so that is what I am looking at doing next….watch this space.

The science of prediction

This blog post is a short introduction to the science of prediction which is a topic that I have been totallt immersed in over the last new months and recently presented about at the 2014 ESOMAR Congress with Hubertus Hofkirchner. I thought I would share some of what I have learnt.


The accuracy of any prediction is based roughly around this formula...

P Accuracy = Quality of information x Effort put into making the prediction x (1 - difficulty of accurately aggregating all the dependent variables) x The level of Objectivity with which you can do this  x The pure randomness of the event

P = QxEx(1-D)xOxR

Here is the thinking behind this:

• If you have none of the right information your prediction will be unreliable
• If you don't put any effort into processing the information your prediction may be be unreliable
• The more complex a task it is to weigh up and analyse the information need to make a  prediction the less likely that the prediction will be correct
• Unless you stand back from the prediction and look at things objectively then your prediction could be subject to biases which to lead to you making an inaccurate prediction 
• Ultimately prediction accuracy is capped by the randomness of the event. For example predicting the outcome of tossing a coin 1 time v 10,000 times have  completely different levels of prediction reliability.

Realize that prediction accuracy is not directly linked to sample size

You might note as a market researcher, that this formula is not directly dependent on sample size i.e. one person with, access to the right information, who is prepared to put in enough effort, has the skills needed to process this data and is able to remain completely objective, can make as good a predictions as a global network of market research company interviewing millions of people on the same subject! I cite as an example of this Nate Silver's achievement of single handedly predicting all 52 US State 2012 election results.

Now obviously we are not all as smart as Nate Silver, we don't have access to as much information, few of us would be prepared to put in the same amount of effort and many of us many not be able to process up this information as objectively.

So it does help to have more than 1 person involved to ensure that the errors caused by one persons lack of info or another person lack of effort or objectivity can be accounted for.

So how many people do you need to make a prediction?

Now this is a good question, the answer obviously is that it depends.

It firstly depends on how much expertise the people making a prediction have on the subject individually and how much effort they are prepared to make. If they all know their stuff or are prepared to do some research and put some thought into it, then you need a lot less than you might think.

16 seems to be about the idea size of an active intelligent prediction group

In 2007, Jed Christiansen of the University of Buckingham took a look. He used a future event with very little general coverage and impact, rowing competitions, and asked participants to predict the winners. A daunting task, as there are no clever pundits airing their opinions in press, like in soccer. However Christiansen recruited his participant pool from the teams and their (smallish) fan base through a rowing community website, in other words, he found experts. He found that the magic number was as little as “16”. Markets with 16 traders or more were well-calibrated, below that number prices could not be driven far enough.

The Iowa Electronic Market, which is probably the most famous of prediction systems out there that has successfully been used to predicted over 600 elections, has I understand involved an average of less than 20 traders per prediction.

Taking account of ignorance

However for every one completely ignorant person you add into the mix who effectively makes a random prediction you will instantly start to corrupt the prediction.  And in many situations this is scarcity of experts means to isolate ignorant and expert predictions this often means you need to interview a lot more people than 16.

Take for example trying to predict tomorrows weather. Imagine that 10% of the people you ask have seen the weather forecast and know it will not rain - these could be described as the experts and the rest simply guess 50% guessing it will rain and 50% not its easy to understand that if by chance more than 10% of the random sample predict it will rain, which is entirely possible the group prediction will be wrong.   Run the maths and for 95% certainty you will need to have a margin of error of less than 10% to be confident which means you will have to ask 86 people.

It gets even harder if the experts themselves are somewhat split in their opinions.  Say for example you were trying to predict who will win a tennis match and 10% of the sample are you ask are keen tennis fans (experts) who predict 2:1 that player A will win, the rest randomly guess 50% player A 50% player B.  Because of division in the experts you now need to a margin of error of less that 7% to be 95% confident which means you will need to interview around 200 people.

Taking account of cognitive bias

It gets even harder if you start to take into account cognitive biases of the random sample.  For example just by asking whether you think it will rain tomorrow more people will randomly say yes than no because of latent acquiescence bias.  We have tested this out in experiments for example if you ask people to predict how many wine drinkers prefer red wine the prediction will be 54%, if you ask people to predict how many wine drinkers prefer white wine the number of people who select red wine drops to 46%.   So its easy to see how this cognitive bias like this make predicting things difficult .

In the above example predicting the weather this effect would instantly cancel out the opinions of the experts and no matter how many people  you interviewed you would never be able to get an accurate weather forecast prediction from the crowd unless you accounted for this bias.

This is just one of a number of biases that impact on the accuracy of our predictions, one of the worse being our emotions.

Asking a Manchester United football fan to predict the result of their teams match is nye on useless as it almost impossible for them to envisage losing a match due to their emotional attachment to the team.

This makes political predictions particularly difficult.

Prediction biases can be introduced simply as a result of how you ask the question

Imagine I were doing some research to get people to predict how often when a  coin is tossed it is heads and I asked the question "If I toss this coin, predict if it will be heads or tails" for the reasons explained above the on average around 68% of people will say heads. The question has been asked in a stupid way so it delivers back a wildly inaccurate aggregated prediction.  If you change the question to "If a coin were tossed 10,000 times, predict how often it would be heads" you probably need no more than a sample of 1 to get an accurate prediction.   Now this might sound obvious, but this issue sits at the route of many inaccurate predictions in market research.

Imagine you asked 15 people to predict the "% chance" of it raining tomorrows and 5 of them happen to have seen the forecast and know there is a 20% chance of rain and the rest randomly guess numbers between 0% and 100%. If their average random guess is 50%,  this will then push up the average prediction to 40% rain.  If there is the same mix of informed in non informed predictors in your sample like this, it does not matter how many more people you interview the average prediction accuracy will never improve and will always be out by 20%.

This runs very much counter to how we tend to think about things in market research, where its nearly all about gathering large robust samples.  In the world of prediction, its all about isolating experts and making calibrations to take account of biases.

The stupid way we ask question often in second hand ways we ask questions can exacerbate this.

"Do you like this ad" for example is not the same question as whether you think its going to be a successful ad. The question is a Chinese whisper away from what you want to know.

A successful ad is not an ad I like its an ad that lots of people will like.  Change the question and motivate the participants to really think and we have found to make a perfect prediction about the success of an ad samples drop from around 150 to as low as 40.

Picking the right aggregation process

The basics

Imagine you were asking people to predict the trading price of a product and a sample of  predictions from participants looks like this.

$1, $1.2,  $0.9, $0.89, $1.1,  $0.99, $0.01, $1.13,  $0.7,  $10,000  

Your Mean = $1,000  .....Whoopse that joker putting in $10k really messed up our prediction.

Now for this reason you cannot use mean averages. For basic prediction aggregation we recommend using Median.  The median average of these is = $1 which looks a lot more sensible.

An alternative might be to simply discard the"outliers" and use all the data that look sensible.  In this example its the $0.01 and the $10,000 that look out of sync with the rest removing these the medium average = £1.03 which seems a bit more precise

Weighting individual predictions

The importance of measuring prediction confidence

In the world of prediction its all about  working out how to differentiate the good and bad predictors and one of the simplest techniques to do this is simply to ask people how confident they are in their prediction.

For example if I had watched the weather forecast I would be a lot more confident in predicting tomorrows weather that if I had not.  So it would be sensible when asking people to predict tomorrows weather to ask them if they had seen the weather forecast and how confident they were. From this information you could easily isolate out the "signal" from the "noise"

The trick is with all prediction protocols to try and find a way of isolating the people that are better informed than others and better at objectively analyzing that information but in most cases its not as easy as asking if they have seen the weather forecast.

For more complex predictions like predicting the result of a sports match, prediction confidence and prediction accuracy is not a direct linear relationship but certainly confidence weighting can help but needs to be carefully calibrated.  How you go about this it a topic for another blog post.

In the mean time if you are interested in finding out more about prediction science read our recently published  ESOMAR paper titled Predicting the future