Clean Data In A Messy World | by Abubakar Mohamed
As Heraclitus once said, “Nothing is permanent except change”. The world is full of changing uncertainties. A virus in bats spreading to humans and causing a worldwide pandemic. Changes in weather patterns, animal migrations, and rising temperatures around the world.
Simultaneously we are being sold a digitised future. Improved, instantaneous communication, less manual labour, and more relevant content than ever before. How did this come to be? Why has the world bought into numbers and algorithms?
Can The Brain Keep Score?
One success story of data is in sports. The ability to make good decisions as an athlete and coach has long been left to gut instinct. Scouting talent has been an art form, perfected by the many front offices in Boston, Barcelona, and London.
A typical football match has 22 players kicking a ball about for 90 minutes. There are, on average, a total of 396 tackles made in a game, with each player making between 90-140 touches with the ball per game. This alone is a dizzying number of things to keep track of!
I don’t know about you, but much of the time I spend on football I’m not actually watching. I’m on my phone, getting a snack, off to the bathroom, or any number of things. How can the human brain hope to understand a game it can’t keep up with?
The answer: it doesn’t. By using some clever heuristics, the brain simplifies the action to make it easier to follow. For example, biases in attention. We’d rather follow the football than watch the positions and actions of all the individual players.
One of the most dangerous biases we fall prey to is outcome bias. The ends justify the means. If we get the desired result, we must have made the best decision, right?
Wrong.
This issue plagues a game like poker all the time, where randomness means weaker players can beat even professional players at a hand. We associate winning was down to our own skill as opposed to luck or other factors. This leads to overconfident “fish” who go on to lose all their money.
Is Data A Double-Edged Sword?
Data can help minimize the effect of biases by precisely documenting each interaction. We have reliable records that aren’t marred by our poor memory – but it comes at a cost. Complexity.
Reliance on data requires careful analysis to come away with accurate conclusions. This isn’t just a theoretical, wishy-washy issue that is combated with “more education on the subject”.
Scholarship suffers from bias all the time. It is well known that scientists tend to report the results of exciting experiments, while dropping trials that fail to dazzle. The result? A potential lack of evidence when forming conclusions. This can be disastrous in fields such as healthcare.
The upside is we get to validate and test hypotheses instead of spouting them into the ether. We can investigate the root causes of a problem, analyse the potential results of a decision, and create fresh perspectives on how we see the world.
In the modern world, the ability to work with numbers and data is a sense akin to touch and sight. Without being aware, it shapes our interactions: how many likes we get on Facebook (or Multiverse!), inflation rates, the time we spend watching movies … the data needed to describe our interactions is potentially endless. What new types of data are you curious about?
Abubakar Mohamed is a Digital Marketing alumnus based in London, UK.