Problems are cherished

If there is no problem, there is no need for science; so embrace problems, cherish them and indulge them. Whatever you do, don't run away. Problems challenge us to raise our game, to rise above what is currently possible, to explore new ground. By meeting problems head on throughout my career I have made significant gains in each field I have operated in.

Dr Beattie has two decades of experience in cutting edge real-world science honing an ability to confront and solve difficult data problems

The key to problem solving is four steps:

• Understand
• Identify
• Plan
• Resolve

Curiosity doesn’t kill cats

Not assessing the dangers does. Curiosity is something fundamental to both cats and humans, but it is something drummed out of us from very young.

But why? What is round there? What would happen if…?

I generally resent so-called lateral thinking tests as most are closed with one ‘correct’ answer. You are expected to think in the same obscure way as the person who dreamed it up. It defeats the purpose – lateral thinking is about being different, not the same.

Nature allows for lateral thinking without a predefined answer and I think that is why I love science. It is complex and vast, with so much left to explore. I never stuck to the paths in forest parks and I don’t ever intend sticking to the well worn paths in science. I want to explore, I have a voracious curiosity that needs fed.

Understanding trumps ignorance

A popular trend in data science and statistics is to trust mathematical models implicitly.

 ‘In Math we trust’.

But to do so is a grave error. The assumption is that maths does not hold the same biases we do, so it must perform  better job if left to its own devices. But even if this was true (the functions that the maths can exploit must be made available by people so that introduces a bias), the data often is horribly biased and so layering maths on top can give biases a veneer of truth. Biases, unlike errors, get worse the more data you throw in and the more complex your math becomes.

Understanding the data opens up the possibility of finding ways to handle or, better, circumvent biases. It also allows anticipation of the breakdown limits of the models, when it is clear that the model will not be appropriate.

Insight is Mightier than Data

Data: Tomato is a fruit

Insight: Do not put a tomato in a fruit salad.

There is a lot of hype today about the value of data to businesses, with dizzying claims about how much it is worth. The one problem with this is that data itself will provide not net value. Data should never be conflated with ‘information’, it will only become informative, and so provide value, if it can be turned into insight.

If big data is rushed and not enough thought is put into the strategy of optimising insight, it will almost certainly provide none.

Despite being so young, the big data world is full of examples of data failing to provide insight and in some cases actually embedding human bias deep in the system (the book ‘Weapons of Math Destruction’ by Cathy O’Neil is a good high level introduction).

Risk is inevitable, failure isn’t

All data analytics is done with the aim of achieving something new. The fact that it is new means that there are unknowns that need to be met and dealt with. This means risk. Risk of developing a wrong solution and missing something that would have worked. Risk of making a wrong guess and it turning out that it was never going to work…

With appropriate planning and ongoing risk evaluation all stage can be managed to minimise disruption and maximise information. Even if it turns out that the original aim is no longer relevant (new regulations, new fads, new technology) a well designed big data project will continuously provide insight and information. This ensures maximal value is extracted.

I have worked on projects were it became clear the original trajectory was no longer viable. I have never worked on a project that did not provide significant knowledge and insight. This is down to design and monitoring with an adaptability to changing needs.