Swimming in data lakes and running through random forests

One of my favourite authors, Haruki Murakami, also happens to share a passion of mine: long-distance running. He once wrote a book about it called What I talk about when I talk about running, in which he does a great job of explaining to non-runners why on earth anyone would want to take up such a ridiculous hobby.

In my last post I touched upon why now is a great time to be talking about all things data science. But as with most technical subjects, the vernacular can be pretty confusing to the uninitiated. So I’ll now try to explain what we talk about when we talk about data science!

What do we really mean by the terms artificial intelligence (AI), machine learning and deep learning? Many sources will show you a diagram that looks something like the one below, and it is…sort of fine in explaining these terms. As the diagram shows, AI is used as a catch-all term for the simulation of human intelligence. Machine learning is then a subset of AI referring to ‘algorithms with the ability to learn without being explicitly programmed’, while deep learning is a further subset of machine learning. 

While technically correct, we think that this framework for defining these terms is actually quite misleading – especially with regards to their application in finance – and often leads to them being used interchangeably.

When you think about AI, what comes to mind? Siri, facial recognition in Facebook, autonomous cars – all of these products and processes in one form or another draw upon deep learning. This is a subset of machine learning whereby layers of artificial neurons – meant to mimic the way our brains function – learn successive pieces of information about a particular data set, building upon what the previous layer has understood as they progress.

Machine learning is a really wide field of study with a huge array of methods and applications. What we think of as AI in its current form is really deep learning, itself a subset of machine learning in the earlier diagram. So instead of thinking of machine learning as a subset of AI and subsequently using these terms vaguely and interchangeably, we prefer to think of an algorithmic spectrum, with tasks separated by the ease with which the problem can be defined.

On the left of the above chart, we group processes that are easy to define but tedious to perform, and are thus prime targets for automation. On the right we place processes that are difficult to define, but in many ways easy to perform. Think here about many of our cognitive functions, like recognising the person sitting next to you using your eyes. This function is easy to perform, but it is a lot harder to define exactly how your brain achieved it.

While deep learning (our current definition of AI) belongs on the right here, what we refer to as ‘classical’ machine learning sits somewhere between the two extremes, and this is where we would group the vast majority of data-science techniques being deployed in investment management today.

Machine learning is really a marriage of traditional statistics and computer science; while there are many complex concepts within this bucket, the range is very broad and it also includes techniques that have simply been rebranded from traditional statistics (or as one of my colleagues put it, “with less of the ‘learning’ bit”). By separating out deep learning (and hence AI) from machine learning, the concept of machine learning immediately becomes less intimidating and more approachable for investment professionals. So next time a broker or adviser tells you about their innovative use of AI, it might be worth taking a closer look at what’s really going on under the hood.

Next time, I’ll further decompose machine learning and talk about some of the difficulties faced when applying data-science techniques to investment decision-making.