Now that the infrastructure to store and process vast quantities of data has become mainstream, the initial wave of hype around ‘big data’ has subsided.

Having access to big data is no longer a differentiator; it’s now the way you exploit your data that sets you apart.

The race is on to get to the point where extracting maximum value from your data is standard. This is all very well for digital native companies that have built their entire infrastructure to achieve this, but it’s much harder for others that need to transform their legacy systems before truly exploiting big data. Many organisations currently sit somewhere on that journey, having made investments under the heading of ‘Big Data’. Some are frustrated that they haven’t gained the value they expected, and others have had success, but are now wondering how to build on it.

Reacting to the coronavirus pandemic introduced new strains on how organisations use data, shaking up the business-as-usual data flows. Suddenly, people needed different data sets and they needed them quickly. For many businesses, these unusual requests pushed them to the limit of their data capabilities and exposed cracks in their data management structures. Organisations struggled to work across the business with consistent, reliable and checked data sets. They were caught on the back foot, and collaboration suffered.

In some cases, businesses found that the people-focused data they needed just didn’t exist, and they were unable to look at the effects of the pandemic from a people perspective. 

The need to analyse specific situations from different, new angles is a lesson that businesses can learn a lot from. It reinforces their ability to do more than react to challenges, enabling them to explore and seize opportunities in a market that’s evolving fast.

The goal of big data is being able to assemble and interrogate data across every aspect of your business, so you can zoom out to understand large scale phenomena and zoom in to understand an individual or to customise an experience – all in a timely way. Many organisations can already do this, but it’s often a slow and painful process that doesn’t support real-time decision-making.

Successfully realising the full value of big data means you can make better decisions, faster and more cost-effectively, supporting more efficient internal performance. And by bringing AI and machine learning into the equation, you’re able to better allocate resources to match demand, moving to preventing events rather than just predicting them. Combining AI and machine learning with ‘human in the loop’ input allows the automation of complex processes that would otherwise be too costly. This frees people from tedious and time-consuming tasks to do something more productive.

From a customer point of view, you’ll be able to use customer profiles and behaviours to tailor your propositions to their needs. You’ll create a rich dialogue with your customer to fine-tune how they’re using your products, offering enrichments where appropriate. And, as your strategic insight into your customer’s experience increases, you’ll be able to target your activity and investment to support your business goals.

Big data is currently fulfilling its potential in a wide range of industries and applications, from drug discovery and genomics, programmed trading and consumer fraud detection in financial services, through to transportation logistics and high-end engineering, such as sensing on modern jet engines. So, what are the challenges businesses like yours need to overcome to achieve similar success?

To become the heartbeat of the business, big data and analytics need to demonstrate the value they bring quickly. This can be difficult, because showing value almost immediately takes a strong foundation that might not be there. Depending on where you start, you may need an initial ‘leap of faith’ investment to cover significant gaps before you can even start demonstrating value.

Moving forward from this position involves developing capability while attacking value. There’s no silver bullet for this challenge, but moving from rigid, on-premise, best-of-breed infrastructure to cloud computing offers the flexibility and scalability that will set up your operations to focus on extracting value. Basing data storage, management and analytics in the cloud opens up fresh cost models for your organisation to explore. In fact, cloud adoption is likely to expose costs around data and data science that have previously been hard to isolate or have been absorbed in other ways. This could potentially transform how your internal stakeholders view and use data analytics.

It sets the scene for viable data analytics experiments. The ability to set up and run data analytics rapidly is an enabler, as is the broader, more democratic access to data analytics that cloud deployment brings. The cloud model removes the need for big, upfront investment and its costs are more likely to scale in a linear way with capacity and usage. This largely removes the cost barrier to entry, opening the field up to testing new applications during production – analytical experiments can run alongside business-as-usual processes, without impacting on them in any way.

Achieving high-quality data analytics isn’t a switch that you can just flick and watch the value flow through the organisation. The right tools and basic data infrastructure need to be in place to get the most from any investment in data. In order to be ready to implement AI and machine learning, we need to move how we deal with data from an art into a science. It’s essential too, that businesses set their data analytics firmly within emerging legal and ethical frameworks, to make the most of their AI and machine learning.

Tools now exist that make the process of developing the models that power AI and machine learning more disciplined, reproducible and traceable. Businesses must be able to manage all the inputs that go into developing their models, including the data they ‘train’ the models on, the structure of the models themselves (so they can be adapted or re-used) and anything created along the way that might be re-used at a future date. This transparency means that the results of AI and machine learning can be held accountable to regulatory and ethical systems.

Currently, many firms only pay lip service to the fact that data is a company’s biggest asset, and this needs to change. To really exploit the benefits of big data and analytics, data needs to run through everything as an integral part of the organisation’s people, culture and skills. The litmus test of success will be when every area of your company has data as part of its strategy.

It’s the time to prioritise getting the right talent to realise your big data goals, and to make sure data skills are evenly spread throughout your organisation. Right now, it’s likely you have islands of expertise, patchily distributed. Changing this will probably involve a broad programme of upskilling and empowering your workforce to create digital citizens who appreciate the importance of accurate data and act as accountable curators for it across its use journey. This data-skilled workforce is essential to achieving positive outcomes throughout every aspect of data, from entering and consolidating it, through to distribution.

When big data becomes business as usual – and when we stop referring to ‘big data’ – we’ll know the size of the data has stopped being the main problem and its adoption journey has come to an end. What will the data environment look like at that point?

We predict that a thorough understanding of the value of data throughout its lifetime will become the norm for businesses. We’re moving into a world that will prize the cutting-edge models and processes that run on the top of data, rather than the data itself.

To get to this point, data must join the trend towards digital first capabilities. A cloud-based data approach will mean that the appetite for gathering data for the sake of it will fade. Instead, businesses will shift to a more cost-effective methodology of dipping into a data lake common to the industry and going back to get more data as and when it’s needed. As companies move to buying in data and mission-critical services as software-as-a-service tools, they’ll find that, increasingly, analytical capabilities are built in. Potentially, this could generate a regression back into a siloed way of working.

The promise of big data and analytics is to give everyone and everything information superpowers, while protecting privacy and our social values. What’s evident now is that making this a reality will involve a careful blend of factors in an environment that both values and respects data. Action needs to start today on upskilling the workforce, making timely, good quality data available and putting in place appropriate infrastructure and tools. 

Discover how your business can exploit technology and innovation both now and in the future by downloading our ‘Winning the innovation race’ brochure.