Home >> Insights  >>

Dealing With The Digital Data Deluge

by Seby Kallarakkal, - CEO, Nabler July-2018

Headquartered in New York, Nabler is an expert in Digital Analytics Operations, capable of empowering clients to make better marketing decisions by leveraging data and technology.

The Size Problem
To set the context, let's contrast the digital world and physical world from a data perspective. Consider a shop that does 500 orders a day with 10 items per order and we have 5,000 records in the database with columns like product ID, rate, quantity, total amount and name of sales person. Contrast that with an e-commerce website which also does 500 orders a day. Here again, you have 5,000 records, but with more columns like customer name, address, phone, email and mode of payment. You will also have data points like traffic source, pages viewed and time spent, making it a substantially wider data set. 

Assuming 3% conversion rate, the website needs to generate 16,666 visits for 500 orders. If every visit has 10 page views, the click stream data would have 166,000 records. A typical e-commerce website also has multiple measurement tags. If you assume 10 such tags, collectively we are talking about 1.6 million records. Add social, mobile, and campaign data to this and you can see the magnitude of data deluge in the digital world.

The Accuracy Problem
The physical world typically generates transactional data. Digital world generates a lot of intent and behavioural data - that tends to be less accurate than transactional data. If your analytics tool shows 1,000 visits a day, are they humans or crawlers? If the same person visits from laptop and mobile, it might appear as two visitors. If somebody leaves the browser open for 5 minutes and comes back later, the tool will include that 5 minutes. 

The Consistency Problem
It’s not just accuracy that will trouble you, but also inconsistency. Different tools will give different numbers for the same set of activities. Which ones would you believe?

The Speed Problem
The speed of the data starts to matter when you have to take decisions fast. You need to take the data in, process it and use it quickly. A really good example is bidding for keywords where the big players use algorithms to do the bidding. Another challenge with the speed of data is that you will run out of space faster.

The New Sources Problem
New devices, new platforms and new channels are constantly emerging. Think about platforms like Instagram, Pinterest or devices like Apple Watch. All of them are generating more data points for the organization. 

The Ownership Problem
Data from the physical world is typically stored within the organization network. In the digital world, data is often stored in the cloud on vendors’ servers. Many a times, an individual within the organization would introduce a new tool that starts generating data without the IT even knowing about it. 

Business stakeholders will have their own way of dealing with the data deluge and leveraging it. But how does IT deal with this? 

Build the Infrastructure
Start building a data infrastructure that can deal with data which is big, not very structured, and can come in fast. Look around at the kind of data other companies in your industry are dealing with. Keep an eye open for the new data sources that are emerging. Consider cloud as an option. Keep scalability in mind as it is really hard to estimate the size of data.

Accelerate Data Intake and Publishing 
One of the often repeated complaints about IT is that it is slow. So you have to work on making your process for data intake and publishing faster. Digital data just like any other data requires governance and IT is best positioned. Therefore, IT can still play a big role in how organizations bring data in and start using the data. But the nature of the data and the nature of usage dictate a certain speed.

Go Ask For Data
Once you have confidence in your infrastructure and your process – start conversations with business stakeholders. Offer your help in handling their data and putting governance in place. It is better for IT to take over, rather than let business users use it under the radar.

Publish Early Wins
All are trying to figure out things as we are still in the early stages of digital data. Start publishing your early wins of taking over data sources, streamlining the process and making it available for decision makers. That would encourage other users to approach you to take over their data. 

The data deluge is both a threat and an opportunity for the IT. It can overwhelm the IT department, equally has the capability to put IT back, firmly in the strategy chair.