Dark Data is a Big Opportunity for Services Companies

dark dataWhy is Big Data big? Obviously not because someone invented a new statistical method of data analysis that can explain everything. It’s because there is a lot more data spewing out of business processes where there were none before. The questions haven’t changed – as a business manager I still want to know how to forecast sales or understand which levers to apply to improve my business outcomes. But where there was once no way to answer these questions, today there actually is data that can be analyzed to provide some of these answers.

So far all the attention has been on business problems whose answers lie in the copiously flowing data from associated business processes. Marketing, for example, where all Marketing is getting subsumed into Digital Marketing which, as anyone will tell you is a data gusher.

But there are innumerable questions in business that need answering where the data just isn’t there to analyze. Or rather, the data is there but it is unusable, just beyond reach.

I am part of a non-profit theatre company in the Bay Area. Theatre companies live and die by their ticket sales. All costs are fixed. Once you decide to stage a play your costs are all locked in. Your revenue, however, is completely variable, by the number of tickets you sell.

In such a business, it would be crucial to understand where one stands on ticket sales. In other words, you should be able to answer the following question at all times “Based upon the ticket sales today, X days before opening night, we are on track to fill Y% of seats”.

Almost all ticket sales are online. Which is a good beginning. The online ticket selling service that we use sends us a daily email with our cumulative ticket sales till that day. But, and here’s the nub, they don’t store a time series of daily ticket sales. So if I wanted to draw a graph with number of days to opening night on the X axis and cumulative tickets sold (by show, of course) on the Y axis, I’m out of luck.

This is not a unique situation. I’m sure you can think of many such examples in your business where you know the data is created but it isn’t kept, or it isn’t in the right form.

Then there’s a form of data that is not captured but can be, with just a little work. At Infosys, when I was trying to wrap my head around how to implement CRM in a company which hadn’t used one for years, I thought that it might be too much to expect the field force to make notes after every client meeting. But perhaps, if they could just log every client meeting, that by itself would be very useful. It would be a measure of business activity which we may be able to correlate to deal value and perhaps, could serve as a rough, early warning forecasting system. There are so many opportunities for squeezing a business process for meaningful data. Analyzing this data, typically doesn’t need Hadoop clusters, but the business outcomes could be quite significant.

Think of data as fossils in sedimentary rock. The fossils in the upper layers are newer, better formed and easier to interpret. The ones in the lower layers are just the opposite. But they are just as important to understanding and improving your business.

IT Services companies will see a lot of opportunity in Big Data and Analytics. But software companies will take away most of the value in the top sedimentary layers. The lower layers will be messy. And straightening out messes, is where IT Services companies thrive.

Meanwhile, I’ll be trying to sort out my ‘messy’ ticket sales data using Google Script. If anybody knows of a script that will help extract a number from an email that follows an identical string of text, please send me a note. Thanks.


  1. ayyappa says:

    Dark Data – very interesting term..:)

    On CRM. I worked with CRM (lead, opportunity, order, contacts, prospects) for a long time – used by a large sales force. For the same organization, I helped build/customize on two different CRM systems over 5 years and when I left they were evaluating a third (Salesforce.com, Microsoft Dynamics, Siebel etc). Finally did Salesforce.com. Just last week, while catching up with an old colleague, he said now they are in the process of moving out – it looks like we changed CRM systems every 3 years or so. Challenge was not with the systems. What is the value that the actual sales force derive from this – they hate systems, asking them to put every lead, every interaction, impressions, who is an influencer, who is decision maker, what are their priorities, opportunity, how they chase it – putting all of it into the system is tedious for them. They don’t get value out of it. There is tremendous value for senior management to get intelligence out of this data, but since the ground field force don’t see the value of entering this data religiously, on time, that intelligence is mostly based on incomplete data. Unless it is IoT, sensor data, automatically generated, it should be extremely painless to capture or there should some reward/value for people who are capturing such data. Long winding comment, but just was skeptical of improving CRM data and cracking the code of Dart Data in that context..


    1. Agree Ayyappa. CRMs thus far have been focused on value to management from the data and less on value to the field force.


  2. You are right, Basab, about the opportunities in straightening out the messy, bottom most layer of data. It can be assumed that anyone who does that level of data sifting will gain tremendous insights into structuring them (not processing them but just laying out their structure). Question is, if the sexy part of the value chain is to unleash software toolsets to process the information leading to business insights why do IT Services company shy away from doing that? Why do they yield that space to specialized software companies?


    1. Because software isn’t their business?


Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s