15pctAISuccess- Part 2

To be in the 15% successful : Part 2 -

Availability of Data

Many people assume that you can just throw in the data and the answers will magically flow out. (I actually had a boss ask me to find him that black box). But that is not really realistic; even in GenAI, today, you need some skill at prompt engineering.

The most obvious issue is that the data that you have simply needs to be related to the problem you are trying to solve.

There must be the breadth of data elements you think have some impact on the problem at hand, and also sufficient depth so you can have enough examples your analytical techniques can use/learn from.

Data is everywhere, how can there be insufficient data?

There are 3 major reasons why data that you have may not be fit for the purpose of solving some business questions

It’s not there: Data simply has not been kept by the business (or not for long enough)
There was no perceived value in keeping it: Certain data elements or data past certain time is routinely not kept since was not seen as crucial to the business/regulations
The data is not owned by the business: Your organisation does not own the data.

The data is not there

For many traditional bricks and mortar/off line/physical shop, the data may not have been kept, or is in such separate systems that it takes a lot of effort to gather the data before analysing it.

A simple example is a retail shop, or a company engaging in renovation works. They will definitely keep records of the goods purchased for sale. These records would include price and date, and may be invoice details. Similarly, there will be data on sales, what was sold and when. For many that is sufficient to run the business.

The record keeping software for sales may be a simple Point of Sale system, therefore by design, very little is captured about the customer making the purchase for example. Or even if you know who made the purchase, you do not have details about the person that could be leveraged say to understand patterns of purchase among different groups.

It’s not because of anything sinister, it could simply that this information was not critical to running the business and nobody had thought of extracting value from the data.

The situation is similar say for an organisation in the construction/renovation sector. The strict minimun will be details on invoices for equipment and material used during business, as well as which project they are charged to.

Very often however, details about the markets that allows timing of purchases and optimising stocks and ash-flows for example may not be captured.

There was no perceived value

In some organisations, it is a deliberate choice not to keep the data. This is often true for older organisations, or organisations set-up by people from organisations with long legacy.

Many years ago, the decisions whether to keep data or not, were driven by regulatory requirements and costs. The idea was to keep the minimum possible for as little time as possible so as to minimise storage costs, hard disks on servers were expensive, even magnetic tapes needed special care.

But before you think newer, especially organisations with online presence would be more modern and less prone to such issues, I can tell you of a customer with a huge online presence that did not capture anything about visitors to the site unless a purchase was made. This of course saves a lot of money in terms of storage, but totally blinds the organisations to patterns prospects have.

You do not own the data

This is very true especially in the online world. There are many platforms that allow users to establish digital shops. Very often these come with canned reports, providing the customer (here a seller) with some basic business intelligence.

But often, the platforms’ business is to sell value-added services to sellers, run marketing campaigns targeting relatively inflexible segments of visitors to the platform. In such cases, it often takes some resources to be able to obtain data with enough granularity so as to allow proper analytics to be performed.

Note however that this is not restricted to only online platforms. Some software systems such as SAP have proprietary connectors whereby it costs organisations a fair coin to be able to access the data in sufficient granularity to conduct analyses outside of the system.

So what can be done?

The first step is to understand what data is available, at what granularity and detail level and for how long. Then compare this to the business issues that have to be solved. In order for this to happen, some collaboration between the business side, who intuitively know the important levers of their specific business and experienced data people who have solved similar problems and have an idea of what data elements are critical, and what elements are good to have an so on.

Basically, a review of the data available as a collaborative exercise between the business and experienced analytics people who know the industry/solved similar issues/have a good grasp of the business issues is critical.

If the perfect set of data is not available, there are strategies that can be employed to approximate what is required, while taking steps to bring the dataset closer to the requirement. This is often a multi-stage process with increasing performance, it’s not magic, it has to be understood, planned for, and executed.

At DataMobius, we have the experience of having solved business problems in various industry verticals as well as horizontal business units. This gives us confidence that we can work with you to avoid this pitfall.

Report abuse