The #1 Reason AI/ML Projects Fail in Your Organization…
Originally published in Molecula blog site on May 20, 2021
The shine on AI and ML projects has never been brighter. The C-suite is excited about the prospects of these projects and is throwing money at them. Why then, with all this enthusiasm, can’t AI/ML projects gain more traction across the business? Why are so many of them failing?
The short answer: your business executives are not getting timely insights or sometimes even the right insights.
Almost every enterprise has accepted that AI can provide a competitive advantage. They have also created a separate org headed by Chief Data Officers (CDO), hired some of the best data scientists, collected the most appropriate data, bought some of the best AI platforms/tools, and yet are struggling with productionizing their AI/ML models.
One of the major issues is that most companies still use data the same way they have used it for years. While we have moved from a “compute economy” to a “data economy” with AI proliferation, organizations deal with data the same old way. The data economy (and AI/ML) will not succeed if it is done this way.
A 2020 survey by Forrester on behalf of Dell, (with responses from 1,600+ data decision-makers) had some very interesting findings:
Data volume does not seem to be the problem in getting the data value in most enterprises.
“71% say they are gathering data faster than they can analyze and use.”
Despite that, most businesses crave more data even though their current tools, processes, and culture cannot handle any more data.
“66% of businesses say they constantly need more data than their current capabilities provide.”
Shockingly, in spite of the demand for more data, a majority of enterprises admit their data teams are overwhelmed.
“62% say their data teams are already overwhelmed by the data they have.”
The most important part of this survey, in my view, was the discovery that 64% of those executives claim to be from a “data-driven” organization and say that data is the life of their organization.
84% of the businesses have a major problem with non-optimized data warehouses, high storage costs of data because of too much data, outdated IT infrastructure, and finally manual or slow processes that do not meet business needs.
So, if I were to summarize:
- Enterprises have a lot of data.
- Some of them need even more data than they already have.
- Data teams are overwhelmed even with the current levels of data.
- Data pipelines are not optimized for AI/ML model creation.
The above points are very much in line with an article I wrote in Forbes last year. Research shows that data teams are not able to handle data properly and get insights in time for the business executives to make decisions. A mind-boggling 67% of CEOs worldwide (78% in USA) ignore data-driven analytics and predictive models provided by their CIO/IT teams because the insights contradicted their experience. Consequently, major enterprise decisions are being made based on intuition.
This should be very concerning. After spending so much money, effort, and time on creating AI/ML models, the very people who wanted—and funded—them are ignoring the results. As I stated at the beginning of the article, business executives are not getting timely insights, or even the right insights, from ML models.
Unsurprisingly, the “digital native” enterprises don’t have such a problem. Not only have they figured it out, but they thrive on real-time insights made possible through ML models. Imagine Uber not being able to find, or properly price, your ride or AirBnB pricing themselves unattractively against hotels because they had a problem in crunching the numbers!
Almost all of the big name data-driven companies have solved their data problem. They not only thrive on data, but they can handle even more. They are successful because they figured out their “data pipeline” to feed their ML models properly and on time, thus resulting in very accurate, timely results.
A major component of all the data-driven companies’ success is a concept called, “feature store.” A feature store is a curated data store for AI/ML model creation, model training, and real-time model serving for decisions, that feeds ML models accurately, consistently, and in a timely manner both for training and decision-making. I will write a detailed follow-up blog specifically on feature stores very soon.
If you find this hard to believe, check out the State of AI in the Enterprise survey from Deloitte Insights. You’ll see 2,700+ LOB executives report their top AI initiative as, “modernizing data infrastructure for AI.”
The bottom line is, fix your data pipeline to service the data economy. This is very different than yesterday’s compute economy for which databases, data stores, data lakes, and the IT infrastructure were all built.
As I’ve stated before, make sure you are giving AI the right amount and quality of data if your executives are going to make major enterprise decisions based on it. If you don’t, they may very well ignore your model outputs and recommendations and make their own decisions which will introduce more risk and devalue the opportunity of creating a truly data-driven business.
This post is sponsored by Molecula
