Predictive AI Must Be Valuated – But Rarely Is. Here's How To Do It

Most predictive AI projects neglect to estimate the potential profit – a practice known as ML ... More valuation – and that spells project failure. Here's the how-to.
To be a business is to constantly work toward improved operations. As a business grows, this usually leads to the possibility of using predictive AI, which is the kind of analytics that improves existing, large-scale operations.
But the mystique of predictive AI routinely kills its value. Rather than focusing on the concrete win that its deployment could deliver, leaders get distracted by the core tech's glamor. After all, learning from data to predict is sexy.
This in turn leads to skipping a critical step: forecasting the operational improvement that predictive AI operationalization would deliver. As with any kind of change to large-scale operations, you can't move forward without a credible estimation of the business improvement you stand to gain – in straightforward terms like profit or other business KPIs. Not doing so makes deployment a shot in the dark. Indeed, most predictive AI launches are scrubbed.
So why do most predictive AI projects fail to estimate the business value, much to their own demise? Ultimately, this is not a technology fail – it's an organizational one, a glaring symptom of the biz/tech divide. Business stakeholders delegate almost every aspect of the project to data scientists. Meanwhile, data scientists as a species are mostly stuck on arcane technical metrics, with little attention to business metrics. The typical data scientist's training, practice, shop-talk and toolset omits business metrics. Technical metrics define their comfort zone.
Estimating the profit or other business upside of deploying predictive AI – aka ML valuation – is only a matter of arithmetic. It isn't the "rocket science" part, the ML algorithm that learns from data. Rather, it's the much-needed prelaunch stress-testing of the rocket.
Say you work at a bank processing 10 million credit card and ATM card transactions each quarter. With 3.5% of the transactions fraudulent, the pressure is on to predictively block those transactions most likely to fall into that category.
With ML, your data scientists have developed a fraud-detection model that calculates a risk level for each transaction. Within the most risky 150,000 transactions – that is, the 1.5% of transactions that are considered by the model most likely to be fraud – 143,000 are fraudulent. The other 7,000 are legitimate.
So, should the bank block that group of high-risk transactions?
Sounds reasonable off the cuff, but let's actually calculate the potential winnings. Suppose that those 143,000 fraudulent transactions represent $18,225,000 in charges – that is, they're about $127 each on average. That's a lot of fraud loss to be saved by blocking them. But what about the downside of blocking them? If it costs your bank an average of $75 each time you wrongly block due to cardholder inconvenience – which would be the case for each of the 7,000 legit transactions – that will come to $525,000. That barely dents the upside, with the net win coming to $17,700,000.
So yeah, if you'd like to gain almost $18 million, then block those 1.5% most risky transactions. This is the monetary savings of fraud detection, and a penny saved is a penny earned.
But that doesn't necessarily mean that 1.5% is the best place to draw the line. How much more might we save by blocking even more? The more we block, the more lower-risk transactions we block – and yet the net value might continue to increase if we go a ways further. Where to stop? The 2% most risky? The 2.5% most risky?
To navigate the range of predictive AI deployment options, you've just got to look at it:
A savings curve comparing the potential money saved by blocking the most risky payment card ... More transactions with fraud-detection models. The performance of three competing models is shown.
This shows the monetary win for a range of deployment options. The vertical axis represents the money saved with fraud detection – based on the same kind of calculations as those in the previous example – and the horizontal axis represents the portion of transactions blocked, from most risky (far left) to least risky (far right). This view has zoomed into the range from 0% to 15%, since a bank would normally block at most only the top, say, two or three percent.
The three colors represent three competing ML models: two variations of XGBoost and one random forest (these are popular ML methods). The first XGBoost model is the best one overall. The savings are calculated over a real collection of e-commerce transactions. So was the previous example's calculations.
Let's jump to the curve's peak. We would maximize the expected win to more than $26 million by blocking the top 2.94% most risky transactions according to the first XGBoost model.
But this deployment plan isn't a done deal yet – there are other, competing considerations. First, consider how often transactions would be wrongly blocked. It turns out that blocking that 2.94% would inconvenience legit cardholders an estimated 72,000 times per quarter. That adverse effect is already baked into the expected $26 million estimate, but it could incur other intangible or longer-term costs; the business doesn't like it.
But the relatively flatness that you can see near the curve's peak signals an opportunity: If we block fewer transactions, we could greatly reduce the expected number wrongly blocked with only a small decrease in savings. For example, it turns out that blocking 2.33% rather than 2.94% cuts the number of estimated bad blocks in half to 35,000, while still capturing an expected $25 million in savings. The bank might be more comfortable with this plan.
As compelling as these estimated financial wins are, we must take steps to shore up their credibility, since they hinge on certain business assumptions. After all, the actual win of any operational improvement – whether driven by analytics or otherwise – is only certain after it's been achieved, in a "post mortem" analysis. Before deployment, we're challenged to estimate the expected value and to demonstrate its credibility.
One business assumption within the analysis described so far is that unblocked fraudulent transactions cost the bank the full magnitude of the transaction. A $100 fraudulent transaction costs $100 (while blocking it saves $100). And a $1,000 fraudulent transaction indeed costs ten times as much.
But circumstances may not be that simple, and they may be subject to change. For example, certain enforcement efforts might serve to recoup some fraud losses by investigating fraudulent transactions even after they were permitted. Or the bank might hold insurance that covers some losses due to fraud.
If there's uncertainty about exactly where this factor lands, we can address it by viewing how the overall savings would change if such a factor changed. Here's the curve when fraud costs the bank only 80% rather than 100% of each transaction amount:
The same chart, except with each unblocked fraudulent transaction costing only 80% of the amount of ... More the transaction, rather than 100%.
It turns out, the peak decreases from $26 million down to $20 million. This is because there's less money to be saved by fraud detection when fraud itself is less costly. But the position of the peak has moved only a little: from 2.94% to 2.62%. In other words, not much doubt is cast upon where to draw the decision boundary.
Another business assumption we have in place is the cost of wrongly blocking, currently set at $75 – since an inconvenienced cardholder will be more likely to use their card less often (or cancel it entirely). The bank would like to decrease this cost, so it might consider taking measures accordingly. For example, it could consider providing a $10 "apology" gift card each time it realizes its mistake – an expensive endeavor, but one that might turn out to decrease the net cost of wrongly blocking from $75 down to $50. Here's how that would affect the savings curve:
The same chart, except with each wrongly-blocked transaction costing only $50, rather than $75.
This has increased the peak estimated savings to $28.6 million, and moves that peak from 2.94% up to 3.47%. Again, we've gained valuable insight: This scenario would warrant a meaningful increase in how many transactions are blocked (drawing the decision boundary further to the right), but would only increase profit by $2.6 million. Considering that this guesstimated cost reduction is a pretty optimistic one, is it worth the expense, complexity and uncertainty of even testing this kind of "apology" campaign in the first place? Perhaps not.
For a predictive AI project to defy the odds and stand a chance at successful deployment, business-side stakeholders must be empowered to make an informed decision as to whether, which and how: whether the project is ready for deployment, which ML model to deploy and with what decision boundary (percent of cases to be treated versus not treated). They need to see the potential win in terms of business metrics like profit, savings or other KPIs, across a range of deployment options. And they must see how certain business factors that could be subject to change or uncertainty affect this range of options and their estimated value.
We have a name for this kind of interactive visualization: ML valuation. This practice is the main missing ingredient in how predictive AI projects are typically run. ML valuation stands to rectify today's dismal track record for predictive AI deployment, boosting the value captured by this technology up closer to its true potential.
Given how frequently predictive AI fails to demonstrate a deployed ROI, the adoption of ML valuation is inevitable. In the meantime, it will be a true win for professionals and stakeholders to act early, get out ahead of it and differentiate themselves as a value-focused practitioner of the art.

Try Our AI Features

Explore what Daily8 AI can do for you:

Learn with AIFact CheckingText to SpeechAI Summary

Related Articles

Bloomberg

25 minutes ago

• Bloomberg

US Could See Potential 'Whiff' of Stagflation: Clarida

Richard Clarida, global economic adviser at Pimco and former vice chair of the Federal Reserve, discusses the inflation situation in the US. "If we do get stagflation, it's not going to be the bad old days of the Seventies," Clarida tells Bloomberg's Francine Lacqua. "We'll have a whiff, potentially a whiff, of stagflation." (Source: Bloomberg)

Yahoo

26 minutes ago

• Yahoo

How to take care of yourself and your finances when you've experienced a layoff

NEW YORK (AP) — Recent layoffs from technology and media companies and government agencies might have you thinking about job security. Losing your job is a difficult thing to process and you might feel the impact in several parts of your life. But there are things you can do to alleviate some of the stress and anxiety. If you've been laid off, experts recommend that you first take a moment to process and then move on with a plan for your job search. 'A layoff can feel so personal but it's not a reflection of your value or what you contributed. Especially in the U.S., the work we do is so tied to our identity,' said Lindsay Bryan-Podvin, a financial therapist. Here are some expert recommendations to help you take care of your finances and your mental health if you've lost your job: Take a break to process Losing your job can cause a lot of stress and financial anxiety, so it's important that you take time to emotionally process. 'Acknowledge and normalize that they're going to feel a range of emotions, whether it's anger or sadness,' Bryan-Podvin said. Whether you are feeling sadness, shock, anger or even relief, taking a moment to feel those emotions can help you move forward. Review your current financial situation To make plans for the future, it's essential that you review your current financial situation, said career coach Marlo Lyons. If you have a budget, take a closer look and review whether you have expenses that you can temporarily cut to save money. Lyons recommends that you know how long your severance package will last if you're receiving one. Applying for unemployment benefits as soon as possible can help alleviate some of the financial stress of a layoff. While the amount you get for unemployment might not be as much as your salary, it can help you to stay afloat for some time, Lyons said. The Labor Department has tips on applying for unemployment. Remember that you are not your job Getting laid off from a job can affect your self-esteem, so Bryan-Podvin recommends that you create a list of good qualities about yourself that add to your overall value as a person. Bryan-Podvin calls this a 'non-financial asset list." 'If I were doing a non-financial asset list on myself I might say I'm a pretty good partner and a fun aunt,' she said. When working with her clients, Bryan-Podvin recommends that they create this list to remind them that their self-worth is part of their net worth. Adjust spending If you lose your job, you'll need to be more strategic about your spending, said Jesse Mecham, founder of the money management app YNAB. 'In a layoff, it becomes even more imperative that you treat every dollar with more attention than you have in the past,' Mecham said. Part of YNAB's budgeting strategy is giving each dollar a job, whether it is to pay rent, buy groceries or add to your savings account. When adjusting your spending, Bryan-Podvin also recommends that you approach this with some compassion. Since cutting back can bleed into not doing activities that make you happy, she recommends reminding yourself that this is a temporary pause rather than a permanent change in your lifestyle. Avoid overusing your credit card While utilizing your credit card to pay for some expenses is almost inevitable, it's best to not completely lean on credit while you're searching for a new job. 'If getting a new job takes them a little longer than they thought, that credit card has just become an anchor for them,' Mecham said. Even though unemployment is temporary, Mecham recommends cutting expenses rather than keeping them at the same level and leaning on credit. Find community resources Look for community resources, whether that means finding a food bank near you or applying for a temporary hardship program. 'Lots of communities offer temporary hardship programs when it comes to necessary bills, such as electricity or water,' Bryan-Podvin said. offers a benefit finder where you can find available programs and how to apply. Approach your job search with a plan As you are applying to new jobs, Lyons recommends that you take some time to assess if your professional goals are still the same. If you're continuing in the same field, Lyons recommends that you make your resume 'forward-looking,' which means showing your future employer what you can do, not just what you have done. 'You want to show the employer what you can do, what unique value you can bring to that particular job that no other candidate can bring because of your previous experience,' Lyons said. Lyons also recommends that you activate your network by reaching out to past colleagues on LinkedIn or attending industry networking events. Online certifications are another great way to make yourself a better candidate, she added. Create and maintain a routine Having a routine can help you take care of your mental health and keep a cadence of applying to jobs. Plan your days so they include eating at your usual time, exercising and applying for jobs for a specific amount of time. 'When we are laid off, unemployment can feel really aimless, especially if it came at us kind of out of nowhere,' Bryan-Podvin said. It's also important to avoid isolating yourself and lean on your support system for emotional support, she added. ___ The Associated Press receives support from Charles Schwab Foundation for educational and explanatory reporting to improve financial literacy. The independent foundation is separate from Charles Schwab and Co. Inc. The AP is solely responsible for its journalism. Adriana Morga, The Associated Press

Bloomberg

27 minutes ago

• Bloomberg

NY Jets Owner Woody Johnson Agrees Deal for Crystal Palace Stake

Robert 'Woody' Johnson IV, the co-owner of the New York Jets and a former American ambassador to the UK, has agreed to buy a significant stake in Crystal Palace, the most recent FA Cup champion, according to people familiar with the situation. Johnson is set to pay about $215 million to John Textor, who owns a more than 40% stake in Crystal Palace via his Eagle Football group, according to one of the people familiar with the situation.