I recently watched Eric Siegel’s talk, The AI Playbook, at Talks at Google (https://www.youtube.com/watch?v=gnVk6ztz6Bs). Siegel has worked in machine learning for over 30 years, founded the Machine Learning Week conference, and wrote the book The AI Playbook.
The talk is about 59 minutes long, and I came away with one idea I can’t shake: the hardest part of AI isn’t building the model. It’s getting the model used.
Here is what he said, and what I think it means for anyone building or selling AI today.
The Core Problem: Great Models, No Deployment
Siegel opens with a blunt claim: most new machine learning projects fail to reach deployment. The models get built, the accuracy numbers look good, and then nothing changes in the actual business.
His explanation is that both sides treat these as technology projects. Data scientists say, “My job is the number crunching. Using it is a management issue.” Business leaders say, “I delegate the technical stuff. That’s why I have data scientists.” So the faucet and the hose never connect.
His reframe is the line I wrote down first:
These are consulting gigs, not technology installs.
An AI project is a business improvement project that happens to use machine learning. The model isn’t the product. The changed operation is.
My take: I see the same pattern with AI tools in general, not just predictive models. A company buys a tool or builds a chatbot, and six months later nobody uses it, because nobody redesigned the workflow around it. The technology works and the organization doesn’t change.
How Predictive AI Actually Creates Value
Siegel explains predictive AI in a way that stripped the mystery out of it for me. The mechanics are simple:
- Take a list of individuals: customers, patients, transactions, even train wheels or satellites.
- Give each one a probability of some outcome (will they buy, cancel, default, commit fraud).
- Rank the list from most to least likely.
- Draw a line, called the decision threshold, and act on everyone above it.
That’s all of it. Insurance companies have always worked this way, pricing risk for each individual. Predictive AI applies the same idea to marketing, fraud, logistics, healthcare, and spam filtering. Siegel calls it the antidote to information overload, because it triages what deserves attention.
He also stresses that this is different from forecasting. Forecasting asks how many ice cream cones we’ll sell. Predictive AI asks who is most likely to buy one.
The “Prediction Effect”: A Little Prediction Goes a Long Way
Many people assume prediction is impossible, so how can anyone build a business on it? Siegel’s answer is the part I found most useful. You don’t need to predict with certainty. You only need to predict better than guessing.
His back-of-the-napkin example:
- You have 1 million prospects.
- Contacting each costs $2, so mass marketing costs $2 million.
- Only 1% respond, and each response is worth $220, so you earn $2.2 million.
- Profit: $200,000.
Now suppose a model identifies the top 25% of prospects, and that group responds at 3% instead of 1%. That’s a “lift of three,” meaning three times better than guessing.
- You contact only 250,000 people, saving 75% of the cost.
- Profit jumps to about $1.15 million, more than five times higher.
And Siegel points out that this model is not impressive. It is nowhere near certain about any individual, and a 3% response rate is still low. But in a numbers game, tipping the odds slightly in your favor changes the bottom line enormously. He jokes that predictive analytics is “a skunk with bling,” and he calls this the prediction effect.
My take: This is a good antidote to perfectionism. I’ve seen people (and I’ve been one of them) wait for a “perfect” system before launching anything. Siegel’s math shows that the value is in being usefully better than the baseline, not in being right every time.
The Three Questions Every Stakeholder Must Understand
Siegel says the semi-technical knowledge that business people need fits into three questions:
- What is predicted?
- How well is it predicted?
- What is done about it?
He argues this is much more accessible than high school algebra, and that business stakeholders must understand it. His line: if the stakeholder doesn’t get their hands dirty, their feet will get cold. Leaders who don’t understand the project won’t approve deployment when the moment comes.
His analogy: you don’t need to know where the spark plug is to drive a car. But you do need to understand momentum, friction, and the rules of the road. Business leaders need that level of understanding of ML, not the engine internals.
Be Specific: Define the Prediction Precisely
“Will the customer buy?” is not a usable question. Siegel’s example of a usable one:
If we send them a brochure, will they make a purchase of at least $125 after shipping, within 13 days, and not return it for a refund within 45 days?
Real business questions carry caveats and qualifiers, and he says there are often at least three times as many as you’d first expect. Data scientists cannot work these out alone in a cubicle. It takes deep collaboration, because the action you take only makes sense if you’re predicting exactly the right thing.
My take: This is the same skill as good discovery in sales. In B2B, a vague problem produces a vague solution. Whoever asks the sharper question wins.
Metrics: Stop Talking About Accuracy, Start Talking About Money
This was my favorite section. Data scientists usually evaluate models with technical metrics such as precision, recall, and area under the curve. Those only tell you how well a model performs in the abstract, compared with random guessing.
Business people need absolute business value: profit, revenue, savings, returns.
Siegel says that in over 30 years in the field he has seen profit curves only about three times outside his own slides. These curves show how profit changes as you move the decision threshold, and they usually have a Goldilocks zone: not too much, not too little.
Business value also depends on how you deploy the model, so you can’t say a model is “worth a million dollars” on its own. The same model produces different profits depending on cost per contact, where you draw the line, and what business conditions apply. A stakeholder might prefer to break even while marketing to 73% of the list, rather than maximize profit. That is a business decision, not a technical one.
My take: If I’m explaining AI to a client, I should never lead with “the model is 92% accurate.” I should lead with “this saves 10 hours a week” or “this recovers X dollars.” That is the language decision-makers use.
The Six-Step BizML Playbook
Siegel calls his framework BizML. It has six steps, and the first three are the ones people skip:
- Establish the deployment goal. What is predicted, and what will be done about it? Start from the end and plan in reverse.
- Define the prediction goal. Make it specific, as in the brochure example.
- Define the metrics. Both the technical and the business metrics, and where they need to land before you deploy.
- Prepare the data.
- Train the model.
- Deploy the model.
Steps 4 to 6 have been the same since the 1960s, when credit scoring and targeted marketing began. Training the model is the “rocket science,” but deployment is the “paydirt.” Everyone is excited about the rocket science, while the real value comes from the launch.
The UPS Story: What Success Looks Like
Siegel’s lead example is UPS. At each shipping center, they must assign thousands of packages to roughly 50 trucks overnight, but they only have partial information, because some packages haven’t arrived yet. So they add predicted packages to the known ones, which gives them a reliable picture of tomorrow’s deliveries. Paired with ORION, their route-optimization system, the results were:
- 185 million fewer miles per year
- $350 million in savings
- 8 million gallons of fuel saved
- 185,000 metric tons fewer emissions
Even UPS had battles getting it approved and deployed, and had to backtrack at one point. If a company that capable struggled, that tells you how hard the last step is.
The Quiet Hardest Part: Data Preparation
In the Q&A, Siegel says the biggest technical bottleneck isn’t the model but the data. Getting data into the right format is hard, and one reason is the time leak problem. For each row of training data, you can only include what you knew at the moment you would have wanted to predict, nothing influenced by later events. Reconstructing the past that cleanly is difficult, and one workaround is to log snapshots over a couple of weeks as things happen.
His summary: the fun part, creating the model, is not nearly as challenging as getting the data clean and meaningful in the first place.
On Hype: “AI First Means Using AI Last”
Siegel is blunt about the hype. He argues that:
- “AI’s original sin was calling it AI.” The name invites anthropomorphism and fuels the idea of human-level machines that you could onboard like employees.
- Nobody asks “what’s your Excel strategy?” Starting from the tool instead of the problem is solutionism: to a hammer, everything looks like a nail.
- His advice: fall in love with the problem, not the solution.
- He thinks generative AI is overhyped “by a factor of 10 or 20,” but that 10% of the hype would still be very valuable.
- He doesn’t believe current progress is a concrete step toward artificial general intelligence.
You can disagree with some of this, and I think reasonable people do. But the practical advice holds whatever you believe about AGI: define the value first.
Generative AI: Same Rules, Rarely Measured
Siegel’s framework is built for predictive AI, but he says the broad principle applies to generative AI too: know from the start how it will deliver value, what operation it will change, and how you’ll measure it. He notes that almost nobody measures this, and gives the rare public examples:
- Ally Financial cut the time spent creating marketing content by 34%.
- A Fortune 500 software company used generative AI to pre-draft customer service replies, improving issues resolved per hour by 14%, and by 34% for novice and low-skilled workers.
Two things stand out. The gains are real but modest, “definitely not nothing” but not a robot doing everything. And generative AI usually needs a human to proofread the output, whereas predictive AI, such as fraud approval, can run fully autonomously.
My Key Takeaways
After watching this, here is what I’m taking with me:
- Deployment is the product. A model that sits unused has produced zero value. Plan for adoption from day one.
- Start from the end. Define what action will change before choosing any tool.
- Speak in business metrics. Profit, savings, and time saved beat accuracy scores every time.
- Good enough beats perfect. A modest lift applied at scale can transform results.
- Be painfully specific about what you’re predicting or automating.
- Non-technical stakeholders need a semi-technical education. This is exactly the gap Siegel calls his “hill to die on,” and I think it’s a real opportunity for anyone who can explain AI clearly to business teams.
- Measure it. Most organizations don’t, which is why the success stories are so rare.
The talk is from a predictive-analytics expert, but its message applies to anyone working with AI in business, including people deploying chatbots, automations, and AI tools in everyday operations. Technology is rarely the bottleneck. Understanding, collaboration, and follow-through are.
If you work with AI in any way, I recommend the full talk: https://www.youtube.com/watch?v=gnVk6ztz6Bs
