Does This Need AI at All?
Before picking a model or a framework, sort each decision in the workflow: a person, a plain rule, classical ML, or a generative model. Every wrong assignment has its own cost, and the costs run in both directions.
Most of the AI build advice I read starts one step too late. It compares models, frameworks and agent harnesses, and it assumes the problem already belongs to a model.
The earlier question is whether it does. Two sources in my research library put that question first. An IBM Technology explainer splits work four ways: a person, a rule, classical machine learning, or a generative model. Katie Bakewell, a data scientist at NLP Logix, gave Jake Van Clief the practitioner version in an interview: do the discovery before touching a tool.
I write embedded C++ for a living, so my bias runs toward code that does the same thing every time. To keep that honest, I wrote down what a wrong assignment costs in both directions.
Four owners, and they are not a ladder
The IBM video's most useful point: the four options are different kinds of problem solving, and none is a more mature version of another. Each comes with a bill.
| The decision needs | Owner | What that owner costs |
|---|---|---|
| Accountability, ethics, judgment under ambiguity | A person | Slow, expensive, hard to scale |
| Exact, stable, testable logic | Rules in plain code | Breaks when conditions change or the logic gets too complex to write down |
| A prediction from structured historical data | Classical ML | Needs monitoring for drift, weak outside the problem it was trained on |
| Interpretation or generation over unstructured input | A generative model | Output varies between runs, hard to test, no guarantee of correctness, expensive at scale |
The thing being sorted is a single decision, never the whole product. IBM's example is a spending-analysis tool. Code does the arithmetic, a statistical model finds the trends, and a language model writes the explanation. Asking whether that product "uses AI" tells me nothing about how to build it.
The questions that sort a task
This is the order I would ask them in, because the early ones can end the conversation before any design work.
- What is the return? Bakewell's rule is that a project with no stated return does not get built. For a business the return is more revenue, lower cost or lower risk. She estimates that about half the companies who come asking for AI are pointing it at the wrong problem. That is her estimate, not a measured figure.
- Are the steps fixed? If the process is always step one, step two, step three with no decision in between, it is software. In her words, an agent on that process "costs you more money than software would."
- Is it a prediction from structured history? Fraud, churn and forecasting are IBM's examples. That is classical ML.
- Is the input unstructured, and how wrong can the answer be? A generative model fits when flexibility matters more than exactness. I would write down the tolerated error before assigning the step.
- Who answers for the outcome? If the answer involves liability or someone's livelihood, a named person owns the decision even when the steps under it are automated.
What a wrong assignment costs
Reaching too far is the direction with the most stories attached.
Put an agent on a fixed process and Bakewell's argument applies: its decisions are non-deterministic, so it adds risk and cost with no better outcome. Jake Van Clief tells a version of this as a teaching story. A team of about 50 built an agent to move data from a warehouse into tickets, replacing a manual spreadsheet upload that already worked. In his telling the agents kept corrupting files, and the effort to automate got in the way of the goal.
Reaching too far inside ML looks different. Bakewell describes a firm that was handed debt data with no explanation and came back after roughly twelve weeks with seven neural networks ensembled by an eighth, taking 18 hours to process one file. The model had found that people who capitalize both letters of their state pay more often. The signal was an artifact of which customer the records came from. More modeling power applied to a problem nobody had understood produced a confident wrong answer.
The other direction is quieter. The IBM video says rules break down once conditions keep changing or the logic is too complex to define by hand, and it describes human judgment as expensive and hard to scale. So a team that refuses a model where the input really is messy language ends up maintaining a pile of special cases, or queueing everything behind a person. I have fewer stories for this side, which probably says more about who tells the stories than about which mistake is more common.
Where I would not lean on the sources
Bakewell works for a vendor. She reports that about 70 percent of her firm's projects reach production, and sets that against an MIT figure that about 5 percent of AI projects get past a pilot. The first number is self-reported and the second was cited secondhand in conversation. I have checked neither. The cautionary tales are anecdotes told by people whose business is the simpler approach.
The IBM video is an educational explainer with no benchmark behind it, and the four-way split is simplified. Optimization, search and simulation do not fit cleanly in any of the four boxes.
The sorting questions hold up without the numbers.
Moving work back out of the model
Bakewell adds a reverse pattern. When a model keeps performing the same operation, write code for that step and hand it back to the model as a tool. It is cheaper and traceable, and it means a step can start out generative and get frozen into a rule later.
My own research pipeline has this mixed shape. A model writes each video note, which is interpretation over a transcript. Verifying that every claim's timestamp anchor holds up is a pass-or-fail check, and the skill refuses to finish until it passes. The retrieval index is generated from note frontmatter, with no vector database. Whether I believe a source's number is still my call.
Key Takeaways
- Sort decisions, not products. One workflow can hold work for a person, a rule, a statistical model and a generative model.
- Ask for the return first and whether the steps are fixed second.
- An agent on a deterministic process adds cost and risk. A rule on messy, changing input turns into a pile of special cases.
- The production-rate figures in these sources are self-reported or secondhand. The heuristics do not depend on them.
Sources
- Jake Van Clief, "A Pagani, a Toyota, and Why 'Best' Is a Trap in AI" (interview with Katie Bakewell)
- IBM Technology, "Knowing When Not to Use AI: AI Agents vs Rules vs ML"
- Jake Van Clief, "200 to 48,000 YouTube Subscribers in Four Months (Afternoon Tea Session 5)"
The two questions I am putting in front of my next build before any code: what is the return, and are the steps fixed?