Where Should the AI Actually Go?

I have been working on an AI product in the legal domain, and one question has taken much more time than I expected:

Where should the AI actually go?

Not which model to use. Not whether to use RAG or an agent. I mean where, inside the real process, should we allow a model to make a judgment?

At first, this did not look like a separate problem. We had a workflow. Parts of it required reading legal documents and making sense of information spread across them. An LLM could help with that, so the natural next step was to put the workflow in front of a capable model and improve the prompt until the result looked right.

That worked well enough to be convincing. It did not work well enough to be dependable.

When a result was wrong, it was hard to say why. Maybe the model missed a fact in a document. Maybe it connected two facts that should have remained separate. Maybe it had the right facts but applied the wrong reasoning. The final answer looked like one output, but too many different things had happened inside the same model call.

We kept looking at the model. The more useful question turned out to be whether we had given the model the right job.

This felt familiar

Earlier in my career, when most of the work was what we now call traditional ML, problem framing could decide the entire project.

A request such as "predict which customers will leave" was not yet a machine learning problem. We had to decide what "leave" meant, how far ahead we wanted to predict it, what information would exist at prediction time, and which mistakes the business could tolerate. A small change in any of those decisions could turn a workable project into an impossible one.

The first serious work was not model training. It was finding a version of the problem that the available data could answer.

Sometimes, after doing that work, the answer was that we did not need ML. A few rules or a database query were enough. That was not a failure. It meant the problem had been understood properly.

LLMs make it easy to skip this stage. Their input and output are both language, and they can attempt almost any task we describe. The model nearly always gives us something. A demo can therefore look successful before we have decided what the model should be responsible for.

The old framing problem has not gone away. Part of it has moved into the architecture.

Capability is not the same as responsibility

Suppose a model can read a set of documents, pull out relevant facts, compare them, apply a rule, and write a conclusion. That does not mean it should own all five steps in production.

Some parts of a process are uncertain because human language is messy. The same name may be written in different ways. A fact may be buried in a paragraph instead of placed in a clean field. Two documents may appear to conflict until their context is considered. Models are useful here because the task involves interpretation.

Other parts are certain once the facts are known. A date is before or after another date. A required field is present or missing. An identifier matches or it does not. A known rule should produce the same result every time.

Using an LLM for those steps adds uncertainty without adding much intelligence.

I think this is where we initially made the problem harder than it needed to be. We were asking what the model was capable of doing. Since the answer was "quite a lot," more of the process kept moving into the model.

But every extra responsibility made the result harder to inspect. A better model improved the output, but it did not make the boundary clearer.

What helped was going through the process step by step and separating two questions:

• Where do we genuinely need interpretation?
• Where do we need the system to behave exactly the same way every time?

The model belongs much more naturally in the first category. Normal software is still better at the second.

Why this matters more in law

In a casual AI product, a plausible answer may be useful even when it is not perfect. Legal work has a different standard. A conclusion may affect a real case, a transaction, or a decision made by a lawyer. "The model usually gets it right" is not enough.

The result needs a trail. We need to know which document a fact came from, what the system did with it, and where judgment entered the process. If a lawyer disagrees with the output, the system should make that disagreement productive. It should be possible to inspect the evidence and the intermediate decisions instead of arguing with a paragraph generated at the end.

That becomes difficult when the AI layer covers the whole process.

It becomes easier when the model has a narrower role. The model can interpret the messy part and return structured evidence. Code can handle checks that have a fixed answer. Another model call may be useful later if a remaining question really needs contextual reasoning. The lawyer can review the conclusion together with the evidence behind it.

This is still an AI system. It may even use the same model as the earlier version. The difference is that the model is no longer treated as the process itself.

Once we thought about the system this way, several things became easier. We could test extraction separately from reasoning. We could tell whether missing evidence came from retrieval or document understanding. We could change a deterministic rule without touching a prompt. We could also see which model calls were not helping and remove them.

The reliability gain did not come from one clever prompt. It came from deciding where probabilistic behavior was acceptable.

The wrong layer can hide behind a good demo

This is the part I now worry about when I see AI demos, including my own.

A demo answers: can the model produce a good result for this example?

A product has to answer harder questions. When the result is wrong, can we locate the failure? Can we rerun one part without rerunning everything? Can we enforce the rules that are not open to interpretation? Can a human review what matters without reading the entire input again?

If the answer to those questions is no, changing the model may not solve the real problem. The AI may simply be sitting too high in the process, with too much hidden responsibility.

There is a temptation to solve this by creating more agents: one to extract, one to reason, one to verify, one to judge the others. Sometimes that separation is useful. Sometimes it only divides one vague AI box into several smaller vague AI boxes.

The boundary is not "one model versus many models." It is between work that benefits from probabilistic judgment and work that should remain explicit.

The question I ask now

Before thinking about prompts or model providers, I try to draw the existing human process as it is. Not the AI version. The boring version: what comes in, what people look for, which decisions they make, and what they do when the information is incomplete.

Then I look for the point where ordinary software stops being enough.

That point might be document interpretation. It might be matching information written in inconsistent ways. It might be reasoning over evidence that cannot be reduced to a stable set of rules. Wherever it is, that is a candidate for the AI layer.

The rest of the process does not have to become AI-shaped just because one part does.

In traditional ML, framing decided whether we had a problem the model could learn.

With LLM products, we have another framing decision to make. We have to decide what part of the process deserves a model, what part needs ordinary code, and where a human must remain visible.

A model being able to do the whole task is not a good enough reason to let it.