How we measure whether an AI feature is worth keeping

Building an AI feature is the easy part now. The harder discipline is deciding, a few weeks in, whether it actually deserves to stay. Plenty of features that demo beautifully turn out to cost more attention than they save. We try to know which is which, with numbers rather than vibes.

The first question is always about time, and on both sides of the ledger. How much time does the feature save the person doing the work, and how much time does it add in review and correction. People remember the saving and forget the review. A draft that takes two minutes to fix when it used to take ten to write is a real win. A draft that takes nine minutes to fix is theatre. You only find out by watching real use, not by asking how people feel about it.

The second question is accuracy in the way that matters for this job, not in the abstract. For an extraction feature, what share of fields are right enough to confirm without changing. For a classifier, how often the routing is correct, and just as importantly what the wrong cases cost. A misrouted newsletter is nothing. A misrouted complaint is something. We weight errors by what they actually do, not by counting them equally.

The numbers we watch

A few measures tell us most of what we need.

• Time saved per item, net of review time, measured on real work rather than estimated.
• Accuracy at the level of trust, meaning the share of output a person accepts unchanged.
• Override rate over time, because if people increasingly correct or ignore the AI, they are voting against it.
• Running cost per item, including the model bill and the human review behind it.

That last point matters. The cost of an AI feature is rarely the model. It is the review time around it. A feature can be accurate and still not worth keeping if every output needs a careful human pass. We count that honestly or we are fooling ourselves.

Keep, fix, or retire

With those numbers in hand the decision is usually clear. If the net time saved is real and the override rate is low and falling, keep it and look for the next workflow. If accuracy is fine but review is heavy, the fix is often in the interface, making confirmation faster, not in the model. If people are quietly working around it, that is the strongest signal of all, and we would rather retire a feature cleanly than let it linger as dead weight.

We bake this into how we work from the Discovery Sprint onward. The prototype is not just to prove the thing can be built. It is to start gathering the honest numbers that tell us whether it should be. A feature that cannot show its value after a fair trial does not get to stay on sentiment. That rule has saved our clients money, and it has saved us from defending things that were never really working.