AI classification acceptance tests: defining success beyond the demo

Before commissioning AI that classifies enquiries, documents or requests, define a correct outcome and the response to insufficient information. A model name or implementation method is not an acceptance test. Request examples evaluated separately from development, a breakdown of error types and conditions for human review. This lets you compare development proposals against the work the system must perform, rather than only the demonstration the supplier chooses to present.

What does bunq illustrate about methods and outcomes?

In October 2022, the Dutch tribunal published a summary of its bunq decision. According to that summary, the regulator had not proved that certain screening methods based on data analysis and AI violated the law, while other violations were established. DNB’s response refers, among other matters, to investigating the source of funds. This describes a historical decision, not general approval of AI systems or legal guidance for another project.

The product lesson we take from the draft is that defending a method does not replace checking the required outcome. A supplier may explain its model choice convincingly and still need to demonstrate that the system handles your cases. Separate technology selection, evidence of output quality and the decision about permitted downstream actions in the specification. These questions are related, but a satisfactory answer to one does not settle the others.

Define what each classification permits

In a hypothetical support-enquiry classifier, the label “urgent” is insufficient. Define what makes an enquiry urgent, who can decide and which queue receives it. If the system merely suggests a label to an employee, an error has different consequences from closing a request or sending a commitment to a customer. Describe the business action following classification and how it is stopped when the result is uncertain.

Include borderline examples alongside easy ones: a brief message without an order number, an enquiry in another language or two separate requests in one message. Decide whether multiple labels are allowed and how conflicting information is handled. Sometimes “needs clarification” is the right result. A specification requiring a category in every case can encourage a confident answer precisely where there is insufficient basis for a decision.

Keep evaluation cases separate from development examples

Collect examples representing expected work, using only information the team is permitted to process. You can begin with synthetic cases and later examine examples with identifying details removed, as appropriate. Retain the input, expected result and a short explanation for each example. Separate cases used for development and improvement from a reserved evaluation set. Otherwise, it is difficult to distinguish solving the task from repeatedly adapting to familiar examples.

When two employees disagree about the right label, do not conceal that disagreement inside an average score. The business definition itself may be unclear. Assign a decision owner, document the rule and decide whether the case is suitable for automation at all. Evaluation sets also need a version and date: a change in service policy may change the expected answer without any change to the model. Preserve that distinction to compare results fairly over time.

Separate missed cases from false alarms

Overall accuracy is not enough. Missing an urgent enquiry and sending an ordinary enquiry to the urgent queue create different work. Ask how many examples of each type were included, correctly classified, missed or incorrectly escalated. Include the number referred for clarification. This prevents an impressive aggregate score from hiding a small category the system handles poorly.

There is no universal threshold for every product. With the process owner, identify which errors block use, which need review and which can be corrected later. Begin with business consequences and employees’ ability to check the result. If most cases require manual review, measure the additional staff time too. A cautious system may be useful, but it should not be described as fully automated when people still perform the work.

Include the review path in acceptance

Request a demonstration where the system declines to classify and the employee receives what they need to decide. Who sees the case, when must it be addressed and what happens if nobody is available? Define what is retained after a correction and how duplicate actions are prevented. Every correction need not automatically become training material: first establish permission, approval of the decision and suitability for the intended use.

Julius’s article-publishing tool produces a quality score, warnings and a decision about whether content meets publication conditions. The runbook additionally requires source review and public-page verification. This is a documented example of a rule-based gate, not an accuracy measurement for a customer’s AI classifier. It illustrates why a score and a passing rule need explicit boundaries: passing one check does not establish that the entire business outcome is complete.

What should a software team receive before quoting?

Add label definitions, borderline examples, a disagreement owner and permitted actions to the technology product specification. Ask the proposal to itemise evaluation-set preparation, error review, the clarification workflow and repeat checks after model or instruction changes. Replacing a component does not automatically require a full relaunch, but you need to know which checks establish that important behaviour is preserved.

Our guide to choosing AI developers covers team selection; the narrower question here is what constitutes evidence that a particular classifier fits your workflow. When approaching Julius for software development, bring easy cases and cases your employees struggle to resolve. Start with a bounded trial, compare against the existing process and identify where automation adds value before connecting it to actions that are difficult to reverse.