The model is not the product

Say “AI in healthcare” and people picture the model. But the model is a small part of any serious health product.

What surrounds it decides safety: clear limits on what it may do, a validation gate before any output, a review trail and escalation to a human.

In short:

• The model organises and suggests. It is an assistant, not a source of decisions.
• Four things built around it decide whether a health product is safe: limits, a validation gate, a review trail and a way to hand over to a person.
• The question to ask is not “how smart is the model?” but “if it is wrong, what catches it?”

The model sits in the middle. Safety is built around it.

What makes AI in healthcare safe?

An AI health product is safe when its design assumes the model will sometimes be wrong, and something other than the model is there to catch it. In practice that means four things: limits on what the model may do, set before it is used; a validation gate that checks every output before anyone sees it; a review trail that keeps each step; and a person who decides when the system is unsure.

A capable model without those four is a demonstration, not a health product. Below: what each one catches, and the questions to ask of any AI health tool, including ours.

Why the model is the small part

A model can be impressive in a demonstration and still be unsafe in a clinic. A demonstration shows what a model does when it is right. A health product has to be designed for the moment it is wrong: a misread word, a missing detail, an answer given with more confidence than it deserves.

Those moments are not rare accidents. They come with how language models work. The same question can come back with a different answer. A summary can contain a detail nobody said. A phrase in a dialect the model rarely met can be read as something else. The World Health Organization’s 2024 guidance on large multi-modal models, the kind of generative AI behind today’s chat tools, points to documented risks of “false, inaccurate, biased, or incomplete statements” that could harm people who use them to make health decisions.

There is a regional reason too. WHO’s 2021 report on AI in health warns that systems trained mainly on data from high-income countries may not perform well in low- and middle-income settings, and asks that AI be designed for the diversity of health-care settings it will meet. A product built for the Middle East and Africa, in Arabic and its dialects, should expect its model to meet words it saw little of in training, and be built for that.

That is why the work that matters most sits outside the model. It decides what the model is allowed to touch, what happens to its output before anyone sees it, and who steps in when it is unsure.

Four things around the model

1. Clear limits on what it may do. Set before it is used, not discovered after. At Medigize, the first limit is that no part of the product gives a diagnosis; we wrote the limits down in three things we won’t build.
2. A validation gate before any output. Every output is checked by something that is not the model, so the check can catch the model being wrong.
3. A review trail. What happened at each step is kept, so it can be looked at again.
4. Escalation to a human. When the system is unsure, a person decides, and urgency goes up rather than down.

Each one answers a different failure, which is why none of them can stand in for another:

Around the model	What it does	What it is there to catch
Clear limits	Says in advance what the tool may and may not do	A tool drifting into decisions that belong to a clinician
A validation gate	Checks every output before it is shown	An output that breaks a limit
A review trail	Keeps what was asked, answered and shown	An error nobody can trace, and so nobody can fix
Escalation to a human	Hands the case to a person when the system is unsure	A guess presented as a decision

A limit that nothing checks is only a wish, which is why the gate matters as much as the limits. The trail is what makes the whole system correctable. Patient safety work treats incident reporting as a way to learn rather than to blame: WHO lists systems for reporting incidents “for learning and continuous improvement” among the features of a safe health system. A review trail does that job for software. Without it, a wrong output is an anecdote. With it, it is a step someone can find, understand and change.

An engineer’s view

We treat the model as an assistant that organises and suggests, not as a source of decisions. Any sensitive output is checked before it reaches anyone.

Each job has its own place, and the model has only one of them.

One habit follows directly from that sentence: the check is never the model marking its own work. A check is only useful if it is able to disagree with what it checks.

The World Health Organization’s guidance on artificial intelligence for health puts the same idea first among its six principles: humans should remain in control of health-care systems and medical decisions.

Read the other five with this article’s question in mind and most describe the surroundings, not the model: quality control in practice, information documented before an AI technology is deployed, ways to question decisions based on algorithms and to seek redress, and continuous assessment during actual use.

Medical device regulators reached the same point from another direction. One of the ten guiding principles for machine learning in medical devices, from the US Food and Drug Administration, Health Canada and the UK’s Medicines and Healthcare products Regulatory Agency, asks for a focus on the performance of the human-AI team, “rather than just the performance of the model in isolation”. It was written for medical devices, but the lesson travels.

The same 2024 WHO guidance adds a warning about the last layer. It describes “automation bias”: health workers and patients overlooking errors they would otherwise have caught, or handing a difficult choice to the machine. A person in the loop only helps if that person is really deciding.

A worked example: one message, four layers

Here is how the four layers meet a single message. It is an illustration, not a real patient, and it leaves the clinical details vague on purpose.

1. The message arrives. Someone writes, half in dialect and half in English, that a pain began a few days ago, that it is getting worse, and that they took “the usual tablet” for it.
2. The model organises and suggests. It sorts what is known (a complaint, roughly since when, getting worse) from what is missing (where exactly, which tablet), and suggests the next question.
3. The limits hold, whatever the model made of it. The tool may not name a condition, and it may not say this is probably minor.
4. The gate checks the output. Before the next question or any summary reaches the person, something that is not the model checks it against those limits. An output that fails does not go out.
5. The trail keeps it. What was asked, answered and shown is kept, so if a question went wrong, someone can find where and why.
6. A person decides when the system is unsure. If the answers leave real doubt, a person decides, and the urgency goes up, not down.

Notice what the model did in that story: it organised, and it suggested. Everything that made the exchange safe happened around it. And if an answer had pointed to a warning sign that needs an ambulance, the next step would have been the call, not more typing.

Common mistakes when judging an AI health product

• Judging it by the demo. A demonstration is chosen to go well. Ask what happens on the day it goes badly.
• Trusting a confident tone. Language models write fluent, certain sentences whether they are right or wrong. Confident wording is not evidence.
• A human in the loop who only clicks “approve”. A person asked to approve a stream of outputs, with no time and no reason to doubt them, is set up for the automation bias WHO warns about. That is not oversight.
• Adding the safety after launch. Limits set after something went wrong are a repair. Limits set before use are a design.
• Quoting the model’s score as the product’s. A model tested on its own says little about how the whole system behaves with real people and real workflows. That is why we make no claim about impact before it is measured, against a starting point measured first.

Questions to ask before you trust an AI health tool

The question underneath is the same for everyone: if it is wrong, what catches it? It just sounds different from each side.

If you are a patient or a family member:

• Does it ever tell me what I have? It should not. A diagnosis needs a clinician who can examine you.
• When it is not sure, does it say so, and who takes over?
• Does it ever tell me something can wait? A navigation tool should not reassure you about something that needs examination.
• Does anything I tell it move without my consent? It should move only with your agreement, as we explain in a record that moves with you.

If you run a clinic or a hospital:

• Ask for the limits in writing: what the tool will never do, decided before it was built.
• Ask what checks each output before it reaches a patient or a clinician, and whether that check is separate from the model.
• Ask to see the trail for one case, from the first message to the hand-over. If nobody can show it, nobody can correct it.
• Ask what has been measured, how, and against what starting point. A number with no method is not a claim to rely on.
• Ask what would change if the model were replaced by a newer one. If the limits, the gate, the trail and the hand-over would all stay where they are, the safety lives in the product.
• Ask whether it works on top of the systems you already use, or asks you to replace them. Our own answer is in the real competitor is WhatsApp and paper.

The model, or what surrounds it?

Six parts of an AI health product. Decide which belong to the model, and which to what is built around it:

The takeaway

The key question is not “how smart is the model?” It is “if it is wrong, what catches it?” That is what we build around the model.

Ask any AI health product which of the two it is.

It is a question we invite you to ask us too. We listed it, with others, in why we build in public. And for what a navigation tool can and cannot do for a patient, see can an app tell you if it’s an emergency? For where this work sits around treatment, see what care operations means.

Frequently asked questions

What is the difference between an AI model and an AI product?

The model is the part that reads and suggests. The product is everything around it: the limits on what it may do, the checks on its output, the trail of what happened, and the people who decide.

Can an AI model diagnose?

Medigize’s tools do not diagnose. Diagnosis belongs to a clinician who can examine the patient.

What is a validation gate?

A check, outside the model, that every output passes before it reaches anyone.

What does “human in the loop” mean in healthcare AI?

That a person, not the model, makes the decisions that matter, with the time and the information to disagree with it. WHO puts humans in control of medical decisions first among its six principles for AI in health.

Is a more accurate model automatically safer?

No. A score measured on the model alone says little about how the whole system behaves with real people. Regulators of AI in medical devices ask for a focus on the human-AI team, not the model in isolation.

Why does Medigize say AI-assisted?

Because the model assists. It organises and suggests; the clinician always leads.

What happens when the system is unsure?

A person decides, and when in doubt, urgency goes up, never down.

Medigize is a non-diagnostic navigation tool. This is general information, not medical advice. In an emergency, call an ambulance or go to the nearest emergency department.

References

1. World Health Organization: WHO issues first global report on AI in health and six guiding principles for its design and use (2021)
2. World Health Organization: Ethics and governance of artificial intelligence for health (2021)
3. World Health Organization: WHO releases AI ethics and governance guidance for large multi-modal models (2024)
4. Medicines and Healthcare products Regulatory Agency, US Food and Drug Administration and Health Canada: Good machine learning practice for medical device development: guiding principles (2021)
5. World Health Organization: Patient safety (fact sheet)
6. World Health Organization: Global Patient Safety Action Plan 2021–2030