Use of AI triage outside its validated population or purpose
ChainBow-tie
- Status
- Approved
- Theme
- Foreseeable use of the system outside its intended purpose
- Owner
- Deploying organisation
- Linked
- INC-014CAPA-008Plan item P-02NHS DCB portal
- Initial risk
- Major severity × medium likelihood, score 12
- Affects
What could go wrong
- 1The intended purpose and exclusion criteria of the Infermedica MGP are not made clear to patients at the point of entry, so excluded groups still attempt triage.
- 2Nothing in the GP Triage workflow actively prevents use by patients outside the validated population, such as young children, proxies entering data for others, or patients in crisis.
- 3Practices under local pressure extend GP Triage to pathways such as urgent same-day or safeguarding triage that it was not configured or assessed for.
- 4Patients and administrative staff assume the AI triage covers all presentation types, including red-flag or mental health emergencies, because no clear scope statement is shown.
- 5Automated routing or booking without prospective clinician review acts on an out-of-scope triage outcome as if it were valid, without a human check.
- 6Staff onboarding new appointment integrations are not trained on which patient groups should bypass AI triage entirely.