An AI agent that can finish a multi-step clinical task without a human watching is useful. It is also a problem if it gets the task wrong. The American Medical Association has now published a governance framework built around that tension, and its core demand is plain: if an agent performs a clinical function, someone has to own the outcome.
The framework, released this month, rests on one principle. Agentic AI tools that perform clinical functions must be backed by evidence and must keep performing at a high level. The AMA says evidence, authority, oversight and accountability “must remain aligned in service of better patient care.”
Four questions for every clinical agent
Rather than a long rulebook, the AMA offers four questions it will use when assessing clinical AI agents:
- What clinical function is the AI performing?
- What demonstrates that it can perform that function, and what happens if it fails?
- What should the AI be permitted to do, and under what conditions?
- Who is responsible for the risks and outcomes?
The approach is deliberately portable. The same questions apply to an ambient documentation tool, a diagnostic aid or a patient monitoring system. The AMA says it will use the framework in its work with other organizations, and it keeps physicians at the center of how these tools are built and deployed.
Why agents raise the stakes
Agentic AI differs from generative AI because it can act on its own and complete complex, multi-step tasks. That autonomy is the appeal. But it also means an inaccurate or biased decision could cause real patient harm before a clinician has a chance to step in. A chatbot that drafts a poor summary is one thing. An agent that acts on a poor conclusion is another.
Adoption is moving faster than supervision
The timing reflects how quickly the tools are spreading. In a recent survey of 250 U.S. healthcare leaders, about 28% said their organization has already deployed agentic AI, and 44% reported running pilots or proof-of-concept projects. A majority expect a transformative effect on clinical workflows.
The supervision numbers are less comfortable. Some 72% of respondents said agentic tools are occasionally implemented without IT approval. More than half named compliance or regulatory violations and unauthorized access to protected health information as major concerns. Shadow deployment of an autonomous tool inside a health system is a very different risk from an employee trying an unapproved app.
A regulatory gap that industry is trying to fill
Federal rules on health AI remain thin, so industry groups are writing the guardrails themselves. States are doing their own thing, too. Some are restricting the technology, while others are experimenting with AI-based care. According to Manatt, 47 states introduced more than 250 AI bills affecting healthcare in 2025.
For health system leaders, the AMA framework is not binding, but it is a practical checklist. Before an agent goes live, can you name its function, show its evidence, define its limits and identify who answers for it? If not, the tool probably isn’t ready. And with deployment already outpacing governance, that is a question many organizations will need to answer sooner rather than later.