CH 1 · CHALLENGE OF THE AGES 3
regulatory writer composing a submission narrative) begin to sit partially
with the system. Not because anyone designed it that way, but because the
system has now done most of the thinking before the human looks. The
reviewer's role shifts from deciding to agreeing or not agreeing, and that is a
different cognitive job, with different failure modes. That quality unit,
presumably asked to certify outputs they had never been the principal
author of, made that kind of mistake.
The speed of decisions accelerates beyond the speed of governance. A
traditional quality management system (QMS) was designed around
decisions made by humans at human pace. A deviation reviewed in days.
A corrective action closed in weeks. A change control completed in
months. AI systems, left to themselves, will produce dispositions in
seconds. If governance can only react in days, the mismatch creates a
structural gap into which anything can fall. The fix is not slowing the AI.
The fix is making governance run at the same speed.
Accountability dissolves into the network. With a traditional system,
when a decision goes wrong, you can trace the fault. You can identify the
reviewer, the standard operating procedure (SOP) they followed or failed
to follow, the training they had or lacked, the data they could or couldn't
access. With an AI-mediated workflow, that traceability evaporates unless
you have built it back in deliberately. Whose decision was the missed
adverse event signal? The model's? The reviewer who agreed with the
model? The data scientist who fine-tuned the model? The vendor who
supplied the foundation? The QMS owner who validated the system?
Without architecture, the answer is no one, and no one is not an acceptable
answer. Regulators have been clear: it must be a named, qualified,
accountable human, every time, and the architecture has to enforce it.
This is why the phrase human in the loop, useful as it has been, is not enough. A
loop describes someone watching from the side, intervening when something
looks wrong. That metaphor was sufficient for systems in which the human did
most of the work and the machine assisted. It is insufficient for systems in
which the machine does most of the work and the human reviews. The geome-
try has inverted, and the language has not caught up.
Read together, the regulations describe a different commitment, one this book
reads as placing the human not in the loop but in the core (human-centered).
Structurally, not procedurally. To give this commitment a name for the pur-
poses of this book, we will call the operational form it takes the Round Table: hu-
mans and agents seated as peers in a single forum, contributing through de-
fined roles, sharing the same context and lineage. Accountability for outcomes
regulatory writer composing a submission narrative) begin to sit partially
with the system. Not because anyone designed it that way, but because the
system has now done most of the thinking before the human looks. The
reviewer's role shifts from deciding to agreeing or not agreeing, and that is a
different cognitive job, with different failure modes. That quality unit,
presumably asked to certify outputs they had never been the principal
author of, made that kind of mistake.
The speed of decisions accelerates beyond the speed of governance. A
traditional quality management system (QMS) was designed around
decisions made by humans at human pace. A deviation reviewed in days.
A corrective action closed in weeks. A change control completed in
months. AI systems, left to themselves, will produce dispositions in
seconds. If governance can only react in days, the mismatch creates a
structural gap into which anything can fall. The fix is not slowing the AI.
The fix is making governance run at the same speed.
Accountability dissolves into the network. With a traditional system,
when a decision goes wrong, you can trace the fault. You can identify the
reviewer, the standard operating procedure (SOP) they followed or failed
to follow, the training they had or lacked, the data they could or couldn't
access. With an AI-mediated workflow, that traceability evaporates unless
you have built it back in deliberately. Whose decision was the missed
adverse event signal? The model's? The reviewer who agreed with the
model? The data scientist who fine-tuned the model? The vendor who
supplied the foundation? The QMS owner who validated the system?
Without architecture, the answer is no one, and no one is not an acceptable
answer. Regulators have been clear: it must be a named, qualified,
accountable human, every time, and the architecture has to enforce it.
This is why the phrase human in the loop, useful as it has been, is not enough. A
loop describes someone watching from the side, intervening when something
looks wrong. That metaphor was sufficient for systems in which the human did
most of the work and the machine assisted. It is insufficient for systems in
which the machine does most of the work and the human reviews. The geome-
try has inverted, and the language has not caught up.
Read together, the regulations describe a different commitment, one this book
reads as placing the human not in the loop but in the core (human-centered).
Structurally, not procedurally. To give this commitment a name for the pur-
poses of this book, we will call the operational form it takes the Round Table: hu-
mans and agents seated as peers in a single forum, contributing through de-
fined roles, sharing the same context and lineage. Accountability for outcomes