CH 2 · THE REGULATORS HAVE SPOKEN 11
F O U N D A T I O N 2
The Regulators Have Spoken
The regulators have written what the architecture has to satisfy.
2.1 Regulators Are Clear
There is a moment in every transformation when the executive picks up the
regulations directly and reads them alongside everything else. It is usually a
clarifying moment. The regulators have been working on the AI question for
some time, and their writing has the specificity, plainness, and durability that
we'd hope for. When you read the AI guidance from the U.S. FDA's Center for
Devices and Radiological Health alongside the EU AI Act, the European
Medicines Agency (EMA) Reflection Paper, the International Medical Device
Regulators Forum (IMDRF) N88, and the U.S. FDA, Health Canada, and the UK's
Medicines and Healthcare products Regulatory Agency (MHRA) Good Machine
Learning Practice (GMLP) guiding principles, you do not encounter five differ-
ent futures. You encounter one future, described five times.
Consider the most consequential single sentence in any of these documents,
Article 14 of the EU AI Act, the human oversight article. It requires that high-risk
AI systems be designed and developed in such a way that they "can be effectively
overseen by natural persons during the period in which they are in use." Now, even
more important than the phrase itself, are the words designed and developed ear-
lier in the sentence. The Article does not say that human oversight must occur.
It says that the system must be designed for human oversight. It is an architec-
tural requirement, not a procedural one. You cannot retrofit it by writing an
SOP.
The Article goes further. It specifies what that design must enable. The natural
person must be able to understand the capacities and limitations of the system,
to remain aware of automation bias, to correctly interpret the system's output,
to decide not to use the system in any particular situation, to override or re-
verse a decision the system has made, and, in the most architecturally specific
F O U N D A T I O N 2
The Regulators Have Spoken
The regulators have written what the architecture has to satisfy.
2.1 Regulators Are Clear
There is a moment in every transformation when the executive picks up the
regulations directly and reads them alongside everything else. It is usually a
clarifying moment. The regulators have been working on the AI question for
some time, and their writing has the specificity, plainness, and durability that
we'd hope for. When you read the AI guidance from the U.S. FDA's Center for
Devices and Radiological Health alongside the EU AI Act, the European
Medicines Agency (EMA) Reflection Paper, the International Medical Device
Regulators Forum (IMDRF) N88, and the U.S. FDA, Health Canada, and the UK's
Medicines and Healthcare products Regulatory Agency (MHRA) Good Machine
Learning Practice (GMLP) guiding principles, you do not encounter five differ-
ent futures. You encounter one future, described five times.
Consider the most consequential single sentence in any of these documents,
Article 14 of the EU AI Act, the human oversight article. It requires that high-risk
AI systems be designed and developed in such a way that they "can be effectively
overseen by natural persons during the period in which they are in use." Now, even
more important than the phrase itself, are the words designed and developed ear-
lier in the sentence. The Article does not say that human oversight must occur.
It says that the system must be designed for human oversight. It is an architec-
tural requirement, not a procedural one. You cannot retrofit it by writing an
SOP.
The Article goes further. It specifies what that design must enable. The natural
person must be able to understand the capacities and limitations of the system,
to remain aware of automation bias, to correctly interpret the system's output,
to decide not to use the system in any particular situation, to override or re-
verse a decision the system has made, and, in the most architecturally specific