169
Software as a Medical Device: Regulatory and Market Access Implications
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
In reality, people often tend to sway. If the first
impression is positive, humans tend not to see
when the AI makes mistakes or may forget or
forgive the AI. How we must design our devices
allowing users to calibrate their trust appropri-
ately and how we must educate them in their
first interactions with the device is an open field
of research.
The radiology domain can inspire. For
example, a radiology study showed that the
human-device combination’s accuracy might
improve when a computer-aided-detection
algorithm identifies more than one choice to
the radiologist.38 Offering multiple options can
maintain trust in the system and mitigate the
risks of overtrust by putting the human expertise
at work.
In other cases, the AI’s performance alone
might be better than the performance of the
human-AI team. Sometimes, it is necessary to
take the human out of the loop altogether to get
the best performance and reduce or eliminate
use error. For example, in a clinical diagnostic
application used to read quantitative Polymerase
Chain Reaction qPCR) assays, FDA requires
manufacturers to deactivate the possibility of
the molecular biologist intervening, because the
precision and reliability of the AI outperforms
that of the human-AI team.39 Taking the human
out of the loop takes away human variability and
mitigates the risk of a lab using fewer control
samples than required by the assay manufac-
turer. On the other hand, when AI is trained on
rare diseases with fewer datasets, it may require
humans to be in the loop to reach maximum per-
formance. Striking the right balance is important
and differs on a case-by-case basis. Most medical
device legislation requires manufacturers to
eliminate or reduce the risk of use error.40 Inap-
propriate levels of trust in automation may cause
suboptimal performance.41
Transparency and Explicability
Only by gaining the user’s trust will machine
learning devices find their way into the care
pathways. One way to gain confidence is to
ensure transparency, both in terms of the
organization that creates the AI and the AI
itself. Transparency is also useful to clarify the
liability, i.e., did the doctor or the user make a
mistake? Was it the AI, incorrect or unforeseen
input data, or a malicious actor, e.g., hackers or
disgruntled employees?
Transparency may also be needed (1) to
allow manufacturers to determine operating
parameters (when the device works or does not
work), limitations to the intended use, con-
tra-indications, inclusion, and exclusion criteria
for input data or (2) to enable debugging of the
system and detect potential issues of bias.
Transparent AI presents core decision-mak-
ing elements in an open, comprehensive, accessi-
ble, clear, unambiguous, and interpretable way.42
The first question that comes to mind when
considering algorithmic interpretability is: ‘Inter-
pretable to whom?’ The very word ‘interpretable’
implies an observer or subjective recipient who
will judge whether they can understand the algo-
rithm’s model or its behaviors. Another challenge
is the question of what we want to be interpre-
table, i.e., the historical data used to train the
model, the model, the performance of the model
found by the algorithm on a population cohort,
or the model’s decisions for a particular case?
Early well-known machine learning mod-
els are rather simple (see Figure 11-8) and
principled (maximizing a natural and clearly
stated objective, such as accuracy), and thus
are interpretable or understandable to some
extent. However, the ‘rules’ used by such models
can be complicated to understand fully. They
may capture complex and opaque relationships
between the variables in what seemed to be a
simple dataset. While radiologists may look for
“a curved tube resembling a ram’s horn, located
in the inner region of the brain’s temporal lobe,”
to identify the hippocampus, AI may use features
and patterns that are not articulable in human
language. While this makes AI an extremely
powerful tool for clinical decision-making, it
Software as a Medical Device: Regulatory and Market Access Implications
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
In reality, people often tend to sway. If the first
impression is positive, humans tend not to see
when the AI makes mistakes or may forget or
forgive the AI. How we must design our devices
allowing users to calibrate their trust appropri-
ately and how we must educate them in their
first interactions with the device is an open field
of research.
The radiology domain can inspire. For
example, a radiology study showed that the
human-device combination’s accuracy might
improve when a computer-aided-detection
algorithm identifies more than one choice to
the radiologist.38 Offering multiple options can
maintain trust in the system and mitigate the
risks of overtrust by putting the human expertise
at work.
In other cases, the AI’s performance alone
might be better than the performance of the
human-AI team. Sometimes, it is necessary to
take the human out of the loop altogether to get
the best performance and reduce or eliminate
use error. For example, in a clinical diagnostic
application used to read quantitative Polymerase
Chain Reaction qPCR) assays, FDA requires
manufacturers to deactivate the possibility of
the molecular biologist intervening, because the
precision and reliability of the AI outperforms
that of the human-AI team.39 Taking the human
out of the loop takes away human variability and
mitigates the risk of a lab using fewer control
samples than required by the assay manufac-
turer. On the other hand, when AI is trained on
rare diseases with fewer datasets, it may require
humans to be in the loop to reach maximum per-
formance. Striking the right balance is important
and differs on a case-by-case basis. Most medical
device legislation requires manufacturers to
eliminate or reduce the risk of use error.40 Inap-
propriate levels of trust in automation may cause
suboptimal performance.41
Transparency and Explicability
Only by gaining the user’s trust will machine
learning devices find their way into the care
pathways. One way to gain confidence is to
ensure transparency, both in terms of the
organization that creates the AI and the AI
itself. Transparency is also useful to clarify the
liability, i.e., did the doctor or the user make a
mistake? Was it the AI, incorrect or unforeseen
input data, or a malicious actor, e.g., hackers or
disgruntled employees?
Transparency may also be needed (1) to
allow manufacturers to determine operating
parameters (when the device works or does not
work), limitations to the intended use, con-
tra-indications, inclusion, and exclusion criteria
for input data or (2) to enable debugging of the
system and detect potential issues of bias.
Transparent AI presents core decision-mak-
ing elements in an open, comprehensive, accessi-
ble, clear, unambiguous, and interpretable way.42
The first question that comes to mind when
considering algorithmic interpretability is: ‘Inter-
pretable to whom?’ The very word ‘interpretable’
implies an observer or subjective recipient who
will judge whether they can understand the algo-
rithm’s model or its behaviors. Another challenge
is the question of what we want to be interpre-
table, i.e., the historical data used to train the
model, the model, the performance of the model
found by the algorithm on a population cohort,
or the model’s decisions for a particular case?
Early well-known machine learning mod-
els are rather simple (see Figure 11-8) and
principled (maximizing a natural and clearly
stated objective, such as accuracy), and thus
are interpretable or understandable to some
extent. However, the ‘rules’ used by such models
can be complicated to understand fully. They
may capture complex and opaque relationships
between the variables in what seemed to be a
simple dataset. While radiologists may look for
“a curved tube resembling a ram’s horn, located
in the inner region of the brain’s temporal lobe,”
to identify the hippocampus, AI may use features
and patterns that are not articulable in human
language. While this makes AI an extremely
powerful tool for clinical decision-making, it