170
Chapter 11: Artificial Intelligence
also brings the risk of the AI reflecting spurious
correlations in the data or overfitting to this par-
ticular dataset at the cost of transferability. Thus,
simple algorithms applied to simple datasets can
nevertheless lead to inscrutable models. However,
it is also entirely possible that even a model we
cannot understand ‘in the large’ or that is hidden
from the user can make specific decisions that
we can understand or rationalize post hoc.43 For
example, a doctor can review the output of an
algorithm that reports on the wound healing
stage (haemostasis, inflammatory, proliferative,
or maturation) by looking at a wound picture
to determine whether the algorithm identified
the healing phase correctly. Alternatively, we can
interrogate the model by having it tell us what it
would do on any input. We can explore coun-
terfactuals such as “What would be the smallest
change in input data that would change the
decision?” This type of explanatory understanding
at the level of individual decisions or predictions
is the basis for some of the more promising
research on interpretability.44
Explained variance, i.e., “Given a blank
sheet, what would be the minimum input
data needed to receive this decision?” is on the
opposing end of counterfactuals, i.e., “Given the
complete picture, what is the minimum change
needed to also change the answer?” Explained
variance involves the AI providing “the mini-
mum set of input data needed to come close to
its decision.” The minimum set of information
depends on the desired level of closeness, which
may differ for novice versus expert users. For
example, an AI predicting the probability of
survival from COVID-19 infection may explain
to the user that ‘age’ contributed to 80% of its
prediction. For some users, this may be sufficient
information. In contrast, other users may want
the AI to explain 99% of its prediction, adding
that patient history contributed to 15%, specific
lab results 3%, symptoms 0.5%, etc.45
‘Inexplicable’ devices are not unusual, as
healthcare has long been known for accepting
‘inexplicable’ devices for certain purposes, as long
as the technical file includes a description of
Figure 11-8. Transparency and Interpretability of AI
Neural networks comprise relatively simple models, with weights, transfer, and activation function. Still, due
to the vast amount of data, a person will not be able to process this information to the point of understanding
it. AI being technically interpretable or transparent does not automatically imply that a doctor or a patient can
interpret it, i.e., to understand cause and effect. A different level of abstraction is required to make the neural
network interpretable to users.
Reality f()
x
0
x
1
x
2
w
0
w
1
w
2
+o
inputs weights transfer
function
activation
function output
performance
(accuracy, precision, ...)
© Koen Cobbaert 2021
Chapter 11: Artificial Intelligence
also brings the risk of the AI reflecting spurious
correlations in the data or overfitting to this par-
ticular dataset at the cost of transferability. Thus,
simple algorithms applied to simple datasets can
nevertheless lead to inscrutable models. However,
it is also entirely possible that even a model we
cannot understand ‘in the large’ or that is hidden
from the user can make specific decisions that
we can understand or rationalize post hoc.43 For
example, a doctor can review the output of an
algorithm that reports on the wound healing
stage (haemostasis, inflammatory, proliferative,
or maturation) by looking at a wound picture
to determine whether the algorithm identified
the healing phase correctly. Alternatively, we can
interrogate the model by having it tell us what it
would do on any input. We can explore coun-
terfactuals such as “What would be the smallest
change in input data that would change the
decision?” This type of explanatory understanding
at the level of individual decisions or predictions
is the basis for some of the more promising
research on interpretability.44
Explained variance, i.e., “Given a blank
sheet, what would be the minimum input
data needed to receive this decision?” is on the
opposing end of counterfactuals, i.e., “Given the
complete picture, what is the minimum change
needed to also change the answer?” Explained
variance involves the AI providing “the mini-
mum set of input data needed to come close to
its decision.” The minimum set of information
depends on the desired level of closeness, which
may differ for novice versus expert users. For
example, an AI predicting the probability of
survival from COVID-19 infection may explain
to the user that ‘age’ contributed to 80% of its
prediction. For some users, this may be sufficient
information. In contrast, other users may want
the AI to explain 99% of its prediction, adding
that patient history contributed to 15%, specific
lab results 3%, symptoms 0.5%, etc.45
‘Inexplicable’ devices are not unusual, as
healthcare has long been known for accepting
‘inexplicable’ devices for certain purposes, as long
as the technical file includes a description of
Figure 11-8. Transparency and Interpretability of AI
Neural networks comprise relatively simple models, with weights, transfer, and activation function. Still, due
to the vast amount of data, a person will not be able to process this information to the point of understanding
it. AI being technically interpretable or transparent does not automatically imply that a doctor or a patient can
interpret it, i.e., to understand cause and effect. A different level of abstraction is required to make the neural
network interpretable to users.
Reality f()
x
0
x
1
x
2
w
0
w
1
w
2
+o
inputs weights transfer
function
activation
function output
performance
(accuracy, precision, ...)
© Koen Cobbaert 2021