173
Software as a Medical Device: Regulatory and Market Access Implications
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
GDPR, causing the tradeoff between fairness
and accuracy. Also, testing AI for non-discrimi-
nation on an ethical basis is at odds with GDPR
and poses risks to users’ privacy. Under current
GDPR requirements, developers should not be
able to access attributes such as ethnicity and,
therefore, could not test for ethnic representation
in a dataset.
Conversely, software can play an essential
role in identifying and minimizing bias. This is
not specific to artificial intelligence but to soft-
ware in general. Historically, it was hard to prove
unintended discriminatory bias based on race, for
example. Software can enable feedback loops that
make it easier to detect and fix bias issues.
Standardization bodies are currently devel-
oping standards to characterize data sets. Manu-
facturers can use these standards to establish data
quality for training or evaluation purposes (see
Chapter 5 for a discussion of clinical evaluation
of software and machine learning devices). They
can use these characteristics to develop bias
and determine whether the AI is suitable for
a specific target population. However, manda-
tory certification of training data against these
standards is not an effective mechanism to assure
the AI is safe and effective for the target popu-
lation or that bias is minimized. Manufacturers
do not always have access to the training data
(see machine learning section). There are forms
of AI that learn without training data. Also, bias
can enter the AI development chain at different
points. Training data is only one of those entry
points. Instead, manufacturers can make a more
comprehensive assessment of bias through the
use of qualitative evaluation data.
EU AI Legislation
Legislators across the world are focusing on
ethical aspects of AI and the presence of bias. A
2019 heat map published by Anna Jobin60 shows
that the number of published AI ethics guide-
lines has increased, especially in Europe and the
US. As ethical guidelines are not enforceable, the
EU is assessing if and how to regulate ethical
aspects of AI (see Figure 11-10).
People often attribute a broad meaning to
ethics. The High-Level Expert Group on AI
(HLEG) ‘Ethics Guidelines for Trustworthy
Bias can creep in at different stages of AI development: 1. data collection, 2. data processing operations such
as cleaning, enrichment, and aggregation or through assumptions in the design of data processing pipelines, 3.
human judgment about the model construction and evaluation metrics can introduce biases in the processes of
algorithm development and testing, and 4. through context changes or user interaction.
algorithm
deployment
algorithm
development
data
processing
data
collection
application
in context
feedback
loops
BIAS?
BIAS?
BIAS?
BIAS?
Figure 11-9. Bias Entry Points in the AI Development Chain
Software as a Medical Device: Regulatory and Market Access Implications
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
GDPR, causing the tradeoff between fairness
and accuracy. Also, testing AI for non-discrimi-
nation on an ethical basis is at odds with GDPR
and poses risks to users’ privacy. Under current
GDPR requirements, developers should not be
able to access attributes such as ethnicity and,
therefore, could not test for ethnic representation
in a dataset.
Conversely, software can play an essential
role in identifying and minimizing bias. This is
not specific to artificial intelligence but to soft-
ware in general. Historically, it was hard to prove
unintended discriminatory bias based on race, for
example. Software can enable feedback loops that
make it easier to detect and fix bias issues.
Standardization bodies are currently devel-
oping standards to characterize data sets. Manu-
facturers can use these standards to establish data
quality for training or evaluation purposes (see
Chapter 5 for a discussion of clinical evaluation
of software and machine learning devices). They
can use these characteristics to develop bias
and determine whether the AI is suitable for
a specific target population. However, manda-
tory certification of training data against these
standards is not an effective mechanism to assure
the AI is safe and effective for the target popu-
lation or that bias is minimized. Manufacturers
do not always have access to the training data
(see machine learning section). There are forms
of AI that learn without training data. Also, bias
can enter the AI development chain at different
points. Training data is only one of those entry
points. Instead, manufacturers can make a more
comprehensive assessment of bias through the
use of qualitative evaluation data.
EU AI Legislation
Legislators across the world are focusing on
ethical aspects of AI and the presence of bias. A
2019 heat map published by Anna Jobin60 shows
that the number of published AI ethics guide-
lines has increased, especially in Europe and the
US. As ethical guidelines are not enforceable, the
EU is assessing if and how to regulate ethical
aspects of AI (see Figure 11-10).
People often attribute a broad meaning to
ethics. The High-Level Expert Group on AI
(HLEG) ‘Ethics Guidelines for Trustworthy
Bias can creep in at different stages of AI development: 1. data collection, 2. data processing operations such
as cleaning, enrichment, and aggregation or through assumptions in the design of data processing pipelines, 3.
human judgment about the model construction and evaluation metrics can introduce biases in the processes of
algorithm development and testing, and 4. through context changes or user interaction.
algorithm
deployment
algorithm
development
data
processing
data
collection
application
in context
feedback
loops
BIAS?
BIAS?
BIAS?
BIAS?
Figure 11-9. Bias Entry Points in the AI Development Chain