164
Chapter 11: Artificial Intelligence
use, within the change boundaries, and according
to the manufacturer’s algorithm change proto-
col (ACP) (see next section for a description of
change boundaries and ACP). The manufacturer
remains responsible for the device. Consider, for
example, an AI for the prediction of sepsis. The
hospital makes a change in its local practices,
now also encoding blood parameters procalci-
tonin and IL-6 and making these available for
the AI. The manufacturer has proved these blood
parameters work as valid inputs, described in the
ACP. The user can now further improve the local
model’s performance by training the AI on these
extra blood parameters, following the manufac-
turer’s ACP.
Also, hybrid scenarios may exist, whereby
the model continues to evolve, but the user has
the ability to revert back to an earlier state of the
model or to the factory default.
Note: Having a human in the loop during
learning to control what model state is put into
clinical use is different from having a human in
the loop to control the device during clinical use.
A continuously changing AI does not have a
human in the loop to control the learning but can
nevertheless have a human in the loop to control
device functions during clinical use, for example,
through a device override or emergency stop.
Postmarket Significant Change
Medical device legislation requires a manufac-
turer to determine whether a new device release
changes significantly (‘substantial’ in the words of
the IMDRF). If a new software release changes
significantly, the manufacturer must perform a
new conformity assessment before placing the
device on the market. Under the EU MDR and
EU IVDR, a health institution developing a
machine learning device for in-house use also
must perform such significant change determi-
nation and perform a new conformity assess-
ment before putting the device into clinical use.
In some cases, the health institution may use a
manufacturer’s machine learning component to
build a new device, or the health institution may
change an existing device in a significant way. A
significant change that occurs postmarket pro-
vides a fourth scenario (see Figure 11-5).
In the fourth scenario, the user intentionally
changes the local model in a way not allowed by
the manufacturer’s change control plan, either by
making a change:
• Beyond the manufacturer’s pre-determined
change envelope (also known as change
boundaries or pre-specifications), e.g., for a
purpose not covered by the intended use or
for more specific purposes than claimed by
the manufacturer
• While not complying with the
manufacturer’s ACP
Alternatively, the user incorporates the device as
a component in a new device.
In the fourth scenario, the device is inten-
tionally changed by the user in a significant way.
The user is misusing the device. Consider, for
example, an AI to read quantitative Polymerase
Chain Reaction (qPCR) assays. The AI can han-
dle qPCR curves generated for assays for plant,
animal, or human specimens. Assume a second
manufacturer or a health institution produces an
assay for the detection of SARS-COV-2. By con-
tinuing the AI training, it becomes especially good
at reading qPCR curves associated with a SARS-
COV-2 assay. ‘Reading qPCR curves for SARS-
COV-2’ is a more specific claim and requires a
higher level of performance before it is considered
state-of-the-art than ‘reading qPCR curves for
assays for plant, animal or human specimen.’
Consequently, the more specific claim is
considered a significant change. A manufacturer
or EU-based health institution (following EU
MDR Art. 5(5) on in-house manufacturing)
performing such change carries the manufacturer
responsibilities under medical device regulations.
If the second manufacturer can prove safety
and performance without having the technical
documentation of the original device, the original
manufacturer’s intellectual property remains pro-
tected if the notified body or competent author-
Chapter 11: Artificial Intelligence
use, within the change boundaries, and according
to the manufacturer’s algorithm change proto-
col (ACP) (see next section for a description of
change boundaries and ACP). The manufacturer
remains responsible for the device. Consider, for
example, an AI for the prediction of sepsis. The
hospital makes a change in its local practices,
now also encoding blood parameters procalci-
tonin and IL-6 and making these available for
the AI. The manufacturer has proved these blood
parameters work as valid inputs, described in the
ACP. The user can now further improve the local
model’s performance by training the AI on these
extra blood parameters, following the manufac-
turer’s ACP.
Also, hybrid scenarios may exist, whereby
the model continues to evolve, but the user has
the ability to revert back to an earlier state of the
model or to the factory default.
Note: Having a human in the loop during
learning to control what model state is put into
clinical use is different from having a human in
the loop to control the device during clinical use.
A continuously changing AI does not have a
human in the loop to control the learning but can
nevertheless have a human in the loop to control
device functions during clinical use, for example,
through a device override or emergency stop.
Postmarket Significant Change
Medical device legislation requires a manufac-
turer to determine whether a new device release
changes significantly (‘substantial’ in the words of
the IMDRF). If a new software release changes
significantly, the manufacturer must perform a
new conformity assessment before placing the
device on the market. Under the EU MDR and
EU IVDR, a health institution developing a
machine learning device for in-house use also
must perform such significant change determi-
nation and perform a new conformity assess-
ment before putting the device into clinical use.
In some cases, the health institution may use a
manufacturer’s machine learning component to
build a new device, or the health institution may
change an existing device in a significant way. A
significant change that occurs postmarket pro-
vides a fourth scenario (see Figure 11-5).
In the fourth scenario, the user intentionally
changes the local model in a way not allowed by
the manufacturer’s change control plan, either by
making a change:
• Beyond the manufacturer’s pre-determined
change envelope (also known as change
boundaries or pre-specifications), e.g., for a
purpose not covered by the intended use or
for more specific purposes than claimed by
the manufacturer
• While not complying with the
manufacturer’s ACP
Alternatively, the user incorporates the device as
a component in a new device.
In the fourth scenario, the device is inten-
tionally changed by the user in a significant way.
The user is misusing the device. Consider, for
example, an AI to read quantitative Polymerase
Chain Reaction (qPCR) assays. The AI can han-
dle qPCR curves generated for assays for plant,
animal, or human specimens. Assume a second
manufacturer or a health institution produces an
assay for the detection of SARS-COV-2. By con-
tinuing the AI training, it becomes especially good
at reading qPCR curves associated with a SARS-
COV-2 assay. ‘Reading qPCR curves for SARS-
COV-2’ is a more specific claim and requires a
higher level of performance before it is considered
state-of-the-art than ‘reading qPCR curves for
assays for plant, animal or human specimen.’
Consequently, the more specific claim is
considered a significant change. A manufacturer
or EU-based health institution (following EU
MDR Art. 5(5) on in-house manufacturing)
performing such change carries the manufacturer
responsibilities under medical device regulations.
If the second manufacturer can prove safety
and performance without having the technical
documentation of the original device, the original
manufacturer’s intellectual property remains pro-
tected if the notified body or competent author-