168
Chapter 11: Artificial Intelligence
interaction problems,32 such as a lack of opera-
tor situational awareness (sensing uncertainties/
limited situational awareness), i.e., the operator
having insufficient knowledge of the state of the
device at the time of intervention.
When specific actions require the device
to perform at high speed or with safety and
accuracy, it may be safer to circumvent the lim-
itations of immediate, real-time human control.
For example, a machine learning robot for eye
surgery33 may require the human to be taken out
of the loop because of the user’s limited decision
making-capacity,34 limited situational aware-
ness, and sensing uncertainties. In this case, the
most effective human supervision will be based
on continuous oversight and periodic retrospec-
tive review of the performance for individual
patients or for cohorts of patients, for example,
through PMCF.
Automation is, of course, not a panacea.
Manufacturers and health institutions must be
aware that automation leads to deskilling in some
circumstances or may require more user train-
ing and higher-skilled individuals than actions
performed without automation.35
Overtrust
Manufacturers of machine learning devices
sometimes require a human in the loop to con-
trol when the AI is less confident of its decision.
Human behavior, however, comes with its risks.
For example, a Level 2 self-driving car36 requires
a human in the loop to drive the car safely. It
might take 100.000 km before the car has an
accident the human may by then have put too
much confidence (overtrust) and automation
bias37 in the car and no longer pays enough
attention to take back control of the car in a
timely and safe manner.
In a perfect world, we would like users who
trust AI when it works at 100% accuracy but
to be hypersensitive and identify when it is not.
Devices that are controlled or supervised by humans are also known as heteronomous devices. Devices that
do not provide a means to intervene in a timely fashion are also known as autonomous devices. Hybrid devices
provide direct control or supervisory control on certain functions and no control on other functions.
SENSE ACT
DECIDE
DECIDE
SENSE ACT
Direct Control Supervisory Control No Control
The AI performs a task and then
waits for the human user to take an
action before continuing
The AI can sense, decide, and act on
its own. The human user supervises
its operation and can intervene
when required.
The AI can sense, decide, and act on
its own. The human user cannot
intervene in a timely fashion.
DECIDE
SENSE ACT
© Koen Cobbaert 2021
Figure 11-7. Types of Controllability
Chapter 11: Artificial Intelligence
interaction problems,32 such as a lack of opera-
tor situational awareness (sensing uncertainties/
limited situational awareness), i.e., the operator
having insufficient knowledge of the state of the
device at the time of intervention.
When specific actions require the device
to perform at high speed or with safety and
accuracy, it may be safer to circumvent the lim-
itations of immediate, real-time human control.
For example, a machine learning robot for eye
surgery33 may require the human to be taken out
of the loop because of the user’s limited decision
making-capacity,34 limited situational aware-
ness, and sensing uncertainties. In this case, the
most effective human supervision will be based
on continuous oversight and periodic retrospec-
tive review of the performance for individual
patients or for cohorts of patients, for example,
through PMCF.
Automation is, of course, not a panacea.
Manufacturers and health institutions must be
aware that automation leads to deskilling in some
circumstances or may require more user train-
ing and higher-skilled individuals than actions
performed without automation.35
Overtrust
Manufacturers of machine learning devices
sometimes require a human in the loop to con-
trol when the AI is less confident of its decision.
Human behavior, however, comes with its risks.
For example, a Level 2 self-driving car36 requires
a human in the loop to drive the car safely. It
might take 100.000 km before the car has an
accident the human may by then have put too
much confidence (overtrust) and automation
bias37 in the car and no longer pays enough
attention to take back control of the car in a
timely and safe manner.
In a perfect world, we would like users who
trust AI when it works at 100% accuracy but
to be hypersensitive and identify when it is not.
Devices that are controlled or supervised by humans are also known as heteronomous devices. Devices that
do not provide a means to intervene in a timely fashion are also known as autonomous devices. Hybrid devices
provide direct control or supervisory control on certain functions and no control on other functions.
SENSE ACT
DECIDE
DECIDE
SENSE ACT
Direct Control Supervisory Control No Control
The AI performs a task and then
waits for the human user to take an
action before continuing
The AI can sense, decide, and act on
its own. The human user supervises
its operation and can intervene
when required.
The AI can sense, decide, and act on
its own. The human user cannot
intervene in a timely fashion.
DECIDE
SENSE ACT
© Koen Cobbaert 2021
Figure 11-7. Types of Controllability