From X-Rays to AI: Navigating US Regulations in Radiological Health
129
laboratory and patients in a clinical setting and
span a variety of acquisition protocols.
Deployment
The story of product development does not end
at obtaining FDA marketing authorization in
fact, in many ways, FDA market clearance is the
beginning. The FDA expects that AI/ML models
follow a TPLC process. Once an AI/ ML model
is trained, validated, and obtains FDA market-
ing authorization, it needs to be deployed and
integrated into a healthcare provider’s existing
information technology infrastructure. It is
important to ensure that the AI/ML clinical
applications are compatible with customers’
existing software systems and does not negatively
impact the clinical workflow.
The deployment step involves careful plan-
ning, implementation and monitoring by both
developers and clinical users to ensure the algo-
rithm’s accuracy, reliability, and safety. Between
regulatory marketing authorization and deploy-
ment at the final clinical sites, best practice is to
perform additional testing at designated clinical
sites (beta sites). This additional testing, though
potentially burdensome to manufacturers as well
as to the clinical facility’s information technology
team, is important to ensure that the software
performs seamlessly and as intended.
Monitoring
It is common to observe performance drifts over
time in real-world use. Therefore, it is necessary
to have a robust post-market monitoring systems
in place to detect any errors or bias that may arise
during the use of the algorithms. By implement-
ing a robust and effective monitoring system,
manufacturers can ensure that AI/ML devices
continue to consistently deliver high-quality care
without compromising the patient’s safety.
Key Ethical Considerations for AI
software
All involved stakeholders should understand the
ethical considerations in SaMD development.
General guidance on key ethical considerations
derived from the literature is overviewed below.
Data Governance
There are different types of data including clin-
ical data, business operational and analytic data,
raw image data, augmented data, and synthetic
data.72The collection, analysis and use of patient
data forms the basis for AI/ML medical prod-
uct development. The successful AI system will
primarily depend on thousands of high-quality
data for training the algorithm and independently
validating the locked models. Limited access, either
due to cost or availability, to data is one of the most
challenging aspects in SaMD development, result-
ing in training data with systemic biases because
of under-representation of a gender, age, race,
sexual orientation, or disease characteristic. These
biases will impact or limit the use of the resulting
algorithm. Lack of proper data governance controls
will result in data inconsistencies and anomalies
resulting in regulatory compliance issues.
Black Box
One of the challenges in the adoption of AI/
ML in healthcare are black-box algorithms
which do not explain or justify the results. It is
important to be able to explain how the output
from an AI algorithm is related to clinical prac-
tice and scientific literature.73 Regardless of the
complexity of the software and whether or not
it is proprietary, the software developer should
describe the underlying data used to develop the
algorithm and should include plain language
descriptions of the logic or rationale used by
an algorithm to render a recommendation. The
FDA’s guidance on clinical decision support
software74 points out that the requirement for
transparency into algorithm decision making
and ensuring that this information is available
to users so that they can “independently review
the basis” for the software’s recommendations.
Healthcare providers may more critically evaluate
recommendations from AI tools and therefore
make better decisions if they better understand
how such technologies work.
AI/ML devices developed based on discrete
data from larger hospitals, scanner models, type
of imaging modality, or certain patient demo-
graphics, may not perform as well in a different
environment. The radiologist may not be aware
129
laboratory and patients in a clinical setting and
span a variety of acquisition protocols.
Deployment
The story of product development does not end
at obtaining FDA marketing authorization in
fact, in many ways, FDA market clearance is the
beginning. The FDA expects that AI/ML models
follow a TPLC process. Once an AI/ ML model
is trained, validated, and obtains FDA market-
ing authorization, it needs to be deployed and
integrated into a healthcare provider’s existing
information technology infrastructure. It is
important to ensure that the AI/ML clinical
applications are compatible with customers’
existing software systems and does not negatively
impact the clinical workflow.
The deployment step involves careful plan-
ning, implementation and monitoring by both
developers and clinical users to ensure the algo-
rithm’s accuracy, reliability, and safety. Between
regulatory marketing authorization and deploy-
ment at the final clinical sites, best practice is to
perform additional testing at designated clinical
sites (beta sites). This additional testing, though
potentially burdensome to manufacturers as well
as to the clinical facility’s information technology
team, is important to ensure that the software
performs seamlessly and as intended.
Monitoring
It is common to observe performance drifts over
time in real-world use. Therefore, it is necessary
to have a robust post-market monitoring systems
in place to detect any errors or bias that may arise
during the use of the algorithms. By implement-
ing a robust and effective monitoring system,
manufacturers can ensure that AI/ML devices
continue to consistently deliver high-quality care
without compromising the patient’s safety.
Key Ethical Considerations for AI
software
All involved stakeholders should understand the
ethical considerations in SaMD development.
General guidance on key ethical considerations
derived from the literature is overviewed below.
Data Governance
There are different types of data including clin-
ical data, business operational and analytic data,
raw image data, augmented data, and synthetic
data.72The collection, analysis and use of patient
data forms the basis for AI/ML medical prod-
uct development. The successful AI system will
primarily depend on thousands of high-quality
data for training the algorithm and independently
validating the locked models. Limited access, either
due to cost or availability, to data is one of the most
challenging aspects in SaMD development, result-
ing in training data with systemic biases because
of under-representation of a gender, age, race,
sexual orientation, or disease characteristic. These
biases will impact or limit the use of the resulting
algorithm. Lack of proper data governance controls
will result in data inconsistencies and anomalies
resulting in regulatory compliance issues.
Black Box
One of the challenges in the adoption of AI/
ML in healthcare are black-box algorithms
which do not explain or justify the results. It is
important to be able to explain how the output
from an AI algorithm is related to clinical prac-
tice and scientific literature.73 Regardless of the
complexity of the software and whether or not
it is proprietary, the software developer should
describe the underlying data used to develop the
algorithm and should include plain language
descriptions of the logic or rationale used by
an algorithm to render a recommendation. The
FDA’s guidance on clinical decision support
software74 points out that the requirement for
transparency into algorithm decision making
and ensuring that this information is available
to users so that they can “independently review
the basis” for the software’s recommendations.
Healthcare providers may more critically evaluate
recommendations from AI tools and therefore
make better decisions if they better understand
how such technologies work.
AI/ML devices developed based on discrete
data from larger hospitals, scanner models, type
of imaging modality, or certain patient demo-
graphics, may not perform as well in a different
environment. The radiologist may not be aware