From X-Rays to AI: Navigating US Regulations in Radiological Health
127
using independent datasets that are new to the
algorithm.
Verification and Validation
Verification and validation testing requirements
for AI/ML clinical applications depend on the
IFU statement and the clinical performance
claims. For example, a device that acts as a
second reader to assist physicians by identifying
a specific image may need a different validation
study to that of a mammography screening
device that can be read without any further
review by a radiologist.69 Manufacturers should
carefully plan their validation studies to support
their desired indications and clinical claims.
The FDA reviews and clears devices based on
the specific indications. Special considerations
are needed if the subject device is indicated for
pediatric population. Hence, it is recommended
to highlight the availability of performance data
from pediatric patients if applicable in the regu-
latory submission.70
Standalone Performance Assessment
A standalone performance assessment can help
demonstrate that an algorithm performs as
intended. The performance study is usually based
on an independent dataset representative of the
target clinical population for which the device is
indicated. It is recommended that the manufac-
turers use the FDA 510(k), De Novo, and PMA
databases to review publicly available information
for recent device marketing authorizations to
Table 8-5. Key Considerations for Successful CAD Regulatory Documentation
1. Understand the real clinical need from the healthcare professionals’ and/or patients’ perspectives
2. Define the intended use, indications for use, and the target patient population
3. Develop good data collection strategies (ensure data include positive, negative, and borderline cases,
and demographics, etc.)
4. When possible, data can be enriched to assess generalizability (e.g., in the case of a rare disease)
5. Ensure there are appropriate quality checks for input data
6. Ensure the objective reference standard is robust with clinical justification (for retrospective studies
consider collecting as much data as possible including biopsy, pathology reports, clinical reports, and
long-term follow-up)
7. If subjective reference standard is considered, ensure at least three expert clinicians with appropriate
training and experience
8. Model development (use appropriate machine learning strategies)
9. Performance assessment should include both standalone performance and clinical validation (if
applicable)
10. Ensure the validation datasets are independent of training datasets
11. Ensure the statistical analysis plan is robust
12. Ensure the bench performance testing includes appropriate evaluation or scoring methods which are
clinically justifiable
13. Get alignment with health authorities on the clinical validation study (where applicable)
• Ensure the design includes multiple-reader, multiple-case design
• Access the appropriateness of the study design (e.g., retrospective/prospective)
• Ensure the clinical endpoints adequately cover the proposed indications
• Ensure the established ground truth and the truthers involved are acceptable
• Ensure the design includes robust statistical comparison between reference, aided and un-aided clinical reads.
• Where possible include a clinical quality control process (confirming the artificial intelligence/machine learning
functionality)
14. Ensure the labeling follows all the regulatory requirements
127
using independent datasets that are new to the
algorithm.
Verification and Validation
Verification and validation testing requirements
for AI/ML clinical applications depend on the
IFU statement and the clinical performance
claims. For example, a device that acts as a
second reader to assist physicians by identifying
a specific image may need a different validation
study to that of a mammography screening
device that can be read without any further
review by a radiologist.69 Manufacturers should
carefully plan their validation studies to support
their desired indications and clinical claims.
The FDA reviews and clears devices based on
the specific indications. Special considerations
are needed if the subject device is indicated for
pediatric population. Hence, it is recommended
to highlight the availability of performance data
from pediatric patients if applicable in the regu-
latory submission.70
Standalone Performance Assessment
A standalone performance assessment can help
demonstrate that an algorithm performs as
intended. The performance study is usually based
on an independent dataset representative of the
target clinical population for which the device is
indicated. It is recommended that the manufac-
turers use the FDA 510(k), De Novo, and PMA
databases to review publicly available information
for recent device marketing authorizations to
Table 8-5. Key Considerations for Successful CAD Regulatory Documentation
1. Understand the real clinical need from the healthcare professionals’ and/or patients’ perspectives
2. Define the intended use, indications for use, and the target patient population
3. Develop good data collection strategies (ensure data include positive, negative, and borderline cases,
and demographics, etc.)
4. When possible, data can be enriched to assess generalizability (e.g., in the case of a rare disease)
5. Ensure there are appropriate quality checks for input data
6. Ensure the objective reference standard is robust with clinical justification (for retrospective studies
consider collecting as much data as possible including biopsy, pathology reports, clinical reports, and
long-term follow-up)
7. If subjective reference standard is considered, ensure at least three expert clinicians with appropriate
training and experience
8. Model development (use appropriate machine learning strategies)
9. Performance assessment should include both standalone performance and clinical validation (if
applicable)
10. Ensure the validation datasets are independent of training datasets
11. Ensure the statistical analysis plan is robust
12. Ensure the bench performance testing includes appropriate evaluation or scoring methods which are
clinically justifiable
13. Get alignment with health authorities on the clinical validation study (where applicable)
• Ensure the design includes multiple-reader, multiple-case design
• Access the appropriateness of the study design (e.g., retrospective/prospective)
• Ensure the clinical endpoints adequately cover the proposed indications
• Ensure the established ground truth and the truthers involved are acceptable
• Ensure the design includes robust statistical comparison between reference, aided and un-aided clinical reads.
• Where possible include a clinical quality control process (confirming the artificial intelligence/machine learning
functionality)
14. Ensure the labeling follows all the regulatory requirements