AI Medical Devices Demand New Quality Control Paradigms
The rapid evolution of artificial intelligence (AI) in healthcare has ushered in a new era of medical innovation, where software is no longer static but adaptive, learning from real-world data to improve diagnostic accuracy and clinical outcomes. However, this very dynamism presents unprecedented challenges for regulatory frameworks and quality management systems traditionally designed for stable, deterministic technologies. As AI-powered medical devices become increasingly prevalent in clinical settings, experts are calling for a fundamental rethinking of how these systems are developed, validated, and monitored throughout their lifecycle.
At the heart of this transformation is the growing recognition that conventional approaches to software validation and regulatory oversight are insufficient for artificial intelligence medical devices (AIMDs). Unlike traditional medical software, which operates on fixed algorithms and produces consistent outputs for identical inputs—a concept known as “locked” algorithms—many modern AIMDs employ machine learning models capable of continuous adaptation. These systems can evolve over time through exposure to new patient data, potentially altering their behavior and performance without explicit human intervention. This characteristic, while promising improved accuracy and personalization, introduces significant risks related to unpredictability, lack of transparency, and potential degradation in performance under unforeseen conditions.
In response to these emerging complexities, regulatory bodies worldwide have begun revising their frameworks. A pivotal development came in 2019 when China’s National Medical Products Administration (NMPA) issued the Guidelines for Quality Management of Independent Software, which took effect in July 2020. This annex specifically targets standalone software used in medical applications, emphasizing rigorous lifecycle management, version control, and post-market surveillance. It mandates comprehensive documentation for software updates, including risk assessments, verification and validation activities, traceability analysis, and user communication protocols. While this represents a critical step forward, experts argue that even these enhanced guidelines fall short when applied to true adaptive AI systems.
Li Shu, Wang Hao, Wang Chenxi, Hao Ye, Li Jiage, and Li Jingli from the Institute for Medical Devices Control at the National Institutes for Food and Drug Control in Beijing have been at the forefront of analyzing these gaps. In a recent publication in China Medical Devices, they outline the unique challenges posed by AIMDs and advocate for a dynamic, robust quality management model tailored to the iterative nature of AI-driven healthcare technologies.
One of the central issues they identify is the frequency and nature of design changes in AI systems. Traditional medical devices undergo infrequent updates, typically requiring full regulatory review with each modification. In contrast, AI models may be retrained weekly or even daily, incorporating new data to refine their predictions. Each retraining cycle constitutes a form of software update, yet not all updates carry the same level of risk. The authors emphasize the need for a tiered approach to change management—one that distinguishes between minor performance tweaks and major shifts in algorithmic architecture or intended use.
For instance, an update that improves detection sensitivity for lung nodules in CT scans using additional cases from the same population may represent a low-risk enhancement. However, expanding the model’s capability to diagnose a different type of cancer—or shifting its role from assisted diagnosis to primary decision-making—introduces far greater clinical implications and thus warrants more stringent evaluation. The challenge lies in establishing clear criteria for determining when a software modification triggers a new round of clinical validation or regulatory submission.
This leads directly to another critical concern: verification and confirmation. In software engineering, verification ensures that “the system was built right”—that it conforms to technical specifications—while confirmation establishes that “the right system was built”—that it fulfills its intended clinical purpose. For AIMDs, both processes must account for the inherent uncertainty and black-box nature of deep learning models.
The research team highlights that traditional test datasets, often static and limited in scope, are inadequate for evaluating systems designed to learn continuously. Instead, they propose a shift toward real-world performance monitoring, where post-market data feeds back into the validation loop. This includes tracking algorithmic drift, assessing performance across diverse patient demographics, and detecting edge cases that were not present during initial training.
Moreover, the issue of data provenance and integrity becomes paramount. AI models are only as good as the data they are trained on. Biases in training datasets—such as underrepresentation of certain ethnic groups or disease variants—can lead to systematic errors in diagnosis. Therefore, manufacturers must implement rigorous data governance practices, documenting the source, quality, and representativeness of all training and testing data. The authors stress the importance of defining acceptable data sources early in the development process and maintaining consistency unless justified by scientific rationale.
Another dimension of quality assurance is software interpretability. As AI systems make decisions that impact patient care, clinicians and regulators demand transparency. However, many high-performing models, particularly deep neural networks, operate as “black boxes,” making it difficult to understand how a particular output was generated. This lack of explainability raises ethical, legal, and practical concerns.
The European Union’s General Data Protection Regulation (GDPR) has already introduced the concept of a “right to explanation,” allowing individuals to request meaningful information about automated decisions affecting them. In healthcare, this translates to a need for clinicians to understand the basis of an AI-generated diagnosis before acting upon it. To address this, the authors suggest integrating interpretability tools into the development pipeline—methods such as saliency maps, feature attribution, or rule extraction that can shed light on the model’s reasoning process.
However, they caution against treating interpretability as a one-size-fits-all solution. Different stakeholders require different levels of explanation: a radiologist may benefit from visual heatmaps highlighting regions of interest in an image, while a regulator might require statistical summaries of model behavior across populations. Thus, the quality management system should include specifications for what constitutes adequate interpretability based on the device’s risk class and intended use.
Risk stratification plays a crucial role in shaping the regulatory pathway for AIMDs. Drawing from the International Medical Device Regulators Forum (IMDRF) framework, the researchers categorize AI applications based on two key factors: the importance of the information provided to clinical decision-making and the severity of the underlying medical condition. For example, an AI system that provides diagnostic recommendations for life-threatening conditions like stroke or sepsis falls into the highest risk category (Class IV), necessitating the most stringent controls. Conversely, a tool that offers secondary notifications for non-urgent findings would be classified lower (Class I or II), allowing for more flexible oversight.
What sets AIMDs apart is that risk is not solely determined by intended use but also by the degree of adaptability. A “locked” algorithm, even if used for high-stakes decisions, can be thoroughly tested and validated before deployment. In contrast, a continuously learning system introduces uncertainty because its future behavior cannot be fully predicted at the time of approval. This necessitates a paradigm shift from pre-market certification to continuous post-market surveillance.
To manage this, the authors advocate for the adoption of Total Product Lifecycle (TPLC) approaches, where regulatory confidence is maintained through ongoing assessment of organizational excellence, software development practices, and real-world performance. Under such a model, manufacturers would be evaluated not just on individual products but on their entire quality ecosystem—development processes, risk management protocols, incident reporting mechanisms, and responsiveness to feedback.
This approach aligns with recent initiatives by the U.S. Food and Drug Administration (FDA), which has proposed a Pre-Certification Program for Software as a Medical Device (SaMD). The idea is to accredit organizations based on their culture of quality and organizational excellence, allowing them to bring lower-risk AI updates to market more rapidly while still ensuring patient safety. Similar models could be adapted within China’s regulatory landscape, fostering innovation without compromising safety.
The paper also explores practical scenarios where these principles come into play. One example involves myoelectric prosthetics controlled by AI. These devices learn from users’ muscle signals over time, adapting to individual movement patterns and intentions. While this personalization enhances functionality, it also means that each user effectively has a unique version of the software. Traditional validation methods, which assume uniformity across devices, are ill-suited for such systems. Instead, robustness testing must account for inter-user variability, and performance metrics should reflect long-term usability rather than one-time accuracy.
Another case study focuses on AI-based medical imaging screening, such as lung cancer detection in chest CT scans. Here, the architecture of the deep learning model itself may evolve during retraining—layers may be added or removed, activation functions changed, or optimization algorithms modified. When both weights and architecture change, the resulting model may behave fundamentally differently from its predecessor, even if trained on similar data. Such transformations require comprehensive regression testing and possibly new clinical trials, depending on the magnitude of change.
The implications extend beyond technical considerations to organizational and cultural shifts within medical device companies. Developing and maintaining AIMDs requires multidisciplinary teams combining expertise in clinical medicine, data science, software engineering, and regulatory affairs. Quality management systems must support collaboration across these domains, ensuring that clinical insights inform model development and that technical decisions are aligned with patient safety goals.
Documentation practices also need to evolve. Traditional software requirements specifications are often insufficient for capturing the nuances of AI behavior. Instead, living documents—continuously updated throughout the product lifecycle—are needed to record model versions, training data characteristics, performance benchmarks, and known limitations. These records serve not only regulatory purposes but also internal learning, enabling teams to trace performance changes back to specific updates or data sources.
Cybersecurity is another critical component of AI quality management. As these systems increasingly connect to hospital networks and cloud platforms, they become targets for malicious attacks. Adversarial inputs—carefully crafted data designed to fool AI models—pose a real threat, especially in high-stakes diagnostic applications. Therefore, security testing must be integrated into the verification process, including penetration testing, anomaly detection, and secure update mechanisms.
Looking ahead, the authors envision a future where AI medical devices are not only intelligent but also self-aware—to some extent. Systems could include built-in monitors that detect performance degradation, distributional shifts in input data, or conflicting outputs, triggering alerts for human review or automatic rollback to a previous version. Such capabilities would enhance reliability and trust, particularly in unsupervised environments.
Ultimately, the goal is to create a balanced ecosystem where innovation thrives without sacrificing patient safety. The current pace of AI development demands agile regulatory frameworks that can keep up with technological change. At the same time, foundational principles of medical device regulation—safety, efficacy, and accountability—must remain inviolate.
The work by Li Shu, Wang Hao, Wang Chenxi, Hao Ye, Li Jiage, and Li Jingli underscores the urgency of building adaptive quality management systems capable of supporting the next generation of AI-driven healthcare solutions. By integrating risk-based oversight, continuous validation, transparent design, and organizational excellence, regulators and manufacturers can ensure that artificial intelligence fulfills its promise to improve patient outcomes while minimizing unintended harm.
As AI continues to transform medicine, the conversation around quality assurance will only grow more complex. But with thoughtful, evidence-based approaches grounded in both technical rigor and clinical relevance, the healthcare community can navigate this transition responsibly. The future of medical AI depends not just on smarter algorithms, but on smarter governance.
Li Shu, Wang Hao, Wang Chenxi, Hao Ye, Li Jiage, Li Jingli, Institute for Medical Devices Control, National Institutes for Food and Drug Control, Beijing; China Medical Devices; doi:10.3969/j.issn.1674-1633.2021.09.003