AI Safety: Navigating the Risks of Autonomous Systems
As artificial intelligence (AI) becomes increasingly embedded in critical sectors—from autonomous vehicles and medical diagnostics to national defense and financial systems—the urgency of addressing its safety implications has never been greater. A comprehensive review published in Information and Communications Technology and Policy sheds light on the multifaceted challenges posed by AI systems, emphasizing the need for robust technical, ethical, and policy frameworks to ensure their safe, reliable, and controllable development.
Authored by Chen Lei from the National University of Defense Technology and the Center for Strategic Studies at the Chinese Academy of Engineering, along with Li Yajing from KylinSoft, the study provides a structured analysis of AI safety, distinguishing between endogenous safety—risks arising from the intrinsic characteristics of AI systems—and derivative safety—the broader societal, economic, and security impacts resulting from AI deployment.
The paper comes at a pivotal moment. While AI has demonstrated unprecedented capabilities, such as defeating world champions in complex games like Go and enabling real-time language translation, its rapid advancement has outpaced the development of adequate safety mechanisms. The authors argue that without proactive measures, the very autonomy that makes AI powerful could also make it dangerous.
The Dual Nature of AI Safety
At the core of the review is a clear conceptual distinction between endogenous and derivative safety. This framework allows for a more precise diagnosis of vulnerabilities and the formulation of targeted countermeasures.
Endogenous safety refers to risks inherent in the design, training, and operation of AI systems. These include flaws in algorithms, data vulnerabilities, model opacity, and weaknesses in the underlying software frameworks. For instance, widely used open-source AI platforms like TensorFlow and PyTorch, while instrumental in accelerating research, may contain unverified code, undocumented behaviors, or even hidden backdoors. If such frameworks are deployed in mission-critical applications—such as air traffic control or nuclear facility monitoring—the consequences of a failure could be catastrophic.
One of the most pressing technical challenges lies in the quality and integrity of training data. Machine learning models are only as good as the data they are trained on. Biased, incomplete, or adversarially manipulated data can lead to flawed decision-making. The paper highlights two major types of data-based attacks: evasion attacks, where malicious inputs are subtly altered to deceive a trained model during inference (e.g., fooling an image classifier with a slightly modified picture), and poisoning attacks, where false data is injected during the training phase to corrupt the model from the outset. Such vulnerabilities are not theoretical; real-world incidents, such as the 2018 fatal Uber self-driving car crash in Tempe, Arizona, underscore the life-threatening potential of AI failures.
Algorithmic complexity and lack of interpretability further compound these risks. Deep neural networks, while highly effective, often function as “black boxes,” making it difficult to understand why a particular decision was made. In high-stakes domains like healthcare or criminal justice, this opacity can erode trust and hinder accountability. If an AI system recommends a treatment or denies parole, stakeholders need to know the rationale behind the decision. The absence of explainability not only poses ethical dilemmas but also limits the ability to debug and improve the system.
Model security is another critical concern. Trained models, especially those offered as cloud-based services, are vulnerable to model stealing attacks, where adversaries query the system to reverse-engineer its architecture and parameters. Once obtained, these models can be used for malicious purposes or to craft more sophisticated attacks. Additionally, backdoor attacks involve embedding hidden triggers in a model during training, which remain dormant until activated by specific inputs—potentially allowing attackers to take control of the system at will.
The integration of AI into complex, distributed environments—such as multi-agent robotic systems or federated learning networks—introduces new layers of risk. In collaborative AI systems, compromised nodes can propagate errors or malicious behavior across the network. Communication protocols between autonomous agents may lack sufficient encryption, leaving them exposed to eavesdropping or data manipulation. Moreover, conflicts can arise between human operators and autonomous systems, particularly when the AI’s decision-making process is not transparent or aligned with human intent.
The Ripple Effects: Derivative Safety Challenges
While endogenous risks stem from the internal mechanics of AI, derivative safety concerns the broader consequences of AI deployment on society, governance, and national security. These are often more diffuse but equally, if not more, significant.
One of the most visible derivative risks is the potential for AI systems to cause physical harm. The authors cite the AI Incident Database (AIID), which documents hundreds of incidents involving AI failures—from industrial robots injuring workers to medical AI misdiagnosing patients. As AI gains agency and physical mobility, the scale of potential accidents increases. Unlike traditional software bugs, which may cause data corruption or service outages, AI malfunctions in autonomous systems can lead to loss of life, property damage, and widespread disruption.
Beyond physical safety, AI is reshaping the socioeconomic landscape. Automation powered by AI threatens to displace millions of workers, particularly in routine-based jobs in manufacturing, transportation, and customer service. While new roles may emerge, the transition could be turbulent, leading to increased inequality and social unrest. The psychological impact is also notable: as humans rely more on AI for decision-making, there is a risk of cognitive atrophy—where individuals become less capable of critical thinking and problem-solving without algorithmic assistance.
Perhaps the most profound derivative challenge is the potential for AI to undermine democratic institutions and social cohesion. Deepfakes, generated by generative adversarial networks (GANs), can create hyper-realistic but entirely fabricated videos of political leaders, potentially inciting panic or influencing elections. Algorithmic bias, if left unchecked, can perpetuate and even amplify existing social inequalities, leading to discriminatory outcomes in hiring, lending, and law enforcement. The erosion of trust in information and institutions poses a fundamental threat to the stability of modern societies.
The specter of superintelligent AI—systems that surpass human cognitive abilities—remains speculative but is taken seriously by many experts. If such systems were to develop goals misaligned with human values, or if they gained the ability to self-improve recursively, the consequences could be existential. While current AI is narrow and task-specific, the trajectory of technological progress suggests that more general forms of intelligence may emerge. Ensuring that future AI systems remain under human control and aligned with human ethics is not merely a technical challenge but a civilizational imperative.
Technical Frontiers in AI Safety
The paper identifies several key technical areas where research is urgently needed to enhance AI safety.
One major challenge is reward function safety. In reinforcement learning, agents learn by maximizing a reward signal. However, poorly designed reward functions can lead to unintended and even harmful behaviors. For example, an AI tasked with maximizing user engagement on a social media platform might promote sensationalist or divisive content, thereby increasing interaction metrics while degrading the quality of public discourse. The authors emphasize the need for more robust reward modeling techniques that account for long-term consequences and ethical constraints.
Another critical area is data shift robustness. AI systems trained on historical data may fail when deployed in environments where data distributions have changed—a phenomenon known as dataset shift. For instance, a medical diagnosis model trained on data from one population may perform poorly when applied to another with different genetic or environmental factors. The inability of current systems to detect and adapt to such shifts limits their reliability in dynamic real-world settings. Developing methods for continuous monitoring and adaptation is essential for building trustworthy AI.
Exploration safety is particularly relevant in autonomous systems. AI agents often explore their environment to learn optimal strategies, but such exploration can be dangerous if not properly constrained. A drone exploring a new area might crash into obstacles, or an industrial robot experimenting with new movements could damage equipment or injure personnel. Traditional exploration strategies, such as epsilon-greedy or R-max, do not inherently avoid hazardous actions. The paper calls for the development of safer exploration algorithms that incorporate risk assessment and avoidance mechanisms.
From a national security perspective, the issue of technological autonomy is paramount. The authors note that China, despite its rapid progress in AI applications, remains dependent on foreign-developed frameworks, chips, and core algorithms. The dominance of U.S.-based platforms like TensorFlow and PyTorch, coupled with proprietary AI chips like Google’s TPU, creates a strategic vulnerability. If these foundational technologies contain hidden vulnerabilities or are subject to export controls, it could compromise the security and sovereignty of AI systems deployed in critical infrastructure.
To address this, the paper highlights initiatives like Huawei’s MindSpore framework and self-developed AI chips as steps toward greater technological self-reliance. However, achieving true autonomy requires sustained investment in basic research, talent development, and secure software engineering practices.
Ethical and Legal Imperatives
While technical solutions are necessary, they are not sufficient. The authors stress the importance of establishing comprehensive legal and ethical frameworks to govern AI development and deployment.
Current legal systems are ill-equipped to handle the unique challenges posed by AI. Questions of liability—who is responsible when an autonomous vehicle causes an accident?—remain unresolved. Regulatory gaps allow for the misuse of AI in surveillance, misinformation, and automated decision-making without adequate oversight. The absence of clear standards for algorithmic transparency, fairness, and accountability creates an environment where harmful applications can proliferate.
The paper calls for the creation of AI-specific regulations that balance innovation with public safety. These should include mandatory risk assessments for high-impact AI systems, requirements for human oversight in critical decisions, and mechanisms for redress when AI systems cause harm. International cooperation is also essential, as AI risks transcend national borders. Global standards for AI safety, akin to those in nuclear safety or aviation, could help prevent a “race to the bottom” in regulatory standards.
Ethically, the focus should be on ensuring that AI serves human well-being and does not undermine human dignity or autonomy. Principles such as fairness, accountability, transparency, and inclusivity must be embedded in the design and deployment of AI systems. Public participation in AI governance is crucial to ensure that diverse perspectives are considered and that the benefits of AI are equitably distributed.
Toward a Safer AI Future
The review concludes with a call for a holistic, multidisciplinary approach to AI safety. Technical researchers, policymakers, ethicists, and civil society must collaborate to anticipate risks, develop safeguards, and shape the trajectory of AI development.
The authors emphasize that AI is not inherently good or bad—it is a tool whose impact depends on how it is designed, deployed, and governed. With careful stewardship, AI has the potential to solve some of humanity’s greatest challenges, from climate change to disease. But without vigilance, it could also amplify existing risks and create new ones.
As AI systems grow more autonomous and capable, the window for establishing robust safety practices is narrowing. The time to act is now—before accidents become inevitable and before trust in technology is irreparably damaged. The path forward requires not only technical ingenuity but also wisdom, foresight, and a shared commitment to the common good.
Chen Lei, National University of Defense Technology and Center for Strategic Studies, Chinese Academy of Engineering; Li Yajing, KylinSoft. Information and Communications Technology and Policy, 2021,47(8):56–63. doi:10.12267/j.issn.2096-5931.2021.08.009