Design and safety requirements for autonomous and semi-autonomous weapon systems

(a) General requirement (1) In general The Secretary shall, acting through each of the Secretaries of the military departments, ensure that whenever the Department designs, tests, develops, procures, deploys, or uses a system described in paragraph (2), the system meets the requirements of this section. (2) Systems A system described in this paragraph is an autonomous weapon system or semi-autonomous weapon system that uses artificial intelligence to create, generate, prioritize, recommend, or engage targets or courses of action in support of use of force decisions. (b) Human responsibility over use of force (1) Accountable individuals For each system described in subsection (a)(2)— (A) not later than 90 days after the date of the enactment of this Act, the Secretary shall promulgate rules for clear chain of command and command hierarchy for military operations involving a system described in subsection (a)(2) to mirror the chain of command and command hierarchy for military operations that do not involve such systems; (B) the Secretary shall ensure that a designated commander is identified as accountable under applicable military and international laws for each engagement or class of engagements involving a system described in subsection (a)(2), regardless of the degree to which artificial intelligence contributed to the identification, development, recommendation, selection, or engagement of the target; and (C) the designated commander identified under subparagraph (B) shall exercise ultimate discretion, judgment, and control over the use of force. (2) System requirements Each system described in subsection (a)(2) shall incorporate the following: (A) A system design that incorporates capabilities and interfaces that require the designated commander to exercise ultimate discretion, judgment and control in the envisioned development, planning, deployment, and use processes for the weapon system, including constraints on each such system’s authorized actions, targets, and geographic, temporal, and contextual scope, which the system may not expand or modify without explicit human authorization. (B) System capabilities, human-machine interfaces, doctrine, tactics, techniques, procedures, and human operator training must require commanders and human operators to use the system with deference to their discretion regarding care, and to analyze the output, in accordance with the Law of Armed Conflict and relevant laws of the United States and international laws, applicable treaties, weapon system safety rules, and rules of engagement that are applicable or reasonably expected to be applicable. (C) The creation and ongoing maintenance of records of target selection data, decision logic, and human operator actions, including the individual designated under subsection (b)(1)(B), sufficiently detailed to enable post-engagement review of compliance. (D) The design, testing, development, procurement, deployment, legal analysis and review, and use of artificial intelligence capabilities in autonomous and semi-autonomous weapon systems shall be consistent with, but not limited to, the Ethical Principles for Artificial Intelligence and the Responsible Artificial Intelligence Strategy and Implementation Pathway. (3) Availability to the public The Secretary shall ensure that— (A) the Ethical Principles for Artificial Intelligence and the Responsible Artificial Intelligence Strategy and Implementation Pathway are available to the public; and (B) any revision to the Ethical Principles for Artificial Intelligence or the Responsible Artificial Intelligence Strategy and Implementation Pathway adopted by the Secretary is made available to the public before the date that is 30 days before the date on which the revision goes into effect. (c) Engagement constraints and termination (1) In general Each system described in subsection (a)(2) shall be designed— (A) to complete engagements within a designated timeframe and designated geographic area and against a designated set of potential targets, as well as other relevant constraints, consistent with commander and human operator intentions; (B) to require independent review and analysis of a designated commander before using force against previously unauthorized targets, materially expanding target sets or geographic scope, taking actions contravening applicable law, rules of engagement, other relevant laws of the United States and international laws, or taking actions likely to result in unintended engagement; and (C) if unable to complete an engagement consistent with the parameters described in paragraph (1), to terminate the engagement until additional human operator and commander evaluation is completed. (2) Evaluation criteria The evaluation criteria used under subparagraph (C) of paragraph (1) shall consist of assessment of deficiencies and recommendations for changes to be compliant with subparagraph (A) of such paragraph before restarting preparation for engagement. (d) Transparency, auditability, and explainability Consistent with the potential consequences of an unintended engagement or unauthorized interference with the operation of a system described in subsection (a)(2), the physical hardware and software of such system shall be designed with— (1) technologies and data sources that are available to, auditable by, and explainable to the greatest extent possible by relevant personnel of the Department with the necessary clearance level; and (2) system safety, anti-tamper mechanisms, and cybersecurity in accordance with Department instructions and military standards governing cybersecurity and system safety. (e) Activation, termination, and human operator interface Each system described in subsection (a)(2) shall be designed so that— (1) system design and human-machine interfaces are readily understandable to trained human operators, including by clearly disaggregating which actions human operators need to perform and which actions the weapon system will perform; (2) clear procedures exist for trained human operators to activate, terminate, and disable all weapon system functions; (3) the system provides timely feedback on system status, including regarding the quality and sufficiency of the data inputs relied upon, to human operators in real time or near-real time; and (4) adequate training, tactics, techniques, procedures, and doctrine are available, reviewed on a quarterly basis, by weapon system human operators and designated commanders to understand the functioning, capabilities, and limitations of the system’s autonomy in realistic operational conditions. (f) Degraded communications safeguard Any autonomous or semi-autonomous weapon system that is, or is part of, an unmanned platform shall be designed such that, in the event of degraded or lost communications, the system does not autonomously select and engage individual targets, specific target groups, or general classes or specific types of targets that have not been previously selected by an authorized human operator. (g) Continuous monitoring The Secretary shall, in coordination with the Director of Operational Test and Evaluation, the Under Secretary of Defense for Research and Engineering, and the appropriate Secretary of a military department or Assistant Secretary for Special Operations and Low-Intensity Conflict, establish and maintain procedures for continuous monitoring of each system, to the greatest extent possible, described in subsection (a)(2) to identify and address circumstances in which changes to the system design or operational environment require additional testing and evaluation or legal review to provide sufficient confidence that the system will continue— (1) to function as intended; (2) to avoid unintended engagements; (3) to resist interference by unauthorized parties; and (4) to remain compliant with relevant laws of the United States, the Law of Armed Conflict, and international laws. (h) Robust artificial intelligence design For any system described in subsection (a) that incorporates artificial intelligence capabilities, such system shall be designed to utilize robust artificial intelligence, in accordance with the Responsible Artificial Intelligence Strategy and Implementation Pathway so that the system is resilient in real-world settings and against adversarial attacks and spoofing.