Policy and guidance related to autonomous weapon systems and artificial intelligence capabilities acquisition

Chapter 345 of title 10, United States Code, is amended by adding at the end the following new section: 4577. Autonomous weapon systems and artificial intelligence capabilities acquisition; planning and oversight processes (a) Policy It is the policy of the Department of Defense to maximize uses of autonomy and artificial intelligence capabilities to the extent practicable, while ensuring and maintaining that implementation of such autonomy and artificial intelligence capabilities provides appropriate human oversight, to ensure that operations are conducted in accordance with the law of war, applicable treaties, weapon system safety rules, applicable rules of engagement, and long-standing frameworks protecting the privacy and civil liberties of United States persons. (b) Requirement In accordance with the policy set forth in subsection (a), the Secretary of Defense shall— (1) ensure personnel exercise appropriate levels of human judgment and consistently monitor any deployed artificial intelligence-enabled autonomous weapon systems, while remaining responsible for the development, deployment, and use of autonomous weapon systems and artificial intelligence capabilities; (2) take deliberate steps to ensure accuracy in autonomous weapon systems and artificial intelligence capabilities; (3) develop and deploy autonomous weapon systems and artificial intelligence capabilities in a manner that promotes an appropriate understanding of the technology, their development processes, and operational methods applicable to autonomous weapon systems and artificial intelligence capabilities; (4) subject prototype and deployed autonomy and artificial intelligence capabilities to routine testing and information assurance across their entire life cycles to ensure that capabilities meet defined safety, security, and effectiveness parameters; and (5) design and engineer autonomy and artificial intelligence capabilities to fulfill their intended functions, and deploy these capabilities such that human operators retain— (A) the ability to detect and avoid unintended consequences or behaviors; and (B) a means for human intervention to disengage or deactivate deployed systems that demonstrate unintended or illegal behavior. (c) Review for autonomous weapon systems and supporting artificial intelligence capabilities (1) Levels of human judgment (A) The Secretary shall ensure that any autonomous weapon system or artificial intelligence capability to be utilized by the Department, except as exempted by subsection (c)— (i) is designed and employed in a manner that enables commanders and operators to exercise ultimate human responsibility over the use of force; and (ii) is categorized by the appropriate level of human judgment required to mitigate risks to life, safety and health of Department personnel or noncombatant civilian harm. (B) For purposes of subparagraph (A), appropriate levels of human judgment for an autonomous weapon system shall consist of two levels as follows: (i) Level 1, which means that the weapon system poses little to no risk to human life or safety should the system fail to act as designed. (ii) Level 2, which means that the weapon system poses moderate to high risk to human life or safety should the system fail to act as designed. (C) Appropriate levels of human judgment for an artificial intelligence capability shall consist of two levels as follows: (i) Level 1, which means that the artificial intelligence capability poses little to no risk as a result of compromise of data integrity or operational support should the system fail to act as designed. (ii) Level 2, which means that the artificial intelligence capability— (I) uses data protected by section 552a of title 5 (commonly known as the Privacy Act of 1974) or regulations promulgated under the Health Insurance Portability and Accountability Act of 1996 (Public Law 104–191); or (II) poses a risk such that the failure of the artificial intelligence system to act as designed would severely affect the ability of the Department to perform the designated mission of the artificial intelligence capability. (D) To meet the requirements under subparagraph (A), the Secretary of Defense shall ensure that any autonomous weapon system or artificial intelligence capability described in such subsection, to the maximum extent practicable— (i) allows human operators to supervise the operation of the weapon system during mission execution when appropriate given system design; (ii) includes mechanisms that allow commanders or operators to intervene in or terminate the use of force by the weapon system at all stages during mission execution when appropriate given system design; (iii) includes fail-safe mechanisms to enable manual control when autonomous systems are degraded, jammed, spoofed, or under adversarial attack when appropriate given system design; (iv) provides sufficient information regarding weapon system status and decision logic to enable operators to understand, oversee, and exercise functional control over system behavior; (v) maintains records of target selection data, decision logic, and human operator actions sufficient to enable post-engagement review of compliance with the requirements under subsection (a); and (vi) operates consistent with applicable United States law and international law, rules of engagement, the law of armed conflict, applicable treaties, and Department of Defense policy. (E) (i) The Secretary of Defense shall establish procedures to ensure that commanders and operators are rigorously trained to provide appropriate levels of human judgment to supervise autonomous weapon systems and artificial intelligence capabilities in operational environments, including regular proficiency assessments in manual target identification, threat assessment, and engagement procedures to ensure operators can effectively execute missions using other capabilities when autonomous systems are unavailable or unreliable. (ii) Such procedures shall include a mechanism for operators to report, without fear of reprisal, concerns regarding weapon system reliability or the adequacy of human-machine interfaces. (2) Review and verification Subject to subsection (d), for any autonomous weapon system or artificial intelligence capability under development or being fielded by the Department, the Secretary shall ensure that such autonomous weapon system or artificial intelligence capability requires— (A) before a decision to enter prototyping or formal development, review and verification by the Under Secretary of Defense for Research and Engineering or a designee of the Secretary of Defense, in consultation with the Vice Chairman of the Joint Chiefs of Staff that— (i) the system design incorporates the necessary capabilities to allow commanders, operators, and analysts to exercise appropriate levels of human judgment over the use of force in the envisioned planning and employment processes for the autonomous weapon system or artificial intelligence capability in such a way that it can accommodate varying levels of human oversight without requiring redesign; (ii) (I) the autonomous weapon system or artificial intelligence capability is designed to complete engagements within a timeframe and geographic area, as well as other applicable environmental and operational parameters, consistent with commander and operator intentions; or (II) if not designed as described in subclause (I), the autonomous weapon system or artificial intelligence capability will terminate engagements or obtain additional operator input before continuing the engagement; (iii) the combination of the design and concept of employment of the autonomous weapon system or artificial intelligence capability, such as its target selection and engagement logic and other relevant processes or measures, accounts for risks to nontargets, consistent with commander and operator intent and the laws of war; (iv) the design of the autonomous weapon system or artificial intelligence capability, including system safety, anti-tamper mechanisms, and the cybersecurity of the autonomous weapon system or artificial intelligence capability, in accordance with Department of Defense Instruction 8500.01 (relating to cybersecurity), or successor instruction, addresses and minimizes the probability and consequences of failures; (v) plans are in place for verification and validation and test and evaluation to establish the reliability, effectiveness, and suitability of the autonomous weapon system or artificial intelligence capability under realistic conditions, including possible adversary actions, to a sufficient standard consistent with the potential consequences of an unintended engagement or unauthorized parties interfering with the operation of the autonomous weapon system or artificial intelligence capability prior to fielding; and (vi) a preliminary legal review of the autonomous weapon system or artificial intelligence capability has been completed— (I) in coordination with the General Counsel of the Department of Defense; and (II) in accordance with Department of Defense Directive 5000.01 (relating to Defense Acquisition System), or successor directive, Department of Defense Directive 2311.01 (relating to Department of Defense Law of War Program), or successor directive, and, where applicable, Department of Defense Directive 3000.03E (relating to Department of Defense Agent for Non-Lethal Weapons and Non-Lethal Weapon Policy), or successor directive; and (B) before fielding, review and verification by the Under Secretary for Research and Engineering or a designee of the Secretary of Defense, in consultation with the Vice Chairman of the Joint Chiefs of Staff that, with respect to the autonomous weapon system or artificial intelligence capability— (i) system capabilities, human-machine interfaces, doctrine, tactics, techniques and procedures, and training have been demonstrated to allow commanders and operators to exercise appropriate levels of human judgment over the use of force and to employ systems with appropriate care and in accordance with the law of war, applicable treaties, weapon system safety rules, and definable rules of engagement that are applicable or reasonably expected to be applicable; (ii) relevant system safety, anti-tamper mechanisms, cyber survivability, operational resilience, and cybersecurity capabilities have been implemented to minimize the probability and consequences of failures; (iii) for autonomous weapon systems or artificial intelligence capabilities that are being fielded that may have gone through the development pipeline and verified under subparagraph (A), an updated legal review of the weapon system or artificial intelligence capability has been completed— (I) in coordination with the General Counsel of the Department of Defense; and (II) in accordance with Department of Defense Directive 5000.01 (relating to Defense Acquisition System), or successor directive, Department of Defense Directive 2311.01 (relating to Department of Defense Law of War Program), or successor directive, and, where applicable, Department of Defense Directive 3000.03E (relating to Department of Defense Agent for Non-Lethal Weapons and Non-Lethal Weapon Policy), or successor directive; and (iv) a monitoring regime is in place to identify and address changes in operational environment, data inputs, and use that could contribute to failure of the system or capability to act in a manner consistent with the intent for the system or capability. (3) Validity of verification (A) The Secretary shall treat each verification under paragraph (2) or paragraph (4) as valid for a period of three years. (B) An autonomous weapon system or artificial intelligence capability that is a substantially similar variant of another autonomous weapon system or artificial intelligence capability that is verified under paragraph (2) or paragraph (4) shall also be treated as verified. (4) Subsequent review and verification (A) For any autonomous weapon system or artificial intelligence capability that was previously verified under paragraph (2)(A) or exempted under any predecessor review process under Department of Defense Directive 3000.09 (relating to Autonomy in Weapon Systems), that does not currently have a valid verification pursuant to paragraph (3), the Secretary shall ensure that it undergoes subsequent review and verification under such paragraph. (B) For any autonomous weapon system or artificial intelligence capability that was previously verified under paragraph (2)(B) or exempted under any predecessor review process under Department of Defense Directive 3000.09 (relating to Autonomy in Weapon Systems) that does not currently have a valid verification pursuant to paragraph (3), the Secretary shall ensure that it undergoes subsequent review and verification under such paragraph as if it had not been deployed. (5) Privacy impact assessments (A) For each artificial intelligence capability classified under subsection (c)(1)(C)(ii), the Secretary ensure that a privacy impact assessment is conducted by the Director for Privacy, Civil Liberties and Transparency. (B) In carrying out a privacy impact assessment under subparagraph (A), the Director may consult with such technical and policy experts in the Department of Defense or elsewhere in the Federal Government as the Director considers appropriate. (d) Exceptions The following categories of autonomous weapon systems are not subject to the policy set forth in subsection (a) or the requirements of subsection (b) and (c): (1) Operator-supervised autonomous weapon systems used to select and engage materiel targets to intercept attempted time-critical or saturation attacks. (2) Operator-supervised autonomous weapon systems used to select and engage materiel targets for defending operationally deployed remotely piloted or autonomous vehicles or vessels. (3) Autonomous or semi-autonomous cyberspace capabilities, reasonably judged to be non-lethal in nature. (4) Unarmed platforms, whether remotely operated or operated by onboard personnel, and whether autonomous or semi-autonomous. (5) Unguided munitions. (6) Munitions manually guided by the operator. (7) Mines. (8) Unexploded explosive ordnance. (9) Autonomous or semi-autonomous systems that are not weapon systems. (10) Weapon systems employing autonomous terminal guidance to acquire and track a target individually designated by a human operator prior to or at the time of launch, where the system's autonomous function does not involve discrimination among potential targets. (e) Prohibitions on certain uses of autonomy or artificial intelligence capabilities Except as may be provided in another statute, the Secretary may not use autonomy or an artificial intelligence capability for any of the following use cases: (1) For the decision to initiate the launch or detonation of a nuclear weapon. (2) For the monitoring, tracking, profiling, or targeting of an individual or group of individuals reasonably believed to be in the United States or the collection, querying, or analysis of information about the same not otherwise permissible under the Constitution of the United States, regardless of the origin of the data used, except for activities conducted in accordance with applicable provisions of law. (3) In the employment of lethal force by autonomous weapon without incorporation of appropriate levels of human judgment. (f) Verification, validation, testing and evaluation of autonomous weapon systems leveraging autonomy or artificial intelligence systems (1) In general For each autonomy and artificial intelligence system that is covered by the policy set forth in subsection (a), regardless of the acquisition pathway or test and evaluation oversight status for an autonomous weapon system or artificial intelligence capability, the Director for Operational Test and Evaluation, or the Director’s designee, shall ensure the autonomous weapon system or artificial intelligence capability functions as anticipated in realistic operational environments against adaptive adversaries and are sufficiently robust to minimize failures, including by ensuring— (A) such autonomous weapon system or artificial intelligence capability goes through rigorous hardware and software verification and validation and realistic system developmental and operational test and evaluation, including analysis of unanticipated emergent behavior to assess system performance, capability, reliability, effectiveness, and suitability under realistic conditions, including possible adversary actions, consistent with the potential consequences of unintended engagement or unauthorized parties interfering with the operation of the system or capability; (B) hardware and software verification and validation include iterative cyber test and evaluation in accordance with Department of Defense Instruction 5000.89 (relating to Test and Evaluation), or successor instruction, to verify that the autonomous weapon system or artificial intelligence capability is resilient and survivable in contested cyberspace, if relevant to the purpose and mission of the system or capability; (C) systems incorporating autonomy or artificial intelligence capabilities go through rigorous developmental and operational test and evaluation to verify and validate that the autonomous weapon system or artificial intelligence capability is robust according to design requirements; (D) test and evaluation of systems incorporating autonomy or artificial intelligence capabilities include testing to confirm that their autonomy or artificial intelligence algorithms can be rapidly reprogrammed on new input data to enable timely correction of any unintended system behaviors that may be observed or discovered during future system operations; (E) adequate training, tactics, techniques, procedures, and doctrine are available, periodically reviewed, and used by system operators and commanders to understand the functioning, capabilities, and limitations of the system’s autonomy or artificial intelligence in realistic operational conditions; (F) system design and human-machine interfaces are readily understandable to trained operators, with clear ability to activate and deactivate system functions; (G) after initial operational testing and evaluation, as directed by the Director, system data is collected and any further changes to the system undergo appropriate verification, validation, test, and evaluation to ensure that critical safety features have not been degraded; (H) system software is tested using the best means and methods available to the Department to validate that critical safety features have not been degraded; (I) automated testing tools, such as modeling and simulation, are used whenever feasible; (J) testing identifies any new operating states and other relevant changes in the autonomous weapon system or artificial intelligence capability; (K) as directed by the Director— (i) each new or substantially revised operating state undergoes appropriate and tailored additional test and evaluation to characterize the system behavior in that new operating state; and (ii) whole system follow-on operational test and evaluation when required due to changes to the state transition matrix; and (L) in coordination with the Under Secretary for Research and Engineering and Director, the owning component of the Department provides for monitoring to identify and address when changes to the system design or operational environment require additional testing and evaluation to provide sufficient confidence that the system will continue to avoid unintended engagements and resist interference by unauthorized parties. (2) Applicability of requirements (A) For systems and capabilities assigned Level 1 under subsection (c)(1)(A)(ii), the requirements of subparagraphs (A) through (L) of paragraph (1) shall apply only to the extent relevant to the system's authorized functions and intended operational environment. (B) For systems and capabilities assigned Level 2 under subsection (c)(1)(A)(ii), the requirements of subparagraphs (A) through (L) of paragraph (1) shall apply in full. (g) Autonomy and Artificial Intelligence Systems Working Group (1) Establishment (A) The Secretary shall establish and charter a working group for the purposes set forth in paragraph (2). (B) The working group established under subparagraph (A) shall be known as the Autonomy and Artificial Intelligence Working Group. (2) Purposes The purposes set forth in this paragraph are as follows: (A) To support the Under Secretary of Defense for Research and Engineering, and the Vice Chairman of the Joint Chiefs of Staff in considering the full range of relevant Department interests during the review of autonomous weapon systems and artificial intelligence capabilities before formal development. (B) To support the Under Secretary of Defense for Research and Engineering, the Under Secretary of Defense for Acquisition and Sustainment, and the Vice Chairman of the Joint Chiefs of Staff, in considering the full range of relevant Department interests during the review of autonomous weapon systems before deployment. (C) When requested by appropriate representatives of the secretaries of the military departments, the Commander of United States Special Operations Command, or, when applicable, a director of a defense agency or a Department of Defense Field Activity— (i) to advise whether a given weapon system requires senior-level approval in accordance with this section; and (ii) to help identify and advise on addressing potential issues presented by a given weapon system during a potential senior-level review in accordance with this section. (D) To develop and issue safety standards for use of autonomy and artificial intelligence capabilities in evaluation of such capabilities. (h) Incident repository (1) Establishment The Secretary of Defense shall establish and maintain a centralized repository for reporting, collecting, and analyzing incidents involving autonomous weapon systems and artificial intelligence capabilities used by the Department of Defense— (A) to document operational incidents, weapon system failures, unintended weapon system behaviors, or near-miss events involving autonomous weapon systems and artificial intelligence capabilities; (B) to enable the Department to identify systemic risks, software errors, or operational vulnerabilities associated with such weapon systems and capabilities; and (C) to promote institutional learning and continuous improvement in the design, testing, and operational employment of autonomous weapon systems and artificial intelligence capabilities. (2) Reporting requirements The Secretary shall establish procedures requiring the reporting of incidents described in paragraph (1), including— (A) operational malfunctions affecting the use of force; (B) unintended or anomalous weapon system behavior; (C) instances in which autonomous weapon systems or artificial intelligence capabilities selected or engaged targets inconsistent with commander intent or rules of engagement, regardless of whether engagement was ultimately executed; (D) human-machine interaction failures affecting command or operator control; and (E) other safety-related events determined appropriate by the Secretary. (3) Safety reporting model In establishing the repository under paragraph (1), the Secretary shall, to the extent practicable, model reporting practices on aviation safety reporting systems used to identify and mitigate systemic risks in complex operational environments. (4) Analysis and dissemination The Secretary shall ensure that information collected through the repository established under paragraph (1) is analyzed to identify trends and lessons learned, and that appropriate findings are disseminated across the Department of Defense to improve system design, training, and operational procedures. (5) Protection of sensitive information Information contained in the repository established under paragraph (1) shall be handled in accordance with applicable classification, operational security, and national security requirements. (i) Annual report Not later than January 31 of each year until January 31, 2037, the Secretary shall submit to the congressional defense committees an annual report on the administration of this section. (j) Applicability (1) In general The requirements of this section apply to— (A) lethal autonomous weapon systems; and (B) artificial intelligence capabilities used in lethal targeting engagements that support the validation of a target, the decision to engage a target, the acquisition of a target during the weapons engagement process, and development of engagement parameters for a designated target. (2) Exclusion Systems and capabilities not described in paragraph (1) are not subject to the requirements of this section. (k) Rule of construction Nothing in this section shall be construed as preventing the lawful use of military capabilities and platforms or preventing the rapid development and deployment of military capabilities and platforms that rely upon or leverage artificial intelligence in a national security crisis, miliary conflict, or war. This section is intended to be temporary in nature until the governed technologies mature to the point of requiring less direct oversight in their development and employment. (l) Definitions In this section: (1) The term artificial intelligence has the meaning given the term section 5002 of the National Artificial Intelligence Initiative Act of 2020 (15 U.S.C. 9401). (2) (A) The term autonomous weapon systems means a weapon system that, once activated, can select and engage targets without further intervention by an operator. Such term includes operator-supervised autonomous weapon systems that are designed to allow operators to override operation of the weapon system but can select and engage targets without further operator input after activation. (B) Such term does not include a weapon system solely because it employs autonomous terminal guidance to acquire and track a target class designated by a human operator prior to or at the time of launch. (3) The term autonomy means a capability (or set of capabilities) that enables a particular action of a system to be automatic or, within specified boundaries self-governing with minimal human oversight. (4) The term decision to engage means the determination by a human official with weapons release authority to authorize lethal or non-lethal force against a validated target. (5) The term development of engagement parameters means the calculation of the conditions, geometry, timing, and constraints under which a designated target will be engaged, consistent with rules of engagement and collateral effect limitations (6) The term target acquisition means the process by which a system locates, identifies, and tracks a designated target in preparation for engagement. (7) The term target validation means the process of confirming that a potential target meets criteria for engagement under applicable law and rules of engagement, including verification of target identity and status. (8) The term ultimate human responsibility, with respect to a weapon system, means the ability of a human commander or operator to exercise informed human agency— (A) to understand the operational context of the weapon system through continuous training, capacity building, and integration of design and testing features that strengthen human understanding and effective oversight; (B) to supervise, intervene in, or terminate the use of force by the system, thereby guaranteeing human responsibility, when appropriate given system design; and (C) to ensure compliance with all applicable United States law and international law, including applicable treaties, weapon system safety rules, the law of armed conflict, and rules of engagement..