Pilot program on testbeds for development of measurement standards for evaluation of artificial intelligence systems

(a) Pilot program required (1) In general The Director shall, in coordination with the Secretary of Energy and after consultation with the Working Group, carry out a pilot program of testbeds to assess the feasibility and advisability of developing measurement standards for the evaluation of artificial intelligence systems used by Federal agencies through the use of testbeds. (2) Process The Director shall carry out the pilot program required by paragraph (1) by building and demonstrating an iterative process whereby measurement science is advanced and standards are developed using testbeds in accordance with subsection (d), and the results of the tests are reviewed by stakeholders from government, the private sector, and academia. (b) Memorandum of Understanding (1) In general Not later than 180 days after the date of the enactment of this Act, the Secretary of Commerce and the Secretary of Energy shall enter into a memorandum of understanding to implement the coordination between the Secretary of Energy and the Director required by paragraph (2). (2) Requirements The memorandum shall be sufficient to ensure the Institute and other Federal agencies have access, as may be necessary and as determined by the Secretary of Energy, to the resources, personnel, and facilities at the Department of Energy, including the cross-cutting research and development programs to support the activities of the pilot program require by subsection (a)(1). (3) Renegotiations Not later than 2 years after the establishment of the pilot program required by subsection (a)(1), and every 2 years thereafter, Secretary of Commerce and the Secretary of Energy shall reassess the memorandum of understanding to ensure that it meets the needs of the agencies and adequately supports the activities of the pilot program. (c) Artificial Intelligence Testing Working Group (1) Establishment (A) In general Not later than 90 days after the memorandum of understanding required by subsection (b)(1) is signed, the Secretary of Commerce shall establish a working group to provide advice to the Secretary of Commerce and the Secretary of Energy on the development of measurement standards for the evaluation of artificial intelligence systems under the pilot program required by subsection (a). (B) Designation The working group established under subparagraph (A) shall be known as the Artificial Intelligence Testing Working Group. (2) Membership (A) In general The Working Group shall be comprised of not more than 10 members that include the following: (i) The Secretary of Commerce or a designee of the Secretary of Commerce from the Department of Commerce. (ii) The Secretary of Energy or a designee of the Secretary of Energy from the Department of Energy. (iii) The Director or a designee of the Director from the Institute. (iv) Representatives from the private sector, as the members described in clauses (i) through (iii) consider appropriate. (v) Representatives from academic institutions, as the members described in clauses (i) through (iii) consider appropriate. (B) Prohibition No citizen of a covered foreign country may serve as a member of the Working Group. (3) Strategy for measurement standards (A) In general Not later than 1 year after the date on which the Working Group is established under paragraph (1), the Working Group shall develop a strategy for the development of measurement standards for the evaluation of artificial intelligence systems. (B) Elements The strategy required by subparagraph (A) shall— (i) identify measurement standards that the members of the Working Group consider necessary, including standards with respect to an artificial intelligence system for— (I) reliability; (II) performance; (III) capability; (IV) interpretability; (V) security, including the propensity of the system for leaking data; (VI) privacy; (VII) data bias; and (VIII) bounds for applicability and use cases; (ii) include a proposed blueprint for developing the measurement standards; (iii) identify the applications or artificial intelligence systems to which the standards should initially apply; and (iv) identify metrics for the assessment of the pilot program. (C) Publication The Secretary of Commerce shall publish on a publicly accessible website of the Department of Commerce the strategy for measurement standards required by subparagraph (A) to provide the information contained in the strategy to stakeholders, including code and model developers, researchers, and industrial users of artificial intelligence systems. (d) Development of Testbeds for measurement standards (1) In general Not later than 2 years after the date of the enactment of this Act, the Director shall, in collaboration with the Secretary of Energy and in coordination with the Working Group, develop, in accordance with the strategy developed under subsection (c)(3)(A), testbeds on which to demonstrate the development of measurement standards for the evaluation of artificial intelligence systems to be used by Federal agencies. (2) Experts In carrying out paragraph (1), the Director may hire any expert the Director considers necessary, including experts from academia, industry, and standards development organizations. (e) Report Not later than 180 days after the date on which the Director completes the first testbed demonstration and development of measurement standards under subsection (d)(1), the Director shall submit to Congress a report that— (1) reviews the findings of the Director with respect to the initial demonstration carried out under this section; (2) recommends revisions to the pilot program plan, including by identifying any resources, funding, or personnel that is needed to improve and continue the development of testbeds and measurement standards for the evaluation of artificial intelligence systems for multiple applications; and (3) makes recommendations for legislative or administrative action to advance the development of measurement standards for the evaluation of artificial intelligence systems.