Support for evaluation of artificial intelligence automation

(a) Prize competition required Not later than 270 days after the date of the enactment of this Act, the Secretary of Commerce, acting through the Director of the National Institute of Standards and Technology (in this section referred to as the Director), shall commence carrying out at least 1 prize competition under section 24 of the Stevenson-Wydler Technology Innovation Act of 1980 (15 U.S.C. 3719) to develop benchmarks or similar reproducible methods to quantitatively measure the ability of artificial intelligence to automate or augment tasks or occupations, with the primary purpose of improving forecasts of the impacts that artificial intelligence may have on workers and the retraining needs of workers. (b) Consultation In carrying out the prize competition required by subsection (a), the Director shall consult with the Secretary of Commerce, the Secretary of Labor, the Commissioner of the Bureau of Labor Statistics, and the Director of the National Science Foundation. (c) Structure and evaluation criteria (1) In general The Director shall develop the structure and evaluation criteria for a prize competition carried out under subsection (a) and a companion grant or cooperative agreement awarded under subsection (d) in accordance with the primary purpose described in subsection (a). (2) Competition scoping The Director may structure a competition under subsection (a) into 1 or more categories, including by the occupation impacted or the capability domain of the artificial intelligence. (3) Category selection In selecting the categories under paragraph (2), the Director shall consider— (A) the informativeness of a category in evaluating the direct or indirect impacts on labor markets, which may include factors such as the number of individuals directly or indirectly impacted by the automation or augmentation and the economic security or ability to find new employment opportunities of the individuals impacted; (B) the quality or efficacy of the submissions for a category; (C) the potential for a category to result in the development of benchmarks that complement existing benchmarks; (D) the potential for a category to result in the development of benchmarks that are underfunded by private contributions relative to the public value of the benchmarks; and (E) other factors determined appropriate by the Director, taking into account input gathered under section 101. (4) Competition phasing The Director may structure a competition under subsection (a) into 1 or more phases (such as design and prototype stages). (5) Evaluation considerations In developing the evaluation criteria for the benchmarks or similar methods submitted for prize competitions under subsection (a), the Director shall, to the extent practicable, seek to avoid or mitigate common problems affecting the value and feasibility of artificial intelligence benchmarks, including— (A) training data contamination that is difficult to detect; (B) rapid loss of discriminatory value over time due to a metric having a low ceiling for performance; and (C) other common problems, including problems identified in the input collected under section 101. (d) Companion grants and cooperative agreements In carrying out this section, the Director may, in addition to carrying out a prize competition under subsection (a), award a grant or enter into a cooperative agreement to support the design, construction, validation, and maintenance of a benchmark or similar reproducible method described in subsection (a), including support for— (1) data collection and labeling; (2) evaluator training and rubric development; and (3) third-party replication and inter rater reliability testing. (e) Program administration The Director may enter into contracts, cooperative agreements, or other agreements with for-profit or nonprofit entities, or State, territorial, local, or Tribal agencies, to design and administer prize competitions under subsection (a) and related grant and cooperative agreement activities under subsection (d). (f) Authorization of appropriations There is authorized to be appropriated to the Secretary of Commerce to carry out this section $7,000,000 for the period of fiscal years 2026 through 2030.