Part 6. Human Resources Management

Chapter 410. Learning and Education

Section 9. Training Evaluation Policy

6.410.9 Training Evaluation Policy

Manual Transmittal

August 16, 2021

Purpose

(1) This transmits revised Internal Revenue Manual (IRM) 6.410.9 Training Evaluation Policy for Servicewide Learning and Education.

(2) This IRM established general policy and guidance needed to conduct Levels 1-4 Training Evaluations.

Material Changes

(1) IRM 6.410.9.1

  1. Paren (5) updated name of program owner.

  2. Paren (6) removed redundant terminology.

(2) IRM 6.410.9.1

  1. Deleted “This IRM provides essential information on the policy and procedures for developing, delivering and managing Levels 1 – 4 training evaluations at the IRS.”

  2. Paren (2) updated IRM reference and hyperlink.

(3) 6.410.9.2

  1. Revised title

  2. Deleted obsolete definitions: Blended learning, Critical training, Curriculum-based training, Mission-critical occupations, Technology-enabled learning,

  3. Added definitions applicable to the new policies and procedures: Item, Learning and Education, Mission Critical Training, Priority 1, Priority 2, Priority 3, Priority 4

  4. Updated hyperlinks.

  5. Added Paren (2)

(4) 6.410.9.2.1 subsection deleted.

(5) 6.410.9.3 New subsection, Roles and Responsibilities.

(6) 6.410.9.3.1 Subsection deleted.

(7) 6.410.9.3.2 Subsection deleted.

(8) 6.410.9.3.4 Revised title, format, and content.

(9) 6.410.9.3.4.1 Revised title, format, and content.

(10) 6.410.9.3.4.2 Revised title, format, and content.

(11) 6.410.9.3.4.3 New subsection, Level 3 Evaluation: Behavior.

(12) 6.410.9.3.4.4 New subsection, Level 3 Evaluation: Results

(13) 6.410.9.5 Revised title, format, content, and procedures.

(14) 6.410.9.5.1 Revised title, format, content, and procedures.

(15) 6.410.9.5.2 Revised title, format, content, and procedures.

(16) 6.410.9.6 Revised title, format, content, and procedures.

(17) 6.410.9.6.1 Revised title, format, content and procedures.

(18) 6.410.9.6.2 New Subsection, Administering Level 2 Evaluations.

(19) 6.410.9.6.3 New Subsection, Analyzing Level 2 Evaluation Results.

(20) 6.410.9.7 New Subsection, Level 3 Evaluation Policies and Procedures.

(21) 6.410.9.7.1 New Subsection, Level 3 Evaluation Development.

(22) 6.410.9.7.2 New Subsection, Administering Level 3 Evaluations.

(23) 6.410.9.7.3 New Subsection, Analyzing Level 3 Evaluation Results.

(24) 6.410.9.8 New Subsection, Level 4 Evaluation Policies and Procedures.

Effect on Other Documents

This IRM supersedes IRM 6.410.9 dated November 27, 2018, Training Evaluation Policy.

Audience

All Servicewide Learning & Education employees or personnel, including program managers, education specialists and others involved in training activities within each business unit.

Effective Date

(08-16-2021)

Robin D. Bailey Jr., IRS Human Capital Officer

6.410.9.1 (08-16-2021)

Program Scope and Objectives

  1. This IRM provides general policy and guidance needed to conduct Levels 1 - 4 training evaluations at the IRS.

  2. Purpose: This transmittal revises IRM 6.410.9, Training Evaluation Policy for Servicewide Learning & Education (SL&E).

  3. Audience: All Business Units.

  4. Policy Owner: The Human Capital Office (HCO), Enterprise Talent Planning and Development (ETPD) Division .

  5. Program Owner: The HCO-ETPD Servicewide Training and Event Management (STEM) office is responsible for the administration, policy development and updates related to this IRM.

  6. Primary Stakeholders: All SL&E organizations.

  7. Program Goals: This IRM supports the ETPD goal of supporting a culture of continuous self-development and mission-focused learning for employees by providing tools for measuring and reporting the overall effectiveness of training programs at the IRS.

6.410.9.1.1 (08-16-2021)

Background

  1. The IRS, through the joint efforts of its learning and education community, develops and delivers a variety of training courses that are designed to contribute to mission accomplishment and meet organizational goals. Effectively evaluating these training courses helps us determine course improvement opportunities and the overall value that training is adding to business results and the organization’s mission.

  2. The IRS follows established procedures outlined in the instructional design model known as the Training Development Quality Assurance System (TDQAS) to administer training evaluations. Specific procedures address the evaluation of learner reaction, and achievement and the evaluation of job performance and organizational impact. Adherence to TDQAS processes ensures procedural consistency in the overall quality of our training systems and products. IRM 6.410.1, Learning and Education Policy, and its subsections address the actions associated with the evaluations phase of TDQAS.

6.410.9.1.2 (08-16-2021)

Authority

  1. Per Title 5 of the Code of Federal Regulations (5 CFR 410.202), agencies must evaluate their training programs annually to determine how well such plans and programs contribute to mission accomplishment and meet organizational performance goals.

6.410.9.2 (08-16-2021)

Definitions and Resources

  1. IRM 6.410.9 uses the following terms and lists specific training definitions in most subsections of this IRM:

    1. Consolidated Report - A report maintained in the course file that includes data analysis, a summary, and recommendations for improvement based on Level 1, 2, and 3 training evaluation results.

    2. Evaluation Management System (EMS) - The official system of record for managing training evaluation data. Certain evaluation data captured in the Learning Management System (LMS) is systemically transferred to EMS to streamline data collection and provide a more strategic approach in analyzing and interpreting evaluation data.

    3. Item - Any event tracked in the LMS for learning history completions, and/or for tracking usage. Items include but are not limited to: Continuing Professional Education (CPE), courses, and workshops.

    4. Learning and Education (L&E) - Organization within a business unit, or designated area within HCO, that develops courseware and provides training and education support.

    5. Learning Management System (LMS) - The official system of record for training. It offers development and delivery of course evaluations. Access LMS and resources at https://irssource.web.irs.gov/Pages/ITM.aspx.

    6. Mission Critical Training - Any training that is required to meet employee and organizational performance goals, or training that, if not continued, would result in significant loss to the agency measured in terms of lost taxes, customer confidence, or employee productivity. Examples of mission critical training include, but are not limited to:

      • Courses developed in the prior (FY) and designated for the first quarter as mission critical,

      • Recruit and basic training for mission critical and non-mission critical positions,

      • Functional and cross-functional leadership training,

      • Classroom instructor training, on-the-job (OJI) instruction, instructor training workshops,

      • Discretionary Out-Service – unanticipated mission critical needs that need out service funds.

    7. New World Kirkpatrick Model - A four-level blended approach to training evaluation that focuses on transferring learning to behavior and aligning training with organizational goals.

    8. Priority 1 - Mission critical training that must be timely delivered or developed during the FY. This training is necessary for the employee to function in their basic position such as filing season readiness, new hire, and tax law.

    9. Priority 2 - Mission critical training that must be delivered or developed during the FY. The employee can still function in their basic position but needs this training to perform at a higher job level such as higher phase training, specialty training, and CPE.

    10. Priority 3 - Training that should be delivered or developed during the FY but may be postponed until the latter part of the FY or the following FY without significantly impacting operations. This training enhances existing employee by enabling employees to operate more effectively.

    11. Priority 4 - Training that could be postponed to the following FY with no negative impact if there are insufficient training funds or resource limitations. This is training that enhances performance but is not necessary to perform daily activities.

    12. Rubric - A set of criteria for grading assignments. Rubrics usually have evaluative criteria, quality definitions for those criteria at particular levels of achievement, and a scoring strategy.

    13. Training - The process of providing employees the programs, courses, or other instruction they need to develop new skills to perform a task or process and/or enhance or improve current skills in their individual job performance. Effective training may result in observably changed behavior.

    14. Training Development Quality Assurance System (TDQAS) - The instructional design model used by the IRS that guides the processes of training assessment, analysis, design, development, implementation, and evaluation. The TDQAS development model ensures high-quality training products and services. For more information see IRM 6.410.1, Learning and Education Policy, and its subsections.

    15. Training Event - Instruction that is conducted in a structured learning environment (eLearning and/or non-eLearning) and contains behavioral objectives linked to or derived from job competencies or tasks.

  2. Regulations & Resources are as follows:

    1. U.S. Office of Personnel Management Training & Development (5CFR410).

    2. U.S. Office of Personnel Management Training Evaluation Field Guide

    3. National Agreement, Article 30

    4. Servicewide Training and Event Management SharePoint

6.410.9.3 (08-16-2021)

Roles and Responsibilities

  1. The Servicewide Training Evaluation Program (STEP) is responsible for:

    1. Establishing training evaluation policies and procedures for the IRS.

    2. Providing training and resources for L&E professionals such as:

      • Levels of Evaluation Training,

      • Training Evaluation & Measurement for Performance Optimization (TEMPO) Training, and

      • Webinars.

    3. Administrating the EMS, which includes:

      • Issuing appropriate accesses,

      • Contract and billing, and

      • Troubleshooting.

    4. Providing technical support and consultation for training evaluation development.

    5. Reporting Servicewide training evaluation results.

    6. Communicating evaluation information.

    7. Creating standardized Level 1 evaluations designed consistently captures, compiles, quantifies and reports data.

    8. Maintaining the STEP SharePoint site and the *HCO Servicewide Evaluation Program mailbox.

  2. The L&E office is responsible for:

    1. Determining Level 2 through 4 evaluation requirements for training courses, in conjunction with business unit customers.

    2. Submitting annual and/or ad hoc training plans for each business unit, including the evaluation requirements, to the Servicewide Training & Event Tracking System (STETS).

    3. Creating and maintaining items in the LMS, to include:

      • Updating evaluation requirements fields and

      • Associating the right standardized Level 1 training evaluation surveys to items.

    4. Communicating training evaluation responsibilities to individuals throughout their business unit (i.e., provide guidance to training coordinators, instructors, etc. regarding their duties in the training evaluation process).

    5. Submitting evaluation results to include:

      • Alerting Virtual Learning Management (VLM) or Classroom Learning Services (CLS) when standardized paper Level 1 training evaluations must be administered.

      • Communicating Level 2 training evaluation requirements to instructors and ensuring instructors complete Form 14156, Level 2 Instructor Data Capture (IDC), and forward it to VLM or CLS.

      • Compiling and forwarding Level 3 training evaluation results not systemically captured in the EMS (i.e., alternative Level 3 training evaluations) to STEP team at *HCO Servicewide Evaluation Program mailbox.

    6. Creating and maintaining consolidated reports in course files.

    7. Adhering to all policies and procedures set forth for training evaluation in this IRM.

    8. If L&E is unable to meet the evaluation requirements in this IRM due to shifting priorities, limited resources, or other unforeseen circumstances, L&E may submit a Training Evaluation Waiver Request to the STEP team.

    Note:

    If a training event includes multiple courses, apply the policies and procedures contained in this IRM individually to each course within the training event.

  3. The VLM or CLS office is responsible for:

    1. Administering paper Level 1 training evaluations.

    2. Batching the results of paper Level 1 training evaluations and forwarding to the EMS vendor for processing.

    3. Entering data from Form 14156, Level 2 Instructor Data Capture, into the LMS.

    4. Informing instructors of Level 2 training evaluation requirements.

6.410.9.4 (08-16-2021)

Training Evaluation Overview

  1. Evaluation is the phase of the TDQAS in which data is systematically collected, analyzed, and interpreted to determine the effectiveness of training.

  2. The IRS uses a four-level approach to training evaluation, based on one of the leading industry standards in the field of training. This systematic approach, known as the Kirkpatrick Model, with New World Kirkpatrick Model enhancements, ensures that the IRS gathers complete data that measures the effectiveness of training, while enabling course owners to identify course improvement opportunities. The Kirkpatrick Model effectively demonstrates the strategic value that training programs add to business results and the organization’s mission.

  3. The four levels of training evaluation in the New World Kirkpatrick Model are:

    1. Level 1: Reaction - The degree to which participants find the training favorable, engaging and relevant to their jobs.

    2. Level 2: Learning - The degree to which participants acquire the intended knowledge, skills, attitude, confidence and commitment based on their participation in the training.

    3. Level 3: Behavior - The degree to which participants apply what they learned during training when they are back on the job.

    4. Level 4: Results - The degree to which targeted outcomes occur as a result of training. Targeted outcomes are training programs’ highest-level goals. These should tie into the organization’s overall purpose and business need.

6.410.9.4.1 (08-16-2021)

Level 1 Training Evaluation: Reaction

  1. Level 1 training evaluation is the degree to which participants find training favorable, engaging and relevant to their jobs. Its purpose is to assess the immediate reaction of learners to the training.

  2. New World Kirkpatrick Model Level 1 dimensions include:

    1. Customer Satisfaction - participants’ satisfaction with the training.

    2. Engagement - the degree to which participants are actively involved in contributing to the learning experience.

    3. Relevance - the degree to which participants believe they will have the opportunity to use or apply what they learned in training on the job.

  3. The Level 1 evaluation process enables business units to gather and interpret participants’ reactions to the training they received.

6.410.9.4.2 (08-16-2021)

Level 2 Training Evaluation: Learning

  1. Level 2 training evaluation is the degree to which participants acquire the intended knowledge, skills, attitude, confidence and commitment based on their participation in training. Level 2 training evaluation can be either knowledge-based or performance-based.

  2. New World Kirkpatrick Model Level 2 training evaluation dimensions include:

    1. Knowledge - “I know it.”

    2. Skill - “I can do it right now.”

    3. Attitude - “I believe this will be worthwhile to do on the job.”

    4. Confidence - “I think I can do it on the job.”

    5. Commitment - “I intend to do it on the job.”

  3. The Level 2 training evaluation process enables business units to gather and interpret participants’ level of learning based on the training they received.

  4. Level 2 training evaluation can be quantitative or qualitative in nature.

    1. Quantitative tests or assessments are those that express results in numbers. Examples of quantitative Level 2 evaluations include, but are not limited to:

      • Traditional knowledge tests (multiple choice, fill in the blank, calculate the equation, complete the form, etc.)

      • Knowledge checks, and

      • Individual case studies (with rubric provided for scoring).

    2. Qualitative methods don’t yield numerical results or are not easily scored. They are any means by which students receive input and guiding feedback on their relative performance to help them improve. Examples of qualitative Level 2 evaluations include, but are not limited to:

    • Group/team exercises,

    • Simulations,

    • Presentations,

    • Observations,

    • Work reviews,

    • Self-assessments,

    • Pass/fail assessments,

    • Pre/post assessments,

    • Questionnaires,

    • Role plays,

    • Interviews, and

    • Action learning.

    Note:

    Developing a grading rubric to calculate grades and scores can convert most qualitative methods to quantitative methods.

6.410.9.4.3 (08-16-2021)

Level 3 Training Evaluation: Behavior

  1. Level 3 training evaluation is the degree to which participants apply what they learned during training when they are back on the job.

  2. New World Kirkpatrick Model Level 3 training evaluation dimensions include:

    1. Required Drivers - processes and systems that reinforce, monitor, encourage and reward performance. Examples of required drivers include work review checklists, job aids, recognition, coaching, mentoring, etc.

    2. Critical Behaviors - the few, specific actions that will have the biggest impact on the desired results if performed consistently on the job. For the IRS, critical behaviors are the same as enabling learning objectives (ELOs) and/or terminal learning objectives (TLOs).

    3. On-the-Job Learning - a culture and expectation that individuals are responsible for maintaining the knowledge and skills to enhance their own performance.

  3. The Level 3 training evaluation process enables business units to gather and interpret the level of impact the training had on participants behavior.

6.410.9.4.4 (08-16-2021)

Level 4 Training Evaluation: Results

  1. Level 4 training evaluation is the degree to which targeted outcomes occur as a result of training. It assesses the impact of the improvements in the trainees’ performance on the mission of the organization. Business units must be able to identify and provide data on measures that have organizational impact.

  2. New World Kirkpatrick Model Level 4 training evaluation dimensions include leading indicators which are short-term observations and measurements suggesting that critical behaviors are on track to create a positive impact on desired results. Examples of leading indicators include balanced measure ratings, compliance ratings, customer response ratings, etc.

  3. The Level 4 evaluation process enables business units to gather and interpret the results that the training had on the organization.

6.410.9.5 (08-16-2021)

Level 1 Training Evaluation Policies and Procedures

  1. All training events require Level 1 training evaluations:

    1. The L&E office must document the Level 1 evaluation requirements for each item in the LMS. The “Level 1 Evaluation” field must be populated with “Level 1 is required (Required).”

    2. The L&E office must associate the right standardized online Level 1 training evaluation to each item in the LMS. The four standardized online Level 1 training evaluation forms currently available are:

      • L1-CLS-3 (Classroom Course Evaluation) - used to evaluate instructor led, classroom training events.

      • L1-BLD-1 (Blended - Classroom, Computer and Self-Directed Evaluation) - used to evaluate instructor-led training events with classroom and technology-based components.

      • L1-VIR-1 (Virtual Evaluation) - used to evaluate virtual training events.

      • L1-SD-1 (Self-Directed Training Evaluation) - used to evaluate self-directed training events.

    3. Feedback from participants, instructors, the L&E community and the National Treasury Employees Union (NTEU) was considered when developing the standardized Level 1 training evaluation surveys.

    4. By agreement, NTEU will be provided information upon request.

  2. Participants completing Level 1 training evaluations is voluntary but strongly encouraged.

  3. Level 1 training evaluations must be completed during normal duty hours.

  4. Level 1 training evaluations must be anonymous and confidential (i.e., no names, SEIDs, and/or other self-identifying data will be used).

6.410.9.5.1 (08-16-2021)

Administering Level 1 Training Evaluations

  1. Level 1 training evaluations are systemically issued through the LMS when a course with an associated standardized online Level 1 training evaluation is recorded in the employee’s learning history. The LMS systemically captures the responses.

  2. Standardized paper Level 1 training evaluations are available for employees with restricted access to computers or as a reasonable accommodation.

    1. The two standardized paper Level 1 training evaluation forms currently available are:

      • L1-CLS-3 (Classroom Course Evaluation) - used to evaluate instructor led, classroom training events.

      • L1-VIR-1 (Virtual Evaluation) - used to evaluate virtual training events.

    2. The VLM or CLS office must manually administer the standardized paper Level 1 evaluations.

    3. The VLMor CLS office must send the results to the EMS vendor for input into the EMS.

6.410.9.5.2 (08-16-2021)

Analyzing Level 1 Training Evaluation Results

  1. On an annual basis, each L&E organization, through discussion with their business unit customer(s), will determine the priority of training events included on their training plans.

  2. The L&E organization must obtain Level 1 training evaluation results from the LMS and/or the EMS.

  3. The L&E organization must analyze the Level 1 training evaluation results for all Priority 1 and Priority 2 training events within their purview. Analyze Level 1 training evaluation results for Priority 3 and Priority 4 training events as resources allow.

  4. Analysis of Level 1 training evaluation results must be compiled and documented in a consolidated report and maintained in the course file.

  5. Documenting Level 1 training evaluation results analysis must include:

    1. Recommendations for course improvements and/or corrective actions, or

    2. Annotation if there are no recommendations for improvement.

  6. Additional documentation may include, but is not limited to:

    1. Analysis and assessment of Level 1 reports with feedback.

    2. Overall training satisfaction scores.

    3. Evaluative information from instructors to ensure high-quality, effective and efficient delivery of training.

  7. Based on a business unit’s program priorities, one-time training events may be excluded from analysis:

    1. Update the associated item(s) in the LMS to “inactive” once the course information is obsolete.

    2. Annotate the consolidated report as a one-time event.

6.410.9.6 (08-16-2021)

Level 2 Training Evaluation Policies and Procedures

  1. On an annual basis, each L&E organization, through discussion with their business unit customer(s), will determine which training courses require Level 2 training evaluations.

    1. Level 2 training evaluations should focus on mission-critical training programs and Priority 1 and 2 training events but may be required for any training program as determined above.

    2. To obtain a statistically significant sample for measuring the Servicewide training program, at a minimum L&E must require Level 2 training evaluations on 5% of courses delivered for each business unit.

      Note:

      For planning purposes, the 5% will be based on the number of courses delivered in the prior FY.

    3. Consider training and program priorities, training course costs, and available resources when determining Level 2 training evaluation requirements.

  2. The L&E organization must document the Level 2 training evaluation requirements for all training events on the annual business units’ training plan and/or ad hoc requests submitted to STETS.

  3. The L&E organization must document the Level 2 training evaluation requirements for each item in the LMS.

    1. When items are created in the LMS, populate the “Level 2 Evaluation” field with “Level 2 is not required (Optional)” or “Level 2 is required (Required),” as appropriate.

    2. If the Level 2 training evaluation requirement for an item changes after it is initially created in the LMS, update the “Level 2 Evaluation” field in the LMS prior to the start of any associated training events.

  4. Learners must complete Level 2 training evaluations during normal duty hours.

  5. Level 2 training evaluation results used for reporting and analytical purposes must be anonymous and confidential (i.e., no names, SEIDs, and/or other self-identifying data will be recorded).

  6. Level 2 training evaluation results for an individual may be included in a developmental guide, or similar document, for employee development purposes or to provide individualized coaching.

  7. An individual’s Level 2 training evaluation results are not required to be anonymous when being used for employee development purposes or to provide individualized coaching but can only be shared with a training coordinator or the employee’s first level manager.

  8. No adverse actions will be taken against learners based on Level 2 training evaluation results.

  9. Information gathered through Level 2 evaluation will not be used to evaluate participants on their annual performance appraisals.

6.410.9.6.1 (08-16-2021)

Level 2 Training Evaluation Development

  1. Since Level 2 training evaluations measure the knowledge obtained from the subject matter of each individual course, there are no standardized forms or questions that are required to be used in all Level 2 training evaluations.

  2. If a course has a Level 2 training evaluation requirement, L&E must develop a Level 2 evaluation and include in the course. See IRM 6.410.9.4.2(4) for types of Level 2 evaluations.

  3. The L&E organization must develop a rubric (i.e., grading scheme) and methods or procedures for capturing results and include in the course.

  4. If using Level 2 training evaluation results for employee development or coaching purposes, methods and/or procedures for capturing the employee’s individual Level 2 training evaluation results must be developed and included in the course.

6.410.9.6.2 (08-16-2021)

Administering Level 2 Training Evaluations

  1. If a training event has a Level 2 training evaluation requirement, the evaluation must be administered. Typically, Level 2 training evaluations are administered during class, however, there are instances when they might be administered after class.

  2. The instructor is responsible for scoring and/or compiling the results of Level 2 training evaluations.

    Note:

    If an instructor doesn’t have administrator access to the Integrated Talent Management (ITM) , they must contact VLM or CLS for help in gathering electronic Level 2 evaluation results.

  3. Certain Level 2 evaluation data must be captured in the LMS for Servicewide analysis and reporting purposes. For instructor led and/or blended training, the course instructor or business unit designee must:

    1. Administer and score the Level 2 evaluation.

    2. Complete Form 14156, Level 2 Instructor Data Capture (IDC).

    3. Submit the form to VLM or CLS for input into the LMS.

  4. Data captured on the IDC Form includes (as applicable):

    1. Whether or not a Level 2 evaluation was administered, including type of delivery, (e.g., online (AOL), oral (O), qualitative (QL), quantitative (QN)),

    2. Reason if not administered,

    3. Average pre-test score,

    4. Average post-test score,

    5. Percentage of improvement,

    6. Average final test score,

    7. Current trainee count,

    8. Number of trainees completing, and

    9. Percentage of trainees completing.

  5. To capture comprehensive Level 2 training evaluation results for more extensive analysis, L&E must communicate any additional reporting requirements to instructors.

  6. If a Level 2 evaluation is required but not developed and/or administered, the instructor or business unit designee must complete the IDC Form, include the reason the Level 2 was not administered and submit the form to VLM or CLS for input into the LMS.

    Note:

    For CPE courses where accreditation (continuing education units or credits) is sought, the courses must adhere to test requirements and standards of the accrediting authority (i.e., American Institute of Certified Public Accountants or National Association of State Board of Accountancy).

    Criminal Investigation (CI) training courses must adhere to the standards set forth by the Federal Law Enforcement Training Accreditation Board.

6.410.9.6.3 (08-16-2021)

Analyzing Level 2 Training Evaluation Results

  1. Limited Level 2 training evaluation results will be available from the LMS and/or EMS. Access additional Level 2 training evaluation results from data housed in alternative systems, in accordance with procedures established by the business unit. Use comprehensive results for analysis when available.

  2. The L&E organization must analyze results from Level 2 training evaluations for all training events with a Level 2 training evaluation requirement.

  3. If Level 2 training evaluation results are available for courses without a Level 2 training evaluation requirement, analyze results as resources allow.

  4. Compile and document analysis of Level 2 training evaluation results in a consolidated report and maintain in the course file.

  5. Documentation of Level 2 training evaluation results analysis must include:

    1. Recommendations for course improvements and/or corrective actions, or

    2. Annotation if there are no recommendations for improvement.

  6. Additional documentation may include, but is not limited to:

    1. Test item analysis,

    2. Average test scores,

    3. Pre/post assessments,

    4. Final test scores,

    5. Performance-based results, and/or

    6. Level 2 final results from pass/fail, complete/not complete assessments, etc.

6.410.9.7 (08-16-2021)

Level 3 Training Evaluation Policies and Procedures

  1. Annually, each L&E organization, through discussion with their business unit customer(s), will determine which training courses require Level 3 training evaluations.

    1. Level 3 training evaluations must focus on mission-critical training programs and Priority 1 and 2 training events, but can be required for any training program, as determined above.

    2. To obtain a statistically significant sample for measuring the Servicewide training program, at a minimum L&E must require Level 3 training evaluations on 5% of courses delivered for each business unit.

      Note:

      For planning purposes, the 5% will be based on the number of courses delivered in the priorFY.

    3. Consider training and program priorities, training course costs, courses scheduled for revision and available resources when determining Level 3 training evaluation requirements.

  2. The L&E organization must document the Level 3 training evaluation requirements for all training events on the annual business unit training plan and/or ad hoc requests submitted to STETS.

  3. The L&E organization must document the Level 3 training evaluation requirements for each item in the LMS.

    1. When items are created in the LMS, populate the “Level 3 Evaluation” field with “Level 3 is not required (Optional)” or “Level 3 is required (Required),” as appropriate.

    2. If the Level 3 evaluation requirement for an item changes after it is initially created in the LMS, update the “Level 3 Evaluation” field in the LMS.

  4. Administering Level 1 and Level 2 training evaluations is recommended (but not required) prior to administering Level 3 training evaluations for a course.

  5. Participants completing Level 3 training evaluations is voluntary, but strongly encouraged.

  6. Completion of Level 3 training evaluation by the participants’ managers is mandatory.

  7. Managers and participants must complete Level 3 training evaluations during normal duty hours.

  8. Level 3 training evaluation results used for reporting and analytical purposes will be anonymous and confidential (i.e., no names, SEIDs or other self-identifying data will be recorded).

  9. No adverse actions will be taken against participants or managers based on results of Level 3 data reports.

  10. Information gathered through Level 3 evaluation will not be used to evaluate participants on their annual performance appraisals.

6.410.9.7.1 (08-16-2021)

Level 3 Training Evaluation Development

  1. If a training event has a Level 3 training evaluation requirement, Level 3 training evaluations must be developed. This should be completed during the design and development phase of TDQAS, but may be completed later, if necessary.

  2. Level 3 training evaluations must be developed in the EMS.

    Note:

    Paper evaluations may be developed for employees with restricted access to computers or as a reasonable accommodation, but the results must be recorded in the EMS.

  3. Corresponding participant and manager Level 3 training evaluations must be developed simultaneously.

  4. All Level 3 training evaluations must include the mandatory training transfer questions to provide results for Servicewide reporting.

  5. Level 3 evaluation instruments must be reviewed and approved by the STEP team prior to delivery.

    Note:

    Consult the STEP team for guidance when program managers want to use alternative Level 3 training evaluations such as focus group interviews.

6.410.9.7.2 (08-16-2021)

Administering Level 3 Training Evaluations

  1. Level 3 training evaluations must be conducted on all training events with a Level 3 training evaluation requirement.

  2. Administer Level 3 training evaluation after completion of the training event and after employees have been given an opportunity to apply course content while back on the job. They are typically delivered one to six months after training but must be administered within nine months of the end of the training event.

  3. If a required Level 3 training evaluation has not been administered within the typical delivery timeframe, L&E must document the course file with a scheduled administration date.

  4. Once a Level 3 training evaluation pair is reviewed and approved in the EMS, survey links will be sent to L&E to forward to participants and their managers. Electronic Level 3 training evaluation responses are systemically captured.

  5. If a paper Level 3 training evaluation is used, a PDF version will be created and sent to L&E. The L&E office is responsible for:

    1. Administering the Level 3 paper evaluation.

    2. Compiling paper Level 3 training evaluation responses using the batch sheet; and

    3. Submitting the paper responses and batch sheet to the EMS vendor for processing.

  6. The L&E organization may delegate administering, compiling and submitting Level 3 paper evaluations; however, they must communicate the requirements to their designee and confirm completion.

  7. Waiving the use of the EMS in its entirety for Level 3 training evaluations requires STEP written approval; complete and submit a Training Evaluation Waiver Request to detail the business need for an exception.

6.410.9.7.3 (08-16-2021)

Analyzing Level 3 Training Evaluation Results

  1. The L&E organization must obtain Level 3 evaluation results from the EMS.

  2. The L&E organization must analyze results from Level 3 training evaluations for all training events with a Level 3 training evaluation requirement.

  3. If Level 3 evaluation results are available for courses without a Level 3 training evaluation requirement, analyze results as resources allow.

  4. Compile and document analysis of Level 3 training evaluation results in a consolidated report and maintain in the course file.

  5. Documentation of Level 3 training evaluation results analysis must include:

    1. Recommendations for course improvements and/or corrective actions, or

    2. Annotation if there are no recommendations for improvement.

  6. Additional documentation may include, but is not limited to:

    1. Level 3 participant training evaluation results.

    2. Level 3 manager training evaluation results.

    3. Analysis of Level 3 training evaluation results.

6.410.9.8 (08-16-2021)

Level 4 Training Evaluation Policies and Procedures

  1. The L&E organization will confer with their assigned business unit executives or designee(s) during the TDQAS assessment phase to determine:

    1. Program expectations including the customer’s desired results of the training,

    2. If training warrants a Level 4 training evaluation and/or it is feasible,

    3. Whether resources and funding are available and will be committed, and

    4. The extent to which the training impact can be isolated against other factors, and if customer-valued data on business results is available.

  2. The L&E organization must document the Level 4 training evaluation requirements for all training events on the annual business unit training plan and/or ad hoc requests submitted to STETS.

  3. The L&E organization must document the Level 4 training evaluation requirements for each course in the LMS.

    1. When items are created in the LMS, populate the “Level 4 Evaluation” field with “Level 4 is not required (Optional)” or “Level 4 is required (Required),” as appropriate.

    2. If the Level 4 evaluation requirement for an item changes after it is initially created in the LMS, update the “Level 4 Evaluation” field in the LMS.

  4. If a Level 4 training evaluation is required, conduct Levels 1, 2 and 3 training evaluations. Feedback from these levels will provide important quantitative and qualitative data supporting conclusions drawn that links training to business results.

  5. There are currently no Servicewide reporting requirements for Level 4 training evaluation results; however, forward results to the STEP team.

  6. Design Level 4 training evaluation executive reports by following guidelines established by each business unit.

  7. Maintain Level 4 training evaluation results in the course file if available.