This two-part blog post was written for learning and development (L&D) leaders and professionals who want to quantify the impact of their training initiatives. Specifically, it shows how to demonstrate this through training evaluations. It provides a six-step process for evaluating training programs at five distinct evaluation levels: 1) learner satisfaction; 2) skills acquisition; 3) on-the-job application; 4) business impact; and 5) return on investment. This article outlines the first three steps: 1) selecting a training program for evaluation; 2) setting expectations using objectives; and 3) designing a data collection plan. The second article tackles the remaining three steps: 4) collecting the data; 5) compiling the data; and 6) presenting the data to stakeholders. In both articles, we use the terms participants and employees to refer to any person who consumes the training program.
Capturing the business impact of training programs is critical to the success of your training department. Training programs, including eLearning courses, exist to “contribute to the business results” (Kirkpatrick 2016, 79). Therefore, it is important to connect performance of training programs to measurable business results. Senior management want to see the payoffs for investing money, material and time into training programs. Evaluations are an effective way to gather data, at each level of the development process, and to demonstrate the business impact of the training initiative.
Although evaluation results have several benefits, you may still view them negatively. Evaluations can help justify training budgets, identify training programs that need to be improved or discontinued. In addition, they can show the contribution training programs make to the company’s financial success. L&D leaders and professionals have mixed views on evaluations related to the business impact of training programs. They may resist performing evaluations due to lack of resources, lack of support, lack of adequate measurement tools and lack of skills. In addition, they may perceive unfavorable results as a threat to their position in the company (Griffin 2014, 55).
Approaching training evaluations in a systematic way can resolve some of these issues. A systematic approach makes the evaluation process manageable and efficient. A manageable and efficient process may reduce the resistance L&D leaders and professionals display toward the process. In time, they may be more inclined to implement evaluation programs at all stages. As mentioned earlier, this evaluation approach involves six steps: 1) select the training program; 2) set expected objectives; 3) design data collection plan; 4) gather data; 5) compile data; and 6) present data. The remainder of this article addresses steps one to three. The second article examines steps four to six.
Step 1: Select the training program
Evaluating every training program in your department may be impractical. The materials, time and resources required to evaluate all the programs would overwhelm your team. Especially if your team is already stretched to the limit. In addition, not all programs deserve to be evaluated. Some programs will align with your company’s business goals and others are nice to haves. Evaluating the nice-to-have programs will not give you the most bang for your buck. Evaluation data from training programs that align with your company’s business goals offer the most impact. Start small with one training program. Select a program whose performance closely aligns with the business goal. This allows you to demonstrate to senior leaders how the program links to the business goal and adds value to the company’s financial success.
Example: A company that makes widgets loses X amount of dollars per year due to defective widgets. They want to reduce this loss from X dollars to 15% of X. They determine the defective widgets are due to inconsistent quality assurance procedures among the employees. They want to create a training program that trains employees on one uniform quality assurance procedure. If successful, the training program contributes to the company’s goal of reducing the financial loss due to defective widgets. This training program aligns with the company’s goal and would clearly demonstrate how it contributes to the company’s financial success.
Step 2: Determine the expected objectives
Establish objectives at each level of evaluation to set your training program up for success. Objectives set expectations for the outcome of the program and ensures all stakeholders are on the same page. They ensure everyone involved perceives the same measure of success. It establishes a metric for measuring the evaluation data.
Think of the evaluation process as a road trip where you make several stops on your way to your final destination. Your final destination is the highest-level business objective. The stops before you reach your final destination are lower-level objectives. Between each stop, you reach landmarks that confirm you are on track to reach the next stop. The landmarks represent indicators that show you are on your way to meet the next objective.
Start with the highest-level objective: level 5. Establish the objective; then identify the indicators required to meet the objective. For example, six months after completing the program, the participant’s monthly sales will increase by 15%. The indicators required to reach this objective would be to have the sales manager record the participant’s sales at one, two, three, four and five months after completion of the program. Repeat the process for levels 4, 3, 2 and 1.
Create objectives that are specific, measurable, achievable, relevant and time bound (SMART). A SMART approach to identifying the objectives ensures they are clear, focused, and productive. Setting SMART objectives increase the likelihood the training initiative will be successful from the perspective of all stakeholders. In addition, it provides a system to measure the results of the evaluation.
Level 5: Return-on-Investment (ROI) Objectives
Consult with senior leaders and other stakeholders to determine the expected ROI (i.e., breakeven or 5% ROI). According to two training evaluation experts (Phillips and Phillips 2016, 22) ROI objectives “… set expectations for the program’s economic contribution …” to the business. There are several options to set objectives for the ROI. One option is to calculate the total cost of the training program and the net gain from the investment. Then, subtract the cost from the gain. For example, an eLearning program about reducing company travel expenses costs $50,000 to develop and distribute. After the launch of the program, the company saw a $60,000 reduction in travel expense costs from the employees who participated in the program. This resulted in a $10,000 ROI ($60,000 Gain – $50,000 Cost). Another option is to identify the intangible benefits of the program. For example, the program increases employee engagement and attracts more top talent to the company.
Sample Objective: Six months after the first participant completes the program, the program will return a net gain of 15% over the total program cost.
Level 4: Business Impact Objectives
Define the business impact the training will have on the organization. The business impact is the benefit realized from the project. It states the change that occurred due to the implementation of the training project. Business impact opportunities are plentiful: reduce expenses, grow product sales, reduce regulatory fines, increase employee retention, increase customer satisfaction and much more.
Sample Objective: Six months after completing the training, participants will score at least an 8 (on a scale of 1 to 10) on their customer satisfaction rating.
Level 3: On-the-Job Application Objective
Set expectations for how participants will apply the training (i.e., use company’s five-step process to resolve customer complaints) on the job. Collaborate with managers and supervisors to determine what success on the job entails. List the performance(s) participants must demonstrate to meet the level 2 objective.
Sample Objective: Three months after completing the program, participants will perform the five-step complaint resolution process when interacting with customers who raise a complaint.
Level 2: Skills Acquisition Objective
Outline standards for when participants reach a specific level of skills acquisition. Use the learning objectives identified during the design phase of the instructional design process. Most instructional designers and other L&D professionals are familiar with these types of objectives.
Sample Objective: At the end of the training, participants will be able to demonstrate the five-step customer-complaint resolution process during a role-play exercise.
Level 1: Learner Satisfaction Objective
Determine expectations regarding how the learners will feel about the relevance of the training, importance of the training and intention to use the training on the job. Get input from previous participants, facilitators and managers to design these expectations.
Sample Objective: After participating in the training, participants will rate the training as having at least a 4 rating (on a scale of 1 to 5) of relevance to their job.
Objectives tell you where you are going and alert you when you arrive. They should be specific, measurable, achievable, relevant and time bound (SMART). Start with the end in mind when identifying objectives. Begin with level 5 and work your way down to level 1.
Step 3: Design the data collection plan
Design the data collection plan during the design phase for the training program. Doing it as an afterthought reduces the probability of success. According to Phillips and Phillips (2016, 43) ‘the data collection plan should answer five fundamental questions: what to ask (data collection questions); how to ask (data collection method); whom to ask (data source); when to ask (data collection timing); and, who is going to ask (data collection team).’
Data Collection Questions
Asking the right questions is a two-part endeavor. First, refer to the objectives created for each level of evaluation in step 2. The objectives determine the questions to ask during each phase of the evaluation process. Then, list the specific indicators for reaching those objectives. For example, to gather data on the business impact objective to reduce employee expenses 15% in six months, you might ask the employee’s manager to provide a monthly expense report. You are looking to see if the expenses are going down over the six-month period. Consistent reduction in expenses indicates you are on track to achieve the objective.
Data Collection Method
Determine the data collection methods and instruments (e.g., surveys, observations, tests or interviews). Identify the tools that will help you collect the highest quality and most relevant data. The collection method should facilitate an easy and efficient way of collecting data. For example, using quizzes may be useful for collecting skills acquisition data from participants but it would not be useful to collect data regarding employee sales for a business impact objective. In this case, a better tool may be the company’s client relationship management software (i.e., Salesforce). This software records each employee’s monthly sales, which indicates if the employee met the objective.
Identify trustworthy sources of data. These data sources may include people and systems (e.g., participants, supervisors, data from the learning management system or data from the human resources information system). Trustworthy sources are those “… that are the closest to the measure of interest” (Phillips and Phillips 2016, 46). For example, a call center may collect customer feedback scores from the company’s call center software. Since the call center software collects data directly from the customer, it is the closest source to the customer who has relevant data.
Data Collection Timeframe
Establish a timeframe for collecting the data (e.g., before the training, during the training or after the training). Each objective written in step 2 should be a SMART objective that identifies a timeline to achieve the objective. Use this timeframe to gather the data. For example, if you expect participants to become fluent in a process after performing it for three months, then gather data about the participants’ performance at the three-month mark. Training evaluation experts suggest considering when the expectations for business measures and behavior changes take place or when stakeholders want the data (Phillips and Phillips 2016, 46). For example, a good time for collecting ROI or business impact data may be just before senior management is scheduled for a budget meeting. The data could provide valuable insight to help them make a better decision. Be sure to allow enough time to complete the evaluation and effectively present that data to stakeholders.
Data Collection Team
List the people who and/or systems that (e.g., managers, facilitators, trainers, HR personnel) will play a role in the data collection process and define their responsibilities. Determine who will be responsible for collecting the data around the convenience of the data collection process. For example, a customer service representative manager may be the best person to collect data regarding the objectives related to on-the-job application of the training because they regularly observe participants on the job. The manager’s responsibility may include observing employees and completing surveys regarding the observations.
Design a data collection plan that includes the data collection questions, method, source, timeframe and team. The plan may look something like the template below.
Start evaluating the ROI on your training program today. Download an MS Word version of the Data Collection Plan template.
DATA COLLECTION PLAN TEMPLATE DOWNLOAD FORM
Use this six-step process to evaluate training programs at five distinct evaluation levels: 1) learner satisfaction; 2) skills acquisition; 3) on-the-job application; 4) business impact; and 5) return on investment. First, select one training program that aligns with your company’s business goal. Second, set expectations using SMART objectives. Begin with level 5 and work your way down to level 1. Third, design a data collection plan that includes data collection questions, methods, sources, timeframe and team. The next article tackles the remaining three steps: 4) collecting the data; 5) compiling the data; and 6) presenting the data to stakeholders.
If you have any questions about this article or our company, please contact us at firstname.lastname@example.org or (727) 897-5456.
Richard Griffin. 2014. Complete Training Evaluation: The Comprehensive Guide to Measuring Return on Investment. London: Kogan Page. https://search-ebscohost-com.cobbcat.idm.oclc.org/login.aspx?direct=true&db=nlebk&AN=816417&site=eds-live&scope=site.
Kirkpatrick, James D., and Wendy Kayser Kirkpatrick. 2016. Kirkpatrick’s Four Levels of Training Evaluation. Alexandria, VA: Association For Talent Development. https://search-ebscohost-com.cobbcat.idm.oclc.org/login.aspx?direct=true&db=nlebk&AN=1364594&site=eds-live&scope=site.
Phillips, Patricia Pulliam, and Jack J. Phillips. 2016. Real World Training Evaluation: Navigating Common Constraints for Exceptional Results. Alexandria, VA: Association for Talent Development. https://search-ebscohost-com.cobbcat.idm.oclc.org/login.aspx?direct=true&db=nlebk&AN=881224&site=eds-live&scope=site.