📘 Lecture 23 — Training Evaluation (continued)
📖 Overview: This lecture continues the exploration of training evaluation by detailing the specific types of outcomes used to measure training effectiveness, moving beyond the foundational Kirkpatrick model. It explains how reaction, learning, behavior, affective, results, and return on investment (ROI) outcomes are defined, measured, and applied, providing practical examples and research insights to demonstrate their importance in justifying training programs.
🗂️ Topics Covered
The lecture covers the six main categories of training outcomes from Kirkpatrick's framework and beyond: Reaction Outcomes (Level 1), Learning or Cognitive Outcomes (Level 2), Behavior and Skill-Based Outcomes (Levels 2 & 3), Affective Outcomes (Levels 2 & 3), Results (Level 4), and Return on Investment (ROI, Level 5). Each outcome is defined, explained with measurement methods, and linked to its practical application and research findings. The lecture concludes with a case study of how Northwest Airlines used a Training Quality Index (TQI) to evaluate and demonstrate the value of its training department.
📝 Lecture Summary
Reaction Outcomes
Reaction outcomes refer to trainees’ perceptions of the program, including the facilities, trainers, and content. They are often called class or instructor evaluations and are typically collected at the program's conclusion via a questionnaire. A reaction measure should include questions about the trainee’s satisfaction with the instructor, training materials, training administration, clarity of course objectives, and usefulness of the content. These outcomes are Level 1 (reaction) criteria in Kirkpatrick's framework.
🔑 Definition — Reaction Outcomes: Trainees’ perceptions of the training program, including the facilities, trainers, and content (also known as “creature comfort”). 📌 Example: You have probably been asked to complete class or instructor evaluations at the end of a college course or a training program at work.
The lecture notes that conventional wisdom suggests trainees who have positive reactions learn more and are more likely to transfer training. However, recent research challenges this, finding that reactions have the largest relationship with changes in affective learning outcomes and are significantly related to changes in declarative and procedural knowledge (contradicting previous research). For courses like diversity or ethics training, reactions are especially important as they affect learner receptivity to attitude change. Furthermore, when technology is used for instructional delivery, reactions have the strongest relationship with post-training motivation, trainee self-efficacy, and declarative knowledge. This implies that for online or e-learning, it is crucial to ensure easy access and meaningful content linked to job experiences.
💡 Why this matters: Trainee reactions are not just about "creature comfort"; they are a valid predictor of learning, motivation, and knowledge acquisition, especially in technology-based training.
Learning or Cognitive Outcomes
Cognitive outcomes are used to determine the degree to which trainees are familiar with principles, facts, techniques, procedures, or processes emphasized in the training program. They measure what knowledge trainees learned and are Level 2 (learning) criteria in Kirkpatrick's framework.
🔑 Definition — Cognitive Outcomes: Outcomes that measure what knowledge trainees learned in the program (principles, facts, techniques, procedures, or processes). 📐 Measurement: Typically assessed using pencil-and-paper tests. 📌 Example: Items from a pencil-and-paper test measure whether a trainee knows how to make a decision (the process they would use), but they do not determine if the trainee will actually use the skill on the job.
Behavior and Skill-Based Outcomes
Skill-based outcomes assess the level of technical or motor skills and behaviors. They include acquisition or learning of skills (skill learning) and the use of skills on the job (skill transfer). These outcomes relate to Kirkpatrick's Level 2 (learning) and Level 3 (behavior).
🔑 Definition — Skill-Based Outcomes: Outcomes used to assess the level of technical or motor skills and behaviors, including both skill learning and skill transfer. 📐 Measurement: Skill learning can be evaluated by observing performance in work samples (e.g., simulators). Skill transfer is determined by observation, self-ratings, or ratings from peers, managers, and subordinates. 📌 Example: A resident medical student performs surgery while the surgeon observes, gives advice, and assists.
The lecture emphasizes that because research suggests using only self-ratings results in an inaccurately positive assessment of skill or behavior transfer, it is recommended to collect ratings from multiple perspectives (e.g., managers, subordinates, or peers). Table 3 is mentioned as showing a sample rating form used to evaluate a training program for school principals' management skills.
Affective Outcomes
Affective outcomes include attitudes and motivation. Examples that might be collected include tolerance for diversity, motivation to learn, safety attitudes, and customer service orientation. These outcomes can be measured using surveys.
🔑 Definition — Affective Outcomes: Outcomes that include attitudes and motivation (e.g., tolerance for diversity, motivation to learn, safety attitudes, customer service orientation). 📐 Measurement: Typically measured using surveys. 📌 Example: A survey might be used to measure career goals, plans, and interests to evaluate training focusing on employees self-managing their careers.
Affective outcomes relate to Kirkpatrick’s Level 2 (learning) or Level 3 (behavior) depending on how they are evaluated. If trainees are asked about their attitudes on a survey, that is considered a learning measure.
Results
Results are used to determine the training program’s payoff for the company. They are Level 4 (results) criteria in Kirkpatrick’s framework.
🔑 Definition — Results: Outcomes used to determine the training program’s payoff for the company. 📌 Example: Examples include increased production, reduced costs related to employee turnover, accidents, and equipment downtime, as well as improvements in product quality or customer service.
The text provides an example from Kroger, the supermarket chain, which compared cashiers who received computer-based training to those trained in the classroom and on the job. The measures of productivity included rate of scanning grocery items, recognition of produce to be identified and weighed, and the time store offices spent helping cashiers with complex transactions like food stamps and checks.
Return on Investment
Return on investment (ROI) refers to comparing the training’s monetary benefits with the cost of the training. It is often referred to as Level 5 evaluation.
🔑 Definition — Return on Investment (ROI): Comparing the training’s monetary benefits with the cost of the training (often referred to as Level 5 evaluation). 📐 Formula: ROI = (Monetary Benefits - Training Costs) / Training Costs → This compares the net gain from training to its cost.
The lecture details that training costs can be direct and indirect. Direct costs include salaries and benefits for all involved employees, program materials and supplies, equipment or classroom rentals or purchases, and travel costs. Indirect costs are not related directly to the design or delivery of the program, such as general office supplies, facilities, management salaries, and administrative staff support. Benefits are the value the company gains from the training program.
The lecture provides a case study from Northwest Airlines’ technical operations training department. The department developed a Training Quality Index (TQI), a computer application that collects data into five categories: effectiveness, quantity, perceptions, financial impact, and operational impact. For example, knowledge gain under effectiveness is measured by the difference in pre-training and post-training exam scores. The TQI allowed the department to:
- Reduce the cost of training an individual technician by 16 percent.
- Increase customer satisfaction through training.
- Make the case for upper management to provide financial resources for training.
- Improve post-course evaluations, knowledge, and performance gains.
- Justify the cost of training and compare training demand against passenger loads and flying routes.
💡 Why this matters: The TQI system demonstrates how training evaluation can be strategically integrated into a company’s overall strategy, proving the value of the training function and enabling data-driven decisions.
⭐ Key Takeaways
This lecture is critical for understanding the full scope of training evaluation. You must remember the six types of outcomes: Reaction (Level 1, trainee perceptions), Learning/Cognitive (Level 2, knowledge tests), Behavior/Skill (Levels 2 & 3, observing skills), Affective (Levels 2 & 3, attitudes/motivation), Results (Level 4, company payoff like reduced costs), and ROI (Level 5, comparing monetary benefits to costs). Be able to define each, state its Kirkpatrick level, and provide a specific measurement method (e.g., survey, test, observation). Crucially, understand that research has shown reactions are significantly related to learning and are especially important for attitude change, and that skill evaluation is more accurate when using multiple perspectives (not just self-ratings). The Northwest Airlines TQI case is a powerful example of how an integrated evaluation system can justify training and link it to business strategy.
🧠 Quick Revision Questions
- What is the definition of "reaction outcomes" and what are they nicknamed?
- According to recent research, what is the relationship between trainee reactions and learning outcomes (declarative and procedural knowledge)?
- How are "cognitive outcomes" typically measured, and what level of Kirkpatrick's framework do they represent?
- Why is it recommended to collect skill or behavior ratings from multiple perspectives (peers, managers, subordinates) rather than just self-ratings?
- What is the primary difference between "results" outcomes (Level 4) and "return on investment" outcomes (Level 5)?
📘 Lecture 24 — Training Evaluation (continued)
📖 Overview: This lecture continues the discussion of training evaluation, focusing on how to select appropriate training outcomes based on relevance, reliability, discrimination, and practicality. It also examines current evaluation practices, the relationship between different outcome types, and evaluation designs that help ensure valid conclusions about training effectiveness.
🗂️ Topics Covered
This lecture covers the criteria for selecting appropriate training outcomes: relevance (including criterion contamination and deficiency), reliability, discrimination, and practicality. It then reviews current evaluation practices in companies, discussing which outcomes should be collected and why, including the limitations of measuring only reaction and cognitive outcomes. Finally, the lecture introduces evaluation designs and threats to validity, including internal and external validity threats that can undermine confidence in evaluation results.
📝 Lecture Summary
How to select appropriate outcomes
An important issue in choosing outcomes is to determine whether they are appropriate. That is, are these outcomes the best ones to measure to determine whether the training program is effective? Appropriate training outcomes need to be relevant, reliable, discriminative, and practical.
Relevance
Criteria relevance refers to the extent to which training outcomes are related to the learned capabilities emphasized in the training program. The learned capabilities required to succeed in the training program should be the same as those required to be successful on the job. The outcomes collected in training should be as similar as possible to what trainees learned in the program. That is, the outcomes need to be valid measures of learning. One way to ensure the relevancy of the outcomes is to choose outcomes based on the learning objectives for the program. The learning objectives show the expected action, the conditions under which the trainee is to perform, and the level or standard of performance.
Figure 1 shows two ways that training outcomes may lack relevance. Criterion contamination refers to the extent that training outcomes measure inappropriate capabilities or is affected by extraneous conditions. For example, if managers’ evaluations of job performance are used as a training outcome, trainees may receive higher ratings of job performance simply because the managers know they attended the training program, believe the program is valuable, and therefore give high ratings to ensure that the training looks like it positively affects performance. Criteria may also be contaminated if the conditions under which the outcomes are measured vary from the learning environment. That is, trainees may be asked to perform their learned capabilities using equipment, time constraints, or physical working conditions that are not similar to those in the learning environment. For example, trainees may be asked to demonstrate spreadsheet skills using a newer version of spreadsheet software than they used in the training program. This demonstration likely will result in no changes in their spreadsheet skills from pre-training levels. In this case, poor-quality training is not the cause for the lack of change in their spreadsheet skills. Trainees may have learned the necessary spreadsheet skills, but the environment for the evaluation differs substantially from the learning environment, so no change in skill level is observed.
Criteria may also be deficient. Criterion deficiency refers to the failure to measure training outcomes that were emphasized in the training objectives. For example, the objectives of a spreadsheet skills training program emphasize that trainees both understand the commands available on the spreadsheet (e.g., compute) and use the spreadsheet to calculate statistics using a data set. An evaluation design that uses only learning outcomes such as a test of knowledge of the purpose of keystrokes is deficient, because the evaluation does not measure outcomes that were included in the training objectives (e.g., use spreadsheet to compute the mean and standard deviation of a set of data).
🔑 Definition — Criterion contamination: The extent to which training outcomes measure inappropriate capabilities or are affected by extraneous conditions. 🔑 Definition — Criterion deficiency: The failure to measure training outcomes that were emphasized in the training objectives.
Reliability
Reliability refers to the degree to which outcomes can be measured consistently over time. For example, a trainer gives restaurant employees a written test measuring knowledge of safety standards to evaluate a safety training program they attended. The test is given before (pre-training) and after (post-training) employees attend the program. A reliable test includes items for which the meaning or interpretation does not change over time. A reliable test allows the trainer to have confidence that any improvements in post-training test scores from pre-training levels are the result of learning that occurred in the training program, not test characteristics (e.g., items are more understandable the second time) or the test environment (e.g., trainees performed better on the post-training test because the classroom was more comfortable and quieter).
🔑 Definition — Reliability: The degree to which outcomes can be measured consistently over time.
Discrimination
Discrimination refers to the degree to which trainees’ performance on the outcome actually reflects true differences in performance. For example, a paper-and-pencil test that measures electricians’ knowledge of electrical principles must detect true differences in trainees’ knowledge of electrical principles. That is, the test should discriminate on the basis of trainees’ knowledge of electrical principles. (People who score high on the test have a better understanding of the principles of electricity than do those who score low.)
🔑 Definition — Discrimination: The degree to which trainees’ performance on the outcome actually reflects true differences in performance.
Practicality
Practicality refers to the ease with which the outcome measures can be collected. One reason companies give for not including learning, performance, and behavior outcomes in their evaluation of training programs is that collecting them is too burdensome. (It takes too much time and energy, which detracts from the business.) For example, in evaluating a sales training program, it may be impractical to ask customers to rate the salesperson’s behavior because this would place too much of a time commitment on the customer (and probably damage future sales relationships).
🔑 Definition — Practicality: The ease with which the outcome measures can be collected.
EVALUATION PRACTICES
Figure 2 shows outcomes used in training evaluation practices. Surveys of companies’ evaluation practices indicate that reactions (an affective outcome) and cognitive outcomes are the most frequently used outcomes in training evaluation. Despite the less frequent use of cognitive, behavioral, and results outcomes, research suggests that training can have a positive effect on these outcomes. Keep in mind that while most companies are conducting training evaluations, some surveys indicate that 20 percent of all companies are not!
Which Training Outcomes Should Be Collected?
From our discussion of evaluation outcomes and evaluation practices you may have the mistaken impression that it is necessary to collect all five levels of outcomes to evaluate a training program. While collecting all five levels of outcomes is ideal, the training program objectives determine which ones should be linked to the broader business strategy.
To ensure adequate training evaluation, companies should collect outcome measures related to both learning (levels 1 and 2) and transfer of training (levels 3, 4, or 5).
It is important to recognize the limitations of choosing to measure only reaction and cognitive outcomes. Consider the previous discussions of learning and transfer of training. Remember that for training to be successful, learning and transfer of training must occur. Figure 3 shows the multiple objectives of training programs and their implication for choosing evaluation outcomes. Training programs usually have objectives related to both learning and transfer. That is, they want trainees to acquire knowledge and cognitive skill and also to demonstrate the use of the knowledge or strategy in their on-the-job behavior. As a result, to ensure an adequate training evaluation, companies must collect outcome measures related to both learning and transfer.
Ernst Young’s training function uses knowledge testing (level 2) for all of the company’s e-learning courses, which account for 50 percent of training. New courses and programs use behavior transfer (level 3) and business results (level 4). Regardless of the program, the company’s leaders are interested in whether the trainees feel that training has been a good use of their time, money, and whether they would recommend it to other employees (level 1). The training function automatically tracks these outcomes. Managers use training and development to encourage observable behavior changes in employees that will result in business results such as client satisfaction and lower turnover, which they also monitor.
Note that outcome measures are not perfectly related to each other. That is, it is tempting to assume that satisfied trainees learn more and will apply their knowledge and skill to the job, resulting in behavior change and positive results for the company. However, research indicates that the relationships among reaction, cognitive, behavior, and results outcomes are small.
Which training outcomes measure is best? The answer depends on the training objectives. For example, if the instructional objectives identified business-related outcomes such as increased customer service or product quality, then results outcomes should be included in the evaluation. As Figure 3 shows, both reaction and cognitive outcomes may affect learning. Reaction outcomes provide information regarding the extent to which the trainer, facilities, or learning environment may have hindered learning. Learning or cognitive outcomes directly measure the extent to which trainees have mastered training content. However, reaction and cognitive outcomes do not help determine how much trainees actually use the training content in their jobs. As much as possible, evaluation should include behavior or skill-based, affective, or results outcomes to determine the extent to which transfer of training has occurred—that is, whether training has influenced a change in behavior, skill, or attitude or has directly influenced objective measures related to company effectiveness (e.g., sales).
How long after training should outcomes be collected? There is no accepted standard for when the different training outcomes should be collected. In most cases, reactions are usually measured immediately after training. Learning, behavior, and results should be measured after sufficient time has elapsed to determine whether training has had an influence on these outcomes. Positive transfer of training is demonstrated when learning occurs and positive changes in skill-based, affective, or results outcomes are also observed. No transfer of training is demonstrated if learning occurs but no changes are observed in skill-based, affective, or learning outcomes. Negative transfer is evident when learning occurs but skills, affective outcomes, or results are less than at pre-training levels. Results of evaluation studies that find no transfer or negative transfer suggest that the trainer and the manager need to investigate whether a good learning environment (e.g., opportunities for feedback and practice) was provided in the training program, trainees were motivated and able to learn, and the needs assessment correctly identified training needs.
💡 Why this matters: Selecting the right outcomes and understanding the timeframes for measurement are critical for accurately determining if training has actually transferred to the job.
EVALUATION DESIGNS
The design of the training evaluation determines the confidence that can be placed in the results, that is, how sure a company can be that training is either responsible for changes in evaluation outcomes or has failed to influence the outcomes. No evaluation design can ensure that the results of the evaluation are completely due to training. What the evaluator strives for is to use the most rigorous design possible (given the circumstances under which the evaluation occurs) to rule out alternative explanations for the results of the evaluation.
This discussion of evaluation designs begins by identifying these “alternative explanations” that the evaluator should attempt to control for. Next, various evaluation designs are compared. Finally, this section discusses practical circumstances that the trainer needs to consider in selecting an evaluation design.
Threats to Validity: Alternative Explanations for Evaluation Results
Table 1 presents threats to validity of an evaluation. Threats to validity refer to factors that will lead an evaluator to question either (1) the believability of the study results or (2) the extent to which the evaluation results are generalizable to other groups of trainees and situations. The believability of study results refers to internal validity. The internal threats to validity relate to characteristics of the company (history), the outcome measures (instrumentation, testing), and the persons in the evaluation study (maturation, regression toward the mean, mortality, initial group differences). These characteristics can cause the evaluator to reach the wrong conclusions about training effectiveness. An evaluation study needs internal validity to provide confidence that the results of the evaluation (particularly if they are positive) are due to the training program and not to another factor. For example, consider a group of managers who have attended a communication skills training program. At the same time that they attend the program, it is announced that the company will be restructured. After the program, the managers may become better communicators simply because they are scared that otherwise they will lose their jobs. Perhaps no learning actually occurred in the training program!
Trainers are also interested in the generalizability of the study results to other groups and situations (i.e., they are interested in the external validity of the study). As shown in Table 1, threats to external validity relate to how study participants react to being included in the study and the effects of multiple types of training. Because evaluation usually does not involve all employees who have completed a program (or who may take training in the future), trainers want to be able to say that the training program will be effective in the future with similar groups.
🔑 Definition — Threats to validity: Factors that will lead an evaluator to question either (1) the believability of the study results (internal validity) or (2) the extent to which the evaluation results are generalizable to other groups of trainees and situations (external validity).
⭐ Key Takeaways
To select appropriate training outcomes, they must be relevant, reliable, discriminative, and practical. Relevance concerns whether outcomes measure learned capabilities, with threats from criterion contamination (measuring wrong capabilities) and criterion deficiency (failing to measure what was taught). While reaction and cognitive outcomes are most commonly used in practice, companies should collect outcomes related to both learning (levels 1 and 2) and transfer of training (levels 3, 4, or 5) for an adequate evaluation, as the relationships among outcome types are small and measuring only learning does not show if knowledge transfers to the job. Finally, evaluation designs must control for threats to validity—internal threats that question the believability of results (e.g., history, maturation) and external threats that affect generalizability—to ensure confidence that observed changes are due to training.
🧠 Quick Revision Questions
- What are the four criteria for selecting appropriate training outcomes, and what does each one mean?
- Explain the difference between criterion contamination and criterion deficiency, using an example for each.
- Why is it important to collect outcome measures related to both learning and transfer, rather than just reactions and cognitive outcomes?
- What is the difference between internal validity and external validity in the context of training evaluation?
- Give two examples of threats to internal validity that could lead an evaluator to incorrectly conclude that training was effective.
📘 Lecture 25 — Training Evaluation (continued)
📖 Overview: This lecture continues the discussion on training evaluation, focusing on methods to control threats to validity, types of evaluation designs, and determining return on investment (ROI). Understanding these concepts is critical for trainers to design rigorous evaluations that provide credible evidence of training effectiveness and justify training expenditures.
🗂️ Topics Covered
The lecture covers three main areas: first, methods to control threats to validity including pretests/posttests, comparison groups, and random assignment; second, the five types of evaluation designs—posttest only, pretest/posttest, pretest/posttest with comparison group, time series, and Solomon four-group—along with considerations for choosing an evaluation design; and third, determining return on investment through cost-benefit analysis, including how to calculate costs, benefits, and ROI with practical examples.
📝 Lecture Summary
Methods to Control for Threats to Validity
Because trainers often use evaluation results to justify training programs or secure funding, minimizing threats to validity is essential. Three primary methods are used: pretests and posttests, which establish a baseline before training and measure outcomes after training to show change; comparison groups, which include employees similar to trainees who do not attend training, helping rule out factors like the Hawthorne effect (high performance due to receiving attention); and random assignment, which assigns employees to training or comparison groups by chance, ensuring similar characteristics across groups. While random assignment is often impractical because companies prefer to train those who need it, statistical procedures like analysis of covariance can control for differences between groups.
🔑 Definition — Comparison Group: A group of employees who participate in the evaluation study but do not attend the training program. They have personal characteristics (e.g., gender, education, age, tenure, skill level) as similar to the trainees as possible.
🔑 Definition — Hawthorne Effect: The phenomenon where employees in an evaluation study perform at a high level simply because of the attention they are receiving.
🔑 Definition — Random Assignment: Assigning employees to the training or comparison group on the basis of chance, without consideration of individual differences (ability, motivation) or prior experiences.
🔑 Definition — Analysis of Covariance: A statistical procedure used to control for factors in which the training and comparison groups differ when random assignment is not possible.
Types of Evaluation Designs
Several evaluation designs exist, varying in who is involved, when measures are collected, costs, time, and ability to rule out alternative explanations. In general, designs using pre-training and post-training measures with a comparison group provide the strongest evidence but are more costly and time-consuming.
Posttest Only: An evaluation design in which only post-training outcomes are collected. This can be strengthened by adding a comparison group. It is appropriate when trainees are expected to have similar levels of knowledge or behavior before training.
Pretest/Posttest: An evaluation design in which both pre-training and post-training outcome measures are collected, but there is no comparison group. This makes it difficult to rule out business conditions or other factors as explanations for changes.
Pretest/Posttest with Comparison Group: An evaluation design that includes both trainees and a comparison group, with pre-training and post-training outcome measures collected from both groups. If improvement is greater for the training group, this provides evidence that training caused the change. This design controls for most threats to validity. An example involved comparing three types of computer training—behavior modeling (watching a video of key behaviors), self-paced study, and lecture—with a comparison group. Measures included learning (11-item test), procedural comprehension (scenarios on screens), skill-based learning (six computer tasks), and satisfaction (six items). Measures were collected pre-training, immediately post-training (time 1), and four weeks post-training (time 2) to assess retention and transfer.
🔑 Definition — Behavior Modeling: A training method involving watching a video showing a model performing key behaviors necessary to complete a task.
Time Series: An evaluation design in which training outcomes are collected at periodic intervals both before and after training. Strength can be improved by using reversal (a period when participants no longer receive training) and a comparison group. This design is frequently used to evaluate programs focused on readily observable outcomes like accident rates, productivity, or absenteeism. An example evaluated a safety training program in a food manufacturing plant using observations of safe work behaviors over 25 weeks. The Wrapping Department (trained group) and Makeup Department (comparison group) were observed. Safe behaviors increased after training and decreased when training was withdrawn, providing strong evidence of training effectiveness.
🔑 Definition — Reversal: A time period in which participants no longer receive the training intervention, used to improve the strength of time series designs.
Solomon Four-Group: Combines the pretest/posttest comparison group and the posttest-only control group design. One training group and one comparison group are measured before and after training; another training group and control group are measured only after training. This controls for most threats to internal and external validity. An example compared integrative learning (IL) training (featuring group discussions, games, stories, poetry) with traditional lecture-based training for manufacturing resource planning. IL group participants learned slightly less but had much more positive reactions.
🔑 Definition — Integrative Learning (IL): A training approach that begins with activities to create a relaxed, positive environment, including group discussions, games, stories, and poetry related to the training topic.
Considerations in Choosing an Evaluation Design: No single evaluation design is always appropriate. A less rigorous design may be chosen when managers are unwilling to commit time, lack expertise, or view training as a low-return investment. A more rigorous design (e.g., pretest/posttest with comparison group) should be considered when: results can change the program; the program is ongoing with large impact on employees/customers; involves multiple classes and many trainees; cost justification is based on numerical indicators; expertise or budget exists; training is costly; sufficient time is available; or there is interest in measuring change or comparing programs. Designs without pretests or comparison groups are appropriate when the goal is to determine if a specific performance level has been achieved, rather than measuring change.
💡 Why this matters: Choosing the right evaluation design directly affects the credibility of the results. A poorly designed evaluation may lead to incorrect conclusions about training effectiveness, wasting resources or missing opportunities for improvement.
DETERMINING RETURN ON INVESTMENT
Return on investment (ROI) is an important training outcome determined through cost-benefit analysis—the process of determining economic benefits using accounting methods. Training cost information helps understand total expenditures, compare alternative programs, evaluate spending distribution, and control costs.
🔑 Definition — Return on Investment (ROI): A measure of the economic value provided by training, calculated by dividing benefits by costs.
🔑 Definition — Cost-Benefit Analysis: The process of determining the economic benefits of a training program using accounting methods that look at training costs and benefits.
Determining Costs: The resource requirements model compares equipment, facilities, personnel, and materials costs across stages of the training process (needs assessment, development, design, implementation, evaluation). Seven categories of cost sources include: program development/purchase, instructional materials, equipment/hardware, facilities, travel/lodging, trainer and support staff salary, and cost of lost productivity (or temporary employees). Costs are classified as one-time costs (needs assessment, program development), costs per offering (site rental, trainer salaries), and costs per trainee (meals, materials, lost productivity).
🔑 Definition — Resource Requirements Model: A method for comparing costs of alternative training programs by comparing equipment, facilities, personnel, and materials costs across different stages of the training process.
Determining Benefits: To identify benefits, review the original training reasons (e.g., reducing production costs, increasing repeat business). Methods include: reviewing technical/academic/practitioner literature, conducting pilot training programs, observing successful job performers, and gathering estimates from trainees and their managers.
Example of a Cost-Benefit Analysis: A wood plant with 300 workers, 48 supervisors, 7 shift superintendents, and a plant manager faced three problems: 2% panel rejection rate due to poor quality, poor housekeeping, and high preventable accidents. Supervisors, shift superintendents, and the plant manager attended training in performance management, interpersonal skills, and rewarding employees. Training was conducted at a hotel using a purchased videotape with a consultant instructor. Benefits were calculated based on improved quality, housekeeping, and reduced accident rate. ROI was calculated as 6.7, meaning each dollar invested returned approximately seven dollars in benefits.
📐 Formula: ROI = Return (Benefits) ÷ Costs → The ratio of economic benefits to costs, indicating the value generated per dollar invested. 📌 Example: In the wood plant, ROI = 6.7. This means for every dollar spent on training, the company received $6.70 in benefits through improved quality, housekeeping, and reduced accidents.
Other Methods for Cost-Benefit Analysis: Utility analysis is a more sophisticated method that assesses the dollar value of training based on estimates of the difference in job performance between trained and untrained employees, number trained, length of training influence, and variability in job performance. It requires a pretest/posttest design with a comparison group. Other economic analyses evaluate training benefits to the firm or government using direct and indirect costs, government incentives, wage increases, tax rates, and discount rates.
🔑 Definition — Utility Analysis: A cost-benefit analysis method that involves assessing the dollar value of training based on estimates of the difference in job performance between trained and untrained employees, the number of individuals trained, the length of time a training program is expected to influence performance, and the variability in job performance in the untrained group of employees.
💡 Why this matters: ROI analysis should be limited to programs that are focused on operational issues, linked to company strategy, expensive, highly visible, have management interest, are attended by many employees, and are permanent. This ensures resources are used efficiently on analysis that provides the most value.
⭐ Key Takeaways
Students must remember that three methods—pretests/posttests, comparison groups, and random assignment—control threats to validity in training evaluation. The five evaluation designs (posttest only, pretest/posttest, pretest/posttest with comparison group, time series, and Solomon four-group) vary in rigor, cost, and ability to rule out alternative explanations, with the pretest/posttest comparison group design being the most practical strong design. Choosing an evaluation design depends on factors like whether results will change the program, the program's scope and cost, available expertise, and the need to measure change versus achieving a performance level. ROI is calculated by dividing benefits by costs, with the resource requirements model and seven cost categories providing a systematic way to determine costs, and benefits identified through literature, pilots, observation, or estimates. Utility analysis offers a more sophisticated cost-benefit analysis method but requires a rigorous evaluation design.
🧠 Quick Revision Questions
- What are the three methods to control for threats to validity in training evaluation, and how does each method help?
- Describe the Solomon four-group design and explain why it is considered the strongest evaluation design for controlling threats to validity.
- In the safety training time series example, what was the purpose of withdrawing training (reversal), and what did the results show?
- List four conditions under which a more rigorous evaluation design (e.g., pretest/posttest with comparison group) should be considered.
- Calculate ROI if a training program costs $50,000 to implement and generates $350,000 in benefits. What does this ROI value mean?