The classic evaluation model developed by Donald Kirkpatrick looks at four levels: student reaction, knowledge transfer, behavioral change, and business results (Kirkpatrick, 1975). Critics of Kirkpatrick model say that it doesn't take the business impact far enough and that the final step in any training program should be a "fifth level" of evaluation -- financial return. This ultimate evaluation determines the financial return on investment (ROI) of the training program.

New Demands for Accountability

For a long time, organizations paid lip service to the human resource motto that "people are an organization's most important asset." Now senior executives have come to believe that employees and the intellectual capital they create can uniquely differentiate their company in the marketplace. Training expenditures now are viewed as critical investments in human capital, and an effective method of increasing employee retention.

Because of these factors, the investment in corporate training programs is large and growing rapidly. Total corporate spending on training in the United States was approximately $60 billion in 1998 (Training, October 1998). The fastest-growing segment of the training budget is expenditures on technology-based training. By the year 2000, it is estimated that 50 percent of all training interventions will be delivered via CD-ROM or corporate Intranets (Ibid.). Specifically, within the Web-based training marketplace, expenditures are expected to go from a mere $197 million in 1997 to over $6 billion in 2002 (International Data Corporation, 1998). This projected 100 percent annual growth rate in technology-based training is being driven by increases in the availability of network bandwidth, the decreasing price of multimedia computers, and the development of more services and products from vendors.

Commensurate with this increase in training expenditures, senior executives are demanding more accountability from their training departments. In fact, 93 percent of training professionals surveyed at a 1996 conference said they are increasingly being asked to show the return-on-investment of their programs (National HRD Executive Survey, 1997). Training managers need to be able to answer direct questions about total costs, benefits, and bottom-line impact. Visionary training managers embrace cost-benefit analysis as a way to justify bigger budgets for technology and new training programs.

The Value of Cost-Benefit Analysis

Proponents of technology-based training have long touted its many benefits: reduction in learning time, increase in knowledge retention rates, cost savings. Brandon Hall, quoted in the August 1998 issue of HRMagazine, makes the generalization: "There's about a 50-percent reduction in time and cost over classroom training." (Roberts, 1998) The power of the cost-benefit analysis process is that it enables you to move from generalizations and assumptions to proof of the value of each and every program you develop.

This type of quantifiable measurement of value is critical in the overall management of a training function, and a powerful tool that can be used to keep or expand available training resources. Brandon Hall, editor of Multimedia & Internet-Based Training Newsletter, has conducted exhaustive research in the area of return on investment. His research and that of others have uncovered some compelling cases:

• A computer storage media company converted a four-day instructor-led course for 1,500 technicians into a multimedia CD-ROM format. Due to a reduction in learning time and elimination of travel expenses, Storage reduced costs over three years by 47 percent and saved $1.5 million (Hall, 1997).
• A major consultancy firm developed and delivered computer-based training for 7,000 consultants in 50 countries. The cost of the training was $106 per student, versus an estimated $760 per student for instructor-led delivery. Over the five-year life span of the program, technology-based training saved the firm more than $4.5 million (Ibid.).
• A computer reseller developed a Web-based training solution for its internal sales force and value-added resellers. Some 40 online courses were developed, complete with self-assessment quizzes. According to their director of strategies technologies, "We saw a 50 percent increase in sales across distribution and integration resellers," (Fickel, 1998).
• A branch of the U.S. military estimated that their technicians' ability to troubleshoot problems increased by 90 percent after the adoption of multimedia training. Over a period of five years, they expect at least a 20-fold return on their investment (Jerram, 1994).

Key Concepts of Cost-Benefit Analysis

At the simplest level, cost-benefit analysis answers the question "Was it worth the money?" In other words, what were the total costs to develop the program, and what were the total benefits realized? Costs include direct costs, such as payments to vendors, as well as indirect costs, such as the value of time. Financial benefits can be in the form of cost savings, or increases in productivity or revenue. The following key concepts are factors in a cost-benefit analysis.

• Life of training. Every project needs to be measured across some time period. Technology-based training programs don't last forever. Their shelf life will be determined by things such as changes to content, changes in technology, and changes in business need. According to Hall's research conducted over the last ten years, most ROI studies show technology-based training is more expensive to develop and deliver over the short-term, but pays off over time. Typically, three-to-five years of use is an accepted time period to apply for evaluating a training program.
• Alternate delivery options. Perhaps the most common method of showing the financial impact of technology-based training is to compare it against the costs for other forms of delivery. To come up with a comparison means, ask the question, "If we don't deliver the training via the Web, what would it cost for us to deliver it in a classroom setting?"
• Size of audience. With technology-based training, the cost of development is not dramatically effected by the number of students using it. The cost is basically the same to develop a two-hour CD-ROM or Web-based training program for 10 people as it is for 1000 people. The only additional costs may be in the form of CD-ROM duplication, student tracking, and end-user support. However, the size of the target audience is extremely relevant when comparing the costs against instructor-led delivery. With live workshops, the number of students has a direct impact on expenses related to instructors, locations, and travel.
• Seat time. The total amount of time students will spend with the course is called seat time -- how long they will be in their seats. Seat time is always specified for instructor-led training, but is an estimate when given for self-paced, technology-based training. After all, a course that takes one student two hours to complete, might take another only 90 minutes. Increasingly, effective Web-based training is blurring the lines between instruction and just-in-time performance support. This factor makes estimates of seat time additionally tenuous.
• Burdened costs. This accounting term refers to the total cost of an item, which may include some hidden costs. For example, you might quickly estimate that a classroom facilitator who earns a $60,000 salary costs $230 per day, simply by dividing the salary by the total number of weekdays ($60,000 ¸ 52 weeks ¸ 5 days). But the burdened cost for the instructor will be higher once you take into account payroll taxes, insurance, and other benefits. Additionally, when calculating day rates, make sure to subtract company holidays, vacation time, and sick days to get an accurate estimate of the burdened cost for each productive workday.
• Estimated revenue impact. Often the impact a training program has on sales and expenses is indirect, or difficult to measure. In these cases, the impact on revenue is projected or extrapolated from known data. For example, assume that a quality control training program was shown to reduce the number of defective cell phones produced each year in a factory from 5000 to 3000 (net reduction of 2000 defective phones a year). Although the training program directly reduced errors, which led to a drop in the number of defective phones, you would have to estimate the revenue impact. To do this, you would need to research costs associated with wasted materials in each defective phone, labor time for the manufacturing, identification of, and disposal of each defective phone. With this methodology, a defect-reduction number can be translated into a revenue-savings number.
• Opportunity costs. These costs are the lost revenues or increased costs associated with opportunities that will be missed because of the training program. This measure is increasingly being used in the competitive world of sales. Traditionally, for a sales rep spending time in training, a main measure of cost is the salary of the rep while in the training program. However, a more advanced analysis measures the opportunity cost of the rep not being out in the field. According to Jim DeMaioribus, associate director of sales training at Knoll Pharmaceuticals, every day a rep spends in the field is worth approximately $8,000 in revenue. Therefore a major advantage of technology-based sales training is its ability to maximize time in the field and minimize the opportunity costs of sales training.