Evaluating the Effectiveness of Training Programs

by Patricia Boverie, Deanna Sanchez Mulcahy, and John A. Zondlo

Change is everywhere. It is inescapable. Hardly a day goes by without news of the new world economy or the shift from a production to a service orientation. Indeed, change is necessary to survive in an uncertain world. Also, today's business environment is highly competitive. Because of the sweeping effects of change and competition, a great deal of interest has been placed on higher education and lifelong learning. Consequently, business is turning to training in order to cut costs and increase productivity among employees. In fact, according to Fulmer (1988) and Eurick (1985), in 1985 the United States corporate training and education efforts were estimated to cost 40 to 60 billion dollars annually, which is close to the amount spent on post-secondary education.

However, in the rush to train and educate people, many organizations have failed to treat the evaluation of such training as a priority. At best, the evaluation of training has been a perfunctory task with little analysis and usefulness. Yet evaluating the effectiveness of costly training efforts is paramount to the success of any program. This article reviews the current HRD literature addressing the evaluation of adult and workplace training programs, based on Donald Kirkpatrick's (1979) evaluation steps.

FACTORS AFFECTING ADULT EDUCATION

Before discussing the evaluation of training, it is important to explain the elements that serve as an impetus for adult education and workplace training. Merriam and Caffarella (1991) identify three major areas of change that influence adult learning:

Demographic Changes

One of the changing elements of demographics is age. According to Merriam and Caffarella, there are more Americans aged sixty-five and older than there are Americans aged twenty-five and younger. The aging of the population will continue well into the next century, and the demand for quality adult education will rise accordingly. The rapid growth of cultural and ethnic diversity in the U.S. is another changing element of demographics. The United States is now experiencing a wave of immigration, primarily from Asia and Latin America, that parallels the influx of Europeans at the beginning of this century. By the year 2000, minorities are expected to compose 29 percent of the U.S. population (Merriam & Caffarella, 1991). In order to tap this valuable resource, employers will need to provide specialized training to help these people adjust to the American workplace.

Economic Changes

Economic changes are also having and will continue to have an impact on adult learning and training. Many experts, such as Naisbitt and Aburdene (1990), contend that the economies of the world are now interdependent. Consequently, major companies are allowing, encouraging, and sometimes subsidizing their employees' education in order to become more competitive and to increase their chances for survival in a world economy.

Another critical economic change is the shift from a manufacturing economy to a service economy, which has produced a change in the job market and has affected the kind of training that employees need. Moreover, changes in the composition of the U.S. work force itself are influencing training. For example, since World War II women have become an integral part of American organizations.

Technological Changes

Finally, the advancements in technology will continue to shape and define adult training needs, primarily because of the advent of the personal computer. Computers have revolutionized every aspect of corporate education, allowing people to produce, analyze, and manipulate data with greater ease than before. By some accounts (for example, Apps, 1988), every seven years the amount of information generated in the world doubles. Furthermore, about half of the information that most professionals learn will be outdated in about five years.

The implication of technological advancements is that learning is a lifelong proposition. Not only will there be a demand for training to keep up with technological advancements, but there also will be a demand to retrain the millions of Americans who will be displaced because of such advancements.

WHY EVALUATE?

Although for many years trainers have attempted to evaluate their programs, until quite recently, there has not been a bona fide effort to use valid and reliable methods to conduct such evaluations. Furthermore, some trainers gather data for evaluation but do not analyze those data for trends or use them to improve existing training programs. Such an oversight can be costly, especially in light of the billions of dollars that have been spent and will continue to be spent annually on training efforts as a result of the demographic, economic, and technological changes just discussed.

It is important to remember that effective evaluation is multifaceted. All of the literature recognizes the importance of evaluation in terms of client orientation and economic return. In other words, most researchers in the field understand that clients, whether they are those who have hired the trainer or those who have participated in the training, must be satisfied with that training. If clients do not perceive a return on their investment, whether measured in terms of time or dollars, they may not be willing to continue to invest in training.

LEVELS OF EVALUATION

There are several components to an effective evaluation program. One of the most comprehensive and widely referenced models of evaluation is Donald Kirkpatrick's (1979). The four levels of this model are as follows:

The balance of this article reviews the current research on evaluation in light of Kirkpatrick's model.

Level 1: Reaction Evaluation

Reaction is the term that Kirkpatrick uses to refer to how well the participants liked a particular training program. Evaluation of participants' reactions consists of measuring their feelings; it does not include a measure of actual learning. Kirkpatrick contends that although the evaluation of reactions is an easy measurement, many trainers do not follow these five essential steps for accurate measurement:

Although Kirkpatrick suggests that participants should feel free and be encouraged to make additional comments, he also contends that this type of qualitative data is extremely difficult to analyze. Thus, it is difficult to discern any patterns or trends in order to revise the training program.

Other researchers have different perspectives regarding the evaluation of participants' reactions. For instance, Antheil and Casper (1986) state that participant reaction is a measure of "customer satisfaction" indicating the level of effectiveness and usefulness of the training program at the time the participants are experiencing it and sometimes weeks or even months afterward. However, they are careful to stress that data collected regarding participant reactions reflect participant opinions and should not be considered proof of learning.

To determine what training-evaluation tools were being used by industry, Fisher and Weinberg (1988) of Bell Communications Research, Incorporated (Bellcore) conducted a phone survey in March of 1986. The data indicated that the typical instrument to gather information regarding reactions was a "short, quickly constructed, open-ended questionnaire" (p. 73). This "happy sheet" (p.73), as Fisher and Weinberg refer to it, provided subjective impressions and no data that could withstand statistical analysis or measures for reliability. Because there was no adequate tool for evaluation, the Bellcore System developed a new instrument with items addressing the trainer's behavior, the participant's experience, and other issues phrased as open-ended questions.

This questionnaire, like most such instruments, focuses on participant reactions-not learning or the transfer of learning. For instance, one item on the questionnaire reads, "The-e course presented useful information" (p. 76). The participants are then asked to rate the statement on a Likert scale. Fisher and Weinberg (1988) warn that while this questionnaire does provide a "general estimate of a particular course's success based upon the views of the participants" (p. 75), the data may be somewhat inaccurate because participants have a tendency to report what a trainer wants to hear. Also, some questionnaires have poorly constructed questions or items that predispose participants to respond in predicted ways.

Some trainers and researchers feel that measurements of participant reactions are inaccurate and counterproductive. For instance, Conway and Ross (1984) found that participants have a tendency to underestimate their pretraining skills and overestimate their posttraining skills in an attempt to justify participating in the training. Their research is consistent with research in the field of social psychology indicating that people have a strong need to justify their behavior and actions and consequently may alter their opinions and their interpretation of past events. Therefor-e, if trainers continue to use participant reactions as the sole means of evaluation-and management continues to allow such use-the outcome can be misleading and extremely costly.

Carnevale and Schulz (1990) go a step further. They claim that "participant reactions are easy to collect but provide little substantive information about training's worth" (p. s-15). They also claim that because data concerning participant reactions do not reveal the actual learning that has taken place, those data do not accurately indicate the return on investment for training efforts. They state that because of such unreliable data, many trainers have stopped using reaction sheets. However, Carnevale and Schulz go on to say that most trainers believe participants' favorable reactions are crucial to a program's success and that participants whose reactions are favorable tend to be more receptive to the material and consequently more likely to use it on the job.

Dixon (1987, p.108) claims that "the use of participant reaction forms can cause more problems than benefits for the training function of an organization". This statement is especially true when participant reactions are the only evaluation method used. Dixon contends that three major problems result from the use of reaction forms:

1. The expectation that training must be entertaining. Because reaction sheets measure how the participants felt about the training, the trainer may tend to emphasize participant enjoyment during the training rather than substantive information. As a trainer is often rewarded with high marks when the participants enjoy themselves, this relationship between evaluation and participant enjoyment can become a vicious cycle. The trainer's ratings are also a major factor in the rewards that the trainer receives from management or the client organization: renewal of a contract or a promotion. Obviously, under these circumstances the use of a reaction sheet can lead to a conflict of interest.

2. Faulty instructional design. The term "faulty instructional design" refers to a questionnaire design that asks for information that participants cannot legitimately provide. As Dixon (1987) states, the art of questionnaire design is to ask questions for which a participant can give informed responses.

3. The perception that learning is passive rather than active. This perception refers to the common belief that it is the trainer's responsibility to ensure that participant learning occurs. Measuring how well this responsibility has been met with a reaction sheet is problematic, as a reaction sheet asks questions about the trainer's performance and the course design without asking about the participants' efforts to learn. Dixon emphasizes that evaluation and learning are not complete unless both functions have been measured. Ultimately, it is the responsibility of the trainer to provide information and the responsibility of the participant and the trainer to process the information. Reaction sheets rarely take into account the participant's role as part of the training program.

Level 2: Learning Evaluation

According to Kirkpatrick (1979), the second level of analysis in the evaluation process is that of learning. Kirkpatrick defines learning as the "principles, facts and techniques that were understood and absorbed by the participants" (p. 82) and identifies the following guidelines or standards for evaluation in terms of learning:

Obviously, evaluation of learning is much more difficult to measure than reaction. According to Kirkpatrick's guidelines, a knowledge of statistical procedures is essential for accurate and meaningful measurement.

Endres and Kleiner (1990) state that pretests and posttests are necessary when evaluating the amount of learning that has taken place. Without a point of comparison, the measurement of learning at the end of the training program will not reveal exactly how much knowledge has been obtained from the training experience. Although paper-and-pencil tests are the most frequently used tools to measure knowledge, there are other means for gathering this kind of data.

For instance, when simulations, role plays, or demonstrations are used to measure knowledge, the trainer can use before-and-after situations in which participants can demonstrate or perform the knowledge and techniques that they have learned. This information is consistent with Kirkpatrick's research on the measurement of learning. In fact, like Endres and Kleiner, Kirkpatrick maintains that simulations and demonstrations can closely approximate the participants' work environment and can help them relate the learning in meaningful ways, especially when specific job skills are the focus of the training.

According to Carnevale and Schulz (1990), the measurement tools used to evaluate learning should reflect each training program's particular objectives. Also, measures of learning changes may be taken during or at the end of a training session. Carnevale and Schulz warn that such a measure of learning changes "may indicate that a program's instructional methods are effective, but it doesn't show whether or how participants' new learning will be applied on the job" (p. s-16).

A useful process for reviewing items on a measurement tool that evaluates learning has been suggested by Cantor (1990):

These steps are consistent with the instructional systems design method and will help ensure that items will be reliable and valid means for determining whether learning has occurred.

Research by Antheil and Casper (1986, p. 58) indicates that "evaluation of learning at this level closely resembles testing" and most often takes the form of paper-and-pencil tests. They suggest that the typical measurement tool includes gathering pretest and posttest data to determine-nine the amount of learning that has been acquired. They also stress that skill demonstrations in a learning situation merely indicate whether a participant can use the skills-not whether he or she will use them.

Level 3: Transfer-of-Learning Evaluation

Kirkpatrick's third level in the evaluation model is transfer of learning. In the HRD literature there are relatively few examples of studies that have specifically attempted to assess the transfer of training skills or knowledge to the job. Even Kirkpatrick (1979, p. 86) warns that "evaluation of training programs in terms of on the job behavior is more difficult than the reaction and learning evaluations...... As a result, much training is delivered without a plan for measuring the transfer of training. Kirkpatrick goes on to suggest a framework for evaluating training programs in terms of behavioral changes:

Antheil and Casper (1986) propose a comprehensive evaluation model based on Kirkpatrick's four levels, which they call "program effects levels." Their three-step procedure for implementing the model is as follows:

Antheil and Casper (1986) emphasize the importance of collecting and presenting the information in a way that will be meaningful and relevant for the specific audience involved. This level of evaluation not only assesses the performance of the person who receives the training, but also provides valuable feedback to those involved in redesigning existing training programs or in designing programs to meet future needs. This information is also useful to those who will be evaluating the effectiveness of the overall training program. The collection of qualitative as well as quantitative data is encouraged by Antheil and Casper. They suggest logs, diaries, and observer narratives, for example.

Endres and Kleiner (1990) use Kirkpatrick's model in suggesting an approach to evaluating the effectiveness of management training. They caution against relying on in-house performance-appraisal systems as the primary measure of transfer of learning, as it is difficult to separate the effects of training efforts from those of other factors. Instead, they suggest setting initial performance objectives and monitoring accomplishment of those objectives after training. They offer an example in which participants write personal and professional objectives at the end of the training experience. These objectives are then sent to the participants approximately a week after the training. Two months later they are sent again, and the participants are asked to comment on their performance against these objectives. A certificate of completion for the training is issued only after each participant's feedback is secured.

Like Kirkpatrick, Endres and Kleiner suggest multidimensional on-the-job evaluations, including feedback from the participant, his or her subordinates, and peers. "By using all three forms of feedback," they say, "the built-in biases of the evaluator can be reduced as the number of evaluators having different perspectives is increased" (p. 6).

Finally, they remind evaluators that other factors can impact the effectiveness of management training and development, including the manager, the trainer, the organization, and the environment. As they state, "All four are complex creatures" (p. 7).

Nanda (1988) also looked at the transference of supervisory skills following training programs and found that most supervisory-training programs are knowledge based. However, to be of value to the trainee and the organization, that knowledge must result in a change of attitude, followed by a change in the supervisor's behavior. Unfortunately, the impact of most supervisory-training programs does not go beyond knowledge and awareness. One factor that often inhibits transference of learning is the organizational climate, which may be inconsistent with what is taught in the training program. This inconsistency often renders such training programs entirely ineffective. As Nanda (p. 28) says, "perhaps changes in attitude among top managers are key to the skill development of supervisors."

The instrumental impact of the on-the-job environment is consistent with Bandura's findings in the studies that led to the development of Social Learning Theory. Bandura (1965) found that any learning that may have been gained by observing the behavior of models was completely wiped out by the subsequent incentives received for the performance of a specific response, leading him to conclude that "mere exposure to modeling stimuli does not provide sufficient conditions for imitative or observational learning" (p. 593).

Kelly (1982) starts with the assumption that typically only 10 percent of a company's training transfers skills to the job. What happens to the other 90 percent of training? She suggests that 40 percent is lost because the training function is often isolated or peripheral: "Therefore, management, who views anyone paid to do a peripheral job as a peripheral person, will not bring that person's ideas into the workplace" (p. 102). An additional 40 percent, she suggests, is lost because most trainers or management educators do not build transfer into the training programs. Finally, 10 percent may be lost when the course designer does not deliver the training.

For skills to be transferred to the job, Kelly believes that they must be built into the training "before the first specific behavioral objective is chosen, before the first course activity is imagined or before a packaged product is selected" (p. 104). In other words, the course should be designed with the specific intent of transfer to the actual job situation.

Kelly's comment stresses an important point. In order to study whether skill transfer related to training has in fact occurred, one must establish a baseline of current skills or knowledge before the training occurs. For example, six months after a two-day workshop on supervisory skills, Swierczek and Carmichael (1985) conducted a survey in which they attempted to measure whether participants in the workshop actually used the learned skills. They found that they were hampered by the lack of baseline information: "Therefore the results cannot be linked definitively to the workshop" (p. 97). Of course, following a good process for instructional system design would suggest pretesting, both to establish such a baseline as well as to determine the need for training in the first place.

Mahoney (1980) suggests that management training be evaluated using three criteria:

To optimize transference of management training, Mahoney suggests that a manager who wishes to train subordinates in supervisory skills should conduct a series of working meetings on specific issues. The issues selected should be ones that have been identified by the subordinates (thus meeting the "targets" criterion). Next, Mahoney suggests that the training be divided into a series of half-day segments. One criticism of training is that it takes too much time both for the participant and for the manager/trainer. However, if the working meetings are limited to four to eight times per year, they represent only a 1- to 2- percent investment of each subordinate's time (thus meeting the "time" criterion). Finally, Mahoney suggests that training be designed with an "action research" process in mind. With proper selection of training topics and content, the manager's subordinates will actually take their jobs with them to each training session. Thus, "the job and the training are separated only by the training setting, not by process and not by content" (p. 29) (thus meeting the "transfer" criterion).

A synthesis of the literature reviewed here suggests the following ten guidelines for designing training that ensures transfer:

It is interesting to note that if transfer of learning is considered at all, this consideration usually occurs after the training has been designed or even delivered. However, most of the guidelines suggested above should be followed during the design phase.

Level 4: Results Evaluation

Kirkpatrick's fourth level of evaluation is results or impact on the organization. Attempting to measure results is not for the fainthearted! Although measuring training programs in terms of results may be the best way to measure effectiveness, Kirkpatrick himself (1979, p. 89) points out that "there are ... so many complicating factors that it is extremely difficult if not impossible to evaluate certain kinds of programs in terms of results." The separation of variables to measure how much of the improvement is due to training is extremely difficult. Instead of offering a specific formula, Kirkpatrick simply reports anecdotal efforts to measure results. He does applaud attempts by researchers such as Likert to use qualitative data in measuring results, but he laments the fact that current research techniques are essentially inadequate and that progress in this area is slow.

Zenger and Hargis (1982) recommend experimental-research designs using pretesting and posttesting of experimentally trained groups with untrained control groups. However, outside an ideal laboratory environment, this approach is not without its challenges.

Ban and Faerman (1990) report on their attempt to measure both skill transference and results following an intensive, twenty-four-day advanced supervisory-training program. They had hoped to study impact with an experimental design by surveying a control group of managers who had not participated in the training program. However, they had to abandon this part of their study because of logistical problems. They conclude that "the literature on training evaluation may be too optimistic in recommending experimental or quasi-experimental design for many field situations" (p. 278).

Similarly, Trapnell (1984, p. 92) remarks that 'impact evaluation is not a science" because of the number of variables other than training that may affect long-term results. Despite this comment, though, Trapnell encourages the use of available secondary data, such as savings resulting from reductions in downtime, accident rates, absenteeism, customer returns, assembly-line rejects, staff turnover, and employee grievances.

In an update to Zenger and Hargis' 1982 article, Kelly, Orgel, and Baer (1984) recommend quasi-experimental designs based on samples and groups that exist naturally in the work environment. An example would be two similar departments, one that receives training and one that does not. Rather than evaluating performance differences statistically and presenting those statistics-which, according to them, few people really understand-they suggest demonstrating results visually through graphic presentations.

The literature offers an account of at least one attempt to apply an econometric model to the evaluation of costs and benefits of training. Schmidt, Hunter, and Pearlman (1982) adapted "linear-regression-based decision-theoretic equations" to estimate the dollar impact of "intervention programs designed to improve job performance" (p. 333). The models they used were originally developed to estimate the dollar impact of valid selection procedures on work-force productivity. Typical studies on the value of selection procedures are highly statistical. However, "in general, organizational decision makers are less able to evaluate these statistics than statements made in terms of dollars" (p. 334). The model developed depends on a number of key assumptions, several of which must be inferred or estimated because they are not typically available from prior research.

Using their model, Schmidt, Hunter, and Pearlman estimated the value of a training program for one hundred programmers at more than one million dollars. In general, they hypothesized that "the economic impact or intervention programs may be greater than industrial/organizational psychologists have realized" (p. 340).

Using Schmidt, Hunter, and Pearlman's procedure, Sheppeck and Cohen (1985) propose a somewhat less statistical "utility" formula:

UTILITY= YD x NT x PD x V-NT x C, where:

This formula still depends on estimates for several variables. The most obscure is the concept of "value," a statistic that is not readily available for most jobs. Sheppeck and Cohen provide several suggested estimates of "value" based on the few actual studies reported in the literature, but they suggest that Schmidt, Hunter, and Pearlman's range of 40 to 70 percent of annual salary is a reasonable estimate when actual figures are unknown. They suggest further studies in a variety of occupational settings to develop more precise, job-specific estimates for each of these variables.

Given the difficulty of results evaluations and the relative lack of objective, valid tools to use, are they worth pursuing? McEvoy and Buller (1990) suggest not only that it would be wise to think twice about pursuing such evaluations but also that not all training is results oriented. They describe their attempts to conduct a comprehensive, four-step evaluation of their training for developing executive leadership, which is similar to Outward Bound. They found that training is often used for purposes other than achieving a measurable impact on the performance of an individual employee or the organization. For example, sometimes training is seen as a perquisite for performance that has already been judged successful or as a cultural "rite of passage" that all those hoping to advance must complete. In these cases the value of the training is more symbolic than technical.

McEvoy and Buller even went so far as to use a utility formula similar to that described by Sheppeck and Cohen to assess the dollar impact of their program for one of their clients. Using the most conservative assumptions for the model, they still estimated a net benefit of over a half-million dollars! They decided not to share the estimate with the client-because they did not think the client would believe it. The formula is not at all intuitive, and they reasoned that sharing the figure would hurt their credibility rather than help it.

These studies suggest that evaluation training on the basis of results or organizational impact may not be the ultimate measure. In the years since Kirkpatrick proposed his model, little has been added in the way of specific, valid tools to objectively measure training impact. Most promising are the quasi-experimental methods suggested by Kelly, Orgel, and Baer (1984) using graphic representations of hard data. Unfortunately, we may see few examples of this approach in the literature, as it lacks the scientific rigor that most journals favor.

It would also be a good idea to conduct further studies in a greater variety of occupational settings to determine reasonable, more precise estimates of performance differences between trained and untrained employees as well as value (that is, the standard deviation of job performance in dollars between trained and untrained employees). This research, however, may have some ethical hurdles to cross if it involves consciously withholding training from some people.

SUMMARY

Kirkpatrick's (1979) four levels of training evaluation-reaction, learning, transfer of learning, and results-are still a useful framework for considering evaluation techniques, as evidenced by the frequency with which they are referenced in the current training-evaluation literature. This article has offered a review of recent contributions in the literature as they pertain to Kirkpatrick's four levels. The experience reflected in the literature suggests that trainers incorporate at least the first three levels routinely in the design of training programs. In fact, many authors emphasize the importance of considering early in the design process how each level of evaluation will be addressed. The ability to track and report regularly on the effectiveness of training programs beyond participant reaction (that is, to documentation of learning, behavioral changes, and transfer of learning) can be critical to the success of a training program. It can also cement organizational recognition of the value of training and can help to ensure continued support.

Kirkpatrick's fourth level of evaluation, results, is still difficult to measure. The difficulty here, as Kirkpatrick himself points out, is the ability to separate training from the multitude of other variables that can impact long-term performance. The econometric and utility models reviewed here may be statistically elegant but are not sufficiently intuitive to warrant widespread application. This fact suggests opportunities for further research into alternative approaches and methodologies for addressing results. The more qualitative, quasi-experimental approaches involving action research, critical incidents, and similar methods appear promising. These approaches offer the advantage of observing and documenting the impact of training activities at the site. The authors hope that the journals that target adult educators and organizational trainers will recognize the value of these approaches and will share studies based on such methods.

As can be seen from the existing literature, more research in the field of training evaluation is necessary. In fact, evaluation is paramount to the success of any training program. Training not only must be cost effective but also must teach participants skills and concepts that they can readily use in their organizations after the training has been completed.

REFERENCES

Antheil, J.H., & Casper, I.G. (1986). Comprehensive evaluation model: A too] for the evaluation of non traditional educational programs. Innovative Higher Education, 11 (1), 55-64.

Apps, J.W. (I 988). Higher education in a learning society. San Francisco: Jossey-Bass.

Ban, C., & Faerman, S.R. (1990). Issues in the evaluation of management training. Public Productivity and Management Review, 8(3), 271-286.

Bandura, A. (1965). Influence of models' reinforcement contingencies on the acquisition of imitative responses. Journal of Personality and Social Psychology, 1(6), 589-,595.

Cantor, J.A. (1990). How to perform a comprehensive course evaluation. Performance & Instruction, 29(4), 8-15.

Carnevale, A.P., & Schulz, E.R. (1990, July Supplement). Return on investment: Accounting for training. Training & Development Journal, pp. 51-531.

Conway, M., & Ross, M. (1984). Getting what you want by revising what you had. Journal of Personality and Social Psychology, 47(4), 738-748.

Dixon,N.M. (1987).Meet training's goals without reaction forms. Personnel Journal, 66(8),108-115.

Endres, G.J., & Kleiner, B.H. (1990). How to measure management training and development effectiveness. Journal of European Industrial Training, 14(9), 3-7.

Eurick, N. (1985). Corporate classrooms. Princeton, NJ: Carnegie Foundation.

Fisher, H.E., & Weinberg, R. (1988). Making training accountable: Assess its impact. Personnel Journal, 67(l), 73-77.

Fulmer, R.M. (1988). Corporate management development and education: The state of the art. Journal of Management Development, 7(2), 57.

Kelly, A.I., Orgel, R.F., & Baer, D.M. (1984). Evaluation: The bottom line is closer than you think. Training and Development Journal, 38(8), 32-37.

Kelly, H.K. (1982). A primer on transfer of training. Training and Development Journal, 36(11), 102-106.

Kirkpatrick, D.L. (1979). Techniques for evaluating training programs. Training and Development Journal, 33(6), 78-92.

Mahoney, F.X. (1980). Targets, time, and transfer: Keys to management training impact. Personnel, 57(6), 25-34.

McEvoy, G.M., & Buller, P.F. (1990). Five uneasy pieces in the training evaluation puzzle. Training and Development Journal, 44(8), 39-42.

Merriam, S.B., & Caffarella, R.S. (1991). Learning in adulthood. San Francisco: Jossey-Bass.

Naisbitt, J., & Aburdene, P. (1990). Megatrends 2000: Ten new directions for the 1990s. New York: Morrow.

Nanda, R. (1988). Organizational performance and supervisor, skills. Management Solutions, 33(6), 22-28.

Schmidt, F.L., Hunter,J.E., & Pearlman, K. (1982). Assessing the economic impact of personnel programs on workforce productivity. Personnel Psychology, 35(2), 333-346.

Sheppeck, M.A., & Cohen, S.L. (1985). Put a dollar value on your training programs. Training and Development Journal, 39(11), 59-62.

Swierczek, F.L., & Carnichael, L. (1985). The quantity and quality of evaluating training. Training & Development Journal 39(l), 95-99.

Trapnell, G. (1984). Putting the evaluation puzzle together. Training and Development Journal, 38(5), 90-93.

Zenger, J.H., & Hargis, K. (1982). Assessing training results: It's time to take the plunge. Training and Development Journal, 36(l), 10-16.

Authors

Patricia E. Boverie, Ph.D., is an assistant professor of training and learning technologies at the University of New Mexico. She teaches courses in adult learning theory, training, critical thinking, and team and group development. Her current research interests include examining efficacy in groups, team development, the evaluation of delivery systems, educational change and reform, and the examination of adult-study skills. In addition to her teaching responsibilities, she has a consulting practice.

Deanna Sanchez Mulcahy received her Master's degree in training and learning technologies from the University of New Mexico in 1992. She currently works at the University and is a private tutor of adult learners. In addition, she has worked with Centerline Enterprises and Zia Imports as a training and management consultant.

John A. Zondlo is a Senior Human Resource Development Consultant for Presbyterian Healthcare Services, an integrated healthcare delivery system headquartered in Albuquerque, New Mexico. His responsibilities include overall coordination of a leadership-development program for more than 300 managers and supervisors across the organizational system. He has more that fourteen years of experience in health-services administration and is a Fellow in the American College of Healthcare Executives.

Reproduced from The 1994 Annual: Developing Human Resources by J.W. Pfeiffer (ed.). Copyright © 1994 by Pfeiffer & Company, San Diego, CA. The 1994 Annual may be purchased from Pfeiffer & Company by contacting the customer service department at 2780 Circleport Drive, Erlanger, KY 41018 or by phoning 800-274-4434.

Back to Zondlo home page


Copyright © 1995 John A. Zondlo/ACCESS Learning Systems. All rights reserved.