Will Thalheimer: I am a learning researcher and consultant. I started in the "training and development field" in 1985 and I played a number of roles since then. I have been an instructional designer, simulation architect, project manger, product manager, a trainer and now I play more of the researcher/consultant role.

I started Work-Learning Research back in 1998 to help bridge the gap that exists between researchers and practitioners. There is a lot of good research out there and a lot of good practice out there, but the two groups really don't talk to each other. They are really speaking completely different languages.

So one of the things that I think needs to be done is to translate the good research into a format that helps instructional designers, e-learning developers, trainers, educators, to build better learning interventions.

Kapp: Why do you think there is not much dialogue between the groups?

Will Thalheimer: First of all, they are rewarded for different things. The researchers are rewarded for coming up with theories and publishing papers that are read only by fellow researchers. The language they use is different and geared toward each other. But on the other hand the practitioners are rewarded for getting a learning intervention out the door, so they don't have a lot of time. Researchers have more time for reflection and to think up complex theories and ideas. Whereas practitioner's want an answer, "what do I need to do?" Because researchers are rewarded for those research papers, they are not rewarded for talking to practitioners. The researchers are not rewarded for writing papers for practitioners. In fact, those kind of efforts are looked down upon.

I heard that Richard Mayer (one of the really good practical researchers) said one time that his fellow research colleagues said he has become a "popularizer" because of the books he writes and that's a dirty word in research circles. You don't want to do that.

So I think there is a great need for a translator or a wholesaler between the groups. Someone to bridge that gap is needed…So that's why I started Work-Learning Research. I thought that there is really a need to translate between the two groups, to bring them together.

I also want to take the learning research and make it more palatable for the practitioners, and to encourage the researchers to think more about the practical aspects-and that is sort of a longer term effort. That has to be done as well.

Kapp: Well we already covered my next question, why do you think most designers are unaware of the hard research. I think I heard you say because of the language.

Will Thalheimer: Also, they don't have time; the language is undecipherable most of the time for practitioners. And, frankly, there are not that many sources of it. There are just a few of us out there that are proselytizing on using research-based practice. I personally feel that 90% of the gurus out there ought to be touting research-based recommendations, and instead it's more like 1%. Even if you are a practitioner you may not know the research is something you ought to be paying attention to.

Kapp: Do you find that it is hard to get clients and are people open to the message?

Will Thalheimer: There are always clients out there, sometimes they don't know until they talk to you that they need what you got. Part of it is finding out what type of delivery the clients are going to be open to and what they really need. This is a new area so there are not a lot of people out there doing it...the research to practice thing. There are no business models to follow, so you sort of have to experiment with things to figure out how things work.

Kapp: To change the topic a little and move more towards e-learning, how would you define e-learning?

Will Thalheimer: I know some people define e-learning as anything that is computer-based, anything that is an electronic or a digital format. I find that definition is not as helpful as one that thinks of e-learning as the interconnectivity between the learning material and the learners that you can access either through an intranet or the Internet.

The reason I think that definition is valuable is because there are special learning methodologies that are enabled when you have that type of connectivity. Particularly, I have argued that because e-learning has the opportunity to connect with learners over time, that aspect of it needs to be taken advantage of. Especially because the learning research is very clear that people learn a lot more, or the learning sticks better, or they forget less if they can get spaced repetitions of meaningful content over time.

Kapp: Yes, I always think that almost everyone you meet can tell you what is on a Big Mac because advertisers are periodically bombarding you with that information. You can't help but learn it. Where as, in the training field, we seem to do a one and done.

Will: Yeah exactly. I like your metaphor. It's true. It is one of the problems in our field. There is this expectation that learning can be simple and effortless and that is ridiculous and silly. It goes against everything we know. Some learning can be effortless and fun, but most meaningful learning requires learners to exert themselves. We are going to get into compliance training later and that training is just a mess because of the lack of repetition.

Kapp: I was talking to Clark Aldrich for this series, and he mentioned the idea of "cognitive dissonance" People get upset or frustrated at the learning process and believe that it isn't working them, but sometimes those are the best outcomes in the long run as opposed to the quick and easy bit sized learning chunks.

Will Thalheimer: Yeah sometimes you have to have an emotional scare that will make you sit up and pay attention and if it is emotional and related to something personal, it will connect to more accessible memory structures that you will have access to later. It can make learning more effective and actually improve the learning in some cases.

Kapp: Speaking of improving learning, you mentioned before that well designed learning can improve learning by much as 220%, yet there is a lot of pressure right now to learn quickly. Can well designed instruction and rapidly designed instruction occupy the same space?

Will Thalheimer: I think yes, however, you have to do it right. You probably have to have some practice doing it in your content area. But the truth is that sometimes there is a trade-off because more thoughtful instructional design, better needs assessment and more time spent upfront is going to get you better results down the road. I know there is a big push for the rapid e-learning thing, not that it's a bad thing, but certainly there is this major push to do everything rapidly and that probably is going to create some problems. We are probably going to over do that a little bit.

I think that comes out of the push from a couple of years ago that e-learning was going to save us money. So now everyone has the mind set that…"well we are trying to save money with e-learning," but that is only one part of the equation. The part of the equation that we forget is that we are trying to get some learning benefits so we can get some performance benefits, so we can make money or save down time or serve customers better or somehow have people go up the learning curve and improve their performance. So the things that are easy like cost savings we tend to do and speeding up delivery is part of the rapid learning tsunami the field is facing.

Kapp: Right, and also, as I was reading some of your work, in terms of pushing the learning time off into the employees' private time.

Will Thalheimer: Yes, that's an ethical issue that we should be facing. I've had colleagues in the field that create compliance training tell me that their clients tell them that one reason they are doing e-learning is because now our employees will do this learning at home during their own personal time. And part of that is ok in that everyone must be responsible for their own development. But on the other hand if it is a conscious effort to take another personal chunk out of an employee's own time it is unethical. And I think it is going to backfire in the long run.

Juliet Shore is writing about the topic of overworked Americans and that Americans are working more than ever before, and how both parents have to work. This is going to take a toll on our families, our health, our parenting, our citizenship, our civic organizations, our society, everything. It is just another thing that is going to hurt us.

Kapp: Do you find a lot of research out there about ethics in our field. One of my students asked me about ethics in our field.

Will Thalheimer: I tend to read more in the learning research area. But frankly, I haven't heard much talk about ethics in the instructional design literature. Roger Kauffman has written some good stuff, and his recent book on evaluation with Guerra and Platt makes the case for considering societal impact, but mostly ethics gets short shrift. It goes back to what is our ultimate goal; is our ultimate goal only "business results." I'm not sure I buy that completely as an instructional designer. I think I have some responsibility to the learner as well. I think we need to balance the needs of both of those stakeholders.

Kapp: Now getting back to "productivity," do you really think e-learning can deliver productivity improvements. The forward looking people are saying that we need to think about performance improvement, and show that e-learning improves employee performance. Do you think it can?

Will Thalheimer: I think it can, absolutely, but it's not simple. It's not like you give them good instruction and that is automatically going to improve employee performance. Anybody that follows the performance improvement model knows that there are other things that affect on-the-job performance. A really good instructional professional is going to take into account the management, the organization behind the improvement effort, potential obstacles and often times we don't have much leverage on those things. Usually we don't have a lot of control over those things. But ultimately we shouldn't even be creating instruction…even the best instruction in the world… if there are too many obstacles to the learner performing well on the job.

Kapp: So it needs to be part of the system?

Will Thalheimer: Yes, it should be part of a system. On the other hand, you can't focus only on the system. On the road from learning to on-the-job performance to business results, one of the things that needs to happen is that the learning should be designed in such a way so the learner can retrieve information they learned while on the job. So that should be one of our goals as instructional designers, to create learning that can be retrieved on the job. Again, that's not the only thing; you need to focus on some of the other variables that affect performance. But, if you don't get retrieval of your learning then your performance will not change, so there is still a place and a role for the instructional designer in that process.

Kapp: So when you are talking about retrieval are you talking about approaches like Gloria Gerry's electronic support system, or are you talking about the learner recalling in his or her mind and what was learned?

Will Thalheimer: Both really. At some level I don't care whether it's retrieval of knowledge or skills from long term memory, or an electronical support system which encourages them to perform well. Usually there is an overlap there and you can have an electronic performance support system that reminds you to retrieve or provides the triggers that make you retrieve the information, it is not a conscious process. What I focus on in my consulting and in my workshops is to design the instruction so it enables the learner to retrieve what they learned. And I use the term "spontaneous remembering," and that is one of the things that we should try to design into our instruction. We need to make sure that when the learners face their performance situation the cues in that physical/mental environment remind them of what they learned. So they are more likely to use their previous learning to achieve the desired results.

Kapp: Can you give me maybe an example of that?

Will Thalheimer: Sure, say you take a course in Microsoft Excel, and maybe someone designs a nice e-learning program to let you practice those skills in an actual simulated Excel Program. Because of the cues (the rows and the columns and the cells) are the same cues that you are going to face in your performance environment, you are going to be reminded and your memory will be triggered to recall what you have learned when you are in the actual environment.

Kapp: So is that like Roger Schank and Clark Aldrich's concept of "learning by doing." So you are actually doing the task in the learning environment.

Will Thalheimer: Yeah, to be honest it's a little more fundamental. With all due respect to Rodger and Clark; it's just a fundamental learning principle. To use a research term it's "encoding specificity." It is just fundamental human psychology, that when you learn something you take in more than just the learning message, you take in background stimuli at the same time, and they are stored together in the memory. So later on, when you encounter the background stimuli, it triggers the actual learning message. It's much easier to explain with a graphic.

Kapp: So basically you are getting conscious or unconscious cues from other things in the learning environment.

Will Thalheimer: Yeah they don't have to be unconscious but they can be.

Kapp: You talk about looking at the instructional design literature, and you say you focus more on the learning theory or cognitive theory.

Will Thalheimer: No, no, no. Wait a minute; let's be a little more precise, because I try to be atheoretical. I don't care if it is a cognitive psychologist, if it comes out of the Behavioiorist School or the research comes from the Constructivists or even the Bolsheviks or whoever. It doesn't matter. When I look at any research study I look at what is actually being manipulated, what's actually being done and see what causes what. And if you do a really close reading on the research, you find many examples where the researcher claims in his or her discussion that a certain theory is supported and you've actually read the methodology closely and you say, I don't think so. So you have to look past the labels and ignore that stuff.

Kapp: So you mainly analyze the results?

Will Thalheimer: I analyze the results, and the procedure. Of course I read the other parts but I am a skeptic. I never look at one study. I always look at multiple studies. I look for a confluence of evidence. You have to look at the situation; you have to be careful that not everything is coming from one lab or one research paradigm because that can be non- representative of the real world. Try to figure out how ecologically balanced (to use a little jargon) the study is. Can it be generalized or relevant to the real world or not.

Kapp: Do you find a lot of educational research hard to generalize. Because I know there is always the balance to keep intact the independent and dependent variables but still have generalizability.

Will Thalheimer: Yes, what I find interesting is that some of the research in the early 1900's or 1950's wasn't as controlled as we would like today, but it was much more real world. And it didn't include many statistics. But the results were very striking, and I often go back to some of that stuff. But there is a balance between how tight the controls can be and how real world it can be. And sometimes you need both types of studies; you need the lab studies and the field studies and analyze them from a thoughtful perspective and draw your conclusions from both types of studies.

Kapp: Do you think there is more valuable educational research going on today, or less, or is it pretty much the same?

Will Thalheimer: That is a really good question, I really haven't thought about that. I don't know (laugh).

Because when I am deeply involved in studying the research I never really think about it in those terms. If I were to make a guess, I would say probably about the same. You know the way the research field works is that it goes through fads. So for example a great deal of ink was spilled over learning objectives back in the 1960's, 70's, and early 80's and now there is virtually no research on them whatsoever. Lately, the constructivist models of learning are where the research is and where the money is. But then that is going to meet some sort of diminishing returns, and it'll move on to the next thing.

The sad part is that even with all those studies on learning objectives the practitioners still don't really know what the research says. How can that be? The definitive studies were done back in the 70s? What does that mean? Now there are some studies that I would like to see done… one interesting question that I have been puzzling lately, is whether prequestions are better than learning objectives at focusing learner attention. In the lab, the results usually show that they are equal. I want to know in the real world whether learners are equally likely to process learning objectives and prequestions. And my guess is that they are much more willing to look at the prequestions because they are more interesting…sets of learning objectives are bor…ing!

But prequestions if they are designed well can really energize a learner while learning objectives have no effect if they are not read and processed. There are some gaps in the research. I have been meaning to come up with a list of gaps to encourage researchers to do a little extra work in certain areas. Someday soon I hope.

Kapp: Do you have a prediction of what the next research trend will be? Or do we wait for the list (laugh).

Will Thalheimer: Yeah, (laugh) wait for the list. Well there is a lot of research going on in the brain sciences. A lot of money is in that area. For example Functional MRI imaging that researchers are using to see what parts of the brain are being activated when people are learning. I think more money will be spent on that. But someone is going to come up with (I don't want to say crazy theory) but some new theoretical angle and everyone will say that sounds good and we will all rush to do that. Then we report something like that to the practitioners like Constructivism and then completely oversell it and because so many practitioners in our field don't have fundamental knowledge of learning, they are not going to know how to interpret it. Then it gets misapplied and that happens way too often in our field.

Kapp: I think this brings us up to interactivity, Gagne's first event of instruction is to gain attention, and you talk about too much interactivity or videos in training.

Will Thalheimer: Gagne's first event of instruction (grab attention) is a perfect example of misapplication, and he knew what he was doing. What that first event meant to the practitioner is that I have to go make silly games and do things in the classrooms and put a lot of sizzle in there and I have to do nutty things. And it really wasn't that simple. It has really created a lot of difficulty.

So to get back to the Interactivity thing…that is another good example, "oh we have to have interactivity," well yeah. But it is not only about energizing the learner in general, its about focusing attention on particular learning material and presenting it in a particular way so they can get the concepts and so they can practice the concepts. I wrote an article called "Stop Aiming for Interactivity!" because it is not that simple. After I wrote that so many developers got in touch with me and said "oh thank you" because our clients keep clamoring for interactivity which didn't make sense for the goals that they were trying to be achieved. It was overkill.

Kapp: A lot of clients say "oh we need interactivity can you put in a video?"

Will Thalheimer: (Laugh) That's a good one. There is no interactivity in that at all.

Kapp: Well a follow up then is with generation X; they are growing up with electronic and digital media, do you think some of the studies about learning would have different results if they were done on today's younger population of kids?

Will Thalheimer: I don't think that the basic fundamentals of human learning will be different. I think there may be some differences in how well young learners are dealing with distractions. The basic tenants of how people learn through repetition and retrieval/practice, through getting feedback, and practicing in the right context are all going to remain the same. One of the challenges will be what is tomorrows' performance situation going to look like. If the performance situation is one where there is more and more multitasking and more multimedia, then new learners may gain an advantage with being able to deal with that type of environment. But I'm not convinced that that's true. I have been talking to people about office space, because I am looking for a place for my company Work-Learning Research. And real-estate people have told me that lots of their clients used to go with that open office space design but it didn't work because people wanted their own offices so they could have quiet and could focus so they can get work done. And these are high-tech companies with young workers so I think there is a lot of that going around. They are finding the special noise cancellation head sets.

Kapp: Yeah, we have an open lab here and a lot of the students wear headphones so they can get work done and focus.

Will Thalheimer: What I say in my consulting and workshops is that the human learning system really hasn't changed that much in the last 10,000 years. The basics are the same, and unless you got mutant learners with some special genetic learning mutation, the fundamentals are still going to work.

Kapp: So do you think learning is more nurture vs. nature, that human beings learn the same regardless of the culture and environment?

Will Thalheimer: I think both nature and nurture are important. As an instructional designer, the things that you change mostly are the environmental side of it and not the biological side of it. I can't wait till the day when we can go to our corporate training mangers and say "ok, what we would like to do to help your employees learn is to change their genetic code." Given the battle on stem cells, I don't think that is going to happen any time soon. I wouldn't sign up for that training class if I was an employee.

Kapp: (Laugh) I wouldn't either. Like the Matrix, you just plug you right in.

Will Thalheimer: (Laugh) That's right, so we need to focus on the nurture side where we can have some level of control.

Kapp: Moving on…you mentioned that Level 2 assessments given right after the learning event can be misleading, why do you say that and how do you think that can be changed?

Will Thalheimer: Well the long answer is really complicated, but the short answer is that, and this actually relates to some research that has been done. Let's take a little digression and then I'll answer the question. Just for your amusement.

Kapp: Ok.

Will Thalheimer: A lot of the research was done where…the researcher brings the learner in and has them learning something and then tests them right after. And there is no delay between the learning and retrieval situation. And a lot of that learning is not relevant to the real world, because in that kind of situation all you have measured is how well they learned and how well they can retrieve information in the exact same mental and emotional context. But usually learners have to wait a day or a week before they can retrieve the real information. And the same is true for a Level 2 Assessment. Measuring a learner right after learning only measures their ability to retrieve in the environment in which they learned in.

Kapp: So is there any value in the Level 2 evaluation?

Will Thalheimer: Well you can delay the level 2 evaluation. That is a recommendation that I often make depending on the situation. If you can delay it, that allows you to measure the learning intervention's ability to, not only create learning, but also to minimize forgetting.

Kapp: So you can use the Level 2 assessment as another education tool?

Will Thalheimer: Well, the Level 2 assessment right after learning can provide your learners with the retrieval practice which is known to minimize the forgetting curve. You can also use a level 2 assessment a week after learning and that will measure a little bit better whether the learner has been able to remember anything over time. Because if you only measure immediately after the learning event or at the end of the day after a training session or right after an e-learning program is completed you are not really able to access whether the learning program helped the learner remember something over time.

So for example you could have a great e-learning or training program, and you give them the assessment instrument right after and they score 100% but if you gave them the same assessment a week later they might score a 40%. Well, then the learning assessment right after the learning is a lie. Right. It is a complete and utter mischaracterization of the learner's ability to be able to remember on the job.

Kapp: Right

Will Thalheimer: That is often what compliance training does. It asks people to answer questions immediately after they have taken an e-learning program. And "yes" they are able to answer them. But is that level 2 assessment predictive of their ability if they are able to remember when they need that information when it will protect the health and safety of clients or themselves or enable them to follow certain regulations? No, that test is completely not predictable of the situation they might face a week or month later.

Kapp: So it would only be predictive if it was given a week or month later?

Will Thalheimer: It would be much more predictive, but none of these things are 100% predictive.

Kapp: So, would it be better for a company to measure the performance of an individual based on a learning event rather than knowledge recall.

Will Thalheimer: Well knowledge recall…if it is trivial knowledge…I would never recommend that, never. Unless that knowledge was needed as a prerequisite for learning something else more complex. But for the most part, a valid assessment requires that you more or less, put people as much as you can into a realistic situation and make them either make a decision or perform a skill in that situation.

Kapp: With your experience with working with companies…before you arrive, do those organizations design instruction in that manner or do they usually design it with a test at the end to recall basic knowledge.

Will Thalheimer: They usually use knowledge recall…well…no, it's usually a smile sheet which is only helpful to get a general sense if the employees like it or thought they learned something but they are not that beneficial. The Level 2 thing is complicated because of a lot of factors. Like…What is the real context in which the person will be performing in? How long is it from the learning? How authentic of a situation can you create? One of the problems with the way we have been designing instruction is that we have used instructional objectives to design tests and these objectives are usually low level. They are things that are easy to measure. I think instead of using low level criteria, we should be using what I call "Performance Benchmarks" which are more broadly defined and are meaningful learning instead of trivial knowledge. It frees the instructional designer to measure meaningful learning instead of trivial knowledge-based information.

Kapp: So what would an example of one of those be?

Will Thalheimer: Well you could put people in a behavioral simulation. You can certainly have a performance benchmark be the actual work performance, that would be ideal, but that is not always possible. So if you can't do that, a simulation assessment is good but that can be difficult. So, the next best thing is a simulation-like question or situation. Actually ask people to perform. It really depends on the skill that you are asking them to learn. That should dictate which performance benchmark performances are appropriate. I think though that the performance-benchmark notion frees us up, because we can't always use a level 3 or 4 assessment because it's too expensive. But if we can use the performance benchmark concept…I sort of think of that as a level 2.5 because it is much more than a silly knowledge-based questionnaire at the end and it is definitely better than level 1 smile sheets. It is fairly predictive of how people will really be able to perform on the job.

Kapp: Interesting, but jumping back for a moment to when we discussed the speed of learning, I know one company that said, "Hey Karl is there any research out there that shows how you can shave 5 minutes off the learning lesson if you do A, B or C?"

Will Thalheimer: Well first of all, I don't know how you would do such research I think if you understand human learning and know that the goal is to provide learners with retrieval practice opportunities set in the similar on-the-job performance context that they will face...we can try to aim for that. That would be my quick and dirty rapid instructional development tool, to provide them with instructional materials and give them the chance to practice in a real world, authentic environment.

Kapp: Are there techniques that a designer can use to make someone learn faster?

Will Thalheimer: Well, yeah, if you want to learn something fast you can start with drill and practice, but the danger with that is burnout and boredom and it probably won't be remembered over time. So there is a tradeoff as well.

Kapp: You have talked about the benefits of questions, prequestions instead of objectives, and authentic questioning. Can you elaborate about the role that questions have in instruction?

Will Thalheimer: Well, I have a whitepaper called the "Learning Benefits of Questions" which has 90 references to research. There is almost too much to say. But…questions are very powerful for many reasons.

Prequestions are powerful because they help focus the learner's attention on the most important part of the learning material. A learner's attention wanders as they focus on learning materials so…Learning objectives and prequestions are good at helping learners focus on the really important stuff. On the other side, when questions are used after learning material is presented, there are a number of benefits. First there is a repetition effect because you are getting the learning material early and then getting a question on it. It provides repetition and it also provides retrieval practice if you set the question up properly. You can have them follow the same routes through memory that they will have to follow on the job. Repetition and retrieval practice are arguably the two most important learning effects that there are. And if you can put those two together, that is a powerful thing. There are other effects of questions, in some environments there is the effect of studying. And studying provides spaced repetition and that is a good thing, but that doesn't often happen in corporate world. Unless there is some type of credentialing.

I see the repetition that my daughter, Alena, uses when learning also, it's amazing to watch. It is a joy to see her learn…as a "learning guy." I am fascinated by it. She is always learning words. You can see the fundamental learning principles at work. She learns language but she learns it slowly over time. Repetition spaced over time in a meaningful context.

Kapp: Are you at all worried about school systems not following sound educational practices?

Will Thalheimer: Well two days a week we have a Spanish speaking nanny come in and only speak Spanish to Alena. And it is remarkable, she is already better at understanding Spanish than her mother and father. Two days a week Alena goes to childcare where she plays with other kids, and one day a week one of her partners stays with her during the work week.

And yes we are very concerned about the school system. We were temporary foster parents and we had talks with the teachers, and some of those talks were very satisfying and some were very frightening. We had science teachers who didn't give kids homework and gave them a project out of the book. Instead of having them think about it deeply on their own. I mean so…yeah I am concerned. My wife and I are both very mission career oriented, so we don't have time to home school our daughter except on nights and weekends. A lot of the learning comes organically through real world situations. Not through book learning or special exercises. Education helps you have a meaningful life, it helps you get a good job, it helps you be a good citizen, it is critical. We are concerned.

Kapp: Totally switching gears. What advice would you give for training managers and developers in terms of how they should use research when they design instruction to help their organization? If you only had a few minutes with them, what kind of broad things would you tell them?

Will Thalheimer: The truth is that they are unlikely to have the time or the background to utilize research on their own, so they need to find someone who can help them. They need an unbiased perspective. One of the things I do at Work-Learning Research is to provide a learning audit, and I will look at your learning designs and program, and I have a 30 page template that allows me to compare your learning intervention to the best practices based on what the research suggests. Then I prepare a fairly intensive report on all these things and try to prioritize. And you can't fix everything, but there are some things you can do.

The other thing besides doing something like a learning audit is to do meaningful tests, and hold yourself accountable by measuring your learning program in a substantive way. Because as a training manager that will give you feedback in terms of whether or not you are making a difference. Because our field in some ways is a big joke. We don't know if we are creating anything of value, we have no feedback loop, we are in the dark, and we don't know what we are doing. Like we are on a driving range but it is completely dark. There are no lights at all…we can't see were the ball goes, there is no way to get better. You are more likely to reinforce your bad swing because you don't know if the ball is sliced. It is the same thing with the training development field, the fact that we don't measure our results in a meaningful way…we should all just be lowering our eyes in shame. It is bad.

Kapp: So is that what is driving the move toward ROI, and do you think that is helpful or harmful?

Will Thalheimer: I think in general a move towards ROI is a good thing. However, the way that we calculate ROI, even the people that are really good at it will admit that it is not a perfect process. It's a good thing, but it's not the only thing. We also need to be focused on the prerequisites of ROI like the learning results and the on-the-job performance. I actually think that the field may not change until we are forced to change, and one of the things that may need to happen is that CEO's may need to take a little more time to understand how training and development works, and to really require meaningful measurements of the learning outcomes. We are still counting butts in seats and most of us don't get burned for that; when in fact, we ought to be fired.

Kapp: Well I was looking at some of the early learning analytics, and they were still looking at how many people took this course or that course.

Will Thalheimer: Yes, and in some aspect that is important, but you have to multiply how many people are in the seats by how valuable the learning is. And if the learning is 100 % valuable then having 30 people in the classroom is good, and 60 is better. But if the learning is only 2% valuable then it doesn't matter how many butts are in the seats, that's not good. Somehow we think we'll make up the difference in volume.

Kapp: My take on ROI is that if you look at the ROI calculations in a managerial accounting book, then at Jack Phillips and what some of the other people are using, it's night and day. And then for the training professional to go in a meeting with a MBA and talk about ROI, I think that is dangerous ground. I think you just muddy the water by throwing out the term "ROI." Certainly we need to talk about cost/benefit, productivity improvements, and learning gains but I think we should avoid the term "ROI."

Will Thalheimer: Yeah, I actually have a MBA, and I never thought of it that way. And if someone looks at the way we assess our numbers they might think it is voodoo.

Kapp: Even though the ROI formula for managerial accounting is voodoo too…that is the sacred voodoo. You can't mess with that. ROI has been bothering me for a while and I think that is what I finally hit on, that is why training ROI is not gaining traction outside of our field

Will Thalheimer: Well I went into a workshop years ago with Jack Phillips and I was completely skeptical saying, "How are you going to calculate the benefits." He convinced me during that session that you can make conservative estimates that are relatively reasonable, but because I know training, development and learning, I can kind of understand that. But a hard nosed operations person will say that sounds kind of funny.

Kapp: Well I think the thing that it makes me nervous is when it comes to "let's call this ROI," I think we can quantify some things and monetize them, but I don't think throwing out a formula and saying, "hey, we achieve 100 percent ROI on this training course," is in the field's best interest.

Will Thalheimer: Yeah that is good. You should publish that somewhere.

Kapp: Yes, well, speaking of publishing, are you working on a book of any type?

Will Thalheimer: Well a book is a matter of being overwhelmed with work and being an entrepreneur and not having the time to do it. But what I decided to do is develop a series of books, and there is a long time line for that. I do not know when it is going to come out, but I am starting to offer some of my writings on the Work-Learning Research website at www.work-learning.com/catalog/. In our field, and even in the e-learning field which you would think "oh wow the information from an electronic document, this would be great" but the fact is that we are stuck in our old models and we need to have it in a book or it has no value. In the meantime a lot of information is incorporated into my research reports, whitepapers and articles. So they are available online.

Kapp: Well I have one more question, what do you think is the future of e-learning?

Will Thalheimer: The future of e-learning? I think in the short term we are going to be focused on cost savings and rapid e-learning. The next phase will focus on proxies of instructional effectiveness, like simulation and interactivity. I think ultimately more and more people in the field will learn about the learning side, and focus on that. And develop instructional e-learning designs that are actually more and more effective. Will we ever have the perfect field, and will we always measure ourselves and base everything on really good research design? No, we probably won't ever get there, but we can move in that direction. This field is based on proxies of instructional effectiveness, but eventually we are going to have to go beyond that. The buck will have to stop somewhere.

Kapp: What do you mean by proxies, do you mean a simulating environment?

Will Thalheimer: Well like a simulation for example doesn't work because it simulates. It works because it provides meaningful practice opportunities in an authentic context and it provides repetition and feedback. Those are the things that make the simulation work. What happens in the field is that a vendor will say "I got a simulation." And everyone will say "oh good that's what I need." And that is what a proxy is; a proxy is not what really makes it work. It's not the naming of it as simulation that makes it work. Looking at the underlying learning methods, that's what makes it work. As long as it's aligned with the human learning system.

Kapp: Great, unfortunately, it looks like we are out of time. Thanks for a great interview.

Will Thalheimer: Thank you for a great interview!! It's been a pleasure Karl.