In this episode of Excellence in Training, Shawn Rogers provides some of this thoughts on measuring training effectiveness.

As a GM employee, Shawn has exposed to the beauty and complexity of today’s automobiles.  Think back to the first automobile that you drove 20 or 30 or 40 years ago, and you’ll recognize how far the technology has progressed to make automobiles more efficient, more comfortable, and more safe. Compliance programs have come a long way as well. Joe Murphy recently tweeted that “in the old days, once a month the General Counsel would walk over to the Marketing Department, open the door, and yell “stop that!”

There are many subsystems that make up an automobile. One subsystem reminds me of a compliance training program — the windshield wiper system. This might seem odd, but it provides an interesting analogy. The windshield wiper system is critical to vehicle safety. Nobody would want a car that didn’t have windshield wipers. The wipers are a subsystem of the car’s overall safety system. The car’s entire safety system is designed to (1) prevent crashes, and (2) protect the occupants when a crash occurs.

Similarly, a compliance training program is very important. No company executive would ever say a company does not need to conduct compliance training. However, compliance training is just one part of the overall compliance program. It is not the entire program. The compliance program is designed to (1) prevent compliance violations, and (2) protect the company when a compliance issue occurs.

The windshield wiper system helps prevent crashes by making sure the driver has good visibility. A compliance training program has a very similar purpose. A good compliance training program provides clarity to the employees on how they are to behave in their daily jobs. When properly implemented, compliance training helps employees stay within the guardrails. The compliance training program fulfils a very specific purpose, but there are other parts of the compliance program that have to be in place as well to prevent compliance lapses to keep the company safe.

If you wanted to measure the effectiveness of a windshield wiper system, you would not try to come up with a metric that measured how many crashes the wiper system prevented. That measurement is too far removed from the specific function of the wipers, but it would be a good measurement of the overall safety system. The effectiveness of the wiper system must be measured by its ability to keep the windshield clear of visual obstruction when it is raining.

You would never measure the effectiveness of the wiper system by counting how many times the wipers were turned on, or how many window swipes were performed. The number of swipes reveals nothing about a wiper’s effectiveness. It could be a completely accurate metric, but it is also irrelevant to the question of whether the windshield was clear.

To measure the effectiveness of a compliance training program, you can’t come up with a metric that measures how many violations it prevented. Everybody knows intuitively that training helps prevent compliance violations. Again, that measurement is too far removed from the purpose of the compliance training program. However, it would be a good metric for the overall training program if you could figure out how to do it.

But how often do you see companies reporting the number of classes that were delivered? Or how many hours of compliance training were completed? It happens all the time. It could be a completely accurate statistic. It could be a measure of compliance program efficiency. It could be an indicator of an active compliance training program. But it in no way shows if the compliance training is effective.

But there are ways to measure training effectiveness. You can show that the training was aligned to the company’s risk profile. With user surveys and focus groups you can measure whether the learners feel that the training is applicable to their role and you can measure user satisfaction. You can ask learners to give examples of how they have changed the way they do their jobs.

Why don’t companies do a better job in measuring the effectiveness of compliance training? Because it’s very challenging to do. But there are ways to do it. Shawn conclude with one of his current ‘most favorites’ implemented at GM this year.

At GM there is a cybersecurity course that explains how to avoid phishing email scams. It is required of all employees that have a GM email account. To measure how effective the training was, the IT function came up with a method of sending out emails to random batches of employees that should have been recognized as phishing emails if they had paid attention to the training. If the employee recognized that the email was suspicious and clicked on the “Report Phishing” button, they were congratulated on reporting the email as suspicious. However, if they clicked on the link in the email, the IT team knew that the training had not met its objective. And, those employees that clicked on the link were kindly informed that they had failed the competency test and were provided with immediate feedback on how to avoid phishing scams.

Disclaimer-As a company, GM uses many training vendors. GM’s compliance function primarily uses two vendors. Rogers has worked with other good vendors that currently do not work with GM. Rogers is not promoting any specific vendors, nor is he disparaging any specific vendors in this podcast. And, of course, these opinions are Roger’s alone and opinions that  developed over almost 15 years. He is not speaking on behalf of GM in any way.