I am very glad that this industry group is getting right to some of the core
issues facing the CRM and Human Factors practitioners such as metrics. I have
given the issue of CRM metrics considerable thought in my pursuit of the CRM
performance measurement grail. Like most pursuits for knowledge, the more you
know, the more you realize you do not know.
However, when one distills all the possible measures to their core, they appear
to fall into one of three bins: 1. Outcomes, 2. Processes, 3. Perceptions.
Each of these measures have useful purposes as well as limitations.
Process Measures:
John Wise highlighted very eloquently the limitations and traps of perception
measures. However, practitioners may not want to remove these perception
measurement tools (surveys, etc.) from their operational toolbox while we
continue our search for better process and outcome measures. With appropriate
caution and respect for the traps and limitations, perception measures can still
serve a useful purpose.
Outcome Measures:
As all researchers know, clean cause-effect outcome measures in the operational
environment are very difficult to obtain because of all the confounding
influences. We know that it is not a simple matter of identifying one
independent variable (CRM training) and one dependent variable (incidents).
However, I think that there are many unexplored areas where we could identify
dependent variables that operational managers use (or could use) then identify a
number of independent CRM / Human Factors variables that appear to have a
predictable influence.
Process Measures:
This is perhaps the area of CRM / Human Factors that is most fertile for
development. When a clear set of CRM process expectations are developed (e.g.
a CRM task list or a set of clearly defined management behaviors), we have a
focal point for observing and assessing the process. Keep in mind that not
every single one of these CRM processes has to be empirically tested any more
than each item on a checklist has to be empirically tested (we do not have to
empirically test whether it is a good idea to put the gear down or set the
altimeter). Researchers and safety analysts continually highlight the
conditions that lead to crew error. When these tasks/behaviors/expectations are
identified, they can be added to the task list or the management behavior
requirements. Many, however, have enough surface validity that an operational
manager can simply make them a requirement. (certain elements of a crew briefing
is one example). There are vast unexplored areas in the measurement of CRM
processes and this forum is a great place to explore them.
Just a few more logs for the CRM metrics fire...
Vince Mancuso, Ph.D.