To address the problems the industry currently faces to develop valid and
reliable CRM metrics, we will have to sufficiently frame the problem. Dave
Wilsons overview is very helpful for beginning to frame the problem. It is my
desire to travel with this group down the path that Dr. John Wise challenged us
all to take to identify the theoretical underpinnings of CRM performance
measurement.
In this listserv response, I will focus specifically on performance measurement.
Since I believe that performance measurement can be reflected in both the
process as well as the outcome, I will further narrow my discussion to the
process (the management skills that "pilots/mechanics/ramp agents/controllers"
use to meet "flight/hangar/ramp/tower" circumstances).
Since I know that there are many folks on the listserver who are not focused
specifically in aircrews, you can simply replace the references to pilot with a
different job position.
I think that it is safe to say that if we hope to measure anything, we have to
have a set of standards or expectations unique to the job position as a gauge
for performance. I think that it is also safe to say that there are very few
organizations that have specific expectations for management skill (CRM)
performance. While I do not pretend that Delta has the management skill
performance expectations that will last for all time, we do have a defined set
of management skill expectations that serve as the foundation for building
training and assessing pilot performance. We also have these defined
expectations for other areas within the company like the ramp. We are
currently working with NASA researchers conducting qualitative analysis of
aircrew LOFT performance to systematically identify the management skill (CRM)
performance expectations that should be on our pilot task list.
It is damn near impossible to measure something for which there are no set
expectations. However, from where should the CRM expectations emerge? Who
determines what is an appropriate management skill expectation? Some of the
more frequently used CRM metrics require the rater to determine if the
performance is unsatisfactory, minimally acceptable, standard, or outstanding
(or some other 4 point scale). The fatal flaw in this approach is that these
assessment tools (to my knowledge) are not anchored to a set of stated
management skill standards. The selection of a grade is based on the observers
perception of what is standard and not by an anchored set of expectations.
***The way that they try to compensate for this fatal flaw is by providing
observer consensus/calibration sessions to converge on this amorphous norm.***
Using consensus to converge on a norm (focus group research sessions) should be
conducted by the CRM program designers to identify and document the norms to be
placed in the task list; not by evaluators to help them determine the norm "on
the fly" for each observation. Once the standards/expectations/norms are
determined, then it is beneficial to have calibration sessions with the
observers.
I do not think that crew management skill expectations necessarily have to come
from the research heavens as an empirically derived set of behaviors. Airline
managers have always set clear procedural expectations without the aid of an
empirical analysis, why not do the same for management skills? It is not
unreasonable for an airline manager to identify a set of management skill
expectations for aircrews.
(e.g. For any distraction that causes the a crew member to divert his or her
attention, the crew members individually insure that someone is always
monitoring the plane, the path, and the people.)
When an expectation is combined with a grading scale criteria (e.g. 1=did not
recognize or correct, 2=recognized late and/or corrected slowly or incompletely,
3=recognized immediately and corrected completely) we have some of the basic
building blocks for measurement.
Building a system that supports error management:
One of the primary reasons for any CRM program is to assist the senior
management with the management and mitigation of error. One of the fundamental
assumptions with error mitigation is that crew members recognize and respond to
error producing conditions (breaking the error chain). It is important to
point out that airlines do not have the luxury of unlimited training so we have
to focus our training curricula on specific skills that will transfer to other
contexts as well as the most likely error producing conditions. The airline
managers challenge takes a different perspective when one considers that we have
to build a crew members ability to recognize and respond to error producing
conditions in just 50 minutes of classroom and 1+30 - 2+00 in a simulator LOFT
session each year.
To strategically move to an error management-based CRM system, we have to be
specific about our expected responses to certain conditions and have a grading
scale that measures the level of recognition and response. The three point
grading scale highlighted above is just a starting point for discussion. It is
not the one that we use.
Once an airline manager defines a set of Management Skill (CRM) expectations and
a grading scale, then the airline has the basis for building training and
assessment. Until an airline documents a clear set of CRM expectations and
describes how they will hold people accountable, I do not believe that the
management skill performance can be adequately measured nor will the desired
change in the organization be as efficient as it could be.
The concept of an audit trail is central to AQP as well as any structured
instructional systems design effort. All audit trails start and end with the
clear set of expectations highlighted in the task list. CRM is no exception.
Before we will be able to assess a pilots management skill, we will need to
define the underpinnings of skill. One simple definition of skill is "having an
appropriate response for a given set of circumstances." I think it is safe to
say that our training programs are designed to build or reinforce a pilots
response to certain circumstances. This is just as true for technical training
as it is for management skill (CRM) training.
Crew vs. Individual Assessment:
There has been a lot of ballyhoo about busting crews. Having worked with the
AQP team at my airline to build the first generation of LOE assessment tools, I
can say that collective crew grading is not desirable. Management skill grading
will need to focus on an individuals performance in the crew setting. As a
manager, you have to ask yourself: "Am I going to use one gradesheet to
determine a collective crew grade?" Think back to college when you had to do
group assignments. What if your entire semester grade was determined only on
what your group did collectively? How would the professor ever know how someone
within that group performed? Would it be possible that someone who had poor
skills was able to slip through because the strength of the other group members
carried the person? What happens when that person who slipped through with
insufficient individual skill takes his or her credentials to the workplace?
When you transfer this logic to pilots, it becomes clear that the assessments
have to be on the individual level.
We might be able to make a case for team assessment for Air Force B-52 bomber
crew that always fly with the same team members. However, I would still want to
know how each individual was performing within the team. In the first year I
flew with the airline, I never flew with the same crew twice.
These CRM management skills are "individual skills in a crew setting." There
are some responses that we expect from all crew members but there are some
unique responses that we expect from captains and first officers.
The Underpinnings of Skill:
If crew member skill is what we are trying to build - and skill is an
appropriate response to a given set of circumstances - perhaps we need to
identify the appropriate response we expect from crews when faced with certain
circumstances. While it would be impossible to document all the possible
combinations of circumstances at a micro level, it is not unreasonable to
identify these conditions or circumstances at a macro level (Automation changes
on arrivals). The safety data analysts have done a good job of building
taxonomies for the conditions that consistently lead to crew error. The
management skill expectations can be IF-THEN statements that are written at the
macro level and apply to all phases of flight.
In fact, the CRM expectations probably should be worded with the following
sequence:
"Given Set of Circumstances"...."Appropriate Response"
I think that it is entirely appropriate for a manager to set expectations
"appropriate responses" to "given set of circumstances". In fact, many airlines
expect the subordinate crew members to speak-up if they are uncertain and will
hold them accountable if something happens and they did not speak-up. As
trainers, we can build the circumstances into our Line Oriented Training "e.g.
ATC directed runway changes" and observe the crews response. For line
evaluations, we cannot control the conditions, but we can observe the crews
response to the conditions.
This is probably the first few steps in a "Journey of a Thousand Miles" but I
hope that this helps to further focus the issues surrounding CRM metrics.
Best Regards,
Vince Mancuso