You write:
"Our program has been running for six months now and I find a crossroad on
the horizon. How does one ensure continuity of the end product at the same
time gauging those results ?"
Greg Deen writes:
"My first thought is, why are you concerned at the six-month point? .....If
your courseware is sound of design, and the motive of the facilitators is
toward the correct goal, the program will survive. Don't worry about a
six-month check. If your program was in trouble, you'd have seen it
sooner. "
Tim, I think that your early focus on assessing program performance is
extremely important and timely! If you wait until you see obvious signs
that your program is in trouble before you begin assessing performance, its
too late!!!
Performance measurement systems could take months to design and take even
longer to show performance trends. Performance data collection designed to
show trends should begin from day 1 of the program. Performance
measurement should be a part of initial CRM program architecture.
Task analysis and performance measurement are the two parts of the
instructional systems design (ISD) process that are glossed over with
disturbing frequency in the CRM community. Airlines that elect to build an
AQP, for example, will not get their programs approved unless the program
has clear and unambiguous CRM task list as well as performance measurement
systems.
With AQP and other performance measurement management systems becoming the
norm, the CRM community can no longer dodge the performance measurement
issue. Dave Wilson's outstanding compilation of Metrics discussions in
the archives of this forum is a "must read" to anyone designing performance
measurement tools for CRM.
The ISD weakness of early generation CRM program design has become
painfully obvious to airlines building AQP programs. The solutions are
not easy but they are not impossible.
Some management cultures will not continually allocate money on a leap of
faith. Many management teams demand to see performance measurements.
When measurements are distilled down to their simplest form, there are
essentially three different measurement focal points (The Process, The
Outcomes, The Perceptions). All three have utility.
Early generation CRM programs relied heavily on perception surveys given to
training participants. Developing perception surveys is much easier than
developing Outcome or Process measurement tools. Consequently, the
measurement of CRM skill in an operational setting (the process) based on a
defined set of standards was almost non existent in early generation CRM
programs. Most organizations had not clearly or specifically identified
the CRM Roles, skills, or standards they were expecting from aircrews.
I dug through the CRM developers archive to a discussion on performance
measurement we had nearly a year ago. I have attached it to this email. I
hope you find it, along with Dave Wilson's writings useful as you design
your performance measurement system.
Keep up the good work Tim.... You are on track.
Vince Mancuso, Ph.D.
***************************************
Hello Folks,
To address the problems the industry currently faces to develop valid and
reliable CRM metrics, we will have to sufficiently frame the problem. Dave
Wilsons overview is very helpful for beginning to frame the problem. It is
my
desire to travel with this group down the path that Dr. John Wise
challenged us
all to take to identify the theoretical underpinnings of CRM performance
measurement.
In this listserv response, I will focus specifically on performance
measurement.
Since I believe that performance measurement can be reflected in both the
process as well as the outcome, I will further narrow my discussion to the
process (the management skills that "pilots/mechanics/ramp
agents/controllers"
use to meet "flight/hangar/ramp/tower" circumstances).
Since I know that there are many folks on the listserver who are not
focused
specifically in aircrews, you can simply replace the references to pilot
with a
different job position.
I think that it is safe to say that if we hope to measure anything, we have
to
have a set of standards or expectations unique to the job position as a
gauge
for performance. I think that it is also safe to say that there are very
few
organizations that have specific expectations for management skill (CRM)
performance. While I do not pretend that Delta has the management skill
performance expectations that will last for all time, we do have a defined
set
of management skill expectations that serve as the foundation for building
training and assessing pilot performance. We also have these defined
expectations for other areas within the company like the ramp. We are
currently working with NASA researchers conducting qualitative analysis of
aircrew LOFT performance to systematically identify the management skill
(CRM)
performance expectations that should be on our pilot task list.
It is damn near impossible to measure something for which there are no set
expectations. However, from where should the CRM expectations emerge? Who
determines what is an appropriate management skill expectation? Some of the
more frequently used CRM metrics require the rater to determine if the
performance is unsatisfactory, minimally acceptable, standard, or
outstanding
(or some other 4 point scale). The fatal flaw in this approach is that
these
assessment tools (to my knowledge) are not anchored to a set of stated
management skill standards. The selection of a grade is based on the
observers
perception of what is standard and not by an anchored set of expectations.
***The way that they try to compensate for this fatal flaw is by providing
observer consensus/calibration sessions to converge on this amorphous
norm.***
Using consensus to converge on a norm (focus group research sessions)
should be
conducted by the CRM program designers to identify and document the norms
to be
placed in the task list; not by evaluators to help them determine the norm
"on
the fly" for each observation. Once the standards/expectations/norms are
determined, then it is beneficial to have calibration sessions with the
observers.
I do not think that crew management skill expectations necessarily have to
come
from the research heavens as an empirically derived set of behaviors.
Airline
managers have always set clear procedural expectations without the aid of
an
empirical analysis, why not do the same for management skills? It is not
unreasonable for an airline manager to identify a set of management skill
expectations for aircrews.
(e.g. For any distraction that causes the a crew member to divert his or
her
attention, the crew members individually insure that someone is always
monitoring the plane, the path, and the people.)
When an expectation is combined with a grading scale criteria (e.g. 1=did
not
recognize or correct, 2=recognized late and/or corrected slowly or
incompletely,
3=recognized immediately and corrected completely) we have some of the
basic
building blocks for measurement.
Building a system that supports error management:
One of the primary reasons for any CRM program is to assist the senior
management with the management and mitigation of error. One of the
fundamental
assumptions with error mitigation is that crew members recognize and
respond to
error producing conditions (breaking the error chain). It is important to
point out that airlines do not have the luxury of unlimited training so we
have
to focus our training curricula on specific skills that will transfer to
other
contexts as well as the most likely error producing conditions. The airline
managers challenge takes a different perspective when one considers that we
have
to build a crew members ability to recognize and respond to error producing
conditions in just 50 minutes of classroom and 1+30 - 2+00 in a simulator
LOFT
session each year.
To strategically move to an error management-based CRM system, we have to
be
specific about our expected responses to certain conditions and have a
grading
scale that measures the level of recognition and response. The three point
grading scale highlighted above is just a starting point for discussion. It
is
not the one that we use.
Once an airline manager defines a set of Management Skill (CRM)
expectations and
a grading scale, then the airline has the basis for building training and
assessment. Until an airline documents a clear set of CRM expectations and
describes how they will hold people accountable, I do not believe that the
management skill performance can be adequately measured nor will the
desired
change in the organization be as efficient as it could be.
The concept of an audit trail is central to AQP as well as any structured
instructional systems design effort. All audit trails start and end with
the
clear set of expectations highlighted in the task list. CRM is no
exception.
Before we will be able to assess a pilots management skill, we will need to
define the underpinnings of skill. One simple definition of skill is
"having an
appropriate response for a given set of circumstances." I think it is safe
to
say that our training programs are designed to build or reinforce a pilots
response to certain circumstances. This is just as true for technical
training
as it is for management skill (CRM) training.
Crew vs. Individual Assessment:
There has been a lot of ballyhoo about busting crews. Having worked with
the
AQP team at my airline to build the first generation of LOE assessment
tools, I
can say that collective crew grading is not desirable. Management skill
grading
will need to focus on an individuals performance in the crew setting. As a
manager, you have to ask yourself: "Am I going to use one gradesheet to
determine a collective crew grade?" Think back to college when you had to
do
group assignments. What if your entire semester grade was determined only
on
what your group did collectively? How would the professor ever know how
someone
within that group performed? Would it be possible that someone who had poor
skills was able to slip through because the strength of the other group
members
carried the person? What happens when that person who slipped through with
insufficient individual skill takes his or her credentials to the
workplace?
When you transfer this logic to pilots, it becomes clear that the
assessments
have to be on the individual level.
We might be able to make a case for team assessment for Air Force B-52
bomber
crew that always fly with the same team members. However, I would still
want to
know how each individual was performing within the team. In the first year
I
flew with the airline, I never flew with the same crew twice.
These CRM management skills are "individual skills in a crew setting."
There
are some responses that we expect from all crew members but there are some
unique responses that we expect from captains and first officers.
The Underpinnings of Skill:
If crew member skill is what we are trying to build - and skill is an
appropriate response to a given set of circumstances - perhaps we need to
identify the appropriate response we expect from crews when faced with
certain
circumstances. While it would be impossible to document all the possible
combinations of circumstances at a micro level, it is not unreasonable to
identify these conditions or circumstances at a macro level (Automation
changes
on arrivals). The safety data analysts have done a good job of building
taxonomies for the conditions that consistently lead to crew error. The
management skill expectations can be IF-THEN statements that are written at
the
macro level and apply to all phases of flight.
In fact, the CRM expectations probably should be worded with the following
sequence:
"Given Set of Circumstances"...."Appropriate Response"
I think that it is entirely appropriate for a manager to set expectations
"appropriate responses" to "given set of circumstances". In fact, many
airlines
expect the subordinate crew members to speak-up if they are uncertain and
will
hold them accountable if something happens and they did not speak-up. As
trainers, we can build the circumstances into our Line Oriented Training
"e.g.
ATC directed runway changes" and observe the crews response. For line
evaluations, we cannot control the conditions, but we can observe the crews
response to the conditions.
This is probably the first few steps in a "Journey of a Thousand Miles" but
I
hope that this helps to further focus the issues surrounding CRM metrics.
Best Regards,
Vince Mancuso