After you have read the reports please give us feedback on whether you find
these research products useful in your work. The hardcopies we mail out
have a questionnaire with a self-addressed envelope. Email comments are
also welcome.
Key
Facilitating LOS Debriefings:
A Training Manual
Lori K. McDonnell1, Kimberly K. Jobe1, R. Key Dismukes2
Ames Research Center
Summary
This manual is a practical guide to help airline instructors effectively
facilitate debriefings of Line Oriented Simulations (LOS). It is based on a
recently completed study of Line Oriented Flight Training (LOFT)
debriefings at several U.S. airlines. As a companion piece to the published
report of that study (LOFT Debriefings: An Analysis of Instructor
Techniques and Crew Participation, by R.K. Dismukes, K.K. Jobe, and L.K.
McDonnell, NASA Technical Memorandum 110442, March 1997), this manual
presents specific facilitation tools instructors can use to achieve
debriefing objectives. The approach of the manual is to be flexible so it
can be tailored to the individual needs of each airline. Part One clarifies
the purpose and objectives of facilitation in the LOS setting. Part Two
provides recommendations for clarifying roles and expectations and presents
a model for organizing discussion. Part Three suggests techniques for
eliciting active crew participation and in-depth analysis and evaluation.
Finally, in Part Four, these techniques are organized according to the
facilitation model. Examples of how to effectively use the techniques are
provided throughout, including strategies to try when the debriefing
objectives are not being fully achieved.
1 San Jose State University Foundation
2 NASA Ames Research Center
LOFT DEBRIEFINGS: An analysis of instructor techniques and crew participation
R. Key Dismukes ,Kimberly K. Jobe , and Lori K. McDonnell2
Summary
This study analyzes techniques instructors use to facilitate crew analysis
and evaluation of their LOFT performance. A rating instrument called the
Debriefing Assessment Battery (DAB) was developed which enables raters to
reliably assess instructor facilitation techniques and characterize crew
participation. Thirty-six debriefing sessions conducted at five U.S.
airlines were analyzed to determine the nature of instructor facilitation
and crew participation. Ratings obtained using the DAB corresponded closely
with descriptive measures of instructor and crew performance. The data
provide empirical evidence that facilitation can be an effective tool for
increasing the depth of crew participation and self-analysis of CRM
performance. Instructor facilitation skill varied dramatically, suggesting
a need for more concrete hands-on training in facilitation techniques.
Crews were responsive but fell short of actively leading their own
debriefings. Ways to improve debriefing effectiveness are suggested.
OVERVIEW
How much crews learn in Line-Oriented Flight Training (LOFT) and take back
to the line depends on the effectiveness of the debriefing that follows the
LOFT. The Crew Resource Management (CRM) literature and the Federal
Aviation Administration's (FAA) advisory circular (AC) 120-35C recommend
that in the debriefing instructors should facilitate self-discovery and
self-critique by the crew rather than lecture on what they did right and
wrong. Self discovery by the crew is believed to provide deeper learning
and better retention. Also, crews are more likely to enhance their
performance of CRM in line operations if they develop their ability to
analyze flight operations in terms of CRM and debrief themselves after line
flights.
In this study 36 LOFT debriefings conducted at five major U.S. airlines
were analyzed. Audiotape recordings of each session were made with the
permission of instructors and crews. The recordings were subsequently
deidentified, coded, and analyzed for more than 70 variables. The
Debriefing Assessment Battery was developed to systematically characterize
instructor effectiveness at facilitation and the nature of crew
participation in debriefings. The data indicate that the Debriefing
Assessment Battery is a reliable and valid instrument for assessing
instructors' skill in facilitation and for analyzing crew participation.
The battery was designed to be used by researchers, however a short form of
the battery that can be used by training departments to evaluate
debriefings in real time is currently being developed and evaluated.
Most instructors at all five airlines followed a similar general format for
debriefing. However, within each airline both instructors and crews varied
widely on many of the specific variables observed. There were also
substantial differences among airlines on several variables for both
instructors and crews, though most of these differences were not
statistically significant due to the large variability within each airline.
The debriefings lasted an average of 31 minutes, with a range of 8 to 82
minutes. However, 31 minutes may not allow adequate time for crews to
analyze their performance thoroughly or learn and practice the skills of
self-debriefing. This study provides no data on the optimal length for
debriefings, however an hour may be a useful rough target, with adjustments
for the needs of individual crews. This suggestion must, of course, be
considered in the context of other demands on instructors' time.
Most instructors appropriately emphasized crew performance in the LOFT and
achieved a balance between CRM and technical issues, although the range of
instructor scores on these variables was very large. Instructors typically
emphasized the things crews did well, but said little about things done not
so well and spent little time suggesting ways to improve. Likewise, crews'
discussions of their performance tended to be factual descriptions of
events and crew actions, with limited evaluation of performance or
discussion of ways to improve.
The content of the debriefings was driven almost exclusively by the
instructors; crew members rarely brought up topics on their own initiative.
Also, discussions revolved around the instructor, even when the instructor
succeeded in getting the crew to do most of the talking: there was little
back-and-forth discussion directly between crew members. The data indicate
that crews were responsive but not very proactive. This may be in part
because few of the instructors explicitly told crews they should take a
proactive role and perform their own analysis without depending on the
instructor to lead them step by step. It may also be that instructors
themselves either do not fully accept or understand the concept of crews
taking initiative and responsibility for the content of the debriefing.
On average, instructors asked a large number of questions to elicit crew
participation, directing their questions evenly among crew members.
Participation by captains and first officers was quite similar.
Participation by flight engineers (in three-person crews) was lower, but
this difference was marginally significant.
Most instructors appeared to be highly competent and conscientious in the
traditional roles of instructors, and most attempted to facilitate crew
participation to some degree; however, their success in facilitation ranged
from very good to poor. Instructors who were effective in facilitation
tended to use a combination of techniques, such as careful phrasing of
questions to encourage crew self-analysis, strategic silence, active
listening, and follow-up on crew-initiated topics. Probably more important
than the use of any particular technique is the instructor's underlying
focus on encouraging the crew to analyze for themselves the situations that
confronted them in the LOFT and how well they managed those situations.
Many instructors unwittingly did things counterproductive to their own
attempts to facilitate crew participation. In addition to failing to
explicitly state expectations for crew participation and allowing the
discussions to revolve around themselves instead of encouraging crew
interaction, some instructors failed to allow crew members enough time to
formulate thoughtful responses to questions. Also, some instructors engaged
in long monologues, gave their own evaluations before eliciting crew
self-evaluation, failed to push the crew to go beyond superficial
description of their actions, and/or failed to encourage crews to analyze
why things went well when they did.
The wide range of instructor effectiveness in facilitation indicates that
the airlines face an issue of standardization of this aspect of debriefing.
The distribution of facilitation scores was distinctly bimodal, with one
group of instructors scoring in the good to very good range and another
group of instructors scoring in the marginal range. Also, instructors who
did well in one aspect of facilitation typically did well in all aspects
(except stating expectations for crew participation), and those who did
poorly in one aspect tended to do poorly in all aspects. These data suggest
instructors' ability to use various techniques is determined at least in
part at the conceptual level: Do they grasp the underlying concept of
facilitation? Do they accept the concept? Is facilitation the type of
approach for which they have ability?
The CRM literature states that debriefings should be led by the crews
themselves, using the instructor as a resource. Our data suggest that this
goal, although worthwhile, is rather idealistic. Instructors become
discouraged when, after a brief and rather abstract course in facilitation,
they attempt to facilitate debriefings and discover that crews often do not
immediately respond. We suggest that it would be more effective to teach
instructors that facilitation should be adapted to the level at which the
particular crew is able to respond. Facilitation can be conducted at levels
ranging from high, which approaches the ideal of the debriefing being led
by the crew, to low, in which the instructor leads the crew substantially,
but in all cases debriefings should emphasize as much self-discovery by the
crew as possible.
Instructors are encouraged to attempt to facilitate at the highest level
possible for a particular crew. Realistically, however, most crews do not
yet have the skills and motivation needed to lead their own debriefings
without substantial assistance from the instructor. It may be possible to
change this situation over time if LOFT instructors consistently encourage
crews to take a proactive role in debriefing their own training.
Instructors sometimes mistakenly assume that using facilitation requires
giving up their role as teachers in the debriefing. On the contrary, good
facilitation in no way precludes the instructor from adding his or her own
perspective to the discussion or from teaching specific points about CRM
and technical issues as appropriate. Effective facilitators can integrate
their teaching points into a group discussion in which the crew members are
full participants.
The study provides empirical evidence that facilitation can be used to
substantially increase crew self-discovery and the depth of crew
participation. Instructors, however, need additional training in
facilitation. Facilitation training should emphasize hands-on practice in
which instructors encounter the kinds of obstacles they are likely to face
in actual debriefings. Initial training should be followed by mentoring by
senior instructors who are themselves expert facilitators. A training
manual that provides detailed suggestions for how to facilitate debriefings
is forthcoming as a companion to this technical report.