Many Thanks,
Paul
>----------
>From: Key Dismukes[SMTP:kdismukes_at_mail.arc.nasa.gov]
>Sent: Friday, April 11, 1997 20:48
>To: crm-devel_at_db.erau.edu
>Subject: LOFT debriefing study reports
>
>We have just received back from the printer copies of the two final reports
>from the FAA/NASA study of LOFT debriefings. Brief summaries of the
>reports are reproduced below. If you have already requested these reports,
>your name is on our mailing list and we will start mailing out the reports
>on monday (long list!). If you are not on our list and want the reports,
>please send me your mailing address and we will send them to you. I will
>also explore with Neil Krey the possibility of putting the reports or
>excerpts on this web site. (The technical report of the study is 92 pages
>with 23 tables and 3 figures. The facilitation training manual for
>instructors is 49 pages long. I don't know if the graphics will reproduce
>through the web).
>
>After you have read the reports please give us feedback on whether you find
>these research products useful in your work. The hardcopies we mail out
>have a questionnaire with a self-addressed envelope. Email comments are
>also welcome.
>
>Key
>
>
>Facilitating LOS Debriefings:
>A Training Manual
>
>Lori K. McDonnell1, Kimberly K. Jobe1, R. Key Dismukes2
>
>Ames Research Center
>
>
>Summary
>
>This manual is a practical guide to help airline instructors effectively
>facilitate debriefings of Line Oriented Simulations (LOS). It is based on a
>recently completed study of Line Oriented Flight Training (LOFT)
>debriefings at several U.S. airlines. As a companion piece to the published
>report of that study (LOFT Debriefings: An Analysis of Instructor
>Techniques and Crew Participation, by R.K. Dismukes, K.K. Jobe, and L.K.
>McDonnell, NASA Technical Memorandum 110442, March 1997), this manual
>presents specific facilitation tools instructors can use to achieve
>debriefing objectives. The approach of the manual is to be flexible so it
>can be tailored to the individual needs of each airline. Part One clarifies
>the purpose and objectives of facilitation in the LOS setting. Part Two
>provides recommendations for clarifying roles and expectations and presents
>a model for organizing discussion. Part Three suggests techniques for
>eliciting active crew participation and in-depth analysis and evaluation.
>Finally, in Part Four, these techniques are organized according to the
>facilitation model. Examples of how to effectively use the techniques are
>provided throughout, including strategies to try when the debriefing
>objectives are not being fully achieved.
>
>1 San Jose State University Foundation
>2 NASA Ames Research Center
>
>
>
>LOFT DEBRIEFINGS: An analysis of instructor techniques and crew
>participation
>R. Key Dismukes ,Kimberly K. Jobe , and Lori K. McDonnell2
>Summary
>This study analyzes techniques instructors use to facilitate crew analysis
>and evaluation of their LOFT performance. A rating instrument called the
>Debriefing Assessment Battery (DAB) was developed which enables raters to
>reliably assess instructor facilitation techniques and characterize crew
>participation. Thirty-six debriefing sessions conducted at five U.S.
>airlines were analyzed to determine the nature of instructor facilitation
>and crew participation. Ratings obtained using the DAB corresponded closely
>with descriptive measures of instructor and crew performance. The data
>provide empirical evidence that facilitation can be an effective tool for
>increasing the depth of crew participation and self-analysis of CRM
>performance. Instructor facilitation skill varied dramatically, suggesting
>a need for more concrete hands-on training in facilitation techniques.
>Crews were responsive but fell short of actively leading their own
>debriefings. Ways to improve debriefing effectiveness are suggested.
>OVERVIEW
>How much crews learn in Line-Oriented Flight Training (LOFT) and take back
>to the line depends on the effectiveness of the debriefing that follows the
>LOFT. The Crew Resource Management (CRM) literature and the Federal
>Aviation Administration's (FAA) advisory circular (AC) 120-35C recommend
>that in the debriefing instructors should facilitate self-discovery and
>self-critique by the crew rather than lecture on what they did right and
>wrong. Self discovery by the crew is believed to provide deeper learning
>and better retention. Also, crews are more likely to enhance their
>performance of CRM in line operations if they develop their ability to
>analyze flight operations in terms of CRM and debrief themselves after line
>flights.
>In this study 36 LOFT debriefings conducted at five major U.S. airlines
>were analyzed. Audiotape recordings of each session were made with the
>permission of instructors and crews. The recordings were subsequently
>deidentified, coded, and analyzed for more than 70 variables. The
>Debriefing Assessment Battery was developed to systematically characterize
>instructor effectiveness at facilitation and the nature of crew
>participation in debriefings. The data indicate that the Debriefing
>Assessment Battery is a reliable and valid instrument for assessing
>instructors' skill in facilitation and for analyzing crew participation.
>The battery was designed to be used by researchers, however a short form of
>the battery that can be used by training departments to evaluate
>debriefings in real time is currently being developed and evaluated.
>Most instructors at all five airlines followed a similar general format for
>debriefing. However, within each airline both instructors and crews varied
>widely on many of the specific variables observed. There were also
>substantial differences among airlines on several variables for both
>instructors and crews, though most of these differences were not
>statistically significant due to the large variability within each airline.
>The debriefings lasted an average of 31 minutes, with a range of 8 to 82
>minutes. However, 31 minutes may not allow adequate time for crews to
>analyze their performance thoroughly or learn and practice the skills of
>self-debriefing. This study provides no data on the optimal length for
>debriefings, however an hour may be a useful rough target, with adjustments
>for the needs of individual crews. This suggestion must, of course, be
>considered in the context of other demands on instructors' time.
>Most instructors appropriately emphasized crew performance in the LOFT and
>achieved a balance between CRM and technical issues, although the range of
>instructor scores on these variables was very large. Instructors typically
>emphasized the things crews did well, but said little about things done not
>so well and spent little time suggesting ways to improve. Likewise, crews'
>discussions of their performance tended to be factual descriptions of
>events and crew actions, with limited evaluation of performance or
>discussion of ways to improve.
>The content of the debriefings was driven almost exclusively by the
>instructors; crew members rarely brought up topics on their own initiative.
>Also, discussions revolved around the instructor, even when the instructor
>succeeded in getting the crew to do most of the talking: there was little
>back-and-forth discussion directly between crew members. The data indicate
>that crews were responsive but not very proactive. This may be in part
>because few of the instructors explicitly told crews they should take a
>proactive role and perform their own analysis without depending on the
>instructor to lead them step by step. It may also be that instructors
>themselves either do not fully accept or understand the concept of crews
>taking initiative and responsibility for the content of the debriefing.
>On average, instructors asked a large number of questions to elicit crew
>participation, directing their questions evenly among crew members.
>Participation by captains and first officers was quite similar.
>Participation by flight engineers (in three-person crews) was lower, but
>this difference was marginally significant.
>Most instructors appeared to be highly competent and conscientious in the
>traditional roles of instructors, and most attempted to facilitate crew
>participation to some degree; however, their success in facilitation ranged
>from very good to poor. Instructors who were effective in facilitation
>tended to use a combination of techniques, such as careful phrasing of
>questions to encourage crew self-analysis, strategic silence, active
>listening, and follow-up on crew-initiated topics. Probably more important
>than the use of any particular technique is the instructor's underlying
>focus on encouraging the crew to analyze for themselves the situations that
>confronted them in the LOFT and how well they managed those situations.
>Many instructors unwittingly did things counterproductive to their own
>attempts to facilitate crew participation. In addition to failing to
>explicitly state expectations for crew participation and allowing the
>discussions to revolve around themselves instead of encouraging crew
>interaction, some instructors failed to allow crew members enough time to
>formulate thoughtful responses to questions. Also, some instructors engaged
>in long monologues, gave their own evaluations before eliciting crew
>self-evaluation, failed to push the crew to go beyond superficial
>description of their actions, and/or failed to encourage crews to analyze
>why things went well when they did.
>The wide range of instructor effectiveness in facilitation indicates that
>the airlines face an issue of standardization of this aspect of debriefing.
>The distribution of facilitation scores was distinctly bimodal, with one
>group of instructors scoring in the good to very good range and another
>group of instructors scoring in the marginal range. Also, instructors who
>did well in one aspect of facilitation typically did well in all aspects
>(except stating expectations for crew participation), and those who did
>poorly in one aspect tended to do poorly in all aspects. These data suggest
>instructors' ability to use various techniques is determined at least in
>part at the conceptual level: Do they grasp the underlying concept of
>facilitation? Do they accept the concept? Is facilitation the type of
>approach for which they have ability?
>The CRM literature states that debriefings should be led by the crews
>themselves, using the instructor as a resource. Our data suggest that this
>goal, although worthwhile, is rather idealistic. Instructors become
>discouraged when, after a brief and rather abstract course in facilitation,
>they attempt to facilitate debriefings and discover that crews often do not
>immediately respond. We suggest that it would be more effective to teach
>instructors that facilitation should be adapted to the level at which the
>particular crew is able to respond. Facilitation can be conducted at levels
>ranging from high, which approaches the ideal of the debriefing being led
>by the crew, to low, in which the instructor leads the crew substantially,
>but in all cases debriefings should emphasize as much self-discovery by the
>crew as possible.
>Instructors are encouraged to attempt to facilitate at the highest level
>possible for a particular crew. Realistically, however, most crews do not
>yet have the skills and motivation needed to lead their own debriefings
>without substantial assistance from the instructor. It may be possible to
>change this situation over time if LOFT instructors consistently encourage
>crews to take a proactive role in debriefing their own training.
>Instructors sometimes mistakenly assume that using facilitation requires
>giving up their role as teachers in the debriefing. On the contrary, good
>facilitation in no way precludes the instructor from adding his or her own
>perspective to the discussion or from teaching specific points about CRM
>and technical issues as appropriate. Effective facilitators can integrate
>their teaching points into a group discussion in which the crew members are
>full participants.
>The study provides empirical evidence that facilitation can be used to
>substantially increase crew self-discovery and the depth of crew
>participation. Instructors, however, need additional training in
>facilitation. Facilitation training should emphasize hands-on practice in
>which instructors encounter the kinds of obstacles they are likely to face
>in actual debriefings. Initial training should be followed by mentoring by
>senior instructors who are themselves expert facilitators. A training
>manual that provides detailed suggestions for how to facilitate debriefings
>is forthcoming as a companion to this technical report.
>
>
>
>