On Fri, 11 Apr 1997, Key Dismukes wrote:
> We have just received back from the printer copies of the two final reports
> from the FAA/NASA study of LOFT debriefings. Brief summaries of the
> reports are reproduced below. If you have already requested these reports,
> your name is on our mailing list and we will start mailing out the reports
> on monday (long list!). If you are not on our list and want the reports,
> please send me your mailing address and we will send them to you. I will
> also explore with Neil Krey the possibility of putting the reports or
> excerpts on this web site. (The technical report of the study is 92 pages
> with 23 tables and 3 figures. The facilitation training manual for
> instructors is 49 pages long. I don't know if the graphics will reproduce
> through the web).
>
> After you have read the reports please give us feedback on whether you find
> these research products useful in your work. The hardcopies we mail out
> have a questionnaire with a self-addressed envelope. Email comments are
> also welcome.
>
> Key
>
>
> Facilitating LOS Debriefings:
> A Training Manual
>
> Lori K. McDonnell1, Kimberly K. Jobe1, R. Key Dismukes2
>
> Ames Research Center
>
>
> Summary
>
> This manual is a practical guide to help airline instructors effectively
> facilitate debriefings of Line Oriented Simulations (LOS). It is based on a
> recently completed study of Line Oriented Flight Training (LOFT)
> debriefings at several U.S. airlines. As a companion piece to the published
> report of that study (LOFT Debriefings: An Analysis of Instructor
> Techniques and Crew Participation, by R.K. Dismukes, K.K. Jobe, and L.K.
> McDonnell, NASA Technical Memorandum 110442, March 1997), this manual
> presents specific facilitation tools instructors can use to achieve
> debriefing objectives. The approach of the manual is to be flexible so it
> can be tailored to the individual needs of each airline. Part One clarifies
> the purpose and objectives of facilitation in the LOS setting. Part Two
> provides recommendations for clarifying roles and expectations and presents
> a model for organizing discussion. Part Three suggests techniques for
> eliciting active crew participation and in-depth analysis and evaluation.
> Finally, in Part Four, these techniques are organized according to the
> facilitation model. Examples of how to effectively use the techniques are
> provided throughout, including strategies to try when the debriefing
> objectives are not being fully achieved.
>
> 1 San Jose State University Foundation
> 2 NASA Ames Research Center
>
>
>
> LOFT DEBRIEFINGS: An analysis of instructor techniques and crew participation
> R. Key Dismukes ,Kimberly K. Jobe , and Lori K. McDonnell2
> Summary
> This study analyzes techniques instructors use to facilitate crew analysis
> and evaluation of their LOFT performance. A rating instrument called the
> Debriefing Assessment Battery (DAB) was developed which enables raters to
> reliably assess instructor facilitation techniques and characterize crew
> participation. Thirty-six debriefing sessions conducted at five U.S.
> airlines were analyzed to determine the nature of instructor facilitation
> and crew participation. Ratings obtained using the DAB corresponded closely
> with descriptive measures of instructor and crew performance. The data
> provide empirical evidence that facilitation can be an effective tool for
> increasing the depth of crew participation and self-analysis of CRM
> performance. Instructor facilitation skill varied dramatically, suggesting
> a need for more concrete hands-on training in facilitation techniques.
> Crews were responsive but fell short of actively leading their own
> debriefings. Ways to improve debriefing effectiveness are suggested.
> OVERVIEW
> How much crews learn in Line-Oriented Flight Training (LOFT) and take back
> to the line depends on the effectiveness of the debriefing that follows the
> LOFT. The Crew Resource Management (CRM) literature and the Federal
> Aviation Administration's (FAA) advisory circular (AC) 120-35C recommend
> that in the debriefing instructors should facilitate self-discovery and
> self-critique by the crew rather than lecture on what they did right and
> wrong. Self discovery by the crew is believed to provide deeper learning
> and better retention. Also, crews are more likely to enhance their
> performance of CRM in line operations if they develop their ability to
> analyze flight operations in terms of CRM and debrief themselves after line
> flights.
> In this study 36 LOFT debriefings conducted at five major U.S. airlines
> were analyzed. Audiotape recordings of each session were made with the
> permission of instructors and crews. The recordings were subsequently
> deidentified, coded, and analyzed for more than 70 variables. The
> Debriefing Assessment Battery was developed to systematically characterize
> instructor effectiveness at facilitation and the nature of crew
> participation in debriefings. The data indicate that the Debriefing
> Assessment Battery is a reliable and valid instrument for assessing
> instructors' skill in facilitation and for analyzing crew participation.
> The battery was designed to be used by researchers, however a short form of
> the battery that can be used by training departments to evaluate
> debriefings in real time is currently being developed and evaluated.
> Most instructors at all five airlines followed a similar general format for
> debriefing. However, within each airline both instructors and crews varied
> widely on many of the specific variables observed. There were also
> substantial differences among airlines on several variables for both
> instructors and crews, though most of these differences were not
> statistically significant due to the large variability within each airline.
> The debriefings lasted an average of 31 minutes, with a range of 8 to 82
> minutes. However, 31 minutes may not allow adequate time for crews to
> analyze their performance thoroughly or learn and practice the skills of
> self-debriefing. This study provides no data on the optimal length for
> debriefings, however an hour may be a useful rough target, with adjustments
> for the needs of individual crews. This suggestion must, of course, be
> considered in the context of other demands on instructors' time.
> Most instructors appropriately emphasized crew performance in the LOFT and
> achieved a balance between CRM and technical issues, although the range of
> instructor scores on these variables was very large. Instructors typically
> emphasized the things crews did well, but said little about things done not
> so well and spent little time suggesting ways to improve. Likewise, crews'
> discussions of their performance tended to be factual descriptions of
> events and crew actions, with limited evaluation of performance or
> discussion of ways to improve.
> The content of the debriefings was driven almost exclusively by the
> instructors; crew members rarely brought up topics on their own initiative.
> Also, discussions revolved around the instructor, even when the instructor
> succeeded in getting the crew to do most of the talking: there was little
> back-and-forth discussion directly between crew members. The data indicate
> that crews were responsive but not very proactive. This may be in part
> because few of the instructors explicitly told crews they should take a
> proactive role and perform their own analysis without depending on the
> instructor to lead them step by step. It may also be that instructors
> themselves either do not fully accept or understand the concept of crews
> taking initiative and responsibility for the content of the debriefing.
> On average, instructors asked a large number of questions to elicit crew
> participation, directing their questions evenly among crew members.
> Participation by captains and first officers was quite similar.
> Participation by flight engineers (in three-person crews) was lower, but
> this difference was marginally significant.
> Most instructors appeared to be highly competent and conscientious in the
> traditional roles of instructors, and most attempted to facilitate crew
> participation to some degree; however, their success in facilitation ranged
> from very good to poor. Instructors who were effective in facilitation
> tended to use a combination of techniques, such as careful phrasing of
> questions to encourage crew self-analysis, strategic silence, active
> listening, and follow-up on crew-initiated topics. Probably more important
> than the use of any particular technique is the instructor's underlying
> focus on encouraging the crew to analyze for themselves the situations that
> confronted them in the LOFT and how well they managed those situations.
> Many instructors unwittingly did things counterproductive to their own
> attempts to facilitate crew participation. In addition to failing to
> explicitly state expectations for crew participation and allowing the
> discussions to revolve around themselves instead of encouraging crew
> interaction, some instructors failed to allow crew members enough time to
> formulate thoughtful responses to questions. Also, some instructors engaged
> in long monologues, gave their own evaluations before eliciting crew
> self-evaluation, failed to push the crew to go beyond superficial
> description of their actions, and/or failed to encourage crews to analyze
> why things went well when they did.
> The wide range of instructor effectiveness in facilitation indicates that
> the airlines face an issue of standardization of this aspect of debriefing.
> The distribution of facilitation scores was distinctly bimodal, with one
> group of instructors scoring in the good to very good range and another
> group of instructors scoring in the marginal range. Also, instructors who
> did well in one aspect of facilitation typically did well in all aspects
> (except stating expectations for crew participation), and those who did
> poorly in one aspect tended to do poorly in all aspects. These data suggest
> instructors' ability to use various techniques is determined at least in
> part at the conceptual level: Do they grasp the underlying concept of
> facilitation? Do they accept the concept? Is facilitation the type of
> approach for which they have ability?
> The CRM literature states that debriefings should be led by the crews
> themselves, using the instructor as a resource. Our data suggest that this
> goal, although worthwhile, is rather idealistic. Instructors become
> discouraged when, after a brief and rather abstract course in facilitation,
> they attempt to facilitate debriefings and discover that crews often do not
> immediately respond. We suggest that it would be more effective to teach
> instructors that facilitation should be adapted to the level at which the
> particular crew is able to respond. Facilitation can be conducted at levels
> ranging from high, which approaches the ideal of the debriefing being led
> by the crew, to low, in which the instructor leads the crew substantially,
> but in all cases debriefings should emphasize as much self-discovery by the
> crew as possible.
> Instructors are encouraged to attempt to facilitate at the highest level
> possible for a particular crew. Realistically, however, most crews do not
> yet have the skills and motivation needed to lead their own debriefings
> without substantial assistance from the instructor. It may be possible to
> change this situation over time if LOFT instructors consistently encourage
> crews to take a proactive role in debriefing their own training.
> Instructors sometimes mistakenly assume that using facilitation requires
> giving up their role as teachers in the debriefing. On the contrary, good
> facilitation in no way precludes the instructor from adding his or her own
> perspective to the discussion or from teaching specific points about CRM
> and technical issues as appropriate. Effective facilitators can integrate
> their teaching points into a group discussion in which the crew members are
> full participants.
> The study provides empirical evidence that facilitation can be used to
> substantially increase crew self-discovery and the depth of crew
> participation. Instructors, however, need additional training in
> facilitation. Facilitation training should emphasize hands-on practice in
> which instructors encounter the kinds of obstacles they are likely to face
> in actual debriefings. Initial training should be followed by mentoring by
> senior instructors who are themselves expert facilitators. A training
> manual that provides detailed suggestions for how to facilitate debriefings
> is forthcoming as a companion to this technical report.
>
>
>
>