January 26, 1996, at the NASA Ames Research Center
February 22, 1996 at Hughes Training Inc.
To identify opportunities for assembling and focusing industry and research resources to create, store, and disseminate CRM and Human Factors products that meet pressing operational needs To identify the structures and processes that NASA could create to support industry CRM and Human Factors product needs. To identify the operational community's role in a joint NASA / Industry developers group To determine if a development group structured under NASA could adequately meet the needs of the industry To identify critical success factors for an industry development group To identify the goals and mission of a joint industry/NASA development group To review and revise draft protocols for the industry development group
What would the mission of a joint industry / NASA development group be? What are some of the possible roles for NASA support of an industry developers group? What are some of the current gaps between researchers and practitioners? How can an industry development group be used to bridge the gap between HF theory and practice? Who should identify the priority of projects, products, services, and support that the industry developers group would work on? How would the projects and priorities be established? How would the group identify goals? Who would be potential contributors to development efforts? (Academia "UAA", FAA, research, non-profit development groups, private companies, etc.) What would the financial requirements of a joint industry development group be? Who would provide financial support sources for a joint industry development group? (NASA, FAA, ???) What communication mediums would be used for developer coordination? What would NASA's role be in supporting developer communications and coordination? Who would be the customers for the products that this group creates? What would be the financial benefit to an airline that commits people and resources to a development team? Would an industry development group be more effective as an independent organization like Battelle? How would we structure an industry development group under NASA that is largely controlled and shaped by industry? Is NASA interested in building and maintaining a compilation of CRM and human factors products for industry? How would NASA financially support the development of CRM and human factors products for general industry use? How would the joint NASA/Industry group disseminate information and resources ?(conferences, online media, mail, newsletters, etc.) How would we identify the programs and products that NASA would like to make available to the entire industry? How would we determine the appropriate balance of personnel on a tiger team ? (operators, researchers, instructional systems designers, media specialists, etc.)
Catalog / index the existing/available industry training courseware Catalog / index the existing HF products (CIR, Accident & Incident investigation, Etc.) Identify gaps in the existing body of available products, courseware, and other resources Identify programs that are outdated
Administration and Support Topics: The structures and processes for leadership of such a group (steering committees?) Administrative (day to day) leadership of such a group The administrative support structure of an industry developers group Membership of the development group Membership of development teams Identify examples of existing developers consortiums Identify existing models where the research community is very close to the practitioners. Identify factors that make such an arrangement work.
Protocols for the Industry Resource Management and Human Factors Program Development Task Force:
To improve the quality and accessibility of CRM and Human Factors programs and resources internationally.
To maintain a product development-orientation in all of our efforts To continually translate the emerging body of RM and HF science and literature into usable products for the aviation industry To have a bi-annual conference to present development efforts To seek and recruit the top industry experts for the development challenges and topics. To continually refine the art and science of instructional systems design as it relates to CRM and Human Factors To encourage the development of RM/HF training programs as opposed to briefing programs To provide products that are not airline specific but can be easily adapted to specific cultures, organizations, and environments To develop, maintain and continuously improve our communication and coordination channels within the groups and throughout the industry on RM/HF topics To provide a focal point for assembling industry talent to address pressing RM and HF development challenges To provide a focal point and forum for Academia and the Research Community to identify pressing CRM and Human Factors concerns. To provide a resource center of publicly accessible CRM and Human Factors materials To provide a focal point and communication medium for CRM developer industry concerns (a top ten hit list) To work in concert with existing CRM development entities and organizations to collectively improve the quality and accessibility of industry CRM resources To promote the availability of task force CRM/HF programs and resources. To continually solicit the contributions and perspectives of industry talent To support the use of structured program development efforts that represent sound training development science
Individual RM/Human Factors program developers (airline, military, general aviation, corporate) from various disciplines (maintenance, dispatch, ramp, ATC, etc.) The research community The academic community Civilian and Military flight schools All non flying personnel such as (maintenance, dispatch, ramp, ATC, etc.) General aviation pilots Corporate pilots The FAA
Internal Customers for the Steering Committee Committee members Contributing CRM program developers Contributing academicians and researchers
Steering Committee: A steering committee will be selected to guide development efforts. The steering committee will be comprised of no more than 7 people. No fewer than 4 will be active operational industry managers. Requisites for steering committee membership will include a current position where their job includes the active development or management of human factors and or resource management programs. Steering committee tasks will include: developing a project prioritization, approving the selection of core development teams, (I could use some ideas here....) Develop suggested approaches and protocols for product development
Core Development Groups: Core development groups will be no larger than 8 core members. No fewer than four will be active operational human factors / resource management developers or managers. Core working groups will be charged with the authority and responsibility to assemble the required talent. Each group will designate a chairperson. This person will be the focal point for that group's development efforts. Working group members are expected to meet group commitments and to contribute regularly. If a member is not contributing, he or she will be asked to assume a "friend of the working group" status and a new group member will be selected.
Friends of the Core Development Group: Development groups can have as many "Friends of the Development Group" as they desire. The friends of the group can participate in the development effort during open forums of the core group. As development progresses, development groups will have open forums where friends of the group will be invited to make contributions and comments.
Training courseware Audit tools Measurement tools Literature searches and bibliographies for certain CRM and HF topics SPOT scenarios with video tapes, briefing guides, and debriefing guides Developers guides that highlight various training methods and examples A compilation of metrics for human performance Customized data searches and analysis for certain topics General data search and analysis tools and methodologies
Maintaining an ongoing list of prioritized user needs and putting voice to them (survey / data collection tools for developing and maintaining a list of industry development priorities) Provide a forum for highlighting the possibilities (technology applications, CDROM, virtual reality, etc.) Provide a forum for training industry developers who are just beginning their development efforts Identify how NASA might support industry development training workshops Identify the steps we would want to take to begin formalizing a development group (start with one project? set an agenda that leads to a developers conference)
NASA sponsored Bi-annual conference for highlighting development products Free access and distribution through the internet and other online services Email and U.S. mail requests for information and products
On behalf of the entire industry developers group, I would like to extend sincere thanks to the folks at Hughes Training Inc. who provided facilities, lunch, and the time of their talented people for our meeting. Since these efforts are not directly funded by anyone (and at the moment the developers prefer to keep it that way to avoid politics and self-interest), their support was greatly appreciated.
Vince Mancuso, Pete Wolfe, Tom Chidister, Neil Krey, Mike Rinehart, Gary Van Hartogh, Kay Richter, Bruce Tessmer, Tom Heinzer, Jay Norelius, Lloyd Murray, David Wilson, Jay Norelius
These draft meeting minutes will be distributed via email or U.S. mail to all members in attendance. Everyone will be encouraged to make comments. The minutes will be finalized at the next industry developers meeting. While a specific date for the next meeting has not yet been established, it will likely happen prior to the AQP meeting in May.
After the opening remarks by our gracious hosts at Hughes Training Inc, each of the members in attendance introduced themselves and highlighted their HF/CRM function at their airline. Vince Mancuso then gave an overview of the days agenda.
The first item on the agenda was a historical overview of how the group formed along with a brief discussion of where the group is heading. This meeting was a derivative of earlier efforts to bring industry developers together. Existing forums were not meeting the needs of the developers. Some of these challenges were political issues and some were simple matters of inefficiency, resources, or clear direction. The bottom line was that industry developers had products they had to develop at their respective airlines and there were significant resources that were either unavailable or unused that could help them. The group largely grew out of the frustration of seeing a better way to develop HF/CRM products.
It is safe to say that the establishment of an online meeting point through email has fundamentally enhanced the work of this group. In fact, if a developer does not have access to online libraries and discussion forums like Compuserve's HF/CRM Developers Group on Aviation Week Group Forum, they are at a significant disadvantage. It is highly recommended that all the individuals who would like to be involved with the developers group effort gain access to email. At the moment, Compuserve appears to be the best service for gaining access to HF/CRM related discussions and materials. McGraw Hill and Aviation Week Group have dedicated online space for our efforts. This is not a paid advertisement for Compuserve. It just happens to be where most of the online activity in HF/CRM development is happening. Even if individuals are on other services, they will be able to correspond with folks on Compuserve via email. Email is quickly becoming a necessity and may soon be a requisite for participation in this developers group.
In January 1996, a group of 5 industry developers met with Dr. Mark Rosekind and other researchers at Ames to identify some of the existing gaps between research products and operator needs. We discussed the mission of the industry developers group and discussed ways that NASA could support or work with this group to enhance industry HF/CRM product development. Through these discussions, the group identified two common needs that could be used as "proof of concept" applications. These included the common need to educate industry developers on how to use online and Internet resources and the need for a Situational Awareness (SA) Management module. For the first industry developer need, NASA agreed to create an educational guide and a video along with a listing of HF/CRM sites. Steve Casner, Kevin Gregory, and Greg Pisanich have formed a group to create a technical manual, the educational materials, and the online site. The group was shooting for the May AQP meeting as the roll out of the first version of the educational module. It is envisioned that the listing of HF/CRM sites would be a living document that would be periodically updated and kept online. The NASA folks also suggested that they could also create a site on their Internet server that would provide a focal point for "Tiger Team" product development efforts.
The next industry need highlighted was an education and training module for SA Management. The group agreed that the creation of a SA Management product would be a good context within which to refine the "tiger team" concept and to see if the industry development group was feasible or practical. Two components of the SA Management were identified. The first component is a one hour training module and the second component is an instructor module for SA Management. Vince Mancuso agreed to take the coordination lead on the one hour module effort with a target roll out of August. Neil Krey and Pete Wolfe offered to take the coordinating lead on the component labeled "Teaching & Observing/Evaluating SA Management -- for Instructors and Check Airmen," with a target roll out of September. Tom Houle agreed to coordinate the overall effort and provide integration between efforts.
The group then discussed how industry HF/CRM program developers have both a common need as well as a common set of industry and government resources from which to draw. The spirit of cooperation and the common interest in improving safety have created a climate of open exchange among industry HF/CRM program developers.
Each HF/CRM developer must translate good science and principles into usable programs and products at their respective airlines. An airline's HF and CRM program and product needs are relatively similar and somewhat predictable regardless of the airline. In fact, with the establishment of the FARs requiring CRM, there is now a regulatory baseline to which all airlines must meet or exceed. One of the common denominators that connects the industry program developers in this group is the need to translate HF/CRM literature, research findings, incident reports, and accident reports, and other human factors related resources into operationally usable products.
There is a tremendous wealth of talent and resources that often go unused by industry HF/CRM program developers. The fact that they go unused is a tremendous source of frustration for many industry developers. There are several problems that create barriers to using the available talent and resources. Some of the large airlines with significant resources have found ways to circumvent the barriers. The individuals or airlines that have the resources or personnel to get around the barriers are the "HF/CRM haves" and the ones that do not have the resources or personnel are the "HF/CRM have nots". If the program developers have a large personal or company human factors libraries, they are HF/CRM information wealthy. If they do not, they are HF/CRM information poor.
There is a tremendous wealth of talent and information that is often underutilized because the bridge between the industry developers and the resources is underdeveloped. It is important to note that resources include people, information and products.
Research transfer, when it does occur in today's aviation human factors environment, usually happens in one of two ways; either the researcher works with the developers at an individual airline to translate resources and findings into products or the carrier has employees who do the translation in-house. While every airline may be able to benefit from a particular HF researcher's work, one cannot reasonably expect these researchers to visit every airline to translate their findings into products. To expect that every operational air carrier will staff positions to translate research into operationally relevant products is also not realistic.
The members of the development group believe that the solution lies in creating a product-oriented focal point for common development efforts and pooling resources to do the value-added work. After the value-added work is done, the developers can bring the products back to their airlines as well as provide the value-added work to the industry. This saves everyone except for the product "Tiger Team" from having to do the value-added work. If the individual developers have a common development need and were going to have to do the work anyway, there is nothing lost on the part of the "Tiger Team" participants. For the aviation organizations that would not or could not have done the value-added work, they can obtain a valuable product that can be easily translated at their organization.
There are some outstanding examples of research findings and reports that are easily assimilated into airline products and programs. Reports from the Aviation Safety Reporting System (ASRS) and the Flight Safety Foundation (FSF) provide excellent examples of value-added resources that are easily translatable into operational products. The group agreed that generally speaking, the current level of products from the aviation human factors research community are often very esoteric and difficult to translate into operationally useful products. These esoteric reports often require significant translation and adaptation to be even remotely usable in an operational program. This value-added work must be done by someone. In many cases (particularly at airlines with insufficient resources to hire full time human factors or training program development specialists) the translation from research to practice is never done.
The HF/CRM program developers in attendance believe that the gap discussed above could be closed through a few different fixes.
Fix #1: Recognize that a program is a combination of content, structure, methods, and devices. Further recognize that the researchers are excellent sources for content to be usable but the content must be combined with structure, methods, and devices. Recognize that the bulk of the work that industry developer's do is not in identifying content but in assimilating content into operationally usable structure, methods, and devices. Require the researchers to use structure, method, and device Subject Matter Experts (SMEs) to add-value to their content and therefore deliver a product closer to the requirements of industry.
Fix #2: Identify unmet industry development needs. Once identified, pool industry developers, researchers, SME's and resources to focus on creating a product that meets the industry needs. Use the pooled resources to do the detailed labor-intensive translation and value-added work once rather than the current model of partial or no translation. Provide an end-product that, while it is not a turn-key or one-size-fits-all product, requires minimal adaptation by a program developer to be placed into operational use.
Fix #3: Do both #1 and #2 concurrently.
One fix is to require that research grants be made in such a way that researchers are required to do the value-added work to bridge the gap between theory and practical application within the context of their research grant. The grants would require researchers to work with airline program development subject matter experts to do the value-added work to create operationally relevant and usable products. This does not mean that the end result must be a turn-key program or one-size-fits-all product that could be used at every airline. The group recognizes that each airline must customize any HF or CRM product to fit their airline requirements. It does mean, however, that findings are put into context and an operationally usable framework. A basic prototype is one example. The products provided by NASA's Fatigue Countermeasures Program are a perfect example of value-added processing that yields a product that can be easily translated into operational use.
The discussion of fix #2 is best embodied in the draft mission statement and goals of the industry developers group. While the group discussed the draft mission statement and goals a little later in the day, it appears to fit best here in the minutes.
Vince Mancuso presented a basic mission statement and goals that he drafted prior to the meeting. Using this early draft as a strawman, the group reworked the draft verbiage. Listed in appendix A of these minutes is a synopsis of the mission statement, goals and protocols as they currently exist. With the limited time the developer's group had in Arlington, they decided that it would be best to edit, add, delete and wordsmith the goals and the protocols individually between meetings. It is important to note that the mission statement, goals, and protocols are still drafts. Participants are welcome to make inputs or suggestions. It is envisioned that these protocols will be put into a more final form when the group meets before the AQP meeting in May (See Appendix A for the Draft Mission Statement, Goals, and Protocols).
The next item on the agenda was to put voice to a development process model that this group could use. Neil Krey highlighted an overall development process. (Please forgive the text only representation of this process loop).
Research (add value) Industry Group Deliverables (distribution) Customized Airline Products (new needs?) ---- Loop back to Research
The group was also interested in identify how far industry group deliverables would be developed before the airline would want to start their own customization. It will be important to identify where in the development process each airline would want to depart the "Tiger Team" development effort and begin their customization. This part of the meeting provided some particularly valuable insights into developer needs and requirements. Using the backdrop of the Situational Awareness Management prototype project, we collectively identified a training program development process that would lead us toward an operationally usable product. It is important to recognize that the development process for non-training products like human factors assessment tools, or incident investigation protocols, for example, would be different. The process outlined below is specifically tailored to training program development.
One question that emerged from our identification of the development process was "At what point in the group development process will airlines want to depart and begin their customization?" Since the whole purpose of the industry development group is to provide value to HF/CRM developers, it is important to identify the point in the development process at which they no longer find value. It is not reasonable to assume that individual airline developers who's group development efforts are being supported by their respective airlines would participate in a group development effort beyond the point where their airline receives no additional value. The attendees agreed that this "jump-off point" will probably be different for every airline. The major airlines in attendance agreed that the jump-off point will likely be just before the development of the ISD framework.
As the tiger team progresses through the various steps of development, they will document their work and their progress. One of the distinct orientations of the tiger team should be to leave an unmistakable paper trail of their work. This assumes more administrative overhead but it is important for several reasons. First, individuals who were not involved in the development effort will be able to see how the product was derived. Second, if they end product does not suit their needs, they can trace the development back to the point where they want to start customizing their own product and begin at that point. Third, as new findings in the human factors field are derived, the industry may find it necessary to revisit a previous development effort and update it. An unmistakable paper trail during the original development effort will allow the subsequent developers to have a baseline for refinement.
It is expected that the airlines with significant development resources will likely begin the customization process earlier in the development cycle than airlines without significant development resources. Background work such as bibliographies, reference lists, notes on how the working theory was derived, end-user focus group minutes and findings, incident analysis work are examples of documentation the program developers might want to use. Since incident reports provide an excellent training tool that can be built directly into the courseware, the compilations of relevant incident data are another resource that the "Tiger Team" will likely produce. If the airline has an "in-house" incident reporting system, they could supplement or complement these reports with airline-specific examples.
The development of an index of industry HF/CRM products and projects To properly identify the gaps that exist in the industry's current CRM and HF products, it will be vitally important to catalog and maintain who is doing what. The members in attendance also pointed out that there are several immediate benefits from a compendium of existing programs. The first is that a HF/CRM program developer can identify airlines that already have a particular program in place or under development and contact that airline or individual directly. The meeting attendees identified the following three questions as a guide for developing the "existing HF/CRM products" chapter.
It became apparent in our discussions that there would be several different areas or breakdowns that this index could cover. The breakdowns quickly started looking like chapters, so the members in attendance basically wrote an outline by chapter and some identified some of the content they would find useful in such a document.
Introduction Philosophy Purpose
Chapter 1 List of products Courseware, protocols, guides For Training Course Descriptions - How long - Target audience - Who teaches it - Methods - Instructional Strategies - Media - Lessons learned
Chapter 2 List of projects in development Title Objective Target Audience How far along in development How is developing it
Chapter 3 Index of products organized by behavioral categories
Chapter 4 HF/CRM Organizational Structures in place at airlines
Appendix 1 Detailed descriptions of Chapter 1 entries
Appendix 2 List of Players Involved
Appendix 3 Priorities
The group identified a few tasks involved in the development of this compendium document
To be a forum to identify needs, coordinate processes, and facilitate development of CRM and HF resources and products.