First, I strongly endorse your comment that:
>>using consensus to converge on a norm (focus group research sessions)
should be conducted by the CRM program designers to identify and document the
norms to be placed in the task list; not by evaluators to help them determine
the norm 'on the fly' for each observation.<<
Well said, and very important.
>>... performance measurement can be reflected in both the process as well as
the outcome.<<
Our experience indicates that we should measure the process, not the outcome.
Our management system (process) was designed to manage the judgment/decision
making process, and control risk, rather than solve specific problems or
circumstances. If the system is properly used, and the crew still experiences
a "negative" outcome, then the system has failed, not the crew. So we must
first fix the system, and then train the pilots in the modifications.
Quite naturally, to fairly evaluate the crew's use of the system, the system
must be easily, objectively and accurately observed.
I also agree with you that most grading scales usually require some sort of
judgment by the evaluator. We prefer a simple "yes" or "no" observation. Was
a required management action taken or not? To do this, of course, you must
have a logical set of clearly observable behaviors. AQP requires CRM based on
task analysis. Observable behavior is the natural output of such an analysis.
Our approach is to design the management system from the top down and then
test it from the bottom up (how well does it work in various specific
situations?)
I got the impression from some of your comments (e.g. "In fact, the CRM
expectations probably should be worded with the following sequence: 'Given
Set of Circumstances'....'Appropriate Response'") that you are taking a
reactive approach to CRM. Wait until something happens, then determine and
use the appropriate CRM tool. Effective CRM, whether it be a management
system or not, should be pro-active, in addition to being reactive. It should
guide the crews to make better decisions at all times, and significantly
reduce the risk of potential problems even developing — not wait until there
is a problem which must then be managed by CRM. It must be pro-active and
reactive.
Hope you had a good Thanksgiving.
Skip Mudge