There are a few thoughts which creep up as I read your analysis of
metrics. One deals with something which I refer to as the "distance"
between performance/behavior and the metrics.
In his book _The Fifth Discipline_, Peter Senge describes two kinds of
complexity. He says "Systems thinking teaches that there are two types
of complexity -- the 'detail complexity' of many variables and the
'dynamic complexity' when 'cause and effect' and not close in time and
space and obvious interventions do not produce expected outcomes."
Some of the activity I am currently involved in has presented a problem
because the intervention we are using is widely separated from the
measurment to measure its effect (an overall operational outcome). By
Senge's definition, that is a dynamically complex situation. As a
result, we are not confident that our training is really responsible for
any improvement we may be seeing. This is leading us to rethink how and
where in the process we measure trainee performance. We may need to
trade the 'dynamic complexity' of the current evaluation system for the
'detail complexity' of a system which measures performance closer to the
actual intervention.
Another thought deals with the presence of "norms" in all of our
operations (Skip Mudge addressed this in another post). It is possible,
and likely, that most of us have two parallel sets of metrics in place.
One used during training and checking, and the 'real' one used on the
line when the check airman isn't around. It is a significant challenge
to identify the differences, unify the parallel expectations, and
implement the single 'standard/norm' in the system. Fly as you train,
train as you fly.
-- Best regards, Neil Krey neilkrey_at_why.net http://users.why.net/neilkrey