I believe that any evaluation system must attempt to be proactive,
responsive and recurrent. In other words, the data we are trying to collect
is as follows:
a) Have the course delegates remembered what they were 'taught'
b) Has there attitute changed as a consequence
c) Has behaviour changed as a consequence
d) What do they need to learn next
a) To take each in turn, whilst 'remembering' is not the same as
believing or doing, it is the first step. If there are aspects of a
training program that simply aren't remembered then it needs to be covered
again and in a different way. It is no use crossing 'things to do' off a
list of HF subjects if no-one took what was said on board.
b) Attitude change is something that can be monitored to a degree
through a recurrent tool. i.e. a survey that goes out e.g. annually with
similar questions as well as different ones. With a statistically
significant sample, there is not the same need to match samples. One
Australian carrier I am working on this subject with has just run phase one
of a three (plus) stage survey to collect the baseline data for a company
wide HF/Team work type training initiative. It includes attitude questions
such as;
"Internal communications within my section of X are generally good..."
(state level of agreement)
"Over the last year, communications in general within X have improved..."
(state level of agreement)
The responses may of course be influenced by initiatives outside of HF
training, but then seeing as an integrated approach toward HF is our true
goal, being able to separate variables is not that important.
c) 'Has behaviour changed' is more difficult as if we ask individuals
about themselves they are likely to be over-positive (i.e. I know I am a
better driver than anyone else in Sydney). Instead we ask more general
questions about "How often in the last 12 months have the following
happened to you..." As long as similar questions are asked in each pass of
the survey, personal bias will be negated somewhat.
d) One of the failings of HF/CRM training has been the temptation to
'teach people to suck eggs' (i.e. tell them things they already know). Some
of the problems with beinging in psychologists, consultants and even
academics has been a reaction from the delegates that they are being taught
things they already know under new 'touchy-feely' names. Who has ever heard
the comment "That's not CRM; that's airmanship and we've done it that way
for years...." A recurrent survey lets you test things you may be
considering adding to the next course syllabus. If people already know the
answer; don't waste resources teaching it (hello AQP!).
There is plenty more to say, but I am conscious of the 'keep it brief'
philosophy.
Hope this helps
Best Wishes
Dr. Graham Braithwaite
Department of Aviation
University of New South Wales
Sydney 2052
Australia
Tel: +61 (0) 2 9385 6757
Fax: +61 (0) 2 9385 6637
Mobile: 0414 736 467