Dear Mr. Talleur:
In a contribution to this mailing list, you recently wrote:
Talleur> I ran WOMBAT (It is capitalized because it stands for
Talleur> something) to try predict performance of new pilots. It
Talleur> failed to predict anything. However, I'm not convinced we
Talleur> were measuring the right things to correlate to the WOMBAT
Talleur> scores.
Talleur> In any event, WOMBAT is a game and unless it has changed
Talleur> dramatically I fail to see how it would predict pilot
Talleur> performance in an emergency better than testing is a real
Talleur> simulator. Also how often do pilot's do a dual tracking task
Talleur> (one with each hand) and in different planes of motion. I
Talleur> flown over 30 types of aircraft and a dozen different types
Talleur> of sims and have yet to find myself doing this type of stuff.
Talleur> But is this a non-similar part-task training device of some
Talleur> sort that, although it has little or nothing to do with
Talleur> flying activities, has good predictive power of how a pilot
Talleur> may perform? I'm not convinced. For regular line flying you
Talleur> might be able to collect data to correlate to WOMBAT scores.
Talleur> But the issue is predicting how pilots will react under real,
Talleur> life-threatening pressure. I still contend that we'll have a
Talleur> hard time producing meanignful data on these events that, in
Talleur> hindsight, can be correlated to WOMBAT scores. Don't think
Talleur> I'm knocking WOMBAT however. It has been proven to have some
Talleur> useful applications in aviation. I just don't believe that
Talleur> this is one of them!
Talleur> Don Talleur
You administered WOMBAT tests to students at the University of Illinois
Institute of Aviation during the Fall of 1993 under the supervision of
Dr. Gavan Lintern. The purpose of the experiment was to find the
correlation between the WOMBAT scores and the flight training required
for ab initio pilots to fly solo. Although the developers of WOMBAT have
always worked to predict distant success (10, 15, 20 years down the line;
see David O'Hare's paper in Dec. 97 issue of the journal Human Factors),
it was interesting to see whether WOMBAT would predict short-term
training success, something of practical importance to training
organizations such as yours.
Consequently, for this study, flight instructors were asked to record the
numbers of landings and flight hours prior to the time they judged their
students were sufficiently proficient to fly solo safely, although most
soloed at a later time due to internal presolo experience requirements.
There was no incentive to take the WOMBAT test other than to participate
in the study, and it has been our experience that candidates take the
test more seriously and generally perform better when they have reason to
believe the scores may affect their acceptance for training or future
employment.
Despite these limitations, the data you collected were analyzed by Dr.
Lintern. He obtained some of the strongest correlations ever found in
this type of study. In fact, WOMBAT scores correlated 0.80 with the
number of landings to presolo proficiency and 0.78 with the number of
flight hours. These results were released by Dr.Lintern, have appeared in
published books and articles, and have been available from Aero
Innovation. It is incorrect to say, "It failed to predict anything."
Whether you were aware of Lintern's findings or not, you have no basis
for such a statement other than your subjective observations.
There is no doubt that WOMBAT has the interface of a game. However, the
principle behind the test is to recognize the complexity of the future
flight performances we are trying to predict and design a test with the
same level of complexity in a culture-free environment. WOMBAT is quite
complex in the sense of its multiple attention demands but is independent
of flying experience in the sense that the component tasks are
individually unlike any component tasks in flying an airplane.
Furthermore, the current version of the test (Version 4.6) is far more
complex and hence more demanding and "stressful" than the version you
used in 1993.
Flight simulators, on the other hand, are the quintessence of a
culture-dependent environment. Manufacturers strive for the ultimate in
verisimilitude to an airplane and the flight environment. Simulators are
designed this way because they are meant for high training transfer and
for assessing an experienced pilot's currency. They are not designed for
assessing an individual's potential for future pilot performance. How one
performs in a simulator depends so much on flight experience and currency
and on the variability in the ratings of different check pilots that it
is a poor basis for selection.
In the four corners of the world we have observed countless breakdowns in
performance under stress during WOMBAT testing, countless attempts by
testees to explain their behavior by blaming the interface, the game-like
environment, their poor comprehension of the tasks to be performed, and
so on. Many of these candidates, when trained by the hiring organization,
have failed to perform acceptably prior to certification, and worse yet,
some have failed on the line. When access to the job of your life depends
on a WOMBAT score, the level of "WOMBAT stress" is comparable to
real-life emergency operational stress. A few thousand people can testify
to this.
Regards,
Jean LaRoche, President
Aero Innovation, inc.
Montreal