eLearning I - evaluation results

In the evaluation of the Web-based Training (WBT), two aspects were differentiated, the usability of the application and the performance of the users in evaluating the computer workstation.

Usability

• Data Source: log file recorded by the web server.
• Sample: 5704 visitors, during 18 month.
• Question: to what degree elements of the WBT were barriers to start using and actually completing the task.

Performance

• Conditions: WBT - users completing the WBT; laboratory - occupational health professionals, evaluating an equivalent physical workstation.
• Data Sources: completion time; responses to checklist items, in terms of nonparametric indices of signal detection theory, i.e., sensitivity A' (accuracy of the responses) and response bias B''d (tendency to report an ergonomic deficiency).
• Hypotheses: the WBT condition shows a shorter completion time and a lower sensitivity, compared to the laboratory condition, due to a speed-accuracy trade-off; regarding bias, no systematic differences between the two conditions were expected.

Usability

The start rate, the proportion of visitors that registered for using the WBT, was 22%. The completion rate as the proportion of users that completed the task was 29%. Optional information, which could be accessed via hotspots and hyperlinks in the model, the checklist, and the feedback, was used comparatively infrequently.

Performance

In the WBT condition, as expected, the task was completed much faster than in the laboratory condition. This did not affect the accuracy in responding to items about the spatial arrangement of elements of the computer workstation; here the sensitivity indices of the two conditions were not significantly different. However, items related to the object details were answered less accurately in the WBT condition, on the average, than in the laboratory condition. Moreover, for both item types significant differences were found for the response bias, the WBT condition had a generally lower tendency for negative responses, i.e., for reporting an ergonomic deficit.

Utility of VR scenarios

Summarizing the results, the WBT was of interest for a large number of people, which showed a performance that was comparable to that of participants which completed the task under equivalent laboratory conditions, evaluating a real-world computer workstation. The judgment of object details, on the other hand, turned out to be more difficult than in the laboratory condition. This result is to be attributed to the limited fidelity of the scenario, to allow its replay on conventional computer systems. But this problem is expected to be less grave in the near future, with the more widespread availability of more sophisticated hardware.

Cursory and intuitive information processing

The results concerning the completion rate, the use of optional information, the task completion time, and the response bias indicate that content provided via the WWW is studied in less depth, compared to conventional media. This might be related to the fact that time-contingent cost-models for accessing the Internet still dominate. Under these conditions, it would be more adequate to break down comprehensive learning modules (here: a complete evaluation of a computer workstation) into smaller units (e.g., by presenting different aspects separately).

The implications of the outlined results were seized in our follow-up project INTEGRAL II.

Gude, D., Branahl, E., Kawalek, P., Prions, A., & Laurig, W. (2003). Evaluation of a virtual reality-based ergonomics tutorial. In D. de Waard, K.A. Brookhuis, S.M. Sommer, & W.B. Verwey (eds.), Human Factors in the Age of Virtual Reality (pp. 117-128). Maastricht, the Netherlands: Shaker Publishing.

In the section Selected publications the text is available as a pdf-file.