MSI centered system development and system evaluation – A review of methods and methodology
Publish date: 2013-12-31
Report number: FOI-R--3804--SE
Pages: 133
Written in: Swedish
Keywords:
- MSI
- methodology
- methods
- system evaluation
- system evaluation
- iterative design
- task analysis
- quantitative methods
- psychophysiological measures
- statistical analysis
Abstract
MSI (Man System Interaction) is a Human Factors-related research area at the Swedish Defence Research Agency (FOI). MSI performs work with, for example, training and education, team cooperation and decision making, interface design, system requirements specifications, support of system development, system design, and system evaluation. However, this report is mainly limited to descriptions of the methodology and methods that MSI uses to support system development, system design, and system evaluation. The main purpose of the report is to provide an enhanced understanding of how MSI-competence can be utilised during the whole system development process. First, a general background is given about the need to include MSI-evaluations in the whole system development process. Thereafter, different aspects of human capabilities and limitations that are relevant for information presentation and man system interaction are discussed. A general overview is provided of methodology to assure that MSI-questions are regarded in system and interface design. Different methods to design and perform experimental studies and system evaluation are described, for example pros and cons with quantitative and qualitative methods. The most frequently used statistical methods in the MSI-area are also described. To illustrate the importance of continuous feedback to designers and systems engineers from performed evaluations during the entire system development process, iterative design is discussed in a separate chapter. The report presents methodology for task- and user analysis, and emphasizes the need for performing systematic task- and user analyses to thoroughly investigate which task the system will perform, how these tasks will be performed, and the characteristics of the users. The main part of the report describes analytical and empirical methods for data collection and evaluations in the MSI-area. Analytical methods refer to methods that are used by usability experts and system engineers without participation of users or subjects, whereas empirical methods refer to methods that are performed with users, or subjects, with the purpose of measuring their performance and subjective experiences. Generally, the methods descriptions are given at an overall level, which means that the descriptions cannot be used as a handbook to actually perform evaluations or experiments. However, readers that need further information about the methods can use the provided references that are provided.