Usability.Ru

Новости

Статьи

Персоналии

Коллективы

Библиотека

Глоссарий

Форум

Ссылки

DRAFT

INTERNATIONAL ISO DIS 9241-11

STANDARD

September 1994




Ergonomic requirements for office work with visual display terminals (VDTs): -

Part 11:

Guidance on Usability


Editor's note:
This version of the standard will be circulated for international vote terminating in mid-1995. If you have any comments on the content, these can be made formally to your national standards body, or directly to me:
Nigel Bevan
NPL Usability Services
National Physical Laboratory
Teddington, Middlesex
TW11 OLW, UK
Tel: +44 181 943 6993
Fax: +44 181 977 7091
Email:
Nigel@hci.npl.co.uk


Table of Contents

Foreword
Introduction
1. Scope
2. Normative references
3. Definitions
4. Benefits and rationale
5. Specifying and measuring the usability of products 5.1. Framework for specifying usability
5.1.1. Purpose
5.1.2. Components of usability
5.1.3. Information needed

5.2. Context of use
5.2.1. Description of users
5.2.2. Description of equipment
5.2.3. Description of environments

5.2.4. Description of goals

5.2.5. Description of tasks

5.2.6. Examples

5.3. Usability measures

5.3.1. Choice of measures

5.3.2. Effectiveness

5.3.3. Efficiency

5.3.4. Satisfaction

5.3.4. Further examples

5.4. Interpretation of measures

6. Specification and evaluation of usability during design

6.1. Specification of overall requirements for a product

6.2. Specification of usability requirements for a product

6.3. Product development

6.4. Specification or evaluation of product attributes

6.5. Usability measurement

6.6. Quality system

6.7. Comparative evaluation of products

6.8. Use with other standards

7. Specifying and measuring the quality of a work system in use

7.1. Relationship to usability

7.2. Examples of application

7.2.1. Design of a complete work system

7.2.2. Diagnostic evaluation

Annex A (informative) Example of how to specify the context of use

Annex B (informative) Examples of usability measures

B.1. Overall usability

B.2. Measures for desired properties of the product

B.3. Choosing usability criteria

B.4. Types of measures

B.5. Measures of effectiveness and efficiency

B.5.1. Measuring effectiveness

B.5.2. Measuring efficiency

B.6. Measures of satisfaction

B.7. Measures of cognitive workload&

Annex C (informative) Example of a usability requirements specification

Annex D (informative) Relationship to other standards


D.1. Definition of usability in ISO 9126

D.2. Use in conjunction with other parts of ISO 9241

D.2.1. Relationship of context of use to usability

D.2.2. Use in conjunction with ISO 9241-10: Dialogue Principles

Annex E (informative) References and bibliography


Foreword

ISO (the International Organisation for Standardisation) is a worldwide federation of national standards bodies (ISO member bodies). The work of preparing international Standards is normally carried out through ISO technical committees. Each member body interested in a subject for which a technical committee has been established has the right to be represented on that committee. International organisations, governmental and non-governmental, in liaison with ISO, also take part in the work. ISO collaborates closely with the international Electrotechnical Commission (IEC) on all matters of electrotechnical standardisation.

Draft International Standards adopted by the technical committees are circulated to member bodies for voting. Publication as an international Standard requires approval by at least 75% of the member bodies casting a vote.

International Standard ISO 9241-11 was prepared by the Technical Committee ISO/TC159, Ergonomics, Sub-Committee SC4, Ergonomics of human-system interaction.

ISO 9241 consists of the following Parts, under the general title Ergonomics requirements for office work with visual display terminals (VDTs):


Part 1: General Introduction
Part 2: Guidance on task requirements
Part 3: Visual display requirements
Part 4: Keyboard requirements
Part 5: Workstation layout and postural requirements
Part 6: Environmental requirements
Part 7: Display requirements with reflections
Part 8: Requirements for displayed colours
Part 9: Requirements for non-keyboard input devices
Part 10: Dialogue principles
Part 11: Guidance on usability
Part 12: Presentation of information
Part 13: User guidance
Part 14: Menu dialogues
Part 15: Command dialogues
Part 16: Direct manipulation dialogues
Part 17: Form filling dialogues

Introduction

The objective of designing and evaluating for usability is to enable users to achieve goals and meet needs in a particular context of use. This part of ISO 9241 explains the benefits of measuring usability in terms of user performance and satisfaction. These are measured by the extent to which the intended goals of use are achieved, the resources that have to be expended to achieve the intended goals, and the extent to which the user finds the use of the product acceptable.

This part emphasises that usability is dependent on the context of use and that the level of usability achieved will depend on the specific circumstances in which a product is used. The context of use consists of the users, tasks, equipment (hardware, software and materials), and the physical and social environments which may all influence the usability of a product in a work system. Measures of user performance and satisfaction assess the quality of the work system in use, and, when a product is the focus of concern, these measures provide information about the usability of that product in the particular context of use provided by the rest of the work system. The effects of changes in other components of the work system, such as the amount of user training, or the improvement of the lighting, can also be measured by user performance and satisfaction.

The term usability is sometimes used to refer to the attributes of a product which make it easier to use (see Annex D). Requirements and recommendations relating to the attributes of the hardware, software and environment which contribute to usability, and the ergonomic principles underlying them, are provided in other parts of ISO 9241. This part of ISO 9241 explains how the resulting usability of a product when used in a particular context can be assessed by measures of user performance and satisfaction.


DRAFT INTERNATIONAL STANDARD ISO DIS 9241-11: 1994 (E)


Ergonomic requirements for office work with visual display terminals (VDTs): -

Part 11:

Guidance on Usability


 

1. Scope

This part of ISO 9241 defines usability and explains how to identify the information which is necessary to take into account when specifying or evaluating usability in terms of measures of user performance and satisfaction. Guidance is given on how to describe the context of use of the product (hardware, software or service) and the required measures of usability in an explicit way.

The guidance in this part of ISO 9241 can be used in procurement, design, development, evaluation, and communication of information about usability. It includes an explanation of how the usability of a product can be specified and evaluated as part of a quality system which conforms to ISO 9001. It applies both to products intended for general application and products being acquired for or being developed within a specific organisation.

This part also explains how measures of user performance and satisfaction can be used to measure how any component of a work system affects the quality of the whole work system in use.

The guidance includes procedures for measuring usability but does not detail all the activities to be undertaken. Specification of detailed user-based methods of measurement is beyond the scope of this part of ISO 9241, but further information can be found in Annex B and the bibliography in Annex E. Some other parts of ISO 9241 contain specific user performance test methods.

This part of ISO 9241 applies to office work with visual display terminals, and other situations where a user is interacting with a product to achieve goals. ISO 9241 parts 14 to 17 provide recommendations which are applicable in given contexts of use. The guidance in this part can be used in conjunction with ISO 9241 parts 14 to 17 in order to identify the applicability of individual recommendations.

 

2. Normative references

The following standards contain provisions which, through reference in this text, constitute provisions of this part of ISO 9241. At the time of publication, the editions indicated were valid. All standards are subject to revision, and parties to agreements based on this part of ISO 9241 are encouraged to investigate the possibility of applying the most recent editions of the standards indicated below. Members of IEC and ISO maintain registers of currently valid International Standards.

 

ISO 9241: Ergonomics requirements for office work with visual display terminals (VDTs):

Part 14: Menu dialogues

Part 15: Command dialogues

Part 16: Direct manipulation dialogues

Part 17: Form filling dialogues

 

3. Definitions

For the purposes of this part of ISO 9241, the following definitions apply:

3.1. usability: The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.

3.2. quality of a work system in use: The extent to which specified goals can be achieved with effectiveness, efficiency and satisfaction in a specified work system.

Note: The difference between usability and the quality of a work system in use is a matter of focus. When usability is evaluated, the focus is on improving a product while the other components of the work system (user, task, equipment, and environment) are treated as given. When the quality of a work system in use is evaluated any component may be subject to modification or improvement.

3.3. effectiveness: The accuracy and completeness with which users achieve specified goals.

3.4. efficiency: The resources expended in relation to the accuracy and completeness with which users achieve goals.

3.5. satisfaction: The comfort and acceptability of use.

3.6. context of use: The users, goals, tasks, equipment (hardware, software and materials), and the physical and social environments in which a product is used.

3.7. work system: A system, consisting of users, equipment, tasks and a physical and social environment, for the purpose of achieving particular goals.

Note: The context of use consists of the goals of the work system and those components of the work system which are treated as given when specifying or measuring usability.

3.8. user: The person who interacts with the product.

3.9. goal: An intended objective.

3.10. task: The activities undertaken to achieve a goal.

3.10. product: The part of the equipment (hardware, software and materials) for which usability is to be specified or evaluated.

 

4. Benefits and rationale

Usability is an important consideration in the design of products because it is concerned with the extent to which the users of products are able to work effectively, efficiently and with satisfaction.

Usable products may be designed by incorporating product features and attributes known to benefit users in particular contexts of use. This process alone is not sufficient to ensure that the product will be effective, efficient and satisfying in use. In order to establish the level of usability achieved, it is necessary to measure the performance and satisfaction of users working with a product. Measurement of usability is particularly important in view of the complexity of the interactions between the user and the task characteristics and the other elements of the context of use, which can result in significantly different levels of usability for the same product when used in different contexts.

Dealing with usability as part of a quality system for design and development of products, as specified in ISO 9001, involves the systematic identification of requirements for usability, including usability measures and verifiable descriptions of the context of use. These provide design targets which can be the basis for verification of the resulting design.

The approach adopted in this part of ISO 9241 has benefits which include:

 

5. Specifying and measuring the usability of products

5.1. Framework for specifying usability

5.1.1. Purpose

The framework describes the components of usability and the relationship between them.

5.1.2. Components of usability

In order to specify or measure usability it is necessary to decompose effectiveness, efficiency and satisfaction and the components of the context of use into sub-components with measurable and verifiable attributes. The components and the relationships between them are illustrated in Figure 1.

5.1.3. Information needed

When specifying or measuring usability, the following information is needed:

Figure 1

Figure 1 - Usability framework

 

5.2. Context of use

5.2.1. Description of users

Relevant characteristics of the users need to be described. These may include knowledge, skill, experience, education, training, physical attributes, and motor and sensory capabilities. It may be necessary to define the characteristics of different types of user, for example with different levels of experience or performing different roles.

5.2.2 Description of equipment

Relevant characteristics of the equipment need to be described. The description of the hardware, software and materials may be in terms of a set of products, one or more of which may be the focus of usability specification or evaluation, or it may be in terms of a set of attributes or performance characteristics of the hardware, software and other materials.

5.2.3 Description of environments

Relevant characteristics of the physical and social environment need to be described. Aspects which may need to be described include attributes of the wider technical environment (e.g. the local area network), the physical environment (e.g. workplace, furniture), the ambient environment (e.g. temperature, humidity) and the social and cultural environment (e.g. work practices, organisational structure and attitudes).

5.2.4 Description of goals

The goals of use of a product should be described. Goals may be decomposed into sub-goals which specify components of an overall goal and the criteria which would satisfy that goal. For example, a telephone sales clerk may have the goal to "Maintain customer orders". This overall goal might then be decomposed into sub-goals such as:

The level at which the overall goal is set is a function of the boundary of the work system which is under consideration and which provides the context of use. In the example above the work system under consideration consists of clerks taking telephone orders.

The specification or measurement of the usability of a particular product should identify the overall goal, specific sub-goals, the relevant context of use, including the tasks and resources involved, and the measures of effectiveness, efficiency and satisfaction which are chosen as being relevant to the goals which have been identified.

5.2.5 Description of tasks

Tasks are the activities undertaken to achieve a goal. Characteristics of tasks which may influence usability should be described as part of the context of use, e.g. the frequency and the duration of performance.

Detailed descriptions of the activities and processes may be required if the description of the context is to be used as a basis for the design or evaluation of details of interaction with the product. This may include description of the allocation of activities and steps between the human and technological resources. Tasks should not be described solely in terms of the functions or features provided by a product or system. Any description of the activities and steps involved in performing the task should be related to the goals which are to be achieved.

For the purposes of evaluating usability, a set of key tasks will typically be selected to represent the significant aspects of the overall task.

NOTE - User tasks and sub-tasks can be identified by task analysis (for more information see the bibliography in Annex E).

5.2.6 Examples

Annex A gives examples of how the components of the context of use can be described in terms of characteristics which may be relevant to usability.

 

5.3 Usability measures

5.3.1 Choice of measures

A description of usability measures consists of target or actual values of effectiveness, efficiency, and satisfaction for the required contexts. It is normally necessary to provide at least one measure for each of effectiveness, efficiency and satisfaction. Because the relative importance of components of usability depends on the context of use and the purposes for which usability is being described, there is no general rule for how measures should be chosen or combined.

The choice of measures and the level of detail of each measure, is dependent on the objectives of the parties involved in the measurement. The importance each measure has relative to the goals should be considered. For example where usage is infrequent, high importance may be given to measures of learning and relearning.

If it is not possible to obtain objective measures of effectiveness and efficiency, subjective measures based on the user's perception can provide an indication of effectiveness and efficiency.

5.3.2 Effectiveness

Measures of effectiveness relate the goals or sub-goals of the user to the accuracy and completeness with which these goals can be achieved.

For example if the desired goal is to accurately reproduce a 2-page document in a specified format, then accuracy could be specified or measured by the number of spelling mistakes and the number of deviations from the specified format, and completeness by the number of words of the document transcribed divided by the number of words in the source document.

5.3.3 Efficiency

Measures of efficiency relate the level of effectiveness achieved to the expenditure of resources. Relevant resources may include mental or physical effort, time, materials or financial cost. For example human efficiency could be measured as effectiveness divided by human effort, temporal efficiency as effectiveness divided by time, and economic efficiency as effectiveness divided by cost.

If the desired goal is to print copies of a report, then efficiency could be specified or measured by the number of usable copies of the report printed, divided by the resources spent on the task such as labour hours, process expense and materials consumed.

5.3.4 Satisfaction

Measures of satisfaction describe the comfort and acceptability of the use.

Satisfaction can be specified and measured by attitude rating scales or measures such as the ratio of positive to negative comments during use. Additional information may be obtained from longer term measures such as rate of absenteeism, health problem reports, or the frequency with which users request transfer to another job.

Measures of satisfaction may assess attitudes to use of the product, or assess the user's perception of aspects such as efficiency, helpfulness or learnability.

5.3.5. Further examples

Further examples of measures that can be used for assessing usability are included in Annexes B and C.

5.4 Interpretation of measures

Care should be taken in generalising the results of any measurement of usability to another context which may have significantly different types of users, tasks or environments. If measures of usability are obtained over short periods of time the values may not take account of infrequent events which could have a significant impact on usability, for example intermittent system errors.

For a general-purpose product it will generally be necessary to specify or measure usability in several different representative contexts, which will be a subset of the possible contexts and of the tasks which can be performed. There may be differences between usability in these contexts.

 

6. Specification and evaluation of usability during design

6.1. Specification of overall requirements for a product

Information about the characteristics of users, their goals and tasks and the environments in which the tasks are carried out provides important information for use in the specification of overall product requirements, prior to development of specific usability requirements.

6.2. Specification of usability requirements for a product

Prior to development of a custom system, a purchasing organisation can use the information in this part of ISO 9241 as a framework for specifying the usability requirements which the system must meet and against which acceptance testing may be carried out. Specific contexts in which usability is to be measured should be identified, measures of effectiveness, efficiency and satisfaction selected, and acceptance criteria based on these measures established (an example is given in Annex C).

6.3. Product development

The definition and framework for usability can be used by product development teams to establish a common understanding of the concept of usability, and can help product development teams address the breadth of issues associated with product usability.

A developer can use the guidance in this part of ISO 9241 to help specify usability targets for the product (see Annex C). At various stages during the development process the developer can measure the usability achieved against these targets. This information enables objective decisions to be taken about the need for design changes to enhance usability, and about trade-offs which may be appropriate between usability and other requirements.

6.4. Specification or evaluation of product attributes

The guidance on context of use can be used to identify the users, tasks and environments so that more accurate judgements can be made about the needs for particular product attributes.

6.5. Usability measurement

This part of ISO 9241 provides information to support measurement of usability. For example, description of the characteristics of users can assist with the selection of users to participate in evaluation. Identification of user's goals can assist with the selection of appropriate tasks for usability testing or reviews. The characteristics of the environment in which a product is likely to be used need to be described if that environment has to be simulated to ensure the validity of test results.

This part of ISO 9241 also provides a basis from which measures of usability can be generated. Product developers can develop appropriate measures of efficiency, effectiveness, and/or satisfaction (see Annex B).

6.6. Quality system

The activities listed in 6.1 to 6.5 can provide a basis for defining, documenting and verifying usability as a part of a quality system. For example, Figure 2 illustrates how usability can be incorporated into a quality system which conforms to ISO 9001.

Figure 2 Quality Plan

6.7. Comparative evaluation of products

This part of ISO 9241 can be used to help structure comparative evaluations of usability. Evaluators can use this part of ISO 9241 to identify which parts of the context of use (the users, their tasks, and the environment of use) need to be controlled when conducting comparative evaluations.

The comparative user performance test methods in the Annexes of parts 3 and 4 of ISO 9241 are examples of usability evaluation for the purpose of equivalence testing of a product (display screen or keyboard). These test methods, which are intended to provide alternative means of testing conformance, enable a manufacturer to demonstrate the usability of a new product by showing that, in a specific context of use, the measures of usability with the product are at least as good as the measures of usability with a product which has physical attributes which are known to meet the requirements of the standard.

A purchasing organisation can use the guidance in this part of ISO 9241 to help choose between products already available. Having specified the usability requirements in terms of the intended context of use and which measures of effectiveness, efficiency and satisfaction are to be used, the guidance may then be used to specify test conditions and evaluation criteria. The test conditions should be representative of important aspects of the overall context of use.

6.8. Use with other standards

Other parts of ISO 9241, such as Part 14, contain recommendations which are applicable in particular contexts of use. The guidance in this part of the standard can be used to provide a framework for identifying the context of use which is relevant to the design decision to be made. Annex D contains more information on the relationship with other parts of ISO 9241 and other standards.

 

7. Specifying and measuring the quality of a work system in use

7.1. Relationship to usability

Measures of usability apply to products used within the overall work system. Measures of effectiveness, efficiency and satisfaction can be used to evaluate how any component of a work system affects the quality of the whole work system in use.

If the aim is to improve the quality of the overall work system in use, any part of the work system may be the subject of design or evaluation. For example it may be appropriate to consider the amount of user training to be provided, changes in lighting, or re-organisation of the task. In this case the element which is the object of design or evaluation is considered to be subject to potential variation, while the other elements of the work system are treated as fixed.

7.2. Examples of application

7.2.1. Design of a complete work system

When designing a complete work system its quality in use may be optimised by changing components of the context of use of a product, such as the operating system, lighting or amount of user training. In this case measures of effectiveness, efficiency and satisfaction can be used to specify or evaluate the effect of different operating systems, different types of lighting, or different amounts of user training respectively.

7.2.2. Diagnostic evaluation

When the quality of a work system in use is recognised to be unsatisfactory it is often necessary to carry out systematic analyses of the contribution of different components of the context of use and the interactions between them, in order to determine what the principle causes of the problems are. This process may also be used to identify which components are amenable to change, in order to bring about improvements in the quality of the work system in use. Diagnostic activity relating to the context of use is often a necessary step in the specification and evaluation of product usability, since it provides the basis for judging whether problems can be resolved by the re-design of the product.

 

Annex A (informative)

Example of how to specify the context of use

Table A.1 [2,12] gives an example of how the context of use can be specified in terms of attributes which may be relevant to the usability. A product to be used in a particular context can be specified under these headings. Specifications may be either in terms of relevant characteristics, or the identification of a specific instance. (When specifying a particular instance of a product which can be customised, it will be necessary to specify what adaptations have been made to the default characteristics of the product.)

Verifiable and repeatable descriptions of usability require measures of usability which can be repeated in a specified context. Unless evaluation of usability can take place in conditions of actual use, it will be necessary to decide which attributes of the actual or intended context of use are to be represented in the context which is specified for evaluation. When specifying or evaluating usability it is therefore important that the context selected is representative of the important aspects of the actual or intended context of use. Particular attention should be given to those attributes which are judged to have a significant impact on the usability of the product.

This information should form part of any report on usability.

Table A.1 Example of attributes of the context of use

USERS

GOALS

ENVIRONMENT

Personal details
User types
Audience and secondary users

Skills & knowledge
Product experience
System knowledge
Task experience
Organisational experience
Training
Keyboard & input skills
Qualifications
Language skills
General knowledge

Personal attributes
Age
Gender
Physical capabilities
Physical limitations and disabilities
Intellectual ability
Attitude
Motivation

 

Goals

Sub-goals

 

Organisational Environment

Structure
Hours of work
Group working
Job function
Work practices
Assistance
Interruptions
Management structure
Communications structure
Remuneration

Attitudes & culture
Policy on use of computers
Organisational aims
Industrial relations

Job design
Job flexibility
Performance monitoring
Performance feedback
Pacing
Autonomy
Discretion

 

EQUIPMENT

TASKS

Technical environment

Basic description
Product identification
Product description
Main application areas
Major functions

Specification
Hardware
Software
Materials
Other Items

Task breakdown
Task name
Task frequency
Task duration
Frequency of events
Task flexibility
Physical and mental demands
Task dependencies
Task output
Risk resulting from error

 

Configuration
Hardware
Software
Reference materials

Physical environment

Workplace conditions
Atmospheric conditions
Auditory environment
Thermal environment
Visual environment
Environmental instability

Workplace design
Space and furniture
User posture
Location

Workplace safety
Health hazards
Protective clothing & equipment

 

Annex B (informative)

Examples of usability measures

B.1. Overall usability

To specify or measure usability, measures of effectiveness, efficiency and satisfaction are required for goals. Usability measures may be specified for overall goals (e.g. produce a letter) or for narrower goals (e.g. perform search and replace). Selecting usability measures for the most important user goals may mean ignoring many functions, but is likely to be the most practical approach. Examples of appropriate measures are given in Table B.1.

Table B.1 Examples of measures of usability

Usability objective

Effectiveness measures

Efficiency measures

Satisfaction measures

Overall usability

Percentage of goals achieved;

Percentage of users successfully completing task;

Average accuracy of completed tasks

Time to complete a task;

Tasks completed per unit time;

Monetary cost of performing the task

Rating scale for satisfaction;

Usage rate over time;

Frequency of complaints

B.2. Measures for desired properties of the product

Additional measures may be required for particular desired properties of the product which contribute to usability. Examples of some of these properties and additional specialised measures are given in Table B.2. In addition, where appropriate the measures given in Table B.1 can also be used for the usability objectives given in Table B.2.

Table B.2 Examples of measures for desired properties of the product

Usability objective

Effectiveness measures

Efficiency measures

Satisfaction measures

Appropriate for trained users

Number of power tasks performed;
Percentage of relevant functions used

Relative efficiency compared with an expert user

Rating scale for satisfaction with power features

Appropriate for walk up and use

Percentage of tasks completed successfully on first attempt

Time taken on first attempt;
Relative efficiency on first attempt

Rate of voluntary use

Appropriate for infrequent or intermittent use

 

Time spent relearning functions;
Number of persistent errors

Frequency of reuse

Minimisation of support requirements

Number of references to documentation;
Number of calls to support;
Number of accesses to help

Productive time;
Time to learn to criterion

 

Learnability

Number of functions learned;
Percentage of users able to learn to criterion

Time to learn to criterion;
Time to relearn to criterion;
Relative efficiency while learning

Rating scale for ease of learning

Error tolerance

Percentage of errors corrected or reported by the system;
Number of user errors tolerated

Time spent on correcting errors

Rating scale for error handling

Legibility

Percentage of words read correctly at normal viewing distance

   

 

B.3 Choosing usability criteria

The choice of criterion values of measures of usability depends on the requirements for the product and the needs of the organisation setting the criteria. Usability objectives may relate to a primary goal (e.g. produce a letter) or a sub-goal (e.g. search and replace) or secondary goals (e.g. learnability or adaptability). Focusing usability objectives on the most important user goals may mean ignoring many functions, but is likely to be the most practical approach. Setting usability objectives for specific sub-goals may permit evaluation earlier in the development process.

It may be necessary to specify criteria both for the minimum acceptable level of usability and for the target level of usability.

When setting criterion values for a group of users, the criteria may be set as an average (e.g. average time for completion of a task to be no more than 10 minutes), for individuals (e.g. all users can complete the task within 10 minutes), or for a percentage of users (e.g. 90% of users are able to complete the task in 10 minutes).

B.4 Types of measures

Measures of usability should be based on data which reflect the results of users interacting with the product or work system. It is possible to gather data by objective means, such as the measurement of output, of speed of working or of the occurrence of particular events. Alternatively data may be gathered from the subjective responses of the users expressing feelings, beliefs, attitudes or preferences. Objective measures provide direct indications of effectiveness and efficiency while subjective measures can be linked directly with satisfaction.

It should be noted that it is possible to obtain data relating to each component of usability from objective or from subjective measures. For example, satisfaction can also be inferred from objective measures of the behaviour of the users, and estimates of effectiveness and efficiency can be derived from subjective opinions which the users express about their work and its outputs.

The validity of the data gathered to predict the level of usability achieved when a product is actually used will depend upon the extent to which the users, tasks and context of use are representative of the real situation and the nature of the measures chosen. At one extreme one may make measurements in the "field" using a real work situation as the basis for the evaluation of the usability of a product. At the other end of the continuum one may evaluate a particular aspect of the product in a "laboratory" setting in which those aspects of the context of use which are relevant are re-created in a representative and controlled way. The advantage of using the laboratory based approach is that it offers the opportunity to exercise greater control over the variables which are expected to have critical effects on the level of usability achieved, and more precise measurements can be made. The disadvantage is that the artificial nature of a laboratory environment can produce unrealistic results.

Evaluations may be conducted at different points along the continuum between the field and laboratory settings depending upon the issues which need to be investigated and the completeness of the product which is available for test. The choice of test environment and measures will depend upon the goals of the measurement activity and their relationship with the design cycle.

B.5 Measures of effectiveness and efficiency

B.5.1 Measuring effectiveness

Effectiveness is defined as the accuracy and completeness with which users achieve specified goals.

To measure accuracy and completeness it is necessary to produce an operational specification of the criteria for successful goal achievement. This can be expressed in terms of the quality and quantity of output, for example, the specification of a required format for output documents together with the number and length of documents to be processed.

Accuracy can be measured by the extent to which the quality of the output corresponds to the specified criteria, and completeness can be measured as the proportion of the target quantity which has been achieved.

If a single measure of effectiveness is required, it is possible to combine measures of accuracy and completeness. For example, completeness and accuracy may be calculated as percentages and multiplied together to give a percentage value for effectiveness [2,12]. In cases where it is not appropriate to trade accuracy off against completeness, the two measures should be considered independently.

B.5.2 Measuring efficiency

Efficiency is measured by relating the level of effectiveness achieved to the resources used. Temporal efficiency can be defined as the ratio between the measure of effectiveness in achieving a specified goal, and the time it takes to achieve that goal. This provides an absolute measure of temporal efficiency in a particular context. Similar calculations can be made with respect to efficiency in the use of mental or physical energy, materials or financial cost.

 

B.6 Measures of satisfaction

Satisfaction (defined as comfort and acceptability of use) is a subjective response of users to interaction with the product. Satisfaction can be assessed by subjective or objective measures. Objective measures may be based on observation of the behaviour of the user (e.g. body posture, body movement, frequency of absences) or can be based on monitoring the physiological responses of the user.

Subjective measures of satisfaction are produced by quantifying the strength of a user's subjectively expressed reactions, attitudes, or opinions. This process of quantification can be done in a number of ways, for example, by asking the user to give a number corresponding to the strength of their feeling at any particular moment, or by asking users to rank products in order of preference, or by using an attitude scale based on a questionnaire.

Attitude scales, when properly developed, have the advantage that they can be quick to use, have known reliabilities, and do not require special skills to apply. Attitude questionnaires which are developed using psychometric techniques will have known and quantifiable estimates of reliability and validity, and can be resistant to factors such as faking, positive or negative response bias, and social desirability. They also enable results to be compared with established norms for responses obtained in the past. QUIS [3,16] and SUMI [10,11] are examples of attitude questionnaires.

 

B.7 Measures of cognitive workload

Workload entails both physical and mental aspects of tasks. Physical workload is the load resulting from movements, walking, carrying, etc. and also arising from constrained postural positions. The activities that are of a mental nature (perception, information processing, thinking, etc.) contribute to 'cognitive workload'. The design of computer hardware should take into account physical demands caused by high rates of input and sustained periods of activity. Interactive dialogues result in cognitive demands which in some circumstances can have a significant effect on usability of software products.

It is possible to measure the cognitive workload on the user. Cognitive effort is one resource expended in relation to the accuracy and completeness with which users achieve goals and can therefore contribute to the measurement of efficiency. However, cognitive workload has certain special characteristics in that both under- and over-loading may result in lowered efficiency. A task demanding too little mental effort may result in a lowered efficiency because it leads to boredom and lack of vigilance, which directly lowers effectiveness. In such a case overall efficiency would be enhanced by increasing demand. Excessive cognitive workload may also result in lowered effectiveness, if it causes information to be missed and results in errors. This is a particularly important issue in situations where safety is critical, e.g. air traffic control and process control. Measures of cognitive workload can be used to predict these types of problems.

Two examples of questionnaires which have been validated for measuring mental effort and perceived work demands are the Subjective Mental Effort Questionnaire (SMEQ) [19] and the Task Load Index (TLX) [13]. The SMEQ is a unidimensional, whereas the TLX distinguishes between six aspects of work demands.

 

Annex C (informative)

Example of a usability requirements specification

This example shows how a usability requirements specification might be written to assist a development team to consider usability in a systematic way during the development process. The usability requirements specification defines the intended context of use and specifies measures and criterion levels for effectiveness, efficiency and satisfaction which must be reached during the development of a product.

The specification is based on the framework provided by this part of ISO 9241 for considering the relevant factors. The specification has two main sections which provide the minimal information required about context of use and measures of usability (see clause 5.1.3). The format of the specification closely follows the structure of clauses 5.2 and 5.3 of this part of ISO 9241. The format is not critical and can be adapted, as appropriate, provided the relevant information is provided in a sufficiently precise form.

The team involved in producing such a usability requirements specification might typically involve a human factors specialist with experience in usability engineering, and other relevant professionals, for example from marketing, product management and software development.

[Comments on the content of the specification are given in italics.]

Name and purpose of the product

[This section identifies the product and describes its overall purpose.]

This specification defines usability requirements for the Videophone professional viewing terminal (VidiPro).

The purpose of VidiPro is to increase the effectiveness of telecommunication by providing a combination of voice and visual information.

Context of Use

[This section provides information following the guidance in clause 5.2 of this part of ISO 9241. Aspects of the context of use are described where these are critical for the usability of the product, for example:

Specification of users

It is assumed that users are familiar with the most common features of business telephone systems. Some potential users would therefore have to acquire this knowledge before they could use VidiPro successfully. However, no skills or knowledge are assumed with regard to Videotelephony (system knowledge and task experience), and this is explicitly stated.

Specification of environments

Access to an ISDN socket and mains power are mentioned because without these VidiPro could not be used at all.

Specification of equipment

The essential parts of VidiPro are listed. This is important where equipment consists of a number of hardware and/or software components, all of which are necessary.

Specification of tasks

A product such as VidiPro might be used for many purposes. However, the product is intended to help the user to achieve certain particular goals and these are stated here.

A number of specific tasks are mentioned where the usability of VidiPro is critical, i.e. installation, call set-up, programming and extended use. At a more detailed level, display legibility is mentioned as this is a condition for usability.]

Specification of users

VidiPro is intended for use by any person with the following characteristics:

Attribute

Requirement

Skills and knowledge

 

Product experience

use of business telephone system with "hold", "transfer" and "call diversion" features

System knowledge

none required

Task experience

none required

Organisational experience

none required

Training

none required

Keyboard and input skills

use of telephone keypad

Qualifications

none required

Linguistic ability

minimum reading age = 11 years1

   

Physical attributes

 

Vision

(corrected) normal vision measured using standard test

Hearing

normal hearing measured using standard screening test2

Manual dexterity

one hand with normal dexterity (VidiPro operable with one hand)

Notes:

1. For reading help and other information on display.

2. Hearing not required for operation of video features.

Specification of environments

The following connections must be available:

In order to meet the usability objectives, VidiPro should be used in an environment which conforms to relevant ergonomics standards, in particular:

Specification of equipment

The main hardware components of VidiPro are an ISDN telephone terminal, a multiplexer, a video codec, a CCD camera and an 20 cm colour CRT.

Specification of tasks

The primary intended goal of VidiPro is to enable two users to communicate by voice and to share visual information about objects which are found in an office environment, e.g. sketches, printed text and graphics, maps, photographs and models.

Secondary goals of VidiPro include general audio and video communication in an office environment, i.e. without viewing objects. VidiPro could also be used in other environments (e.g. domestic) to provide enhanced interpersonal communication.

This usability requirements specification applies to the following specific goals:

An additional usability requirement for a desired property of the product is:

 

Specification of MEASURES OF USABILITY FOR PARTICULAR CONTEXTS

[This section specifies the usability goals, targets or objectives for the product by defining both measures and criterion values of the measures for each of the critical tasks listed above. To supplement the general description of the intended context of use, additional information is provided about the specific context of use for each task. At least one measure and criterion value is provided for effectiveness, efficiency and satisfaction following the guidance given in clause 5.3 of this part of ISO 9241.

The particular measures chosen and the criterion levels specified will depend on the priorities set by the development team taking the needs and requirements of potential customers into account. In this example the criterion for the efficiency of installation has been specified as completion of the installation task within 10 minutes. (This part of ISO 9241 does not and cannot provide guidance with respect to particular criterion levels.)

By producing a specification which follows the guidance of this part of ISO 9241, it is made clear to all parties concerned in what way and under what conditions the product will be usable, and how to check whether this level of usability has been achieved during the evaluation of the product.]

Goal:
Make VidiPro operational, i.e. ready to make or receive calls, or to be programmed.

Specific context:
Start with VidiPro in original packaging. No additional support or  tools are provided.
Users are completing the procedure for the first time.

Effectiveness:
Accuracy: All components are correctly connected to each other.

Completeness:
VidiPro is connected to a mains power supply and an ISDN line.

Efficiency:
Installation to be completed by the user within 10 minutes.

Satisfaction:
Less than 10% of users report dissatisfaction with installation procedures.

Goal:
Set up a video call to enable visual information to be shared.

Specific context:
Standard on-line and off-line user guidance are available.
Users are completing the procedure for the first time.

Effectiveness:
Accuracy and completeness: Called party can see object (e.g. A4 document) within field of view of camera.

Efficiency:
Video call set-up to be completed within 2 minutes.

Satisfaction:
Less than 10% of users report dissatisfaction with video call set-up procedures.

Goal:
Change VidiPro default setting or information stored:
1) set up call diversion
2) enter name and number in the directory

Specific context:
Standard on-line and off-line user guidance are available.
Users may be completing the procedure for the first time.

Effectiveness:

Accuracy:
No errors in information entered.

Completeness:
No omissions in information to be entered or deleted.

Efficiency:
Call diversion to be programmed within 2 minutes.
New name and number to be entered into directory within 3 minutes.

Satisfaction:
Less than 10% of users report dissatisfaction with programming procedures.

Goal:
Communicate by voice and share visual information about bjects in an office environment over an extended period.

Specific context:
Users have used VidiPro to share visual information at least 60 times over a period of at least one month.

Effectiveness:

Accuracy: Less than 2% of all calls placed result in connection to wrong numbers.

Completeness: At least 95% of intended calls are completed successfully.

Efficiency: Average video call set up time to be no more than 30 seconds.

Satisfaction: At least 90% of users prefer VidiPro to alternative equipment, e.g. telephone plus fax machine.

Goal: Read system messages and instructions displayed on screen.

Specific context: Range of illumination levels from 50 to 5000 lux.

Effectiveness: Accuracy: At least 98% of words used in system messages and instructions can be read correctly at normal viewing distance.

 

Annex D (informative)

Relationship to other standards

D.1. Definition of usability in ISO 9126

The term usability is often used to refer to the capability of a product to be used easily. This corresponds with the definition of usability as a software quality in ISO/IEC 9126:

"a set of attributes of software which bear on the effort needed for use and on the individual assessment of such use by a stated or implied set of users".

However the attributes which a product requires for usability depend on the nature of the user, task and environment. A product has no intrinsic usability, only a capability to be used in a particular context. Usability cannot be assessed by studying a product in isolation.

There are therefore three potential ways in which the usability of a product could be measured.

1. By analysis of the product and the context of use.

Usability could be measured by assessing the product features required for usability in a particular context. Appropriate features are specified in other parts of ISO 9241. However ISO 9241 only gives partial guidance. Of the many potential design solutions compatible with ISO 9241, some will be more usable than others.

2. By analysis of the process of interaction.

Usability could be measured by modelling the interaction between a user carrying out a task with a product. However, current analytic approaches do not give very precise estimates of usability. As the interaction is a dynamic process in the human brain, it cannot be studied directly. However, measures of mental effort and acceptability are important indirect measures of factors contributing to usability.

3. By analysing the effectiveness and efficiency which results from use of the product in a particular context and measuring the satisfaction of the users of the product. As these are direct measures of usability, they are the ultimate test of usability. If a product is more usable in a particular context, usability measures will be better.

It should be noted that usability as defined in this part of ISO 9241 also depends on software qualities which are distinct from usability as defined in ISO 9126, such as functionality, reliability and computer efficiency. These software qualities all contribute to quality of the work system in use.

Usability defined in terms of the quality of a work system in use necessarily depends on all factors which may influence use of a product in the real world, including organisational factors such as working practices and the location or appearance of a product, and individual differences between users including those due to cultural factors and prejudice. This broad approach has the advantage that it concentrates on the real purpose of design of a product - that it meets the needs of real users carrying out real tasks in a real technical, physical and organisational environment. This is consistent with the objectives of ISO 9241 described in ISO 9241-1.

 

D.2 Use in conjunction with other parts of ISO 9241

D.2.1 Relationship of context of use to usability

Any component of the context of use (user, equipment, task or environment) may be manipulated in order to change the usability of a product. The user interface may be improved by conforming to good dialogue design practices (e.g. ISO 9241 parts 10, and 12-17). In addition the fit between the user and the rest of the context of use may be improved through means such as selection and training. The task may be designed appropriately (e.g. ISO 9241 part 2). Aspects of the working environment such as lighting, noise, workstation design might be improved (e.g. ISO 9241 parts 3-9). However, when evaluating the usability of a product, the focus is on optimising the product for a given context.

A software designer needs to identify the anticipated users, tasks and environments using this part of ISO 9241, before designing appropriate usability attributes into the software following the guidance and requirements of ISO 9241 parts 10, and 12-17. However, this alone will not guarantee that a product reaches a required level of usability, as ISO 9241 does not provide an exhaustive specification of how to apply the general principles that make a product usable.

D.2.2 Use in conjunction with ISO 9241-10: Dialogue Principles

D.2.2.1 Relevance of the context of use

The way in which each dialogue principle in ISO 9241-10 should be applied to the design or evaluation of a product will depend on the context of use. It is necessary to identify the relevant characteristics of the intended users, tasks and environments before applying the dialogue principles for design or evaluation. Although all aspects of the context of use should be considered for every dialogue principle, "suitability for the task" deals with design issues which are closely associated with the task characteristics. When applying this principle particular consideration should be given to those tasks which particular types of user may need to perform to meet the goals of the user organisation. "Suitability for learning", "suitability for individualisation", and "conformity with user expectations" deal with design issues which are closely associated with the user characteristics. When applying these principles particular consideration should be given to the needs of different types of intended users when performing intended tasks in particular situations.

D.2.2.2 Suitability for learning and individualisation

The dialogue principle "suitability for learning" refers to the attributes of a product which facilitate learning. Actual learnability in a specific context can be measured by comparing the usability of a product for one user over time, or comparing the usability of a product for an experienced and inexperienced user.

The dialogue principle "suitability for individualisation" refers to attributes of the product which facilitate adaptation to the user's needs for a given task. Actual flexibility of use by different users for different tasks can be assessed by measuring the usability in a number of different contexts.

D.2.2.3 Use of dialogue principles as usability objectives

The dialogue principles are stated in terms of characteristics of software. Some of the principles are related to usability objectives which are desired properties of the product, e.g. achievement of goals, error tolerance and learnability (see Annex B).

Annex E (informative)

References and bibliography

[1] Bevan N (1995) Measuring usability as quality of use. Journal of Software Quality.

[2] Bevan N and Macleod M (1994) Usability measurement in context. Behaviour and Information Technology, 13, 132-145.

[3] Chin,J.P., Diehl,V.A., and Norman,K.L.(1988). Development of an instrument measuring user satisfaction of the human-computer interface. Proc. ACM CHI'88 Conf. (Washington, DC 15-19 May), 213-218.

[4] Dumas S D and Redish J C (1994) A practical guide to usability testing. Ablex Publishing Corporation.

[5] ETSI (1991) Guide for usability evaluations. ETSI/TC-HF(91)4.

[6] Houwing E.M., Wiethoff M., and Arnold A.G. (1993). Introduction to cognitive workload measurement. Delft University of Technology, Laboratory for Work & Interaction Technology (WIT Lab).

[7] ISO/IEC 9001: Quality systems - Model for quality assurance in design/development, production, installation and servicing

[8] ISO/IEC 9126 Software Product Evaluation - Quality Characteristics and Guidelines for the User.

[9] Johnson P J (1992) Human Computer Interaction: psychology, task analysis and software engineering. McGraw Hill.

[10] Kirakowski J & Corbett M, 1993, SUMI: the software Usability Measurement

Inventory, BJEdTech 24.3 210-214

[11] Kirakowski J (in press) The software usability measurement inventory: background and usage. In: P Jordan, B Thomas, & B Weerdmeester, Usability Evaluation in Industry. Taylor & Frances, UK.

[12] Macleod M, Thomas C, Dillon A, Maissel J, Rengger R., Maguire M, Sweeney M, Corcoran R. and Bevan N. (1993) Usability Context Analysis handbook, Version 3. National Physical Laboratory, Teddington, UK.

[13] Nasa-Ames Research Center, Human Performance Group (1986) Collecting NASA Workload Ratings: A Paper-and-Pencil Package. Moffet Field, CA: NASA-Ames Research Center

[14] Nielsen J (1993) Usability Engineering. Academic Press

[15] Rengger R, Macleod M, Bowden R, Drynan A and Blaney M. (1993) MUSiC Performance Measurement Handbook. National Physical Laboratory, DITC, Teddington, UK.

[16]Shneiderman, B (1992) Designing the User Interface. Strategies for Effective Human-Computer Interaction, Reading, MA: Addison-Wesley

[17] Stammers R B, Carey M S, Astley J A (1990) Task Analysis. In: Evaluation of Human Work - A practical ergonomics methodology, Wilson J R and Corlette E N, (eds), pps 134-160. Taylor and Francis.

[18] Whiteside J, Bennett J, Holzblatt K (1988) Usability engineering: our experience and evolution. In: Handbook of Human-Computer Interaction, Helander M (ed). Elsevier.

[19] Zijlstra, F.R.H. (1993) Efficiency in Work Behaviour: a Design Approach for Modern Tools. Delft: Delft University Press.

Usability.Ru

Новости

Статьи

Персоналии

Коллективы

Библиотека

Глоссарий

Форум

Ссылки