Chapter 11

Evaluation Techniques

11.1 In groups or pairs, use the cognitive walkthrough example, and what you know about user psychology (see Chapter 1), to discuss the design of a computer application of your choice (for example, a word processor or a drawing package). (Hint: focus your discussion on one or two specific tasks within the application.)

answer

This exercise is intended to give you a feel for using the technique of cognitive walkthrough (CW). CW is described in detail in Chapter 11 and the same format can be used here. It is important to focus on a task that is not too trivial, for example creating a style in a word processing package. Also assume a user who is familiar with the notion of styles (and with applications on the same platform (e.g. Macs, PCs, UNIX, etc.)) but not with the particular word processing package. Attention should be given to instances where the interface fails to support the user in resolving the goal and where it presents false avenues.

11.2 What are the benefits and problems of using video in experimentation? If you have access to a video recorder, attempt to transcribe a piece of action and conversation (it does not have to be an experiment - a soap opera will do!). What problems did you encounter?

answer

The benefits of video include: accurate, realistic representation of task performance especially where more than one video is used; a permanent record of the observed behaviour.

The disadvantages include: vast amounts of data that are difficult to analyse effectively; transcription; obtrusiveness; special equipment required.

By carrying out this exercise, you will experience some of the difficulties of representing a visual record in a semi-formal written format. If you are working in a group, discuss which parts of the video are most difficult to represent, and how important these parts are to understanding the clip.

11.3 In Section 11.5.1 (An example: evaluating icon designs), we saw that the observed results could be the result of interference. Can you think of alternative designs that may make this less likely - read the description of the experiment carefully. Remember that individual variation was very high, so you must retain a within-groups design, but you may perform more tests on each subject.

answer

Three possible ways of reducing interference are:

Notice that all the above measures require additional subject time and one has to constantly weigh up the advantages of richer experiments against those of larger subject groups.

11.4 Choose an appropriate evalution method for each of the following situations. In each case identify

  1. The subjects.

  2. The technique used.

  3. Representative tasks to be examined

  4. Measurements that would be appropriate.

  5. An outline plan for carrying out the evaluation.

answer

Note that these answers are illustrative; there are many possible evaluation techniques that could be appropriate to the scenarios described.

Spreadsheet package

1SubjectsTypical users: secretaries, academics, students, accountants, home users, schoolchildren
2TechniqueHeuristic evaluation
3Representative tasksSorting data, printing spreadsheet, formatting cells, adding functions, producing graphs
4MeasurementsSpeed of recognition, accuracy of recognition, user-perceived clarity
5Outline planTest the subjects with examples of each icon in various styles, noting responses.

Theatre booking system

1SubjectsTheatre-goers, the general public
2TechniqueThink aloud
3Representative tasksFinding next available tickets for a show, selecting seats, changing seats, changing date of booking
4MeasurementsQualitative measures of users' comfort with system, measures of cognitive complexity, quantitative measures of time taken to perform task, errors made
5Outline planPresent users with prototype system and tasks, record their observations whilst carrying out the tasks and refine results into categories identified in 4.

New game system

1SubjectsThe game's target audience: age, sex, typical profile should be determined for the game in advance and the test users should be selected from this population, plus a few from outside to see if it has wider appeal
2TechniqueThink aloud
3Representative tasksWhatever gameplay tasks there are - character movement, problem solving, etc.
4MeasurementsSpeed of response, scores achieved, extent of game mastered.
5Outline planAllow subjects to play game and talk as they do so. Collect qualitative and quantitative evidence, follow up with questionnaire to assess satisfaction with gaming experience, etc.

Group decision support system

1SubjectsSolicitors, legal assistants, possibly clients
2TechniqueCognitive walkthrough
3Representative tasksAnything requiring shared decision making: compensation claims, plea bargaining, complex issues with a diverse range of expertise needed.
4MeasurementsAccuracy of information presented and accessible, veracity of audit trail of discussion, screen clutter and confusion, confusion owing to turn-taking protocols
5Outline planEvaluate by having experts walk through the system performing tasks, commenting as necessary.

Exam result management

1SubjectsExams officer, secretaries, academics
2TechniqueThink aloud, questionnaires
3Representative tasksStoring marks, altering marks, deleting marks, collating information, security protection
4MeasurementsEase of use, levels of security and error correction provided, accuracy of user
5Outline planUsers perform tasks set, with running verbal commentary on immediate thoughts and considered views gained by questionnaire at end.

11.5 Complete the cognitive walkthrough example for the video remote control design.

answer

Continue to ask the four questions for each Action in the sequence. Work out what the user will do and how the sytem will respond. If you can analyse B and C, you will find that Actions D to I are similar.

Hint: Remember that there is no universal format for dates.

Action J: Think about the first question. Will the user even know they need to press the transmit button? Isn't it likely that the user will reach closure after Action I?

Worked exercises

The following appear as worked exercises in Chapter 11:


HCI 2e home page || changes and additions || resources || search || contents || authors || ordering
feedback to hcibook@hiraeth.com
http://www.hcibook.com/hcibook/
designed and hosted by hiraeth mixed media
webmaster@hiraeth.com