EXERCISE 9.4
Choose an appropriate evaluation method
for each of the following situations. In each case
identify
(i) The participants.
(ii) The technique used.
(iii) Representative tasks to be examined.
(iv) Measurements that would be appropriate.
(v) An outline plan for carrying out the evaluation.
(a) You are at an early stage in the
design of a spreadsheet package and you wish to test
what type of icons will be easiest to learn.
(b) You have a prototype for a theatre booking system
to be used by potential theatre-goers to reduce queues
at the box office.
(c) You have designed and implemented a new game system
and want to evaluate it before release.
(d) You have developed a group decision support system
for a solicitor's office.
(e) You have been asked to develop a system to store
and manage student exam results and would like to
test two different designs prior to implementation
or prototyping.
answer
Note that these answers are illustrative;
there are many possible evaluation techniques that
could be appropriate to the scenarios described.
Spreadsheet package
(i) Subjects |
Typical users: secretaries, academics, students,
accountants, home users, schoolchildren |
(ii) Technique |
Heuristic evaluation |
(iii) Representative tasks |
Sorting data, printing spreadsheet, formatting
cells, adding functions, producing graphs |
(iv) Measurements |
Speed of recognition, accuracy of recognition,
user-perceived clarity |
(v) Outline plan |
Test the subjects with examples of each icon
in various styles, noting responses. |
Theatre booking system
(i) Subjects |
Theatre-goers, the general public |
(ii) Technique |
Think aloud |
(iii) Representative tasks |
Finding next available tickets for a show, selecting
seats, changing seats, changing date of booking |
(iv) Measurements |
Qualitative measures of users' comfort with
system, measures of cognitive complexity, quantitative
measures of time taken to perform task, errors
made |
(v) Outline plan |
Present users with prototype system and tasks,
record their observations whilst carrying out
the tasks and refine results into categories identified
in (iv). |
New game system
(i) Subjects |
The game's target audience: age, sex, typical
profile should be determined for the game in advance
and the test users should be selected from this population,
plus a few from outside to see if it has wider appeal |
(ii) Technique |
Think aloud |
(iii) Representative tasks |
Whatever gameplay tasks there
are - character movement, problem solving, etc. |
(iv) Measurements |
Speed of response, scores achieved,
extent of game mastered. |
(v) Outline plan |
Allow subjects to play game and talk
as they do so. Collect qualitative and quantitative
evidence, follow up with questionnaire to assess satisfaction
with gaming experience, etc. |
Group decision support system
(i) Subjects |
Solicitors, legal assistants, possibly
clients |
(ii) Technique |
Cognitive walkthrough |
(iii) Representative tasks |
Anything requiring shared decision
making: compensation claims, plea bargaining, complex
issues with a diverse range of expertise needed. |
(iv) Measurements |
Accuracy of information presented and
accessible, veracity of audit trail of discussion,
screen clutter and confusion, confusion owing to turn-taking
protocols |
(v) Outline plan |
Evaluate by having experts walk through
the system performing tasks, commenting as necessary. |
Exam result management
(i) Subjects |
Exams officer, secretaries, academics |
(ii) Technique |
Think aloud, questionnaires |
(iii) Representative tasks |
Storing marks, altering marks,
deleting marks, collating information, security protection |
(iv) Measurements |
Ease of use, levels of security and
error correction provided, accuracy of user |
(v) Outline plan |
Users perform tasks set, with running
verbal commentary on immediate thoughts and considered
views gained by questionnaire at end. |
|
|