|
Class |
Method |
Mechanism |
Advantages |
Disadvantages |
|
Analytic Methods |
Task analysis [1-3] |
The evaluator
analyses a task that a user required to do. There are about 40
specific techniques of task analysis. One example is Goals, Operators,
Methods and Selection rules (GOMS). According to GOMS, a task is analyse
into Goals, Operators, Methods and Selection rules to predict human
learning and performance for interface design quantitatively.
|
-
Applicable to all
development stages.
-
Can collect qualitative and
quantitative data.
|
-
Task analysis can be a very
time consuming activity if used with a high degree of detail on complex
problems.
-
The evaluator need to have
competence background in many disciplines e.g. philosophy, psychology,
sociology, ergonomics and computing. |
|
Specialist Reports
(Usability
Inspection) |
Heuristic
[4, 5] |
The inspector reviews
the product interface (in this study, process description) and try to
to determine
the good and bad of its features. Rules or usability principles are used
as guidelines. |
-
Involve small number of
inspectors (3-5).
-
Supported by general
usability rules as guidelines.
-
Can inspect a paper
prototype. |
-
Inspector can misinterpret
the rules especially if it is too general.
-
Inspector decides their own
way to inspect.
-
Can be hard to control.
|
|
Cognitive
Walkthroughs [4] |
A
group of inspectors reviews the product interface. During the
walkthrough, the inspectors consider, in sequence, each of the user
actions needed to accomplish the task. A set of tasks/scenarios is
provided to guide the activity. |
-
Support by strong theory of
learning by exploration.
-
Can inspect a paper
prototype. |
-
Focus on just one attribute
of usability - ease of learning. |
|
Pluralistic
Walkthroughs [4] |
A
group of inspectors reviews the product interface and write what actions
they would take in attempting the specified task. After each inspector
has written their independent response, one of the inspectors
(administrator) announces their answer for the task and the inspectors
can discuss their responses and potential usability issues. |
-
Can inspect a paper
prototype. Promote and support the concept of redesign “on-the-fly”.
-
Detect usability issues in
group can be more productive than working alone. |
-
Always performed by a group
of inspectors – need to wait for all participants to finish writing the
action on paper before the review discussion can start.
-
Cannot simulate all possible
actions on paper. |
|
Feature Inspections
[4] |
The inspector reviews the product features or functions by using a set
of tasks/scenarios to guide the review. |
-
Suitable for product that
ready to be deployed. |
-
Certain features are hard to
describe in scenarios and also probably hard to find by the inspector in
the first place. |
|
Guideline Checklist
[3, 4] |
The inspector reviews the product by using specific style guidelines
(e.g. Apple Human Interface Guideline). |
-
Checklist helps to ensure
that usability principles are covered.
-
Following a style guide will
increase the consistency between screens. |
-
May inhibit inspector
capability from detecting usability problems beyond the provided
guidelines. |
|
Perspective-based
inspection [6] |
The inspector reviews the product from one of several perspectives (e.g.
novice user or expert user). Each perspective provides the inspector
with a point of view, a list of inspection questions, and procedures as
guidelines. |
-
Can reduce the difficulties
for inspector to detect all different usability problems at the same
time.
-
Each perspective can detect
unique set of usability problems. |
-
Inspector may have problems
to understand the perspective. |
|
Observational Methods
(Usability Testing) |
Co-discovery [7,
8] |
Two users attempt to perform tasks together by using the product while
being observed by the evaluator. |
-
Testers may make more
comments about the product when they are trying to solve the tasks
together. |
-
Testers may have different
strategies for learning and using the product.
-
Difficult to train pair of
testers to work together. |
|
Question asking
protocol [7, 9] |
The user learns to use the product by completing given tasks without any
instructional manuals. Tester will be assisted by a tutor, who will
help on a one-to-one basis via question-answering.
|
-
More natural than thinking
aloud protocol method.
-
Suitable for collecting
qualitative data.
-
Suitable for tester with any
level of product background knowledge (novices, little knowledge or
experts). |
-
Not easy for the tutor to
make a proper judgment at all times when respond to tester’s questions.
-
The success of the testing
depends to a large extent on the tutor’s skills. |
|
Think aloud protocol
[7] |
The user verbalises
his/her thoughts, feelings, and opinions while using the product based
on provided tasks. The evaluator records potential problems.
|
-
Effective in collecting
qualitative data from fairly small number of tester.
-
Better understand of the
tester’s mental model and interaction with the product. |
-
May give a false impression
of the cause of usability problems if too much weight is given to the
tester.
-
Unnatural to most tester –
normally tester do not tell others what they think as they work. |
|
Performance
measurement [7, 8] |
The user performs a set of tasks/scenarios while the evaluator collects
metrics for performance measurement (e.g. time and error data) |
-
Suitable for collecting
quantitative data.
-
Objective usability
evaluation. |
-
Difficult to collect
relevant data for evaluation.
-
Focus only on performance
issues. |
|
Logging use
[7] |
The user uses the
product and the product automatically collects statistics about its use. |
-
Trace the usage trend –
which features are highly used (or unused).
-
Can run continuously. |
-
Need many testers.
-
Potential violation of
tester’s privacy. |
|
Field Observation
[3, 7, 10]
|
The evaluator visits the representative user’s workplace to observe them
work and understand how they use the product. |
-
View users’ in real context.
-
Simple to implement – visit
one or more users and doing as little as possible in order not to
interfere with their work. |
-
Users may change their
behavior.
-
Hard to analyse data from
note taking, video or screen capture. |
|
User Reports
(Survey) |
|
Questionnaire [3, 7,
11-14] |
The evaluator writes and distributes usability questionnaires to the
product users. |
-
Data are easy to handle.
-
Can be conducted via email.
-
Easy to repeat. |
-
Lack of dialogue.
-
Need many participants.
-
Is sometimes misleading. |
|
Interview
[3, 7] |
The evaluator obtains
users’ opinions about the product usability via face-to-face or online
meeting. |
-
Opportunity to understand
users closely.
-
Flexible, two-way
communication |
-
Difficult to arrange
suitable time and users commitment.
-
Hard to analyse data. |
