๐ Lecture 27 โ Behavior & Form Part II
๐ Overview: This lecture continues the study of Human Computer Interaction by exploring how to apply behavioral and formal design principles to different platforms, including the web, mobile devices, and appliances. It emphasizes the importance of creating transparent, flow-inducing interfaces through careful orchestration of interaction elements.
๐๏ธ Topics Covered
This lecture covers postures for web design including information-oriented sites and transactional web applications, web portals, postures for other platforms such as kiosks, handheld devices, and appliances, flow and transparency in interaction design, and orchestration methods including finesse, distinguishing possibility from probability, providing comparisons, and avoiding unnecessary reporting.
๐ Lecture Summary
27.1 Postures for the Web
Designers may be tempted to think the web is different from desktop applications, but the basic four stances cover most needs. Information-oriented sites must balance sovereign attributes (detailed information display using full-screen space) with transient attributes (ease and clarity of navigation). The dominant stance depends on target personas and their behavior patterns โ infrequent users need transient stances, while repeat users benefit from sovereign stances. Web designers must decide early what lowest common denominator screen resolution to support.
๐ Formula: Sovereign stance = full-screen detailed display โ maximizes information density and navigational tools ๐ Example: E-commerce sites like Amazon.com balance sovereign and transient stances by providing one-click ordering, good search, browsing, reviews, recommendation lists, persistent shopping cart, and tracking of recently viewed items.
๐ก Why this matters: The choice between sovereign and transient posture directly impacts user satisfaction and task completion rates.
Transactional sites and Web applications
Transactional Web sites must balance sovereign and transient stances similarly to informational sites. Consumer-oriented sites (e-commerce, online banking) have heavy informational aspects as users research and compare products. Navigational clarity, access to supporting information, and streamlining of transactions are critical.
๐ Formula: Consumer transactional sites = balance sovereign + transient + heavy informational aspects ๐ Example: Amazon.com addresses these issues via one-click ordering, persistent shopping cart, and tracking of recently viewed items.
27.2 Web portals
Early search engines served as navigational portals โ ways to get somewhere else. Nothing really happens; you get in, go somewhere, get out. The appropriate posture is transient for infrequent access and auxiliary for frequent access (like the Windows taskbar).
Environmental portals provide unified access to content and functionality related to a specific topic; actual work is done here. Information is gathered from disparate sources and acted upon. The appropriate posture is sovereign โ the portal becomes a destination in itself.
Within environmental portals, individual elements function as small applications with postures: โข Auxiliary elements: aggregated information sets the user wants constant access to โข Transient elements: minimal complexity, used briefly on demand
๐ Example: A portal composed of dynamic status monitors (auxiliary) plus temporary to-do lists (transient).
27.3 Postures for Other Platforms
Kiosks โ Despite large full-screen nature, most kiosks should be transient because users are first-time (except ATMs), spend minimal time, use touch screens or bezel buttons, and stand in public places with bright light and distractions. Educational/entertainment kiosks can allow more data density and complex interactions.
๐ Definition โ Handheld devices: Often satellite systems of a desktop, used more for viewing than heavy input. The appropriate posture is auxiliary for frequently used applications (PIM, email, browsing) and transient for less frequent ones (alarms).
Cellular telephones are primary communication devices, not satellites. The best interface is transient โ place a call as quickly as possible, then abandon the interface. Voice activation is perfect for placing calls.
๐ Definition โ Convergence devices (e.g., Handspring Treo): Combine handheld data and phone functions. Data functions should be auxiliary, phone functions transient.
Appliances โ Most should be transient posture interfaces. Users want simple, straightforward interfaces to perform specific tasks. Status information (cycle status, VCR settings) should be provided as daemonic icons โ minimal status quietly in a corner.
๐ก Why this matters: Designers often mistakenly put dozens of unwanted features into digital appliances, making LCD touchscreens confusing arrays of unworkable controls.
27.4 Flow and Transparency
Flow is a state of deep, nearly meditative involvement where people concentrate wholeheartedly on an activity, losing awareness of distractions. The concept was first identified by Mihaly Csikszentmihalyi. Tom DeMarco and Timothy Lister describe flow as inducing a "gentle sense of euphoria" where users become unaware of time passage.
To create flow, interaction must become transparent โ the interface must not call attention to itself but serve the user. Four methods to achieve this:
- Follow mental models โ Different users have different mental models of processes. The interface should match users' existing mental models rather than computer processes.
- Direct, don't discuss โ Users prefer interacting with software like using a tool (car, hammer) rather than having a dialog. Direct manipulation idioms are better.
- Keep tools close at hand โ Tools should be on palettes or toolbars for easy access with single clicks.
- Provide modeless feedback โ Information built into the main interface that doesn't stop normal flow.
๐ Definition โ Modeless feedback: Feedback that is built into the main interface and doesn't stop the normal flow of system activities and interaction.
๐ Example: Word shows page number, section, total pages, cursor position, and time modelessly via the status bar. Jet fighters use heads-up display (HUD) to superimpose critical instrument readings onto the forward view.
๐ Example: In a hospital system, physicians and nurses have mental models based on patient records โ finding patients by name makes sense. Business clerks think about overdue bills โ sorting by time overdue and amount due makes sense.
๐ Formula: Direct manipulation = point to what you want โ click and drag โ immediate visual feedback
27.5 Orchestration
Orchestration is "harmonious organization" โ when all interface elements work coherently towards a single goal, the interaction becomes almost invisible. The interaction designer must train to hear "sour notes" in software interaction.
Adding finesse: Less is more โ Constantly strive to reduce interface elements without reducing system power. Do more with less. Example: Instead of File Open dialog boxes in every application, replace with the shell program itself.
๐ Definition โ Finesse: Coordinating and controlling all product power without letting the interface become a jumble of windows and dialogs covered with unrelated controls.
Distinguishing possibility from probability โ Programmers tend to treat possibilities as equal to probabilities. Example: A user has equal mathematical possibility of saving or discarding six hours of work, but the probability of discarding is thousands to one against. Yet typical programs always ask.
๐ Formula: Good design = segregate possible from probable โ don't ask for unlikely scenarios
Providing comparisons โ Quantitative information should answer the question "Compared to what?" Visual presentation should show data rather than just telling about it.
๐ Example: Windows File Manager showed exact bytes free (231,728 KB) โ hard to interpret. A pie chart showing 22% free space with used space in blue and free space in magenta makes comprehension immediate.
Using graphical input โ Software should let users enter graphs and convert them to numbers, not just enter numbers to make graphs. Example: Word processors let users set tabs by dragging markers on rulers.
Reflecting program status โ Programs should visually show when asleep, awake, or busy. Status should be communicated using rich modeless feedback.
๐ Formula: Program state communication = asleepโlooks asleep, awakeโlooks awake, busyโlooks busy
Avoiding unnecessary reporting โ Don't stop proceedings to report normalcy. Save dialogs for events outside normal course. Don't bother users with non-serious problems.
๐ Example: If program has trouble getting through a busy signal, build a status indicator rather than a dialog box.
Avoiding blank slates โ Users prefer seeing what the program thinks is right and manipulating it, rather than explaining what they want from scratch.
๐ Example: Word creates a blank document with preset margins rather than asking for every detail. PowerPoint asks users to choose base style each time โ less adequate.
๐ Formula: Blank slate โ program makes statistically good assumption โ user adjusts โ better than asking for permission
Command invocation versus configuration โ There's a big difference between invoking a function and configuring it. Invocation should be immediate without interrogation; configuration should be separate.
๐ Example: Word's print toolbar button prints immediately; Print dialog from File menu allows configuration. Users invoke commands ten times for every one configuration.
Asking questions versus providing choices โ Asking questions makes users feel inferior; offering choices empowers them. Dialog boxes ask questions; toolbars offer choices.
๐ Definition โ Ejector seat levers: Controls for significant, irreversible actions (like ejecting from a jet fighter). These must be hidden from inexperienced users to prevent accidental deployment. They come in two varieties: those causing significant visual dislocation and those performing irreversible actions.
๐ Example: A function that changes layout of all tools or performs irreversible actions like deleting data should be hidden and require deliberate action to access.
โญ Key Takeaways
The critical concept is that posture choice (sovereign, transient, auxiliary) depends on user personas and behavior patterns โ infrequent users need transient stances, repeat users benefit from sovereign stances. Flow and transparency are essential for productivity, achieved through following mental models, direct manipulation, keeping tools close at hand, and providing modeless feedback. Orchestration requires finesse (less is more), distinguishing possibility from probability, providing comparisons rather than raw numbers, and avoiding unnecessary reporting. Programs should avoid blank slates by making good assumptions, separate command invocation from configuration, and hide ejector seat levers from casual users.
๐ง Quick Revision Questions
- What are the four basic postures for user interfaces, and how do they apply to web design?
- What is "flow" and what four methods help create it?
- Why should designers distinguish between possibility and probability when designing dialogs?
- What is the difference between command invocation and configuration, and how should each be handled?
- Why should "ejector seat levers" be hidden from inexperienced users?
๐ Lecture 28 โ Behavior & Form Part III
๐ Overview: This lecture continues the study of Human Computer Interaction by examining how to eliminate unnecessary user effort (excise) in software interfaces. It explores the nature of excise tasks versus goal-directed tasks, and provides strategies for reducing navigational complexity to improve usability and user satisfaction.
๐๏ธ Topics Covered
The lecture covers two main sections: first, an in-depth analysis of exciseโthe extra work users must perform that doesn't directly contribute to their goalsโincluding GUI excise, visual excise, and training wheels. Second, navigation as a form of excise is examined, along with types of navigation (between windows, panes, tools, and information), and strategies for improving navigation through reducing places to go, providing signposts, overviews, proper mapping, and inflection.
๐ Lecture Summary
28.1 Eliminating Excise
Software often contains interactions that are top-heavy with extra work for the user. Excise is the extra work that satisfies either the needs of our tools or those of outside agents as we try to achieve our objectives. Programmers typically focus so intently on enabling technology that they don't carefully consider the human actions required from a goal-directed point-of-view. The result is software that charges users a cognitive and sometimes physical tax every time it is used. Where we can eliminate the need for excise tasks, we make the user more effective and productive and improve the usability of the software.
๐ Definition โ Excise: Extra work that satisfies the needs of tools or outside agents rather than directly contributing to achieving the user's goal.
What Is Excise?
Any large task involves many smaller tasks. Some work directly toward achieving the goal (goal-directed tasks), while excise tasks don't contribute directly to reaching the goal but are necessary to accomplish it just the same. For example, when driving to the office: opening the garage door, starting the engine, and stopping at traffic lights are excise tasks, while steering down the road toward your office is goal-directed. The distinction is sometimes hard to see because we get so used to excise being part of our tasks.
Software has a clear dividing line between goal-directed tasks and excise tasks. Some software excise tasks are trivial, while others are as obnoxious as fixing a flat tireโinstallation, configuring networks, making backups, and connecting to online services. As a software designer, you should become sensitive to the presence of excise and take steps to eradicate it with the same enthusiasm a doctor would apply to curing an infection.
Virtually all window management falls into this category. Dragging, reshaping, resizing, reordering, tiling, and cascading windows qualify as excise actions.
๐ก Why this matters: Recognizing excise is the first step to eliminating it. Every excise task you remove makes your software more efficient and user-friendly.
GUI Excise
One main criticism of GUIs is that getting where you want is made slower by extra effort manipulating windows and icons. With command lines, users can just type commands and the computer executes them immediately. With windowing systems, they must open folders, find files, then stretch and drag windows into desired configurations. However, the command-line interface forces an even more expensive excise budget: the user must first memorize commands and cannot easily configure their screen. The excise of the command-line interface becomes smaller only after significant time and effort in learning it.
For casual or first-time users, the visual explicitness of the GUI helps them navigate and learn what tasks are appropriate. The step-by-step nature of the GUI helps users who aren't yet familiar with the task or system, and benefits those who must use more than one program at a time.
Excise and expert users
Any user willing to learn a command-line interface automatically qualifies as a power user. These users will easily learn each nuance of programs they use, starting each program with a clear idea of exactly what they want to do. To this user, the assistance offered to casual or first-time users is just in the way. We must be careful when eliminating excise: we must not remove it just to suit power users, but we also must not force power users to pay the full price of providing help to new or infrequent users.
Training wheels
Software designers can inadvertently introduce significant excise in support for first-time or casual users. Training wheelsโfacilities added to make it easy for newer users to learnโquickly become excise as users become perpetual intermediates (familiar with the program). Such facilities must be easily turned off. Training wheels are rarely needed for extended periods and, although a boon to beginners, are a hindrance to advanced learning and use when left on permanently.
"Pure" excise
There are actions that are excise of such purity that nobody needs them, from power users to first-timers. These include most hardware-management tasks the computer could handle itself, like telling a program which COM port to use. Any demands for such information should be struck from user interfaces and replaced with more intelligent program behavior behind the scenes.
Visual excise
Designers sometimes paint themselves into excise corners by relying too heavily on visual metaphorsโdesktops with telephones, copy machines, staplers, and fax machines, or file cabinets with folders in drawers. While these metaphors may make it easy to understand relationships between program elements, after these fundamentals are learned, management of the metaphor becomes pure excise. The screen space consumed by images becomes increasingly egregious, particularly in sovereign posture applications (programs used for extended periods). The little telephone that charmingly told us how to dial on that first day is now a barrier to quick communications.
Transient posture applications (programs used infrequently) can tolerate more training and explanation excise than sovereign applications because their users need more assistance. For sovereign applications, the slightest excise becomes agonizing over time.
The second type of visual excise became significant with the Web: overemphasis of visual design elements to the extent they interfere with user goals and comprehension. The late 90s attracted graphic designers to the Web who viewed it as a predominantly visual medium. This might be appropriate for brochure-ware Web sites (serving as marketing collateral) but is highly inappropriate for transactional Web sites and Web applications, which have far more in common with sovereign desktop applications.
A pre-eminent example was Boo.com, a fashion e-tailor that used hip visuals and flash-based interactive agents but didn't address user goals. The site was sluggish due to flash, visually distracting, confusingly laid out, and difficult to navigate. Users' goals were simply to buy products more quickly, cheaply, and easily online. By the time problems were remedied, customers had abandoned them.
Determining what is excise
The only way to determine whether a function or behavior is excise is by comparing it to the user's goals. If the user needs to see two programs at a time to compare or transfer information, the ability to configure windows to share screen space is not excise. If the user doesn't have this specific goal, a requirement to configure the main window is excise.
๐ Example โ Determining Excise: A user needs to transfer data between two programs. Allowing them to arrange windows side-by-side is not exciseโit directly supports their goal. However, if the user only works in one program at a time, forcing them to configure windows is pure excise.
28.2 Navigation and Inflection
Desktop applications, Web sites, and devices all share one attribute that, if improperly designed, becomes a critical obstacle to usability: navigation. The user must be able to navigate efficiently through features and facilities and stay oriented as they move from screen to screen. A user can navigate if they always understand what to do next, know what state the program is in, and know how to find the tools they need.
Navigation Is Excise
The most important thing to realize about navigation is that, in almost all cases, it represents pure excise or something close to it. Except in games where the goal is to navigate through a maze, navigating through software does not meet user goals, needs, or desires. Unnecessary or difficult navigation becomes a major frustration. Poorly designed navigation presents the number-one problem in the design of any software application or systemโdesktop, Web-based, or otherwise. It is also where the programmer's implementation model is made most apparent to the user.
Types of Navigation
Navigation through software occurs at multiple levels:
-
Navigation between multiple windows or screens โ most disorienting kind, involves gross shifting of attention disrupting user flow and forcing new context. The act of navigating to another window often obscures the original window's contents, requiring window management. If users must constantly shuttle back and forth between windows, productivity drops and disorientation rises. If the number of windows is large enough, users may experience navigational traumaโgetting lost in the interface. Sovereign posture applications avoid this by placing all main interactions in a single primary window with multiple independent panes.
-
Navigation between panes within a window โ windows can contain multiple adjacent panes (separated by splitters) or tabbed panes (stacked on top of each other). Adjacent panes can solve many navigation problems by placing supporting functions directly adjacent to the primary work area. Problems arise when supporting panes become too numerous or don't match the user's workflow. Too many adjacent panes result in visual clutter and confusion. Tabbed panes bring navigational excise and potential disorientation because they obscure what was on screen before. However, tabbed panes are appropriate when multiple supporting panes aren't used simultaneously. Microsoft Internet Explorer for Macintosh uses a variant where tabs can deselect, shutting the adjacent pane like a drawer.
โ ๏ธ Caution: Some programmers interpret tabs as permission to break complex facilities into smaller chunks. This actually increases excise while decreasing understanding and orientation. A dialog box is another roomโhave a good reason to go there.
- Navigation between tools or menus in a pane โ spatial organization is critical to minimizing extraneous mouse movements. Tools used frequently and in conjunction should be grouped together spatially and immediately available. Menus require more navigational effort because contents aren't visible prior to clicking. Frequently used functions should be provided in toolbars or palettes; menu use should be reserved for infrequently accessed commands.
๐ Example โ Poor Tool Navigation: Adobe Photoshop 6.0 forces users to navigate between the Paint Bucket tool and Gradient tool, which occupy the same location on the tool palette. Users must click and hold to select between them, even though both are fill tools used frequently. Better design would place each on the palette next to each other.
- Navigation within information displayed in a pane โ accomplished by scrolling (panning), linking (jumping), and zooming. Scrolling is often a necessity but should be minimized. For 2D visualization and drawing applications, thumbnail maps ease navigation. Linking is the critical navigational paradigm of the Webโbecause it is visually dislocating, extra care must be taken to provide visual and textual cues. Zooming and panning are appropriate for 2D or 3D drawings but typically fall short for arbitrary or abstract data. Spatial zoom is difficult enough for users; logical zoom (displaying more attribute details about objects) is arcane to all but visualization researchers. Panning and zooming together create enormous navigation difficultiesโhumans are not used to moving in unconstrained 3D space.
Improving Navigation
The most effective methods for improving navigation:
1. Reduce the number of places to go
- Keep pages and windows to a minimum: one full-screen window with two or three views maximum
- Keep adjacent panes limited to the minimum needed for users to achieve their goals (three panes is a good maximum for sovereign applications)
- Keep controls limited to what users really need
- Minimize scrolling; default views should allow orientation without too much panning
- Zooming (especially continuous zooming) should be discretionary, not required
Many e-commerce sites present confusing navigation because designers try to serve everyone with one generic site. Personalization can helpโif a user buys books but never CDs, access to the CD portion could be de-emphasized, simplifying navigation for that user.
2. Provide signposts Signposts are persistent objects placed in the user interface that serve as points of reference. In desktop applications, these include the program's window, menu bars, toolbars, palettes, status bars, and rulers. On the Web, Amazon.com makes careful use of persistent objectsโthe tab bar along the top and Search/Browse areas on the left. Their consistent presence and layout help orient customers.
Menus: The most prominent permanent object. Unexpected changes to menus can deeply reduce user trust. It's okay to add items to the bottom of a menu, but the standard suite of items should change only for clearly demonstrable need.
Toolbars: Should be considered recognizable signposts. Removing the toolbar is a dislocating change. Some programs put controls on the toolbar that made the toolbar disappearโthis is a completely inappropriate "ejector seat lever."
Other interface signposts: Tool palettes and fixed data display areas should also be considered persistent objects. Judicious use of white space and legible fonts is important so signposts remain clearly evident.
๐ Example โ Web Signposts: Amazon.com uses a tab bar that remains constant throughout shopping, with Search and Browse areas on the left. These don't just provide navigational optionsโtheir consistent presence and layout help orient customers.
3. Provide overviews Overviews help orient users within the content rather than within the application as a whole. The overview area should itself be persistent, though its content depends on the data being navigated. Overviews can be graphical (like the Navigator palette in Adobe Photoshop) or textual (like breadcrumb displays on the Web). Annotated scrollbars provide location information about selections, highlights, and other attributes of textโMicrosoft Word uses a variant showing page number and nearest header in a ToolTip during scroll.
4. Provide appropriate mapping of controls to functions Mapping describes the relationship between a control, the thing it affects, and the intended result. Poor mapping occurs when a control does not relate visually or symbolically with the object it affects, requiring the user to stop and think about the relationship, breaking flow. Poor mapping increases cognitive load and can result in potentially serious user errors.
5. Inflect your interface to match user needs Inflecting an interface means organizing it to minimize typical navigationโplacing the most frequently desired functions and controls in the most immediate and convenient locations, while pushing less frequently used functions deeper into the interface. Rarely used facilities shouldn't be removed from the program, but should be removed from the user's everyday workspace.
The most important principle in proper inflection is commensurate effort: people will willingly work harder for something more valuable to get. Value is in the eye of the beholder and has entirely to do with the user's goals. If the user really wants something, they will work harder to get it. A program's user interface can't be complex to achieve simple results, but it can be complex to achieve complex results (as long as such results aren't needed very often).
Controls and displays should be organized according to three attributes:
- Frequency of use: Most frequently used items (many times a day) should be immediately in reach. Less frequently used items (once or twice a day) should be no more than a click or two away. Other items can be two or three clicks away.
- Degree of dislocation: The amount of sudden change caused by invoking a function. Generally, put these functions deeper into the interface.
- Degree of exposure: Functions that are irreversible or have dangerous ramifications should be more difficult for users to stumble across.
As users get more experienced, they will search for shortcuts, and you must provide them. When software follows commensurate effort, the learning curve doesn't go away, but it disappears from the user's mind.
โญ Key Takeaways
The central concept of this lecture is that exciseโextra work required by tools rather than directly supporting user goalsโmust be identified and eliminated from software interfaces. Navigation is the most common form of excise, and poorly designed navigation presents the number-one problem in software design. To improve navigation, designers should reduce the number of places users must go, provide persistent signposts like menus and toolbars, offer overviews such as breadcrumbs, ensure proper mapping between controls and functions, and inflect the interface to organize controls by frequency of use, degree of dislocation, and degree of exposure. The principle of commensurate effort reminds us that users will tolerate complexity only when the rewards justify it, and that training wheels for beginners must be easily removable once users become perpetual intermediates.
๐ง Quick Revision Questions
-
What is excise in the context of user interface design, and how does it differ from goal-directed tasks?
-
Explain the difference between sovereign posture applications and transient posture applications in terms of how much visual excise they can tolerate.
-
What are the four main types of navigation described in the lecture, and which type is considered most disorienting for users?
-
List five strategies for improving navigation in applications, Web sites, and devices.
-
What is the principle of commensurate effort, and how should it guide the organization of controls in an interface?
๐ Lecture 29 โ Evaluation โ Part I
๐ Overview: This lecture introduces the fundamental concepts of evaluation in Human Computer Interaction. It explains what evaluation is, why it is necessary, and when it should be conducted during the development process. The lecture also introduces four core evaluation paradigms and various techniques used to assess interactive products.
๐๏ธ Topics Covered
The lecture covers the key concepts of evaluation, including what to evaluate, why evaluation is needed, and when to conduct it. It then introduces four core evaluation paradigms: "quick and dirty" evaluation, usability testing, field studies, and predictive evaluation. Finally, it describes the main evaluation techniques, including observing users, asking users, asking experts, user testing, and modeling users' task performance.
๐ Lecture Summary
What to evaluate?
There is a huge variety of interactive products with a vast array of features that need to be evaluated. Some features, such as the sequence of links to be followed to find an item on a website, are often best evaluated in a laboratory, since this setting allows evaluators to control what they want to investigate. Other aspects, such as whether a collaborative toy is robust and whether children enjoy interacting with it, are better evaluated in natural settings, so that evaluators can see what children do when left to their own devices.
John Gould and his colleagues recommended three principles for developing the 1984 Olympic Message System: focus on users and their tasks; observe, measure, and analyze their performance with the system; and design lucratively. Since that study, a number of new evaluation techniques have been developed, along with a growing trend towards observing how people interact with the system in their work, home, and other settings. The goal is to obtain a better understanding of how the product is (or will be) used in its intended setting, as real-world tasks are often interrupt-driven and very different from laboratory observations.
๐ก Why this matters: The context of evaluation (laboratory vs. natural setting) significantly impacts the validity of the findings, as real-world usage is often messy and interrupt-driven.
Why you need to evaluate?
Just as designers shouldn't assume that everyone is like them, they also shouldn't presume that following design guidelines guarantees good usability. Evaluation is needed to check that users can use the product and like it. Nowadays, users look for much more than just a usable system, as the Nielsen Norman Group points out: the first requirement for an exemplary user experience is to meet the exact needs of the customer, without fuss or bother, followed by simplicity and elegance.
Bruce Tognazzini points out that there are five good reasons for investing in user testing:
- Problems are fixed before the product is shipped, not after.
- The team can concentrate on real problems, not imaginary ones.
- Engineers code instead of debating.
- Time to market is sharply reduced.
- Upon first release, the sales department has a rock-solid design it can sell.
The diversity of interactive products, coupled with new user expectations, poses interesting challenges for evaluators, who must now adapt existing techniques and develop new ones. As well as usability, user experience goals can be extremely important for a product's success.
When to evaluate?
The product being developed may be a brand-new product or an upgrade of an existing product. For new products, considerable time is invested in market research, often using mockups to elicit reactions from potential users. For upgrades, attention is focused on improving the overall product, which is well suited to usability engineering where evaluations compare user performance and attitudes with those for previous versions.
Evaluations done during design to check that the product continues to meet users' needs are known as formative evaluations. Evaluations done to assess the success of a finished product, such as to satisfy a sponsoring agency or check a standard, are known as summative evaluations.
29.1 Evaluation paradigms and techniques
Any kind of evaluation is guided either explicitly or implicitly by a set of beliefs that may be underpinned by theory. These beliefs and the practices (methods or techniques) associated with them are known as an evaluation paradigm. In this lecture, we identify four core evaluation paradigms.
"Quick and dirty" evaluation
A "quick and dirty" evaluation is a common practice in which designers informally get feedback from users or consultants to confirm that their ideas are in line with users' needs and are liked. The emphasis is on fast input rather than carefully documented findings. This approach is often called "quick and dirty" because it is meant to be done in a short space of time. The data collected is usually descriptive and informal, and it is fed back into the design process as verbal or written notes, sketches, and anecdotes.
Usability testing
Usability testing involves measuring typical users' performance on carefully prepared tasks that are typical of those for which the system was designed. Users' performance is generally measured in terms of number of errors and time to complete the task. The defining characteristic of usability testing is that it is strongly controlled by the evaluator. Tests typically take place in laboratory-like conditions that are controlled. Quantifying users' performance is a dominant theme in usability testing. Some evaluators summarize this data in a usability specification so that developers can use it to test future prototypes or versions of the product against it.
๐ Definition โ Usability specification: A document that summarizes performance data so that developers can test future prototypes or versions of the product against it, often specifying optimal performance levels and minimal levels of acceptance.
Field studies
The distinguishing feature of field studies is that they are done in natural settings with the aim of increasing understanding about what users do naturally and how technology impacts them. In product design, field studies can be used to: (1) help identify opportunities for new technology; (2) determine requirements for design; (3) facilitate the introduction of technology; and (4) evaluate technology. The data takes the form of events and conversations that are recorded as notes, or by audio or video recording, and later analyzed using a variety of analysis techniques such as content analysis, discourse analysis, and conversational analysis.
๐ Definition โ Ethnography: A particular type of insider evaluation in which the aim is to explore the details of what happens in a particular social setting. In HCI, it is a means of studying work or other activities to inform the design of information systems and understand aspects of their use.
Predictive evaluation
In predictive evaluations, experts apply their knowledge of typical users, often guided by heuristics, to predict usability problems. The key feature of predictive evaluation is that users need not be present, which makes the process quick, relatively inexpensive, and thus attractive to companies. In recent years, heuristic evaluation in which experts review the software product guided by tried and tested heuristics has become popular. It is important to note that with the advent of new interactive products, original sets of heuristics have been found insufficient, and new sets of heuristics are needed that are tailored to evaluating different classes of interactive products.
The lecture provides a table summarizing key aspects of each evaluation paradigm, comparing them across the following issues: the role of users, who controls the process, the location of the evaluation, when the evaluation is most useful, the type of data collected and how it is analyzed, how findings are fed back into design, and the underlying philosophy.
Techniques
There are many evaluation techniques, and the lecture examines techniques for:
- Observing users: Observation techniques help to identify needs leading to new types of products and help to evaluate prototypes. Notes, audio, video, and interaction logs are well-known ways of recording observations.
- Asking users: Asking users what they think of a product is an obvious way of getting feedback. Interviews and questionnaires are the main techniques for doing this.
- Asking experts: Guided by heuristics, experts step through tasks role-playing typical users and identify problems. This approach is usually relatively inexpensive and quick to perform.
- User testing: Measuring user performance to compare two or more designs has been the bedrock of usability testing. Generally, the time taken to complete a task, the number of errors made, and the navigation path through the product are recorded.
- Modeling users' task performance: These techniques attempt to model human-computer interaction to predict the efficiency and problems associated with different designs at an early stage without building elaborate prototypes. GOMS and the keystroke model are the best-known techniques.
โญ Key Takeaways
Evaluation is essential to verify that a product meets user needs and is enjoyable to use, as relying solely on design guidelines is insufficient. There are four core evaluation paradigms: quick and dirty evaluation for fast, informal feedback; usability testing for controlled, quantitative measurement of user performance; field studies for understanding natural user behavior in real-world settings; and predictive evaluation for expert-driven, inexpensive identification of usability problems. The context of evaluationโlaboratory versus natural settingโsignificantly impacts the findings, and different paradigms are suited for different stages of development (formative vs. summative). Finally, various techniques exist to gather data, including observing users, asking users or experts, testing performance, and modeling tasks, each with its own strengths and limitations.
๐ง Quick Revision Questions
- What are the three principles recommended by John Gould and his colleagues for developing the 1984 Olympic Message System?
- Distinguish between formative evaluation and summative evaluation.
- Name the four core evaluation paradigms discussed in this lecture and briefly describe the key characteristic of each.
- What is the defining characteristic of usability testing, and where does it typically take place?
- What is the key feature of predictive evaluation that makes it attractive to companies, and what is its main limitation?
๐ Lecture 30 โ Evaluation โ Part II
๐ Overview: This lecture introduces the DECIDE framework, a structured checklist for planning and conducting evaluations in Human-Computer Interaction. It is essential because it provides novice evaluators with a systematic approach to ensure evaluations are well-organized, ethical, and yield reliable, valid data that can be interpreted and presented effectively.
๐๏ธ Topics Covered
The lecture covers the DECIDE framework's six steps: determining the goals of an evaluation, exploring specific questions to be answered, choosing the evaluation paradigm and techniques, identifying practical issues such as users and equipment, deciding how to deal with ethical issues, and finally evaluating, interpreting, and presenting data, with a focus on reliability, validity, biases, scope, and ecological validity.
๐ Lecture Summary
DECIDE: A framework to guide evaluation
Well-planned evaluations are driven by clear goals and appropriate questions. To guide evaluations, we use the DECIDE framework, a checklist for novice evaluators. It consists of six steps: 1) Determine the overall goals, 2) Explore the specific questions, 3) Choose the evaluation paradigm and techniques, 4) Identify practical issues, 5) Decide how to deal with ethical issues, and 6) Evaluate, interpret, and present the data.
Determine the goals
The first step is to identify the high-level goals of the evaluation. For example, goals can be to clarify user needs, determine the best metaphor for a conceptual design, fine-tune an interface, or examine how technology changes working practices. These goals guide the evaluation approach, such as using usability testing for quantitative engineering or a field study for exploring innovative products.
Explore the questions
To make goals operational, specific questions must be identified. For instance, the goal of understanding why customers prefer paper tickets over e-tickets can be broken down into questions about trust, computer access, security, or poor interface design. Questions can be further decomposed into sub-questions, such as asking if the system is difficult to navigate or if the terminology is confusing.
Choose the evaluation paradigm and techniques
After identifying goals and questions, the next step is to choose the evaluation paradigm and techniques. The paradigm determines which techniques are used. Practical and ethical issues must be considered, and trade-offs are often necessary due to cost, time, or lack of expertise or equipment.
Identify the practical issues
Several practical issues must be considered before starting an evaluation, including users, facilities and equipment, schedules and budgets, and evaluators' expertise.
- Users: It is crucial to involve appropriate users. They must be screened to ensure they represent the target population, considering factors like experience, age, gender, and cultural diversity. Evaluators must put users at ease by treating them courteously and explaining that the system is being tested, not them.
- Facilities and equipment: Practical concerns include how to set up recording equipment like cameras without making users uncomfortable, and ensuring spare supplies like batteries are available.
- Schedule and budget: Time and budget constraints are important. Planning evaluations that can be completed on schedule and within budget is key, often requiring compromises.
- Expertise: The evaluation team must have the necessary expertise. For example, using models requires prior experience, and analyzing video requires appropriate skills and equipment.
Decide how to deal with the ethical issues
Ethical issues are paramount. Institutional review boards often require researchers to submit plans explaining how human participants will be treated. Key guidelines include:
- Tell participants the goals, process, time commitment, data collected, and any payment offered.
- Ensure confidentiality of demographic, financial, or health information, using a coding system to store data separately from identifiers.
- Make sure users know they can stop the evaluation at any time.
- Pay users when possible to create a formal relationship with mutual commitment.
- Avoid quotes or descriptions that could reveal a person's identity, and ask permission before quoting them.
The general rule is: "Do unto others only what you would not mind being done to you."
The lecture also notes ethical concerns with online research, where user interactions can be logged without their knowledge, raising issues of privacy, confidentiality, and informed consent.
Evaluate, interpret, and present the data
Key decisions involve what data to collect, how to analyze it, and how to present findings. General questions to ask include: Is the technique reliable? Is it valid? Are there biases? Are the results generalizable (scope)? Is the evaluation ecologically valid?
- Reliability: How consistently a technique produces the same results under the same circumstances. A controlled experiment has high reliability, while an unstructured interview has low reliability.
- Validity: Whether the technique measures what it is supposed to measure. For example, a lab experiment is not valid for studying product use in homes; an ethnographic study would be more valid.
- Biases: Distortions in results, such as expert evaluators being sensitive to certain flaws or interviewers unconsciously influencing responses.
- Scope: How much findings can be generalized. For example, a keystroke model predicts expert, error-free behavior, so results cannot describe novice learning.
- Ecological validity: How the evaluation environment influences results. Lab experiments have low ecological validity because they differ from real-world settings, whereas ethnographic studies have high validity. The Hawthorne effect describes how participants alter their behavior when they know they are being studied.
โญ Key Takeaways
The DECIDE framework is a systematic, six-step guide for planning evaluations. A successful evaluation must start with clear goals, then be broken down into specific, operational questions. Practical issues like selecting the right users, managing budgets and schedules, and having the right expertise are critical and often require compromises. Ethical treatment of participants is non-negotiable, requiring informed consent, confidentiality, and respect for their rights. Finally, when interpreting data, evaluators must rigorously assess the reliability, validity, biases, scope, and ecological validity of their techniques to ensure the results are trustworthy and meaningful.
๐ง Quick Revision Questions
- What are the six steps of the DECIDE framework for guiding evaluation?
- Why is it important to break down general evaluation goals into specific questions and sub-questions?
- What are three key practical issues an evaluator must consider, and what kind of trade-offs might be needed?
- List four ethical guidelines that must be followed when conducting an evaluation with human participants.
- Define reliability, validity, and ecological validity, and explain why they are important for interpreting evaluation data.