Category Archives: UI

Anatomy of a Usability Testing Plan: Dana Chisnell at EDUI2009 (Part Three)

(Originally published in 2009 and still highly relevant.)

In her all-day day session at EDUI2009, “Usability Testing Without the Scary,” Dana Chisnell laid out the entire process for usability testing. Breaking the process down into small parts that are easy to understand, Dana showed us how to be comfortable with the process, rather than intimidated. Here’s what she shared about creation of the plan for usability testing.

Someone who will try the design, somewhere to test and something to study.

As Dana puts it, the essence of testing is “sit next to someone and watch them do stuff.” Pretty straightforward. You just have to know what you want them to try out, and how to watch and ask questions so you can learn what they are thinking as they work with your design and functionality.

To support great site design, each phase of the development process should be supported by input from users. It’s also essential to have multidisciplinary teams working on development (ie. marketing, IT, management, etc.).

Although it’s highly useful to look at best practices, guidelines and conventions for usability, they aren’t enough to guarantee usability and accessibility.The nuances of implementation can conflict with best practices, and best practices can even conflict, cancel each other out, or magnify certain issues.  “Informed designs come from data,” says Dana.

The goals of usability testing should be to identify problems in design that lead to

  • misinformation
  • incomplete transactions
  • need for support from administration, management or staff

What tests and measurements to do when

–When you want to map out what the design should do, you observe and listen to users through user testing, focus groups, participatory design, surveys, heuristic evaluations and setting benchmarks.

–To figure out how the design should work, use participatory design, paper prototyping, walk-throughs, usability testing and heuristic evaluation

–To determine whether the design actually does what you want it to do, conduct usability tests, do heuristic evaluation, follow-up studies and comparison of benchmarks.

In the early part of the process, do things that help you learn, that are exploratory and formative. In the middle, do things that help you assess and summarize. And at the end, validate and verify when you’re close to launch but have enough time to incorporate changes.

The User Testing Plan

It’s important to create a test plan. It serves as your blueprint for testing, is a communication vehicle, clarifies needed resources, is a focal point for each test, and lays out milestones. It should include:

  • Goals and objectives**
  • Research questions**
  • Participant characteristics**
  • Description of method**
  • List of tasks**
  • Description of test environment
  • Say what the moderator will do
  • List of the data you’ll collect
  • Description of how the results will be reported

**If your time is limited, focus on the starred tasks.

Next: Selecting participants

EDUI 2009: Dana Chisnell on Taking the “Scary” Out of User Testing (Part Two)

(Originally published in 2009 and still highly relevant.)

In her EDUI 2009 Workshop, Usability Testing Without the Scary, Dana Chisnell discussed how to best support great site design. User testing is essential, even if you only have time to spend one hour with one person. If one person has a problem, it’s legitimate to assume that others do too, and to make design changes based on that data. It’s also important to have management buy-in to doing testing–they must be able to see the benefits of doing the research and implementing what is learned from the tests. Multi-disciplinary teams also support implementation of great design, provided they speak one anothers’ languages and understand one anothers’ skills. Finally, it’s essential to be willing to learn as you go and make changes accordingly. (Note that these criteria are completely in alignment with Jared Spool’s research showing that vision, feedback and culture are the most important elements of developing great design.)

Great designs are :

  • Useful–What’s valuable to your user about having this information or this tool.
  • Efficient–Users can accomplish their goal quickly (that means you have to know what your users’ goals are).
  • Learnable–It should be easy for users to apply what they’ve learned about how to use other sites to using your site (your site should conform to usabilities conventions as much as possible).
  • Satisfying–Your site should be one that people are happy to use, one that seems to be easy and doesn’t take too much time.
  • Accessible–Implementing accessibility guidelines actually makes your site easier for everyone to use.

You should identify design problems that lead to misinformation, incomplete transactions, and/or necessitate support from administrators or staff.

What kind of assessment to use to answer fundamental site design questions

Dana segments analysis into three areas and defines certain types of assessments to use for each area.

What should the site design do? When you want to answer this question, do usability testing, conduct focus groups, have users participate in the design process, do surveys, conduct heuristic evaluations and set benchmarks.

How should the site design work? To answer this question, use participatory design, paper prototyping, walk-throughs, usability testing and heuristic evaluation.

Does the site design do what we want it to do? Answer this via usability testing, heuristic evaluation, follow-up studies and comparison to benchmarks.

Next: Dana’s pointers on putting together a test plan

EDUI2009:Dana Chisnell Shows How to Do Usability Testing “Without the Scary” (Part One)

(Originally published in 2009 and still highly relevant.)

Dana Chisnell presented an all-day session at the EDUI 2009 Conference held in Charlottesville  September 21 & 22, 2009. She founded Usability Works and does usability research, user interface design, and technical communication consulting and development. She is co-author of the Handbook of Usability Testing.  The focus of the workshop was to show how eminently doable user testing can be, if you let go of the idea that you have to do it “by the book.”  Classic, by-the-book user testing can seem intimidating and scary. Instead, Dana spent the day showing how to identify and do the essentials and make it easier to test, and to do it  on an on-going basis. Her approach takes into account that there usually isn’t a lot of time to test. The following is a list of all elements “by the book,” and I’ve noted which ones Dana says are essential:

  • Develop a test plan ESSENTIAL
  • Choose a testing environment–Don’t test in the lab—go to where the users are
  • Find and select participants–ESSENTIAL
  • Prepare test materials
  • Conduct the sessions–ESSENTIAL
  • Debrief with participants and observers–ESSENTIAL–Debrief only with observers
  • Analyze data and observations
  • Create findings and recommendations

User testing works best when you’re early in the design process (that’s where testing using paper prototypes comes in very handy). What you need is someone who will try the design; somewhere to test (preferably where the user typically would use your site, such as at home, the office, or in a dorm room); something to study. Neat and simple. And you’ll learn things about your website interface that you would never have seen on your own.

For example, in conducting user tests of the University of Pennsylvania website,  Dana learned that users were looking for admissions “dates.” But they couldn’t find the information because the University used a different term: “timetable.” User testing provided the context in which they learned about this very simple labelling problem. Your site has to accommodate the way your users think and work, and you can’t be certain about that unless you actually watch them use your site. You must learn whether the user grasps the conceptual design of your site and if not, what the user’s concept is.

 Pointers for moderating user tests

There’s tremendous value in sitting next to someone and watching them do something. Dana recommends saying to the tester things like “tell me what you’re doing–think aloud… tell me what you’re thinking…” When the tester is quiet, be careful about when you ask a question–sometimes they’re quiet because they’re working on a problem, and you may not want to interrupt that process. At the end of the test, review the experience with the tester. Walk through what happened, what happened next. Ask “how did that go? What was confusing or frustrating.” Dana conducted a demonstration, working with an attendee. I found the moderator questions she asked to be very helpful:

  • How would you describe the information you’re looking for?
  • How far do you think you are from getting it?
  • Are you warm or cool?
  • You were hovering there–tell me what you were thinking.
  • What question do you have about the site right now?
  • What do you think the site is about?
  • What one main thing should be improved?

It’s essential for the moderator to talk with the tester and create a situation that feels natural to him or her. As moderator, you should be neutral and objective, but also friendly. Remember that this is not a psychology experiment, it’s a test of user-centered design.

NEXT: What Dana says about design.

Wesch, “Mediated Culture” EDUI Talk, Part Two

(Originally published in 2009 and still highly relevant.)

Twitter’s limit of 140 characters is one way that the medium shapes the message and the conversation. Wesch talked about the ways that digital media affect self-awareness and the construction of identity. Twitter’s “What are you doing?” contributes to “lifecasting” or “mindcasting.” In creating YouTube videos, we have no idea who, exactly, will be watching. We talk to the webcam. And when we look at YouTube videos, we are watching others without staring at them. Context collapses, as we saw in the “Bomb Iran” performance of John McCain, who thought his behavior was perfectly acceptable to the audience he was addressing in person, military veterans. But when the performance, captured digitally, appeared on YouTube, the context changed completely, as did the reception of his act.

I was particularly struck by Wesch’s comment that the anonymity and physical distance inherent in both traditional and digital media have enabled what he calls “hatred as public performance.” We see this on reality TV, in flame wars and vicious comments posted on YouTube, blogs and other places. It’s one outcome of the tide of narcissism that afflicts our culture, combined with a sense of hopelessness and impotence, the behavioral amplification of “whatever.” At the same time, he says that it’s important to have sites where comments can be posted anonymously, as a mechanism for great creativity and as a safeguard for free speech.

In spite of what seems to be a bleak analysis, Wesch is moved by what he sees as real and caring community on YouTube, and the emergence of more civil discourse through crowd ratings of the content of posts on Reddit (Wikipedia notes that “When there are enough votes against a given comment, it will not be displayed by default… Users who submit articles which other users like and subsequently “vote up” receive “karma” points as a reward…”). He’s amused and encouraged by the Free Hugs video that went viral on YouTube (as of this writing, the video has been viewed 50,748,157 times), and the kindness off the responses to a YouTube creator who spoke about her hopelessness.

There’s opportunity and danger in digital media, Wesch says, and we need to see and seize the opportunity or we’ll be washed up thanks to mainstream media. We need to be aware how the machine is using us, so we can guide the future.

Michael Wesch Looks at “Mediated Culture” at EDUI 2009

(Originally published in 2009, moved here 9/2013, and still highly relevant.)

Professor Michael Wesch’s talk at EDUI 2009, “Mediated Culture,” may well have been attendees’ favorite session.

Looking at relationships between individuals, communities, traditional mass media and the state, Wesch showed how blogging, YouTube, Twitter and other social media provide us with a new way of relating to one another, a way that has not been available in television and print.

Wesch builds on his own anthropological research in New Guinea, explaining that the creation of a state book of law changed the way individuals related to one another. When the law book was published, communities began to gather to measure up individuals against the rules and apply punishment accordingly. The book shifted focus from individuals’ relationships to one another to individuals’ relationship to the state. He says that the media are not just tools or means of communications but that they mediate our relationships. When media change, our relationships change.

Drawing on Neil Postman’s book, Amusing Ourselves to Death, Wesch says that our culture is far more the way Aldous Huxley envisioned it, rather than George Orwell. Huxley commented that it wouldn’t be necessary to ban books, because no one would want to read them.

Television became the primary medium for  the conversations of our culture, controlled by a few people and designed for “the masses.”  The conversations are always entertaining, even the serious ones. They’re punctuated by commercials and they create a culture of irrelevance, incoherence and impotence. As Huxley said, we are amusing ourselved to death. At the same time, because so much of the media is aimed at entertaining us, we feel special, flattered, as if shows have been created just for us.

Wesch shows a funny picture of school children demonstrating various states of boredom in the classroom and explores the origins and meanings of the word, “whatever.” He explains that reality TV has shaped the meaning of the word as, “who cares about you–I’ll do whatever I want.” In addition, kids have been told by their parents the world is their oyster. Becoming a contestant on reality TV has become the way to prove that it’s true. And, as all but the shows’ winners discover (and as they head into the world in their twenties), they see that it isn’t true. We see that discovery happening on every episode of American Idol and the other reality shows.

Moving to a discussion of digital media, Wesch comments that it is fundamentally different from anything we’ve seen before. We still use concepts from the past, such as referring to web “pages.” But Tim Berners-Lee said, long ago, that the Web is not supposed to be a glorified television channel.” And thanks to blogging, YouTube and other social media, anybody can create and publish anything for “the masses.” Evolving Web standards for web browsers have helped make that possible. It’s no longer necessary to know HTML and other aspects of digital technology to have visibility online.

(To be continued…)