At the beginning of 2019, we had almost finished the design of our Customer Service Saas solution for Zappos.

Read more: How we used a Design Sprint to create a Customer Service Saas solution

We were at a point where we needed user guidance to make sure we were on the right path, especially around some long-term patterns we were introducing in the tool.

I quickly created a protocol to go and test our designs with the users. Our ops team helped with the scheduling, and we were ready to start!

Method

Schedule meetings with 5 Customer Service members of varying experience who have some sort of familiarity with our Customer Service Product, for 1/2 hour interviews. 

In an interview setting, ask the interviewees to evaluate the concepts presented to them. 

Right after that, ask them to perform tasks with the given interface.

Our line-up is following these guidelines:

  • one facilitator: run the session
  • one note-taker: document the session for both conversational and behavioral takeaways
  • one observer (non-essential): this is a learning position for Research mentee
    • the observer should facilitate the next session
  • the facilitator is the only speaker. 
    • the facilitator will open the forum for questions halfway through and at the end
    • if the note taker and observer want to ask a question outside of the above forum, they should signal it to the facilitator, who will introduce the question so not to overwhelm the participant

Housekeeping

  • Remind the participant not to share any details outside of the test, as to not spread bias
  • Remember to ask tenure at Zappos and previous Customer Service experience
  • Ask if they are willing to come back to test again in the future
  • Try not to book rooms with Solstice, because of the connection difficulties and latency

Prototype to test

InVision link: [withheld]


Concept Test Primary Goals

  • Understand how the current Customer Service tool Home is valuable to agents, and why
  • Introduce the new concept for the Customer Service Platform (CSP) header interface and identify the action areas
  • Understand whether users find tabs valuable

Concepts to test

  • Do they understand the options in the header?
    • Do we need to spell it out or are icons enough?
    • What essential options are we missing?
  • Are the Status Popover and Notification badge meaningful to them?
    • Can they actually see it?
  • Is the tab-based structure of Contact handling valuable?

Success Criteria

  • They understand the header structure and can tell which option is which
  • They find the workflow broken into tabs valuable

Questions

  • Let’s assume this is the Homepage of CSP in the future, which is the first thing you see after you login and after you close a Contact. What would you like to see in it?
  • How is the current Customer Service tool Homepage useful to you?
  • Look at the header: what options do you think we have up there? Is there anything missing?
  • Tabs: how often do you have to open new tabs in your current workflow?
  • What do you think about this breakdown (point at screen)? 

Usability Test Primary Goals

  • Can agents claim an incoming email?
  • Can agents navigate through the entire flow easily?
  • When guided, would have they found the options otherwise easy to access?

Usability Test Secondary Goals

  • Can the user find and explain secondary features?

Success Criteria

  • Agents can claim an email
  • Agents can compos a response
    • include adding a blurb
  • Agents can fix issues and review the response
  • Agents can wrap-up the contact

Scenarios to test

  • A customer is sending an email about asking for a coupon. They are a popular Youtuber and want to be sponsored by Zappos.
  • Release promo and reply to customer

Questions to ask

  • How easy was going through the workflows?
  • What did you find the most painful?
  • What did you find hard to understand and how much training you think you would need to operate this new Phase of CSP?
  • How would you change what you just went through?

Limitations

  • We are testing a happy path. If this is clear, we’ll add edge cases
  • Limited click path in InVision
  • Designed with our new Design System, which is a brand new look and feel
  • 30 minutes to ask questions and go through the tests
  • Scheduled participants come from various Customer Service teams, and have different backgrounds and level of knowledge