Background: Amazon’s invisible problem
You know Amazon.
You’ve probably used their apps and devices.
You may not be aware of the army of customer service agents they educate and employ.
For more than ten years, Amazon has created eReaders, Tablets, and Apps at a staggering pace. Every year, they introduce new products and update their software. With millions of devices and apps running hundreds of different software versions, the complexity customer service agents deal with is monumental and ever-growing.
This is the invisible problem:
Amazon has created a staggeringly complex global ecosystem of devices and software.
As a result, it needs to equip customer service agents to effectively help customers who may call with any combination of device and OS version.
Amazon’s training tools were simply not up to this task.
Problem
Customer service agents (CSAs) need quick access to each device and each OS version to serve customers who are calling in with a problem. These customers could call with brand new tablets or Kindles that are more than a decade old.
Experience showed Amazon that lockers of physical devices were not scalable or cost effective and caused long, frustrating calls for customers. “...please hold while I track down a Kindle from 2008 and attempt to load the OS you’re using…”
To combat this, Amazon hired a company to create browser based device simulations. They fired this company after it couldn’t keep its simulations up-to-date. A second company was hired and subsequently fired. Both companies delivered generic simulations that took far too long to build and couldn’t be updated without costly engineering hours.
Prior to the Project
CSAs were using device simulations that were years out of date. This resulted in frustrated customers whose needs were not met.
Lockers with physical devices existed, but during product launches there was often ONE device to TEN THOUSAND agents. While this is an extreme example, it demonstrates the systemic problem Amazon faced.
Of course, written help documentation existed, but, as CSAs and their managers will tell you, there is no replacement for interacting with a device.
Research
Prior to my arrival, Amazon tried to solve their physical device shortage by hiring two agencies to build browser based simulators that could be accessed by agents.
Interviewing management, I learned that simulators were browser based to support...
Consistency:
Almost all of the tools available to CSAs were browser-based.
Almost all of the tools available to CSAs were browser-based.
Performance:
Emulations couldn’t be run on computers with such poor GPUs and it would cost Amazon huge amounts of money to upgrade these computers.
Emulations couldn’t be run on computers with such poor GPUs and it would cost Amazon huge amounts of money to upgrade these computers.
Security:
Amazon didn’t want emulators running actual device code and calling real Amazon services
Amazon didn’t want emulators running actual device code and calling real Amazon services
Brainstorming/Exploration
No Code:
The failure of previous simulators showed us that front-end code was not a scalable or sustainable approach for this project.
The failure of previous simulators showed us that front-end code was not a scalable or sustainable approach for this project.
Editable/Approachable:
We needed a less technical approach so that non-engineers could build the simulations and, just as importantly, keep them updated.
We needed a less technical approach so that non-engineers could build the simulations and, just as importantly, keep them updated.
Scalable:
We need projects that could easily branch with OS updates, run in every browser which CSAs use and accommodate future localization.
We need projects that could easily branch with OS updates, run in every browser which CSAs use and accommodate future localization.
Choosing the Right Tool
We evaluated UX prototyping platforms to find one that was simple enough for our designers and powerful enough to provide accurate simulations for our CSAs. Here’s what we found:
InVision was too simplistic.
We liked its drag and drop interface, but its limited functionality and inability to handle variables or multi state interactions eliminated it from consideration.
Framer was too complex.
While powerful, Framer requires knowledge of Javascript. It was apparent that we would either need to hire front-end engineers or try to train designers to code. With the previous simulator failures in mind, we moved on.
Proto.io was just right.
We chose Proto.io because it had a simple drag and drop interface and was robust enough to handle complex interactions. Furthermore, projects could be exported and hosted on Amazon’s servers easing security concerns. Finally, projects could easily be branched to accommodate OS updates and localization.
Customer Validation
Over the next 2 months we built a minimum viable product: A Fire Tablet simulation with 500 screens and user flows for all of the CSA’s most common contact drivers.
User testing showed us that...
Content is more important than fidelity.
All content and interactions must exist in the simulation, in the correct place, but that polish does not lead to better learning outcomes. For example, we found that perfectly recreating motion within the device was not necessary.
All content and interactions must exist in the simulation, in the correct place, but that polish does not lead to better learning outcomes. For example, we found that perfectly recreating motion within the device was not necessary.
Depth of interactivity is key.
We built a menu system that showed CSAs the location of every accessibility feature. However, CSAs let us know they were still not able to help customers learn to use these features. As a result we built much more interactive accessibility features and menus. CSAs reported that they were better able to serve customers.
We built a menu system that showed CSAs the location of every accessibility feature. However, CSAs let us know they were still not able to help customers learn to use these features. As a result we built much more interactive accessibility features and menus. CSAs reported that they were better able to serve customers.
Performance is paramount.
CSAs want tools that work instantly and were very frustrated with long load times. That said, our simulators can have as many as 1,500 screens some loading is inevitable. To ease this issue, I worked to optimize the files we used and replaced heavy elements (hi-res JPGs) with two-color PNGs. We were able to maintain the learning experience while cutting load times by 50 to 80 percent.
CSAs want tools that work instantly and were very frustrated with long load times. That said, our simulators can have as many as 1,500 screens some loading is inevitable. To ease this issue, I worked to optimize the files we used and replaced heavy elements (hi-res JPGs) with two-color PNGs. We were able to maintain the learning experience while cutting load times by 50 to 80 percent.
Metrics
Post use survey for CSAs - 4 of 5 rated it as very useful.
Each week, more than 75% of CSAs return to use the simulators.
Next Steps
In the year and a half following our successful test, we created simulators for 16 devices and Apps.
We were asked to localize our simulators to 8 languages in addition to English. I asked for permission to access the text strings and server Amazon uses for localization, but was unequivocally denied. Thus, I created a system for gathering and implementing localized content. I was the sole localizer at first, but in time was allowed to hire, train and oversee a small team of production designers.
Results
When I left, Amazon had an up to date, scalable and growing ecosystem of device simulations.
- 81% of CSAs believed that the simulators reduce call times.
- 76% of CSAs believe simulators allow them to help customers more effectively.
- CSAs have used the simulators on more than 750k calls.
CSAs feel better equipped to help customers, call times are lower, and Amazon has saved huge amounts of money on shipping and maintaining devices. While I don’t know exactly how much the simulator project is responsible for these trends, I do know that Amazon has ordered simulators for each new device they make and that the team has grown from just me to 4 full-time designers. So was this project successful? Amazon seems to think so.
If you'd like a demo of these simulators, just drop me a line or just check out the screen capture of my simulators below: