In August 2017, Intercom launched Operator - a bot to help simple tasks in business conversations. It helps businesses to ask for contact details when they’re not around; Recommend articles to answer simple questions; And asks for feedback after support teams resolved the conversation.
I designed the product system and interaction for the article suggestions skill, and put the few Operator skills together into one holistic system for launch.
Few weeks since launch, over 1600 customers adopted the features and helped resolving ~10% of support conversations on average.
In this case study I’ll be mainly highlighting the design details that went into designing the article suggestion system. It’s the most valuable Operator feature, and it’s the main starting point for me in this project.
The idea of suggesting relevant articles to answer people’s question has always been something that we wanted to do for Educate. In fact, we’ve already built a system to suggest articles for support teams as part of the initial launch. But It wasn’t ready to be used for suggesting directly to end-users yet, due to some uncertainties regarding suggestion quality, and how end-users will perceive the idea of a bot in business communication.
Post Educate launch, we’ve decided to refocus on this product area as it was the most exciting and impactful next step for the product. With the available time, we had the opportunity to explore design solutions more throughoutly, and perform researches to make sure our solution would do the job without harming the end-users experience.
In the ideal scenario, when someone asks a question, the system would be smart enough to suggest the right answer directly to the users.
Our initial strategy was that it would only surface an article when it has a high confidence score because we assumed people will have a low tolerance for bad suggestions. For articles with lower confidence score, we’d suggest them to support teams who can humanly filter the right article to answer with.
However, shortly after implementing the experiment, we quickly learnt that this strategy suffers from two main problems:
Informed by the learnings, we’ve decided to adjust our approach:
This approach brings an interesting interaction design challenge for me to solve - what’s the best experience for delivering multiple article cards in the messenger?
Taking inspirations of how Facebook Messenger handles multiple content cards, we could try to introduce a carousel design pattern for the job.
Alternatively, we could introduce a stacked cards pattern that expands to reveal multiple suggestions.
As much as we thought people would be familiar with the carousel design pattern, results from user tests suggested that people actually preferred the second prototype, because it allows them to scan all the results quickly.
With regards to the experience of receiving less relevant articles - surprisingly, none of the participant actually really put off by the fact that they received automatic suggestions that don’t answer their questions. In fact, they were pretty positive that they received some attempts of answers while the team wasn’t available for an immediate respond. As long as they understood that their questions will reach the support teams, they actually don’t mind trying self-serve while they wait.
The learnings really gave us a lot of confidence to validate our approach, and we’ve even double down on this route:
In terms of design, knowing that being able to scan all the article suggestions quickly provides better user experience, I’ve iterate the design of it with the concept of bundling multiple content card into a single card (Similar to how WeChat delivers multiple content card).
This solution feels much closer to what we wanted to achieve both in terms of design and technical solution. In the background, we were running beta and tracking performances to make sure we’re on the right track.
Providing suggestions isn’t the end state of the interaction. We needed a way for users to inform the system if their question has been answered by the suggestions, so that:
The design challenge lies in how might we design the most appropriate interaction for asking if people got their answer, and how might we let the system understand which article out of the multiple articles was the correct answer?
We came up with ideas such as:
Having evaluated the pros and cons of each approach, we clearly felt that we had a consent on a product principle - We shouldn't let users do the work for us.
Interestingly, the interaction with Zendesk’s Answerbot (that launched few weeks after Operator) went with the opposite path.
This is the interaction we went with in the end:
After users opened, read, and closed an article, the bot would wait for few seconds (to make sure they’re not about to read another suggestion) and ask a simple question to see if their question is answered. The UI would then present a Yes/No structured reply options above the reply composer.
If they reply with Yes, we’d count that as a successful suggestion and close the conversation. In the follow up response, the bot will inform users that the conversation is now ended unless they send another message.
If they reply with No, or with any free-form reply, we’d assume they are still seeking support from human, and would therefore show a follow up response to let them know again when they would expect a reply from the team.
This interaction has been tested well in qualitative research, and we’re seeing ~56% of response rate consistently (Which is pretty high!).
Throughout the design process, there were actually a lot of design thoughts into how the bot should behave. These details aren’t immediately visible, but made the product more considered and behaving with “manners” at a result.
To highlight a few manners that we’ve “designed”:
These manners, we believe, are the key to make the bot as user-friendly as possible.
Up until this point, we’ve developed a decent bot feature for automatically answering people’s questions. However, there’re still some further work to be done before it’s ready to launch.
The concept of a bot existed within Intercom for other purposes like capturing contact detail from a visitor, and collecting feedback at the end of a conversation. They were designed by different teams during different time, and so there wasn’t a holistic system to describe how these features would work together. The settings for these features are scattered across different places.
We took the opportunity to rationalise the different skills together as a single system.
Initially we tried to use a single decision tree to encapsulate how it could chain together the set of different flows.
However, it was hard to achieve because the system had a few important branching logic based on certain context. For example:
These multiple possible context introduces multidimensional complexity that quickly outgrows the single dimension decision tree.
In the end, we found it more manageable to lay out the possible context condition combinations, and then map out the appropriate Operator flows.
This may not be the most scalable way to map the all the Operator flows, but it worked for us to be able to visualise all the different possible paths and Operator responses in each scenario. It has also been extremely useful for engineers to implement the system to work as mapped.
Lastly, to solve the problem of settings being scattered across different places, we’ve designed and built a centralised place for companies to learn about what Operator can do and manage the settings for each individual skill.
Finally, we felt the product was ready to go. As the industry was hyped about bots that try to do too much too helpful, our bot focuses on a few things well. It was really exciting for us to get it out there and see how the market reacts to our approach.
Launching was just a beginning. The ground work sets us up to extend on the skills that Operator offer in the future, and to think about how might we potentially enable our customers to build new skills.
This product is another massive effort from the team. I had the pleasure to collaborate with:
As usual, let me just finish the case study with some nice sentiments from Twitter 😃
7/ Killer example of Intercom keeping humans at the center of support paying off in a big way. Great job!— Reza Khadjavi (@rezakhadjavi) 9 August 2017
A great day! @intercom released their Operator bot today 😎 This is exactly what I've been waiting for.— Scott Underwood (@ThunderwoodSays) 9 August 2017