Intercom Operator

In August 2017, Intercom launched Operator - a bot to help simple tasks in business conversations. It helps businesses to ask for contact details when they’re not around; Recommend articles to answer simple questions; And asks for feedback after support teams resolved the conversation.

I designed the product system and interaction for the article suggestions skill, and put the few Operator skills together into one holistic system for launch.

Few weeks since launch, over 1600 customers adopted the features and helped resolving ~10% of support conversations on average.

In this case study I’ll be mainly highlighting the design details that went into designing the article suggestion system. It’s the most valuable Operator feature, and it’s the main starting point for me in this project.

Article suggestions

Motivation

The idea of suggesting relevant articles to answer people’s question has always been something that we wanted to do for Educate. In fact, we’ve already built a system to suggest articles for support teams as part of the initial launch. But It wasn’t ready to be used for suggesting directly to end-users yet, due to some uncertainties regarding suggestion quality, and how end-users will perceive the idea of a bot in business communication.

Post Educate launch, we’ve decided to refocus on this product area as it was the most exciting and impactful next step for the product. With the available time, we had the opportunity to explore design solutions more throughoutly, and perform researches to make sure our solution would do the job without harming the end-users experience.

First round solution

In the ideal scenario, when someone asks a question, the system would be smart enough to suggest the right answer directly to the users.

Our initial strategy was that it would only surface an article when it has a high confidence score because we assumed people will have a low tolerance for bad suggestions. For articles with lower confidence score, we’d suggest them to support teams who can humanly filter the right article to answer with.

Learnings

We ran concept tests on the design prototype to find out how people would perceive this interaction. In terms of user experience, it was really positive when the article was the answer they needed:

However, shortly after implementing the experiment, we quickly learnt that this strategy suffers from two main problems:

  1. Cold start problem - The suggestion system learns from past conversations. For some beta customers who didn’t have too much conversations yet, it takes a long time for the system to learn before it could provide highly relevant articles.
  2. Low chance of providing the right answer with a single suggestion - With the way the design only allowed for one article suggestion, and that we’ve set a high threshold for suggestion, the occasion where the system could actually suggest articles is very low. Almost close to zero.

Second round solution

Informed by the learnings, we’ve decided to adjust our approach:

This approach brings an interesting interaction design challenge for me to solve - what’s the best experience for delivering multiple article cards in the messenger?

Design explorations

Taking inspirations of how Facebook Messenger handles multiple content cards, we could try to introduce a carousel design pattern for the job.

Alternatively, we could introduce a stacked cards pattern that expands to reveal multiple suggestions.

Learnings

As much as we thought people would be familiar with the carousel design pattern, results from user tests suggested that people actually preferred the second prototype, because it allows them to scan all the results quickly.

With regards to the experience of receiving less relevant articles - surprisingly, none of the participant actually really put off by the fact that they received automatic suggestions that don’t answer their questions. In fact, they were pretty positive that they received some attempts of answers while the team wasn’t available for an immediate respond. As long as they understood that their questions will reach the support teams, they actually don’t mind trying self-serve while they wait.

Third round solution

The learnings really gave us a lot of confidence to validate our approach, and we’ve even double down on this route:

In terms of design, knowing that being able to scan all the article suggestions quickly provides better user experience, I’ve iterate the design of it with the concept of bundling multiple content card into a single card (Similar to how WeChat delivers multiple content card).

This solution feels much closer to what we wanted to achieve both in terms of design and technical solution. In the background, we were running beta and tracking performances to make sure we’re on the right track.

Suggestion feedback loop

Providing suggestions isn’t the end state of the interaction. We needed a way for users to inform the system if their question has been answered by the suggestions, so that:

The design challenge lies in how might we design the most appropriate interaction for asking if people got their answer, and how might we let the system understand which article out of the multiple articles was the correct answer?

We came up with ideas such as:

Having evaluated the pros and cons of each approach, we clearly felt that we had a consent on a product principle - We shouldn't let users do the work for us.

Interestingly, the interaction with Zendesk’s Answerbot (that launched few weeks after Operator) went with the opposite path.

This is the interaction we went with in the end:

After users opened, read, and closed an article, the bot would wait for few seconds (to make sure they’re not about to read another suggestion) and ask a simple question to see if their question is answered. The UI would then present a Yes/No structured reply options above the reply composer.

If they reply with Yes, we’d count that as a successful suggestion and close the conversation. In the follow up response, the bot will inform users that the conversation is now ended unless they send another message.

If they reply with No, or with any free-form reply, we’d assume they are still seeking support from human, and would therefore show a follow up response to let them know again when they would expect a reply from the team.

This interaction has been tested well in qualitative research, and we’re seeing ~56% of response rate consistently (Which is pretty high!).

Behavioural design

Throughout the design process, there were actually a lot of design thoughts into how the bot should behave. These details aren’t immediately visible, but made the product more considered and behaving with “manners” at a result.

To highlight a few manners that we’ve “designed”:

These manners, we believe, are the key to make the bot as user-friendly as possible.

Operator system design

Up until this point, we’ve developed a decent bot feature for automatically answering people’s questions. However, there’re still some further work to be done before it’s ready to launch.

The concept of a bot existed within Intercom for other purposes like capturing contact detail from a visitor, and collecting feedback at the end of a conversation. They were designed by different teams during different time, and so there wasn’t a holistic system to describe how these features would work together. The settings for these features are scattered across different places.

We took the opportunity to rationalise the different skills together as a single system.

Decision tree

Initially we tried to use a single decision tree to encapsulate how it could chain together the set of different flows.

However, it was hard to achieve because the system had a few important branching logic based on certain context. For example:

These multiple possible context introduces multidimensional complexity that quickly outgrows the single dimension decision tree.

Context mapping

In the end, we found it more manageable to lay out the possible context condition combinations, and then map out the appropriate Operator flows.

This may not be the most scalable way to map the all the Operator flows, but it worked for us to be able to visualise all the different possible paths and Operator responses in each scenario. It has also been extremely useful for engineers to implement the system to work as mapped.

Operator Home

Lastly, to solve the problem of settings being scattered across different places, we’ve designed and built a centralised place for companies to learn about what Operator can do and manage the settings for each individual skill.

🤖LAUNCH🤖

Finally, we felt the product was ready to go. As the industry was hyped about bots that try to do too much too helpful, our bot focuses on a few things well. It was really exciting for us to get it out there and see how the market reacts to our approach.

Launching was just a beginning. The ground work sets us up to extend on the skills that Operator offer in the future, and to think about how might we potentially enable our customers to build new skills.

This product is another massive effort from the team. I had the pleasure to collaborate with:

As usual, let me just finish the case study with some nice sentiments from Twitter 😃