Chatbot Pitfalls and Solid Advice on Getting it Right

According to Gartner research, it is estimated that 60.5 million Americans use a virtual assistant at least once a month. If this trend continues, chatbots will likely power 85 percent of all customer service interactions by the year 2020. 

It is said that chatbots are the new FAQ. Every second, 40,000 search queries are made online worldwide. That adds up to 3.5 billion per day or 1.2 trillion each year. That’s a lot of people looking for a lot of answers. If chatbots are built right they definitely enhance the customer experience and make it easier for a customer to find the content they need. Technical support is the most common type of chatbot content. Chatbots provide a bridge between technical documentation teams and customer support call centers. Salesforce’s Chief Digital Evangelist, @ValaAfshar predicts that “The line-of-business that is most likely to embrace AI first will be the customer service – typically the most process-oriented and technology savvy organization within most companies.” AI here refers to NLP and NLU enabled content structures.

From my vantage point at SDL, it has become increasingly clear that content really matters. The modern digital customer journey is a journey that is marked and defined by interaction with different content.  The conversational interaction is needed everywhere, not just with Alexa and other voice-based virtual assistants. Imagine searching for technical support content and finding 20 large PDFs that may contain the answer to your question. Wouldn't it be useful to have the ability to zero in on the right content within these PDFs through a series of clarifying questions?  The real question, then, is not where does the content for chatbots come from, but rather how do we prepare, organize and structure that content so the chatbots can use it? This superior delivery of the right content to many customer questions can only happen if you have a proper content organization model. At SDL we call this GCOM and it involves content architecture, organization and process to make it happen.

Source:5 ways chatbots are revolutionizing knowledge management

 Read the full post here.

However, it is very easy to build chatbots that are frustrating, useless and inefficient. We have all experienced websites where the only question the IVA (interactive virtual assistant or chatbot) is able to ask is "Hey dude, do you wanna buy my s*&t?" Chatbots are purely a reflection of the capability, fastidiousness and patience of the person who created them; and how many user needs and inputs they were able to anticipate. When done badly, which is very often at this stage of evolution, chatbots will kill customer service.

This is not so different from DIY machine translation 10 years ago. Many really bad systems were built and forced upon poor unsuspecting translators. Like MT, this too will take competence and skill and it is wise to work with experts.

Digit’s Ethan Bloch sums up the general consensus: “I’m not even sure if we can say ‘chatbots are dead,’ because I don’t even know if they were ever alive.”  

Our guest post by Ultan O'Broin is a look at how to do the right things the right way to make chatbots that are indeed useful in providing the right content to the right people and provides insight from the design and UX perspective in particular.  While the content organization issues are discussed often, it is worth a review and is well summarized in this article. Ultan has several other posts on Medium on the subject of chatbots and is worth a closer look.

What excites me the most is that once you have the IVA working properly and actually useful, it then makes sense to make it multilingual with optimized machine translation. But first things first.

===== 

Ultan O’Broin looks at how the Jobs To Be Done framework and simple build approaches for the minimum viable product can counter chatbot product management failure and fatigue.

Guerrilla testing of Snapchat Spectacles at Hook Head in Wexford, Ireland. We can try out products in the wild, but are we as focused on the job we’re hiring our product or service to do?

 

One of the compelling hopes for chatbots was that utter power of an agency to increase digital participation in a natural way, partly through reducing app fatigue. Ironically, I now believe we have reached the stage of chatbot fatigue.

Just like everyone was “doing” a smartwatch solution two years ago, suddenly chatbots are the tech “game changer” du jour.

And botification is being done just as badly as the smartwatch overkill.

Chatbot Game Chancers

There’s bot rot out there. The problem is often there is no design thinking about whether a new or existing job is worth botifying or not.

I have seen some criminal chatbots out there. From a barista training chatbot aimed at millennials that delivered a 50 page PDF manual in response to the utterance “how can I be a barista?” to that fitness chatbot telling me that my nearest Parkrun was 8,000 kilometres away, the assumption being that anyone asking was within a 300 kilometres radius of an Irish location. I was in San Francisco at the time, where there is a local run.

None of the arrivistes responsible for such rotbots have asked: “Why is this person hiring this chatbot to do this job?”

The problem is the wrong chatbots are being built, wrong. Fundamentally, begin by asking, “Why would anybody hire my chatbot to do this job?”

This article is about designing the right chatbot, right. But the profound principles to do so can be applied to any app or service.

The “User” and the Damage Done

First, the term “user” must go from the design conversation! User? WTF!

I hate that term “user”; only the usability community and illegal drug business uses it. Let’s reclaim “user” as a role in real life with a real job (unpaid or otherwise) to do. So, our “user” becomes a sales rep, a technician, a concert-goer, a mother, a parent, a patient, and so on.

But how do we find such people, especially if you’re a small startup or innovator? The answer is to think smart, get out on the street, watch real people, and ask them about what they’re doing and encourage them to articulate reasons they would use a chatbot, app, or service instead.

 

Job Profile Persecution

Forget about the job profile approach to identify your “users”.

Those unwieldy tl;dr profile documents of job titles, descriptions, qualifications, experience, org chart position, motivations, IT expertise, and so on. These tomes owe more to recruitment agencies and HR departments than to design thinking and product management.

Profiles list many tasks that the role performs, not just the focussed critical 80/20 reason why your conversational UI might really be a game changer and around which a killer solution can be built.

A job profile is not a real person. In the user experience business, it’s a statement for the prosecution of the mediocrity of design.

 

Don’t Take It Persona-ally, But…

Personas are next to go onto that “user” bonfire of the inanities.

Personas are stereotypical people, dumbed down versions of “a day in the life”, complete with stock photographs, typical responsibilities and tasks, personal characteristics, motivations, and tools.

Again, a persona is not a real digital adopter, but a distant idealisation, performing a non-prioritized list of tasks. These persona peeps may use many types of technology too; not good for innovation disruption by understanding why they might switch or integrate with your innovation.

There’s a reason the word “stereotype” usually comes with a negative connotation.

 

Get Close and Personal. Swipe Right for The Right Design

Instead of “user” personas and profiles, move closer to the customer, and discover what they do.

How? Great chatbot design begins with a contextual conversation.

Take a walk in their shoes. Try a little fun UX ethnography, guerrilla research, and the “Wizard of Oz” technique to identify that killer question or job that might inspire someone to continually turn to a conversational UI to actually do something.

Research and design a chatbot that resonates emotionally and culturally, as well as functionally for your region. Check out Hofstede’s country comparison dimensions to help you shape that experience.

For example, if we were building a chatbot for fitness enthusiasts we might hit the local gym session schedule, post to a Facebook fitness group or fitness app discussion section, or engage on Twitter or Instagram with an appropriate hashtag.

Use a deep-listening, “tell me more about that”, approach.

Original “The Wizard of Oz” movie poster. From Wikipedia; image in the public domain.

Amazon, in the run-up to the French launch of its Echo, for example, introduced Alexa to employees in its French fulfillment centres who interacted with the emergent voice assistant to help the voice chatbot learn French, cultural nuances and behaviours, and how to respond.

The launch of the French version of the Amazon Echo was preceded by real people to learn the language, cultural norms, and how they actually behaved!

Fundamentally, focus on the person’s behaviour, and not on what they say or think they want. Watch their actions. Think of this as a Lean product management approach, a way to quickly design and build a solution to determine the certainty about its value. As Eric Ries says:

“The minimum viable product (MVP) is that version of a new product which allows a team to collect the maximum amount of validated learning about customers with the least effort.”

I encourage you to read Eric’s book, “The Lean Startup” and apply the thinking to chatbot product management and design.

What Features to Design?

Why do drivers of cars buy milkshakes at drive-in takeaways? There could be many reasons. By way of Harvard professor Clay Christensen’s (and others) Jobs To Be Done framework, it was found that people don’t buy and use things because they fit a persona or user profile.

Instead, people hire a service or product to do a job for them. In our milkshake example, the drink was hired because the drivers were bored and in danger of nodding off on a long journey.

So, use the Jobs To Be Done (JTBD) approach for designing the right thing, right!

Again, read (or listen to the podcast) about Clay’s theory of disruptive innovation and Google JTBD for more.

Think of it this way, as mentioned on the startup Intercom’s blog: JTBD allows you to focus on the essential product feature, and to generate a user story about why it’s needed. The story plot line structure for a product story, using the JTBD method, would be something like:

“When X happens, I want to Y, so I can Z.”

An example from the sales rep job world might be:

“When new sales collateral comes online, I want an SMS on my phone, so I can take it on the road with me.”

Designing using JTBD enables a template approach, listing the job to be done, the role, needs, and expected job outcome. We can add outcome performance goals in there too, as a way of testing and proving ROI.

Here’s a one-page enterprise JTBD example you can turn into a template and use yourself: 

A JTBD template completed for a mobile sales rep. You can use this template approach to frame the job to be done and as a way to shape a story about the job itself.

In our example, a sales rep on the road might hire a chatbot to enter new sales prospect details rapidly without using a special device, easily capture meeting locations and images of the opportunity, and share that data in SaaS for work later. She might want to do in 5 seconds. So, a voice-driven chatbot using smartphone out-of-the-box features such as the camera and GPS capability seems like a good opportunity for a chatbot solution.

Using the JTBD approach also helps shape and communicate the job story. A story untold is not a story. Here’s your storytelling formula: SUCCESS — Simplicity, Unexpectedness, Concreteness, Credibility, Emotion, Story.

Phrase your job story well; using bullet points is fine. As John Dewey put it:

“A problem well put is half solved.”

You can tell you JTBD story to customers, product managers, developers, investors, designers, or any stakeholder (although C-level management will probably just want an executive picture on the return on investment for their budget spend).

Competition is Fierce, from Unexpected Combatants

Remember: There is competition with other tools for hiring your product or service.

In the case of our sales rep, she will still carry a notebook and pen — it helps recall — sticky/Post-it notes, an Apple iPad, a Windows laptop, and a Samsung smartphone, maybe.

This hire competition is at the job level, it’s not about the category the tool fits into. As Intercom points out, Microsoft Skype, for example, addresses the same purpose as an airline seat for a business trip— communicating with colleagues.

Chatbots compete against other tools and methodologies on the job hire level in many arenas — communicating, scheduling engagements, ordering, onboarding of employees, managing things to do, marketing, educating, entertaining, finding simple solutions and fixes — and must do so without special training, equipment, and so on.

But fundamentally, the JTDB approach really means the end of design as many people have come to think they know it. Design now becomes about the job the chatbot is being hired to perform by a human.

tl;dr? OK, Summary

So, to design and build the right product right, remember these six simple points.

 

Build the Right Thing

1. Use the JTBD framework: Why is this chatbot (or other product or service) being hired to do the job by this person? Watch their actual job behaviour, ask, and listen.

 

Build It Right

2. Get to the MVP stage fast. Apply proven UI principles for chatbots such as creating prompts and messages with a conversational style, shape a personality, pick use cases based on speed, simplicity, pain alleviation, and that requires no special device or training.

3. Have a clear primary job to be done. What is the 80/20 effort of the job? Avoid corner cases, nice-to-haves, and “what ifs?”.

Consider Microsoft Excel; a super-popular desktop spreadsheet and now service-offering, packed with features. But how much of that functionality do you use and how often? Probably 20% of the features or less to do 80% of the jobs you need Microsoft Excel for. There are other use case examples from the enterprise to get you thinking along those lines for chatbots.

4. Sketch and wireframe your solution first. Balsamiq, for example, offers great digital solutions, and there are now conversational UI specific options. But all you really need to start is a pen, paper, and ideas. You do not need to be an artist.

Using wireframes means you can also apply familiar UX design patterns, making for productive development. Document any open questions, use an Agile backlog and try collaborative, integrated cloud tools like Slack, or whatever suits your context of work, to agree your sketch with stakeholders.

Balsamiq wireframing stencils. There are awesome tools for designers, developers, product managers to sketch and agree on that chatbot JTBD

 5. Use an accelerator design kit or platform with suitable backend AI and NLP capability to innovate fast. Leverage usability heuristics, and use real text and voice scripts in your designs — in the language of the hirer— and iterate with the key stakeholders to agree on a chatbot solution before you code anything. This agreement eliminates surprises later!

Smartly.ai, a platform for creating conversational UI chatbots for SaaS, across different devices.

6. Don’t dribblise your design. With chatbots and conversational interactions stay true to the idea that no UI is the best UI of all. The UX design toolkit is now API calls, AI, NLP, ML, integrations with services and regular device features (GPS, SMS, camera, and the microphone, for example). Design platforms and kits often provide any UI widgets if and when needed (for maps, attachments, avatars, and so on).

No UI is the best UI to create that killer new user experience.

 

Job Satisfaction? Test It!

Finally, do remember to test your innovation.

UI heuristics and baked-in platform usability can do some heavy lifting to prove your idea’s value, but for JTBD the most reliable testing of having solved a design problem are tests that focus on real tasks by real people doing real jobs.

Plan to test your innovation before going live and after, and iterate if needed. Keep focused on the JTBD. If that is wrong, then no amount of fancy visual UI design is going to fix it and improve switching or adoption from another app. As Seymour Chwast puts it:

“If you dig a hole and it’s in the wrong place, digging it deeper isn’t going to help.”

Remember, people are hiring your chatbot to do a job. Hiring. In the age of the cloud, it’s easy to switch chatbots, apps, and services because of a subscription model.

There’s competition for that apps, services, and chatbot job hire. So, be competitive!

=======

Ultan O’Broin (@ultan) is a digital transformation and user experience consultant working with startups and the STEAM community, specialising in conversational UI and PaaS4SaaS. With over 20 years of product development and pre-sales design thinking outreach with major IT companies in the U.S. and Europe, he also speaks and blogs about UX and technology.

Watch out for ConverCon 2018 in Dublin, Ireland in September 2018 (previously reviewed), an event with chatbots at its heart. I might see you there to chat in person about the JTBD of your chatbot!

https://www.linkedin.com/in/ultanobroin/

All images and screen captures in this article are by Ultan O’Broin. Copyright or trademark ownership vested in any screen capture is acknowledged, where applicable.

Post-Editing and NMT: Embracing a New Age of Translation Technology

This is a guest post by Rodrigo Fuentes Corradi and Andrea Stevens from the SDL MT  project management team. 

SDL is unique amongst LSPs as they have both deep MT development expertise, and also have a large pool of in-house professional translators who can communicate much more easily with the MT development and management team on engine specifics, and provide the kind of feedback that leads to best practices in MTPE projects and delivers overall superior quality. MT in many localization contexts only works if it indeed delivers output that is useful to translators rather than frustrates them. This quality is best achieved by an active and productive (constructive) dialogue between translators, project managers, and developers, which is the modus operandi at SDL. 

While most MT developers have a strong preference for using automated quality metrics like BLEU and LEPOR, the SDL MT project management teams have discovered many years ago, that competent human assessments are much more reliable to determine if an MT system is viable or not for an MTPE project. They have developed very reliable methods to make this determination efficiently and effectively.  NMT presents special challenges in MTPE use scenarios, as the automated metrics are generally even less reliable than they have been with SMT. NMT fluency can sometimes veil accuracy and thus special training and modified strategies are needed as this post describes. Also, we often see that the quality of the MT output can be significantly better than a BLEU score might suggest as this metric is often lower for NMT systems for various reasons.

We are beginning to see other research that suggests that NMT does often provide productivity benefits over SMT, but that it requires special updated training and at least some understanding of the new challenges presented to established production processes by the compelling but sometimes inaccurate fluency of NMT systems.

 =====

Post-editing means different things to different people. To the corporate world, it means making localization budgets go further by translating more for less. To freelance translators, it may well mean an infringement of their craft and livelihood. We know that freelance careers are the result of several years of study, hard work and perseverance to create a client portfolio based on a reputation for quality, consistency and on-time delivery.

Language Service Providers like SDL are often stuck in the middle, working to satisfy the complex demands of our clients while nurturing the linguistic and creative talent in our supply chain. The truth is that in today’s localization market, Machine Translation Post-Editing (MTPE) is very much a reality and an answer to the unstoppable content explosion that we are experiencing.

How do we find a balance and how can we, as an LSP committed to machine translation (MT) and post-editing, do the impossible and manage both client expectations and supply chain needs?
At the heart of the conundrum lies content. What content needs to be developed and translated, what is its purpose, who does it serve, and how can our clients bring it to their customers in the most appropriate and cost-efficient way possible? Content and how well it communicates to a local audience is what defines a company in today’s fast-moving markets. As a business changes and evolves, so does content, creating the need for almost constant innovation and reinvention. This undoubtedly poses a challenge for translators who need to commit to life-long learning to understand new concepts, trends, and challenges and produce materials in the target language that are fully adapted to local markets and audiences.

Content keeps growing exponentially and our challenge is to understand the vast amounts of diverse content that our customers create and how to best deal with it. All content has value and answers a specific requirement, but for example, there is a difference between content created for a technical knowledge base, content for an advertising feature or even regulatory content. For content that has a short shelf-life, a straight MT solution without human intervention may be perfectly acceptable whereas, at the other end of the spectrum, more creative content may require specialized translators or transcreation. In between, there is a wide range of content that could be best served with a hybrid human and machine approach.

As a translation work-giver and machine translation provider, it’s up to us at SDL to navigate the challenges posed by the ongoing content explosion in partnership with our clients and our supply chain. We need to be realistic about the role that MTPE plays in today’s translation marketplace and acknowledge the advantages for all involved. At the same time, challenges and constraints cannot be swept under the rug, but need to be openly addressed and discussed for full transparency.

“While NMT is inspired by the way the human brain works, it does not learn languages quite like humans do – humans learn to speak to communicate with each other in a wide social context.”

One of the challenges is to make post-editing sustainable. Post-editing is established as a standard solution for a wide variety of content types and language pairs. The application of MT is constantly pushed further by the commercial need to respond to the overall content growth while dealing with limited or unchanged localization budgets. As a result, the MTPE footprint continues to expand beyond initially successful languages and domains into new territories, such as regulatory life sciences content or even marketing.

To do this successfully, we rely on technology advances and improvements, some of which have only become possible over the last one to two years. Customized solutions and real-time adaptive machine translation are among the tools that improve the post-editing experience for translators, but the biggest step forward is surely the arrival of Neural Machine Translation.

Neural Machine Translation

Neural Machine Translation (NMT) has rightly been described as a revolution rather than an evolution. With its core developed entirely from scratch, NMT offers amazing opportunities for innovation. Its powerful architecture paradigm does not only capture text or syntax information but actual meaning and semantics, leading to the improvements in translation quality that we are seeing across the board. This is something all MT providers, including SDL, agree on after extensive testing across language pairs and combinations.

How is NMT achieving this? In short, NMT uses artificial neural networks which are based on mimicking the human brain with its interconnected neurons that help us understand the world around us, what we see, touch, smell, taste or hear. An NMT system learns from observing correlations between the source and the target text and modifies itself to increase the likelihood of producing a correct translation. While NMT is inspired by the way the human brain works, it does not learn languages quite like humans do – humans learn to speak to communicate with each other in a wide social context. NMT systems are still trained on bilingual data sets but promise noticeable uplift in translation quality through a more efficient framework for learning translation rules. NMT systems use an input layer where the text for translation is fed into the system, a hidden layer or multiple hidden layers where processing takes place and an output layer where we obtain the translation.

“When meaning and semantics are represented through math, words with similar meanings tend to cluster together.”

 

The hidden layer contains a vast network of neural nodes where the input is encoded into a vector of numbers to give a predictive output. Essentially, we are applying math to the problem of language and translation. When meaning and semantics are represented through math, words with similar meanings tend to cluster together. This is how we know that the NMT system starts to learn the semantics of words. When words have several meanings, they appear in different clusters; for example, ‘bank’ can appear in the geography or the finance cluster. It is then even possible to apply further math to the vectors, as shown in the graphic below: if you take the vector ‘king,’ subtract the vector ‘man’ and add the vector ‘woman,’ the result is a vector that is exactly or very close to the vector ‘queen.’

 For NMT, we use deep neural networks, which are better for long-range context and dependencies. This is particularly important when it comes to languages for which the benefits of statistical machine translation were limited. Good examples are language pairs with long distance word traveling such as Japanese, where the clause structure is very different from English, or languages with long-distance dependencies such as German and Dutch.

One of the main advantages of NMT is the very fluent translation output. However, it is important to understand that the very fluent output can sometimes mask the fact that the content of the automated translation is not correct.

This is just one of the reasons why post-editing is still so important, even when working with NMT output. It is essential that translators understand the behaviors and patterns of NMT to be able to take full advantage of this promising technology.

To prepare translators for working with NMT, training and active engagement with the supply chain is essential. This is part of a much larger training effort that includes our soon to be updated Post-Editing Certification Program. Training will not only help our vendors to prepare for working with new technologies but also ensure that everyone is ready for the technological expansion of MTPE.

Working Together

Being a good work giver is key to jointly face new challenges along with our supply chain. Increasingly, this means guiding projects with an expert understanding of tools, domains, processes and intended audience. The combination of these factors will bring sustained success and quality continuity.

New technologies such as NMT can prove disruptive; transparency and training will be key to reassure and prepare our vendors. One key challenge will be to align the improved NMT technology with post-editing experience and a strategy for on-boarding our supply chain.

The intent is to utilize NMT technology across a wide range of domains and content types, and it is important to collect valid data points to proactively assess the chances of success before reaching out to freelancers. Furthermore, when approaching freelancers, it has always been helpful to share these findings and provide guidelines for MT behaviors that can help their post-editing decisions.

In summary, so much of the success is centered on good communication, which is dependent on openness, sharing materials and providing channels to discuss issues and answer questions. In this respect, we see a responsibility to drive these initiatives with a structured communication plan that includes webinars and open days held in our language offices.

 We are witnessing the ongoing growth of Artificial Intelligence and Machine Learning in many aspects of our daily lives, from self-driving cars to medical diagnosis and intervention. Technology is a huge part of how we live and what we do, and this particularly holds true for translators. Post-editing works at the intersection of humans and machines, and machine translation is one of the most advanced tools in the translators’ toolbox to future-proof the profession for a new generation of translators. MTPE is, of course, a choice that everyone needs to make for themselves, but with new technologies such as Adaptive or Neural MT working for translators and the growing reach of MT into new domains, this is not a choice to be taken lightly. Technology developments are not reducing the role of translators, but rather, are changing and enhancing it, opening a host of new opportunities. This is a journey we need to embark on together for continued learning, support, and feedback.

Rodrigo Fuentes Corradi

MT Business Consultant, SDL

Andrea Stevens

 MT Translation Manager, SDL

 

  The authors have produced a white paper on "Best Practices for Enterprise Scale Post-Editing" that can be accessed at this link.