Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Leveraging conversational technology to answer common COVID-19 questions

Leveraging conversational technology to answer common COVID-19 questions Downloaded from https://academic.oup.com/jamia/article/28/4/850/6017172 by DeepDyve user on 19 July 2022 Journal of the American Medical Informatics Association, 28(4), 2021, 850–855 doi: 10.1093/jamia/ocaa316 Advance Access Publication Date: 1 February 2021 Brief Communication Brief Communication Leveraging conversational technology to answer common COVID-19 questions 1 1 1 1 Mollie McKillop, Brett R. South, Anita Preininger , Mitch Mason, Gretchen 1,2 Purcell Jackson 1 2 IBM Watson Health, Cambridge, Massachusetts, USA and Vanderbilt University Medical Center, Nashville, Tennessee, USA Corresponding Author: Mollie M. McKillop, PhD, MPH, 51 Astor Place, Floor 4, New York, NY, USA (mollie.mckillop@ibm.- com) Received 3 September 2020; Revised 17 October 2020; Editorial Decision 18 November 2020; Accepted 28 November 2020 ABSTRACT The rapidly evolving science about the Coronavirus Disease 2019 (COVID-19) pandemic created unprecedented health information needs and dramatic changes in policies globally. We describe a platform, Watson Assistant (WA), which has been used to develop conversational agents to deliver COVID-19 related information. We char- acterized the diverse use cases and implementations during the early pandemic and measured adoption through a number of users, messages sent, and conversational turns (ie, pairs of interactions between users and agents). Thirty-seven institutions in 9 countries deployed COVID-19 conversational agents with WA be- tween March 30 and August 10, 2020, including 24 governmental agencies, 7 employers, 5 provider organiza- tions, and 1 health plan. Over 6.8 million messages were delivered through the platform. The mean number of conversational turns per session ranged between 1.9 and 3.5. Our experience demonstrates that conversational technologies can be rapidly deployed for pandemic response and are adopted globally by a wide range of users. Key words: telemedicine, COVID-19, public health, pandemics, chatbots, conversational technologies INTRODUCTION Information needs Timely and accurate public health information related to COVID- A new coronavirus causing severe acute respiratory syndrome 19 is universally needed across stakeholders. Organizations have (SARS-CoV-2) has infected millions worldwide with coronavirus been asked to provide information on COVID-19, its symptoms, disease 2019 (COVID-19) and caused significant mortality. SARS- how it spreads, strategies for prevention, and how each organization CoV-2 is a highly contagious pathogen, with widely variable clinical is responding to the pandemic. Trusted sources of health informa- manifestations. In March 2020, the World Health Organization tion, like medical practices, have limited in-person visits to focus on (WHO) classified COVID-19 as a pandemic. In the absence of treating the sick and reducing disease spread. Staff reductions have proven therapies or a vaccine, public health departments, govern- further compounded availability to answer questions. The pervasive- ments, employers, and healthcare institutions have taken measures ness of the pandemic has resulted in organizations assuming new to control the spread of the disease, including providing information and promoting nonpharmaceutical interventions, such as social dis- roles related to the dissemination of public health information. tancing and hand washing. Given the novelty of the disease, infor- Given the enormous demand for information about COVID-19, mation is rapidly evolving, with new evidence often contradicting many stakeholders have leveraged emerging conversational technol- earlier findings. These inconsistencies create uncertainty, leading to ogies to automate responses to common COVID-19 related ques- a need for trustworthy, health-related information. tions and information needs specific to their organizations. V The Author(s) 2021. Published by Oxford University Press on behalf of the American Medical Informatics Association. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted reuse, distribution, and reproduction in any medium, provided the original work is properly cited. 850 Downloaded from https://academic.oup.com/jamia/article/28/4/850/6017172 by DeepDyve user on 19 July 2022 Journal of the American Medical Informatics Association, 2021, Vol. 28, No. 4 851 Chatbots in healthcare dition) that answer the question within the dialogue interface or find the most relevant answer in its knowledge base. One way to scale dissemination of COVID-19 related information is WA is built upon a core set of functionalities with 3 main com- through technologies that employ natural language conversation. ponents that facilitate dialogue with users. The first component is Chatbots, sometimes called conversational agents or virtual assis- the intent, which defines the type of information sought. The second tants, often differ in functionality. Consensus on a taxonomy of 2,3 component includes an entity that is used to provide a precise re- these conversational technologies is lacking. The simplest chat- sponse for an intent. The final component is dialogue, which is the bots are capable of matching a predetermined set of topics with pre- actual conversation a user has with the conversational agent. WA’s defined answers, whereas more sophisticated conversational agents proprietary NLP capabilities facilitate creation and training of con- expand on their functionalities to employ machine learning and nat- versational agents with a minimal amount of data. Agents can be de- ural language processing (NLP) to understand questions in everyday livered in any cloud environment, allowing users to maintain language and engage users in increasingly complex conversations. ownership and privacy of their data. The technical details of func- For example, conversational agents may understand meaning, main- tionalities and implementation are beyond the scope of this brief tain context in dialogue, and learn with time to improve their per- communication but are provided elsewhere. formance. Some describe conversational agents that aid users in performing specific tasks as virtual assistants; they often have a characteristic personality expressed by tone, dialect, or style in con- Conversational agents for COVID-19 versation. Such conversational tools have demonstrated promise in Beginning in March 2020, IBM offered a program called Citizen As- clinical applications, including chatbots for determining social sistant to any organization worldwide, including WA for COVID- 5 6 needs and panic disorder, as well as conversational agents for irri- 19 and assistance with initial setup at no charge for at least 90 days, 7 8 table bowel syndrome and behavior change. For the COVID-19 as part of IBM’s corporate social responsibility initiatives in re- pandemic, conversational agents have been deployed to answer sponse to the pandemic. WA is also free to use for anyone, for up to questions and to triage symptoms, but studies of their adoption and 10 000 messages per month and 1000 users per month. Conversa- use to address questions surrounding COVID-19 have been limited tional agents built using WA were trained to understand and re- 9–11 to single institution experiences. spond, through both voice and text, to common COVID-19 questions, leveraging evidence-based sources where possible, such as Study objective guidance from the United States (US) Centers for Disease Control We sought to characterize the diverse use cases of COVID-related and Prevention (CDC). Basic COVID-19 content was made avail- conversational agents built using the Watson Assistant (WA) plat- able in both English and Spanish. form between March 30, 2020 and August 10, 2020. We measured WA provides both human-curated, predetermined responses as the adoption through the number of users and messages sent. We de- well as capabilities to dynamically search for and identify informa- termined the average number of conversational turns, with 1 turn tion from unstructured documents or websites on a scheduled basis. representing 1 question–response pair. This architecture provides users with access to the most up-to-date information as science evolves and ensures some level of quality in the information provided through expert validation when needed. To dynamically search for and provide up-to-date information, WA MATERIALS AND METHODS treats the user input as a search query. It finds information that is WA description and capabilities relevant to the query from an external data source, such as the WA is a platform for developing, training, and customizing conver- CDC, and returns it to the user. sational agents. Although not specific to the healthcare domain, this Conversational agents built using WA can be customized for spe- platform has previously been applied to medication prescribing, cific use cases. For example, conversational agents can be trained to 12–14 mental health, and Parkinson’s disease. Core natural language include information related to a specific language, locale, or organi- capabilities include: (1) understanding input, (2) classifying topics, zation, such as links to local school closings, local news, and state (3) state management and maintaining a structured dialog (eg, func- websites. Once the assistant is live and users ask questions, a human tions to support dynamically collecting multiple pieces of informa- will typically review subsets of conversations for knowledge gaps. tion, digressions for allowing users to change topics without losing The assistant is retrained to answer any questions it was not initially their place in the conversation, and disambiguation to clarify when trained on to cover these gaps. users say something for which the system has multiple relevant responses ); and (4) retrieving information from a knowledge base through search. WA uses NLP and machine learning in the intent Conversational intents understanding, entity extraction, query expansion, and finding An initial catalog of COVID-19 intents was created by experts in answers through estimating document relevancy. Search capacities conversational agent design to cover areas including testing, case also have NLP in its natural language understanding capabilities counts, travel restrictions, preventative behaviors, symptoms, and both when breaking down the user’s query as well as finding contact information. The content was based on current evidence and answers in documents. WA supports building a conversational inter- best practices retrieved from the CDC, Department of Labor, World face into any application, device, or channel such as a website or in- Health Organization (WHO), and USA.gov. Intents were imple- teractive voice recognition system. mented as static responses or dynamic searches, depending on the WA conversational agents allow users to initiate a conversation types and sources of information, as well as how often this informa- by entering questions. For example, when a user enters a question tion changes. Relatively consistent, high priority COVID-19 knowl- about COVID-19, a conversational agent built using WA will inter- edge intents were curated by humans, with intents and responses pret the question to identify the intent (target of a user’s query) and independently reviewed and evaluated by 2 physicians for face valid- match it to an internal list of intents and entities (for example, a con- ity related to public health and clinical acceptability. Downloaded from https://academic.oup.com/jamia/article/28/4/850/6017172 by DeepDyve user on 19 July 2022 852 Journal of the American Medical Informatics Association, 2021, Vol. 28, No. 4 Figure 1. Countries with implementations of Watson Assistant for COVID-19 response. Disagreements were resolved through discussions with a third widely including: (a) COVID-19 symptoms, (b) testing information, physician to reach consensus. Biweekly reviews to iteratively refine (c) information on preventative behaviors, (d) local and national in- all intents and responses with clinicians and public health experts formation about the disease, (e) response initiatives, (f) availability of were also conducted and are ongoing. Intents with reliable sources services and how to access them, (g) guidelines, restrictions, closures, of information and rapidly-changing answers (eg, case counts) were and reopening information, (h) course and exam information, implemented with dynamic search or lookup functions, with data (i) unemployment benefits and information, (j) stimulus payments, sources routinely reviewed by experts. Additional intents specific to (k) business assistance, and (l) volunteer opportunities. organizational information needs and use cases were developed, As of 8/10/2020, 101 organizations had used WA for COVID-19 such as intents covering physician and medical center access for pro- to develop their own conversational agent; of these, usage data for viders, intents for testing coverage and premium payments for health this study were available for 37 institutions in 9 countries. The types plan members, and intents relating to when and how employees may of organizations implementing conversational agents through the work or return to work for employers (see Supplementary Table 1 WA for COVID-19 platform were primarily governmental (N¼ 24), file for a full characterization of intents). employers (N¼ 7), providers (N¼ 5), and health plans (N¼ 1). Most organizations leveraging this technology were located in the U.S. and Canada (N¼ 29), Europe (N¼ 4), and Asia Pacific (N¼ 4). Evaluation of usage Figure 1 shows countries where organizations implemented conver- We assessed the initial success of the WA platform in delivering infor- sational agents using WA for COVID-19. The number of estimated mation for COVID-19 across use cases by measuring: (1) adoption of potential users ranged from 26 000 to 212 000 people. The types of WA conversational agents through number and diversity of users, (2) users included patients, health plan beneficiaries, students and staff, the total number of messages (ie, number of times a conversational business owners and employees, and the general public (country, agent provides text to the user); and (3) the average number of conver- state, county, and city residents). Supplementary Table 2 further sational turns per session. These metrics were collected over a 4- describes the organizations and their users. month period between March 30, 2020 and August 10, 2020. Analy- Total message usage and average number of conversational turns sis was performed in RStudio version 3.6.1. For any organizations per session are presented in Table 1, with a visualization of these that joined the free trial after March 30, 2020, we calculated usage data over time for each organizational type in Figure 2. A total of 6 metrics from the date of initial use of WA for COVID-19. 872 021 messages were sent in conversations about COVID-19 us- ing the conversational platform. Mean conversational turns were highest for provider organizations (mean, 3.5 turns) and lowest for RESULTS health plans (mean, 1.9 turns). Usage All institutions achieved end-to-end deployment in approximately 3 weeks or less; the average time to initial use was 5 business days. Two DISCUSSION implementations were voice-based, requiring users to call the imple- menting organization’s contact number, while the rest were web chat This brief communication describes rapid and widespread deploy- integrations. Each web-based agent was made available either through ment, adoption, and usage of a set of conversational agents to ad- webchat on the organizations’ home sites or internal landing page (for dress the overwhelming information needs created by COVID-19. employer organizations). The type of information provided ranged We show that conversational agents built to answer many different Downloaded from https://academic.oup.com/jamia/article/28/4/850/6017172 by DeepDyve user on 19 July 2022 Journal of the American Medical Informatics Association, 2021, Vol. 28, No. 4 853 Table 1. Usage metrics from March 30, 2020 to August 10, 2020 Organization Type Total Number of Messages Mean Number of Messages* Mean Conversational Turns* Government 5 702 811 11 880 3.0 (N¼ 24) (min¼ 1; max¼ 24 557) (min¼ 2.56; max¼ 3.8) Employer 1 159 304 9742 2.9 (N¼ 7) (min¼ 109; max¼ 22 334) (min¼ 1.8; max¼ 8.2) Provider 11 379 120 3.5 (N¼ 5) (min¼ 4; max¼ 295) (min¼ 2.1; max¼ 4.9) Health Plan 10 710 714 1.9 (N¼ 1) (min¼ 71; max¼ 1 382) (min¼ 1.4; max¼ 2.9) *Calculated by week Figure 2. Usage metrics over time by organizational type: a) mean number of messages by week and b) mean number of conversational turns by week. types of questions for COVID-19 pandemic response can be The COVID-19 pandemic generated an urgent need to provide deployed quickly and were broadly adopted during the early stages answers to questions based on rapidly evolving scientific evidence. of the pandemic. Citizens continue to want quick access to information in a manner Downloaded from https://academic.oup.com/jamia/article/28/4/850/6017172 by DeepDyve user on 19 July 2022 854 Journal of the American Medical Informatics Association, 2021, Vol. 28, No. 4 that allows them to make informed decisions on how to protect user conversations and how conversations change over time during themselves, their families, and their communities. To address the course of a pandemic. We are also investigating user satisfaction these needs, several institutions have reported leveraging conversa- and experience with COVID-19 conversational agents. tional agent technologies. Most of these agents focused on symptom 19,20 self-checking for patient triage or mental health, while auto- FUNDING mating answers to common questions was more limited. This man- uscript describes a platform used to deploy conversational agents to This study is funded by IBM Watson Health. address a diverse set of information needs for a wide variety of stakeholders including governments, employers, providers, and health plans. AUTHOR CONTRIBUTIONS Thirty-seven organizations in 9 different countries implemented MMc contributed to the conception of the work; the acquisition, analysis, agents and delivered over 6.8 million messages, indicating wide- and interpretation of data; the drafting of the work; and critical revision. BS spread geographic adoption of these conversational agents and de- and AP contributed to the interpretation of data, the drafting of the work, mand for public health information related to the COVID-19 and critical revision. MMa contributed to the acquisition and interpretation pandemic. Published studies of conversational agents to address of data; the drafting of the work; and critical revision. GP contributed to the conception of the work, drafting of the manuscript, and critical revision. COVID-19 have previously been limited to single-institution experi- ences with a single conversational agent. Our experience demon- strates the ability of a conversational technology platform to ACKNOWLEDGMENTS support varied COVID-19 information needs across multiple institu- tions, representing diverse stakeholders and users. The authors gratefully acknowledge Yull Arriaga, Rubina Rizvi, Kristen Sum- Further, we report on conversational turns, which are used to as- mers, for their subject matter expertise. sess the amount of interaction between a user and a system. The mean number of conversational turns per session was 2 to 3, indicat- SUPPLEMENTARY MATERIAL ing engagement with agents and suggesting they can answer most user questions efficiently. The relative number of turns may also un- Supplementary material is available at Journal of the American Medical Infor- derscore the complexity of some user questions, particularly clinical matics Association online. ones, since provider organizations had the most turns per session. Yet, across organizations, the number of conversational turns is not CONFLICT OF INTEREST STATEMENT reflective of highly complex conversations. Due to the novel and rapidly evolving context in the early stages of a pandemic, most The authors of this manuscript are employed by IBM Watson Health. users probably asked simple, transactional types of questions such as “Is the hospital open?” and “What is COVID-19?” This trend is likely to change as the pandemic evolves. For example, in the later DATA AVAILABILITY weeks of this study, conversational length among employers spiked The data underlying this article will be shared on reasonable request to the (see Figure 2). We hypothesize that as workers returned to work, corresponding author. more complex conversations around workplace safety and reopen- ing policies occurred. REFERENCES 1. Cucinotta D, Vanelli M. WHO declares COVID-19 a pandemic. Acta Limitations Biomed 2020; 91 (1): 157–60. This preliminary work has several limitations. This description of 2. Diederich S, Brendel A, Kolbe L. Towards a Taxonomy of Platforms for initial usage did not measure outcomes such as user satisfaction, fre- Conversational Agent Design. Wirtschaftsinformatik 2019 Proceedings quency of intents, whether user questions were answered, or time Published Online First: 28 February 2019.https://aisel.aisnet.org/wi2019/ and cost savings; these are topics of ongoing research. The manu- track10/papers/1 Accessed September 3, 2020. script reports the adoption and use of a system that is commercially 3. Bavaresco R, Silveira D, Reis E, et al. Conversational agents in business: a available for enterprise solutions. However, this manuscript systematic literature review and future research directions. Comput Sci Rev 2020; 36: 100239.doi:10.1016/j.cosrev.2020.100239 reported usage only during the period for which the platform was 4. Laranjo L, Dunn AG, Tong HL, et al. Conversational agents in healthcare: freely available as part of a philanthropic response to the pandemic, a systematic review. J Am Med Inform Assoc 2018; 25 (9): 1248–58. and the platform is freely available to anyone for low to medium 5. Kocielnik R, Agapie E, Argyle A, et al. HarborBot: a chatbot for social volume applications. needs screening. AMIA Annu Symp Proc 2019; 2019: 552–61. 6. Oh J, Jang S, Kim H, et al. Efficacy of mobile app-based interactive cogni- tive behavioral therapy using a chatbot for panic disorder. Int J Med In- CONCLUSION form 2020; 140: 104171.doi:10.1016/j.ijmedinf.2020.104171 7. Zand A, Sharma A, Stokes Z, et al. An exploration into the use of a chat- We have demonstrated the ability of a wide variety of organizations bot for patients with inflammatory bowel diseases: retrospective cohort including governments, employers, providers, and payers to use con- study. J Med Internet Res 2020; 22 (5): e15589. versational technologies to provide current information related to 8. Piao M, Ryu H, Lee H, et al. Use of the healthy lifestyle coaching chatbot COVID-19 to their citizens, employees, patients, and beneficiaries. app to promote stair-climbing habits among office workers: exploratory The WA platform enabled rapid implementation of a set of conver- randomized controlled trial. JMIR Mhealth Uhealth 2020; 8 (5): e15085. sational agents for a wide variety of use cases, and usage data show 9. Espinoza J, Crown K, Kulkarni O. A guide to chatbots for COVID-19 demand for and adoption of these technologies during a rapidly screening at pediatric health care facilities. JMIR Public Health Surveill evolving public health crisis. Our ongoing research aims to examine 2020; 6 (2): e18808. Downloaded from https://academic.oup.com/jamia/article/28/4/850/6017172 by DeepDyve user on 19 July 2022 Journal of the American Medical Informatics Association, 2021, Vol. 28, No. 4 855 10. Battineni G, Chintalapudi N, Amenta F. AI chatbot design during an epi- 15. Building a conversational flow. https://cloud.ibm.com/docs/assistant?top- demic like the novel coronavirus. Healthcare (Basel) 2020; 8 (2): 154. ic¼assistant-dialog-overview Accessed September 3, 2020. 11. Judson TJ, Odisho AY, Young JJ, et al. Case report: implementation of a 16. Watson Assistant. 2020. https://cloud.ibm.com/docs/assistant?top- digital chatbot to screen health system employees during the COVID-19 ic¼assistant-getting-started Accessed September 3, 2020. pandemic. J Am Med Inform Assoc 2020; 27 (9): 1450–55. 17. Creating a search skill. https://cloud.ibm.com/docs/assistant?topic¼assist- 12. Preininger AM, South B, Heiland J, et al. Artificial intelligence-based con- ant-skill-search-add Accessed September 3, 2020. versational agent to support medication prescribing. JAMIA Open 2020; 18. Tangcharoensathien V, Calleja N, Nguyen T, et al. Framework for managing 3 (2): 225–32. doi:10.1093/jamiaopen/ooaa009 the COVID-19 infodemic: methods and results of an online, crowdsourced 13. Linden B, Tam-Seto L, Stuart H. Adherence of the #Here4U app–military WHO technical consultation. J Med Internet Res 2020; 22 (6): e19659. version to criteria for the development of rigorous mental health apps. 19. Judson TJ, Odisho AY, Neinstein AB, et al. Rapid design and implementa- JMIR Form Res 2020; 4 (6): e18890. tion of an integrated patient self-triage and self-scheduling tool for 14. Macedo P, Pereira C, Mota P, et al. Conversational agent in mHealth to COVID-19. J Am Med Inform Assoc 2020; 27 (6): 860–6. empower people managing the Parkinson’s disease. Procedia Comput Sci 20. Miner AS, Laranjo L, Kocaballi AB. Chatbots in the fight against the 2019; 160: 402–8. COVID-19 pandemic. NPJ Digit Med 2020; 3 (1): 1–4. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Journal of the American Medical Informatics Association Oxford University Press

Leveraging conversational technology to answer common COVID-19 questions

Download PDF
6 pages

Loading next page...
 
/lp/oxford-university-press/leveraging-conversational-technology-to-answer-common-covid-19-HAxVwiiRll

References (39)

Publisher
Oxford University Press
Copyright
Copyright © 2022 American Medical Informatics Association
ISSN
1067-5027
eISSN
1527-974X
DOI
10.1093/jamia/ocaa316
Publisher site
See Article on Publisher Site

Abstract

Downloaded from https://academic.oup.com/jamia/article/28/4/850/6017172 by DeepDyve user on 19 July 2022 Journal of the American Medical Informatics Association, 28(4), 2021, 850–855 doi: 10.1093/jamia/ocaa316 Advance Access Publication Date: 1 February 2021 Brief Communication Brief Communication Leveraging conversational technology to answer common COVID-19 questions 1 1 1 1 Mollie McKillop, Brett R. South, Anita Preininger , Mitch Mason, Gretchen 1,2 Purcell Jackson 1 2 IBM Watson Health, Cambridge, Massachusetts, USA and Vanderbilt University Medical Center, Nashville, Tennessee, USA Corresponding Author: Mollie M. McKillop, PhD, MPH, 51 Astor Place, Floor 4, New York, NY, USA (mollie.mckillop@ibm.- com) Received 3 September 2020; Revised 17 October 2020; Editorial Decision 18 November 2020; Accepted 28 November 2020 ABSTRACT The rapidly evolving science about the Coronavirus Disease 2019 (COVID-19) pandemic created unprecedented health information needs and dramatic changes in policies globally. We describe a platform, Watson Assistant (WA), which has been used to develop conversational agents to deliver COVID-19 related information. We char- acterized the diverse use cases and implementations during the early pandemic and measured adoption through a number of users, messages sent, and conversational turns (ie, pairs of interactions between users and agents). Thirty-seven institutions in 9 countries deployed COVID-19 conversational agents with WA be- tween March 30 and August 10, 2020, including 24 governmental agencies, 7 employers, 5 provider organiza- tions, and 1 health plan. Over 6.8 million messages were delivered through the platform. The mean number of conversational turns per session ranged between 1.9 and 3.5. Our experience demonstrates that conversational technologies can be rapidly deployed for pandemic response and are adopted globally by a wide range of users. Key words: telemedicine, COVID-19, public health, pandemics, chatbots, conversational technologies INTRODUCTION Information needs Timely and accurate public health information related to COVID- A new coronavirus causing severe acute respiratory syndrome 19 is universally needed across stakeholders. Organizations have (SARS-CoV-2) has infected millions worldwide with coronavirus been asked to provide information on COVID-19, its symptoms, disease 2019 (COVID-19) and caused significant mortality. SARS- how it spreads, strategies for prevention, and how each organization CoV-2 is a highly contagious pathogen, with widely variable clinical is responding to the pandemic. Trusted sources of health informa- manifestations. In March 2020, the World Health Organization tion, like medical practices, have limited in-person visits to focus on (WHO) classified COVID-19 as a pandemic. In the absence of treating the sick and reducing disease spread. Staff reductions have proven therapies or a vaccine, public health departments, govern- further compounded availability to answer questions. The pervasive- ments, employers, and healthcare institutions have taken measures ness of the pandemic has resulted in organizations assuming new to control the spread of the disease, including providing information and promoting nonpharmaceutical interventions, such as social dis- roles related to the dissemination of public health information. tancing and hand washing. Given the novelty of the disease, infor- Given the enormous demand for information about COVID-19, mation is rapidly evolving, with new evidence often contradicting many stakeholders have leveraged emerging conversational technol- earlier findings. These inconsistencies create uncertainty, leading to ogies to automate responses to common COVID-19 related ques- a need for trustworthy, health-related information. tions and information needs specific to their organizations. V The Author(s) 2021. Published by Oxford University Press on behalf of the American Medical Informatics Association. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted reuse, distribution, and reproduction in any medium, provided the original work is properly cited. 850 Downloaded from https://academic.oup.com/jamia/article/28/4/850/6017172 by DeepDyve user on 19 July 2022 Journal of the American Medical Informatics Association, 2021, Vol. 28, No. 4 851 Chatbots in healthcare dition) that answer the question within the dialogue interface or find the most relevant answer in its knowledge base. One way to scale dissemination of COVID-19 related information is WA is built upon a core set of functionalities with 3 main com- through technologies that employ natural language conversation. ponents that facilitate dialogue with users. The first component is Chatbots, sometimes called conversational agents or virtual assis- the intent, which defines the type of information sought. The second tants, often differ in functionality. Consensus on a taxonomy of 2,3 component includes an entity that is used to provide a precise re- these conversational technologies is lacking. The simplest chat- sponse for an intent. The final component is dialogue, which is the bots are capable of matching a predetermined set of topics with pre- actual conversation a user has with the conversational agent. WA’s defined answers, whereas more sophisticated conversational agents proprietary NLP capabilities facilitate creation and training of con- expand on their functionalities to employ machine learning and nat- versational agents with a minimal amount of data. Agents can be de- ural language processing (NLP) to understand questions in everyday livered in any cloud environment, allowing users to maintain language and engage users in increasingly complex conversations. ownership and privacy of their data. The technical details of func- For example, conversational agents may understand meaning, main- tionalities and implementation are beyond the scope of this brief tain context in dialogue, and learn with time to improve their per- communication but are provided elsewhere. formance. Some describe conversational agents that aid users in performing specific tasks as virtual assistants; they often have a characteristic personality expressed by tone, dialect, or style in con- Conversational agents for COVID-19 versation. Such conversational tools have demonstrated promise in Beginning in March 2020, IBM offered a program called Citizen As- clinical applications, including chatbots for determining social sistant to any organization worldwide, including WA for COVID- 5 6 needs and panic disorder, as well as conversational agents for irri- 19 and assistance with initial setup at no charge for at least 90 days, 7 8 table bowel syndrome and behavior change. For the COVID-19 as part of IBM’s corporate social responsibility initiatives in re- pandemic, conversational agents have been deployed to answer sponse to the pandemic. WA is also free to use for anyone, for up to questions and to triage symptoms, but studies of their adoption and 10 000 messages per month and 1000 users per month. Conversa- use to address questions surrounding COVID-19 have been limited tional agents built using WA were trained to understand and re- 9–11 to single institution experiences. spond, through both voice and text, to common COVID-19 questions, leveraging evidence-based sources where possible, such as Study objective guidance from the United States (US) Centers for Disease Control We sought to characterize the diverse use cases of COVID-related and Prevention (CDC). Basic COVID-19 content was made avail- conversational agents built using the Watson Assistant (WA) plat- able in both English and Spanish. form between March 30, 2020 and August 10, 2020. We measured WA provides both human-curated, predetermined responses as the adoption through the number of users and messages sent. We de- well as capabilities to dynamically search for and identify informa- termined the average number of conversational turns, with 1 turn tion from unstructured documents or websites on a scheduled basis. representing 1 question–response pair. This architecture provides users with access to the most up-to-date information as science evolves and ensures some level of quality in the information provided through expert validation when needed. To dynamically search for and provide up-to-date information, WA MATERIALS AND METHODS treats the user input as a search query. It finds information that is WA description and capabilities relevant to the query from an external data source, such as the WA is a platform for developing, training, and customizing conver- CDC, and returns it to the user. sational agents. Although not specific to the healthcare domain, this Conversational agents built using WA can be customized for spe- platform has previously been applied to medication prescribing, cific use cases. For example, conversational agents can be trained to 12–14 mental health, and Parkinson’s disease. Core natural language include information related to a specific language, locale, or organi- capabilities include: (1) understanding input, (2) classifying topics, zation, such as links to local school closings, local news, and state (3) state management and maintaining a structured dialog (eg, func- websites. Once the assistant is live and users ask questions, a human tions to support dynamically collecting multiple pieces of informa- will typically review subsets of conversations for knowledge gaps. tion, digressions for allowing users to change topics without losing The assistant is retrained to answer any questions it was not initially their place in the conversation, and disambiguation to clarify when trained on to cover these gaps. users say something for which the system has multiple relevant responses ); and (4) retrieving information from a knowledge base through search. WA uses NLP and machine learning in the intent Conversational intents understanding, entity extraction, query expansion, and finding An initial catalog of COVID-19 intents was created by experts in answers through estimating document relevancy. Search capacities conversational agent design to cover areas including testing, case also have NLP in its natural language understanding capabilities counts, travel restrictions, preventative behaviors, symptoms, and both when breaking down the user’s query as well as finding contact information. The content was based on current evidence and answers in documents. WA supports building a conversational inter- best practices retrieved from the CDC, Department of Labor, World face into any application, device, or channel such as a website or in- Health Organization (WHO), and USA.gov. Intents were imple- teractive voice recognition system. mented as static responses or dynamic searches, depending on the WA conversational agents allow users to initiate a conversation types and sources of information, as well as how often this informa- by entering questions. For example, when a user enters a question tion changes. Relatively consistent, high priority COVID-19 knowl- about COVID-19, a conversational agent built using WA will inter- edge intents were curated by humans, with intents and responses pret the question to identify the intent (target of a user’s query) and independently reviewed and evaluated by 2 physicians for face valid- match it to an internal list of intents and entities (for example, a con- ity related to public health and clinical acceptability. Downloaded from https://academic.oup.com/jamia/article/28/4/850/6017172 by DeepDyve user on 19 July 2022 852 Journal of the American Medical Informatics Association, 2021, Vol. 28, No. 4 Figure 1. Countries with implementations of Watson Assistant for COVID-19 response. Disagreements were resolved through discussions with a third widely including: (a) COVID-19 symptoms, (b) testing information, physician to reach consensus. Biweekly reviews to iteratively refine (c) information on preventative behaviors, (d) local and national in- all intents and responses with clinicians and public health experts formation about the disease, (e) response initiatives, (f) availability of were also conducted and are ongoing. Intents with reliable sources services and how to access them, (g) guidelines, restrictions, closures, of information and rapidly-changing answers (eg, case counts) were and reopening information, (h) course and exam information, implemented with dynamic search or lookup functions, with data (i) unemployment benefits and information, (j) stimulus payments, sources routinely reviewed by experts. Additional intents specific to (k) business assistance, and (l) volunteer opportunities. organizational information needs and use cases were developed, As of 8/10/2020, 101 organizations had used WA for COVID-19 such as intents covering physician and medical center access for pro- to develop their own conversational agent; of these, usage data for viders, intents for testing coverage and premium payments for health this study were available for 37 institutions in 9 countries. The types plan members, and intents relating to when and how employees may of organizations implementing conversational agents through the work or return to work for employers (see Supplementary Table 1 WA for COVID-19 platform were primarily governmental (N¼ 24), file for a full characterization of intents). employers (N¼ 7), providers (N¼ 5), and health plans (N¼ 1). Most organizations leveraging this technology were located in the U.S. and Canada (N¼ 29), Europe (N¼ 4), and Asia Pacific (N¼ 4). Evaluation of usage Figure 1 shows countries where organizations implemented conver- We assessed the initial success of the WA platform in delivering infor- sational agents using WA for COVID-19. The number of estimated mation for COVID-19 across use cases by measuring: (1) adoption of potential users ranged from 26 000 to 212 000 people. The types of WA conversational agents through number and diversity of users, (2) users included patients, health plan beneficiaries, students and staff, the total number of messages (ie, number of times a conversational business owners and employees, and the general public (country, agent provides text to the user); and (3) the average number of conver- state, county, and city residents). Supplementary Table 2 further sational turns per session. These metrics were collected over a 4- describes the organizations and their users. month period between March 30, 2020 and August 10, 2020. Analy- Total message usage and average number of conversational turns sis was performed in RStudio version 3.6.1. For any organizations per session are presented in Table 1, with a visualization of these that joined the free trial after March 30, 2020, we calculated usage data over time for each organizational type in Figure 2. A total of 6 metrics from the date of initial use of WA for COVID-19. 872 021 messages were sent in conversations about COVID-19 us- ing the conversational platform. Mean conversational turns were highest for provider organizations (mean, 3.5 turns) and lowest for RESULTS health plans (mean, 1.9 turns). Usage All institutions achieved end-to-end deployment in approximately 3 weeks or less; the average time to initial use was 5 business days. Two DISCUSSION implementations were voice-based, requiring users to call the imple- menting organization’s contact number, while the rest were web chat This brief communication describes rapid and widespread deploy- integrations. Each web-based agent was made available either through ment, adoption, and usage of a set of conversational agents to ad- webchat on the organizations’ home sites or internal landing page (for dress the overwhelming information needs created by COVID-19. employer organizations). The type of information provided ranged We show that conversational agents built to answer many different Downloaded from https://academic.oup.com/jamia/article/28/4/850/6017172 by DeepDyve user on 19 July 2022 Journal of the American Medical Informatics Association, 2021, Vol. 28, No. 4 853 Table 1. Usage metrics from March 30, 2020 to August 10, 2020 Organization Type Total Number of Messages Mean Number of Messages* Mean Conversational Turns* Government 5 702 811 11 880 3.0 (N¼ 24) (min¼ 1; max¼ 24 557) (min¼ 2.56; max¼ 3.8) Employer 1 159 304 9742 2.9 (N¼ 7) (min¼ 109; max¼ 22 334) (min¼ 1.8; max¼ 8.2) Provider 11 379 120 3.5 (N¼ 5) (min¼ 4; max¼ 295) (min¼ 2.1; max¼ 4.9) Health Plan 10 710 714 1.9 (N¼ 1) (min¼ 71; max¼ 1 382) (min¼ 1.4; max¼ 2.9) *Calculated by week Figure 2. Usage metrics over time by organizational type: a) mean number of messages by week and b) mean number of conversational turns by week. types of questions for COVID-19 pandemic response can be The COVID-19 pandemic generated an urgent need to provide deployed quickly and were broadly adopted during the early stages answers to questions based on rapidly evolving scientific evidence. of the pandemic. Citizens continue to want quick access to information in a manner Downloaded from https://academic.oup.com/jamia/article/28/4/850/6017172 by DeepDyve user on 19 July 2022 854 Journal of the American Medical Informatics Association, 2021, Vol. 28, No. 4 that allows them to make informed decisions on how to protect user conversations and how conversations change over time during themselves, their families, and their communities. To address the course of a pandemic. We are also investigating user satisfaction these needs, several institutions have reported leveraging conversa- and experience with COVID-19 conversational agents. tional agent technologies. Most of these agents focused on symptom 19,20 self-checking for patient triage or mental health, while auto- FUNDING mating answers to common questions was more limited. This man- uscript describes a platform used to deploy conversational agents to This study is funded by IBM Watson Health. address a diverse set of information needs for a wide variety of stakeholders including governments, employers, providers, and health plans. AUTHOR CONTRIBUTIONS Thirty-seven organizations in 9 different countries implemented MMc contributed to the conception of the work; the acquisition, analysis, agents and delivered over 6.8 million messages, indicating wide- and interpretation of data; the drafting of the work; and critical revision. BS spread geographic adoption of these conversational agents and de- and AP contributed to the interpretation of data, the drafting of the work, mand for public health information related to the COVID-19 and critical revision. MMa contributed to the acquisition and interpretation pandemic. Published studies of conversational agents to address of data; the drafting of the work; and critical revision. GP contributed to the conception of the work, drafting of the manuscript, and critical revision. COVID-19 have previously been limited to single-institution experi- ences with a single conversational agent. Our experience demon- strates the ability of a conversational technology platform to ACKNOWLEDGMENTS support varied COVID-19 information needs across multiple institu- tions, representing diverse stakeholders and users. The authors gratefully acknowledge Yull Arriaga, Rubina Rizvi, Kristen Sum- Further, we report on conversational turns, which are used to as- mers, for their subject matter expertise. sess the amount of interaction between a user and a system. The mean number of conversational turns per session was 2 to 3, indicat- SUPPLEMENTARY MATERIAL ing engagement with agents and suggesting they can answer most user questions efficiently. The relative number of turns may also un- Supplementary material is available at Journal of the American Medical Infor- derscore the complexity of some user questions, particularly clinical matics Association online. ones, since provider organizations had the most turns per session. Yet, across organizations, the number of conversational turns is not CONFLICT OF INTEREST STATEMENT reflective of highly complex conversations. Due to the novel and rapidly evolving context in the early stages of a pandemic, most The authors of this manuscript are employed by IBM Watson Health. users probably asked simple, transactional types of questions such as “Is the hospital open?” and “What is COVID-19?” This trend is likely to change as the pandemic evolves. For example, in the later DATA AVAILABILITY weeks of this study, conversational length among employers spiked The data underlying this article will be shared on reasonable request to the (see Figure 2). We hypothesize that as workers returned to work, corresponding author. more complex conversations around workplace safety and reopen- ing policies occurred. REFERENCES 1. Cucinotta D, Vanelli M. WHO declares COVID-19 a pandemic. Acta Limitations Biomed 2020; 91 (1): 157–60. This preliminary work has several limitations. This description of 2. Diederich S, Brendel A, Kolbe L. Towards a Taxonomy of Platforms for initial usage did not measure outcomes such as user satisfaction, fre- Conversational Agent Design. Wirtschaftsinformatik 2019 Proceedings quency of intents, whether user questions were answered, or time Published Online First: 28 February 2019.https://aisel.aisnet.org/wi2019/ and cost savings; these are topics of ongoing research. The manu- track10/papers/1 Accessed September 3, 2020. script reports the adoption and use of a system that is commercially 3. Bavaresco R, Silveira D, Reis E, et al. Conversational agents in business: a available for enterprise solutions. However, this manuscript systematic literature review and future research directions. Comput Sci Rev 2020; 36: 100239.doi:10.1016/j.cosrev.2020.100239 reported usage only during the period for which the platform was 4. Laranjo L, Dunn AG, Tong HL, et al. Conversational agents in healthcare: freely available as part of a philanthropic response to the pandemic, a systematic review. J Am Med Inform Assoc 2018; 25 (9): 1248–58. and the platform is freely available to anyone for low to medium 5. Kocielnik R, Agapie E, Argyle A, et al. HarborBot: a chatbot for social volume applications. needs screening. AMIA Annu Symp Proc 2019; 2019: 552–61. 6. Oh J, Jang S, Kim H, et al. Efficacy of mobile app-based interactive cogni- tive behavioral therapy using a chatbot for panic disorder. Int J Med In- CONCLUSION form 2020; 140: 104171.doi:10.1016/j.ijmedinf.2020.104171 7. Zand A, Sharma A, Stokes Z, et al. An exploration into the use of a chat- We have demonstrated the ability of a wide variety of organizations bot for patients with inflammatory bowel diseases: retrospective cohort including governments, employers, providers, and payers to use con- study. J Med Internet Res 2020; 22 (5): e15589. versational technologies to provide current information related to 8. Piao M, Ryu H, Lee H, et al. Use of the healthy lifestyle coaching chatbot COVID-19 to their citizens, employees, patients, and beneficiaries. app to promote stair-climbing habits among office workers: exploratory The WA platform enabled rapid implementation of a set of conver- randomized controlled trial. JMIR Mhealth Uhealth 2020; 8 (5): e15085. sational agents for a wide variety of use cases, and usage data show 9. Espinoza J, Crown K, Kulkarni O. A guide to chatbots for COVID-19 demand for and adoption of these technologies during a rapidly screening at pediatric health care facilities. JMIR Public Health Surveill evolving public health crisis. Our ongoing research aims to examine 2020; 6 (2): e18808. Downloaded from https://academic.oup.com/jamia/article/28/4/850/6017172 by DeepDyve user on 19 July 2022 Journal of the American Medical Informatics Association, 2021, Vol. 28, No. 4 855 10. Battineni G, Chintalapudi N, Amenta F. AI chatbot design during an epi- 15. Building a conversational flow. https://cloud.ibm.com/docs/assistant?top- demic like the novel coronavirus. Healthcare (Basel) 2020; 8 (2): 154. ic¼assistant-dialog-overview Accessed September 3, 2020. 11. Judson TJ, Odisho AY, Young JJ, et al. Case report: implementation of a 16. Watson Assistant. 2020. https://cloud.ibm.com/docs/assistant?top- digital chatbot to screen health system employees during the COVID-19 ic¼assistant-getting-started Accessed September 3, 2020. pandemic. J Am Med Inform Assoc 2020; 27 (9): 1450–55. 17. Creating a search skill. https://cloud.ibm.com/docs/assistant?topic¼assist- 12. Preininger AM, South B, Heiland J, et al. Artificial intelligence-based con- ant-skill-search-add Accessed September 3, 2020. versational agent to support medication prescribing. JAMIA Open 2020; 18. Tangcharoensathien V, Calleja N, Nguyen T, et al. Framework for managing 3 (2): 225–32. doi:10.1093/jamiaopen/ooaa009 the COVID-19 infodemic: methods and results of an online, crowdsourced 13. Linden B, Tam-Seto L, Stuart H. Adherence of the #Here4U app–military WHO technical consultation. J Med Internet Res 2020; 22 (6): e19659. version to criteria for the development of rigorous mental health apps. 19. Judson TJ, Odisho AY, Neinstein AB, et al. Rapid design and implementa- JMIR Form Res 2020; 4 (6): e18890. tion of an integrated patient self-triage and self-scheduling tool for 14. Macedo P, Pereira C, Mota P, et al. Conversational agent in mHealth to COVID-19. J Am Med Inform Assoc 2020; 27 (6): 860–6. empower people managing the Parkinson’s disease. Procedia Comput Sci 20. Miner AS, Laranjo L, Kocaballi AB. Chatbots in the fight against the 2019; 160: 402–8. COVID-19 pandemic. NPJ Digit Med 2020; 3 (1): 1–4.

Journal

Journal of the American Medical Informatics Association – Oxford University Press

Published: Mar 18, 2021

There are no references for this article.