Usefulness of Artificial Intelligence-based Virtual Assistants in Oral and Maxillofacial Radiology Report Writing
1Department of Oral and Maxillofacial Diagnostic Sciences, UConn School of Dental Medicine, Farmington, Connecticut, USA
2Department of Oral and Maxillofacial Diagnostic Sciences, University of Florida, College of Dentistry, Gainesville, Florida, USA
3Department of Oral Pathology, Radiology and Medicine, The University of Iowa, Iowa City, Iowa, USA
4Graduate Research, Education and Training, UConn School of Dental Medicine, Farmington, Connecticut, USA
Corresponding Author: Kavya S Muttanahally, Department of Oral and Maxillofacial Diagnostic Sciences, UConn School of Dental Medicine, Farmington, Connecticut, USA, Phone: +1-8607252104, e-mail: firstname.lastname@example.org
How to cite this article Muttanahally KS, Vyas R, Mago J, et al. Usefulness of Artificial Intelligence-based Virtual Assistants in Oral and Maxillofacial Radiology Report Writing. World J Dent 2021;12(2):97–102.
Source of support: Nil
Conflict of interest: None
Aim and objective: This study aimed to evaluate the usefulness of four voice-based virtual assistants in oral and maxillofacial radiology report writing.
Materials and methods: A questionnaire consisting of 100 questions was queried to 4 commercially available voice-based virtual assistants namely Alexa, Siri, Cortana, and Google Assistant. The questions were divided based on five categories. The categorization was based on the frequency and reason for a radiologist to refer to either a textbook or an online resource before diagnosing and finalizing a radiology report. Two evaluators queried the devices and rated them on a 4-point modified Likert scale.
Results: In the order of efficiency, Google Assistant was the most efficient followed by Cortana, Siri, and Alexa. A significant difference between the examiners was observed with Cortana in anatomy, dental anatomy, differential diagnosis, and pathology.
Conclusion: In this small study that queried only four voice-powered virtual assistants, it showed that they were helpful and convenient in responding to questions regarding oral and maxillofacial radiology. But there is significant scope for expansion in the number of topics and type of information delivered before these can be used specifically in oral and maxillofacial radiology report writing.
Clinical significance: Oral radiologists often gather additional and updated information regarding various topics like disease-specific features, genetic mutations, and differential diagnoses which they typically get from a textbook or a website. Artificial intelligence-based virtual assistants offer radiologists a simple voice-activated interface to gather this information and can immensely help when additional information is required.
Keywords: Alexa, Artificial intelligence, Cortana, Google Assistant, Siri..
Artificial intelligence (AI) is being used in various devices, solutions, and codes that range from a simple calculator to complex machine learning devices.1 A very simple and successful AI-based application that is increasingly being used is voice-activated virtual assistant technology, especially in the consumer-grade and domestic use area. Several technology enthusiasts predicted that voice commands will 1 day allow us to control various digital helpers including administrative assistants and librarians.2 Currently, there are several such devices in the market like Alexa, Siri, Cortana, Google Assistant, and several others. Many of these voice-activated virtual assistants are embedded in smartphones, tablet computers, and/or are standalone devices. These devices offer a very simple voice-activated front end powered by a database on the back end.3 Voice-based virtual assistants can answer various questions from geography to science and can perform tasks like switching on lights to generating shopping lists and playing music and even answer simple questions. Their responses are primarily based on simple one-tier machine learning types of platforms. Since the cost of this technology has significantly come down in recent years, these devices are now being seen in many households and offices. Some of these devices can answer some medical and dental-related questions and can be helpful to both the doctors and the patients in providing quick access to some information that may not always be easy to recall for the human brain. The combination of this technology and an established workflow can lead to significant alterations in the medical field and especially in accessing radiographic images and in radiology report writing.4
There are several instances where radiologists will either refer to a textbook, an atlas with normal anatomic measurements, and a drug manual for drug interactions or visit the equipment manufacturer’s website to get technical specifications. There may be instances where radiologists also go online to look for disease-specific features, genetic mutations, and differential diagnoses. These virtual devices can answer questions about medical topics using information compiled from multiple data points and locations using specific features like symptoms, causes, or treatment for certain diseases. The complexity and the accuracy of voice recognition technology and voice-driven software have grown exponentially in the last few years. Currently, available voice assistant products from Apple, Amazon, Google, and Microsoft allow users to ask questions and issue commands to computers in natural or normal day-to-day language.5 These AI-powered virtual assistants need a substantial amount of information comprising of natural language processing, machine learning, and speech identification platforms.
These commercially available voice assistants were designed to perform specific tasks for which they were developed. An example is that “Alexa” was primarily developed to generate shopping lists and to provide an easy-to-use voice interface for customers to buy products on Amazon. This primary application eventually evolved to expand its use in controlling household devices and gadgets. As an extension of this ever-developing array of applications, we wanted to evaluate the usefulness of the above-mentioned four commercially available voice-driven virtual assistants, in dentistry is whether these virtual assistants can answer queries specific to medicine and dentistry. If they do, how accurate and reliable are those answers, and if they can support report writing in oral and maxillofacial radiology. Since there are no well-designed studies specific to the utility and the value of AI-powered voice assistants in oral and maxillofacial radiology report writing, this study will serve as a proof of concept for this application and will be valuable in further developing AI-based digital assistants.
The objective of this study was to evaluate the usefulness of four voice-based virtual assistants in the field of dentistry in general and oral radiology in particular.
MATERIALS AND METHODS
Two oral radiology residents in training (KM and RV) queried a questionnaire consisting of 100 questions to 4 commercially available voice-based virtual assistants: Alexa, Siri, Cortana, and Google Assistant. The questions were categorized based on the five categories. These categories were developed based on discussions with two senior board-certified oral and maxillofacial radiologists and two radiology residents in training. The categorization was based on the frequency and reason for a radiologist to refer to either a textbook or an online resource before diagnosing and finalizing a radiology report.
- Patient history and syndromes.
- Medications and short forms of diseases.
- Machine and techniques.
- Anatomy, dental anatomy, and dimensions.
- Pathology and differential diagnoses.
The evaluators queried the devices and rated them on a 4-point scale using a modified Likert scale: (1) The device responded with adequate information, (2) The device responded but did not provide adequate information, (3) The device did not know the response to the question, (4) The device recognizes but does not provide adequate information instead it leads to series of website pages. The results of the retrieved queries were recorded on a Microsoft Excel sheet for statistical analysis to do the comparative evaluation.
A sample of questions related to dentistry and oral radiology is provided in Table 1. The full list of questions has been provided in the supplemental material.
Two investigators (KM and RV) queried all four virtual assistants regarding: (1) Patient data and syndrome-related questions, (2) Medications, (3) Anatomy, (4) Differential diagnosis, (5) Dentistry-related general questions, and (6) Knowledge about the radiographic acquisition modality/machine and different acquisition techniques and technical factors. A comparative statistical evaluation of the retrieved queries was done using SPSS version-20 (IBM Inc.).
The mean opinion score for each of the questions between both the evaluators was calculated using one-way ANOVA and post hoc test. Of all the devices queried, Google Assistant was the device that responded to all the questions in the most efficient manner. The mean score for Google Assistant was significantly less for all the questions showing the highest efficiency, followed by Microsoft and Alexa. Siri is the least with the highest mean score (Figs 1 and 2). Tables 2 and 3 show the comparison of the mean response score of the first (KM) and second (RV) investigator, respectively. The dissimilar alphabets denote the significant difference between the groups. In this the lower the mean opinion score, the more efficient the device would be.
Table 4 shows there was a significant difference between the examiners was observed in the categories of (3) Anatomy and (4) Differential diagnosis. Also, there was a significant difference between the examiners for the Google Assistant in the (2) Medication category.
Several innovations are being powered by AI and can perform tasks that one could not have imagined a few years ago. Voice-powered personal assistants are a classic example of how life has changed in the last few years because of access to information via gadgets like these. Question and answer-based gadgets like voice-activated personal assistants powered by large databases are an example of AI. Artificial intelligence is a branch of computer science that addresses simulation of intelligent behavior in computers, or the capability of a machine to imitate intelligent human behavior. Voice-powered assistants mainly work via cloud-based software programs that use internet-based data archives to respond to queries. There are currently several devices dedicated to providing virtual assistance. The most popular ones are available from Apple, Amazon, Google, and Microsoft.6 Alexa is a standalone voice-activated virtual assistant. Alexa is supported by Amazon, Cortana by Microsoft. Siri is available on all Apple devices including smartphones and tablet computer devices. Cortana is available on both Microsoft-supported smartphones and computers. Google Assistant is available on google supported smartphones as well as on independent devices.
Voice-powered virtual assistants can help oral radiologists in various situations where they are seeking additional information that requires referring to textbooks or browse the internet for specific information. While this is a fairly logical concept to use these devices to provide a simple voice-based interface into the workflow, currently there are no such studies, and this is the first of its kind. In this study, we looked at the status of these four-voice activated virtual assistants, specifically in the field of oral and maxillofacial radiology. While writing a radiology report, an oral and maxillofacial radiologist encounters various situations in the clinical requisition forms and clinical history, like new abbreviations, new terminology, or newer or unfamiliar tests or diseases. They typically use the internet to learn more about these terms and many times radiologists refer to a textbook or a published article about certain medical and dental conditions and the current status of research or treatment about the condition. Radiologists often refer to normal anatomic measurements to determine if the findings are consistent with normal morphological measurements or if they are a variant of normal to confirm or rule out pathology. Many radiologists refer to the internet for position papers or white papers on a topic to understand the status of science in a particular area or procedure and for updated guidance. This consumes an enormous amount of time and it would be convenient if there was a voice-activated assistant that could provide accurate information.
|S. no.||Sample questions||Siri||Alexa||Cortana||Google Assistant|
|1||What condition causes multiple impacted teeth?||4||3||4||1|
|2||What condition causes multiple missing teeth?||4||2||4||1|
|3||What are the causes for bell’s palsy?||4||1||1||1|
|4||What causes trigeminal neuralgia?||4||1||1||4|
|5||What does a flu shot contain?||4||1||1||1|
|6||What causes combine immunodeficiency?||4||3||1||1|
|7||What is the antibiotic prophylaxis for cardiac patients before teeth extraction||4||1||1||1|
|8||What is antibiotic prophylaxis?||1||1||1||1|
|9||Who is affected by sickle cell anemia?||4||2||4||1|
|10||Which is the common medication that causes hyperplastic gingiva?||4||3||4||1|
|11||What is the generic name of Tylenol?||4||1||1||1|
|12||What is the generic name of Fosamax?||4||3||1||1|
|13||What is the generic name of Advil?||4||1||1||1|
|14||What is the generic name of procardia?||4||3||1||1|
|15||What is the generic name of coumadin?||4||1||1||4|
|16||what is the drug interaction of lorazepam?||2||1||4||1|
|17||What is the mechanism of action of penicillin?||4||3||1||1|
|18||Why is a bitewing radiograph acquired?||4||3||2||1|
|19||What is the use of sialography?||4||3||3||1|
|20||What is the pathophysiology of osteomyelitis?||4||2||4||1|
Siri is Apple’s voice-based virtual assistant manufactured by Apple Inc., which was launched in 2011 and has become a very commonly used voice assistant especially since apple has a significant market share in the smartphone user’s market.7 Out of 100 questions, Siri was able to recognize 85 questions but failed to provide adequate information. Instead, it led the evaluators to a series of websites. Though it could answer a few questions, it could not provide specific responses to the questions posed regarding commonly available medications, generic names of the medicines and chromosomes associated with specific syndromes, etc. Siri was unable to provide specific information regarding imaging equipment used in oral radiology and imaging techniques. Siri was accurate for only 7 out of 100 questions.
Alexa was launched in November 2014 by Amazon Lab126 and is the first voice-activated virtual assistant linked to a stand-alone home device rather than integrated into existing electronic devices.8 Though it answered around 22 questions accurately, it failed to respond to nearly 69 questions. It failed to answer most of the medication-related questions and many dental-related queries. While it did recognize a significant number of questions, an in-depth answer was not a straightforward retrieval of the query.
|Siri||Alexa||Microsoft||Google Assistant||p value|
|Patient data/syndromes/short forms (n = 33)||3.71 ± 0.79a||2.68 ± 0.76b||2.28 ± 1.46b||1.28 ± 0.87c||<0.01*|
|Medications (n = 13)||3.84 ± 0.55a||2.76 ± 0.59b||2 ± 1.35 b||1 ± 0c||<0.01*|
|Anatomy, dental anatomy (n = 24)||3.5 ± 0.93a||2.83 ± 0.56b||2.41 ± 1.47 b||1.08 ± 0.40c||<0.01*|
|Differential diagnosis and pathology (n = 7)||4 ± 0a||2.14 ± 0.83b||2.28 ± 1.27b,a||1 ± 0c,a||<0.01*|
|Dentistry and general questions (n = 6)||3.6 ± 0.89a||2.6 ± 0.89a||2.8 ± 1.63 a||1.6 ± 1.34a||0.073|
|Machine and techniques (n = 14)||3.71 ± 0.72a||2.85 ± 0.53a||2.92 ± 1.49 a||1.35 ± 0.92b||<0.01*|
* p < 0.01 is statistically significant (one-way ANOVA and post hoc test)
|Siri||Alexa||Microsoft||Google Assistant||p value|
|Patient data/syndromes/short forms (n = 33)||3.54 ± 1.03a||2.24 ± 0.93b||2.48 ± 1.46b||1.21 ± 0.73c||<0.01*|
|Medications (n = 13)||3.76 ± 0.59a||2.30 ± 1.10b||1.84 ± 1.34b||1.92 ± 1.44b||<0.01*|
|Anatomy, dental anatomy (n = 24)||3.82 ± 0.65a||2.60 ± 0.76b||3.93 ± 1.01a||1.08 ± 0.40c||<0.01*|
|Differential diagnosis and pathology (n = 7)||4 ± 0a||2.57 ± 0.78b||3.85 ± 0.37a||1.42 ± 1.13c||<0.01*|
|Dentistry and general questions (n = 6)||3.8 ± 0.40a||2.8 ± 0.40b||4 ± 0a||1.8 ± 1.21c||<0.01*|
|Machine and techniques (n = 14)||3.78 ± 0.80a||3.07 ± 0.26a,a||3.64 ± 0.63a||2.21 ± 1.41b,a||<0.01*|
* p < 0.01 is statistically significant (one-way ANOVA and post hoc test)
|Patient data/syndromes/short forms (n = 33)||3.71 ± 0.79||3.54 ± 1.03||2.68 ± 0.76||2.24 ± 0.93||2.28 ± 1.46||2.48 ± 1.46||1.28 ± 0.87||1.21 ± 0.73|
|Medications (n = 13)||3.84 ± 0.55||3.76 ± 0.59||2.76 ± 0.59||2.30 ± 1.10||2 ± 1.35||1.84 ± 1.34||1 ± 0*||1.92 ± 1.44|
|Anatomy, dental anatomy (n = 24)||3.5 ± 0.93||3.82 ± 0.65||2.83 ± 0.56||2.60 ± 0.76||2.41 ± 1.47*||3.93 ± 1.01||1.08 ± 0.40||1.08 ± 0.40|
|Differential diagnosis and pathology (n = 7)||4 ± 0||4 ± 0||2.14 ± 0.83||2.57 ± 0.78||2.28 ± 1.27*||3.85 ± 0.37||1 ± 0||1.42 ± 1.13|
|Dentistry and general questions (n = 6)||3.6 ± 0.89||3.8 ± 0.40||2.6 ± 0.89||2.8 ± 0.40||2.8 ± 1.63||4 ± 0||1.6 ± 1.34||1.8 ± 1.21|
|Machine and techniques (n = 14)||3.71 ± 0.72||3.78 ± 0.80||2.85 ± 0.53||3.07 ± 0.26||2.92 ± 1.49||3.64 ± 0.63||1.35 ± 0.92||2.21 ± 1.41|
* p < 0.01 is statistically significant (independent t test)
Microsoft Cortana was launched in January 2015.7 Cortana was demonstrated for the first time at the Microsoft BUILD Developer Conference in San Francisco.9 Cortona could not respond to 67 questions. When a specific question was asked, this device was able to point to a scientific article instead of showing multiple websites especially regarding questions about differential diagnosis. A significant difference between the examiners was observed in the categories of anatomy and differential diagnosis. This could be due to the difference in pronunciation as there is a variation of accents in different regions around the world. Among the four virtual assistants evaluated, Cortona was the only one that pulled up images and videos for a few queries.
Google Inc. launched the Google Assistant in May 2016.7 The proposed advantage of Google Assistant was that it provided an ambient two-way conversational experience that uses natural language to provide an audio and voice search. In this study, Google Assistant could answer 81 questions and proved to be more efficient compared to the other three virtual assistants. There was a significant difference between the examiners in the questions in the Medication category. Most of the time Google Assistant provided a reference to the article. The answers were more accurate in response to the query compared to the rest of the virtual assistants. Google failed to answer only the questions, which require human interference where the specific answer had to be retrieved from the material provided.
A study done by Nobles et al. looked at responses to addiction and help-seeking from Alexa, Siri, Google Assistant, Cortana, and Bixby intelligent virtual assistants where they found only Google Assistant provided a referral to a mobile cessation “App” for smoking or tobacco use.10
A similar study done by Miner et al. posed a standardized panel of questions related to mental health, interpersonal violence, and physical health to four widely used AI voice assistants: Siri, Google, S-Voice (Samsung), and Cortana. When asked simple questions about mental health, interpersonal violence, and physical health, most of them responded inconsistently so they concluded that if these conversational agents are to respond fully and effectively to health concerns, their efficiency needs to be improved.11 This conclusion is similar to our study where the results were helpful for the task but needed significant improvements for it to be effectively integrated into the workflow.
At this point, most of the commonly used voice-based virtual assistants appear to recognize a significant number of topics related to dentistry and oral and maxillofacial radiology but do not seem to provide accurate information relative to the queries used in this study. The responses to queries are based on the database that supports the front end of any virtual assistant. It is important to note that devices like Alexa were specifically developed for facilitating and streamlining shopping lists on Amazon and eventually expanded to control household electronic gadgets but were not designed to answer medical and dental-related queries. In contrast, the Google Assistant is supported by Google’s database, which is a much more generic and a much more robust search engine with a large database. The Google virtual assistant in this study pointed to scientific literature in response to certain questions because Google also hosts a significant number of scientific textbooks and manuscripts on the Google Scholar portal which is a scholarly segment of the larger database. This technology is in its evolutionary stage and is typically being developed for specific market needs and so the cross-functional queries may not always yield the necessary responses unless the devices are supported by a broad and inclusive database. A front-end interface is only as good as the backend database that supports it. If voice-based assistants need to carve out a role in medical and dental practices, then field-specific modules or larger databases need to be developed to support the front end of these devices.
While there are no comparable studies specific to our study’s objective, a study done by Feng Lui et al. in 2014, compared the intelligence quotient (IQ) of internet search engines to human beings. To test this idea, they proposed the stipulation of the 2014 internet intelligence scale and designed an IQ test question bank to test the search engine. By comparing 7 classic search engines, such as Google, Baidu, Sogou, Bing, Zhongsou, panguso and 20 children whose ages are 6, 12, 18, the paper has found that the current IQ of an Internet search engine to be more than that of human beings.12
As a continuation, similar research was conducted by Feng Lui et al. in 2016, where they tested the IQ of artificially intelligent systems like Google, Baidu, Sogou, and others as well as Apple’s Siri and Microsoft’s Xiaobing to human intelligence. The results show Google as the most sophisticated AI with an absolute IQ of 47.28 compared to an average 6-year-old human with 55.5. However, human IQ scales are typically adjusted by age, so this is not a clean comparison, but it does provide some perspective on the trends of virtual search engines and their abilities.13
Another recent advancement is the evolution of smart speakers and it is increasing in use is majorly due to changing consumer preferences toward AI-driven products. Another example is the voice-enabled and AI-powered digital assistant, launched by United Physician Management known as Suki. It was designed to help doctors with medical charting during patient visits. In July 2019, Amazon and the UK Government announced Alexa devices would use the NHS website to provide information to users seeking health advice. Using NHS data will help in giving confidence to all those accessing the data that the information they receive is from a reliable source.14
More recent advances in the use of AI and AI-based voice technology got a significant boost during the COVID-19 pandemic where several touchless technologies were explored, and applications were developed to avoid personal contact. Voice-based virtual assistants proved to be very helpful in providing screening options at several entry points in buildings and were also powered to answer some basic questions. In a recent study by Sezgin et al., the authors explored the role of voice assistants and chatbots and their role in healthcare and telemedicine and concluded that voice assistants have played a major role during the COVID pandemic and they hypothesize those voice assistants will continue to play a significant supporting role in healthcare delivery.15 Another recent study by Hollander and Carr studied strategies for healthcare surge control and discussed the concept of forward triaging where sorting of patients and their emergency needs can be better managed if they are adequately screened and the problem is better understood. This can be managed using voice-based assistants that can record patient complaints, symptoms, and analyze the best possible triage options. This minimizes personal contact and allows the delivery of care efficiently.16 There is emerging evidence regarding the utility and application of AI-powered voice assistants but further studies need to be done to establish this concept better.
There are several limitations to the application of AI-powered voice assistants in radiology report writing, largely because the currently available AI assistants were developed for commercial applications like shopping lists or recreational activities, perhaps a medical-based module’s development will help its utilization in healthcare. The current status indicates that there is a long road ahead of us before we develop computing and internet systems that can perform better than human beings. But the efforts to produce humanoid-like AI systems are well underway. In this study, we queried only 100 questions limited to a specific field in dentistry and for devices that were not designed for this purpose. Despite this, most of the devices did reasonably well. We also used only two evaluators in a single site setting which could be a limitation of the study and understanding the scope of this application. This study with all its limitations could serve as the one that explored the proof of concept in this area.
Future directions should include conducting similar studies with multiple users in broader and multiple settings and with a wider range of questions. Overall, there is a promising future for voice-powered virtual assistants. Integration of an increased two-way conversation with natural language processing and more robust back-end databases will help in making their presence useful in the field of medicine and dentistry.
In this small study that queried only four voice-powered virtual assistants, voice-powered virtual assistants were helpful and convenient to use in responding to questions regarding oral and maxillofacial radiology. But there is significant scope for expansion in the number of topics and information before these can be used specifically for oral and maxillofacial radiology.
1. Perez JA, Deligianni F, Ravi D, et al. Artificial intelligence and robotics. arXiv preprint arXiv: 1803.10813. 2018 Mar 28.
2. Vlahos J. Talk to me: How voice computing will transform the way we live, work, and think. Eamon Dolan Books; 2019;.
7. Chen LC, Shih IC, Liu JS. Identifying the main paths of knowledge diffusion in the voice over internet protocol. J Internet Technol 2020;21(1):85–98.
8. Grudin J, Jacques R. Chatbots, humbots, and the quest for artificial general intelligence. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems 2019 May 2 (pp. 1–11).
10. Nobles AL, Leas EC, Caputi TL, et al. Responses to addiction help-seeking from Alexa, Siri, Google Assistant, Cortana, and Bixby intelligent virtual assistants. NPJ Digit Med 2020;3(1):1–3 DOI: 10.1038/s41746-019-0215-9.
11. Miner AS, Milstein A, Schueller S, et al. Smartphone-based conversational agents and responses to questions about mental health, interpersonal violence, and physical health. JAMA Inter Med 2016;176(5):619–625 DOI: 10.1001/jamainternmed.2016.0400.
14. Topol E. Preparing the healthcare workforce to deliver the digital future the topol review: An independent report on behalf of the secretary of state for health and social care. NHS Health Education: London, UK. 2019;.
15. Sezgin E, Huang Y, Ramtekkar U, et al. Readiness for voice assistants to support healthcare delivery during a health crisis and pandemic. NPJ Digit Med 2020;3(1):1–4 DOI: 10.1038/s41746-020-00332-0.
© The Author(s). 2021 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (https://creativecommons.org/licenses/by-nc/4.0/), which permits unrestricted use, distribution, and non-commercial reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.