Usefulness of Artificial Intelligence-based Virtual Assistants in Oral and Maxillofacial Radiology Report Writing

MATERIALS AND METHODS

Two oral radiology residents in training (KM and RV) queried a questionnaire consisting of 100 questions to 4 commercially available voice-based virtual assistants: Alexa, Siri, Cortana, and Google Assistant. The questions were categorized based on the five categories. These categories were developed based on discussions with two senior board-certified oral and maxillofacial radiologists and two radiology residents in training. The categorization was based on the frequency and reason for a radiologist to refer to either a textbook or an online resource before diagnosing and finalizing a radiology report.

These include:

• Patient history and syndromes.
• Medications and short forms of diseases.
• Machine and techniques.
• Anatomy, dental anatomy, and dimensions.
• Pathology and differential diagnoses.

The evaluators queried the devices and rated them on a 4-point scale using a modified Likert scale: (1) The device responded with adequate information, (2) The device responded but did not provide adequate information, (3) The device did not know the response to the question, (4) The device recognizes but does not provide adequate information instead it leads to series of website pages. The results of the retrieved queries were recorded on a Microsoft Excel sheet for statistical analysis to do the comparative evaluation.

A sample of questions related to dentistry and oral radiology is provided in Table 1. The full list of questions has been provided in the supplemental material.

RESULTS

Two investigators (KM and RV) queried all four virtual assistants regarding: (1) Patient data and syndrome-related questions, (2) Medications, (3) Anatomy, (4) Differential diagnosis, (5) Dentistry-related general questions, and (6) Knowledge about the radiographic acquisition modality/machine and different acquisition techniques and technical factors. A comparative statistical evaluation of the retrieved queries was done using SPSS version-20 (IBM Inc.).

The mean opinion score for each of the questions between both the evaluators was calculated using one-way ANOVA and post hoc test. Of all the devices queried, Google Assistant was the device that responded to all the questions in the most efficient manner. The mean score for Google Assistant was significantly less for all the questions showing the highest efficiency, followed by Microsoft and Alexa. Siri is the least with the highest mean score (Figs 1 and 2). Tables 2 and 3 show the comparison of the mean response score of the first (KM) and second (RV) investigator, respectively. The dissimilar alphabets denote the significant difference between the groups. In this the lower the mean opinion score, the more efficient the device would be.

Table 4 shows there was a significant difference between the examiners was observed in the categories of (3) Anatomy and (4) Differential diagnosis. Also, there was a significant difference between the examiners for the Google Assistant in the (2) Medication category.

DISCUSSION

Several innovations are being powered by AI and can perform tasks that one could not have imagined a few years ago. Voice-powered personal assistants are a classic example of how life has changed in the last few years because of access to information via gadgets like these. Question and answer-based gadgets like voice-activated personal assistants powered by large databases are an example of AI. Artificial intelligence is a branch of computer science that addresses simulation of intelligent behavior in computers, or the capability of a machine to imitate intelligent human behavior. Voice-powered assistants mainly work via cloud-based software programs that use internet-based data archives to respond to queries. There are currently several devices dedicated to providing virtual assistance. The most popular ones are available from Apple, Amazon, Google, and Microsoft.⁶ Alexa is a standalone voice-activated virtual assistant. Alexa is supported by Amazon, Cortana by Microsoft. Siri is available on all Apple devices including smartphones and tablet computer devices. Cortana is available on both Microsoft-supported smartphones and computers. Google Assistant is available on google supported smartphones as well as on independent devices.

Voice-powered virtual assistants can help oral radiologists in various situations where they are seeking additional information that requires referring to textbooks or browse the internet for specific information. While this is a fairly logical concept to use these devices to provide a simple voice-based interface into the workflow, currently there are no such studies, and this is the first of its kind. In this study, we looked at the status of these four-voice activated virtual assistants, specifically in the field of oral and maxillofacial radiology. While writing a radiology report, an oral and maxillofacial radiologist encounters various situations in the clinical requisition forms and clinical history, like new abbreviations, new terminology, or newer or unfamiliar tests or diseases. They typically use the internet to learn more about these terms and many times radiologists refer to a textbook or a published article about certain medical and dental conditions and the current status of research or treatment about the condition. Radiologists often refer to normal anatomic measurements to determine if the findings are consistent with normal morphological measurements or if they are a variant of normal to confirm or rule out pathology. Many radiologists refer to the internet for position papers or white papers on a topic to understand the status of science in a particular area or procedure and for updated guidance. This consumes an enormous amount of time and it would be convenient if there was a voice-activated assistant that could provide accurate information.

Siri is Apple’s voice-based virtual assistant manufactured by Apple Inc., which was launched in 2011 and has become a very commonly used voice assistant especially since apple has a significant market share in the smartphone user’s market.⁷ Out of 100 questions, Siri was able to recognize 85 questions but failed to provide adequate information. Instead, it led the evaluators to a series of websites. Though it could answer a few questions, it could not provide specific responses to the questions posed regarding commonly available medications, generic names of the medicines and chromosomes associated with specific syndromes, etc. Siri was unable to provide specific information regarding imaging equipment used in oral radiology and imaging techniques. Siri was accurate for only 7 out of 100 questions.

Alexa was launched in November 2014 by Amazon Lab126 and is the first voice-activated virtual assistant linked to a stand-alone home device rather than integrated into existing electronic devices.⁸ Though it answered around 22 questions accurately, it failed to respond to nearly 69 questions. It failed to answer most of the medication-related questions and many dental-related queries. While it did recognize a significant number of questions, an in-depth answer was not a straightforward retrieval of the query.

Microsoft Cortana was launched in January 2015.⁷ Cortana was demonstrated for the first time at the Microsoft BUILD Developer Conference in San Francisco.⁹ Cortona could not respond to 67 questions. When a specific question was asked, this device was able to point to a scientific article instead of showing multiple websites especially regarding questions about differential diagnosis. A significant difference between the examiners was observed in the categories of anatomy and differential diagnosis. This could be due to the difference in pronunciation as there is a variation of accents in different regions around the world. Among the four virtual assistants evaluated, Cortona was the only one that pulled up images and videos for a few queries.

Google Inc. launched the Google Assistant in May 2016.⁷ The proposed advantage of Google Assistant was that it provided an ambient two-way conversational experience that uses natural language to provide an audio and voice search. In this study, Google Assistant could answer 81 questions and proved to be more efficient compared to the other three virtual assistants. There was a significant difference between the examiners in the questions in the Medication category. Most of the time Google Assistant provided a reference to the article. The answers were more accurate in response to the query compared to the rest of the virtual assistants. Google failed to answer only the questions, which require human interference where the specific answer had to be retrieved from the material provided.

A study done by Nobles et al. looked at responses to addiction and help-seeking from Alexa, Siri, Google Assistant, Cortana, and Bixby intelligent virtual assistants where they found only Google Assistant provided a referral to a mobile cessation “App” for smoking or tobacco use.¹⁰

A similar study done by Miner et al. posed a standardized panel of questions related to mental health, interpersonal violence, and physical health to four widely used AI voice assistants: Siri, Google, S-Voice (Samsung), and Cortana. When asked simple questions about mental health, interpersonal violence, and physical health, most of them responded inconsistently so they concluded that if these conversational agents are to respond fully and effectively to health concerns, their efficiency needs to be improved.¹¹ This conclusion is similar to our study where the results were helpful for the task but needed significant improvements for it to be effectively integrated into the workflow.

At this point, most of the commonly used voice-based virtual assistants appear to recognize a significant number of topics related to dentistry and oral and maxillofacial radiology but do not seem to provide accurate information relative to the queries used in this study. The responses to queries are based on the database that supports the front end of any virtual assistant. It is important to note that devices like Alexa were specifically developed for facilitating and streamlining shopping lists on Amazon and eventually expanded to control household electronic gadgets but were not designed to answer medical and dental-related queries. In contrast, the Google Assistant is supported by Google’s database, which is a much more generic and a much more robust search engine with a large database. The Google virtual assistant in this study pointed to scientific literature in response to certain questions because Google also hosts a significant number of scientific textbooks and manuscripts on the Google Scholar portal which is a scholarly segment of the larger database. This technology is in its evolutionary stage and is typically being developed for specific market needs and so the cross-functional queries may not always yield the necessary responses unless the devices are supported by a broad and inclusive database. A front-end interface is only as good as the backend database that supports it. If voice-based assistants need to carve out a role in medical and dental practices, then field-specific modules or larger databases need to be developed to support the front end of these devices.

While there are no comparable studies specific to our study’s objective, a study done by Feng Lui et al. in 2014, compared the intelligence quotient (IQ) of internet search engines to human beings. To test this idea, they proposed the stipulation of the 2014 internet intelligence scale and designed an IQ test question bank to test the search engine. By comparing 7 classic search engines, such as Google, Baidu, Sogou, Bing, Zhongsou, panguso and 20 children whose ages are 6, 12, 18, the paper has found that the current IQ of an Internet search engine to be more than that of human beings.¹²

As a continuation, similar research was conducted by Feng Lui et al. in 2016, where they tested the IQ of artificially intelligent systems like Google, Baidu, Sogou, and others as well as Apple’s Siri and Microsoft’s Xiaobing to human intelligence. The results show Google as the most sophisticated AI with an absolute IQ of 47.28 compared to an average 6-year-old human with 55.5. However, human IQ scales are typically adjusted by age, so this is not a clean comparison, but it does provide some perspective on the trends of virtual search engines and their abilities.¹³

Another recent advancement is the evolution of smart speakers and it is increasing in use is majorly due to changing consumer preferences toward AI-driven products. Another example is the voice-enabled and AI-powered digital assistant, launched by United Physician Management known as Suki. It was designed to help doctors with medical charting during patient visits. In July 2019, Amazon and the UK Government announced Alexa devices would use the NHS website to provide information to users seeking health advice. Using NHS data will help in giving confidence to all those accessing the data that the information they receive is from a reliable source.¹⁴

More recent advances in the use of AI and AI-based voice technology got a significant boost during the COVID-19 pandemic where several touchless technologies were explored, and applications were developed to avoid personal contact. Voice-based virtual assistants proved to be very helpful in providing screening options at several entry points in buildings and were also powered to answer some basic questions. In a recent study by Sezgin et al., the authors explored the role of voice assistants and chatbots and their role in healthcare and telemedicine and concluded that voice assistants have played a major role during the COVID pandemic and they hypothesize those voice assistants will continue to play a significant supporting role in healthcare delivery.¹⁵ Another recent study by Hollander and Carr studied strategies for healthcare surge control and discussed the concept of forward triaging where sorting of patients and their emergency needs can be better managed if they are adequately screened and the problem is better understood. This can be managed using voice-based assistants that can record patient complaints, symptoms, and analyze the best possible triage options. This minimizes personal contact and allows the delivery of care efficiently.¹⁶ There is emerging evidence regarding the utility and application of AI-powered voice assistants but further studies need to be done to establish this concept better.

There are several limitations to the application of AI-powered voice assistants in radiology report writing, largely because the currently available AI assistants were developed for commercial applications like shopping lists or recreational activities, perhaps a medical-based module’s development will help its utilization in healthcare. The current status indicates that there is a long road ahead of us before we develop computing and internet systems that can perform better than human beings. But the efforts to produce humanoid-like AI systems are well underway. In this study, we queried only 100 questions limited to a specific field in dentistry and for devices that were not designed for this purpose. Despite this, most of the devices did reasonably well. We also used only two evaluators in a single site setting which could be a limitation of the study and understanding the scope of this application. This study with all its limitations could serve as the one that explored the proof of concept in this area.

Future directions should include conducting similar studies with multiple users in broader and multiple settings and with a wider range of questions. Overall, there is a promising future for voice-powered virtual assistants. Integration of an increased two-way conversation with natural language processing and more robust back-end databases will help in making their presence useful in the field of medicine and dentistry.

CONCLUSION

In this small study that queried only four voice-powered virtual assistants, voice-powered virtual assistants were helpful and convenient to use in responding to questions regarding oral and maxillofacial radiology. But there is significant scope for expansion in the number of topics and information before these can be used specifically for oral and maxillofacial radiology.

REFERENCES

1. Perez JA, Deligianni F, Ravi D, et al. Artificial intelligence and robotics. arXiv preprint arXiv: 1803.10813. 2018 Mar 28.

2. Vlahos J. Talk to me: How voice computing will transform the way we live, work, and think. Eamon Dolan Books; 2019;.

3. Merenda M, Porcaro C, Iero D. Edge machine learning for AI-enabled IoT devices: a review. Sensors 2020;20(9):2533 DOI: 10.3390/s20092533.

4. Harmon SA, Tuncer S, Sanford T, et al. Artificial intelligence at the intersection of pathology and radiology in prostate cancer. Diagn Interv Radiol 2019;25(3):183 DOI: 10.5152/dir.2019.19125.

5. Hoy MB. Alexa, Siri, Cortana, and more: an introduction to voice assistants. Med Refer Serv Quart 2018;37(1):81–88 DOI: 10.1080/02763869.2018.1404391.

6. Yu KH, Beam AL, Kohane IS. Artificial intelligence in healthcare. Nat Biomed Engineer 2018;2(10):719–731 DOI: 10.1038/s41551-018-0305-z.

7. Chen LC, Shih IC, Liu JS. Identifying the main paths of knowledge diffusion in the voice over internet protocol. J Internet Technol 2020;21(1):85–98.

8. Grudin J, Jacques R. Chatbots, humbots, and the quest for artificial general intelligence. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems 2019 May 2 (pp. 1–11).

9. Ford M, Palmer W. Alexa, are you listening to me? An analysis of Alexa voice service network traffic. Perso Ubiquit Comput 2019;23(1):67–79 DOI: 10.1007/s00779-018-1174-x.

10. Nobles AL, Leas EC, Caputi TL, et al. Responses to addiction help-seeking from Alexa, Siri, Google Assistant, Cortana, and Bixby intelligent virtual assistants. NPJ Digit Med 2020;3(1):1–3 DOI: 10.1038/s41746-019-0215-9.

11. Miner AS, Milstein A, Schueller S, et al. Smartphone-based conversational agents and responses to questions about mental health, interpersonal violence, and physical health. JAMA Inter Med 2016;176(5):619–625 DOI: 10.1001/jamainternmed.2016.0400.

12. Liu F, Shi Y. The search engine IQ test based on the internet IQ evaluation algorithm. Procedia Comp Sci 2014;311066–1073 DOI: 10.1016/j.procs.2014.05.361.

13. Liu F, Shi Y, Liu Y. Intelligence quotient and intelligence grade of artificial intelligence. Ann Data Sci 2017;4(2):179–191 DOI: 10.1007/s40745-017-0109-0.

14. Topol E. Preparing the healthcare workforce to deliver the digital future the topol review: An independent report on behalf of the secretary of state for health and social care. NHS Health Education: London, UK. 2019;.

15. Sezgin E, Huang Y, Ramtekkar U, et al. Readiness for voice assistants to support healthcare delivery during a health crisis and pandemic. NPJ Digit Med 2020;3(1):1–4 DOI: 10.1038/s41746-020-00332-0.

16. Hollander JE, Carr BG. Virtually perfect? Telemedicine for COVID-19. N Eng J Med 2020;382(18):1679–1681 DOI: 10.1056/NEJMp2003539.