University of Groningen Quality improvement in radiology reporting by imaging informatics and machine learning Olthof, Allard

(1)

University of Groningen

Quality improvement in radiology reporting by imaging informatics and machine learning

Olthof, Allard

DOI:

10.33612/diss.168901920

IMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite from

it. Please check the document version below.

Document Version

Publisher's PDF, also known as Version of record

Publication date:

2021

Link to publication in University of Groningen/UMCG research database

Citation for published version (APA):

Olthof, A. (2021). Quality improvement in radiology reporting by imaging informatics and machine learning.

University of Groningen. https://doi.org/10.33612/diss.168901920

Copyright

Other than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons).

Take-down policy

If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim.

Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons the number of authors shown on this cover page is limited to 10 maximum.

(2)

558259-L-sub01-bw-Olthof 558259-L-sub01-bw-Olthof 558259-L-sub01-bw-Olthof 558259-L-sub01-bw-Olthof Processed on: 14-4-2021 Processed on: 14-4-2021 Processed on: 14-4-2021

Processed on: 14-4-2021 PDF page: 7PDF page: 7PDF page: 7PDF page: 7

GENERAL INTRODUCTION

(3)

“Health care is a sacred mission. A moral enterprise and a scientific enterprise but not fundamentally a commercial one. We are not selling a product. We do not have a consumer who understands everything and makes rational choices—and I include myself here. Doctors and nurses are stewards of something precious. ... Ultimately, the secret of quality is love. You have to love your patient, you have to love your profession, you have to love your God. If you have love, you can then work backward to monitor and improve the system.”

Avedis Donabedian (7 January 1919–9 November 2000) was a physician and founder of the study of quality in health care and medical outcome research, most famously as the creator of the Donabedian Model of care. [1]

10 | Chapter 1

(4)

General introduction

Radiology is essential in many diagnostic processes and, therefore, plays a central role in healthcare. The trends that are applicable in healthcare in general are also applicable in radiology. Therefore, patient-centered and value-based healthcare should be provided in a transparent and evidence-based manner, in an environment of limited resources and hospital mergers [2, 3]. To deal with these challenges, a focus on quality improvement and workflow efficiency is vital.

At the intersection of clinical radiology, data science, and information technology, a relatively young medical subspecialty has emerged: medical imaging informatics. This thesis’ objective is to explore the role of medical imaging informatics in the quality improvement of radiology reporting.

Quality improvement

Humans make errors, indicating the necessity of quality management. To err is human: Building a safer health system [4] is a landmark paper in the rich history of quality improvement science [5]. A key figure and pioneer in this field is professor Avedis Donabedian. The concept of evaluating healthcare in the three domains of structure, process, and outcome is his renowned framework for quality improvement [6].

Healthcare quality improvement can learn from industries such as aviation and car manufacturing, wherein quality improvement has long been common practice to evaluate technical equipment for its ease of use, effectiveness and safety [7, 8]. A systematic and multidisciplinary approach in quality improvement is advocated, where, for example, the plan−do−check−act−model and root cause analysis can help professionals to improve quality and safety [9, 10]. When an adverse event happens, it is important to learn from it to improve quality and safety [11]. The chances to learn are improved in a non-punitive environment with a culture of safety [12]. The importance of quality management is not only reflected by all the efforts to improve safety itself, but also by the broader context of legislation, certifications, governmental regulations, and malpractice liability [13]. The incentive to prioritize quality is further illustrated by the transition from volume- to value-based healthcare in general, as well as in radiology reporting [14–16].

Chapter 1 | 11

(5)

Radiology reporting

The first radiology report was a handwritten letter in 1896 [17]. After the rise of the Dictaphone from the 1960s onward and the later development of digitalization, dictation and transcription were made possible. A new technique was introduced in the late eighties of the twentieth century: voice recognition-based radiology reporting [18]. Milestones of radiology reporting not only encompass the method, but also the structure and content. Traditionally, free text is used to compose radiology reports. It offers radiologists the opportunity to use their creativity and personal preferences. Drawbacks of free text radiology reports are lack of standardization, ambiguous wording, incompleteness and difficulty to extract information from the reports [17, 19]. Structured reporting has been introduced to improve report consistency and, as such, improve communication. Structured reporting is not only about the content of the report, but also how the items of the report are processed and can be applied to divergent levels of structured layout and content [20]. Structured reporting has proven its value in many subspecialties in radiology [19, 21–24]. It contributes to data retrieval from reports supporting research and quality initiatives [25]. As carriers of medical information, the radiology report is a topic of research. The variation in the length and content of radiology reports among radiologists illustrates the potential for improvement [26]. The content of radiology reporting has been investigated to assess its role in adding value to clinical practice [27, 28].

Radiology reporting cannot be separated from radiology workflow in general (Figure 1). This starts with the referring physician making an order for a radiological examination for the patient and ends with that same physician discussing the results of the radiology report with the patient for decision making.

Imaging informatics

The radiology workflow and the development of imaging informatics are deeply related. In the film-based era, the impact of informatics on radiology was limited. The development of the picture archiving and communication system (PACS) in the 1980s fostered the relationship between radiology and information technology. The digital delivery of radiologic images to the radiologist’s workstation changed the professional life of all employees of radiology departments and had a major impact on the whole hospital. Client-server-based post-processing technologies made the 3D visualization of CT and MRI datasets possible. Radiomics extracts information from images that surpass the human eye and has the

12 | Chapter 1

(6)

potential to provide prognostic information [29]. This illustrates that the impact of imaging informatics on radiology is broader than just the storage and delivery of images; it is also about the radiologic data itself and methods to turn data into actionable insights. Data science, big data, artificial intelligence . . . radiology is changing rapidly.

Artificial intelligence is the broad umbrella term used to describe the field of computer science dealing with systems that perform tasks that usually require human intelligence [30]. In machine learning, those systems are trained by exposure to data. Deep learning is a form of machine learning where neural networks with multiple layers and architectures are applied. In radiology, these techniques can be applied to images and radiology reports, concerning the domains of computer vision and natural language processing [31].

Figure 1. Schematic of radiology workflow, including tools investigated in this thesis and applied to this workflow and

the purpose of this application.

Chapter 1 | 13

(7)

Since 2019, medical imaging informatics has been included in the European Training Curriculum for Subspecialisation in Radiology [31]. In its 2020 update among technical, organizational, and regulatory topics, the following quality-related topics are also described:

• To learn how to use tools and procedures to assist radiologists, technicians, and referring physicians with continuous quality improvement and dose management • To set up and maintain a quality management program related to imaging

informatics, including all modalities used for medical imaging

• To guide radiologists in implementing the usage of extracted quantitative data to enhance the value of radiological services and to improve patient care

This indicates the role of the medical imaging informatics radiologist in the intersection of radiology, imaging informatics, and quality management.

Scope

The scope of this thesis is radiology reporting in clinical practice and the adaptation of existing pathways and workflows to improve quality. Quality in this thesis refers to the added value of the radiology report from the viewpoint of referring physicians and radiologists. Regarding referring physicians, this is elaborated concerning the degree to which the expectations of the referring physician are met and the usefulness of information passed on by a radiology report. Regarding radiologists, the scope is the level of agreement among radiologists concerning a case. Subsequently, the thesis regards quality in the sense of the uniformity of radiology reporting and the degree to which the report’s content adheres to guidelines. The quality of data can refer to the qualitative or quantitative pieces of information embedded in it. From this perspective, quality is investigated in the sense of the addition of information to or the extraction of information from radiology reports by AI and natural language processing (NLP).

Radiology reporting is the core business of radiologists; therefore, research on quality improvement is essential. The 332 radiology PhD theses in the Dutch research database NARCIS cover a vast area of important topics. However, none is exclusively about the quality improvement of radiology reporting [32]. This thesis tries to fill this gap by providing examples of evidence-based quality improvement projects applicable to all radiology subspecialties. The implementation of quality improvement initiatives has great potential because of the increasing radiology volume [33]. Therefore, reducing errors and improving the quality of radiology reporting can have a major impact on healthcare.

14 | Chapter 1

(8)

Research questions, methodology, and thesis layout

The aim of this thesis is to develop and explore imaging informatics and machine learning tools to improve the quality of radiology reporting. Figure 1 depicts the four tools and four outcome domains. The two central research questions of this thesis are as follows:

• How can feedback systems, structured reporting, natural language processing, and artificial intelligence be applied to radiology workflow?

• What are the results of this application on four domains of quality: guideline implementation, workflow improvement, quality assessment, and epidemiological insight?

Therefore, in part I, we assess the quality of radiology reporting by feedback.

In part II, we develop structured reporting methods and investigate how these impact

radiology reporting.

In part III, we develop and evaluate natural language processing methods to extract

information from radiology reports. Furthermore, we assess the impact of artificial intelligence on radiology. The research questions and the applied methodology are described in Table 1 and correspond to the thesis’ layout. In the general discussion, the findings of all projects are integrated and put into perspective.

Setting

The implementation studies of parts I and II are performed in a three-location hospital group. The tools are implemented in the daily workflow of clinical practice. The evaluation studies are based on data retrieved from the PACS. The studies on artificial intelligence in part III were performed in collaboration with researchers from the University Medical Center Groningen and the Vrije Universiteit Amsterdam.

Questions Methodology Chapter Part I: Feedback

What is the quality of radiology reports, as perceived by referring clinicians? What can radiologists learn from them to improve radiology reporting?

Enquiry among neurologists in general and university hospitals to obtain feedback about radiology reporting.

2

How can radiologists provide peer feedback about missed diagnoses? How can this contribute to quality improvement? How can this feedback process seamlessly be integrated into the radiology workflow?

Development, implementation, and testing of a PACS-integrated feedback system for radiologists.

3

Chapter 1 | 15

(9)

Questions Methodology Chapter Part II: Structured reporting

How can radiology reporting for response evaluation in oncology be improved? How can this be acceptable both for the referring physicians and radiologists?

Multidisciplinary workflow optimization, including the usage of structured reporting in the setting of a clinical cancer network.

4

How can the communication of critical findings in radiology be improved? How can this be done according to guidelines and without interfering in the workflow?

Development and implementation of a structured reporting template for the communication of critical findings.

5

Part III: Natural language processing and artificial intelligence

What is the best method to extract information from radiology reports by natural language processing?

Development and testing of a natural language processing pipeline for radiology reports with rule-based, machine-learning, and deep-learning methods.

6

What are the effects of dataset size and prevalence on the performance of deep neural networks for the analysis of radiology reports?

Comparison of four deep-learning natural language processing methods and analyses of the impact of dataset size and prevalence.

7

How can natural language processing retrieve aggregated information from collections of radiology requests and reports? How can this be implemented in practice?

Application of deep-learning-based natural language processing on chest-imaging radiology requests and reports.

8

What is the impact of artificial intelligence on the job of the radiologist and radiology reporting? Can the quality of radiology reporting be improved or automated by artificial intelligence?

Systematic technographic review of artificial intelligence applications in neuroradiology and analyses of the impact on the profession of radiologists.

9

16 | Chapter 1

(10)

References

1. Donabedian A (2001) A founder of quality assessment encounters a troubled system firsthand. Interview by Fitzhugh Mullan. Health Aff 20:137–141

2. Itri JN (2015) Patient-centered Radiology. Radiographics 35:1835–46

3. Beaulieu ND, Dafny LS, Landon BE, Dalton JB, Kuye I, McWilliams JM (2020) Changes in Quality of Care after Hospital Mergers and Acquisitions. N Engl J Med 382:51–59

4. Kohn L, Coorigan J (1999) To err is human: Building a safer health system. Summary.

5. Berwick DM, Cassel CK (2020) The NAM and the Quality of Health Care - Inflecting a Field. N Engl J Med 383:505–508

6. Ayanian JZ, Markel H (2016) Donabedian’s lasting framework for health care quality. N Engl J Med 375:205–207 7. Siewert B, Hochman MG (2015) Improving Safety through Human Factors Engineering. Radiographics 35:1694–

705

8. Rosier AS, Tibor LC, Turner MA, Phillips CJ, Kurup AN (2020) Improving Root Cause Analysis of Patient Safety Events in Radiology. Radiographics 40:1434–1440

9. Lee CS, Wadhwa V, Kruskal JB, Larson DB (2015) Conducting a Successful Practice Quality Improvement Project for American Board of Radiology Certification. Radiographics 35:1643–51

10. Bhaludin BN, Shelmerdine SC, Arora S, Senbanjo T, Parthipun A (2014) Delays and errors in abnormal chest radiograph follow-up: a systems approach to promoting patient safety in radiology. J Eval Clin Pract 20:453–9 11. Brook OR, Kruskal JB, Eisenberg RL, Larson DB (2015) Root Cause Analysis: Learning from Adverse Safety Events.

Radiographics 35:1655–67

12. Zygmont ME, Itri JN, Rosenkrantz AB, et al (2017) Radiology Research in Quality and Safety: Current Trends and Future Needs. Acad Radiol 24:263–272

13. Srinivasa Babu A, Brooks ML (2015) The malpractice liability of radiology reports: minimizing the risk. Radiographics 35:547–554

14. Boland GW, Duszak R (2016) A Roadmap for Value-Based Payments. J Am Coll Radiol 13:170–2

15. Sarwar A, Boland G, Monks A, Kruskal JB (2015) Metrics for Radiologists in the Era of Value-based Health Care Delivery. Radiographics 35:866–76

16. Goldberg-Stein S, Chernyak V (2019) Adding Value in Radiology Reporting. J Am Coll Radiol 16:1292–1298 17. Brady AP (2018) Radiology reporting—from Hemingway to HAL? Insights Imaging 9:237–246

18. Robbins AH, Horowitz DM, Srinivasan MK, Vincent ME, Shaffer K, Sadowsky NL, Sonnenfeld M (1987) Speech-controlled generation of radiology reports. Radiology 164:569–573

19. European Society of Radiology (ESR) (2018) ESR paper on structured reporting in radiology. Insights Imaging 9:1–7

20. Nobel JM, Kok EM, Robben SGF (2020) Redefining the structure of structured reporting in radiology. Insights Imaging. doi: 10.1186/s13244-019-0831-6

21. Herts BR, Gandhi NS, Schneider E, Coppa CP, Mody RN, Baker ME, Remer EM (2019) How we do it: Creating consistent structure and content in abdominal radiology report templates. Am J Roentgenol 212:490–496

Chapter 1 | 17

(11)

Processed on: 14-4-2021 PDF page: 16PDF page: 16PDF page: 16PDF page: 16 22. Brown PJ, Rossington H, Taylor J, Lambregts DMJ, Morris E, West NP, Quirke P, Tolan D (2019) Standardised

reports with a template format are superior to free text reports: the case for rectal cancer reporting in clinical practice. Eur Radiol 29:5121–5128

23. Khurana A, Nelson LW, Myers CB, et al (2020) Reporting of acute pancreatitis by radiologists-time for a systematic change with structured reporting template. Abdom Radiol 45:1277–1289

24. Gassenmaier S, Armbruster M, Haasters F, Helfen T, Henzler T, Alibek S, Pförringer D, Sommer WH, Sommer NN (2017) Structured reporting of MRI of the shoulder – improvement of report quality? Eur Radiol 27:4110–4119 25. Ganeshan D, Duong PAT, Probyn L, Lenchik L, McArthur TA, Retrouvey M, Ghobadi EH, Desouches SL, Pastel D,

Francis IR (2018) Structured Reporting in Radiology. Acad Radiol 25:66–73

26. Bosmans JML, Weyler JJ, Parizel PM (2009) Structure and content of radiology reports, a quantitative and qualitative study in eight medical centers. Eur J Radiol 72:354–358

27. Jay Kabadi S, Krishnaraj A (2017) Strategies for improving the value of the radiology report: a retrospective analysis of errors in formally over-read studies. J Am Coll Radiol 14:459–466

28. Shaikh F, Hendrata K, Kolowitz B, Awan O, Shrestha R, Deible C (2017) Value-based assessment of radiology reporting using radiologist-referring physician two-way feedback system; a design thinking-based approach. J Digit Imaging 30:1–8

29. van Timmeren JE, Cester D, Tanadini-Lang S, Alkadhi H, Baessler B (2020) Radiomics in medical imaging—“how-to” guide and critical reflection. Insights Imaging. doi: 10.1186/s13244-020-00887-2

30. Pesapane F, Codari M, Sardanelli F (2018) Artificial intelligence in medical imaging: threat or opportunity? Radiologists again at the forefront of innovation in medicine. Eur Radiol Exp 2:35

31. (2020) Training Curricula | European Society of Radiology. 92–96

32. NARCIS. https://www.narcis.nl/search/Language/en/uquery/radiology. Accessed 21 Jul 2020

33. Rosenkrantz AB, Chaves Cerdas L, Hughes DR, Recht MP, Nass SJ, Hricak H (2020) National Trends in Oncologic Diagnostic Imaging. J Am Coll Radiol. doi: 10.1016/j.jacr.2020.06.001

18 | Chapter 1

(12)

Chapter 1 | 19

(13)

(14)

PART I

(15)