Improving the quality of data of NICE submissions by
means of automated data quality assessment
Daniël Schepers, BSc.
11
Master Medical Informatics, Amsterdam University Medical Centers, location
AMC, University of Amsterdam
Master Thesis
Improving the quality of data of NICE submissions by means of automated data quality
assessment
Author
D. Schepers (Daniël), BSc.
Department Medical Informatics, University of Amsterdam
Meibergdreef 9, 1105 AZ Amsterdam
Supervisor
Dr. ir. S. Brinkman (Sylvia)
Ir. J.L. Boessenkool-Pape (Juliët)
Department Medical Informatics, University of Amsterdam
Meibergdreef 9
1105 AZ Amsterdam
Mentor
T. van Bladel (Teun), BSc.
R. Piening (Remco), MSc.
L. Minne (Lilian), PhD.
Furore
Bos en Lommerplein 280
1055 RW Amsterdam
SRP Duration:
May 2019 – December 2019
SRP Location:
Furore
Bos en Lommerplein 280
1055 RW Amsterdam
Preface
This thesis describes how automated data quality assessment can improve the quality of
data of NICE submissions. I can proudly say that this was my final challenge for the
master Medical Informatics (Amsterdam University Medical Center, location AMC). The
research was conducted under the supervision of Furore Informatica, where I performed
this scientific research project.
I would express my gratitude to Sylvia and Juliët. Late in the evenings, I can still hear those
wise words echoing through the room: "Keep on narrowing your scope, Daniël", "Stick to
the core when writing your thesis", and "Is this really manageable in 8 months?". You
helped me staying focused on the research topic. Without your helpful and constructive
criticism, I perhaps would have been graduated in the distant future.
Teun, Remco, Lilian, thank you all for your guidance and feedback during this research.
The knowledge you were willing to share about the technical facets were extensive and very
welcome. You have familiarized me with a profession in which I would pursue my career.
I was privileged working with you. In addition, I would like to thank everyone at Furore
that provided me with all the necessary resources, valuable insights, and the great times.
My family, friends, beloved girlfiend, and dad -who we’ve lost on the way-, your support
and compassion is inexhaustible. A thousand times thanks for listening to the laments and
giving the feeling that I can come home to you.
I hope you enjoy your reading.
Daniël Schepers
Abstract
Introduction: The National Intensive Care Evaluation (NICE) registry contains data of
patients that are admitted to Dutch intensive care units and provides insight into the
effec-tiveness and efficiency of Dutch intensive care. The submission of intensive care unit-related
patient data comes at a cost. Hospital staff involved in the NICE submission experiences
issues concerning the data quality of patient data and increased administrative burden. In
this study the quality of data of the Sequential Organ Failure Assessment (SOFA) module
is assessed and the support of automated data quality assessment is evaluated.
Methods: A data quality assessment and improvement methodology was adopted in two
hospitals. First, data flows and processes on the intensive care units that are related to the
NICE submission were reconstructed. Second, a literature research was executed to find
previous studies that investigated the data quality of electronic health records (EHRs).
Subsequently, an automated data assessment tool was developed in order to quantify the
quality of data. Finally, data quality improvement activities were defined and evaluated.
Results: The automated assessment tool ran in two hospitals that used software from
the same EHR vendor. The tool showed that 9,46% of the SOFA module is subjected
to data quality issues. It was estimated that 60% of these errors were the result of EHR
software-related issues while the remainder of the errors were due to registration flaws.
Ini-tial evaluation with hospital staff indicated that they are willing to adopt the assessment
tool in the workflow.
Discussion: Structured data quality assessment proved to be useful in identifying data
quality issues that were otherwise submitted to the NICE registry. The automated
assess-ment tool provides insight into causes and potential improveassess-ment areas. The tool can be
used in different settings and can be interpreted by different hospital staff members.
Keywords— EHR data quality, Complete Data Quality Methodology, automated data assess-ment, NICE submission
Samenvatting
Introductie: De Nationale Intensive Care Evaluatie (NICE) registratie verzamelt gegevens van patiënten die zijn opgenomen op Nederlandse intensive care-afdelingen en biedt inzicht in de effectiviteit en efficiëntie van deze afdelingen. Het verzamelen van intensive care-gerelateerde patiëntgegevens heeft ook zijn prijs. Ziekenhuispersoneel dat betrokken is bij de NICE-registratie ondervindt problemen met de datakwaliteit van patiëntgegevens en verhoogde administratieve lasten. In dit onderzoek wordt de kwaliteit van de gegevens beoordeeld die bij de NICE ten behoeve van de SOFA registratiemodule worden ingediend. Daarnaast wordt de ondersteuning van geautomatiseerde datakwaliteitsevaluatie onderzocht.
Methode: Een beoordelings- en verbetermethodologie van de datakwaliteit is in twee ziekenhuizen toegepast. Allereerst zijn datastromen en processen op de intensive cares, die gerelateerd zijn aan de NICE-inzending, gereconstrueerd. Ten tweede werd een literatuuronderzoek uitgevoerd om eerdere studies te vinden die de datakwaliteit van elektronische patiëntendossiers (EPD’s) onderzochten. Vervolgens werd een geautomatiseerd hulpmiddel voor data-controle ontwikkeld om de kwaliteit van data te kwantificeren. Ten slotte werden activiteiten ter verbetering van de datakwaliteit gedefinieerd en geëvalueerd.
Resultaten: De geautomatiseerde beoordelingstool werd geïmplementeerd in twee ziekenhuizen die software van dezelfde EPD-leverancier gebruikten. De tool toonde aan dat 9,46% van de SOFA-module onderhevig is aan problemen met de datakwaliteit. Naar schatting was 60% van deze fouten het gevolg van problemen met de EPD-software, terwijl de rest van de fouten te wijten waren aan registratiefouten.
Discussie: Gestructureerde beoordeling van de datakwaliteit bleek nuttig te zijn bij het identificeren van problemen. Deze problemen zouden anders naar de NICE-registratie toegestuurd zijn. De geautomatiseerde beoordelingstool biedt inzicht in oorzaken en mogelijke verbeteringsgebieden. De tool kan in verschillende instellingen worden gebruikt en kan door verschillende ziekenhuismedewerkers worden gebruikt en geïnterpreteerd.
Steekwoorden— EPD data kwaliteit, Complete Data Kwaliteit Methodologie, geautomatiseerde data-controle, NICE aanlevering
Contents
Preface 2 Abstract 3 Samenvatting 4 List of abbreviations 6 1 Introduction 7 1.1 Context . . . 71.2 Main research question . . . 8
1.3 Research methodology and objectives . . . 8
1.3.1 Objective one – Description of data flows and processes . . . 8
1.3.2 Objective two – Investigating data quality issues . . . 8
1.3.3 Objective three – Choice of improvement activities and evaluation . . . 9
1.4 Chapter organization . . . 9
2 Background 11 2.1 CDQM, a data quality assessment and improvement method . . . 11
2.2 The NICE registry . . . 11
2.3 Data quality – A multidimensional concept . . . 12
2.4 Impact on ICU staff and workflow – Administrative burden . . . 12
3 Methods 14 3.1 Methods objective 1 – Reconstructing data flows and processes . . . 14
3.2 Methods objective 2 – Investigating data quality issues . . . 15
3.3 Methods objective 3 – Process improvement . . . 19
4 Results 20 4.1 Results objective 1 . . . 20 4.2 Results objective 2 . . . 23 4.3 Results objective 3 . . . 31 5 Discussion 38 5.1 Main Findings . . . 38
5.1.1 Reconstruction of data flows and processes . . . 38
5.1.2 Investigation of data quality issues . . . 38
5.1.3 Optimal improvement activities . . . 38
5.2 Interpretations . . . 39
5.3 Strengths and Limitations . . . 39
5.4 Implications . . . 40
5.5 Future Research . . . 40
References 41
Appendices 44
A Interview script phase 1 (Dutch) 44
B Full search term literature review 45
C Full text assessment literature review 47
D Data quality assessment methods per SOFA element 50
List of abbreviations
NICE = National Intensive Care Evaluation ICU = Intensive Care Unit
EHR = Electronice Health Record HIS = Health Information System
CDQM = Complete Data Quality Methodology GCS = Glasgow Coma Scale
FiO2 = Fraction of inspired oxygen
1
Introduction
1.1
Context
Today, the wide-spread adoption of electronic health record (EHR) systems in clinical practice has made it easier to collect, access, and aggregate clinical data. This has led to a growing interest in conducting evidence-based research with data collected during the course of clinical care [37]. The variety of health data stored in EHRs resulted - among others - in the establishment of different clinical registries that entail health information about specific populations. The main purposes of such registries are: gaining insight into the process of patient care, health outcomes for specific patient groups, and treatment plans to improve and monitor patient outcomes [31]. Healthcare providers are enabled, or sometimes obliged, to securely send encrypted patient data to clinical registries based on data derived from the EHR. Participating sites must send their data in a prede-fined format to ensure standardization amongst all data providers. In almost all cases this involves a process of collecting, transforming, assessing, and extracting patient data from the EHR. It is by all means clear that this process can be subjected to issues, in particular data quality concerns [25]. Some of these issues include inconsistent data recording, data conversion or transformation failures, and incompatibility between data models. Another caveat includes the fact that EHRs are designed for clinical care, not specifically as input for quality registries in order to improve the quality of care. This will lead to many object-linking operations, transformation, and calculation activities in order to make the data fit for secondary use in clinical registries and research. In case of inconsequent periodic checks and lack of other validation methods, the possibility arises that inaccurate data might be stored in the clinical registry, and therefore misrepresenting the reality. The assurance of high quality of data in clinical registries is therefore in the interest of all parties. In this research project we will focus on the data submission to one of those clinical data registries: National Intensive Care Evaluation (NICE) registry. This is a large clinical registry that collects data of all IC units in the Netherlands. The registry aims to monitor and optimize the quality of care in IC units. In Chapter two a more comprehensive description of the NICE registry can be found.
To prevent erroneous data to be entered in the registry, clinical registry foundations perform several data quality checks on the submitted data. Although the NICE has an extensive set of activities and data quality checks to optimize the data quality, these cannot assure the accuracy and representativeness of the data without refined data quality checks on the local (hospital) level as the NICE registry receives certain aggregated values derived from the EHRs instead of the whole data set [29]. Preliminary assessment based on interviews with users of two reputable EHR suppliers showed that the EHR suppliers often do not provide sophisticated data quality checks. As a result, activities related to data quality assurance at the hospital often involves increased administrative burden for the healthcare provider [46], therefore automated data quality checks should be developed in order to lower the administrative burden while increasing the quality of data. Data extraction, assessment, and visualization tools may support healthcare providers in analyzing data extracted from electronic health records by means of applying selected techniques that are able to detect data quality issues [36]. Support of these kinds of tools may translate into improved initial recording of patient data and lower administrative burden when preparing the data for the NICE registry. However, there are some challenges that need to be bridged concerning the quality of data delivered to the clinical registry and the administrative burden for healthcare providers.
1.2
Main research question
The main goal of this research project is to improve the quality of data delivered to the NICE registry by means of automated data quality assessment. To reach this goal a data quality as-sessment methodology will be adopted. Within the context of this methodology, the researchers will investigate what data quality is and define approaches to automatically extract, analyze, and visualize NICE-related data quality issues. These approaches will translate into the development of automated data extraction and assessment methods. Automated data extraction and assess-ment methods can be adopted by healthcare providers in the hospital setting and will have two main functionalities: (1) identification of data quality issues found in the EHR by means of auto-mated data quality checks and (2) the visualization of the data quality issues found to support the healthcare provider. The main research question that will be answered in this thesis is, therefore:
Research Question: How can automated data quality assessment support the healthcare provider in improving data quality assurance and control regarding data required for the NICE
registry?
1.3
Research methodology and objectives
Up to now, a number of noteworthy papers defined different methodologies or frameworks for stan-dardizing assessment and reporting of the multidimensional concept data quality [22, 50]. However, relevant literature regarding methodologies for the entire process of identification, assessment, and improvement of EHR data quality is still lacking in the sense that very little approaches have been applied to clinical or health services research studies [22].
A literature search was conducted for identifying existing data quality management method-ologies in different types of fields. A literature review of Batini et al. (2009) identified a number of data quality methodologies and compared them with each other [3]. Batini et al. compared different assessment steps and whether the methodology was extensible to other dimensions and metrics. Especially the latter was considered to be an important criterion for our research design because the methodology should be able to include data quality dimensions, attributes, and mea-sures specific to EHR data. Of the 13 methods that were subjected to the review, the CDQM (Complete Data Quality Methodology) was selected and adopted in this research project because this methodology incorporated all stated assessment steps and was extensible to other dimensions and metrics [2, 3].
CDQM can be applied to all types of data and works in both intra- and inter-organizational contexts by organizing methodological phases within a complete framework. In addition, the methodology supports the selection of the most suitable techniques and tools within each phase. The sequence of activities for CDQM is composed of three phases: (1) state reconstruction, (2) assessment, and (3) improvement process and control.
A research objective had been constructed for each individual CDQM phase in order to answer the main research question, namely (1) description of the current data registration flows and processes, (2) investigating EHR data quality issues in the data required for the NICE submission and (3) choosing data quality improvement activities and evaluation. The methodology is quite new to the field of medical informatics but includes some promising phases that could be well-aligned with the aforementioned research objectives.
Figure 1 includes an in-depth overview and summary of methods and expected results per research objective.
1.3.1
Objective one – Description of data flows and processes
The first objective is reconstructing data flows and processes on the ICU related to the NICE submission. We aim to gain insight in how patient information is processed before submission to the NICE registry. Gaining insight into these subjects will eventually support the initiation of data and process improvements to enhance the data quality of the NICE submission.
1.3.2
Objective two – Investigating data quality issues
The second objective is to investigate which types and causes of data errors in EHRs or medical registry databases can be identified and what data quality assessment measurements or techniques could be applied to measure the data quality. Two sub-questions were defined:
• How to evaluate data quality features in the context of electronic health records?
• What methods are available for measuring the quality of data of electronic health records or clinical registries?
1.3.3
Objective three – Choice of improvement activities and evaluation
The third objective is to assess the expected and actual data quality values. Significant differences will be subjected to data- and process-driven improvement activities. Herewith related data qual-ity issues should be visualized to healthcare providers and this process should be evaluated and validated. The following questions are related to this objective:
• How will assessment of data quality target values compare to actual data quality issues? • What actions should be initiated in order to reach expected or target data quality target
values?
1.4
Chapter organization
The following chapter -background- of this thesis will first outline background information on the methodology that is adopted in this thesis. Subsequently, the NICE registry is elaborated on, followed by information on the concept of EHR data quality. The background section concludes with an in-depth review of the administrative burden for healthcare professionals related to registry submissions. The third chapter describes the methods used which aim to achieve the stated research objectives. Chapter four contains the results per research objective. The concluding chapter includes an overall discussion of this research project, followed by recommendations and statements about future research topics.
2
Background
This chapter provides background information on the Complete Data Quality Methodology (CDQM), the NICE registry, secondary use of clinical (EHR) data, administrative burden related to registry submissions, and the quality of EHR data.
2.1
CDQM, a data quality assessment and improvement method
The Complete Data Quality Methodology (CDQM) introduced in the previous chapter is described by Batini et al. (2006) [2]. CDQM aims to integrate the phases of traditional general-purpose data quality assessment and improvement methodologies like Total Data Quality Management (TDQM) and Total Information Quality Management (TIQM) [14, 39]. By combining these approaches this methodology overcomes a number of limitations like setting explicit quality targets in critical areas and the inclusion of the initial requirements elicitation phase. The core concept of these methodologies is the ‘information product’, which incorporates a model that considers data as an output of a particular process. The CDQM combines both data-driven and process data quality strategies. Data-driven strategies aim to improve the quality of data by directly modifying the value of data while process-driven strategies improve quality by redesigning the processes that create or modify data. A fundamental goal of the CDQM is to support the implementation of a data quality program for all types of organizational data [3].
CDQM is a data quality management methodology that consists of three phases: state re-construction, assessment, and improvement process phase. Applying the CDQM in this research context only required modest adaptations. The following studies were used to adapt the CDQM: (1) EHR data quality dimensions were identified by means of the study of Weiskopf et al. (2013) [50], (2) Concepts of EHR data quality were further assessed by means of the conceptual framework of Chen et al. (2014) [10] and the framework of data quality attributes described by Kahn et al. (2016) [21].
2.2
The NICE registry
Each hospital in the Netherlands is obliged to deliver care that meets predefined quality require-ments. Since 1996, hospitals are required by Dutch law to be accountable in terms of commu-nicating specific standards and numbers of the delivered quality in order to ensure that quality requirements are being met [24, 35].
The National Intensive Care Evaluation (NICE) registry is a Dutch clinical registry and was set up by intensive care medical specialists in 1996. The NICE registry aims to gain insight into how the quality of care of Dutch intensive care can be improved [1]. NICE collects ICU related registration data from participating ICUs and stores acquired data in a national registry database. This database serves multiple purposes as seen from the perspective of the ICU, namely: (1) Providing opportunities for Dutch ICUs to adhere to health and safety legislation by offering services designed by and for ICU staff, (2) ICUs can track their performance over time and are enabled to compare their results to the pooled mean of other comparable ICUs and (3) ICUs can gain insight into possible areas for improvement as a result of conducted research based on submitted data.
Throughout the years the NICE foundation developed several ‘registration modules’ consisting of several data elements. The variety of NICE registration modules ICUs can register to is quite extensive, table 1 provides an overview of the registration modules and the type of data entailed. The medical data included in these modules ranges from structured data values (like vitals) to semi-or unstructured data types (recsemi-ording of neurological assessment of the patient). The impsemi-ortance lies in setting directions to encourage –or rather oblige– participating ICUs to collect the required data correctly and uniformly to prevent data quality issues. The online website of the NICE foundation includes a data dictionary describing which data have to be collected and which data verification rules the ICUs should adhere to [29].
Currently, all Dutch ICU are voluntary submitting at least the so-called minimal dataset (MDS) with anonymized patient data to the NICE registry with the aim to monitor and improve their quality of care. The MDS module consists of 94 data elements and must be submitted once (first 24 hours) when a patient is admitted to the ICU. The Sequential Organ Failure Assessment
Table 1: Registration sets of NICE
NICE registration set Includes variables that describe
Minimal Dataset (MDS) Demography, physiology, reason, and severity of ICU hospitalization within the first 24 hours and several clinical outcome measures.
Sequential Organ Failure
Assessment (SOFA) Daily assessment of six organ systems to monitor and identify organ failure. Quality indicators (KIIC) Quality indicators describing structure-, process- and outcome indicators. Complication Identify the incidence of complications in Dutch ICUs.
Sepsis In case of suspected sepsis, related organ systems, treatment and administration of medication are described.
Nursing capacity module Based on Nursing Activities Scores (NAS), Therapeutic Intervention Scoring System 76 (TISS-76) complemented with variables related to the Dutch situation.
quality issues and administrative burden to a greater extent when compared to other modules. Checking and controlling the data quality of the submission is therefore often cumbersome. The data quality issues that can possibly arise may substantially affect how differences are ascribed in terms of effectiveness among ICUs. Data quality is, in general, ensured at the local hospital site and eventually enforced by the clinical registry operator [45]. Controlling data quality requires certain validation concepts, e.g. audits or periodic checks on data set samples extracted from the EHR [23, 43]. The NICE registry assists participating ICUs by means of a report on the quality of their delivered SOFA data. This is done for each dataset directly after the monthly submission and four times a year about the data that has been delivered till then in the current year. Identifying inaccurate data items in an early stage might, therefore, be crucial because alterations in current collections or extraction procedures could improve data quality [1, 4].
2.3
Data quality – A multidimensional concept
The reuse of EHR data is often limited by several factors, including the data quality and their suitability for research purposes. The adoption of EHRs in the field of medical informatics has not led to considerable improvements concerning data quality, it rather involved an increase in bad quality data being recorded, as stated by Burnum et al. [6]. Concerns about quality of data are present since the introduction of EHRs, however, there is still no consensus on quality of electronic health data or how data quality is conceptualized and defined in the context of EHRs [50]. A broadly adopted conceptualization of data quality is defined through ‘data that is fit for use by data consumers and serve the needs of a given user pursuing specific goals’, as derived from Reeves et al. (1994) [34].
In most literature data quality is recognized as a multi-dimensional concept across the medical informatics -and several other- fields [2, 7, 10, 49]. Whether specific data quality requirements are met is commonly measured along various data quality dimensions and corresponding attributes. These data quality dimensions represent or capture a specific aspect or construct of data quality [49]. Examples of five common dimensions of data quality identified by Weiskopf et al. (2013) include completeness, correctness, concordance, plausibility and currency [50]. Data quality di-mensions on their own do not provide quantitative measures and should, therefore, be associated with one or more metrics. Studies conducted that investigated EHR data quality revealed variable results. The correctness of data ranged between 44% and 100% and completeness varied between 1.1% and 100% according to Hogan and Wagner (1997) [19]. Variability in these results was de-pendent on the clinical concepts being studied. In addition, results of the same clinical concepts also varied across multiple institutions, e.g. Chan et al. (2010) identified that the completeness of blood pressure recordings fell anywhere between 0.1% and 51% [9].
The aforementioned findings conclude that EHR data quality is highly variable which could be explained by differences in recording, information systems, clinical focus, and the fact that EHR data quality assessment will yield different outcomes for a given research task.
2.4
Impact on ICU staff and workflow – Administrative burden
The process of collecting, extraction, and data quality checks for the reuse of EHR data must take place along other daily administrative matters incorporated in the ICU workflow. In addition, the process of checking and controlling data for clinical quality registers is taking up a considerable amount of physician’s valuable time and therefore contributes to an increase in the administrative
burden [46]. Procedures to assure data quality on the local (hospital) level have to be performed simultaneously along with other day-to-day ICU activities. The perceived amount of administrative burden is further influenced by how EHR data is initially recorded and stored [31], the collection methods needed to aggregate the data, and required transformations in order to adhere to data definitions of all NICE data items.
How EHR data is initially recorded and stored depends on the implementation of the hos-pital information system. The adoption of integrated electronic health records in the hoshos-pital information system on the ICU - and in general - facilitated aggregation, standardization, and transition processes of digital patient data. Prerequisite is usually that patient data is recorded in a structured, unambiguous, and easily available manner. However, medical data can be stored in different ways: structured (same format every time), semi-structured (requiring some rules to find the details), and unstructured (does not follow a particular format) [33]. Generally, about 80% of medical data remains unstructured and untapped after it is created [11]. Clinicians gave a number of reasons why they sometimes consider structured recording onerous: Choices available in the software may be limited and may not allow for the expression of nuances [42]. The cognitive load is considerably heavier in the process of selecting and assigning appropriate answers or codes and may, therefore, take longer when compared to summarizing the consultation or findings in text [48]. On the contrary, recording of unstructured narrative free text is highly variable, can be considered more engaging, and will allow expression of feelings from different perspectives [20, 44]. In the context of the ICU, a continuous stream of data input/dataflow is created every day from other related systems (E.g. the patient data management system, PDMS) and ICU staff that both record raw data or information in the electronic health record. Healthcare professionals that record or file data in the EHR also identify different ways of registering data.
In summary, different ways of recording data, continuous streams of data entering the EHR, and healthcare provider-specific ways of registering patient information may further impede the convenience of systematically extracting medical data needed for the NICE submission. The ad-ditional administrative burden associated with NICE related activities in the ICU will therefore increase.
3
Methods
This chapter is subdivided into three sub-sections and includes methods for each research objective stated in Figure 1. Remember that each phase of the CDQ methodology elaborates on one specific research objective.
3.1
Methods objective 1 – Reconstructing data flows and processes
Phase 1
The first research objective was derived from the first phase of the CDQM: reconstructing data flows and processes. In the line of this objective, the subsequent research question was to be answered:
• How and by whom is patient information processed before delivering to the NICE registry?
The first phase of CDQM was about providing insight into the main data users, data providers, and responsibilities of healthcare providers who are involved in NICE submission related activities. The following results were considered relevant: (1) The description of data flows must provide insight into whether ICU staff members either create or use a set of data required for the NICE submission and (2) the identification of processes should describe the responsibilities related to the NICE submission.
The researchers initially investigated data quality issues related to the SOFA module because intensivists experienced data quality issues and administrative burden to a greater extent when compared to other modules.
Interviews
In the first step of this phase semi-structured interviews were conducted by one investigator, the main researcher. During the interview, questions were asked about the following themes: ICU staff involved in the NICE submission, their proceedings, data collection, transformation and submission processes, and the view of ICU staff on possible areas of improvement. The interview forms are included in Appendix I.
In April and May 2019, these semi-structured interviews were prepared and conducted at two Dutch hospitals, a public and a teaching (university) hospital. The interviews were conducted with hospital staff affiliated with the NICE submission. a data manager in the university hospital and two intensivists from the public hospital. Both the data manager and intensivists were largely involved in registering, collecting, and controlling data needed for the NICE submission.
Based on the performed interviews, a data / actor matrix was constructed to get insight in the different responsibilities of intensivist, nurses, data managers, and other medical specialist during the data registration process (e.g. creating, updating data). These responsibilities were defined for grouped NICE elements, which were defined by consulting the NICE data dictionary and several meetings with researchers of the NICE foundation [29].
Process identification and data users
Distinguishing activities and responsibilities amongst the actors of the hospital is important in data quality issues since we can assign responsibilities and/or propose direct suggestions later on in the third phase concerning data- and process-driven improvements. Therefore, in this step, we focused on reconstructing the relationship between processes that are related to data needed for the NICE registry submission and actors of the hospital. The reconstruction of such processes were also used to assess related administrative burden concerning healthcare professionals.
Results of the interviews were reused. Answers related to ICU processes were translated into a process / actor matrix.
3.2
Methods objective 2 – Investigating data quality issues
Phase 2
Phase 2 described what data quality assessment measurements or techniques could be applied and investigated SOFA submission-related data issues. The following research questions were to be answered:
• How to evaluate types and causes of data errors of electronic health records?
• What methods are available for improving or enhancing quality of data of electronic health records or medical registries?
Assessment of data quality issues related to NICE data elements - questionnaires The main goals of the problem definition were (1) the identification of relevant problems and causes of poor data quality and (2) the identification of relevant data quality dimensions for every SOFA data element.
Data quality issues and perceived administrative burden were identified by means of a ques-tionnaire that was disseminated and filled out by two intensivists at a Dutch general hospital. During the assessment period in which the questionnaires were disseminated, the main researcher also visited the intensivists to conduct semi-structured interviews. The questionnaire is included in appendix VI.
Intensivists, the participants, were asked to (1) assess every SOFA element whether data quality issues were present during collection and aggregation, and (2), in case data quality issues were (as-sumed to be) present, what actions they performed in order to retrieve erroneous-free information. Qualitative and quantitative results were then combined into:
• First, the SOFA data elements were grouped based on similar characteristics. Then, based on the interviews, identified data quality issues were assigned to these groups.
• Second, other potential data quality issues were identified. These issues were not explicitly mentioned by the intensivists.
• As a result, occurring and potential data quality issues were assigned to relevant data quality dimensions that were identified by assessing systematic literature reviews. In a following step, assessment methods and techniques to assess these dimensions were constructed.
A schematic overview of this process is shown in Figure 2.
Based on the synthesized results of the problem definition, a systematic literature review was conducted to identify and construct assessment methods and techniques for every relevant data quality dimension. Subsequently, a metric or a method/technique was assigned to every SOFA data element and relevant data quality dimension(s). This step was concluded by measuring the quality of data of the submission data. Data quality assessment was performed through automated data extraction and assessment. Therefore, a data extraction and assessment tool was designed and developed in which the methods identified for quantifying data quality issues were adopted. The tool is used as a method to measure the data quality of SOFA related data.
Methods to quantify EHR data quality issues – Systematic literature review
The metrics, methods or techniques for measuring data quality were identified by means of a systematic literature research. A literature search was performed in Ovid on April 29th 2019. The MEDLINE and EMBASE databases were queried using predetermined search strings that included the terms “data / registration AND Issue/error”, “electronic health record OR medical registry”, and “measure/assessment”. For each term synonyms and related definitions were identified. Terms were mapped to subject headings when appropriate. The search strategy was translated in order to search between the databases.
Full search terms and the structure of the query used are included in appendix II. The PRISMA guidelines on conducting and reporting of systematic reviews were applied [27].
Eligibility
The inclusion criteria were:
1. Discussing methods to assess relevant DQ dimensions (completeness, correctness, concor-dance, plausibility, currency, and other which could be mapped on relevant dimensions); 2. The study should focus on data derived from an EHR, a related system or a medical registry,
or medical data reused for research purposes;
3. The full text of the study must be available and written in English.
No restrictions were placed on publication date. The reference list of relevant studies was reviewed to further identify articles of interest that met our inclusion criteria. Conference papers were excluded from this study.
We excluded studies which discussed irrelevant data quality dimension with regard to this re-search context. Studies questioning the quality of EHR (rere-search) data but not assessing methods or techniques for quantification were also excluded.
Article review
Articles retrieved from the search were first screened based on title and abstract. The reviewing process was performed by the main researcher. Full-text studies were assessed by a single reviewer.
Data extraction
A data collection form was developed. This form was used to extract data from the articles that were included for full-text review. Data abstracted included data quality dimensions assessed, type of EHR data, methods and techniques to assess data quality issues for these dimensions and whether the authors aimed and succeeded to validate their adopted methodology.
Automated data quality assessment - Tool development
A data extraction, transformation, and data assessment tool was designed and developed in order to automatically investigate data quality issues related to SOFA submissions in specified EHRs. This tool includes the attributes and measures identified by means of the systematic literature research.
The extraction part included the aggregation and collection of all relevant patient data during the stay of an ICU patient, whereas the data-related transformation activities were performed to clean up the data. The cleaned up data were used to create a dataset to perform data quality assessment and compose an overview of SOFA data elements which was free of errors and ready to be submitted. Figure 3 provides a schematic overview of how these different elements are inter-related with each other.
Figure 3: Overview related components data extraction and assessment tool
Data quality checks were performed in twofold. (1) The data quality in the context of NICE-related data in the EHR and (2) data quality of the NICE submission.
1. First, the data was tested in the context of the data quality dimensions utilizing the measure-ments identified during the systematic literature review (phase 2 – step 1). All ICU-related data in the EHR were used to determine the data quality of the initial registration process. 2. The EHR automatically aggregates the data that are required for the NICE submission. Initial assessment had shown that this dataset is subjected to data quality issues. In order to identify and quantify these data quality issues the dataset should be compared to another dataset which is error-free. The tool was used to create this error-free dataset required for the NICE submission. The two sets were compared and discrepancies between these sets were recorded and reported. This also enabled the researchers to rule on potential aggregation issues caused by the EHR.
Figure 4 includes an overarching view of the steps taken to measure the data quality of SOFA data elements. Note that the left-upper block is result of the previous step, elaborated on in Figure 2.
Summary of data quality measurement protocol plan:
1. Extracted all SOFA-related data from the hospital information system database of ICU patients hospitalized in a specific period;
2. Transformed data;
3. Executed scripts to assess both the EHR data quality and data quality of the NICE submis-sion;
4. Identification of registration (healthcare provider) or aggregation-related (EHR) data quality issues.
Validation of automated data quality assessment
The automated data quality assessment tool was developed at a Dutch public hospital and was validated concurrently at this site. Validation of the tool was performed in order to rule on the correctness of extraction and transformation queries. Test cases were constructed for every SOFA data element and included a comparison between manually obtained data from the EHR and data that were automatically extracted with the tool. The results were also checked by application specialists. Queries were revised in case of noted discrepancies between the two results. In addition, performance related results like required runtime and memory load were reported.
The aforementioned validation process was repeated at a second test site because the re-searchers wanted to investigate whether the tool can be seamlessly implemented in any hospital using the same EHR.
3.3
Methods objective 3 – Process improvement
Phase 3
In this phase actual and target data quality values were analyzed. The researcher evaluated which data- and process-driven improvement activities should be adopted. Furthermore, the visualization aspect of the data assessment process was investigated. The concluding research activity was the evaluation of the proposed improvement activities which included the usability of the tool (used for automated data quality assessment). The following research questions were to be answered:
• How will assessment of data quality target values compare to actual data quality issues? • What actions should be initiated in order to reach expected or target data quality target
values?
Data quality requirement definition
The purpose of this step was to derive target data quality values based on actual data quality identified in the previous phase. Expected and actual data quality scores were required to elaborate on possible data quality issues.
Actual data quality values based on NICE related data in the EHR were derived for every combination of relevant SOFA element and data quality dimensions: DQij, where ‘i ’ was defined
as the SOFA element and ‘j ’ represented a data quality dimension.
Expected or target data quality levels were defined with the help of the NICE foundation. The researchers were provided with an overview of statistics including averages of data that was missing. Where possible, target data quality values for the data quality dimensions completeness and in some cases correctness were deduced from this set. Other target data quality values for the correctness, concordance, plausibility, and currency data quality dimensions were constructed by the researchers. The data quality target values were based on averaged actual data quality values and logical reasoning.
Data quality improvement activity selection
Significant deviations (> 10 %) between actual and expected data quality levels were further analyzed in this step. The aim was to define improvement activities to reach data quality target values. Activities related to data quality improvement were subdivided into either process-driven activities or data-driven activities.
According to the CDQ methodology, the third step includes implementing a reengineered pro-cess and improve it iteratively. Instead of implementing the reengineered propro-cess, suggestions and an advisory plan were being put forward.
Choice and evaluation of improvement processes
Improvement processes were constructed based on data quality improvement activities described in the previous step. The activities were linked with each other and transformed into possible improvement processes.
Intensivists and other persons involved in the NICE submissions were asked to give their view on the perceived added value and cognitive load of proposed improvement activities. Answers were retrieved as a result of qualitative research in which informal interview sessions were held.
4
Results
This chapter is subdivided into three sub-sections and includes results for each research objective stated in Figure 1.
4.1
Results objective 1
Reconstructing data flows and processes
Interviews
Results of the semi-structured interviews with intensivists from a public hospital (Hospital 1) and a data specialist of a Dutch university hospital (Hospital 2) are summarized according to the following themes: Involved ICU staff and their proceedings, data collection, transformation and submission processes, issues encountered and possible areas of improvement.
Involved ICU staff and their proceedings
Hospital 1: The ICU secretary registers admission-related information for a patient. The scoring period entails the period the patient is admitted to the ICU. Intensivists create admission status reports and fill out NICE submission related forms included within the hospital information system. Other ICU staff (including residents, senior house officers, and nurses) are also accountable for registering patient data in the EHR. These actors are not involved in further proceedings concerning NICE submissions.
Hospital 2: An information technology specialist is responsible for extracting raw patient data. A data manager then performs a set of data related activities and converts the anonymized patient data into a database file ready to be submitted to the NICE registry. Intensivists are sometimes involved in case the data manager is not totally certain on how particular data elements should be scored. Other ICU staff (including residents, senior house officers and nurses) are also account-able for registering patient data in the EHR. These actors are not involved in further proceedings concerning NICE submissions. However, these actors are not involved in further proceedings con-cerning NICE submissions.
Data collection, transformation, and submission processes
Hospital 1: Several times a week the intensivists look up the NICE submission related forms and update the data fields which are not yet filled out. In case of missing data items, the intensivists are often required to retrieve this information from different places in the EHR. No other ICU related staff is involved complementing the NICE submission related forms. Nor are actors other than intensivists accountable for submitting the datasets to the NICE registry.
Hospital 2: A clinical data specialist, or data manager, receives data relevant for the NICE submission from an information technology (IT) specialist. The IT specialist aggregates and ex-tracts data from the hospital information system database. Residents, SHOs, and nurses register patient data from a clinical perspective, not according to the scoring definitions constructed by the NICE registry. Therefore different kinds of data is extracted because a vast amount of NICE data elements have to be deducted from multiple data points. Subsequently, the data manager performs data cleaning activities in order to filter out potential data issues.
Issues encountered and possible areas of improvement
Hospital 1 & 2: Caveat is that prepopulated NICE elements data fields in the EHR –which are specifically designed forms for the NICE submission– are not being checked by the intensivists or the data manager. The idea that “values will be correctly collected and converted if they already have been registered by the system“ prevails among hospital staff members. This is especially the case for laboratory results and values that are too time-intensive to subject to additional manual quality control checks, i.e. mean arterial blood pressure which is continuously measured and validated.
Hospital 1 & 2: Data managers and application managers pointed out that they don’t have insight into all of the software specifications due to software restrictions or lack of authorization. This includes functions in the EHR for aggregating and calculating SOFA data elements.
Hospital 1 & 2: The focus lies merely on decreasing the administrative burden rather than performing extensive data quality analysis.
Hospital 2: At the time of writing the hospital only submits a modest set of NICE modules whilst the aim is to do additional submissions in the near future. The hospital is not submitting NICE modules that require a substantial amount of administrative burden (more than several hours per week). Having to consult free-text elements in order to fill out NICE submission forms increases the administrative burden.
Data - actor matrix
The results of the interviews were synthesized and used to construct the data-actor matrix included in table 2. Each cell of the data - actor matrix specifies whether an actor or a related system unit either creates or just uses or modify (update) a set of data.
Table 2: Data - actor matrix SOFA data elements
Data/
Actor Intensivists Intern/SHO/Registrar Nurse Secretary
Application manager
Laboratory source system/ technician PDMS Administrative data on hospitalization Create Update Create Update Physiological
values (blood sample)
Create Update Vitals Create Update Create Update Create Update Create Update Neurological (GCS) Create
Update Create Create Medication administration Create
Update
Create Update
Create
Update Update Event registration Create
Update Create Update Create Update Update Create Update Patient history/daily status Create
Update
Create Update
Create Update
In this data-actor matrix data elements are grouped based on similarities regarding the creation, collection, and use of this data. The data elements were grouped with the aim to enforce conciseness and coherency. The description of the grouped elements is shown in table 3.
As explained previously, the research scope is limited to assessing data quality for the SOFA module (Table 1). Therefore, table 3 includes grouped NICE data elements concerning the SOFA module. The grouped data elements subjects are also used in other SOFA modules like MDS and KIIC. Hence, this table can be reused when assessing the data quality of additional NICE modules.
Table 3: Grouped SOFA data elements
Data group Required for
Administrative
data Hospital number, IC-number, admission number, scoring date, scoring day, level of treatment Physiological
values (blood samples) PaO2, thrombocytes, bilirubin, creatinine Event
registration
FiO2, respiratory support, special respiratory support methods, artificial liver
support, dopamine, phosphodiesterase inhibitors, adrenalin, noradrenalin, cardiac assists device, intracranial pressure monitoring, renal replacement therapy
Measurement
registration Diuresis, FiO2
Vitals Mean arterial blood pressure
Neurological Eye score, movement score, verbal score (GCS)
Data - actor matrix
Several processes have been identified which are related to creating and updating data needed for the SOFA submission. Each cell indicates how an actor or related system unit is related to the specific process. Being an ‘owner’ implies being responsible or accountable and ‘participates’ specifies process involvement. Nine processes have been identified:
1. Creating data on ICU patient admission: Defining admission date, assigning an unique admission number and department;
2. Updating admission data regarding patient discharge: Defining definitive ICU patient dis-charge date;
4. Updating information on vitals. (Establishing a connection between real-time monitoring systems and HIS in order to store patient information);
5. Registering information on events related to patient care;
6. Updating information on start and end date and time of patient related events (e.g. respi-ratory support or infusion duration);
7. Updating information on daily status of the patient;
8. Updating and transforming data required for submission to the NICE registry. The process - actor matrix is shown in table 4.
Table 4: Process - actor matrix SOFA data elements
Process/Actor Intensivist
Intern/SHO/
Registrar Nurse Secretary
Application
manager Data manager
Not specifically related to ICU workforce Process 1 Owner Participates Participates Owner
Process 2 Owner Participates Participates Owner
Process 3 Owner
Process 4 Owner Participates Participates Participates Process 5 Owner Participates Participates
Process 6 Owner Participates Participates Participates Process 7 Owner Participates Participates
4.2
Results objective 2
Investigating data issues and data quality assessment measurements or techniques
Problem definition - Questionnaire results
The questionnaire, which aimed to identify problems in the process of registering patient-related data for the NICE registry, was filled out by two intensivists. Data quality issues were described in the context of grouped SOFA elements. The grouped SOFA elements were included in table 3. Physiological values, event registration, registration of measurement, and vitals were consis-tently aggregated data within the hospital information system. However, intensivists pointed out that they were not able to verify each and every data field due to the administrative burden, mean-ing that they are not entirely sure whether the data values contain quality issues. Administrative data and neurological data values were often not automatically aggregated by the EHR. A signif-icant amount of values must be registered manually because they were incomplete or incorrectly registered in the EHR. This is a result of a discrepancy between the clinical relevancy and the scor-ing definitions stated by the NICE registry. More concrete: Glasgow Coma Scale (GCS) scores, for example, must be assigned on the basis of specific (scoring definition) rules. In most cases, hospital staff assigns GCS scores from a clinical perspective instead of the scoring rules of the NICE registry.
Relevant data quality dimensions per SOFA data element
Now that we’ve gained insight in frequently occurring data quality issues, it’s essential to identify other potential data quality issues. These data quality issues are further concretized by means of data quality dimensions. The data quality dimensions defined by Weiskopf et al. (2013) are adopted [50]. Weiskopf et al. reviewed clinical literature discussing aspects and methods for assessing the quality of data of elements present in the EHR. However, not every data quality dimension is considered relevant for EHR-related data. The dimensions that were considered relevant were:
• Completeness: Is a truth about a patient represented in the EHR? • Correctness: Is an element that is present in the EHR true?
• Concordance: Is there agreement between data elements in EHR, or between other data sources? Do elements recording different information have values that make sense when considered together?
• Plausibility: Does the element in the EHR makes sense in the context of other knowledge about what that element is measuring?
• Currency: Is the element in the EHR a relevant representation of the patient state at a given point in time?
All the SOFA data elements were analyzed in light of each of the aforementioned data quality dimensions. The relevancy and necessity to measure the quality of data were considered for each SOFA data element and data quality dimension. The results of this analysis are included in table 5.
In some cases, performing data quality checks on completeness and correctness dimensions for particular SOFA data elements was irrelevant and illogical. For example, partial pressure of oxygen (PaO2) is not always determined.
Table 5: Relevant data quality dimension per SOFA data element
Completeness Correctness Concordance Plausibility Currency
SOFA_FiO2 SOFA_FiO2 SOFA_FiO2 SOFA_FiO2
Dobutamine Dobutamine Dobutamine
SOFA_Eye SOFA_Eye SOFA_Eye
SOFA_Motor SOFA_Motor SOFA_Motor
SOFA_Verbal SOFA_Verbal SOFA_Verbal
Renal replacement Therapy
Renal replacement Therapy Respiratory_ support Respiratory_
support Special ventilation Special ventilation
Cardiac assist device Cardiac
assist device
Phosphodiesterase inhibitors Phosphodiesterase inhibitors Phosphodiesterase inhibitors Mean_arterial_ blood pressure Mean_arterial_
blood pressure
Mean_arterial_ blood pressure
Mean_arterial_ blood pressure
Scoring day Scoring day
Scoring date Scoring
date
Artificial liver support Artificial liver support
Level of treatment Intracranial monitoring
Dopamine Dopamine
Adrenalin Adrenalin
Noradrenalin Noradrenalin
SOFA_PaO2 SOFA_PaO2 SOFA_PaO2
SOFA_ Thrombocytes SOFA_ Thrombocytes SOFA_ Thrombocytes
SOFA_Bilirubin SOFA_Bilirubin SOFA_Bilirubin
Creatinine Creatinine Creatinine
Diuresis Diuresis
Results literature review - Methods to quantify EHR data quality issues required for the NICE submission
Now that we have gained insight into the relevant dimensions that are deemed important for each data element, the subsequent step was to define methods and techniques that can be adopted to quantify these data quality dimensions.
The study of Weiskopf et al. (2013) [50] constructed a set of methods for each specific data quality dimension. According to the authors of the study, there is currently little consistency or potential generalizability in the methods used to assess EHR data quality. The stated methods may not always be applicable for different research questions. Therefore, a systematic literature review was conducted to identify generic methods and techniques to quantify data quality of data elements related to the NICE submission. The goal of this literature review was to complement the results of the study of Weiskopf et al. (2013) [50] in order to thrive towards a more widely applicable set of methods and techniques to assure and control the quality of EHR data that is being reused for research.
Figure 5 shows the results of the literature search. 878 records were identified by database searching and one record by means of assessing references of other relevant literature. After re-moving the duplicates, titles and abstracts of 587 articles were scanned. This resulted in 68 articles eligible for full-text assessment, see Appendix III. Finally, 15 articles were included.
Figure 5: PRISMA flow chart
Data extraction and synthesis
The data extraction table of the studies included can be found in appendix III. The methods to quantify data quality in the context of the data quality dimensions were identified for each study. A number of studies mentioned these dimensions explicitly while other studies described those dimensions more implicitly. In the latter case, the dimensions had to be deduced in order to create an unambiguous and consistent overview of data quality dimensions and methods. The study of Kahn et al. (2016) was used for this mapping process because they constructed a crosswalk table between data quality dimensions [21]. The results of the data extraction process are shown in table 6, in which the assessment methods per data quality dimension are described.
Completeness
Nine studies described methods or assessment techniques that fall under the completeness data quality dimension. The element presence assessment technique was mentioned most [12, 18, 32, 47]. Another interesting data quality assessment method included the use of data quality probes utilizing clinical knowledge [5]. These probes are presented as knowledge rules: When ‘A’ presen-t/absent then ‘B’ presenpresen-t/absent. For example, a patient record stating that the patient received invasive respiratory support should also include a non-null validated FiO2 value. If this is not the
Table 6: Assessment methods per data quality dimension. (*) included measures to assess
the quality of data of the NICE submission
Dimension Assessment
method Study
Completeness Data quality probes utilizing clinical knowledge * [5]
Element presence * [12, 18, 32, 47]
Element presence with reference to data values * [40]
Compare to alternative data sources [16]
Attribute domain constraints * [22]
Record strength score [28]
Correctness Attribute domain constraints * [8, 13, 22]
Data quality probes utilizing clinical
knowledge/ logical data rules * [5, 31]
Measuring central tendency and dispersion [8, 16]
Mathematical formulae for quality
metrics [8, 38]
Relational integrity [13]
Compare to alternative data sources [16]
Absolute threshold analysis [17]
Error of commission data check [18]
Frequencies check * [47]
Concordance Cross-checks or crosstabs * [31, 40, 47]
Integrity data rules * [22, 38]
Data quality probes utilizing clinical
knowledge * [5]
Check with external terminology standards [12]
Adopting machine learning / autoencoders
for outlier detection * [47]
Examining coding errors [18]
Plausibility Clinical, time-related or
person-specific validation rules * [16, 32]
Attribute dependency [13]
Change threshold analysis * [17]
Outlier detection - Machine learning / autoencoders and funnel plots [31]
Threshold analysis based on literature * [38, 47]
Internal measurement on the basis of local knowledge [40]
Currency Historical data rules * [13, 22]
State-dependent object rules [13]
Time stamps examination of registered
events * [16, 47]
Change threshold analysis * [17]
Out-of-range time values * [18]
Correctness
Methods to assess correctness were described by ten studies. Attribute domain constraints were described and assessed by three articles [8, 13, 22]. Data values were checked for different domain constraints, whether valid values were entered. Examples include matching predefined coding pat-terns (are float values stored as float instead of integers) and other anomalies in data values that don’t match the expectations. Using data quality probes and logical rules was mentioned by two studies [5, 31]. For example, the timestamps of events registered in the clinical information system may not exceed discharge time. Another interesting find was the use of frequencies check [47]. This assessment method could be used to determine whether the count of SOFA days is equal or less than the total amount of days the patient was admitted to the ICU.
Concordance
Concordance was concretized utilizing cross-checks or crosstabs [31, 40, 47]. Two or more val-ues/variables were compared to determine logical concerns. Examples include: performing a data quality check to determine whether special respiratory support data value registered in the NICE
submission questionnaire is in concordance with respiratory events included in the patient record. Data quality probes utilizing clinical knowledge [5] were adopted to determine the concordance of the medication administered and whether the patient was sedated.
Plausibility
Methods assessing plausibility included change threshold analysis and threshold analysis based on literature [17, 38, 47]. Change threshold analysis determines whether the change between the cur-rent and other related values are as expected, given a specific time range. For instance, bilirubin usually is a slowly changing value. When two data records of patients include bilirubin levels of 10 and 1000, within a time difference of less than 6 hours, then a data quality issue is identified. Threshold analysis and clinical, time-related or person-specific validation rules include biological range checks or standard deviation calculations that can be performed for almost every physiolog-ical or laboratory SOFA data element [16, 32].
Currency
The final data quality dimension assessed was currency. Six studies described methods and tech-niques to concretize this quality dimension. The studies of Dziadkowiec et al. (2016) and Kahn et al. (2012) elaborated on historical data rules that could be used to investigate whether laboratory values were determined in a timely manner [13, 22]. In case of values that are determined dispro-portionately late then the data element in the EHR is not a relevant representation of the patient state at a given point in time. The time stamps examination of registered events methodology could be used to evaluate the time between two validated data points, for example, FiO2or mean
arterial blood pressure values [16, 47].
Data quality assessment methods included in table 6 that are highlighted with an asterisk (*) were used to measure the quality of data of related SOFA data elements, described in the subse-quent section. A detailed scheme of which method is used for each SOFA data element can be found in appendix IV. Some methods were not adopted for data quality assessment. One example included a comparison to alternative data sources or gold standard. These methodologies could not be applied because an alternative data source does not currently exists, aside from the different systems integrated with the clinical information system itself. Another example is outlier detection by means of machine learning techniques and funnel plots. These state-of-the-art methods could be perfectly applied to a selection of SOFA data elements and have proven to be very valuable. Clinical text mining of patient records can be used to identify neurological data (SOFA Glasgow Coma Scale scores). However, training a machine learning model requires an abundance of research data, which is not yet accessible in this research project.
NICE data extraction and assessment tool – technical specifications
Based on the reconstructed data processes, problem definition, and the results of the literature review, we established a base for extracting and assessing data quality. A data extraction and assessment tool was used as a method to extract SOFA data from the EHR to perform data quality assessment. The tool was specifically developed for this purpose and included the assessment techniques identified in table 6.
The tool provides an user interface and several functionalities to extract, assess and export data from the hospital database. The user selects the NICE module of choice and the tool subsequently runs all the related SQL and SQLite scripts in a predefined order, generating a SQLite database as an output.
The extraction scripts were written in the SQL language because hospital data was stored on a MSSQL server. Extracted data from the hospital database were stored in a local serverless SQLite database. There was no intermediary server process because the processes accessing the database read and write directly from the database files on the disk [41]. This was deemed important be-cause patient-related data had to remain on the local hospital network. Transformation and data assessment scripts were written in the SQLite language. The tool is granted read-only database permissions.
Tool functionalities
The tool performs the following operations:
1. Data extraction
Data was retrieved for every patient admitted to the ICU at a specific point of time. Some challenges were identified due to minor deviations in how the hospital information systems were implemented. The vendor offers a default configuration and a package that enable hospitals to implement their solution. As a result, differences between hospitals might exists regarding how data is stored in the hospital database. A SQL script was constructed that extract and analyze every potential data value issues that could occur during extraction in order to take care of these challenges.
The extraction scripts were written in such a manner that they could be applied to different hospitals using the same EHR. The tool is able to read a file document in which hospital-specific column-IDs were included. These column-IDs are then automatically applied in the SQL script code. The file also offers the option to set the extraction period, causing the tool to extract relevant data only and therefore increasing the performance.
2. Data transformation
The second step of the process included the transformation of the extracted data. Three consec-utive tasks were performed in order to create the tables required for data assessment. The first task included the retrieval of data from records containing unstructured or semi-structured raw data. Data retrieved from the first task was joined with general SOFA data like SOFA days and dates because the SOFA module has to be submitted for every day the patient was admitted to the ICU. The third task included data operations that were needed in order to adhere to the scoring definitions stated by the NICE registry [30] (e.g. conversion of kPa to mmHg or deleting adrenalin values that were administered less than an hour on a specific SOFA day).
3. Data assessment
In this step data quality assessment methods defined in table 6 were executed for every SOFA element, see appendix IV.
Quantitative evaluation of data quality issues
Two types of results were reported based on the outputs of the data extraction and assessment tool: correctness of the NICE submission and the data quality of patient data in the EHR required for the NICE submission. These results are described per hospital site. The tool was tested at two different hospitals (Hospitals A and B).
First, the submission file created by the EHR was compared to a submission file generated from the tool. This submission file created by the EHR software is a database file that is normally submitted to the NICE registry. Automated data quality assessment was performed per quarter for the first three quarters of the year 2019. The extractions periods were: January-February-March, April-May-June, and July-August-September. The results are included in table 7. On average the percentage of erroneous data submitted to the NICE registry was 9.46%. The researchers found significant differences (> 50% incorrect submissions) in three reported SOFA elements for both hospitals, namely: PaO2, FiO2, and bilirubin. In addition, around 30% of urine output,
mean average blood pressure and respiratory support, were incorrectly submitted by hospital B. Based on the analysis the researchers identified several errors in the EHR. To a great extent, discrepancies in PaO2, FiO2, urine output, and bilirubin were attributed to faulty calculations
of values and misinterpretation of the NICE scoring definitions. Some other discrepancies in PaO2, FiO2, bilirubin, mean average blood pressure and respiratory support could be a result
of inconsistent or incomplete registering of patient data in the EHR. Neurological values weren’t thoroughly assessed because some sort of text mining/interpretation algorithm was needed to deduce the Glasgow Coma Scale scores, which isn’t yet implemented in the tool. A slight decrease in the percentage of incorrectly submitted data elements can be noticed. This might be a result of software fixes that already had been taken place. In the meantime, hospitals started correcting errors that were identified. This explains why the tool retrospectively detects these errors. Another remarkable finding is the difference between hospital A and B. Better initial registration of patient data has an effect on the number of errors found, which might explain these differences.
