Developing Quality Indicators
to Measure the Adoption of
the Collect Once, Use Many
Times Paradigm in Dutch
University Hospitals
Hans Joris Teunisse, BSc
June 2017, Amsterdam
Master Thesis
Medical Informatics
Developing Quality Indicators to
Measure the Adoption of the
Collect Once, Use Many Times
Paradigm in Dutch University
Hospitals
Student
Hans Joris Teunisse, BSc Student-ID: 10375880
E-mail: hjteunisse@gmail.com
Research Project Address Academic Medical Center
Department of Medical Informatics Meibergdreef 9
1105 AZ Amsterdam-Zuidoost Mentor
Erik Joukes, MSc
Department of Medical Informatics Academic Medical Center
Tutor
Prof. Dr. N.F. de Keizer
Department of Medical Informatics Academic Medical Center
Period
C
ONTENTS
Acknowledgments ... - 1 - Summary ... - 3 - Samenvatting ... - 5 - Chapter 1: Introduction ... - 7 - General Introduction ... - 7 - Background ... - 9 - Chapter 2: Methods ... - 11 - Literature ... - 13 - Expert Panel ... - 14 - Translation of Results ... - 14 -Rating the Indicators ... - 16 -
Discussing the Indicators ... - 17 -
Defining the Final Indicator Set ... - 17 -
Proof of Concept – Feasibility for Implementation ... - 17 -
Chapter 3: Results ... - 19 -
Literature ... - 19 -
Expert Panel ... - 23 -
Translation of Results ... - 23 -
Rating the Indicators ... - 23 -
Discussing the Indicators ... - 24 -
Defining the Final Indicator Set ... - 24 -
Proof of Concept - Feasibility for Implementation ... - 26 -
Chapter 4: Discussion ... - 31 -
Statement of principal findings ... - 31 -
Strengths and weaknesses of the study ... - 31 -
Strengths and weaknesses in relation to other studies ... - 32 -
Meaning of the study ... - 33 -
Unanswered questions and future research ... - 34 -
Conclusion ... - 35 -
References... - 36 -
Appendix A ... - 40 -
A
CKNOWLEDGMENTS
First of all, I would like to express my gratitude to my mentor Erik Joukes and tutor Nicolette de Keizer for all the effort they have put into my project. All your useful suggestions and feedback on my thesis, in preparation of meetings and presentations, and basically everything helped me in completing my thesis.
Furthermore, I would like to thank everyone who participated in the indicator development during the course of the project the final result together. With a special thanks to Lindsay Chang and Melchior Pot for their time throughout the project and sharing their knowledge on the ins and outs of the AMC.
And lastly, I want to thank all the other research interns, Macy, Liz, Debby and Femke in particular, who made the intern room a fun place to be.
S
UMMARY
Introduction: With the rapid increase of digitally available medical data, new secondary use possibilities arise. To enable reuse, data should be collected once, at the point of care, and in a structured and standardized way. This is referred to as the ‘Collect Once, Use Many Times’ (COUMT) paradigm. Several initiatives have been started by hospitals, the Netherlands Federation of University Medical Centers (NFU), the Dutch Association of Hospitals (NvZ), and the National Institute for ICT in Healthcare (Nictiz) to improve the uptake of the COUMT paradigm. Within this context, it is interesting to investigate to what extent Dutch university hospitals have adopted the COUMT paradigm. However, till now there is no way to measure this. The aim of this research was to develop quality indicators (QIs) to measure the adoption of the COUMT paradigm and to assess the feasibility of these QIs in practice.
Methods: A modified RAND method was used to develop the QIs. Literature was reviewed, and an expert panel of employees from the eight university hospitals in the Netherlands was consulted for possible QIs. Two reviewers independently combined and removed duplicate QIs. Hereafter, the reviewers discussed the possible QIs until consensus was reached on a draft set of QIs. Next, EHR implementation directors rated the draft set of QIs on a 9 point Likert scale on relevance, feasibility, and actionability. The QIs were discussed in an expert meeting until consensus was reached on the final indicator set. The final indicator set was defined with the AIRE instrument. As a proof-of-concept, we performed a feasibility study in one university hospital in the Netherlands to assess a selection of the QIs and find barriers in measuring these.
Results: Our literature study resulted in 62 potential QIs and experts provided 53 potential QIs. Combining, duplicate removal, and discussion by the two reviewers resulted in a draft set of 38 QIs. The expert panel of EHR implementation directors rated eleven of these QIs as relevant, feasible and actionable. The set included two structure indicators, seven process indicators, and two outcome indicators. The QIs cover communication to healthcare professionals, structured entry of allergies, medication, diagnoses, and procedures, and the use of routinely collected data for secondary purposes such as quality registrations and patient portals. These QIs were defined with the AIRE instrument. Seven QIs were investigated in a university hospital on their actual feasibility to be calculated and use in developing improvement strategies. The three main barriers we found during the feasibility study were: 1) the data needed to calculate the indicator is hard to extract from the EHR, 2) some QIs are not defined with enough detail, 3) some QIs are less actionable with their current definition.
Conclusion: In cooperation with the NFU and all university hospitals in the Netherlands, we developed eleven QIs which are relevant to assess the degree of COUMT adoption in a hospital. However, our feasibility pilot study shows that some indicators have to be further reviewed and formalized before hospitals can use them to benchmark themselves against their peers and to actually improve on the COUMT paradigm.
S
AMENVATTING
Introductie: In de afgelopen jaren is de hoeveelheid digitaal beschikbare medische data snel gestegen. Deze data kan worden hergebruikt voor meerdere doeleinden. Hiervoor moet deze data wel gestructureerd en gestandaardiseerd worden vastgelegd tijdens het zorgproces. Deze manier van werken wordt ook wel het ‘eenmalig registreren voor meervoudig gebruik’ (COUMT) paradigma genoemd. Ziekenhuizen, de Nederlandse Federatie van Universitair Medische Centra (NFU), de Nederlandse Vereniging van ziekenhuizen (NvZ), en het Nationaal Instituut voor ICT in de Zorg (NICTIZ) zijn verschillende projecten gestart waarmee de adoptie van dit paradigma verbeterd moet worden. Momenteel is niet bekend in hoeverre de adoptie van COUMT in de verschillende ziekenhuizen verloopt. Hiervoor zijn nog geen meetmethoden ontwikkeld. Het doel van dit onderzoek is het ontwikkelen van kwaliteitsindicatoren waarmee gemeten kan worden in hoeverre ziekenhuizen het COUMT paradigma hebben geadopteerd en het beoordelen van de haalbaarheid om deze indicatoren te meten in de praktijk.
Methode: Met behulp van een aangepaste RAND methode hebben wij kwaliteitsindicatoren ontwikkeld. Met behulp van een literatuurstudie en experts uit de universitair medische centra (UMCs) in Nederland is een eerste groep mogelijke indicatoren opgesteld. Deze indicatoren zijn vervolgens door twee reviewers gecombineerd en ontdaan van duplicaten om tot een voorlopig set te komen. Deze set is vervolgens beoordeeld door EPD directeuren op relevantie, haalbaarheid en actiegerichtheid met een 9 punts Likert schaal. De indicatorenset is in een expertbijeenkomst bediscussieerd totdat er consensus was over een definitieve set van indicatoren. De indicatoren zijn vervolgens beschreven met het AIRE instrument. Als laatste hebben we, als proof-of-concept, een aantal indicatoren op haalbaarheid getest in een universitair ziekenhuis om mogelijke barrières in het meten van de indicatoren te vinden.
Resultaten: Uit de literatuur werden 62 indicatoren geselecteerd en de experts formuleerden in totaal 53 indicatoren. Combineren, dedupliceren en discussie door de twee reviewers leidde tot een voorlopige set van 38 indicatoren. De EPD directeuren beoordeelde vervolgens elf indicatoren als relevant, haalbaar en actiegericht na beoordeling en discussie. In deze set zitten indicatoren over communicatie met zorgprofessionals, gestructureerde invoer van allergieën, medicatie, diagnoses, en verrichtingen data, en het hergebruik van data uit het zorgproces voor kwaliteitsregistraties en patiëntenportalen. Na het uitwerken van de indicatoren met behulp van het AIRE instrument zijn zeven indicatoren getest in het haalbaarheidsonderzoek. De drie grootste barrières waren: 1) Data voor de indicator kan moeilijk uit het EPD te extraheren zijn, 2) definities zijn niet voldoende gedetailleerd beschreven en 3) sommige indicatoren zijn beperkt actiegericht in de huidige vorm.
Conclusie: Door samen te werken met de NFU en alle UMCs in Nederland is een set van elf kwaliteitsindicatoren ontwikkeld die relevant is om de mate van adoptie van COUMT in een ziekenhuis te meten. Ons haalbaarheidsonderzoek laat wel zien dat sommige indicatoren verder moeten worden getest en geformaliseerd voordat ziekenhuizen ze kunnen gebruiken om te benchmarken en om de adoptie van het COUMT paradigma te verbeteren.
C
HAPTER
1:
I
NTRODUCTION
G
ENERALI
NTRODUCTIONThe rise of electronic health records (EHRs) in medical centers has brought a vast amount of digital medical data [1–3]. The primary use of these data is to support direct patient care. However, digital medical data also createsopportunities to reuse them for other purposes than patient care. Medical data can be secondarily used for research, quality management, public health, clinical decision support systems (CDSS) and several other purposes [2,4–6]. In order to realize these benefits, data should be collected once at the point of care in a structured, i.e. use of structured entry forms, and standardized way, i.e. code data by an (inter)national terminology system [7]. This is also referred to as the ‘Collect Once, Use Many Times’ (COUMT) paradigm [8]. Adopting the COUMT paradigm can lead to a reduction of total administrative effort for the organization and preferably for healthcare providers, increase the correctness and completeness of medical data and also enable reuse of the data [9–11].
Even though the COUMT paradigmis desirable, there are still barriers which prevent the paradigm from being adopted [12]. Some physicians still collect medical data in free-text, which has several downsides. Use of specialty based jargon can lead to misinterpretation of information [13] and unstructured in combination with structured data might lead to inconsistencies between those two sources [13,14]. Even when data is entered in a structured and standardized way, there are still other barriers to theCOUMT paradigm. There is a proliferation of information and terminology standards and hospitals use EHRs developed by different providers [15]. Development of EHRs without a priority on interoperability has created a situation where sharing data between different care providers within and between hospitals, as well as between hospitals and for example quality registries, costs more effort than necessary [15]. The lack of structured and standardized data recording can lead to a higher risk of medical errors, higher costs of integration of systems, and frustration and capacity loss of care providers who re-enter data multiple times [12].
In recent years, several initiatives have been started by different organizations such as the Netherlands Federation of University Medical Centers (NFU), the Dutch Association of Hospitals (NvZ), and the National Institute for ICT in Healthcare (Nictiz) to improve the uptake of the COUMT paradigm [16]. Terminological systems, information models, and message standards such as SNOMED CT, ICD, CCR/CCD, and HL7v3 can be utilized to collect and exchange data in such a way that others can reuse this data [17]. Moreover, the European Union in cooperation with the European Patients Smart Open Services (epSOS) project developed a recommended standard data set, the epSOS Patient Summary, which offers a starting point for hospitals in the standardization of medical datasets [18]. The epSOS Patient Summary [19] contains the minimum set of information needed to support healthcare coordination and the continuity of care [18]. Based on the epSOS Patient Summary, the Clinical Documentation at the Point of Care program (in Dutch: ‘Registratie aan de Bron’) in the Netherlands developed the Basic Data Set for Care. The Clinical Documentation at the Point of Care program has been set up by the NFU to improve documentation and reuse of patient information, for example by developing national information standards for healthcare data such as Health and Care Information Models [5]. Besides developing information standards, there have also been some initiatives to change the mindset and behavior of physicians to collect medical information in a structured way. For example by providing educational programs on the subject [20].
After starting several of these initiatives, the NFU now wants to investigate to what extent Dutch university hospitals have adopted COUMT. However, till now there is no way to measure this. The aim of this research is to develop quality indicators (QIs) to measure to what extent hospitals have adopted the COUMT paradigm. Furthermore, we want to test the feasibility of measuring these QIs in a university hospital. By developing QIs to measure the adoption of COUMT, hospitals will be able to benchmark their performance against their peers and possibly improve on areas they are still lagging behind. These aims resulted in the following research questions:
Main question:
How can we measure the extent to which COUMT is adopted in Dutch university hospitals and what is the feasibility of performing this measurement in practice?
Subquestions
1. Which QIs can be used to measure the adoption of COUMT?
2. How feasible is the calculation of those QIs in practice, measured as a proof of concept in one of the university hospitals?
This introduction ends with some background information on recent developments on the COUMT paradigm in the Netherlands, as well as background information on the definition
B
ACKGROUND1.2.1 Clinical Documentation at the Point of Care
In the Netherlands, a national program was started with the purpose to increase awareness for the COUMT paradigm. The ‘Clinical Documentation at the Point of Care’ program of the NFU supported by Nictiz started in 2014. The vision of the program is to ensure that medical data is collected unambiguous and only once, in a way it can be exchanged and used for multiple purposes. By developing products such as the Basic Data Set for Care (in Dutch: Basisgegevensset Zorg), Health and Care Information Models (HCIMs, in Dutch ZIBs=ZorgInformatie Bouwstenen), a national diagnosis thesaurus and a national medical procedures thesaurus, Clinical Documentation at the Point of Care is trying to facilitate the adoption of the COUMT paradigm [5]. These products are developed to ensure that the healthcare sector uses the same data standards. This will ease the exchange of data between different institutions which potentially can improve the quality of care.
Besides the development of products for standardization of IT infrastructure, Clinical Documentation at the Point of Care is also providing education for physicians. Healthcare professionals are responsible for collecting medical data. For this reason, it is important that they are aware that the data they collect might be reused for secondary purposes. Therefore, they need to have the attitude to collect data once in a standardized and structured way to support reuse of data for multiple purposes. By providing E-learning courses, games and other forms of education on the COUMT paradigm, Clinical Documentation at the Point of Care is trying to educate healthcare professionals, information managers and healthcare related IT professionals [20].
The Clinical Documentation at the Point of Care program has set several goals they want the university hospitals to achieve before the year 2020 [21]:
1. 80% of healthcare data of the Basic Data Set for Care is recorded in a structured and unambiguous way by healthcare providers and patients.
2. All patients have access to their healthcare data, and a significant portion of patients uses this information, controls their information and adds new healthcare data. 3. 80% of referrals between healthcare providers within the university hospitals and to
other healthcare providers is facilitated by reusing routinely collected healthcare data.
4. Provision of data needed for quality control to important organizations will be done by reusing at least 80% of the routinely collected data routinely collected.
5. Over 50% of healthcare data relevant for patient-related research is available from reusing routinely collected healthcare data.
1.2.2 Quality Indicators
Quality indicators are defined as ‘quantitative measures that can be used to monitor and evaluate the quality of important governance, management, clinical, and support functions that affect patient outcomes’ [22]. QIs offer a quantitative basis to document or increase the quality of e.g. the care process or, in our study, the process to implement COUMT. QIs are often used to benchmark organizations, i.e. to compare the performance of an organization against its peers. According to Donabedian, QIs can be categorized into three categories: structure, process, and outcome indicators [23]. These three types of QIs are connected in a one-way direction from structure to process to outcome indicators (Figure 1).
Figure 1. Relation structure, process and outcome indicators
Structure indicators include attributes of the setting in which the QIs are measured. This consists of materials (facilities, equipment, and finance), human resources (qualifications and number of personnel) and organizational structure (staff training and guidelines). For the COUMT paradigm, this can be rephrased as: ‘Are the materials, human resources and a
facilitating organizational structure available in the hospital to collect data once and use it for multiple purposes?’. Process indicators include what is actually performed in the field.
‘Are physicians collecting data once and in an unambiguous way?’. And lastly the outcome indicators, where the actual effect of the efforts on the subject is measured. ‘Is medical data
collected in the hospital reused for multiple purposes?’.
QIs should be defined with high precision in order to prevent misinterpretation of the QIs leading to incomparable measures which cannot be compared between different institutions. An instrument to help with defining precise QIs is the Appraisal of Indicators through Research and Evaluation (AIRE) Instrument [24] has been developed by the Social Medicine Department of the Academic Medical Center. This tool consists of twenty statements which relate to the quality of the QIs. These statements are divided into four categories: 1) purpose, relevance, and organizational relevance, 2) involvement of stakeholders, 3) empirical evidence and 4) further formulation and substantiation of the QIs. The instrument can be used as a checklist of criteria that a good QI should fulfill.
C
HAPTER
2:
M
ETHODS
This chapter describes the development of a quality indicator set to measure the adoption of the COUMT paradigm in hospitals. For the development, we used a modified RAND method [25]. The original RAND/UCLA Appropriateness method (RAND method) was developed in the 1980s by the RAND Corporation and the University of California Los Angeles [26]. The method has originally been applied tocreate measures which assess the quality of medical or surgical procedures. RAND combines scientific evidence and expert opinions to develop QIs. According to this method, first, literature needs to be reviewed, and experts need to be asked to provide input on possible QIs for the subject. The results are combined into a draft set of QIs. Next, an expert panel is asked to rate the potential QIs on their appropriateness. Then, the experts meet in a discussion meeting to discuss the QIs. Last, the QIs are rated again to make a final selection. The QIs on which the expert panel agrees that it is appropriate to assess the subject are selected and are bundled in a final set of QIs. In our study, we used a modified version of the RAND method. The RAND method has been modified by van Engen-Verheul et al. [25]. The rating and discussion elements have been modified to increase the reliability of the rating and discussion process by defining appropriateness. Van Engen-Verheul et al [25] specified the appropriateness by defining it based on judgment criteria from the Organization for Economic Co-operation and Development [27]. The QIs in the modified RAND method are rated on 1) relevance, 2) actionability and 3) feasibility. Figure 2 gives an overview of the modified RAND method and the corresponding sections in this thesis.
Figure 2. modified RAND Method. The numbers refer to the corresponding methods section in which the phase is discussed.
L
ITERATUREFor our literature study, we defined key terms and MeSH terms to search literature in MEDLINE. These terms were defined in three categories: data collection, secondary use, and indicators. Synonyms were combined with the OR operator, and the categories were combined with the AND operator to find articles that describe possible QIs of the adoption of COUMT. See Table 1 for the complete search query. The search in MEDLINE has been performed on 14-12-2016. All articles without abstract and full text available in English were excluded. The included articles were screened on title and abstract. Articles on healthcare sectors other than hospitals and articles not investigating documentation or reuse of healthcare data were excluded. Selected full-text articles deemed relevant were read to extract possible QIs to measure the adoption of COUMT. The QIs were categorized conform the theoretical framework developed by Joukes et al. for COUMT adoption (Figure 3) [8].
Table 1. Search Query in MEDLINE.
Category Terms
Data collection
("Data Collection"[Mesh]
OR "Documentation"[All Fields] OR "Documentation"[Mesh] OR "Quality Assurance, Health Care"[Mesh]
OR "Electronic Health Records"[All Fields] OR "Electronic Medical Records"[All Fields] OR "Electronic Health Record"[All Fields] OR "Electronic Medical Record"[All Fields])
AND
("reuse"[All Fields] OR "reusing"[All Fields] OR "re-use"[All Fields] OR "re-using"[All Fields] OR "Secondary Use"[All Fields] OR "Secondary Uses"[All Fields] OR ("meaningful use"[MeSH Terms] OR "meaningful use"[All Fields])
OR ("collect"[All Fields] AND "once"[All Fields]))
AND
("Feasibility Studies"[Mesh]
OR "success"[All Fields] OR "succes"[All Fields] OR "successful"[All Fields] OR "indicators"[All Fields] OR "requirements"[All Fields]
OR "Quality Improvement"[Mesh] OR "barrier"[All Fields] OR "barriers"[All Fields] OR "Quality assessment"[All Fields] OR "Measurements" OR "Measures") Secondary Use
E
XPERTP
ANELIn parallel with the extraction of QIs on the adoption of COUMT from the literature, members of an expert panel were asked to provide possible QIs on the subject. We invited employees from the eight university hospitals in the Netherlands involved in the management and support of the EHR to participate in the expert panel. This included directors of EHR implementation programs and managers of services to maintain and further develop the EHR as well as its use in their organization. We also invited four physicians involved in a COUMT-related working group in one of the university hospitals, who had practical experience with record keeping and could provide insights into the practical aspects of COUMT adoption, to participate.
All potential panelists were contacted by email in December 2016 with the request to answer the question: “What would you like to know from your own or another hospital to determine whether medical data is registered once and unambiguous, in such a way that the data can be reused for other purposes?”. The panelists were asked to forward the email to others within their hospital who might be interested in participating in the process of developing QIs to measure the adoption of COUMT. The members of the expert panel were given four weeks to reply. A reminder was sent after two weeks.
The elicitation of the expert panel resulted in a second set of possible QIs for the adoption of COUMT. Similarly to the QIs from the literature, the QIs provided by the expert panel were categorized conform the theoretical framework developed by Joukes et al. [8] for COUMT adoption (Figure 3).
T
RANSLATION OFR
ESULTSThe QIs found in the literature and suggested by the expert panel were reviewed by two reviewers (HT, EJ). Both reviewers independently joined the QIs from both sources and removed duplicates if necessary. Next, the two reviewers discussed and rephrased the QIs until consensus was reached. Furthermore, each indicator was classified as structure, process or outcome indicator. This resulted in a first draft set of QIs to measure the adoption of COUMT.
R
ATING THEI
NDICATORS 2.4.1 QuestionnaireIn the next phase of the modified RAND method, we invited the seven directors of the EHR implementation programs from the eight university hospitals in the Netherlands to rate the draft set of potential QIs. One director was responsible for a joint EHR implementation program of two university hospitals. Hence all Dutch university centers were represented. The expert panel individually rated the draft set of all potential QIs via an online questionnaire sent in February 2017. Each participant was asked to rate each indicator on a 9 point Likert scale (1 = strongly disagree, 5 = neutral, 9 = strongly agree) on three criteria: 1) relevance (Is the indicator relevant for the COUMT paradigm?), 2) feasibility (Is it possible to calculate the indicator without much effort?), and 3) actionability (Are there actions possible which will lead to improvement on the indicator?). Participants were given four weeks to rate the draft set of potential QIs. A reminder was sent after two weeks. To further increase the response rate we presented the list of potential QIs in one of the monthly meetings of the EHR implementation directors and offered the possibility to fill in the questionnaire on a paper-based form.
2.4.2 Analysis
To determine the most relevant, feasible and actionable QIs according to the participants, we calculated the median of the answers and the agreement between participants. According to the RAND method, an indicator is accepted when the median of the answers is a minimum of seven and 80% of the participants answered within the 7-9 score range on all criteria. After analysis, the QIs were categorized into three groups: 1) QIs which had a median of seven on all criteria and agreement among the member of the expert panel, 2) QIs with disagreement between the panel on at least one of the criteria, and 3) QIs with a low rating and agreement among panel members on the low score.
D
ISCUSSING THEI
NDICATORSIn the discussion phase of the RAND method, the results of the individual rating process were presented in a face-to-face discussion meeting of the expert panel who rated the QIs. The QIs were presented in the three groups as described in section 2.4.2. The expert panel was given the opportunity to comment on QIs. In this discussion meeting, mainly QIs from the second group were discussed since opinions differed most on these QIs. Based on the discussion, QIs on which the expert panel unanimously agreed after discussion were added to the group of selected QIs. All other QIs were rejected. At the end of the discussion, based on consensus, the final set of QIs to measure the adoption of COUMT was determined.
D
EFINING THEF
INALI
NDICATORS
ETThe final set of QIs was defined according to the AIRE instrument [24]. For each indicator we described the relation to quality, the definition of the indicator with numerator and denominator, inclusion and exclusion criteria, type of indicator, the quality domain, background information, possible actions to improve on the indicator, limits in interpretation, possible bias, the data source, measuring frequency, and measuring period. The AIRE instrument also states that the validity, precision and discriminatory nature of the indicator should be described. To describe these categories, we would need to pilot the indicator set to see whether the measures show what it was intended to do, whether the indicator is consistent between measurements and whether it is showing differences between hospitals.
P
ROOF OFC
ONCEPT–
F
EASIBILITY FORI
MPLEMENTATIONAfter developing the QIs, as a proof-of-concept, we validated the indicator set on their feasibility to be measured in practice and use in developing improvement strategies. In cooperation with two employees from the EHR implementation program employees from one of the participating university hospitals, we chose several QIs from the final set to be calculated for the university hospital. The selected QIs were most likely to be available within reports generated by the hospital and were also deemed as most interesting to know for the hospital. By calculating these QIs, we wanted to reveal possible barriers in calculating these QIs. To reveal these barriers, we interviewed employees who were most knowledgeable on the data needed for each indicators.
C
HAPTER
3:
R
ESULTS
L
ITERATUREThe search query resulted in 1128 articles. We excluded 339 (30.1%) articles that had neither abstract nor full text available in English. This resulted in 789 (69.9%) unique articles for further screening. Articles were screened on title and abstract which led to 111 (9.8%) articles included for full-text screening. After screening full texts, a total of 32 (2.8%) articles were deemed relevant to extract QIs for measuring the adoption of the COUMT paradigm in hospitals. These articles were further analyzed to extract QIs on the adoption of COUMT. See Figure 4 for an overview of the included and excluded articles in each phase.
The 32 analyzed articles described 62 QIs on the adoption of the COUMT paradigm. We clustered the QIs based on the framework by Joukes et al. [8] describing concepts of the adoption of the COUMT paradigm. The results are shown in Table 2. This table also shows the results of the expert which will be further discussed in section 3.2. QIs found were categorized on the concepts System Quality (n = 14), Information Quality (n = 19), Awareness (n = 2), Facilitating Conditions (n = 2), Institutional Trust (n = 2), and Behavior (n = 8). We did not identify any indicators in this literature search for the main concepts: Subjective Norm, Self-efficacy, Perceived behavior control, Perceived Risk, Information Satisfaction, System Satisfaction, Perceived Usefulness and Compatibility.
Table 2. Concepts on the adoption of the 'Collect Once, Use Many Times' paradigm, found in the literature and suggested by experts, mapped to the framework of Joukes et al. [8].
Concepts # of references [ref] # of experts System Quality Integration 8 [28–35] 3 Accessibility 8 [28,29,32,35–39] 1 Information Quality Completeness 9 [28,32,38–45] 5 Format 5 [28,32,33,38,46] 0 Accuracy 5 [28,30,40,42] 0 Currency 6 [32,36,38,40,46,47] 0
Perceived Ease of Use 1 [48] 4
Awareness 1 [49] 0
Institutional Trust 2 [32,49] 1
Facilitating Conditions 2 [49,50] 0
Below, we will shortly address the indicators mentioned in articles found in our literature study organized according to the main concepts of the model by Joukes et al. [8].
System Quality
We found fourteen [28–39] articles which studied an indicator related to System Quality. The QIs could be categorized to Integration (n = 8) and Accessibility (n = 8). The implementation of data definitions, terminologies and message standards in EHRs are stated as core QIs which show the integration of multiple sources of data, in such a way it can be used for multiple purposes [28–35]. Other studies suggested the ease of accessing and extracting medical information from the system as QIs, for example by comparing data elements definitions of EHRs and quality registries [28,29,32,35–39]. We did not find any articles studying the reliability, flexibility, and timeliness of a system in our literature search. Information Quality
We found nineteen articles [28,32,38–45] that studied one or more sub-concepts of Information Quality. There is no full consensus on what dimensions information quality encompass. Different frameworks have been suggested with the different naming and definitions of the dimensions [28,38,42]. All versions of the dimensions of information quality were mapped to the information quality concept and sub-concepts of the framework. Suggested QIs are for example the consistency of data in the EHR with codes in terminologies [30] and the completeness of medical data such as blood pressure and laboratory test results [44]. We did not find any articles on Information Reliability.
Perceived Ease of Use
We found one article [48] on the perceived ease of use of the system used to store data. The article suggested the efficiency of physicians entering medical data as an indicator for the Ease of Use concept within the COUMT paradigm [48].
Awareness
We found one article [49] discussing an indicator on the awareness concept. Employees should be provided with information on how and why they should collect patient data only once and unambiguous. This could be done by providing educational programs to the subject, one-on-one training, and feedback on performance. Information could also be communicated by newsletters, e-mail and other forms of information provision to all medical and nursing staff. An indicator such as the number of physicians educated on COUMT can show how aware physicians are about why they are collecting their data in a standardized and structured way.
Institutional Trust
We found two articles [32,49] including an indicator pointing out the need for persons who are responsible for data collection within hospitals and who can make high-level decisions to make sure data is collected once and unambiguously. Having someone who is responsible for the COUMT paradigm in a hospital shows the involvement of the hospital with the paradigm and will provide physicians with trust in their institution that the taken actions are necessary. This indicator could also be categorized as structural assurance since there will be someone who assures that safety-nets are in place. We did not find any articles on the concept of Situational Normality.
Facilitating conditions
We found two articles [49,50] which mention actions that can be done to improve on the facilitating conditions within a hospital. An example of an indicator reflecting those actions is the availability of a help-desk where physicians can ask questions on how and where to collect data once and unambiguous.
Behavior
We found eight articles investigating the COUMT paradigm by studying the possibility to reuse routinely collected data. Studies investigated the possibility to automatically produce referral letters from routinely collected data [39,44,45,51,54], the application of clinical decision support systems which use standardized health data [39,45,54] and extraction of medical data for quality registries [47,52,53] as secondary use purposes. Another opportunity for secondary use of medical data is the provision of patient data to patients themselves [39,45,54]. This should make it easier for patients to be informed on their health status. The Meaningful Use program in the United States, as well as the Clinical Documentation at the Point of Care program in the Netherlands, have set patient access to EHR data as a goal to be completed by the hospitals in their country [5,39]. Patients accessing their data within 48 hours, increasing the number of patients using this information, providing more patient-centered information through a patient portal, and giving the patient the possibility to enter his self-generated data into the patient portal are mentioned as QIs which show the use of patient portals.
E
XPERTP
ANELEight employees from four of the eight invited university hospitals responded to the elicitation of possible QIs on the adoption of the COUMT paradigm. Two directors of an EHR implementation program, two information architects, and four physicians provided a total of 53 possible QIs. The two QIs mentioned most often were the completeness of data (n = 5) and the possibility to extract data from the EHR for important quality registries (n = 5). The other suggested QIs concerned the ease of use of the EHR software (n = 4), the implementation of national terminology and message standards in the EHR (n = 3), the availability of clinical decision support that uses routinely collected data (n = 1), referral letters automatically generated from routinely collected data (n = 1), use of routinely collected data collected for scientific research (n = 1) and the number of financial encoders available (n = 1). Table 2 shows on which categories of the framework by Joukes et al. [8] the experts provided possible QIs.
T
RANSLATION OFR
ESULTSTwo reviewers (HJT, EJ) independently reviewed the 62 QIs found in the literature and 53 possible QIs provided by the expert panel. After discussion, the reviewers reached consensus on a total of 38 unique QIs for the draft set. We categorized the draft set of QIs into nine structure indicators, fifteen process indicators and fourteen outcome indicators. Appendix A provides an overview of the draft set of QIs and how they fit into the framework of Joukes et al. [8].
R
ATING THEI
NDICATORSFive directors of an EHR implementation programs responded to the online questionnaire to rate the indicators on relevance, feasibility, and actionability. The 38 QIs of the draft set were divided into three groups based on the rating on those three criteria. The first group consisting of two structure indicators, six process indicators and two outcome indicators had a median rating of seven or higher on relevance, feasibility, and actionability with an agreement between the expert panel members. The second group consisted of eight QIs on which, despite high scores on relevance, feasibility, and actionability, substantial disagreement existed among the expert panel members. The remaining twenty QIs were categorized in the group where the panel unanimously rated QIs with low scores on at least one of the three criteria.
The QIs with the highest median rating (eight on all three criteria) were three QIs on the structured entry of medication, medical procedures, and diagnosis. Appendix A provides a full overview of the ratings on relevance, feasibility, and actionability for each indicator.
D
ISCUSSING THEI
NDICATORSThe rated indicator set was discussed with four directors of EHR implementation programs of the eight university hospitals and the project manager of Clinical Documentation at the Point of Care program. We discussed the QIs according to the order of the three rating categories described in section 3.4.
No additional comments were given on the category of QIs with a median higher than seven. These were all accepted for the final set of QIs of the adoption of COUMT.
Each indicator in the second category of QIs was discussed separately. Of this group of QIs, only the indicator ‘The average percentage of items of the Basic Dataset for Care which has a value (per patient)’ was accepted for the final set of QIs after discussion by the experts. The Basic Data Set for Care is a product of the Clinical Documentation at the Point of Care program. The completeness of this product was seen as one of main the goals of the program. Therefore, the expert panel decided it should be accepted as a quality indicator for COUMT.
The expert panel commented that the QIs on the availability of message standards within the EHR and the possibility of generating discharge letters with the correct information within the EHR were phrased in such a way that the expert panel did not see any possible actions to improve on these two QIs. For this reason, the actionability was scored low and the QIs were rejected. For the QIs on automated extraction of the quality of care indicators, the automatic extraction of the LBZ and HSMR and the indicator on the possibility for patients to access the patient portal, the main reason for disagreement was the ambiguous formulation of the indicator. The expert panel decided that the QIs on the extraction of the LBZ and HSMR were similar to the QIs on the structured entry of the LBZ and HSMR and that they should be rejected. The indicator on the possibility to access the patient portal was deemed too vague and was also rejected.
The QIs in the third category, i.e. those with low scores, were all rejected after the discussion meeting. There was some discussion on the QIs concerning the completeness of the Basic Dataset for Care, but the expert panel concluded that other accepted QIs already covered this subject.
The discussion meeting resulted in eleven QIs in the final set with consensus among the expert panel members. This set is shown in Table 3.
Table 3. The final quality indicator set with numerator and denominator.
Definition Indicator Numerator and Denominator
Structure Indicators
There is a central point of contact within the hospital where physicians and nurses can go to when they have questions about collecting data once and unambiguous.
Yes / No
The hospital provides patients with the possibility to enter their own information in the patient portal
Yes / No
Process Indicators
Percentage of medication entries which have been entered in a structured way, based on a predefined value-set
𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑚𝑒𝑑𝑖𝑐𝑎𝑡𝑖𝑜𝑛 𝑒𝑛𝑡𝑟𝑖𝑒𝑠 𝑤ℎ𝑖𝑐ℎ ℎ𝑎𝑣𝑒 𝑏𝑒𝑒𝑛 𝑐𝑜𝑙𝑙𝑒𝑐𝑡𝑒𝑑 𝑖𝑛 𝑎 𝑠𝑡𝑟𝑢𝑐𝑡𝑢𝑟𝑒𝑑 𝑤𝑎𝑦, 𝑏𝑎𝑠𝑒𝑑 𝑜𝑛 𝑎 𝑝𝑟𝑒𝑑𝑒𝑓𝑖𝑛𝑒𝑑 𝑣𝑎𝑙𝑢𝑒𝑠𝑒𝑡
𝐴𝑙𝑙 𝑐𝑜𝑙𝑙𝑒𝑐𝑡𝑒𝑑 𝑚𝑒𝑑𝑖𝑐𝑎𝑡𝑖𝑜𝑛 𝑒𝑛𝑡𝑟𝑖𝑒𝑠 𝑤𝑖𝑡ℎ𝑖𝑛 𝑡ℎ𝑒 𝐸𝐻𝑅
Percentage of medical procedures which have been entered in a structured way, based on a predefined value-set
𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑚𝑒𝑑𝑖𝑐𝑎𝑙 𝑝𝑟𝑜𝑐𝑒𝑑𝑢𝑟𝑒 𝑒𝑛𝑡𝑟𝑖𝑒𝑠 𝑤ℎ𝑖𝑐ℎ ℎ𝑎𝑣𝑒 𝑏𝑒𝑒𝑛 𝑐𝑜𝑙𝑙𝑒𝑐𝑡𝑒𝑑 𝑖𝑛 𝑎 𝑠𝑡𝑟𝑢𝑐𝑡𝑢𝑟𝑒𝑑 𝑤𝑎𝑦,
𝑏𝑎𝑠𝑒𝑑 𝑜𝑛 𝑎 𝑝𝑟𝑒𝑑𝑒𝑓𝑖𝑛𝑒𝑑 𝑣𝑎𝑙𝑢𝑒𝑠𝑒𝑡
𝐴𝑙𝑙 𝑐𝑜𝑙𝑙𝑒𝑐𝑡𝑒𝑑 𝑚𝑒𝑑𝑖𝑐𝑎𝑙 𝑝𝑟𝑜𝑐𝑒𝑑𝑢𝑟𝑒 𝑒𝑛𝑡𝑟𝑖𝑒𝑠 𝑤𝑖𝑡ℎ𝑖𝑛 𝑡ℎ𝑒 𝐸𝐻𝑅
Percentage of diagnosis which have been entered in a structured way, based on a predefined value-set
𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑑𝑖𝑎𝑔𝑛𝑜𝑠𝑖𝑠 𝑒𝑛𝑡𝑟𝑖𝑒𝑠 𝑤ℎ𝑖𝑐ℎ ℎ𝑎𝑣𝑒 𝑏𝑒𝑒𝑛 𝑐𝑜𝑙𝑙𝑒𝑐𝑡𝑒𝑑 𝑖𝑛 𝑎 𝑠𝑡𝑟𝑢𝑐𝑡𝑢𝑟𝑒𝑑 𝑤𝑎𝑦, 𝑏𝑎𝑠𝑒𝑑 𝑜𝑛 𝑎 𝑝𝑟𝑒𝑑𝑒𝑓𝑖𝑛𝑒𝑑 𝑣𝑎𝑙𝑢𝑒𝑠𝑒𝑡
𝐴𝑙𝑙 𝑐𝑜𝑙𝑙𝑒𝑐𝑡𝑒𝑑 𝑑𝑖𝑎𝑔𝑛𝑜𝑠𝑖𝑠 𝑒𝑛𝑡𝑟𝑖𝑒𝑠 𝑤𝑖𝑡ℎ𝑖𝑛 𝑡ℎ𝑒 𝐸𝐻𝑅
Percentage of allergies which have been entered in a structured way, based on a predefined value-set
𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑎𝑙𝑙𝑒𝑟𝑔𝑦 𝑒𝑛𝑡𝑟𝑖𝑒𝑠 𝑤ℎ𝑖𝑐ℎ ℎ𝑎𝑣𝑒 𝑏𝑒𝑒𝑛 𝑐𝑜𝑙𝑙𝑒𝑐𝑡𝑒𝑑 𝑖𝑛 𝑎 𝑠𝑡𝑟𝑢𝑐𝑡𝑢𝑟𝑒𝑑 𝑤𝑎𝑦, 𝑏𝑎𝑠𝑒𝑑 𝑜𝑛 𝑎 𝑝𝑟𝑒𝑑𝑒𝑓𝑖𝑛𝑒𝑑 𝑣𝑎𝑙𝑢𝑒𝑠𝑒𝑡
𝐴𝑙𝑙 𝑐𝑜𝑙𝑙𝑒𝑐𝑡𝑒𝑑 𝑎𝑙𝑙𝑒𝑟𝑔𝑦 𝑒𝑛𝑡𝑟𝑖𝑒𝑠 𝑤𝑖𝑡ℎ𝑖𝑛 𝑡ℎ𝑒 𝐸𝐻𝑅
Percentage of data-items of the LBZ which are entered in the EHR in a structured way
𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑑𝑎𝑡𝑎 𝑒𝑙𝑒𝑚𝑒𝑛𝑡 𝑒𝑛𝑡𝑟𝑖𝑒𝑠 𝑤ℎ𝑖𝑐ℎ ℎ𝑎𝑣𝑒 𝑏𝑒𝑒𝑛 𝑐𝑜𝑙𝑙𝑒𝑐𝑡𝑒𝑑 𝑖𝑛 𝑎 𝑠𝑡𝑟𝑢𝑐𝑡𝑢𝑟𝑒𝑑 𝑤𝑎𝑦 𝑓𝑜𝑟 𝑡ℎ𝑒 𝐿𝐵𝑍 𝑇𝑜𝑡𝑎𝑙 𝑜𝑓 𝑚𝑎𝑛𝑑𝑎𝑡𝑜𝑟𝑦 𝑑𝑎𝑡𝑎 𝑒𝑙𝑒𝑚𝑒𝑛𝑡𝑠 𝑤ℎ𝑖𝑐ℎ 𝑠ℎ𝑜𝑢𝑙𝑑
𝑏𝑒 𝑐𝑜𝑙𝑙𝑒𝑐𝑡𝑒𝑑 𝑓𝑜𝑟 𝑡ℎ𝑒 𝐿𝐵𝑍
Percentage of data-items of the HSMR which are entered in the EHR in a structured way 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑑𝑎𝑡𝑎 𝑒𝑙𝑒𝑚𝑒𝑛𝑡 𝑒𝑛𝑡𝑟𝑖𝑒𝑠 𝑤ℎ𝑖𝑐ℎ ℎ𝑎𝑣𝑒 𝑏𝑒𝑒𝑛 𝑐𝑜𝑙𝑙𝑒𝑐𝑡𝑒𝑑 𝑖𝑛 𝑎 𝑠𝑡𝑟𝑢𝑐𝑡𝑢𝑟𝑒𝑑 𝑤𝑎𝑦, 𝑏𝑎𝑠𝑒𝑑 𝑜𝑛 𝑎 𝑝𝑟𝑒𝑑𝑒𝑓𝑖𝑛𝑒𝑑 𝑣𝑎𝑙𝑢𝑒𝑠𝑒𝑡 𝑓𝑜𝑟 𝑡ℎ𝑒 𝐻𝑆𝑀𝑅 𝑇𝑜𝑡𝑎𝑙 𝑜𝑓 𝑚𝑎𝑛𝑑𝑎𝑡𝑜𝑟𝑦 𝑑𝑎𝑡𝑎 𝑒𝑙𝑒𝑚𝑒𝑛𝑡𝑠 𝑤ℎ𝑖𝑐ℎ 𝑠ℎ𝑜𝑢𝑙𝑑 𝑏𝑒 𝑐𝑜𝑙𝑙𝑒𝑐𝑡𝑒𝑑 𝑓𝑜𝑟 𝑡ℎ𝑒 𝐻𝑆𝑀𝑅
The average percentage of items of the Basic Dataset for Care which has a value (per patient)
∑ (𝐷𝑎𝑡𝑎 𝑒𝑙𝑒𝑚𝑒𝑛𝑡𝑠 𝑜𝑓 𝑡ℎ𝑒 𝐵𝐷𝐶 𝑤𝑖𝑡ℎ 𝑎 𝑣𝑎𝑙𝑢𝑒 𝐷𝑎𝑡𝑎 𝑒𝑙𝑒𝑚𝑒𝑛𝑡𝑠 𝑜𝑓 𝑡ℎ𝑒 𝐵𝐷𝐶 ) 𝑇𝑜𝑡𝑎𝑙 𝑜𝑓 𝑖𝑛𝑐𝑙𝑢𝑑𝑒𝑑 𝑝𝑎𝑡𝑖𝑒𝑛𝑡𝑠
Outcome Indicators
Percentage of data-items of a set of specific quality registry which can be automatically extracted from routinely collected data in the electronic health record
𝑇𝑜𝑡𝑎𝑙 𝑜𝑓 𝑑𝑎𝑡𝑎 𝑒𝑙𝑒𝑚𝑒𝑛𝑡𝑠 𝑒𝑥𝑡𝑟𝑎𝑐𝑡𝑒𝑑 𝑓𝑜𝑟 𝑡ℎ𝑒 𝑞𝑢𝑎𝑙𝑖𝑡𝑦 𝑟𝑒𝑔𝑖𝑠𝑡𝑟𝑦 𝑓𝑟𝑜𝑚 𝑑𝑎𝑡𝑎 𝑐𝑜𝑙𝑙𝑒𝑐𝑡𝑒𝑑 𝑖𝑛 𝑡ℎ𝑒 𝑐𝑎𝑟𝑒 𝑝𝑟𝑜𝑐𝑒𝑠𝑠 𝑖𝑛 𝑡ℎ𝑒 𝐸𝐻𝑅
𝑇𝑜𝑡𝑎𝑙 𝑜𝑓 𝑚𝑎𝑛𝑑𝑎𝑡𝑜𝑟𝑦 𝑑𝑎𝑡𝑎 𝑒𝑙𝑒𝑚𝑒𝑛𝑡𝑠 𝑤ℎ𝑖𝑐ℎ 𝑠ℎ𝑜𝑢𝑙𝑑 𝑏𝑒 𝑠𝑒𝑛𝑑 𝑡𝑜 𝑡ℎ𝑒 𝑞𝑢𝑎𝑙𝑖𝑡𝑦 𝑟𝑒𝑔𝑖𝑠𝑡𝑟𝑦
Percentage of patients who use the patient portal 𝑎) 𝑃𝑎𝑡𝑖𝑒𝑛𝑡𝑠 𝑤𝑖𝑡ℎ 𝑎𝑛 𝑎𝑐𝑐𝑜𝑢𝑛𝑡 𝑓𝑜𝑟 𝑡ℎ𝑒 𝑝𝑎𝑡𝑖𝑒𝑛𝑡 𝑝𝑜𝑟𝑡𝑎𝑙 𝑇𝑜𝑡𝑎𝑙 𝑜𝑓 𝑝𝑎𝑡𝑖𝑒𝑛𝑡𝑠 𝑖𝑛 𝑡ℎ𝑒 ℎ𝑜𝑠𝑝𝑖𝑡𝑎𝑙 𝑏) 𝑃𝑎𝑡𝑖𝑒𝑛𝑡𝑠 𝑤𝑖𝑡ℎ 𝑎𝑛 𝑤ℎ𝑜 𝑙𝑜𝑔𝑔𝑒𝑑 𝑖𝑛𝑡𝑜 𝑡ℎ𝑒 𝑝𝑎𝑡𝑖𝑒𝑛𝑡 𝑝𝑜𝑟𝑡𝑎𝑙 𝑇𝑜𝑡𝑎𝑙 𝑜𝑓 𝑝𝑎𝑡𝑖𝑒𝑛𝑡𝑠 𝑤𝑖𝑡ℎ 𝑎𝑛 𝑎𝑐𝑐𝑜𝑢𝑛𝑡 𝑓𝑜𝑟 𝑡ℎ𝑒 𝑝𝑎𝑡𝑖𝑒𝑛𝑡 𝑝𝑜𝑡𝑎𝑙
P
ROOF OFC
ONCEPT-
F
EASIBILITY FORI
MPLEMENTATIONAs a proof of concept, we selected seven QIs to investigate them in one university hospital on their feasibility in order to find possible barriers in calculating them and using them in developing improvement strategies. These seven QIs were selected to be measured because they were deemed as most interesting by the hospital and most feasible to be measured within the restricted time period of this project. The QIs we investigated were: 1) ‘there is a central point of contact within the hospital where physicians and nurses can go to when they have questions about collecting data once and unambiguous.’, 2) ‘the hospital provides patients with the possibility to enter their own information in the patient portal’, 3) ‘the percentage of allergies which have been entered in a structured way based on a predefined value-set’, 4) ‘the percentage of data-items of the LBZ which are entered in the EHR in a structured way’, 5) ‘the percentage of data-items of the HSMR which are entered in the EHR in a structured way’ 6) ‘the percentage of patients who use the patient portal’, and 7) ‘the percentage of data-items of a set of specific quality registry which can be automatically extracted from routinely collected data in the electronic health record’. A summary of general barriers found is shown in Table 4.
Error! Reference source not found. describes the results of these QIs for the one selected university hospital. The resulting QI scores were only available for internal use only and are prohibited to share and therefore not available for online publication. Below we will discuss the challenges encountered when calculating each of those four QIs.
Indicator 1 & 2
We did not find any barriers in calculating the first two indicators.
Indicator 3
The third indicator we wanted to calculate was the percentage of allergies which have been collected in a structured and standardized way. The main barrier we found in calculating this indicator was the fact that we could not automatically calculate the number of allergies entered in an unstructured way in the EHR (the denominator of this indicator). A possible solution to this problem would be to take a sample of patients records and manually review free-text fields for unstructured entries of allergies of a patient. This would be a labor-intensive procedure. A more feasible interpretation of the indicator was to calculate the percentage of patients with either at least one allergy recorded within a standardized field or having the indication that no allergies are known for the patient. This will give an indication whether there is a structured entry of allergies in every patient file. The percentage for this latter interpretation of the indicator could be calculated within the investigated university hospital.
Indicator 4 & 5
The fourth and fifth indicator we wanted to calculate were the percentage of structured entries of data-items needed for the LBZ and HSMR. The data needed for the LBZ and HSMR are similar and include data items such as diagnosis, medical procedures, survival state, admission date, date of birth, patient number and several other data items. Most of these data are logged in the backend of the EHR, and if errors in data-entries occurred, they were due to technical errors rather than as a result of recording errors by physicians. With the current formulation of the definition, hospitals could score lower when the EHR does not function correctly, while the intend of the indicator was to show problems with structured and standardized data collected in the care process. Main diagnosis and main medical procedures are the most important aspect of the LBZ and HSMR which should be entered in a structured way and should be coded for the LBZ and HSMR. For this reason, we redefined the indicator to the percentage of main diagnosis, comorbidities, main medical procedures and other medical procedures which are coded and are labeled as a primary or secondary finding during the hospital admission. At the moment, medical encoders are still needed to check all data entries before sending the data set to the LBZ and HSMR. According to a manager of the medical encoding department, the correct entry of medical data needed for LBZ and HSMR by physicians was still a problem in all hospitals and percentages on this indicator will be low in all hospitals.
Indicator 6
The third indicator we calculated was on the use of the hospital’s patient portal by patients. The indicator was split into two sub-indicators: 1) the percentage of patients with an account and 2) the percentage of patients with an account and who have accessed the patient portal. After meeting with the expert on patient portals in the university hospital, we found a possible barrier in the interpretation of the indicator. Since a patient might only go to the hospital once every several years, it might be unclear whether a patient has accessed the patient portal only once because he had only one appointment or whether (s)he is not interested in using the patient portal anymore. A suggestion in defining a possible indicator was to include only chronic patients to indicate multiple uses of the patient portal. However, a barrier in calculating this percentage in the investigated university hospital was that the accounts of patients were not coupled to diseases of patients.
Indicator 7
Lastly, we wanted to calculate the indicator which measures whether data for mandatory data elements of quality registries data sets can be extracted directly from routinely collected data. It was decided to restrict and stratify this indicator for the quality registries CVAB, DSCA, and NICE. For the NICE registry, it was not possible to calculate the indicator on the short term.
A complete set of the mandatory data elements has to be sent to each of the quality registries. Though, there is a difference in how hospitals are collecting the necessary data. For some quality registries, separate forms are made for the mandatory data within the EHR. In the university hospital which we investigated, the EHR management tried to get as much data as possible from the routinely collected data. Certain data elements were not relevant for the care process itself, for example, the number of family members who are younger than 60 years old is a mandatory data element, but not relevant for the care process. These data elements could be entered in a separate form in the EHR.
The manager responsible for extracting data for secondary use of medical data suggested that the indicator should be split into two separate indicators: 1) the number of mandatory data elements extracted from the EHR, and 2) the percentage of mandatory data elements extracted from routinely collected data in the EHR. By dividing the indicator into two sub-indicators, there will be a nuance if all data is extracted from the EHR, but not all data elements were collected based on routinely collected data, but in a separate form.
Table 4. General barriers found in the feasibility study for each indicator
# Barriers
Indicator 1
There is a central point of contact within the hospital where physicians and nurses can go to when they have questions about collecting data once and
unambiguous. and
No barriers
Indicator 2
The hospital provides patients with the possibility to enter their own information in the patient portal.
No barriers
Indicator 3
Percentage of allergies which have been entered in a structured way with a predefined value-set
Denominator not feasible to be calculated
Indicator 4&5
Percentage of data-items of the LBZ and HSMR which is entered in the EHR in a structured way
Ambiguity in definition
Indicator 6
Percentage of patients who use the patient portal Limited actionability in current definition
Indicator 7
Percentage of data items for certain quality registries which can be automatically extracted from the data collected in the care process in the EHR
Ambiguity in definition
Not feasible to calculate numerator for some quality registries
C
HAPTER
4:
D
ISCUSSION
S
TATEMENT OF PRINCIPAL FINDINGSBy using the modified RAND method, we developed a set of eleven QIs to assess the degree of COUMT adoption in a hospital. The set includes two structure indicators, seven process indicators, and two outcome indicators and covers communication to healthcare professionals, structured entry of allergies, medication, diagnoses, and medical procedures and use of routinely collected data collected for secondary purposes such as quality registrations and patient portals.
In our pilot study to test the feasibility of QI measurement and their use in developing improvement strategies, we found several barriers: 1) the data needed to calculate the indicator can be hard to extract from the EHR, 2) some QIs are not defined with enough detail, 3) some QIs are less actionable with their current definition. The feasibility study showed that the QIs in their current form are not ready to be used in practice and need some further refinement.
S
TRENGTHS AND WEAKNESSES OF THE STUDYOne of the strengths of this study is the use of a modified version of the proven RAND method. The RAND method is a widely used method to develop quality indicators. The modified version developed by van Engen-Verheul [25] provided a more elaborate way to measure the appropriateness of the QIs. By using this modified RAND method, there was a more explicit focus on the feasibility to measure the QIs and the actionability of the QIs. The RAND method suggests a range of sources to develop the QIs. We reviewed the literature for possible QIs of the adoption of COUMT. Furthermore, we asked physicians, employees, managers, and directors from EHR implementation programs of all university hospitals to provide possible QIs. The elicitation of several types of professionals gave a more diverse insight in potential QIs to measure the adoption of COUMT. The use of the modified RAND method in combination with developing these QIs on a national level, should improve the applicability of the QIs for hospitals in the Netherlands and increases the likelihood that the QIs are used to benchmark between the different hospitals.
Our study has several limitations. Throughout the course of the study, the composition of the expert panel differed. During the elicitation of QIs, we included physicians and employees and directors of EHR implementation programs. During the rating and discussion phases, only the directors of the EHR implementation programs were included in the process. This might have influenced the outcome of these phases since different experts have different priorities. However, we deemed the directors as most knowledgeable on relevance, feasibility, and actionability of the QIs, since they are responsible for the adoption of COUMT by their own hospital and should have the best overview of the different rating criteria. Another limitation was the number of participants we included. The RAND method suggests at least nine to eleven participants to be included in the expert panel during the course of the indicator development [26]. Due to the fact that only eight university hospitals, led by seven EHR implementation directors are present in the Netherlands, we could not increase the number of members of this type in our expert panel during the rating and discussion phase. All university hospitals have recently implemented a central EHR or are still in the process of implementing a new EHR. For this reason, experts with the same knowledge on the criteria as the EHR implementation directors had only limited availability and could not be included in the rating and discussion phase.
In contrast to what the RAND method recommends, we used the discussion meeting to reach consensus on the final set of QIs. The RAND method suggests to use this round as a discussion of the results of the rating phase without rejecting QIs and to use a second round of rating of the QIs to select the final QIs. Due to time constraints of the project and an expected low response rate, we decided to reach consensus on the final indicator set in the discussion meeting. Reaching consensus in a face-to-face expert meeting can be influenced by dominant individuals and group pressure for conformity [55]. Some participants were more dominant during the meeting, but all participants mentioned their concerns with the discussed indicators, and this should not have affected our results.
S
TRENGTHS AND WEAKNESSES IN RELATION TO OTHER STUDIESOther studies often investigate only one of the aspects of the COUMT paradigm. Terminology models and information quality of medical data in EHRs are subjects that are often explored [17]. Other concepts such as awareness of physicians and organizational barriers are less frequently described in the literature [17,56]. We included QIs in our draft set on a larger set of concepts of COUMT which should provide a broader view of the extent to which a hospital has adopted the COUMT paradigm.
Our final QI set covers three out of five goals set by the Clinical Documentation at the Point of Care program [5]. The goals on the structured and unambiguous recording of healthcare
The COUMT paradigm is also envisioned in other countries with programs such as the Meaningful Use program in the USA [57] and the InfoWay program in Canada [58]. Studies investigating the subject in these countries elaborate on what elements should be considered when trying to implement the paradigm. The Meaningful Use program in the United States of America has set up several QIs investigating whether hospitals are using their EHR in a meaningful way [57]. Some of these QIs also show the extent of adoption of the COUMT paradigm in hospitals. Something that stands out as QIs recommended by these countries is the implementation of terminologies such as SNOMED CT and ICD [17,57,58]. Even though we included QIs on the subject of implemented standards in the EHR in our draft set, the expert panel gave a lower rating on relevance and feasibility. The panel might have been more focused on process and outcome indicators and less on the structure indicators, since they are more interested in the results of their efforts. Another reason could be that they are already informed on whether they have the right materials, human resources, and organizational structure available and that indicators on these elements are less relevant.
The total number of QIs in our draft set was lower than regularly found in clinical QI development studies. These studies included up to 116 QIs in their draft set, compared to our 38 QIs [59–61]. The lower number of QIs might be due to the fact that our expert panel was smaller than others. Our expert panel also seemed to be focused on information quality, system quality and extractability of medical data, and less on structure QIs such as awareness of physicians of the COUMT paradigm.A larger expert panel could have led to a bigger and more diverse draft set of QIs. Furthermore, the expert panel was motivated from the beginning to come up with a limited set to start with and which can be extended in a later phase.
M
EANING OF THE STUDYThe development of a quality indicator set to measure the adoption of the COUMT paradigm is another step which can help with the structured and standardized collection of medical data at the point of care for multiple purposes. Our literature study on possible QIs provides a short overview of what has already been investigated on this subject. The quality indicator set is intended to be used as a tool to measure how well and on which aspects hospitals have adopted the COUMT paradigm. By measuring these QIs for each individual hospital, hospitals can improve on factors they underperform compared to their peers. Furthermore, the NFU and the different directors within the university hospitals can use the QIs to benchmark nationally between hospitals to see how far hospitals in the Netherlands are with the adoption of the COUMT paradigm. By benchmarking between hospitals, best-practices can be discovered and shared among institutions.
U
NANSWERED QUESTIONS AND FUTURE RESEARCHDeveloping the quality indicators and defining them with the AIRE instrument is only a first step before the QIs can be used in practice. Up to 20% of QIs are determined not to be feasible in practice because the indicators are insufficient valid and reliable and data sources are not suitable for extraction [62]. Our feasibility study also showed that, even though we used to RAND Method to develop our QIs, There were still barriers in the feasibility to measure the QIs with their current definition. The entire indicator set should be tested for feasibility to redefine the indicator to make sure they are valid and precise and discriminatory. These categories of the AIRE instrument were not yet described in our study. Moreover, to improve the reproducibility, validity, interpretability, traceability, timeliness, and comparability of the quality indicator set, the set should be formalized with a formalization method such as the Clinical Indicator Formalization (CLIF) [63].
C
ONCLUSION
In summary, in cooperation with the NFU and all university hospitals in the Netherlands, we developed a set of eleven QIs which is relevant to assess the degree of COUMT adoption in a hospital. However, ourpilot study on the feasibility to measure the QI set and their use in developing improvement shows that some indicators have to be further reviewed and formalized before hospitals can use them to benchmark themselves against their peers and to actually improve on the COUMT paradigm.
R
EFERENCES
[1] Prokosch HU, Ganslandt T. Perspectives for medical informatics - Reusing the electronic medical record for clinical research. Methods Inf Med 2009;48:38–44. doi:10.3414/ME9132.
[2] Safran C. Reuse of clinical data. Yearb Med Inform 2014;9:52–4. doi:10.15265/IY-2014-0013.
[3] Raghupathi W, Raghupathi V. Big data analytics in healthcare : promise and potential 2014;2:1–10. doi:10.1186/2047-2501-2-3.
[4] Sandhu E, Weinstein S, McKethan A, Jain SH. Secondary uses of electronic health record data: benefits and barriers. Jt Comm J Qual Patient Saf 2012;38:1,34-40.
[5] Registratie aan de Bron. Basic document The basic principles of health and care information models (HCIMs) and how they can be used. vol. 1. 2017.
[6] Price Waterhouse Coopers. Transforming healthcare through secondary use of health data. 2009.
[7] Skentzos S, Shubina M, Plutzky J, Turchin A. Structured vs. unstructured: factors affecting adverse drug reaction documentation in an EMR repository. AMIA Annu Symp Proc 2011;2011:1270–9.
[8] Joukes E, Cornet R, de Keizer N, de Bruijne M. Collect Once, Use Many Times: End-Users Don’t Practice What They Preach. Stud Heal Technol Inf 2016:252–6. doi:10.3233/978-1-61499-678-1-252.
[9] Kush R, Alschuler L, Ruggeri R, Cassells S, Gupta N, Bain L, et al. Implementing Single Source: The STARBRITE Proof-of-Concept Study. J Am Med Informatics Assoc 2007;14:662–73. doi:10.1197/jamia.M2157.
[10] El Fadly AN, Rance B, Lucas N, Mead C, Chatellier G, Lastic PY, et al. Integrating clinical research with the Healthcare Enterprise: From the RE-USE project to the EHR4CR platform. J Biomed Inform 2011;44:S94–102. doi:10.1016/j.jbi.2011.07.007.
[11] Nivel. Onderzoek naar de overdracht van patiëntinformtussen ziekenhuizen en VVT. 2012.
[12] Hicken VN, Thornton SN, Rocha RA. Integration challenges of clinical information systems developed without a shared data dictionary. World Conf Med Informatics 2004;107:1053–7.
[13] Hogan WR, Wagner MM. Free-Text Fields Change the Meaning of Coded Data. Am. Med. Informatics Assoc., 1996, p. 517–21.
[14] Christoph J, Griebel L, Leb I, Engel I, Kopcke F, Toddenroth D, et al. Secure Secondary Use of Clinical Data with Cloud-based NLP Services. Towards a Highly Scalable Research
