Integrating a smart CDSS in Preventive Child
Healthcare: a mixed-method approach to
study acceptance factors and improve design
June 2020
by
Heleen Werges
University of Amsterdam
Medical Informatics
2
Integrating a smart CDSS in Preventive Child Healthcare: a
mixed-method approach to study acceptance factors and
improve design
Master thesis
Period: December 2019 - June 2020 Student
Heleen Werges Student ID: 10727094
h.werges@amsterdamumc.nl Location
TNO, the Netherlands Organisation for applied scientific research Healthy Living - Child Health department
Schipholweg 77, 2316 ZL Leiden
Mentor Tutor
Dr. O.A. (Olivier) Blanson Henkemans Dr. L.W. (Linda) Dusseljee - Peute Healthy Living - Child Health department,
TNO
Department Medical Informatics, University of Amsterdam
Schipholweg 77, 2316 ZL Leiden Meibergdreef 9, 1105 AZ Amsterdam olivier.blansonhenkemans@tno.nl l.w.peute@amsterdamumc.nl
3
Preface
This thesis was the final assignment of my Scientific Research Project (SRP) and final product of my Master Medical Informatics at the University of Amsterdam. This thesis was undertaken at the request of TNO, where I started an internship. This internship allowed me to perform extensive investigation on the use of the smart CDSS and it provided me with new insights in the possibilities of innovation technologies in the setting of the preventive child healthcare. The diversity of the internship enabled me to develop skills in both training and engaging with end-users, pitching new ideas to other stakeholders and management, back-end coding, usability testing and interface de-sign. Despite the shift from TNO office to home-office (due to the coronavirus), I am grateful for the support and advice received from my TNO colleagues at distance.
There are a number of people that deserve sincere gratitude, who have helped and contributed to my thesis and (scientific) development during my Master. First, I want to thank Linda Dusseljee-Peute. Not only for the efforts she took to help me during my SRP but also for all the things she taught me during my Master. I cherish the relationship we have built during these two years. Your enthusiasm has always urged me to deliver the best possible result.
Second, I especially like to thank my mentor, Olivier Blanson Henkemans. From day one you have given me freedom to implement my own ideas during the internship. Besides performing my SRP, you gave me the responsibility to improve the back-end of the smart CDSS and to contribute to the success of the SRM. You really treated me as a colleague instead of an intern, which I highly value. Besides that, I appreciate you picking up the phone at any time.
Also, I would to express my gratitude to my family for their support and feedback. And at last I want to thank my boyfriend, Rik. Whenever the road got bumpy, you were there to support me.
Dear reader, I hope you will enjoy reading this thesis. Heleen Werges
4
Summary
Introduction Since 1996, the Preventive Child Healthcare (PCH) develops and implements evi-dence-based guidelines to contribute to the quality and uniformity of the preventive care for chil-dren ranging from 0-19 years old. However, since the introduction of these PCH guidelines, it has been demonstrated that the feasibility and implementation of the guidelines are not guaranteed. To overcome these problems, TNO1 is developing a smart clinical decision support system (CDSS)
integrated in an electronic health record. A CDSS is designed to deliver patient-specific recom-mendations to the health care provider, to aid clinical decision making (1). However, previous research also showed that the uptake of the CDSS in a clinical setting is low (2–6). Most imple-mentations of CDSS still suffer from usability issues (7) and it is challenging to design a CDSS that optimally supports the healthcare providers in clinical practice (2). To prevent these problem from happening, the smart CDSS needs to 1) correct for recognised CDSS acceptance factors, 2) needs to fulfil the end users’ needs and 3) should consider the PCH workflow in which it is inte-grated. The human-centred design (HCD) has potential to improve the CDSS usability (2,8–10) from the perspective of the end user. The overall aim of this study was to assess whether an HCD implementation approach contributes to the acceptance of smart CDSS within the PCH.
Methods This thesis consists of three studies performed in line with HCD. The first two studies relate to the first two HCD steps and focus gaining insight into the context of use and the users’ requirements (Chapter 3 & 4). In combination with the HCD approach, an acceptance factor analysis was performed per HCD step, to prevent an over-designed and over-complex smart CDSS. The set of examined acceptance factors were derived from the UTAUT, MIDI2 and a literature
review on specific CDSS acceptance factors. In addition to the acceptance factor analysis, 11 in-terviews, 28 observations, 3 questionnaires were utilized to obtain a detailed view on the context of use of the current implemented paper-based guideline and the new introduced smart CDSS dur-ing the pilot period. Sequentially, eight (re)designs were developed in computerized mock-ups and were validated in an online focus group, which relate to the last two HCD steps (Chapter 5). Results Overall, the results of this thesis showed that the participants positively responded to the acceptance factors. Of the 30 factors that could potentially influence the smart CDSS ac-ceptance and use, 90% were assessed as drivers rather than a barrier. The extent of use of the smart CDSS varied between the PCH disciplines and the use of the smart CDSS was most useful for less experienced PCH professionals. In addition, the use of the smart CDSS supports the professional’s awareness of incomplete patient records. Users were alerted to add data in order to obtain a recom-mendation from the smart CDSS. However, the evaluation of context of use and users’ require-ments showed that the originally designed smart CDSS does not yet optimally supports an efficient workflow in clinical practice. Barriers that affected smart CDSS acceptance were: incorrect presen-tation of user interface, no perceived client relevance, technical issues, time constraints, unclarity about system components, and undesirable system behaviour. All barriers had links to slowing down the task efficiency during clinical practice. Therefore, the redesigns focussed primarily on improving the efficiency to quickly gain information and quickly navigate from and to the smart CDSS. Finally, the evaluation of the redesigns in the online focus group, showed that all seven PCH professionals in the focus group agreed that the usability of the smart CDSS improved. Discussion In summary, this thesis showed that the HCD approach in combination with ac-ceptance analysis, positively contributes to the usability of the smart CDSS within the PCH. The smart CDSS was well-received by the pilot participants. Furthermore, by following all steps of the HCD cycle, the acceptance factors that emerge in a specific step can be omitted in a later HCD step. An example is that the smart CDSS was not shared during the pilot to enable shared decision making. But by incorporating the context of use and the users’ needs in the redesign step, it became clear that the professionals expect to use shared decision making with the redesigns. Future research should look at the implementation of the smart CDSS including more PCH guidelines, that use unstructured data, in more PCH organizations. Mainly because the COVID-19 situation reduced the sample size and biased the use of the smart CDSS.
1 the Netherlands Organization for Applied Scientific Research (TNO)
5
Samenvatting
Introductie Sinds 1996 ontwikkelt en implementeert de Jeugdgezondheidszorg (JGZ) ‘evi-dence-based’ richtlijnen om bij te dragen aan de kwaliteit en uniformiteit van preventieve zorg voor jeugdigen. Sinds de introductie van de JGZ-richtlijnen is echter gebleken dat de toepassing van de richtlijnen in het dagelijks werk nog niet is gegarandeerd. Om bij te kunnen dragen aan succesvolle implementatie van richtlijnen, is vanuit TNO3 een project gestart om een slim
beslissingsonder-steunend systeem (CDSS) te integreren in een digitaal dossier. Deze slimme CDSS heeft als doel ondersteuning te bieden aan een meer uniforme, wetenschappelijk-onderbouwd en efficiënt manier van werken. Uit eerder onderzoek naar het succes en implementatie van CDSS is echter gebleken dat het gebruik van een CDSS in klinische setting nog niet volledig wordt doorgevoerd. De bruik-baarheid van CDSS-systemen blijft een veel voorkomend probleem en daarom is het een uitdaging om een CDSS te ontwerpen dat de zorgverlener in de klinische praktijk optimaal ondersteund. Om te voorkomen dat de nieuw ontworpen slimme CDSS dezelfde problemen ondervindt, moet het rekening houden met erkende CDSS acceptatiefactoren, voldoen aan de behoeften van de eindge-bruikers en aansluiten op de werkwijze van de JGZ. De Human Centered Design (HCD) methode biedt uitkomsten om de bruikbaarheid van de CDSS te verbeteren vanuit het perspectief van de gebruiker. De doelstelling van deze studie is om te beoordelen of de HCD-methode kan bijdragen aan de acceptatie van een slimme CDSS binnen de JGZ.
Methode Deze thesis is onderverdeeld in drie studies die de HCD-cyclus volgen. De eerste twee HCD stappen hebben betrekking tot het verkrijgen van inzicht in de gebruikscontext en de gebruikersbehoeften. In combinatie met de HCD-cyclus, werd een acceptatieanalyse verricht om te voorkomen dat de CDSS te complex en te gericht is op wensen van één gebruikersgroep. De set onderzochte acceptatiefactoren is afgeleid van de UTAUT, MIDI4 en literatuuronderzoek. Naast
de acceptatieanalyse, werden 11 interviews, 28 observaties en 3 vragenlijsten gebruikt om een ge-detailleerd inzicht te verkrijgen in de gebruikscontext van de huidig geïmplementeerde papieren richtlijn en de slimme CDSS. Opeenvolgend, werden acht (her)ontwerpen ontwikkeld en geëvalu-eerd in een online focusgroep, tijdens de laatste twee stappen van de HCD-cyclus.
Resultaten De resultaten toonden aan dat deelnemers over het algemeen positief reageerden op de acceptatie factoren. Van de 30 factoren die mogelijk de acceptatie van de slimme CDSS konden beïnvloeden werd 90% beoordeeld als drijfveer in plaats van barrière. De mate van gebruik varieerde tussen JGZ disciplines en het gebruik was het meest nuttig voor de minder ervaren pro-fessionals. Bovendien ondersteund het gebruik van de slimme CDSS de professionals om inzicht te krijgen in incomplete patiëntendossiers. Uit de evaluatie van de gebruikscontext en gebruikers-behoeften bleek echter ook dat de slimme CDSS een efficiënte workflow in de klinische praktijk nog niet optimaal ondersteunde. Barrières die van invloed waren op de acceptatie waren: onjuiste presentaties, geen waargenomen cliënt relevantie, technische problemen, tijdsdruk, onduidelijkhe-den in het systeem en ongewenst systeemgedrag. Alle barrières hadonduidelijkhe-den een vertragend effect op de werk efficiëntie. De herontwerpen zijn daarom voornamelijk gericht op het verbeteren van de effi-ciënte om snel gericht informatie te verkrijgen en snel te navigeren van en naar de slimme CDSS. Uiteindelijk beoordeelden alle zeven professionals de herontwerpen als meer gebruiksvriendelijk. Discussie Deze thesis concludeerde dat de HCD benadering in combinatie met de accepta-tieanalyse positief bijdraagt aan de gebruikersvriendelijkheid van de slimme CDSS. De slimme CDSS werd positief ontvangen door de pilot deelnemers. Bovendien kunnen bij het volgen van alle stappen van de HCD-cyclus de acceptatiebarrières die in een specifieke stap naar voren komen, in een latere HCD-stap worden weggenomen. Een voorbeeld is dat de slimme CDSS tijdens de pilot niet werd gebruikt voor gezamenlijk besluitvorming. Maar door de gebruikscontext en de gebrui-kersbehoeften in de herontwerpstap op te nemen, werd duidelijk dat de herontwerpen wel een bij-drage konden leveren aan gezamenlijke besluitvorming. Toekomstig onderzoek moet kijken naar de implementatie van meerdere slimme richtlijnen die ongestructureerde data gebruiken, in meer-dere JGZ organisaties. Vooral omdat door de COVID-19 het aantal deelnemers heeft verlaagd en het gebruik van de slimme CDSS heeft beïnvloed.
3 Nederlandse Organisatie voor toegepast-natuurwetenschappelijk onderzoek
6
Content
List of abbreviations ... 8 General introduction ... 9 1.1 Background ... 11 1.2 Setting ... 11 1.3 The PCH guidelines ... 111.4 The system – smart CDSS ... 12
1.4.1 The configurator ... 12
1.4.2 Interaction with the CDSS server and external applications ... 12
1.5 Human centred design approach ... 13
1.6 Research questions and thesis outline ... 13
1.7 Study design ... 15
1.8 COVID-19 ... 16
Theoretical background ... 17
2.1 Measurement Instrument for Determinants of Innovations ... 17
2.2 Unified Theory of Acceptance and Use of Technology ... 17
2.3 Barriers known for CDSS success ... 18
The use and implementation of the smart CDSS compared to the paper-based guideline ... 20
3.1 Introduction ... 21 3.2 Methods ... 21 3.2.1 Participants ... 21 3.2.2 Measures ... 21 3.2.3 Data collection ... 22 3.2.4 Statistical analysis ... 23 3.3 Results ... 23
3.3.1 Overall participant inclusion ... 23
3.3.2 Participant characteristics ... 24
3.3.3 Use of the paper-based guideline at baseline... 24
3.3.4 Use of the smart CDSS during the pilot ... 25
3.3.5 Determinants of implementation ... 26
3.4 Discussion ... 28
3.4.1 Strengths and limitations ... 30
3.4.2 Comparison with other studies ... 30
3.4.3 Further research ... 31
Barriers and user needs for smart CDSS acceptance ... 32
4.1 Introduction ... 33
4.2 Methods ... 33
4.2.1 Participants ... 33
4.2.2 Measures ... 33
7
4.2.4 Data analysis ... 34
4.3 Results ... 34
4.3.1 Participant inclusion ... 34
4.3.2 Participant characteristics ... 35
4.3.3 Drivers for smart CDSS use... 35
4.3.4 Barriers for smart CDSS use ... 36
4.3.5 New identified needs for smart CDSS use ... 38
4.4 Discussion ... 39
4.4.1 Main findings ... 39
4.4.2 Strength and limitations ... 40
4.4.3 Comparison to other studies ... 40
4.4.4 Future research ... 41
A redesign of the smart CDSS: development and testing phase ... 42
5.1 Introduction ... 43
5.2 Redesign solutions ... 43
5.2.2 Initial redesign of the Configurator ... 44
5.2.3 Approach of the redesign ... 45
5.2.4 Mock-ups... 46
5.3 The online focus group ... 50
5.3.1 Methods ... 50
5.3.2 Results ... 51
5.4 Discussion ... 53
5.4.1 Strengths & Limitations ... 54
5.4.2 Comparison to other studies ... 55
5.4.3 Future research ... 56
Overall discussion ... 57
6.1 Overall findings ... 58
6.2 Strength and Limitations ... 59
6.3 Relation to other work ... 61
6.4 Implications & Future research ... 61
6.5 Overall conclusion... 62 References ... 63 Appendices ... 67 Appendix A ... 67 Appendix B ... 69 Appendix C ... 70 Appendix D ... 71 Appendix E ... 72 Appendix F ... 73 Appendix G ... 74 Content
8
List of abbreviations
BDS
Basic dataset (Dutch: Basisdataset)
CDSS
Clinical Decision Support System
EHR
Electronic Health Record
HCD
Human Centred Design
MIDI
Measurement Instrument of Determinants of Innovations
PCH
Preventive Child Healthcare
SRM
Dutch: Slimme Richtlijnmodule, Smart Guideline Module
UTAUT
Unified Theory of Acceptance and Use of Technology
9
Chapter 1
General introduction
In the Netherlands, there is a unique system to deliver preventive care to each child aging from 0 to 19 years old, concerning approximately 3.6 million children (source: CBS StatLine 2019). The Preventive Child Health Care (PCH) is responsible for providing information and additional care to children when required. In the Netherlands, parents and children are invited to receive consult from the PCH professionals; this concerns about thirteen consultations during the age of 0 to 3 years at well-baby clinics and about three consultations during the age of 4 to 18 at the primary- and high school (11). About 95% of the invited (parents of) children make use of the available services of the PCH (12). The consultations of the PCH are designed to detect a wide range of disorders, such as: deviating length growth; overweight; visual and hearing impairments; and psy-chosocial problems. The consultations also offer guidance to parents who require aid with the up-bringing of their child.
Since 1996 the PCH developed and implemented evidence-based guidelines, to assure the quality of PCH (13). These guidelines are important assets to the work of professionals in the PCH and are defined as “Statements that include recommendations intended to optimize patient care that are informed by a systematic review of evidence and an assessment of the benefit and harms of alternative care options” (Institute of Medicine, 2011). Guidelines in healthcare practices are known for their benefits to improve health outcomes, consistency of care, and quality of clinical decisions and care (14,15). PCH guidelines have been developed to promote the physical, psycho-logical and social health of children, e.g. how to handle when a child is involved with child abuse, or when a child has problems with sleeping.
To determine whether a guideline can have impact on health outcomes and the quality of care, evaluation of the adherence to the guideline by the health professional is essential. Since the introduction of the PCH guidelines, the feasibility and implementation of the guidelines are still not guaranteed: 70% of the PCH professionals are aware of the PCH guidelines, but only 7% use the PCH guidelines as intended (13). A study performed in 2012, showed a high variance in guide-line use among PCH professionals and only one out of the nine examined PCH guideguide-lines met the criteria of both 90% guideline awareness and 80% guideline use (13,16). The PCH experiences problems to keep up to date with the guidelines and to implement them in daily practice. In 2019, the GGD Fryslân examined the process of development and implementation of PCH guidelines and found barriers such as: insufficient coherence between guidelines; low ownership of the guidelines by professionals; and differences in interpretation of the too broad explanation of the guideline. Other identified barriers to use the PCH guideline in practice are: the great number of guidelines which are being updated on a regular basis; the inconsistent structure of a guideline; a rather theo-retical character of the guideline which makes it harder to apply; and the lack of standardization to uniformly register the data in the EHR (17). Furthermore, the PCH professionals expressed a need for a clear view about the flow charts and coherence of guidelines, the possibility to act according to their own professional perception and providing feedback on PCH guidelines regarding the con-tent and feasibility.
To overcome implementation problems of guidelines in clinical practice, a clinical decision support system (CDSS) may offer opportunities (2,18–21). Clinical decision support is defined as “a process for enhancing health-related decisions and actions with pertinent, organized, clinical knowledge, and patient information to improve health and healthcare delivery.”(22) CDSS are designed to provide health care providers with patient-specific assessments of recommendations to aid clinical decision making (1). Previous research showed that a CDSS improves guidelines ad-herence (23–27). If the CDSS is integrated within a medical software this can aid to increase the use and success of the guideline-based CDSS (1,24). Furthermore, a CDSS reduces medical errors, improve health care process measures relevant to preventive services, and improves health care providers’ performance by using preventive care reminder systems (23,28).
10
To aid implementing guidelines in Dutch PCH, the Netherlands Organization for Applied Scientific Research (TNO) is developing a smart CDSS, called the Slimme Richtlijnmodule (Smart Guideline Module, SRM). The smart CDSS is designed to overcome the difficulties with guideline imple-mentation by offering support for the uniform and efficient use of the PCH guidelines. The devel-opment of the smart CDSS is part of a project funded by the Netherlands Organisation for Health Research and Development (ZonMw), titled ‘Development of uniform and evidence-based work by PCH through integration of Smart Guideline Module and Digital Dossier (DD) JGZ’ (Dutch title: ‘Ontwikkeling van uniform en wetenschappelijk-onderbouwd werken door JGZ middels integratie van Slimme Richtlijnmodule en Digitaal Dossier (DD) JGZ’)(project 729310005). Prior research into the needs of 13 PCH professionals, showed that professionals expect that the smart CDSS can contribute to work more efficiently and effectively.
However, despite of the positive benefits of a CDSS, the uptake of the CDSS in a clinical setting is still not guaranteed (2–6). Resistance to use and accept a CDSS can have several reasons, such as: insufficient computer ability, lack of eagerness to change user behaviour, the integration into practice workflow, or alert fatigue [21, 25–27].
The clinical acceptance and usage determines the efficacy of the CDSS (32). According to Davis et al. (2003) user acceptance, the willingness or intention to use a technology, is one of the factors determining the successful adoption of a new technology. Technology acceptance is not related to the technology on its own, but has dimensions such as user attitude and personality, social influence, trust, and numerous facilitating conditions (33). A variety of models and frameworks have been developed to describe user adoption to new information systems, and to provide factors that can affect the user acceptance. In healthcare, the use of models the Technology Acceptance Model (TAM) and the Unified Theory of Acceptance and Use of Technology (UTAUT) is wide-spread (34). The UTAUT is able to explain 70% of the variance while other theories explain only 30-40% variance in adoption behaviour (35). Therefore, using the UTAUT can provide insight in the determinants of user acceptance of the smart CDSS in PCH. Previous studies showed that socio-organizational-cultural factors play a significant role with the acceptance of health care technolo-gies (36). Whereas the UTAUT implies that the user acceptance is influenced by perceived useful-ness and ease of use, previous literature addressed the need for a focus on the use of the technology in context, including: social, cultural, professional, organizational and political influences (37). The use of a published instrument in the PCH, the Measurement Instrument of Determinants of Inno-vations (MIDI) could account for this. It is particularly characterized with the inclusion of the or-ganizational and socio-political contexts in which the organization operate.
In addition to instruments used for the acceptance and implementation of new innovations, such as the UTAUT and the MIDI, understanding the reasons for the low uptake to CDSS in spe-cific is essential. In the last decades, widespread studies have been performed to search for barriers and facilitators in CDSS implementation. One of the leading factors influencing the success of the CDSS is the fit within the professionals’ workflow (4,6,21,24,29,37,38). The CDSS should be con-tribute to a more efficient workflow instead disrupting the workflow itself. To improve the work-flow efficiency an analysis of the requirements and an evaluation of the CDSS prototype is required (39). This evaluation could be performed by following the methodology of Human Centred Design (HCD). HCD is method to include the user’s perspective into the software design process in order to achieve a more usable system, which leads to wider acceptance, reduced errors and increased productivity (40). A usable system, is critical for the effective and efficient use (24) and CDSS acceptance (41). However, most CDSS still suffer with problems in usability (7) and it is challeng-ing to design a CDSS that optimally supports the healthcare providers in clinical practice (2). Poorly designed CDSS can lead to usability problems, dissatisfaction with the system and may disrupt normal flow of clinical activities and alert fatigue (7,9,42).
In conclusion, CDSS involves (I) the adoption of a new health information technology (including technology adoption and the technology usability) and (II) the challenge to integrate clinical evidence into routine practices (5). It has been demonstrated that understanding the reasons behind the low uptake is essential for effective implementation of a CDSS (5). Therefore, the ex-pectation is that the introduction of this CDSS in PCH comes with several challenges regarding the attitude of professionals towards a decision-support system (acceptance to adopt), the incorporation of the CDSS during clinical practice, and the usability of the CDSS.
11
Based on the perceived benefits of previous performed studies in CDSS and the needs assessment performed in July 2019, the smart CDSS showed potential to overcome the problems identified in the PCH guideline implementation. But to overcome these problems, the smart CDSS needs to fulfil the users’ needs and should be integrated into the PCH workflow, to improve its usability and its acceptance. This study was performed to evaluate the first experiences with the smart CDSS and to examine the barriers and demands for smart CDSS use, by using the Human Centred Design methodology.
1.1 Background
At the end of 2019, TNO started an initiative to build a guideline-based CDSS, to overcome the identified problems with the PCH guidelines (‘Development of uniform and evidence-based work by PCH through integration of Smart Guideline Module and Digital Dossier (DD) JGZ’). The goal of the smart CDSS is to contribute to a more uniform and scientifically substantiated way of work-ing, with sufficient room to act according to the professional’s perception. Furthermore, the smart CDSS aims to contribute to greater flexibility for daily practice within the PCH. As part of the project, a needs assessment was performed in July 2019 by 13 PCH professionals to discuss the implementation of the guideline in daily practice. Several mock-ups and story boards were used during the needs assessment to show the potentials of the smart CDSS. All professionals expected that the smart CDSS can contribute to work more efficiently and effectively. Furthermore, 12 par-ticipants expected that the smart CDSS can help the professional to provide feedback on the guide-lines, and thereby the smart CDSS supports improving the guidelines. With the introduction of the smart CDSS, some requirements are essential according to the 13 professionals: the professional should still act according professional perception; the professional knowledge is essential (e.g. identifying risks, asking correct questions); and it should not cause over-registration and unclarity about what steps the systems has taken.
Before the start of the development of the smart CDSS, professionals performed a needs assessment prioritizing which PCH guideline needed to be translated into a ‘smart’ guideline first. The most suitable guideline to include first was the guideline Length Growth. Therefore this study will focus exclusively on the experiences with the paper-based and computerized smart guideline Length Growth (2019) (43).
1.2 Setting
Each region in the Netherlands (28 regions in total) includes a PCH organization served by a Mu-nicipal Health Services (Dutch title: GGD) and/or a health institution. Each PCH organization in-cludes specialized community physicians, nurses and doctor’s assistants (the PCH professionals). The PCH physicians and nurses determine whether a child needs to be further examined or referred to a specialist. In this thesis, two PCH organizations were recruited to evaluate the smart CDSS in a pilot setting.
1.3 The PCH guidelines
Following the request of the Dutch Ministry of Health, Welfare and Sport in 1996, evidence based guidelines are developed for the PCH to assure the quality of care (13). The PCH guidelines are being requested and financed by the ZonMw (programs ‘Guidelines PCH’, 2013 – 2018 and 2019-2024, Dutch title: ‘Richtlijnen JGZ’). Guideline developers are responsible of the development and the quality of the requested PCH guidelines. A guideline developer is accountable to the ZonMw. Since 2010 the Dutch Centre of Child Health (NCJ) is responsible for managing the guide-line development, implementation and evaluation. TNO is one of the organizations who develops, implements and evaluates of the PCH guidelines by request, and to date contributes to more than 20 PCH guidelines. Currently, there are 33 published guidelines, and two guidelines are in testing phase (44). The guidelines are based on evidence-based literature in specific themes. These themes consist of information about e.g. background, signalling, monitoring, conducting anamnesis, refer-ral criteria, and research of the Dutch children (e.g. charts or prevalence). Each guideline includes a ‘Basis dataset’ (Basic dataset, BDS) protocol. The BDS is a standard for the PCH EHR, which defines what data needs to be registered in what format. The goal of this BDS is to provide better
12
registration for better care (45). Based on the key recommendations of a PCH guideline, a BDS protocol provides per guideline what child data needs to be stored in what BDS element.
1.4 The system – smart CDSS
The SRM, as proposed and evaluated in the need assessment, is a mid-level CDSS, which implies that it executes guidelines rules or algorithms to compute decisions about possible diagnoses or interventions (46). The SRM, further referred as ‘smart CDSS’, consists of two parts i.e., a con-figurator and a server, which are discussed below.
1.4.1 The configurator
The smart CDSS contains a configurator, which is developed for the guideline developers to trans-late the paper-based guideline into a computerized smart guideline. The smart guideline is struc-tured by a decision tree structure, including IF-THEN rules (see Figure 1). Each decision tree con-sists of one or more of the following blocks: a data request block (e.g. what is the height of the mother?), information block (e.g. showing the data), R-script blocks (for data analysis and algo-rithms, e.g. calculating the target height of the child), reference blocks (e.g. to navigate to another guideline), and recommendation blocks (containing the recommended action for the child, e.g. ‘no referral’). To use a smart guideline, data is requested from the development and growth of the child and specific (family) situation(s) (e.g. the presence of disorders in the family). The PCH profes-sionals store this data in the PCH EHR.
1.4.2 Interaction with the CDSS server and external applications
After publication of a smart guideline in the configurator, the guideline can be used by external applications, such as the PCH EHR or parental applications. In this study the PCH EHR will be used only as the external application. The communication with the smart CDSS and the external application is based on a data request-response mechanism. This communication is built on a Ja-vaScript Object Notation (JSON) format. This JSON format contains the data structured by the BDS standard. Starting with a data request by the PCH EHR, the configurator will receive the patient data and a specification of the requested guideline. After receiving a data-request, the con-figurator will execute the guideline with the retrieved patient data, by addressing the decision tree. Depending on the completeness of the data, the configurator will send a data-response including a guideline recommendation or a request for additional patient data. This data-response is shown within the EHR system. In conclusion, the smart CDSS consist of the following three components; (1) the knowledge base; in which IF-THEN rules of the guidelines are compiled, (2) the reasoning mechanism; in which the rules of the knowledge base is combined with the received patient data, and (3) the communication mechanism; which presents the output of the system to the user (shown in Figure 1) (47).
Figure 1. The overall flow of the request-response mechanism
13
The interface of the smart CDSS was developed by the PCH EHR software supplier itself. In the current situation, the smart CDSS is visible by clicking on a banner, presented in one specific PCH EHR screen: Growth curves. When clicking on the banner, a screen opens including: the recom-mendation, the used and missing data, and the decision tree of the guideline in PDF format (see example, Figure 2).
Figure 2. An example of the ‘Growth curves’ screen in which the smart CDSS banner is presented
1.5 Human centred design approach
Research shows that poorly designed Health Information Technology (HIT) may negatively influ-ence the patient care, clinical workflow and professional work activities (48). The usability of the CDSS is critical for effective and efficient CDSS use (2). Usability is defined as ‘the effectiveness, efficiency and satisfaction with which specified users achieve specified goals in particular envi-ronments’ (ISO- 9241 definition). Yet, poor usability is one of the leading problems to CDSS adop-tion and to use it in clinical practice (5,7,9).
According to Gong et al. (2016), human factors, usability, and human-computer interaction principles are fundamental to successful CDSS. A method used within Human Factors (HF) is human-centred design (HCD), also known as user-centred design. Using user-centred frameworks, contribute to a CDSS that is more likely to fit within specific workflows and which subsequentially will be used more often (49). A user-centred design approach is essential to find the right balance of minimizing disruption to workflow and user burden (49).
The HCD is a iterative methodology that focuses on the end-users needs and users’ mental processes, -limitation and -preferences, to design a system that meet user requirements (2). The HCD model, shown in Figure 3, is adapted from the ISO 9241-210:2010: Ergonomics of human-system interaction - Part 210: Human-centred design for interactive human-systems. This modified HCD model is adapted especially for HIT systems. The iterative model includes: the analysis and under-standing of the context of use (e.g. by performing observation of the system in daily practice); specification of the users’ requirements; the (re)design solutions; and the evaluation of the usability of the (re)design. The iteration is performed again until all user requirements are fulfilled.
1.6 Research questions and thesis outline
Based on previous research in CDSS, the success of a CDSS could be influenced by several ac-ceptance factors, such as: system usability, user attitude, social- or organizational influences (33). To prevent that the smart CDSS does not experience low acceptance by its end users, finding the potential reasons for low uptake in clinical practice is required. Previous research showed that the HCD approach has potential to improve the CDSS usability (2,8–10,50). Engaging the profession-als in the design process from the earlies stage is essential to support a system that optimally sup-ports clinical workflow (9). However, a study performed in 33 individual clinics (general internal medicine and family medicine clinics), performed by Mann et al. (2019), showed that using an iterative user-centred design, over-designed the iterative developed CDSS. This overdesigned
14
CDSS was less utilized compared to the original developed CDSS (49). Furthermore, solely con-sidering the user needs in a user-centred design of one specific PCH group could result in an over-complex CDSS, which make the CDSS less usable for other specific PCH groups (50). It is there-fore advised to examine the CDSS adoption more broadly by validating how problems with poor CDSS usability affect the adoption of the CDSS in the current setting (49). It is therefore advised to examine the CDSS adoption more broadly by validating how problems with poor CDSS usability affect the adoption of the CDSS in the current setting (49). This study combines the HCD approach with validating CDSS acceptance factors by not only performing an HCD cycle, but also continu-ously checking CDSS acceptance factors per HCD step. This approach aims to minimize the risk of delivering a smart CDSS that will not be used in daily practice. Therefore, the overall aim of this study was to assess whether a Human Centred Design implementation approach contributes to the acceptance of smart CDSS within the PCH.
Figure 3 shows the overview of the HCD approach which was used to structure this thesis. This thesis has been divided into six chapters. In Chapter 2 the theoretical background is clarified, sum-marizing the theories and literatureused to set up the CDSS acceptance criteria tested within each HCD step. Chapters 3 through 5 follow the steps of a single iteration of the HCD model (Figure 5). Within these chapters, an HCD step is complemented with a theory, instrument or literature, to provide a more in-depth view. This thesis includes one iteration of the HCD methodology, step A to E. Future evaluation of the smart CDSS interface is already honoured by a new ZonMw project. For this reason, this thesis will not focus on the HCD steps: Designed solution meets user require-ments, and long-term monitoring (Step F & G, Figure 3). Step A, Plan the HCD process, was initialized prior to the execution of the HCD iteration by three researchers (HW, OBH, LDP). The used research questions per HCD step are described below.
Figure 3. The Human Centered Design approach including the used steps for this thesis, adapted from (48) Chapter 3: Understand and specify the context of use
Q1. In what context is the smart CDSS used by the PCH professionals during implementation in the clinical workflow?
Q2. How do PCH professionals evaluate the use of and experience with the smart CDSS com-pared to the implemented paper-based guideline?
The first study addressed the context of use (HCD) of the original designed smart CDSS (version 1.0) (see step B, Figure 3). The MIDI and open-ended questions were used to provide a more in-depth insight in the usage of new supplementary smart CDSS during the pilot (Q1) and to compare the results of use and experience with the currently implemented paper-based guideline in the PCH (Q2). Observations and interviews during the pilot enabled a more detailed context of use of the smart CDSS which was redesigned (Q4).
15
Chapter 4: Specify users’ requirements
Q3. How do PCH professionals evaluate the use of the smart CDSS, Version 1.0, in terms of usa-bility, barriers, drivers and needs?
The second study, Chapter 4, studied the usability, barriers, drivers and newly identified needs of successful CDSS implementation and acceptance. Altogether, those barriers, drivers and needs summarized the users’ requirements (HCD) for the originally designed SRM (Version 1.0). It pro-vides insight in whether the originally designed system meets the user needs and expectations. To test the users’ requirements, predefined acceptance factors for CDSS acceptance and implementa-tion were examined by using UTAUT and MIDI determinants, and CDSS success factors known from previous research.
Chapter 5: Produce design solutions to meet user requirements & Evaluate designs against requirements
Q4. How can a redesigned user interface (Version 2.0) contribute to the implementation of the smart CDSS and the implementation of PCH guidelines in the PCH?
Q5. Is the redesigned user interface of the smart CDSS (Version 2.0) more usable than the origi-nally designed smart CDSS (Version 1.0) and what are recommendations for further develop-ment and impledevelop-mentation?
If the context of use and users’ requirements are described, the following step in HCD is to make (re)design solutions. In chapter 5 the redesign of the smart CDSS (Version 2.0) is described by presenting the developed computer-based mock-ups. It is based on the barriers and needs of chapter 4 and incorporates new possible functionalities to include shared decision making and machine learning in the future. Furthermore, design principles and heuristics were used to ensure that the system supports an adequate amount of interaction between the user and the computer.
In addition to the presentation of the redesigns, chapter 5 includes the evaluation of the mock-ups. An evaluation was performed to verify if the redesign did indeed take into account the user needs and improved the CDSS’ usability (Q5). The results of this evaluation provide infor-mation for a next HCD iteration in the future.
1.7 Study design
This study was performed from January 2020 until June 2020. The participants were examined over time; a baseline measurement prior to the introduction of the smart CDSS (T0), a one-month experience with the smart CDSS (T1), and a two-month experience with the smart CDSS (T2) (see Figure 4).
Figure 4. An overview of the measurements used in the study period T: moment in time
This study used a mixed-method approach. Questionnaires, interviews and observations were used to understand the context of the paper-based guideline, currently implemented and used in the PCH, and the use of computerized smart guideline during the pilot (Chapter 3). The same methods were used to identify the barriers, drivers, and needs of the originally designed smart CDSS (design 1.0) (Chapter 4). The results of Chapter 3 and 4 were incorporated with the redesign of the smart CDSS (design 2.0). This redesign was examined in a one-week semi-structured online focus group, to test whether the changes met the user requirements (T3, Figure 4). In Figure 5, the overall structure of the study design is presented.
16
Figure 5. A flowchart of the thesis’ structure and study design HCD: Human Centred Design, Q: research question, T: moment in time
1.8 COVID-19
In 2020, COVID-19 affected the way healthcare was organized within the Netherlands (including the PCH), and thereby also influenced this study. In March 2020, it became clear that most of the regular consultations of the PCH were not performed and therefore the end users were not able to use the new CDSS as intended. This affected the experience with the CDSS and the results of this study. In the study period, one pilot group was almost at the end of the 2nd month pilot period. The other pilot group was at the starting phase of the pilot.
17
Chapter 2
Theoretical background
The HCD approach was used to enable a more usable CDSS. In combination with this approach, CDSS acceptance factors were tested to provide insight what significant role acceptance factors play role in each HCD step. The set of used (CDSS) acceptance factors contained factors from the UTAUT, MIDI, and literature in CDSS acceptance. By validating this set of acceptance factors to the use of smart CDSS, it enables to identify new user’s needs which were not addressed by the HCD steps ‘context of use’ and ‘users’ requirements’. The theoretical background of the literature used in the context of this study are described below.
2.1 Measurement Instrument for Determinants of Innovations
In 2004, Fleuren et al. published a framework to introduce and evaluate innovation in healthcare and education (51). Potentially desirable change may not occur in one of four main stages in inno-vation processes: dissemination, adoption, implementation, and continuation. The transition from one stage to the other can be influenced by several determinants, categorized in four groups: deter-minants associated with (1) the innovation itself, (2) the adopting user, (3) the organization, and (4) the socio-political context. In 2014 the MIDI was developed to improve the understanding of the determinants of innovations that may affect its implementation and to better align with the innovation strategy (52). The instrument, containing 29 determinants, is based on pooled analysis of eight studies and expert consultations. All eight studies concerned the implementation of an innovation within PCH, primary and secondary schools.
The MIDI was used in this study, mainly because of successful implementation in the PCH. Moreover, the MIDI is often used in studies examining the implementation of evidence-based in-novations such as PCH guidelines. Chapter 3, which concerned the HCD step context of use, was extended with the validation of several MIDI determinants, potentially influencing the acceptance of both the paper-based guideline and the smart CDSS. Implementation determinants experienced by the participants provided a more in-depth view about the context of use. For example: a deter-minant associated with the organization, time availability, provides context about possible time limitations and whether the organizations enable time to use the smart CDSS during clinical prac-tice. The MIDI was also used in Chapter 4 to gain insight in the identified users’ requirements of the smart CDSS implementation in the PCH. These insights provide an understanding on the user needs to be taken along at Chapter 5. The total of items used from the MIDI were 12 of the total 29 determinants, covering the determinants associated with the innovation, the adopting user, and the organization. Socio-political context does not often result in differentiation and is more likely to be a relevant factor in international studies. The smart CDSS was implemented within only two PCH organizations. Therefore, socio-political context was excluded from the list of determinants.
2.2 Unified Theory of Acceptance and Use of Technology
Until today, numerous theories are developed with roots in information systems, psychology and sociology that typically explain over 40% of the variance in individual intention to use a technology (35). In 2003 Venkatesh acknowledged that there was a need for a review and synthesis in order to progress toward a unified view of user acceptance, rather than using a “pick and choose” or “fa-voured model” approach. Therefore, Venkatesh designed and validated the UTAUT (35). The UTAUT is based on eight models: TAM/TAM2, Diffusion of Innovation, Social Cognitive Theory, Theory of Reasoned Action, Motivation Model, Theory of Planned Behavior, and Model of Per-sonal Computer use. This theory states that four constructs play a significant role as direct deter-minants of user acceptance and usage behaviour; performance expectancy (PE), effort expectancy (EE), social influence (SI), and facilitating conditions (FC). Each determinant can be influenced by one or more of the key moderators: gender, age, experience and voluntariness of use.
18
Figure 6 displays which moderator can influence which construct. The intention to use a technology is determined by the three constructs; PE, EE, and SI. Subsequently, the intention of use (= behav-ioural intentions) influences the actual use of the technology (=use behaviour).
Figure 6. The Unified Theory of Acceptance and Use of Technology, reprinted from (35)
The included UTAUT determinants, were used to validate which acceptance factors play a signif-icant role during specification of the fourth HCD step ‘users’ requirements’. For example, by ask-ing the participants about the compatibility of the system (=FC), provides information about if the system should improve its integration with other systems. All four constructs of the UTAUT: PE, EE, SI, and FC, as well as behavioural intention were included in this study. The key moderators: age, experience, and voluntariness of use were included, but key moderator gender was excluded because of the absence of men within each pilot group. A total of ten questions were included covering the UTAUT constructs, key moderators and behavioural intention (Appendix A).
2.3 Barriers known for CDSS success
In addition to considered determinants for innovation acceptance and implementation, covered by the UTAUT and MIDI, a small- scale literate search was performed. This literature search gained insight in determinants influencing the use of CDSS in specific. Factors identified from the litera-ture, were used to validate which CDSS acceptance factors play a significant role during the spec-ification of fourth HCD step ‘users’ requirements’ (Chapter 4). Furthermore, the CDSS acceptance factors were measured over time, to check for differences in acceptance factors when users have greater experience with the smart CDSS.
Within the scope of this study, the literature search resulted in three articles. The first sys-tematic review by Kilsdonk et al. (2017) covered the factors influencing the implementation suc-cess of guidelines based CDSS (24). The authors mapped 421 factors from 25 publications into the Human Organization and Technology-fit framework. The second systematic review by Khong et al. (2015) addressed the important concepts leading to adoption to a CDSS by healthcare providers (4). Khong included 16 articles in which 15 theoretical or conceptual models were identified. From these articles, concepts that significantly affected the behaviour of intention or actual use were selected and categorized in 9 major concepts. The final review by Deveraj et al. (2014) searched for barriers and facilitators of CDSS adoption which then were categorized according the UTAUT model (38). Deveraj et al. reviewed 26 publications which resulted in 35 unique barriers and 25 facilitators as important determinants of CDSS adoption in clinical practice. Thereafter, Deveraj et al. categorized these determinants under the four constructs of the UTAUT (38).
In total the literature review included a review on 117 determinants. Nine of the 117 deter-minants were excluded from the list, because they were only noted once in one of the three articles. The set of determinants were categorized by corresponding theme and characteristics (see Table 1). This resulted in 20 categories, grouped by: user-, system-, supporting- and clinical practice components. User components includes determinants associated with: autonomy, users’ character-istics, motivation, physician-patient relationship, skills, trust and social barriers. System compo-nents includes: information completeness, information format, efficiency, compatibility, complex-ity and ease of use. The supporting components consists of: training, resources & support and sys-tem evaluation. Clinical practice components includes: relevance, time, clinical process and timing.
19
Table 1. An overview of the included categorized determinants originating from three articles (4,24,38). +:facilitator, - : barrier, CPOE: Computerized physician order entry, EMR: Electronic Medical Record
Category Kilsdonk Khong Deveraj
Physician – patient rela-tionship
− Belief that CDSS use hampers the physician-patient relationship − Belief that CDSS might disturb the patient
Relationships- patient users’ relationship, interprofessional re-lationship
- Reluctance to use system in front of patients
Users’ characteristics − Belief that CDSS is an intrusion to clinical practice User’s attitude - agreement with CDSS Users optimism
Users inertia to change Users changing behaviour
- Lack of agreements with the system
- User or physician’s attitude towards the system + Positive user attitude / - Prior bad experience + Good prior experience using a CDSS Skills − Unskilled and lack of experience in computer use
− Lack of familiarity with the new system or similar systems
+ Inexperienced (or young) staff are likely to benefit most of system implementa-tion
User’s efficacy - Computer experience or computer skill User’s efficacy – familiarity with the CDSS or with patients’ conditions
User’s efficacy – confidence
User's demographic profile - age, qualifications
- Lack of knowledge of a system or its content - Poor computer skills
+ Computer literacy of younger generation and good computer skills
+ Enhancing user/provider knowledge Motivation − Physicians consider themselves experts not needing decision support
− Presence of obvious medical indication
− No system benefit because of uncertain patient cases/patient complexity
User's motivation – extrinsic / intrinsic Perceived benefits - general/patient/user User’s intention
- Lack of motivation or incentives
Training + Users receive initial hands-on training + Users receive concise and tailored education
Education or training or knowledge + Good training / - Lack of training
Time −Time constraint to use CDSS during a patient consultation Duty time - Lack of time or time constraints. Clinical process + System is fitted to routine care Disruption to workflow - Difficulty of competing clinical demands
+ Integration of the CDSS into the workflow Autonomy Physicians are often worried about obscured responsibilities and loss of their own
clinical autonomy
Sense of being in control User's responsibility Relevance −Recommendations lack a holistic perspective of the patient
+ Recommendations are relevant for the clinical situation at hand
Relevance – fitness of task Relevance - importance of task
+ Providing or collecting relevant information for user and patient + Applicability to practice
Information completeness + System provides explanation for a recommendation System performance - Output quality
System performance - Capabilities to handle complex condi-tions
- Less authenticity and reliability of system information + Reliability of data and information
Information format + System design is visually oriented
+ Recommendations are clearly visible on the screen (positioning)
System performance - System output + Good information presentation
Timing + Recommendations are available at the point of decision making - Obscure workflow issues
Trust Physicians’ trust in the knowledge-base of a guideline-based CDSS Trust in the system Credibility of system Efficiency: − The time required to use the system is too long
+ User interactions are minimized
- Loss of productivity / + improvement of productivity - Too many unwanted alerts
+ Fast information retrieval and transfer Compatibility: + CDSS is integrated with CPOE or EMR System performance - Compatibility - Less interoperability and standards
+ Integration of the CDSS into the existing computer network sys-tems
Ease of use: + System is intuitive + System is user-friendly
System Usability - Ease of use System performance - Usefulness
- Less user-friendly (3) /+ User-friendliness + Ease of finding information within the CDSS
Complexity: Complex guidelines - Complexity of a system / + Reducing complexity
System Evaluation: System evaluation need + Usability testing
Social barriers Social pressure
Peers/colleagues’ leadership
- Social barriers and lack of social acceptance
Resources & support Resource availability: computer, facilities limitation, structure Implementation support from technical staff
+ Discrete accessibility of resources + Proper documentation of procedures
20
Chapter 3
The use and implementation of the smart
CDSS compared to the paper-based guideline
21
3.1 Introduction
Since the introduction of the PCH guidelines, it had been demonstrated that the feasibility and implementation of the guidelines is still not guaranteed. To overcome the problems of using the guidelines in clinical practice, a CDSS can offer opportunities (2,18–20). CDSS can improve ad-herence, reduce medical errors, can improve providers’ performance. (23–28). However, to suc-ceed, some aspects should be considered such as; technical issues and -limitations, inadequate tech-nology integration within workflow, and poor user-interface (37). One of the leading factors influ-encing the success of the CDSS is the fit within the professionals’ workflow (4,6,21,24,29,37,38). The CDSS should contribute to a more efficient workflow instead disrupting the workflow itself.
The CDSS will be used within a certain range of technical, social, organizational conditions that may affects the use of the system. The quality of the system, depends on having good under-standing of the context of use (40). One of the responses of the needs assessment, performed in July 2019, was the urge to not only focus on the technical aspects of the smart CDSS during the implementation but to look on how to integrate the smart CDSS with the conversations between PCH professionals and their clients. The acceptance of innovations, such as the PCH guideline and the smart CDSS, could be influenced by a set of critical acceptance factors known from previous performed research in innovation implementation (35,52). Understanding the acceptance factors help to target the innovation strategy better (52). Besides the examination of the context of use by user, testing the acceptance factors could provide an additional view to other acceptance factors that were not addressed by the user but are critical for the success of the smart CDSS. Therefore, acceptance factors experienced with the paper-based guideline and the smart CDSS were tested by a set of potential acceptance factors from the MIDI, discussed in Chapter 2. The overall objective of this chapter was (1) to understand in what context the smart CDSS was used by the PCH pro-fessionals during pilot period in the clinical workflow and (2) to evaluate the use of and the expe-rience with the smart CDSS compared to the implemented paper-based guideline.
3.2 Methods
The experiences with and the use of the paper-based guideline were compared to the experiences with the smart CDSS. This was measured by quasi-experimental study design. The context of use of the paper-based guideline was measured by the questionnaires only. Semi-structured interviews and observations were performed to provide an in-depth view on the context of use of the smart CDSS. Questionnaires were used to test the implementation acceptance factors for both guideline formats. The usage of CDSS was logged to provide an overall view about how often the users actually (could have) used the smart CDSS.
3.2.1 Participants
This study was performed amongst two pilot groups. Participants of the first pilot group were re-cruited from the PCH organization ‘Veiligheids- en Gezondheidsregio Gelderland-Midden’ (VGGM) in the east of The Netherlands. The second pilot group was recruited from the PCH or-ganization ‘Jeugdgezondheidszorg Zuid-Holland West’ (JGZ ZHW), in the west of The Nether-lands. The participants were recruited by PCH managers of the two PCH organizations. Prior to the study, the participants of both PCH organization were trained in a one to two-hour session. Informed consent was conducted prior to the training session via an online questionnaire or during this training session via a paper-based questionnaire. Due to the coronavirus, the second group, was not questioned and observed during the pilot period.
3.2.2 Measures
Firstly, the experience with the currently implemented paper-based guideline was measured at baseline. The experiences with the complemented smart CDSS, was measured during the pilot and the at the end of the pilot. Experience was covered by several variables; self-reported skills, appli-cation, complexity, user benefits, ownership, client satisfaction, and social influence. The variables are based on both MIDI determinants and on the users’ requirements initialized in the needs as-sessment prior to this study. The definition of each variable is summarized in Table 2.
22
Secondly, the self-reported awareness of the guideline was measured pre and post the introduction of the smart CDSS, defined as: ‘the degree of knowing the PCH guideline existence’ (13). Self-reported awareness was categorized in the following categories; ‘I am not familiar with the guide-line’, ‘I am aware of the guideline, but I have not read it’, ‘I read (a part of) the guideline once but never again’, ‘I have read (parts of the) guideline more than once’.
Thirdly, the use of the smart CDSS was recorded by the PCH EHR software supplier. The recorded use is described as: ‘the user clicking on the banner in the health record, to see the pro-vided recommendation’.
Table 2. Descriptives of measures used to examine experience. The two right colums show the used questions for each measure in Appendix A. LG: paper-based guideline Length Growth
Variable Description Nr. of linked question in Ap-pendix A LG Smart CDSS Qualitive data
Application Describing the deployment of the guideline in practice, covering the when and why the guide-line/smart CDSS was used and whether additional resources for this guideline were used
2 & 4 41 & 43
Consultation impact
The influence of the use of the guideline/smart CDSS on the conversation between physician and client during the consultation.
3 42 Quantitative data
Complexity Defined as the complexity to apply the items from the guideline in practice. 8 x
Client satisfac-tion
The degree in which the professional thinks that the client is satisfied with the use of the guide-line/smart CDSS by the professional.
10 31
Ownership The degree in which the users can act according his/her professional perception by using the guideline/smart CDSS
5 26
Self-reported skills
The degree in which the professional thinks he/she has sufficient skills to be able to use the guideline/smart CDSS
7 17
Social influence The degree in which the organization enables time to make use of the guideline/smart CDSS 9 20
User benefits The perceived complements of the provided recommendations of the guideline/smart CDSS 6 25
3.2.3 Data collection
Questionnaires
This study performed the analysis at three moments in time (T0,T1,T2). At baseline, the partici-pants were using the implemented paper-based guideline solely, whereas during the pilot they used the smart CDSS as add-on to the paper-based guideline. Therefore, the questionnaire at baseline (T0) included questions on the experience and implementation regarding the paper-based guideline Length Growth, and during the pilot (at T1 and T2) regarding the smart CDSS, by using the name ‘SRM’ instead of the ‘guideline Length Growth’. An overview of the used questions can be found in Appendix A. The T0 questionnaire also consisted of questions regarding the demographical in-formation, including: experience in the PCH, PCH organization, job title, and the number of work-ing hours per week. The self-reported awareness was measured with the T0 and T2 questionnaires (question 1, Appendix A). The quantitative data identified in Table 2 was measured by using a 7-point Likert scale, ranging from 1 (strongly disagree) to 7 (strongly agree). The T0 questionnaire was administered on paper and online by using the Survalyzer software. The T1 and T2 question-naires were administered online by using the Survalyzer software program only.
Electronic health record
The EHR logged the use of the smart CDSS by each participant. This resulted in the following measures per PCH organization per day: the total of banners containing no recommendation due to missing data, the total of banners containing a recommendation, the total number of clicks on the banners. This data was recorded anonymously and was only traceable to the PCH organization.
Interviews
During the pilot, semi-structured interviews were performed to discuss the first experiences with the smart CDSS. Open-ended questions were composed by the observer in advance. The semi-structured interview took place face-to-face after the observations or via the telephone if the par-ticipant was not observed during clinical practice. The answers to the questions were noted down
23
on site. A summary of the interview was made afterwards. The interviews in pilot group 1 took place from 6 February 2020 until 10 March (2 - 7 weeks of experience with the smart CDSS).
Observations
During the pilot period, observations were conducted to perceive the context of smart CDSS use during clinical practice. The participants of pilot group 1 were observed from 6 February 2020 until 26 February (2 - 4 weeks of experience with the smart CDSS). A request to attend the consultation was sent to the participants by mail. In advance to the observations, an observation form was made to identify the subjects and items that needed to be observed. With the use of this observation form, the researcher of this study (HW) observed and reported on site, by writing on the form, what procedures took place during a consultation in which growth length was measured. The observer sat next to or behind the PCH professional, with a sufficient distance to see the computer screen containing the interface of the PCH EHR. Before each consultation, consent was asked from the parent(s) or child, allowing the researcher to attend the consultation. A summary of the observa-tions per participant was made afterwards.
3.2.4 Statistical analysis
Descriptive scores (mean, median, SD) were calculated for the demographical information. To use the same scale for all 7-Likert scales, the ratings for the guideline complexity (negative phrased Likert scales) were transformed into the same scale for the positive phrased Likert scales: e.g. rate 1 (strongly disagree) was transformed in rate 7 (strongly agree).
Analysis of quantitative variables, regarding experiences with the guideline Length Growth, only included the participants who perform consultation. This resulted in the exclusion of two PCH assistants. According to Chernick et al. (2008) a sample size of < 10 does not represent the population. In a too small sample size, use of repeated sampling is incorrect (53). The scores in this group are most likely not be equally distributed as in the overall population. Therefore, boot-strap was performed only for the T0 questionnaire (n=17). Correlations between the T0 variables were tested, using the Spearman’s correlation with a bootstrap of 1000 samples with a 95% confi-dence interval. Correlations with a conficonfi-dence interval (CI) containing zero were excluded. For the mean rating of the 7-Likert scales at T0, the CI for the mean was calculated through a 1000 sample bootstrap (95% CI). The overall mean complexity of the guideline, asked by using four 7-Likert scale questions, was only calculated if the Cronbach’s alpha was > 0.7.
Other statistical hypothesis, such as significance between T0, and T2, could not be per-formed because of the small sample size of T1 and T2 (n=7). Therefore, the differences between PCH organizations, and differences between T0, T1, and T2 were compared by descriptions. The confidence intervals for the mean Likert score and figures were constructed in R version 3.5.1, (RStudio version 1.2.5033). All other analysis was performed in SPSS version 26.
3.3 Results
3.3.1 Overall participant inclusion
A total of 19 participants were included at baseline (T0). Of those, six participated in the observa-tions (group 1), seven were surveyed (group 1) and 11 where interviewed (group 1 & 2) at T1. The researcher attended a total of 28 consultations for children ranging from 0-4 years old and 12+ years old. All 19 participants were logged during the pilot period. Eight of the 19 participants from pilot group 1 were surveyed at T2. The PCH assistants that did not perform consultations and could not answer question regarding the guideline, were excluded for the quantitative measures of all questionnaires. This resulted in the exclusion of two assistants at T0 (n=17) and one assistant at T1/T2 (see a, in Figure 7). One participant did not respond to the T1 questionnaire (see b, in Figure
7) and one participant answered the T2 questionnaire partly. However, the assistant of pilot group 1 was included for open-ended questions of T1 and T2, which provided information about the context of use of smart CDSS in daily practice.
24
Figure 7. Included participants per method and per moment in time (T0/T1/T2)
3.3.2 Participant characteristics
The 19 participants that were included for baseline and CDSS logging had a mean age of 43 years (SD=11). The participants worked for a mean of 9.1 year, with a median of 24 hours per week (SD=5). In this group of seven were physician, eight were nurse, three were assistant and one was resident. One of the three PCH assistants conducted consultations at high schools. The character-istics per pilot group and the charactercharacter-istics per group used per method are presented in Table 3. Table 3. The participants’ characteristics per group included for each method.
SD: standard deviation a: one assistant conducts consultations
T0 T1 T2 Group 1 (n=8) Group 2 (n=11) Survey Guideline awareness & use (n=17) Survey (n=7) Inter-view (n=11) Obser-vations (n=6) Survey Guideline awareness (n=6) Survey CDSS use (n=8) Mean age (SD) 40 (10) 45 (11) 41 (9.8) 38 (9.0) 39 (9.7) 37 (9.8) 41 (8.9) 40 (9.8) Discipline Physician Nurse Assistant Resident 3 3 2 0 4 5 1 1 7 8 1a 1 2 3 2a 0 4 4 2a 1 1 3 2a 0 3 2 1 0 3 3 2 0 Mean years of experience (SD) 9.3 (6.4) 9.6 (10.9) 9.2 (9.2) 7.7 (5.1) 8 (6.6) 7.3 (5.5) 11.5 (5.7) 9.3 (6.4) Median working
hours per week (SD)
26 (6) 24 (5) 24 (5.5) 28 (6.6) 27 (6.7) 28.4 (7.2)
26 (5.9) 28 (6.2)
3.3.3 Use of the paper-based guideline at baseline
Self-reported guideline awareness
At baseline 37% of the participants read (parts of) the guideline more than once, 42% of the par-ticipant read (parts of) the guideline once but never again, 11% were aware of the guideline but did not read it, and 11% was not familiar with the guideline. The participants that were not familiar with the guidelines were PCH assistants, who did not perform consultations (see Figure 8). There-fore, the assistant are excluded for all other questions regarding guideline use. The majority of pilot group 1 read the guideline more than once and the majority of pilot group 2 read the guideline once (see Figure 8).
