Training and support for medical researchers in the usage of Electronic Data Capture tools

(1)

Training and support for medical researchers

in the usage of Electronic Data Capture tools.

Master’s thesis by Jolien Chi Chin Lam

Supervised by

D.L. Arts, MD, PhD

R. Cornet, PhD

(2)

(3)

Training and support for medical researchers in the usage of

Electronic Data Capture tools.

Master’s thesis

Master Medical Informatics

University of Amsterdam

November 2017 – July 2018

Student Mentor

Jolien Chi Chin Lam D.L. Arts, MD, PhD

Student number 10327266 CEO

jccl@live.nl Castor

derk@castoredc.com

Location of Scientific Research Project Tutor

Castor R. Cornet, PhD

Paasheuvelweg 25 Associate Professor

1105 BP Amsterdam Department of Clinical Informatics

Amsterdam UMC r.cornet@amc.uva.nl

(4)

(5)

Summary

Background: Health information technology, such as electronic health records (EHR) and electronic data

capture (EDC) tools, promises to improve efficiency, effectiveness, safety and quality of medical care and research. However, a lack of training and support for the usage of these information systems can be a barrier for healthcare organizations to adopt these systems and to use them effectively, efficiently, and securely. Medical researchers indicate that they lack time or experience to learn how they can use new IT efficiently on their own. Therefore, the current study aims to gain insight into the needs and preferences of medical researchers and professionals for training and support in the usage of data management solutions, such as EDC tools and EHRs. Besides, the study aims to identify user groups from user event logs of an EDC tool. This can be used for tailoring support to these users.

Methods: This thesis consists of three parts. First, a systematic literature review was performed to get

insight into the state of the art of training and support for the usage of electronic health records. In the second part, semi-structured interviews were performed with medical researchers to get insight into their preferences and needs for training and support. In the third part, a data analysis was performed with user event logs of an EDC tool in order to get insight into how users use an EDC tool, to identify different user groups based on what functionalities they use within the EDC tool, and to visualize their user flows.

Results: Though previous studies suggested that training and support were essential for the adoption and

implementation of information systems in healthcare, the number of studies that had investigated this topic was small. The results from the literature review and interviews suggested that training or e-learning were preferred, adapted to the way medical researchers worked in practice. For the use of EHRs, formal or informal personal support were preferred. Among EDC users, e-learning and digital support were preferred, followed by personal consults at the start of a study. The dataset for the user event logs analysis consisted of 124,284 sessions of 4,739 users that occurred over the period of 1 October 2017 to 25 April 2018. The results of the user event logs analysis indicated that there were different user groups identifiable from these data, such as data managers and data entry personnel. User flows were visualized for these groups. Furthermore, events that occurred most frequently before users consulted impersonal support materials, e.g. a manual, were related to building case report forms, management of patient records, and data entry.

Conclusion: It is recommended that training or blended learning and formal face-to-face support are

provided during the implementation of EHRs. After the implementation, EHR users should receive support from peers or key users, or formal, remote support via intranet, email, or telephone. For EDC users, e-learning should be provided as well as personal consults at the start of a study. For future research, it is recommended to investigate this further with quantitative research. The user event logs analysis shows that user group identification is possible, but research into this topic should be expanded, e.g. by comparing user behavior between groups and over time, as well as by using clustering to automatically identify user groups.

(8)

(9)

Samenvatting

Achtergrond: Informatietechnologie in de zorg beoogt efficiëntie, effectiviteit, veiligheid en kwaliteit van

medische zorg en onderzoek te verbeteren. Voorbeelden van informatietechnologieën zijn elektronische patiëntendossiers (EPDs) of “electronic data capture” (EDC) tools. Een gebrek aan training en ondersteuning voor het gebruik van dit soort informatiesystemen kan voor gezondheidsorganisaties een barrière zijn om deze systemen te implementeren en effectief, efficiënt en veilig te gebruiken. Medische onderzoekers geven aan dat zij een gebrek aan tijd of ervaring hebben om zelf te leren hoe zij nieuwe informatietechnologieën kunnen gebruiken. Daarom is het doel van dit onderzoek om inzicht te verkrijgen in de behoeften en voorkeuren van medische onderzoekers en professionals met betrekking tot training en ondersteuning voor het gebruik van “data management solutions”, zoals EDC tools en EPDs. Daarnaast is het doel om gebruikersgroepen te identificeren uit “user event logs” van een EDC tool, die gebruikt kunnen worden om ondersteuning aan te passen aan deze gebruikers.

Methoden: Deze thesis bestaat uit drie delen. Ten eerste is een literatuuronderzoek uitgevoerd om inzicht

te verkrijgen in de huidige situatie van training en ondersteuning voor het gebruik van EPDs. Ten tweede zijn semi-gestructureerde interviews uitgevoerd met medisch onderzoekers om inzicht te krijgen in hun voorkeuren en behoeften voor training en ondersteuning. In het derde deel is een data analyse uitgevoerd met “user event logs” van een EDC tool om inzicht te krijgen in het gebruik van een EDC tool, verschillende gebruikersgroepen te identificeren en hun “user flow”te visualiseren.

Resultaten: Hoewel eerder onderzoek aangeeft dat training en ondersteuning essentieel zijn voor de

adoptie en implementatie van informatiesystemen in de zorg, is het aantal onderzoeken dat dit heeft onderzocht klein. De resultaten van het literatuuronderzoek en de interviews geven aan dat training of e-learning de voorkeur hebben, bij voorkeur aangepast aan de manier waarop medische onderzoekers in de praktijk werken. Voor het gebruik van EPDs hadden formele of informele ondersteuning de voorkeur. Onder EDC gebruikers hadden e-learning en digitale ondersteuning de voorkeur, gevolgd door persoonlijke consulten aan het begin van een onderzoek. De dataset voor de user event logs analyse bestaat uit 124,284 sessies van 4,739 gebruikers die plaatsvonden tussen 1 oktober 2017 en 25 april 2018. Verschillende gebruikersgroepen zijn geïdentificeerd, zoals data managers en data-invoer medewerkers. “Events” die het vaakst voorkwamen voordat gebruikers materialen voor ondersteuning raadpleegden, bijv. een handleiding, waren gerelateerd aan onder andere de bouw van “case report forms”, management van patiëntendossiers en data invoer.

Conclusie: Het wordt aangeraden om training of “blended learning” en formele, persoonlijke support te

bieden tijdens de implementatie van EPDs. Na implementatie moeten EPD gebruikers ondersteuning krijgen van collega’s of “key users”, of formele ondersteuning op afstand, zoals via email of telefoon. Voor EDC gebruikers moeten e-learning en persoonlijke consulten aan het begin van een studie aangeboden worden. Het is aanbevolen om dit onderzoek te vervolgen met kwantitatief onderzoek. De “user event logs” analyse laat ook zien dat verder onderzoek nodig is. Dit kan bijvoorbeeld door gedrag van gebruikers te vergelijken met andere gebruikersgroepen, gedrag over de tijd heen te analyseren en “clustering” gebruiken om automatisch gebruikersgroepen te identificeren.

(10)

(11)

Chapter 1. Introduction

1.1 Background

1.1.1 Traditional data capture

Health information technology (IT) promises to improve efficiency, effectiveness, safety and quality of medical care and research (1-3). Consequently, the number of healthcare organizations that have switched from paper-based records to electronic health records (EHRs) has been increasing (3). Medical data collected with such information technologies are primarily used for medical care, but can also be used for medical research (4). Collecting data for medical research is traditionally done with paper-based case report forms (CRFs), which is regarded as a simple and inexpensive way to collect data (5, 6). However, using paper-based records requires entering data into a database manually to create electronic records, so the data can be analyzed (5). This can be time-consuming and error-prone, especially for larger studies (5). Furthermore, paper-based data capture is not suitable for large studies, because the data are not secure, customizable, nor accessible to geographically distributed researchers or study participants (6, 7).

In a study on data management needs of academic biomedical researchers, some researchers report that they use general software tools instead of paper-based CRFs, such as spreadsheets or text files in Microsoft Office (8). These general software tools are used because of their simplicity in user interface, short learning curve, and availability. However, these tools have limitations as they are not developed to address the needs of biomedical research specifically, e.g. storage for large amounts of data and capabilities to ensure data quality (8). Additionally, these tools are reported to be limited in terms of organizational capabilities. Researchers have no standardized method for organizing data, but report that they do whatever suits their individual needs (8). Because of the lack of standardized methods for organizing data, effective data retrieval and reusability of data are limited.

1.1.2 Electronic data capture

Electronic Data Capture (EDC) tools promise to address the problems that occur with paper-based CRFs and general software applications by providing customization, improving the quality of data, and saving time (5, 6, 9). In medical research they are used for collecting, managing, and reporting clinical and laboratory data (10). Unlike paper-based case report forms, EDC tools can be used for large registries and clinical trials (11), and can facilitate remote data collection and accessibility for users that are distributed geographically (6, 7). The quality and completeness of collected data can be improved at the point of data capture with data validation checks, such as range checks or logical rules (7, 12, 13). Furthermore, EDC systems improve efficiency compared to paper-based CRFs (6, 9, 12). With EDC tools data are available without delay after the data capture, which enables researchers to monitor, evaluate, review, and analyze data in real-time (5, 14).

However, researchers have faced several problems with the use of EDC tools (15). Firstly, researchers may not have sufficient financial resources to purchase EDC tools themselves or lack financial support from their organization (10, 15, 16). Secondly, researchers may perceive that they do not have enough time to change their data management practices, or they may not know how to incorporate EDC tools into research (8). Thirdly, they may lack support staff or support materials that can help them with

(12)

using EDC tools, e.g. from their IT department or from vendors (8, 16, 17). The current study focuses on this kind of support among medical researchers, as well as trainings that can help them in using electronic information systems, such as EDC tools.

1.2 Relevance

In a previous study on support, Shachak et al. define support as “any information or activity that is intended to help users solve problems with, and better utilize, the software”(18). The availability of training and technical staff with adequate expertise in IT is considered to be an essential facilitator in the adoption of electronic information systems (17). Conversely, a lack of support can be a barrier for healthcare organizations to adopt information systems and to use these information systems effectively, efficiently, and securely (16, 17, 19). Especially among relatively smaller healthcare organizations that have fewer resources for support, this can lead to lower adoption rates of electronic information systems (20). Along with the trend of increasing size and complexity of datasets, the need for support and training increases (8).

Researchers need this support and training, because they would rather spend time actually performing research, instead of solving issues related to data capture and management (16). Researchers also indicate that they lack time or experience to learn how they can use new IT efficiently on their own, as they perceive that the time required for this is not an integral part of research (8). Furthermore, the perceived quality and trustworthiness of a vendor is crucial for the acceptance of electronic information systems (20). As EDC tools are relatively new in the medical field, medical researchers may fear that vendors will fail to provide proper services, or will go out of business, after which the researchers will not be able to receive support anymore (20). Therefore, research should be performed to investigate researchers’ perceptions and needs for using electronic information systems (10, 15, 21).

1.3 Research objective and questions

The aim of this study is to gain insights into the needs and preferences of medical researchers and medical professionals for training and support of data management solutions. This study will be performed within the context of hospitals. Additionally, this study aims to get insight into how users actually use an EDC tool and how this information can be used to tailor training and support to users.

1. What are the perceptions of medical professionals at hospitals with respect to training and support in the usage of electronic information systems?

2. What requirements do medical researchers at hospitals have with regards to training or support for the usage of digital systems for electronic data capture?

a. To what extent do medical researchers currently use support material? b. What do they think of the current support material?

c. What barriers or obstacles do they perceive when they search for support? d. For what aspects or functionalities do they need support?

e. How can the current training and support materials be improved from researchers’ perspective?

3. How can user groups be identified in usage logs of an EDC tool? 4. How can we use these user groups to tailor support material to users?

(13)

The research questions will be answered by using different methods. Research questions 1 and 2 will be answered by a systematic review and semi-structured interviews. Research questions 3 and 4 will be answered with an analysis of user event logs data. Because of time constraints, this thesis does not answer research question 4 completely, but provides preparations that are needed to be able to answer these questions in future research.

1.4 Outline

This thesis consists of three parts, namely a) a systematic literature review; b) semi-structured interviews with medical researchers, and c) the analysis of user event logs (Figure 1). The thesis starts off the current introduction in chapter 1, followed by a theoretical framework in chapter 2. Then the three parts are discussed in chapter 3, 4, and 5, respectively. In each of these chapters, the methods, results, and discussion of that part are described. Chapter 6 consists of the discussion and comparison of the results of the three parts altogether. In the final chapter, the conclusions of the scientific research project, as well as recommendations for practice and for future research are described.

(14)

Chapter 2. Theoretical framework

2.1 Technology Acceptance Model

The success of both EDC tools and their support is closely related to users’ acceptance and use of the technology. One of the most frequently cited theories about the use of technology is the Technology Acceptance Model (TAM) (22). The theory assumes that the attitude of a user influences behavioral intention, leading the user to either adopt or reject a technology (22, 23). This attitude is influenced by perceived usefulness and perceived ease of use, which are influenced by system design characteristics. Even though the TAM is used frequently, other researchers have criticized the model (24, 25). Shortcomings of TAM include a poor theoretical foundation and the neglect of social processes and cultural aspects of decision making, as the model is perceived from an individual perspective (24, 26).

2.2 Unified Theory of Acceptance and Use of Technology

In an attempt to integrate the main competing user acceptance models, including the TAM, Venkatesh et al. have developed the Unified Theory of Acceptance and Use of Technology (UTAUT) (27). The UTAUT assumes that there are four key constructs, namely performance expectancy, effort expectancy, social influence, and facilitating conditions, which influence behavioral intention and use behavior (Figure 2). The UTAUT assumes that the higher the values of the four constructs, the higher the value of behavioral intention to use the tool (27). Performance expectancy is the degree to which an individual believes that using an electronic information system will help to attain gains in job performance (27). Effort expectancy is the “degree of ease associated with the use of the system” (27). Facilitating conditions are defined as “the degree to which an individual believes that an organizational and technical infrastructure exists to support use of the system”. The model also includes moderators, namely gender, age, experience, and voluntariness of use, which can moderate the relationships between the four key constructs, and behavioral intention and use behavior.

Venkatesh et al. found that the UTAUT was able to account for 70 percent of the variance in intention to use technology. Several empirical studies found supporting evidence, reporting that performance expectancy, effort expectancy, social influence, and facilitating conditions were related to technology acceptance and use (28-30). However, some empirical studies did not find associations between the factors. For example, Liu et al. (31) found that performance expectancy and facilitating conditions were related to technology acceptance and use, but effort expectancy and social influence were not.

This current study mainly focuses on how to optimize facilitating conditions in order to improve actual use. As mentioned previously, empirical studies also recommend implementing training and support in order to improve actual adoption and use (17, 21). As training and support are meant to assist end users in the usage of a system, this study assumes that they can be seen as part of facilitating conditions. However, users can also have a performance expectancy and effort expectancy with regards to these facilitating conditions. If the performance expectancy and effort expectancy of these facilitating conditions are not optimal, it is possible that the facilitating conditions are present, but do not contribute positively to actual use.

(15)

Figure 2. The UTAUT model. From: Venkatesh V, Morris MG, Davis GB, Davis FD. User Acceptance of Information Technology: Toward a Unified View. MIS Quarterly. 2003 Sep;27(3):425-478.

2.3 Diffusion of innovations

2.3.1 The innovation-decision process

The theory of diffusion of innovations may also give insight into the use of training and support (Figure 3) According to this theory, the process of deciding to adopt an innovation or new technology, the innovation-decision process, consists of five stages, namely 1) Knowledge; becoming aware of a new technology, 2) Persuasion; forming a favorable attitude toward the technology, 3) Decision; making the decision whether to adopt the technology or not, 4) Implementation; using the technology, 5) Confirmation; confirming to continue or discontinue using the technology (32). The optimization of training and support for an EDC tool can impact the stages of the innovation-decision process. For example, during the fourth and fifth stages, a researcher may need support to use the tool and decide whether he or she wants to continue using the tool based on the experience with the tool and support. Thus, it is important that support materials positively contribute to the stages of the diffusion of innovations theory. Performance Expectancy Effort Expectancy Social Influence Facilitating Conditions

Gender

Age

Experience Voluntariness _{of Use}

Use Behavior Behavioral

(16)

2.3.2 Attributes of innovations

Besides the stages of the innovation-decision process, there are also attributes of innovations that can influence the rate of adoption, namely the relative advantage, compatibility, complexity, trialability, observability (32). These five variables have been found to explain half of the variance in innovations’ rates of adoption. They can also apply to both EDC tools and, sometimes indirectly, to their training and support. Attributes that are particularly important for support materials are compatibility and complexity. Compatibility means that the training and support are consistent with users’ needs, values, or experiences. In other words, the training and support should fit the users. Complexity concerns users’ perception about the difficulty to understand and make use of training and support. The perceived relative advantage and trialability of EDC tools can be influenced by the corresponding training and support. Relative advantage is the degree to which users perceive that an EDC tool is better than the idea it supersedes, and trialability is the degree to which users can experiment with the EDC tool (32).

Figure 3. The innovation-decision process and attributes of innovations. Adapted from: Rogers EM. Diffusion of innovations. New York: The Free Press; 1983.

1. Knowledge

2.

Persuasion 3. Decision _{Implementation}4. _Confirmation5.

Attributes of the innovation 1. Relative advantage 2. Compatibility 3. Complexity 4. Trialability 5. Observability 1. Adoption 2. Rejection 2. Rejection Continued adoption Later adoption

(17)

Chapter 3. Systematic Literature Review

3.1 Methods

3.1.1 Search

Literature was reviewed with the objective to gain insight into what support and training for electronic information systems was currently given to medical professionals, as well as the effects or outcomes of that support and training. The literature search was conducted in the databases Library Information Science & Technology Abstracts, PubMed, and Scopus. The search terms and queries are shown in Appendix A. The search was limited to articles in Dutch or English published in between 1 January 2000 and 31 December 2017.

3.1.2 Study selection

Firstly, duplicates of articles were removed. Secondly, titles and abstracts of articles were screened on eligibility according to inclusion and exclusion criteria, which led to the first selection of articles. Inclusion criteria were studies in Dutch or English about electronic information systems used by medical professionals or medical researchers, for which the content or format of the training or support had been addressed, explained, and evaluated. Training or support included all measures that were taken in order to help or prepare end-users in the use of electronic information systems. Articles were excluded if the users were neither medical professionals nor medical researchers, if the content of the training or support was not specified or explained, and if only hardware was addressed rather than software. Thirdly, the eligibility of articles included in this selection were assessed based on full texts. This resulted in the second and final selection of studies used in this systematic review. Extracted data were first author, publication year, country, type of electronic healthcare system, study participants, the format or content of support and training, outcomes or evaluations with regards to the support or training.

3.2 Results

In total 252 articles were identified with the literature search. After removing duplicate articles, 234 unique articles were screened based on titles and abstracts. The screening process led to the selection of 74 articles, of which full texts were assessed for eligibility. In total, 14 articles were included in the systematic review. The flow diagram of the selection of articles is shown in Figure 4.

(18)

Figure 4. PRISMA flow diagram of the selection process of articles. Adapted from: Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group (2009). Preferred Reporting Items for Systematic Reviews and Meta Analyses: The PRISMA Statement. PLoS Med 6(7): e1000097.

3.2.1 Study characteristics

The characteristics of the included studies are shown in Table 1. In total 14 studies were included, of which 11 concerned the use of either an EHR (21, 33-40) or a health management information system (41, 42). Two studies concerned a clinical decision support system (43) and a picture archiving communication system(44), respectively. Petersen et al. (45) did not specify the electronic health information system that had been used, only mentioning the use of IT in general. In 6 studies, both training and support were provided, in 5 studies only support was provided, and in 3 studies only training was provided. Additionally, in two studies, impersonal support materials, such as manuals, help menus, and websites, were used to convey instructions for system use, though these materials have not been evaluated, only mentioning that they were generic in nature (21, 36). In all studies, users included physicians. Most studies also included other staff, such as nurses, managers or administrative staff.

Records identified through database searching

(n = 252)

Records after duplicates removed

(n = 234)

Records screened (n = 234)

Full-text articles assessed for eligibility ( n = 74 ) Studies included in qualitative synthesis (n = 14) Identification Screening Eligibility Inclusion

Records excluded, with reasons ( n = 160) - Did not address technical support or training in the study: 52

- Technical assistance was not related to the use of an electronic healthcare information system: 52

- Did not address the use or implementation of an electronic healthcare information system: 34 - Neither content nor effects of technical support or training were explained: 15 - Users were neither medical researchers or healthcare professionals: 4

- No abstract available 3

Full-text articles excluded, with reasons (n = 60) - Content of support/training not described: 43 - No full text available: 10

- Support was not related to using health information systems: 4

- No evaluation on support/training: 2 - Systematic review: 1

(19)

Table 1. Study characteristics

Authors,

reference Year Country

Electronic information system Intervention Training Face-to-face support Remote support Informal support Im-personal support

Baum et al. (41) 2004 Argentina HIS x

Boas et al. (33) 2014 US EHR x

Bredfeldt et al. (34) 2013 US EHR x Crivianu-Gaita et al. (44) 2000 Canada PACS x Crosson et al. (35) 2011 US EHR, e-prescribing application x x x

Goetz et al. (36) 2012 US EHR x x x

Holden et al. (37) 2011 US EHR x x x

Lagebo et al. (42) 2010 Ethiopia HIS x x Petersen et al. (45) 2010 Denmark IT (not specified) x

Ryan et al. (38) 2014 US EHR x

Shachak et al. (21)

2012 Canada EHR x x x x

Struik et al. (39) 2014 NL EHR x x

Tweya et al. (40) 2016 Malawi EHR x

Wright et al. (43) 2014 US CDS x x

CDS = Clinical Decision System, EHR = Electronic Health Record, HIS = Health Information System, PACS = Picture Archiving and Communication System.

3.2.2 Training

Five studies mentioned that instructions on the use of the system were given during the training, though it is possible that they also used hands-on exercises but did not report this (21, 36, 37, 39, 44). Training led to improved acceptance and use (36, 37, 44). In one study, user acceptability was higher among users that

(20)

received a complete training compared to users that received limited training. However, six months later, no difference was found between these groups in terms of acceptance (44).

In a study by Holden et al. (37), though some users were positive about the training, others indicated that the number of trainings was insufficient, or that the ‘classroom style” of the training did not fit their needs. They preferred either learning hands-on with one on one support given by an expert or colleague or practicing with scenarios that resembled clinical practice. In a study by Struik et al. (39), users preferred receiving support from an IT helpdesk over attending training.

Four studies explicitly mentioned that instructions (or demonstrations) and hands-on exercises were given during trainings and that users were positive about these trainings (34, 35, 40, 42). In the study by Bredfeldt et al., users suggested that the trainings should be offered more frequently and should cover a wider range of topics, the amount of usage of certain functionalities in the EHR increased by 2% to 4% after training (34). In the same study, users indicated that the hands-on exercises were the most useful part of the training, as they were able to familiarize themselves with the system (34, 40). In several studies, trainings have been combined with the use of other methods for delivering support, such as a telephone helpdesk (21, 35-37, 39, 42).

3.2.3 Remote support

3.2.3.1 Telephone and digital support

Support can be given remotely, e.g. via telephone or email. In 5 studies, support was provided via telephone (21, 35, 41, 43, 45). In two studies telephone support was part of the first tier of support and was the first point of contact with users (21, 45). At this level, relatively simple problems were solved, while more complex problems were escalated to a higher level of technical support.

The use of telephones as a communication tool to provide support was perceived both positively and negatively in different studies, depending on the communication tool it was compared to. For example, Struik et al. found that users preferred to receive help via a telephone helpdesk rather than receiving training, because training required extra time (39). However, compared to digital support, users preferred to contact support staff via their organization’s intranet rather than via telephone. In one study, users contacted support staff via intranet in 94% of the cases, compared to 6% of the cases via telephone (41). In this study, support was previously delivered via telephone only, and users were said to be dissatisfied with the accessibility and responsiveness of support staff. This led to the implementation of an institutional intranet. After implementing institutional intranet, at least 88% of the users were satisfied or very satisfied with both the accessibility and response time of the support staff. However, the researchers did not explain why the users were satisfied, but suggest that contacting support staff via telephone became a contingency procedure, which improved the accessibility via telephone (41).

3.2.3.2 Downsides of communicating remotely

Petersen found that users perceived that the communication with support staff was problematic (45). Users indicated that they wanted to know their support staff personally, as this could help to ‘continue to build on an understanding that was already established’ (45). Clinicians also let secretaries or nurses pass on their problems, because doing this themselves would cause a break in their workflow. Petersen suggests that this likely led to misinterpretations or the omission of details, which can slow down or complicate solving the problems (45). Therefore, they suggest that support staff need to fit in with the practice in order to communicate with users efficiently, by adapting to the way users generally communicate, which is

(21)

face-to-3.2.4 Formal face-to-face support

Personal, face-to-face support or assistance was provided in several studies (33, 35-38, 42). Users that received face-to-face support were generally positive about it (33, 35, 37, 38). As mentioned previously, users indicated that they benefited more from personal, one-on-one assistance on demand compared to training (37). Moreover, Boas et al. found that users would like to have face-to-face assistance for months or even years after they adopted an electronic information system, especially after software updates or changes (33). However, some studies faced a lack of support staff, which led to the inability to provide support timely (42, 45). In this way, the needs of the users remain unsatisfied, because the support staff were helping others and were not available (42).

3.2.4.1 Key users

In several studies, it was recommended that a person with an intermediary role should be present. This person should have knowledge about the local setting of the clinical practice, can help bridge the gap between the information technology and users, can give users support on site, and can function as a key contact for the vendor (21, 44, 45). For this reason, ‘super users’ or ‘key users’ were appointed (21, 35, 43). Their intermediary role made them highly valued by their organizations, as both clinical staff and support staff considered them to be essential to the success of the implementation of electronic information systems (43). These key users received more training than others, had access to more features and settings compared to regular users, assisted with troubleshooting problems, and helped new users, preventing interruptions in the workflow (35).

3.2.4.2 Informal peer support

Holden found that users also received informal support from peers and colleagues by observing their colleagues using an electronic information system, or asking them for assistance (37). Especially in smaller, independent practices that lack internal access to support that is often present in larger healthcare systems, users benefited from peer support (36). Lagebo et al. found that a lack of peer support led to a lack of motivation to apply things learnt during trainings to practice (42). In one study, users said they would benefit from ways to discuss the information system with others, e.g. via a forum or email list. In another study, a vendor facilitated an opportunity to share peer support by organizing a user conference, where users were able to exchange knowledge and learn more about using the information system (21).

3.3 Discussion

The aim of this systematic review is to assess the extent to which training and support concerning the use of electronic information systems for medical professionals have been studied and evaluated. This review shows the advantages and disadvantages of different methods of delivering training and support. In total, 14 articles have been included in this review. The small number of included articles indicates that little research has been done on the content (or form) and evaluation of training and support, even though many studies suggest that they are essential for the adoption and implementation of information systems in healthcare (16, 19, 46). In Figure 5, the different types of training and support are categorized according to sufficiency, based on the results, as well as means and their confidence intervals. Every filled dot represents an article that has described the form of training or support, while the circles with a horizontal stripe indicate the means. The levels of sufficiency are insufficient, sufficient, and excellent. Types of support that have been investigated in only one or two studies do not have confidence intervals.

(22)

Figure 5. Sufficiency of different types of training and support

The results suggest that medical professionals perceive a need for training, as the complexity of electronic information systems can be a barrier to adoption (20). Training can increase acceptance and use (44). Similarly, according to the UTAUT, which has been explained previously in Chapter 2, facilitating conditions, such as training, can contribute to actual use behavior. They also contribute to the implementation and confirmation, which are phases of the diffusion of innovations.

3.3.1 Increased acceptance and use through training

Even though some medical professionals are already satisfied with training that provides instructions only, others prefer to learn using a system by practicing with hands-on exercises. In this way, they can interact and familiarize themselves with the system. Also in the literature it is reported that hands-on exercises are of great significance (47, 48). Furthermore, medical professionals need to spend time on learning how to use a newly implemented electronic information system in order to prevent that their workflow slows down and their workload increases subsequently (20). The results suggest that the time spent on using information systems has also led to increased acceptance and use, e.g. in the study of Bredfeldt et al., in which the amount of usage increased with 2% (34), as well as in the study by Crivianu-Gaita et al. In their study, no difference was found between users that received complete training and users that received

(23)

3.3.2 Preferences for support

In the articles, different forms and characteristics of support have been described. For example, support can be provided personally or impersonally, face-to-face or remotely, and formally or informally. Medical professionals primarily prefer to communicate face-to-face (49). Even if medical professionals were offered training via telephone, they would request to have training in person, so that the trainer was able to see their expressions, if they did not understand something during training (49). Another explanation for their preference for face-to-face support can be that they are used to explain and clarify poorly defined clinical problems by telling their colleagues face-to-face (47, 50).

In some studies it was not possible to provide this kind of support optimally due to lack of support staff (42, 45). Alternatively, peer colleagues or key users may provide face-to-face support as opposed to support staff from vendors. Key users can identify and manage problems and motivate others, because of their additional training, broader experience, or special interest (47, 48). They are regarded as a critical factor in successful implementations of electronic information systems in healthcare (49). One study also mentioned that a conference facilitated by a vendor can help users to learn more about a system and can be an opportunity for them to exchange their knowledge and experiences (21).

Next to face-to-face support, support can also be provided remotely, which can either be a second line of support or an alternative to face-to-face support. The results suggest that medical professionals prefer digital support, such as intranet or email, over the use of telephones, because they perceived the responsiveness and accessibility of support staff was greater with digital support. Telephones could be used as a contingency procedure, e.g. for more urgent, possible life-threatening issues (41, 51). These findings are also found in studies on interprofessional communication. Medical professionals perceived that the use of email yielded a faster response, increased accessibility to support staff, and helped them to convey patient information quickly and efficiently (51, 52). If a telephone helpdesk is the first point of contact, it is possible that support staff are not readily available for all users that need help. In this case, users are not able to convey their problem to the support staff directly. Users can also use impersonal support tools, such as manuals, though no evaluations have been done on these tools. This opens up more opportunities to investigate how support can be best provided with impersonal materials.

3.3.3 Limitations

The possibility to analyze the results of this review quantitatively remains limited, due to the small number of included articles and the limited amount of quantitative data measured and reported in the articles. A low number of studies can cause findings to be too broad or incomparable (53). As this systematic review is mainly an explorative study, future studies may focus on measuring needs of medical professionals and the effects of training and support quantitatively in order to add more knowledge to this field. Another limitation of this review is that only one researcher was involved. Including more than one reviewer in a study is recommended to ensure elimination of personal bias and to reduce the possibility of excluding relevant articles (53, 54).

3.3.4 Recommendations for practice

It is recommended that the training for the use of EHRs includes both instructions and hands-on exercises, so that medical professionals can familiarize themselves with the system and can try using it already. Additionally, support should be provided. One of the most preferred ways to communicate is face-to-face.

(24)

support can be provided remotely. Digital support, such as email or intranet, is preferred over the use of a telephone helpdesk, because of the opportunity to convey information directly to support staff. Telephones can then be used as a contingency procedure for more urgent cases. As for other forms of support, such as a user conference or impersonal support materials like manuals, little research has been performed. Future research should be performed to assess whether these forms of support can also be effective.

3.3.5 Recommendations for research

In this review, the state of the art with regards to the perceptions of medical professionals on the content and forms of training and support has been shown. This review can serve as a starting point of research in this subject and may motivate other researchers to investigate this aspect of the adoption of IT in healthcare more thoroughly in order to optimize training, support, implementation, and adoption. To vendors and other parties that develop electronic information systems or assist in the implementation of these systems, this review may give insight into how training and support can be provided in order to optimize the innovation-decision process, the implementation, and the adoption of electronic information systems in healthcare.

(25)

Chapter 4. Interviews

4.1 Methods

4.1.1 Study design

Semi-structured interviews were performed by a Master’s student Medical Informatics at the University of Amsterdam from December 2017 to February 2018. Respondents were selected by convenience sampling. In total, 147 medical researchers and professionals were contacted by email in December to January 2017, of which 2 refused and 130 did not respond. Finally, 15 interviews were performed with medical researchers from 7 different healthcare organizations, of which 4 interviews were performed by telephone because the respondents were either geographically distributed or indicated that they did not have time to participate in face-to-face interviews. Eleven face-to-face interviews were performed at the healthcare organization that the respondents worked at. The interview guide and questions were created based on the UTAUT and the innovation-decision process (Appendix B). During the analysis, follow up questions were sent via email in March, which yielded responses from 10 respondents.

4.1.2 Respondents

The target population was divided into three groups. Two groups consisted of medical researchers who either used the EDC tools Castor EDC or OpenClinica. In this thesis the two groups will be called EDCA and EDCB, concealing which group relates to what EDC tool. The third group consisted of medical professionals that used EHRs. These groups were chosen, so that the support for different EDC tools could be evaluated and compared with each other and with EHRs. Inclusion criteria for respondents were aged 18 years or older, able to converse in Dutch or English, having experience with using an EDC tool or EHR, and employed at a healthcare organization. EDC tools were considered to be all software that was dedicated to electronic data capture and management in medical research. Demographics of the respondents are shown in Table 2.

4.1.3 Data collection

Interviews lasted 30 minutes and were audiotaped, after a respondent had given verbal consent. The interviews were transcribed verbatim using Express Scribe version 6.10. Names of persons and institutions were concealed to ensure anonymity. The interviewer asked the respondents whether they could be contacted after the interview via email for member checking in order to improve the validity of the results (Birt, 2016 #71). This meant that summaries of each interview and additional follow up questions were sent to the respondent after the data analysis of the transcripts, so that respondents could add or modify any answers they were not satisfied with. The follow up was performed in March. If no consent was given for audiotaping the interview, only a summary was made and, together with follow up questions, sent to the respondent for member checking.

(26)

Table 2. Characteristics of interview respondents

Respondent Gender Age Occupation Department

EDCA1 Female 34 Senior researcher Alzheimer Center

EDCA2 Female 29 Clinician researcher Vascular Medicine

EDCA3 Female 29 Clinician researcher Experimental Cardiology

EDCA4 Female 33 Research assistant Alzheimer Center

EDCA5 Male 35 Bio informaticus and data analyst Alzheimer Center

EDCB1 _Male ₂₇ _{PhD student} _Chirurgy

EDCB2 Female 27 Clinician researcher Obesitas Center

EDCB3 _Female ₂₆ _{PhD student} _Neurology

EDCB4 _Female ₃₄ _{Senior researcher} _Rheumatology

EDCB5 Female 30 Clinician researcher Chirurgy

EDCB6 _Female ₃₁ _Resident _Psychiatry

EHR1 _Male ₃₁ _Resident _{Anesthesiology}

EHR2 Male 31 Resident Lung department

EHR3 Male 31 Resident Addiction rehabilitation

EHR4 _Female ₃₀ _Resident _Pediatrics

4.1.4 Data analysis

An inductive, thematic content analysis was conducted, which means that common patterns in the data were identified. The analysis was performed with a bottom up approach, which means that open codes were identified first. The codes were categorized according to emergent themes. The themes included knowledge of EDC or EHR tools, voluntariness of use, relative advantage, experience with starting to use an information system, knowledge of available support, experience with support, and preferred way to seek and receive help. The coding scheme is shown in Appendix C. Current formats of training and support as used by respondents were compared to preferred formats of training and support.

4.2 Results

4.2.1 Awareness and voluntariness of use

There were several explanations as to why certain information systems were chosen for research studies. All EHR users indicated that they had no influence on the decision which EHR would be implemented and used. Usually, healthcare organizations made these decisions, as EHRs needed to be implemented throughout an organization. Three EDC users also indicated that they were not able to choose which EDC

(27)

Six of the remaining eight EDC users indicated that they chose a certain EDC tool because of the support provided by their healthcare organization, as explained by respondent EDCA1. Three of them suggested they were not aware of other EDC tools at the start of their study. Finally, two EDC users said they had chosen which EDC tools they wanted to use themselves, based on usability and on the presence of desired functionalities, such as the possibility to export data, or to link a database with an EHR.

“When this study went live, [EDC A] had just become the EDC tool for the [hospital]. We were

unlucky in that aspect, because one year later, it was [EDC B]. Now [EDC A] is also supported less [at the hospital]. No, we did not have a choice. At that time, we had to use whatever was compliant to GCP and was financed by the hospitals. And at that time, it was [EDC A].”

— Respondent EDCA1

“I heard many people say that [EDC B] was easier, more user friendly. (...) So then I decided to

use [EDC B] for my study. And of course I consulted my supervisors, and they agreed with me.“

— Respondent EDCB3

4.2.2 Experience with support

4.2.2.1 Training and education

Three EHR users and one EDC user followed formal trainings provided by their healthcare organization. These trainings consisted of classroom style trainings with instructions and exercises in a sandbox environment. Two respondents indicated that they also had access to e-learning modules, which they needed to complete before attending a training. The respondents indicated that the trainings were useful for getting to know the basics of how to work with an electronic information system. However, three out of four respondents also found that the training was too long and that the pace of the training was too slow. The respondents indicated that the content of the trainings was not always relevant for them. For example, some functionalities were explained and included in exercises, though the respondents would not use these in practice.

“I found [the training] useful, because you’ll get the basics, but - maybe I’m just like this - you learn better by just doing it. (...) Someone can explain some kind of hypothetical situation to you, but your study may not be like that at all. (...) So the training was quite useful, but when I look back at it, it didn’t have to be for three days. I think it’s nicer to just do it and then be able to ask questions. Maybe if you have a training in which someone can already work on his own study, then it may be more useful.”

“I know I received an explanation about how [the EHR] works at the outpatient clinic, and that was quite weird, because I don’t work at the outpatient at the moment. It was quite useful to get feeling with the system, (...) The training consisted of practicing in a playground version of [the EHR]. And you could do exercises (...) you learn all steps of [the EHR]. But you might think that that’s not the way you will be using it,”

(28)

— Respondent EHR3 EDC users that did not follow trainings mostly learned to use EDC tools by trying it out in test environments. Six out of ten EDC users used online manuals or videos provided by the vendor to learn how to use the EDC tools. Though most of them were positive about these support materials, two respondents stated that the manuals and videos didn’t help. One of them indicated that the issues they wanted to solve were too study-specific, while the other stated she did not understand the structure of the eCRF even after using these support materials. Two respondents didn’t use manuals, because they perceived that they contained too much text and information to read, though one of them said to herself she should read them, as shown in the citation below. EDC users who did not follow any trainings nor used impersonal support materials asked colleagues to help them learning how to use the system.

“I think the layout is a little bit weird. There are also many tabs which I never use, so maybe

I’m missing out on something. Maybe I should read the manual. [It could be] that it’s all possible, and I just don’t do it.”

4.2.2.2 Personal support

After training, the respondents, especially EHR users, asked for support from their colleagues, as they were quickly accessible. For some EHR and EDC users, there were key users, or data management or IT departments which were able to provide support. Most users were satisfied with the help they received from these key users or departments. Three respondents indicated that they were not satisfied, either because the key user or department was not able to solve their problems, and sometimes, delegating complex issues to another party led to prolonged waiting time for the issue to be solved.

EDC users mostly sought help online, i.e., by email or chat, and by telephone from the vendor, an external party, or an internal department responsible for data management in clinical research. They indicated that they were satisfied about the accessibility and responsiveness of these parties, as the support staff usually responded within a day. One respondent said that he communicated via chat for “smaller” problems, and for “bigger” problems, he communicated via email. Some respondents that were only able to contact their vendor digitally indicated that they would also like to contact the support helpdesk via telephone, as they perceived that this would make it easier to explain their question or problem and to exchange information faster:

“Sometimes you just want to tell [your problem via telephone], because you can immediately

answer a question [from the helpdesk], and then it goes smoothly. Then you don’t have to put a whole story on the chat - because sometimes it’s quite complicated how everything works and what I’m doing specifically. (...) I think that making a phone call would help me the most, because then you just have someone on the phone — unless you have to wait in line for an hour

— and then it is just faster.”

(29)

4.2.3 Preferred training and support

Before using a new system, most respondents would like to use impersonal support materials to get to know the basics of the system, in particular by learning. Furthermore, one respondent explained that e-learning enabled users to choose themselves when and where they would follow the e-e-learning course. EHR users also preferred to receive training before using a new system. However, as mentioned previously, most respondents that received training found that the pace of the training was too slow. They would rather receive a training that shows the basics of an electronic information system and is tailored to what they want or need to work with, e.g. by showing and hiding certain parts of the training, depending on the type of user follows the training or the department the user works at.

After e-learning, EDC users would like to have a personal consult to receive advice on how to build their study. During the actual use of the system, both EDC and EHR users wanted to receive peer support. In case of problems, they would also like support staff to be able to look into their computer or study. After training with e-learning and personal consult, EDC users wanted to be able to get in contact with print(support staff in case of problems. Mainly contact via telephone and email were preferred, while chat was preferred to a lesser extent. Five respondents also preferred to have the possibility to share their screen with support staff, which would make it easier to explain a problem.

4.2.3.1 Responsiveness and accessibility

Characteristics that respondents found important for support specifically were identified during the interviews. These characteristics included responsiveness, accessibility, applicability of information, friendliness of support staff, the extent to which feedback about the status of a problem or question was given, and the extent to which priorities were given to issues so that urgent ones are solved before less urgent ones. One respondent also mentioned the importance of language, i.e., a preference of communication in Dutch over English, but this view was generally not shared between other respondents.

Responsiveness and accessibility were considered to be the most important characteristics for support. Respondents that used EDC A or EDC B indicated that the responsiveness of support staff was good, as they usually received a response from support staff within half an hour up to one day. One respondent indicated that it was not possible for her to wait a week for questions or problems to be solved, although another respondent indicated that requirements for responsiveness also depended on the size and urgency of a certain problem. As for accessibility, two respondents tried to send in problems and feature requests, but the key user or IT department were not able to help them. They either sent the issues to another party or higher organizational level of support, or did not help at all. This frustrated the users, because they were not able to continue their work temporarily or had to work inefficiently. This reduced accessibility as perceived by these respondents.

“I once wrote down bugs and passed them to the IT [department] that are responsible for [the

EHR]. But they say, ‘That’s difficult, (...) They [the vendor] won’t change anything anyway, so it’s no use.’ There’s a little bit of a culture like “Well, forget it.” So it gets stuck, and at some point you also just continue working and you think, ‘Well, I’ll do a few clicks extra then’” —

EHR4

4.2.3.2 Applicability of information and friendliness of support staff

(30)

they did not want to receive more information than asked for. For example, two EHR users said they received information about departments at which they would not work at, while two EDC users received information about functionalities they would not use for their particular study anyway. Furthermore, respondents also appreciated friendliness of support staff. For example, respondents said that they appreciated it when they did not have the feeling they were asking a dumb question, or that they had the feeling that the support staff would help until they were satisfied, even if that meant that the support staff went out of their way to help. Researchers also wanted to know the status of the problem, so they were able to plan to do other things in the meantime and know when their problem had been solved.

4.2.3.3 Status of problem and priority system

Relatively less important characteristics included receiving information about the status of a question or problem and having a priority system in place which ensures that urgent matters are solved before less urgent matters. Few respondents also indicated they would appreciate it if support staff would give them feedback about the status of a problem or question they had sent in, e.g. information about how much time would take approximately to solve a problem. As one argued, a user could plan to do something else for the time being, and not have the feeling they had to wait and lost time. One respondent also mentioned that she would like to see a priority system in place for the questions and problems, so that support staff would be able to handle urgent issues before less urgent issues. However, other respondents did not found this as important as other characteristics mentioned before.

“Because then as support staff you can (...) filter the priorities. And back then I had the feeling that I always put on the same stack [of problems], and that they didn’t really look further, you were just treated according to the queue. So a priority system, maybe. And having multiple ways to contact [support staff] as individual researcher.” — EHR1

4.3 Discussion

This study aims to explore the perspectives of medical professionals, and researchers in particular, on different forms of training and support for EDC tools. Not only the EDC tool should fit users, but also the training and support should fit with the way medical professionals work, as they are facilitators for the adoption of electronic information systems (17, 27). The results suggest that currently, EHR users mainly receive trainings or ask their colleagues for help but would like to see that these trainings would be more optimized to the way they would work at their particular department. On the other hand, EDC users try out EDC tools in test environments, learn with videos and manuals, or have colleagues help them to learn how to use the system. EDC users would also like to receive e-learning to learn the basics, followed by a personal consult to learn how they can build their specific study into the EDC tool.

4.3.1 Training and support as facilitating conditions

In Chapter 2 it is assumed that training and support can be seen as facilitating conditions in the UTAUT. In line with this model, the results suggest that training and support can contribute to the use of a system, as they help users to learn how they can work with the system. Effort expectancy also plays a role, as users do

(31)

the maximum extent to which a user wants to put effort in trying to learn using a system can differ for every individual. For example, some of the users did not read manuals, because they perceived they were too extensive and too much to read, while other users used these manuals extensively and found them useful.

Furthermore, in Chapter 2 the innovation-decision process has also been described, which consists of the phases knowledge, persuasion, decision, implementation, and confirmation. It is assumed that these stages of this theory would also be applicable to EDC tools. However, most respondents indicate that they do not make the decisions to use a certain EDC tool. In only two out of fifteen cases, respondents have indicated that they have chosen to use certain EDC tools themselves without any influence from their healthcare organization.

The first step, knowledge, may be important for EDC tools in particular, as this can lead to the subsequent phases of the innovation-decision process. Moreover, during the interviews most respondents have failed to come up with EDC tools other than the one they used or was supported by their organization. In the literature it has also been reported that medical professionals have had little exposure to different EHRs in their educational curriculum (55). Though the other stages, persuasion, decision, implementation, and confirmation have not been investigated thoroughly in this study, it is clear that training and support can influence most of these stages in one way or another.

Attributes of innovations that have also been also found to be important for support include complexity and compatibility. As for complexity, instructions should not be too difficult for users. Compatibility means that the channels of communication, information, and the presentation of information should fit the users. The results suggest that different ways of communicating may also fit different users. For example, some users preferred learning with a manual, while others preferred watching videos.

4.3.2 Comparison of training with literature

In line with the findings of the systematic review of this thesis, respondents have indicated that they find the trainings with exercises in a playground or test environment useful, because they are able to get to know the basics of an information system and to get “feeling” with the system (34). However, some of them perceive that the trainings are too extensive or too slow, as some of them had the feeling they knew a lot of things already during the training. One explanation is the relatively young age of the respondents, which is 31 years old on average. Younger people generally have fewer problems in accepting and learning new technologies compared to older people (49, 56).

Respondents that have followed trainings or have watched videos also indicate that sometimes, they learn about functionalities they would not actually use for their particular department or study. A similar issue has been found in a study by Anderson et al., in which researchers have specific, individual needs and perceive that support provided by their institute has shortcomings (8). These issues can be addressed by assessing users’ skills, needs and by matching varied trainings to these user groups (57).

Respondents prefer to receive e-learning in order to learn how to use a new system, as this can give them increased flexibility, e.g. remote and rapid access (58). However, some EHR users would still like to receive training additionally, which is also mentioned in the literature (58). The combination of e-learning and training is called blended e-learning, and provides the “best of both worlds” (58). Furthermore, blended learning can also benefit and help users with low levels of computer skills who may have difficulty following e-learning only (58).