Workable ingredients for effective VBHC dashboards:
Framework development and first validation
Jelle Carlo Verbraak, BSc. 10726969
Master Medical Informatics, Amsterdam University Medical Centres, Location AMC, University of Amsterdam
13th September 2019
2 Master Thesis
Workable ingredients for effective dashboards: Framework development and first validation
Author
J.C. Verbraak (Jelle Carlo), Bsc. 10726969
Department Medical Informatics, University of Amsterdam Meibergdreef 9, 1105 AZ Amsterdam
Supervisor
Dr. L.W. Dusseljee-Peute (Linda)
Department Medical Informatics, University of Amsterdam Meibergdreef 9, 1105 AZ Amsterdam Mentor L. Spadon (Leandra), Msc. Furore Bos en Lommerplein 280 1055 RW Amsterdam SRP Duration December 2018 – September 2019 SRP Location Furore Bos en Lommerplein 280 1055 RW Amsterdam &
Leids Universitair Medisch Centrum Albinusdreef 2
3
Table of contents
Preface ... 6 Abstract ... 7 Samenvatting ... 8 1. General Introduction ... 91.1. Value-based Health Care and the relevance of giving feedback ... 10
1.2. Setting ... 12 1.3. Research goal ... 13 1.4. Research questions ... 13 1.5. Approach ... 13 1.5.1. Literature phase ... 14 1.5.2. Theory phase ... 14 1.5.3. Evaluation phase ... 14 1.6. Thesis Outline ... 14 1.7. References ... 16
2. Modelling factors and characteristics of effective dashboards from healthcare literature using Grounded Theory ... 18
Abstract ... 19
2.1. Introduction ... 20
2.2. Methods ... 20
2.2.1. Study inclusion criteria ... 20
2.2.2. Search strategy ... 21
2.2.3. Study selection ... 21
2.2.4. Data extraction, analysis and synthesis... 21
2.3. Results ... 21
2.3.1. Study selection ... 21
2.3.2. Grounded Theory coding... 23
2.3.3. Grounded Theory themes ... 23
2.3.4. Initial model ... 25
2.3.5. Initial framework ... 25
2.4. Discussion ... 26
2.5. References ... 29
4
Abstract ... 32
3.1. Introduction ... 33
3.2. Methods ... 33
3.2.1. Study inclusion criteria ... 34
3.2.2. Search strategy ... 34
3.2.3. Study selection ... 34
3.2.4. Data extraction, analysis and synthesis... 34
3.3. Results ... 34
3.3.1. Study selection ... 34
3.3.2. CP-FIT ... 34
3.3.3. Similarities initial framework and CP-FIT ... 36
3.3.4. Differences initial framework and CP-FIT ... 37
3.3.5. Consequences for the initial model and code list ... 37
3.3.6. Definitive framework ... 38
3.3.7. Effective Dashboard Development Process ... 40
3.4. Discussion ... 42
3.5. References ... 43
4. Mapping dashboards with the Framework of Effective Dashboards ... 44
Abstract ... 45
4.1. Introduction ... 46
4.2. Methods ... 46
4.3. Results ... 47
4.3.1. Description Brughoek Dashboard ... 47
4.3.2. FCL results Brughoek dashboard ... 47
4.3.3. Description Schildklier dashboard ... 51
4.1.1. FCL results Schildklier dashboard ... 54
4.3.4. Comparison between Value-based Health Care and the FOED ... 56
4.4. Discussion ... 57
4.5. References ... 59
5. Determining the effectivity of VBHC dashboards in the LUMC ... 60
Abstract ... 61
5.1. Introduction ... 62
5 5.3. Results ... 63 5.3.1. Log data ... 63 5.3.2. Brughoek dashboard ... 63 5.3.3. Schildklier dashboard ... 63 5.3.4. Interviews ... 64 5.4. Discussion ... 64 5.5. References ... 66 6. Discussion ... 67 Appendix ... 69 Appendix 1 ... 69 Abbreviations ... 70
6
Preface
This thesis is my final assignment for the Master Medical Informatics at the University of Amsterdam. This final assignment took place at Furore in Amsterdam, at the Amsterdam University Medical Centre (location Amsterdam Medical Centre), and Leiden University Medical Centre. The scientific research project which was the basis of this thesis, was an incredible educational journey. I developed not only scientific but also practical skills that will certainly help me in my future career.
This thesis is, for now, the end of my life as a student. This is a huge achievement and I could not have done it without the help of certain individuals. First of all, I would like to sincerely thank my supervisor Linda Dusseljee-Peute for her incredible support throughout this scientific research project. Linda, you were an amazing sparring partner, teacher and I can say that without you, I would not have the ability to develop the framework.
Second, I would like to especially thank Leandra Spadon for her support and feedback. You were always available for feedback and gave solid advice which I will certainly use in my future career. Furthermore, I would like to thank my colleagues and the management of Furore. Everyone at Furore was willing to help and I can honestly say that Furore has the nicest work environment I’ve seen to date.
Besides my colleagues at Furore, I would also like to thank my colleagues in the LUMC, and especially Barbara Schooneveldt. Thank you for giving me the time and place to perform my research and giving me a hands-on experience in the field of dashboards.
I also want thank my fellow student Maikel Sing. Maikel, you were an amazing co-student, I’ve had lots of fun with you during the master and I can safely say that without you, getting my master’s degree would have been a lot harder.
On top of that, I would like to thank my family for their support and feedback. You were all great motivators! Finally, I would like to sincerely extend my gratitude to my girlfriend Anja. During this scientific research project some interesting things came our way and despite the stress and difficulty that came with it, you supported me unconditionally.
7
Abstract
Introduction:
An increasing amount of hospitals in the Netherlands are adopting Value-based Health Care (VBHC). VBHC aims to improve the delivered quality of care by measuring and improving outcomes that reflect value instead of volume in order to reach its goal: creating value for the patient. To increase the value for the patient, care teams have to improve the quality of care. An often-used method of activating care teams to improve is providing audit and feedback through dashboards. However, the effect of audit and feedback dashboards is often very little or non-existent. Therefore, the need exists for a framework that describes the how factors around dashboards and the characteristics of dashboards interact with each other and lead to an effective dashboard. In this research, a Framework of Effective Dashboards (FOED) is proposed and tested.
Methods:
First, a literature study was executed that used Grounded Theory to develop a code list. Based on the code list, a basic framework and model were made that describe the characteristics of effective audit and feedback dashboards and the factors that influence effective dashboards. Second, the initial framework and model were complemented with the Clinical Performance Feedback Intervention Theory. Thirdly, the Brughoek and Schildklier dashboards of the LUMC were mapped to determine their compliance to the FOED and to start with a first validation of the items in the framework. Also the effect of VBHC on the FOED was determined. Finally, the usage and perceived effectiveness of the Brughoek and Schildklier dashboards were determined by looking at the usage in log data together with interviews.
Results:
The FOED was successfully established, supplemented with hypotheses from Clinical Performance Feedback Intervention Theory and tested. However, both dashboards did not comply with the FOED. Only 53% of the items in the framework could be validated in both dashboards. The effectiveness of both dashboards was tested to determine whether not FOED complying dashboards still could be effective. Both dashboards were used seven to eight times by four different members of the care team. Both dashboards were currently used as tools to reduce the administrative burden of the case managers. When asked whether the interviewees perceived the dashboards to be slightly versus completely effective in increasing value for the patients.
Discussion:
Audit and Feedback dashboards are often not effective. The FOED shows that there are many factors and characteristics that interact with each other and affect the effectivity of dashboards. VBHC dashboards should comply to certain characteristics to be as effective as possible to activate users to improve their processes and thus increase value for the patient. While the Brughoek and Schildklier dashboard were both perceived as effective VBHC dashboards, it is disputed that both dashboards are true VBHC dashboards in their current state. IT teams that are responsible for the development of future VBHC dashboards in the future should be aware from the start of development that many factors and characteristics affect the effectivity of dashboards and define when they see the dashboard as effective.
8
Samenvatting
Introductie:
Steeds meer ziekenhuizen in Nederland adopteren Value-based Health Care (VBHC). Het doel van VBHC is om de waarde van geleverde zorg te verbeteren door middel van het meten van zorg-, proces- en door patiënt gerapporteerde -uitkomsten die van waarde zijn voor de patiënt en deze te delen met zorgverleners, zodat zorgverleners hun zorgprocessen verbeteren. Om zorgverleners inzicht te geven in de kwaliteit van geleverde zorg, worden steeds vaker dashboards ontwikkeld. Het doel van deze dashboards is om de gebruikers te activeren om hun zorgprocessen te verbeteren. Alleen is het effect van dashboards vaak minimaal of afwezig. Daarom bestaat de behoefte voor een framework die beschrijft hoe factoren en karakteristieken van dashboards elkaar beïnvloeden en leiden tot een effectief dashboard. In dit onderzoek wordt daarom het Framework van Effectieve Dashboard (FOED) voorgesteld en getest.
Methode:
Allereerst is een literatuurstudie uitgevoerd die door middel van Grounded Theory heeft geresulteerd in een codelijst. Op basis van deze codelijst is een initieel framework en model ontwikkeld die de karakteristieken van effectieve dashboards beschrijft samen met de factoren die de effectiviteit zouden kunnen beïnvloeden. Vervolgens zijn het framework en model uitgebreid met de Clinical Performance Feedback Intervention Theory (CP-FIT) en gevormd tot de FOED. Met behulp van de FOED is daarna gekeken naar welke mate de Brughoek en Schildklier dashboards in het LUMC voldoen aan het ontwikkelde framework om zo een eerste validatie te doen van de items in het framework. Daarnaast is gekeken naar het effect van VBHC op het FOED. De ervaren effectiviteit van de dashboards in Leiden bepaald door te kijken naar log data en interviews.
Resultaten:
De literatuurstudie resulteerde in een codelijst van 188 codes, verdeelt in zestien thema’s. Vervolgens is het FOED beschreven en het model gemaakt dat de relaties tussen de thema’s visualiseert. Het FOED is vervolgens aangevuld met 22 hypotheses van CP-FIT. Geen van beide dashboards voldeden aan de FOED, waarbij het Brughoek dashboard niet voldeed aan de Benchmarks, Co-interventions, Dashboard Functionality, Development, Indicator, Usage, Implementation en Evaluation thema’s. Het Schildklier dashboard voldeed niet aan de Benchmarks, Co-interventions, Dashboard Functionality, Development, Evaluation, Implementation en Indicator thema’s. Tot slot is gekeken of de dashboards regelmatig werden gebruikt en of deze door de gebruikers werden gezien als effectieve VBHC dashboards. Vanaf het moment van de releases van de dashboards tot het moment van schrijven waren beide zeven tot acht keer gebruikt door vier verschillende zorgverleners. De dashboards werden gebruikt als middel om de administratieve last van de casemanagers te verminderen, alsook om het aantal ‘eigen lijstjes’ te verminderen. Beide personen die waren geïnterviewd vonden dat de dashboards een beetje of volledig effectief waren in het verhogen van de waarde van zorg voor de patiënt.
Discussie:
De FOED laat zien dat veel factoren en karakteristieken met elkaar interacteren en de effectiviteit van dashboards beïnvloeden, wat er mogelijk toe leidt dat veel dashboards niet effectief zijn. Daarnaast laat de FOED zien dat veel thema’s effect op elkaar hebben en daardoor wellicht te complex is. De Brughoek en Schildklier dashboards worden door de gebruikers ervaren als deels of volledig effectieve VBHC dashboards, maar of de dashboards in hun huidige staat daadwerkelijke VBHC dashboards genoemd kunnen worden kan worden betwist. Ontwikkelteams die in de toekomst dashboards gaan ontwikkelen moeten vanaf het begin van de ontwikkeling bewust zijn van de vele factoren en karakteristieken die effect hebben op de effectiviteit van dashboards. Daarbij kunnen zij de FOED en de bijbehorende codelijst als eerste aanzet tot een raamwerk gebruiken.
9
1. General Introduction
10
1.1. Value-based Health Care and the relevance of giving feedback
In 2008 the world suffered an economic crisis [1].The economic crisis caused a reduction of the resources invested in health services in most countries [2]. In the Netherlands this was despite a continuous growth in healthcare expenses [3]. In the past 20 years techniques, frameworks and theories have been developed to ensure that only interventions with strong evidence of cost effectiveness, quality improvement and cost reduction were implemented. An example of such a technique is evidence-based medicine, which is widely used in the Netherlands [1, 4]. However, evidence based medicine has not been sufficient in reducing costs and improving quality of care. Because of this insufficiency, an increasing amount of hospitals in the Netherlands are adopting Value-based Health Care (VBHC) [5].
The concept of VBHC is relatively new in the Netherlands [6]. It is developed by Michael Porter and Elizabeth Teisberg and was first published in 2006. VBHC is a new rationale for the arrangement and financing of healthcare [7 – 9]. It aims to improve the delivered quality of care by measuring and improving outcomes that reflect value instead of volume in order to reach its goal: creating value for the patient [10, 11]. In this instance, value is defined as ‘the effect that a treatment has for a medical condition on short and long term for a patient, as function of the costs that are accompanied by it.’
In order to transform healthcare systems to value-based healthcare systems, Porter described the “value agenda”. The value agenda is an overarching strategy that requires restructuring of how health care delivery is organised, measured and reimbursed. The value agenda exists of six components, shown in Figure 1. Each of the six components are independent and mutually reinforcing. The progress of moving to a high-value health care delivery system will be greatest if multiple components advance together [8].
The core of the value transformation is changing the way clinicians are organised to deliver care. In order to structure the care around the patient and its needs, it is required that healthcare organisations shift from a siloed organisation by
speciality department to an organisation around the patient’s medical condition. This is called an integrated practice unit (IPU). IPUs do not only treat diseases but also related conditions, complications and circumstances that occur. The personnel in an IPU work together as a team toward a common goal: to maximize the patients overall outcomes as efficiently as possible. They improve their care by meeting frequently, formally and informally, and review data on their own performance.
In order for teams to review data on their performance, this data needs to be collected. More specific, outcomes and costs for every patient have to be measured. That way, teams can improve and excel by tracking their progress over time and compare their performance to other care professionals inside and outside their organisation. An important note is that the outcomes that are measured matter to the patient. Porter described a hierarchy of different outcomes shown in Figure 2 [12]. Since the publication of the outcomes hierarchy of Porter, several standard outcome sets have been developed in recent years to support outcome measurement [13-16].
11 But how can the measured outcome data be gathered, transformed into information and presented in such a way that the care professionals are triggered to improve? At this moment no literature exists that describe how outcome measures ideally should be presented with regards to VBHC. This while the need exists to give the care providers effective feedback on their performance and the fact that there are multiple ways to present information: with face-to-face contact, the use of electronic devises, results printed on paper etc.
An example of an upcoming digital solution to present information is the use of dashboards.
The term ‘dashboard’ is ambiguous because a multitude of different types of dashboards exist. Dashboards can be used as a clinical decision support tool, learning tool, monitoring tool etc. [17 – 19]. When a dashboard is part of a management information system, it is a real-time user interface providing graphical and tabular representations of historical trends and the
current status of key performance indicators [20]. Dashboards seem to have caught on as a management tool [21], but also might be suitable to give insight in outcome measures that represent patient value. An example of the use of dashboards in VBHC initiatives is the implementation of Van Egdom et al. They used a dashboard to show the collected outcomes to both the patients and the physicians [11]. The question that remains however is how dashboards for VBHC initiatives can be developed and designed in such a way that the users are triggered to improve their care processes. No studies have been performed that determined what the optimal methods are. This is worrisome, because it is important that the VBHC dashboards reach their goal, whether the goal is directly increasing value by showing patients information or for care professionals to gain insight in their care processes. Insufficient dashboards may have different consequences, it may be that the dashboard will decrease value showing incorrect information or that the dashboard shows information in such a way that processes which need improvement are marked as processes which do not need improvement.
As mentioned before, the term
‘dashboard’ is ambiguous and thus a single widely used definition does not exist. Therefore, in this study we will define dashboards as “a visual display of the most important information which represents patient value so that care professionals as users are able to gain insight into their processes, where the information is consolidated and arranged on a single screen so the information can be monitored at a glance with the possibility to drill down into the information”. With this definition, we will focus on VBHC dashboards with the goal on increasing the value of care for the patient.
Dashboards can, when well designed, help consolidate the data in meaningful ways so it is clear, consistent and accessible [22]. Dashboards also can help identify trends and opportunities, and aid in performing quick and thorough assessments of hospitals [23]. According to Pauwels et al. the adoption and success of dashboards is driven by five main factors [24]:
- Demand side of the dashboard
12 - Supply side of the dashboard
- The fit between demand and supply - Implementation process of the dashboard - The predisposition of users towards the
dashboard
At this moment, it is unclear how outcome measures can be presented in such a way that care professionals are triggered to change their processes.
Within the healthcare domain research has been performed on different types of dashboards. Clarke et al. performed a literature review on clinical decision support dashboards which showed that “early end-user involvement is an iterative process with a clear-cut return on investment”. Also, Clarke et al. mentioned that a lack of fit between a clinical decision support dashboard and its users can create inefficiencies and prevent achievement of intended results [25]. Wilbanks et al. showed a multitude of interesting benefits and limitations of dashboards, as well as evaluation methods for dashboards in a review of on dashboard for data analytics in nursing[26]. Buttigieg et al. encountered in their literature review on hospital performance dashboards a limitation that aforementioned reviews did not
mention, which was the occurrence of
measurement fixation. When measurement fixation occurs, the emphasis is on meeting the targets instead of the overarching purpose of the organisation [17]. Dowding et al. came to the conclusion that future research should be conducted to establish guidelines for the designs of improvement of patient care dashboards [27]. What stands out in the reviews is that none of the reviews mention how the dashboards work, present the information or give feedback to the user about their processes and key performance indicators. As Dowding et al. mentioned future research should focus on establishing guidelines for designs of improvement dashboards, which indicates that there is a lack of evidence on how to design improvement dashboards.
To summarize the findings above: there are numerous dashboards in the healthcare domain and the effect of the dashboards on healthcare processes has been studied. There is however a lack of research on: VBHC dashboards, how
dashboards should present information (i.e. give feedback) to the users in order to trigger the users to change their processes as well as a how to design VBHC improvement dashboards.
Understanding issues involved in the
development and implementation processes that either enhance or are known barriers to development and implementation may support future VBHC dashboards.
1.2. Setting
This study was conducted at the Leiden University Medical Centre (LUMC) in the Netherlands. The LUMC is a large academic hospital that delivers top clinical care and has 882 beds available. In 2018, the LUMC started to embrace the VBHC principles within its care processes and started rearranging the care processes [28]. This means that care is going to be organised in multidisciplinary care units. Besides that, Patient reported outcome measures (PROMS) as well as clinical outcomes which represent value for the patient will be defined. Also, results for the individual patient and for the cohort will be used to improve the care for both the individual patient and the cohort.
In order to rearrange its care processes, the LUMC designed a VBHC trajectory for each medical condition (Figure 3). In the design phase of the trajectory, the desired care process is designed and outcomes are defined. After the design phase the building phase will start, in which the dashboard is build and a PROMs tool is implemented. In the implementation phase, the team will be guided in monthly management team meetings [10]. It is important for the LUMC that the dashboards are effective, otherwise the initiative might be a loss of resources. To prevent this, the LUMC would like to benefit from a scientific basis for gaining guidance in developing, designing and evaluating VBHC improvement dashboards, in which end-users are activated to improve their care processes.
13
1.3. Research goal
The first objective of this study is to gain insight in the facilitating and impeding characteristics and factors that influence the effectivity of VBHC dashboards. The second aim of this study is to develop a framework that describes the aforementioned characteristics and factors and which draws on a theory of effective feedback: the Feedback Intervention. The third objective is to develop a practical list of factors and characteristics that dashboards should consider and consist of, which can be used during development or after development, to determine how dashboards can be improved based upon the framework.
1.4. Research questions
The research questions that are addressed in this master thesis are as followed:
Main research question:
1. What are the factors that affect the effectivity of Value-based Health Care dashboards and how do they influence the
effectivity in their goal of increasing value for the patient?
Sub questions:
1. What are factors of Audit and Feedback dashboards within healthcare literature that showed healthcare improvements and how are these factors interrelated? 2. To what degree are the uncoveredfactors
in accordance with the Clinical
Performance Feedback Intervention
Theory and how can we draw on this theory to enhance its basis?
3. Which practical codes can be seen as recommendations that are associated with
perceived effectiveness of two
implemented VBHC dashboards in the LUMC?
1.5. Approach
This study exists of three phases (Figure 4). In the first phase, a literature study is performed. to gain
Figure 3: The VBHC trajectory for each medical condition designed by the Leiden University Medical Center. MT: Management Team.
14 insight in the factors and characteristics that are associated with the effectivity of A&F dashboards. The results of the first phase will be a basic framework that describes the different factors, characteristics and interrelations. In the second phase, the framework will be complemented by a Feedback Intervention Theory. In phase three, the framework will be tested by mapping it to two dashboards in the LUMC and determining the effectivity of both dashboards.
1.5.1. Literature phase
The main aim of the literature phase is to gain insight into the factors and characteristics of effective A&F dashboards. A literature study will be performed and based on a Grounded Theory approach, factors and characteristics will be modelled into a framework of interrelations. The codes which are the result of the Grounded Theory represent the first recommendation list.
1.5.2. Theory phase
In the second phase, the Clinical Performance Feedback Intervention Theory (CP-FIT) will be mapped with the framework. The goal is to bring the framework to a higher level by incorporating a feedback intervention theory. First the CP-FIT is studied to determine how the theory can complement the framework. After this, the CP-FIT is mapped on the framework. The goal is to develop a framework that describes the factors, characteristics, interrelations and theory of how effective A&F dashboards work and to compile a list of recommendations specific for A&F dashboards.
1.5.3. Evaluation phase
In the final phase, an initial validation of the framework has been performed by mapping two VBHC improvement dashboards in the LUMC with the dashboards and to determine the effectivity of both dashboards. Log data will be scrutinized and interviews with care professionals will be performed to determine the perceived effectivity of the dashboards The characteristics in the framework require validation across a variety of
organizations and VBHC projects to ensure that the characteristics are valid and complete and a scale can be developed on which to determine the effectiveness of the implementation. Therefore in this study, two practical situations will be assessed for which a VBHC has been developed. Both usage data and experienced effectiveness will be measured by use of the recommendations lists.
1.6. Thesis Outline
The thesis exists of five chapters excluding the introduction chapter. The first chapter describes the literature study that determines the characteristics of effective audit and feedback dashboards and describes the initial theory and model based on the results. In the second chapter, the initial theory and model are extended with the Clinical Performance Feedback Intervention Theory. The third chapter describes the mapping of the current VBHC dashboards in the LUMC with the Theory of Effective Dashboards (FOED). Also, this chapter adds some additional, VBHC specific, codes to the FOED. In the fifth and final chapter, the current VBHC dashboards are tested on their effectivity by performing interviews with care professionals.
15
16
1.7. References
1. Karanikolos M, Mladovsky P, Cylus J, Thomson S, Basu S, Stuckler D et al. Financial crisis, austerity, and health in Europe. The Lancet. 2013;381(9874):1323-1331.
2. Gray M. Value based healthcare. BMJ. 2017;:j437.
3. CBS Statline [Internet]. Opendata.cbs.nl. 2019 [cited 23 June 2019]. Available from: https://opendata.cbs.nl/statline/#/CBS/nl/dataset/84047NED/line?ts=1556194325837 4. De toekomst van evidence-based medicine — KNAW [Internet]. Knaw.nl. 2019 [cited 21 June
2019]. Available from: https://www.knaw.nl/nl/actueel/agenda/de-toekomst-van-evidence-based-medicine
5. 'Value based healthcare nog in kinderschoenen in Nederland' - Actueel - Skipr [Internet]. Skipr.nl. 2019 [cited 21 June 2019]. Available from: https://www.skipr.nl/actueel/id36472-%27value-based-healthcare-nog-in-kinderschoenen-in-nederland%27.html
6. Value-Based Health Care (VBHC) - Santeon [Internet]. Santeon. 2019 [cited 21 June 2019]. Available from: https://www.santeon.nl/vbhc/
7. Porter M, Teisberg E. Redefining health care. Boston, Mass.: Harvard Business Press; 2006.\ 8. Porter M, Lee T. The Strategy That Will Fix Health Care [Internet]. Harvard Business Review. 2013 [cited 21 June 2019]. Available from: https://hbr.org/2013/10/the-strategy-that-will-fix-health-care
9. Porter M. What Is Value in Health Care?. New England Journal of Medicine. 2010;363(26):2477-2481.
10. Schooneveldt B. Inzicht in Waarde, een How-to. Een blauwdruk voor waardegedreven verbeterdashboards in de zorg. Leiden; 2019
11. van Egdom L, Lagendijk M, van der Kemp M, van Dam J, Mureau M, Hazelzet J et al.
Implementation of Value Based Breast Cancer Care. European Journal of Surgical Oncology. 2019;45(7):1163-1170.
12. Porter M. What Is Value in Health Care?. New England Journal of Medicine. 2010;363(26):2477-2481.
13. Martin N, Massey L, Stowell C, Bangma C, Briganti A, Bill-Axelson A et al. Defining a Standard Set of Patient-centered Outcomes for Men with Localized Prostate Cancer. European Urology. 2015;67(3):460-467.
14. de Roos P, Bloem B, Kelley T, Antonini A, Dodel R, Hagell P et al. A Consensus Set of Outcomes for Parkinson’s Disease from the International Consortium for Health Outcomes Measurement. Journal of Parkinson's Disease. 2017;7(3):533-543.
15. McNamara R, Spatz E, Kelley T, Stowell C, Beltrame J, Heidenreich P et al. Standardized Outcome Measurement for Patients With Coronary Artery Disease: Consensus From the International Consortium for Health Outcomes Measurement (ICHOM). Journal of the American Heart Association. 2015;4(5).
16. Kampstra N, Grutters J, van Beek F, Culver D, Baughman R, Renzoni E et al. First patient-centred set of outcomes for pulmonary sarcoidosis: a multicentre initiative. BMJ Open Respiratory Research. 2019;6(1):e000394.
17. Wilbanks B, Langford P. A Review of Dashboards for Data Analytics in Nursing. CIN: Computers, Informatics, Nursing. 2014;32(11):545-549.
18. Schwendimann B, Rodríguez-Triana M, Vozniuk A, Prieto L, Boroujeni M, Holzer A et al. Understanding learning at a glance. Proceedings of the Sixth International Conference on Learning Analytics & Knowledge - LAK '16. 2016
19. Simpao A, Ahumada L, Rehman M. Big data and visual analytics in anaesthesia and health care. British Journal of Anaesthesia. 2015;115(3):350-356.
17
20. Mould D, D'Haens G, Upton R. Clinical Decision Support Tools: The Evolution of a Revolution. Clinical Pharmacology & Therapeutics. 2016;99(4):405-418.
21. Yigitbasioglu O, Velcu O. A review of dashboards in performance management: Implications for design and research. International Journal of Accounting Information Systems. 2012;13(1):41-59.
22. Baskett L, LeRouge C, Tremblay MC. Using the dashboard technology properly. Health Progress. 2018:89(5):16-23.
23. Stadler J, Donlon K, Siewert J, Franken T, Lewis N. Improving the Efficiency and Ease of
Healthcare Analysis Through Use of Data Visualization Dashboards. Big Data. 2016;4(2):129-135. 24. Pauwels K, Ambler T, Clark B, LaPointe P, Reibstein D, Skiera B et al. Dashboards as a Service.
Journal of Service Research. 2009;12(2):175-189.
25. Clarke S, Winson M, Terhaar M. Using Dashboard Technology and Clinical Decision Support Systems to Improve Heart Team Efficiency and Accuracy: Review of the Literature. Nursing Informatics. 2019;225:364-366.
26. Buttigieg S, Pace A, Rathert C. Hospital performance dashboards: a literature review. Journal of Health Organization and Management. 2017;31(3):385-406.
27. Dowding D, Randell R, Gardner P, Fitzpatrick G, Dykes P, Favela J et al. Dashboards for improving patient care: Review of the literature. International Journal of Medical Informatics.
2015;84(2):87-100.
18
2. Modelling factors and characteristics of effective dashboards
from healthcare literature using Grounded Theory
19
Abstract
Introduction:
Research has shown that human behaviour can be influenced extensively by feedback. Providing feedback through the use of dashboards is an often used method. However, an extensive Cochrane review showed that most audit and feedback (A&F) dashboards have little or no effect. From this it may be concluded that A&F feedback through dashboards is not always an effective tool to improve quality of healthcare by changing human behaviour. Therefore, this chapter determined the characteristics of effective A&F dashboards and described an initial framework.
Methods:
A literature study was performed to retrieve articles about effective A&F dashboards. The articles were scrutinized for characteristics and factors that may affect the effectivity of A&F dashboards. Grounded theory was used to form the gathered results into the initial theory and model.
Results:
A total of 31 articles were identified, of which eleven were included in the study. Out of the eleven articles, 188 codes were extracted. The average number of found codes per article was 17, however one article contained 51 codes, whereas two other codes contained only five codes. These codes were categorised into sixteen identified themes: Dashboard, Design, Indicator, Culture, Users, Data, Support, Barrier, Evaluation, Feedback, Implementation, Benchmarks, Usage, Development, Organisation and Effective dashboard. Each theme affects at least one other theme and is also affected by at least one theme. It was possible to form the results into a list of characteristics and factors of dashboards, the initial framework of effective dashboards and the model visualising the interrelations between the themes.
Discussion:
The literature study resulted in the first framework depicts sixteen factors that affect the effectivity of A&F dashboards. Each factor affected at least one other factor. The literature study was elaborate and thorough which contributed to the results. However, the number of A&F studies done is limited. When defining effectiveness, this implicates the positive effect of a dashboard on care processes. The initial framework of this chapter is limited to depicting characteristics and interrelations of reported effective dashboards. To enhance the framework, a theoretical foundation on effective feedback is needed.
20
2.1. Introduction
Research has shown that human behaviour can be influenced extensively by feedback. Aspects of human behaviour that are affected by feedback are motivation, learning and goal orientation [1]. Within healthcare, feedback on performance can be provided in order to improve quality [2]. An example of this type of feedback is given by Dawn et al. Dawn et al mentions the example of the Centres for Medicare & Medicaid services, which uses data collected from hospitals and post-acute care settings to publicly report and provide feedback to organizations on their provided care quality [2]. The transcending goal of reporting these measures is to incentivize quality improvement initiatives and by doing that, raising the quality of the care provided by organizations overall. As mentioned in chapter 1 the LUMC is building dashboards in light of the VBHC trajectory. The dashboards will be used as Audit and Feedback (A&F) tools to give the care professionals insight in outcomes so that quality improvement initiatives can be initiated. However, it is unclear how dashboards can optimise their way of providing feedback and thus maximising their effectiveness in activating users to initiate improvement initiatives.
Studies have shown that feedback not always leads to improvement. According to Hattie et al, feedback could be seen as one of the most powerful influences on learning and performing. However, the impact could be positive and negative [3]. This is in line with the study of Kluger at al., which states that feedback interventions have highly variable effects on performance. In some cases feedback interventions improve performance and yet in others feedback interventions have no apparent or even debilitating effects on performance [4]. If we look at the design of e-A&F systems such as the dashboards built by the LUMC, there is a lack of evidence regarding how to best design the interfaces [5]. From this we may conclude that feedback is not always an effective tool to improve quality and that e-A&F systems may not be designed to maximise usability and thus effectiveness.
Therefore in this chapter a literature study was performed to gain insight into the characteristics of effective audit and feedback dashboards. The definition of characteristic that is used in this study is: “Any property, attribute and
functionality that is part of, is inherent to, or affects the dashboard directly or surrounding mechanisms affecting the dashboard”. A broad definition of
characteristic is used in order to identify as many results in a broad spectrum of aspects that may affect the effectivity of the dashboard.
Grounded Theory was used to gather, analyse and synthesise the results of the literature study into an initial model and framework. The model describes the dependencies between different aspects that affect the effectiveness of audit and feedback dashboards at incentivizing users to start quality improvement initiatives. The literature study aims to propose the initial model and framework by answering the following research questions:
What characteristics of e-A&F dashboards
are common in effective e-A&F
dashboards?
Which overarching themes of the identified characteristics affect the effectiveness of e-A&F dashboards? Which relations occur between previously
discussed overarching themes that affects the effectiveness of e-A&F dashboards?
2.2. Methods
A literature review related to e-A&F dashboards was conducted. The goal was to gain an overview of existing evidence on the aspects and characteristics that affect the effectivity of e-A&F dashboards. In order to reach the goal, relevant literature about e-A&F dashboards (in healthcare) was assessed. From the found literature, characteristics from the dashboards were extracted. The characteristics were categorised and formed into the initial model.
2.2.1. Study inclusion criteria
Studies were included based on the type of intervention, the availability of a full text version of the study, an abstract check and a final relevance judgement. During the abstract check, at least two
21 of the following four words should be mentioned: Dashboard, Design, Implementation or Data.
2.2.1.1. Intervention
Studies were included in the review if they described an evaluation of the use and impact of e-A&F dashboards. The following definition of
dashboard was used: “a performance
measurement system that provides data on structure, process and outcome variables and incorporates the following functions: reporting on a selection of performance indicators; comparing performance to established ideal levels; and providing alerts when performance is sub-optimal to trigger action.“[6]
2.2.1.2. Study design
All study designs were included in the review as long as they took place within a healthcare organisation.
2.2.1.3. Characteristics
All reported characteristics of dashboards, organisations and users reported in reviewed literature were considered. This included qualitative and quantitative data on both measurable effects and perceptions of users.
2.2.2. Search strategy
PubMed and Google Scholar were used in order to obtain relevant articles about e-A&F dashboards in healthcare. On June 3, 2019 the following search query was used for Google Scholar:
(“Electronic Audit and Feedback System” OR “Computerized Audit and Feedback System” OR “Computerized Audit and Feedback Intervention” OR “Electronic Audit and Feedback Intervention”) AND (“Dashboard”) excluding patents and citations
On June 3, 2019 the following search query was used for PubMed:
"Audit and Feedback"[tiab] AND "dashboard"[tiab]
2.2.3. Study selection
All retrieved articles were screened on duplicates. If a full text copy was not available, the article was rejected. An abstract check determined whether the articles were relevant. An article was deemed relevant when two or more of the following words
were mentioned: Dashboard, Design,
Implementation, Data. Finally, each article was reviewed for relevance.
2.2.4. Data extraction, analysis and
synthesis
Grounded Theory (GT) was used as a bottom-up approach to extract, analyse and synthesise the data from the articles. GT defines a collection of techniques and procedures that can be in research to identify concepts and to develop theories based on the qualitative data found [7]. The most used form of GT exists of qualitative interviews [8]. However, GT can also be used to review literature [9]. In this study, literature was scrutinized by one reviewer which gathered the data. Hereafter, the data was coded in three steps conform GT: open, axial and selective coding [8, 10, 11]. During the open coding phase, citations of the articles were collected which consisted characteristics of the dashboards. The complete list of characteristics was refined into codes. Two types of codes were identified: characteristic and factor codes. Characteristic codes are that can be measured or answered with yes or no when formed into a question. Factor codes are factors that affect the dashboard and cannot be measured. During the axial coding phase, the list of codes was checked to determine whether the codes sufficiently cover the collected data. After this step, during an iterative process the codes were clustered into themes. Also, (temporary) relationships between the themes were identified. Finally, during the selective coding phase, the core themes were and all variables of all relationships that were found in the second phase were identified.
2.3. Results
2.3.1. Study selection
A total of 31 articles were identified; of these 6 were excluded due to duplicates; 4 were excluded because there was no full text available; 8 were excluded because the abstract did not meet the requirements and a further 2 were excluded due to not being relevant for this literature study, leaving 11 articles included in the final review (Table 1)
which described effective dashboards. A
22 caused the healthcare professionals to improve the quality of their delivered care.
Article number
Writer Title Study Design Year Reference
1 Patel et al. Next-generation audit and feedback for inpatient quality improvement using electronic health record data: a cluster randomised controlled trial
Cluster randomised control trial
2018 12
2 Gude et al. Electronic audit and feedback intervention with action implementation toolbox to improve pain management in intensive care: protocol for a laboratory experiment and cluster randomised trial
Study protocol 2017 13
3 Weiss et
al.
Effect of a population-level performance dashboard intervention on maternal-newborn outcomes: an interrupted time series study
Interrupted time series
2017 14
4 Reszel et
al.
Use of a maternal newborn audit and feedback system in Ontario: a collective case study
Collective case study
2019 15
5 Dunn et al. A mixed methods evaluation of the maternal-newborn dashboard in Ontario: dashboard attributes, contextual factors, and facilitators and barriers to use: a study protocol
Study protocol 2016 16
6 van Deen
et al.
Involving end-users in the design of an audit and feedback intervention in the emergency department setting – a mixed methods study
Mixed methods (interviews, surveys) 2019 17 7 Williams et al.
SMASH! The Salford medication safety dashboard Case study 2018 18
8 Jeffries et al.
Developing a learning health system: Insights from a qualitative process evaluation of a pharmacist-led electronic audit and feedback intervention to improve medication safety in primary care
Qualitative process evaluation
2018 19
9 Jeffs et al. Insights from staff nurses and managers on unit-specific nursing performance dashboards: a qualitative study
Qualitative study 2014 20
10 Tuti et al. A systematic review of electronic audit and feedback: intervention effectiveness and use of behaviour change theory
Systematic review
2017 21
11 Brown et
al.
Multi-method laboratory user evaluation of an actionable clinical performance information system: Implications for usability and patient safety
Mixed methods 2018 22
23
2.3.2. Grounded Theory coding
The bottom-up approach using GT resulted in 190 different codes (File 1). Of the 190 codes, fifteen were mentioned in multiple articles. Article 11 contained with 51 codes the most codes. Article 3 and 5 contained the least codes, each of the articles had five codes. Each article contained an average of seventeen codes.
2.3.3. Grounded Theory themes
The codes were categorised into sixteen identified overarching themes. Between the 16, 75 codes were found within themes that influenced other themes. The themes will be discussed in descending order of number of codes. The largest theme ‘Dashboard’ is comprised of 74 different codes within five subcategories: Patient Lists, Suggested Actions, Patient-level information, Functionality and Dashboard Properties. The focus of this theme lies within the properties of the dashboard and the functionality of the dashboard. Examples of codes within the Dashboard theme are: the dashboard should be online available; users should be able to drill down, suggested actions are within control of the users, patient lists can be ordered by patient-level clinical variables & patient-level information should be displayed on a single screen. Some of the codes of the ‘Dashboard’ theme affect other themes: ‘Design’, ‘Usage’, ‘Feedback’ and ‘Effective dashboard’ are the themes that are affected by the codes.
The theme “Design” is comprised of 24 codes within three subcategories: Dashboard Appearance, Dashboard Layout and Design Considerations. The subcategory Dashboard Appearance relates to codes that describe how the dashboard should look in terms of colours and how the dashboard the data should present. The subcategory Dashboard Layout refers to the layout of the interface, for example that the dashboard should have a menu on the left side of the screen. The codes that are part of the Design considerations relate to surrounding factors that are part of the design of the dashboard that may affect the effectivity of the dashboard. Some example codes of Design Considerations are: The dashboard should be designed so that the data is represented non-judgementally; the dashboard
should base the design of the intervention on theoretical and empirical evidence from A&F literature.
The theme “Indicator” is comprised of 16 codes within two subcategories: Indicator Properties and Indicator Considerations. The subcategory Indicator Properties relate to how to choose the indicators, which information should present and how the indicator should present the
information. The subcategory Indicator
Considerations relate to surrounding factors affecting the indicators effectivity. Of the sixteen codes, twelve codes were characteristic and only one code affected other themes. The code “Clinical performance summaries (i.e. Quality indicators) are a key interface component” affects the ‘Design’ and ‘Dashboard’ themes.
The theme “Culture” consists of thirteen codes within two subcategories: Practical implications and Culture Considerations. Of the thirteen codes, only three were characteristic codes. These characteristic codes were also the Practical implications for the dashboard regarding culture. The Culture Considerations exists of codes that may influence the effectivity of the dashboard but are not measurable. An example of a practical implication code is: make use of an organisation
wide engagement strategy targeting the
stakeholders across all levels of the organisation. All but one code of the theme ‘Culture’ affect other themes: ‘Organisation’, ‘Users’, ‘Effective
Dashboard’, ‘Implementation’, ‘Data’ and
‘Feedback’ are themes that are affected by the culture.
The theme “Users” consists of twelve codes, of which eight were factors and four were characteristics. The codes were divided into two
subcategories: User Properties and User
Considerations. Codes within the subcategory User Properties relate to the properties of the user that affect the dashboard. An example of a code within user properties is: Users often do not have the time, capacity or skills to interpret feedback and formulate what improvement action is necessary. Codes within the subcategory User Considerations are mostly Factor of nature and relate to how the user affects the effectivity of the dashboard. An example of a User Considerations code is:
24 Involvement of end-users in the design can lead to higher acceptability and increased motivation to change.
The theme “Data” consists of ten codes which were not divided into subcategories. All codes could be describe actions that can be performed with the possible result of an effective dashboard. Examples of the codes are: Use of good data quality, review the data weekly, data should be available online. Of the ten codes, three are characteristics. One of the characteristic codes is ‘timely data may increase effectiveness of audit and feedback’. This implies that an effective dashboard may be affected by data.
The theme “Support” is made up of eight codes which relate to how the dashboard and users should be supported to maximize the effectivity of the dashboard. Most of the codes relate to a form of training that has to be given so that the dashboard is effective. Only one of the eight codes is Factor and is not related to any form of training: ‘build awareness of the effectivity of the intervention and ease of use, to diminish resistance’. This implies that giving support in form of building awareness may effect users by diminishing their resistance to use the dashboard. The theme “Barrier” consists of five codes that describe how barriers affect the effectivity of dashboards. The codes are divided into two subcategories: barrier effects and barrier consequences. Barrier effects consists of codes that describe how and/or what barriers do to affect the effectivity of the dashboard. Barrier consequences describe the consequences of dashboard, for example: ‘Barriers to behaviour change are a common factor influencing uptake of evidence into practice’. All of the codes are Factor and thus not measurable.
Five codes make up the theme “Evaluation”, of which one is Factor and four are characteristic. The codes were not divided into subcategories because the one category would consist of one code. Two of the three codes describe how (parts of) the dashboard should be evaluated, the other codes only mentioned what parts of the dashboard should be evaluated.
The theme “Feedback” consists of five codes, relating to the properties of the feedback
given by the dashboard, as well as the review interval, the effect of format and frequency of the feedback as well as feedback considerations. Of the five codes, only one is characteristic: that the method of feedback should be reviewed quarterly. One of the codes affects another theme: the format and frequency may influence the effectiveness of audit and feedback, and thus the effectivity of the dashboard.
The theme “Implementation” is also made up of five codes which relate to how to implement a dashboard, what should be done when implementing a dashboard, and the actions during implementation which aid to the success of the dashboard. Two of the codes affects two other themes; ‘Barrier’ and ‘Usage’. The code affecting the theme ‘Barrier’ states that a pre-implementation hospital level assessment should be performed to learn about the organisational barriers that may be encountered during implementation. The code that affects ‘Usage’ describes that implementations of other IT-based interventions had encountered problems through limited user acceptance, issues with ownership and problems embedding the intervention into existing work practices. Being aware of this may affect usage by improving the embedding the use of the dashboard in the work practices.
The theme “Benchmarks” consists of four codes of which three are characteristic and one is Factor. The codes describe how benchmarks should be established, what the codes should do and when benchmarks motivate change in practice. Two codes affect other themes: ‘Dashboard’, ‘User’ and ‘Effective dashboard’. The code affecting the theme ‘Dashboard’ states that benchmarks should offer local site performance and national site performance. The code that affects ‘User’ and ‘Effective dashboard’ states that performance closer to benchmarks motivated change in practice. Thus what may be concluded is that benchmarks made up of national performance data has an effect on the dashboard because the dashboard should have this functionality, and when results of users are closer to the benchmarks, the users are more motivated to change their processes and thus result in an effective dashboard.
25 The theme “Usage” is made up of three codes, which are all characteristic. The codes describe how the dashboard should be used, how long the dashboard should be used each month and how to facilitate usage. The code that stated that the teams should spent at least four hours per month on the intervention affects the theme ‘Effective Dashboard’.
The theme “Development” consists of two codes which relate to actions that should be performed before or during development of the dashboard. Both codes affect other themes. One code states: ‘Before creating a dashboard: ensure accurate measurement and attribution to the right level of intervention’, which affects the data used by the dashboard. The code ‘During development process, review biweekly to make iterative improvements in the design’ affects the design of the dashboard.
The theme “Organisation” consists of two codes which are both Factor and affect other themes. The code that affects the theme ‘Indicator’ states that organisational priorities affect the success of sites on dashboard indicators. From this may be concluded that the organisation has an effect on the indicators that the dashboard shows. The code that affects the theme ‘effective dashboard’ states that the implementation setting in which an audit and feedback intervention takes place also affects the desired behaviour change. From this may be concluded that the organisation affects the desired behaviour change.
The theme “Effective dashboard” is the final theme, consisting of two Factor codes which both affect other themes. The code that affects the theme ‘Culture’ stated that the intervention helped to build respect, improve trust and develop relationships between the various healthcare professionals. The code that affected the theme ‘Barrier’ stated that barriers to behaviour change can be influenced by Audit and Feedback. So not only do some themes directly lead to an effective dashboard, but an effective dashboard also affects themes that lead (indirectly) to an effective dashboard.
2.3.4. Initial model
As mentioned in the previous paragraph, some of the characteristics described in the articles affected other themes than their own. To give insight in how the themes affect each other and lead to an effective dashboard, the themes and dependencies were visualised (Figure 5). Each number on the line corresponds with the article number in Table 1. Figure 5 shows a large number of dependencies between the themes, of which the effective dashboard theme being the most dependent theme. The theme ‘User’ affects the most themes with 10 themes being dependent. The theme ‘Effective Dashboard’ is the most affected dashboard with 11 other affecting themes. The model shows no theme that does not affect another theme. However, the themes ‘Indicator’, ‘Usage’ and ‘Feedback’ only affect one other theme.
The model shows the complexity and dependencies between the themes leading to an effective dashboard. Most of the themes affect other themes and even the intended result, an effective dashboard, may affect other themes. What the model also shows is that most of the lines are based on less than three articles. This may be due to the low number of articles within the audit and feedback field that focus on dashboards.
2.3.5. Initial framework
Based on the results, the following framework is composed: An effective dashboard depends on sixteen interacting themes which are interrelated. The dashboard should incorporate the 71 characteristic codes from the theme ‘Dashboard’ that will affect the ‘Effective Feedback’, ‘Design’, ‘Usage’ and ‘Feedback’. The remaining four Factor codes should be kept in mind and when possible implemented. When developing, implementing and supporting the dashboard, the five codes from the theme ‘Barrier’ in order to affect the themes ‘Usage’, ‘Indicator’, ‘Users’, ‘Feedback’ and ‘Effective dashboard’. With regards to the theme ‘Benchmarks’, the dashboard should comply to the three characteristic codes and should keep the single Factor code in mind in order to positively affect the themes ‘User’, ‘Dashboard’ and ‘Effective dashboard’. Also, the theme culture affects a great
26 number of other themes with ten Factor codes which should be kept in mind. The remaining three codes should be implemented. Together, the thirteen codes should have a positive effect on the
themes ‘Organisation’, ‘Users’, ‘Effective
dashboard’, ‘Implementation’, ‘Data’ and
‘Feedback’. The ‘Data’ theme has seven codes that have to be implemented and three Factor codes that should be kept in mind so that the themes ‘Barrier’ and ‘Effective Dashboard’ are positively affected. Of the theme ‘Design’ should 22 characteristic codes be implemented in the dashboard and two Factor codes should be kept in mind. This may have a positive effect on the themes ‘Feedback’, ‘Usage’ and ‘Effective
Dashboard’. During development of the
dashboard, one characteristic code should be implemented and one should be kept in mind in order to have a possible positive effect on the themes ‘Data’ and ‘Design’. The dashboard should be evaluated by implementing the four characteristic codes and the one Factor code should be kept in mind when evaluating. This may affect the ‘Design’, ‘Implementation’ and ‘Dashboard’ themes in a positive way. Concerning feedback, there is one characteristic code that should be implemented and four Factor codes should be kept in mind to increase the effectiveness of the dashboard. The theme ‘Feedback’ only affects the theme ‘Effective dashboard’. During implementation of the dashboard, three Factor codes should be kept in mind and two characteristic codes should be implemented in order to positively affect the themes ‘Barrier’ and ‘Usage’. The theme ‘Indicator’ has twelve characteristic codes which have to be implemented and four Factor codes which should be kept in mind, so that the themes ‘Design’ and ‘Dashboard’ may be positively affected. With regards to the organisation, two Factor codes should be kept in mind to maximize the effect of the dashboard, by positively affecting the themes ‘Indicator’ and ‘Effective dashboard’. Of the theme ‘Support’, seven characteristic codes should be implemented and one Factor code should be kept in mind in order to maximize the effectiveness of the support and thus the effect on the themes ‘Users’, ‘Usage’ and ‘Effective dashboard’. To
maximize the effectiveness of the dashboard, three characteristic codes of the theme ‘Usage’ should be implemented to affect the theme ‘Effective dashboard’. From the theme ‘Users’, four codes can be implemented and eight have to be kept in mind. This will affect multiple other themes: ‘Design’, ‘Feedback’, ‘Data’, ‘Barrier’, ‘Indicator’, ‘Design’, ‘Support’, ‘Usage’ and ‘Effective dashboard’. Finally, an effective dashboard affects two themes through two Factor codes. This should be kept in mind in order to maximize the effectiveness of the dashboard.
2.4. Discussion
The aim of this literature study was to gain insight into the characteristics of effective audit and feedback dashboards and to propose a conceptual model that visualises the aspects that affect the effectiveness of audit and feedback dashboards. The literature search initially resulted in 25 unique articles of which 11 full-text articles were included in this study. Nine articles reported the effects of the dashboards, characteristics of dashboards and the setting in which the dashboards were implemented. Two included articles were study protocols which described the study designs, characteristics of dashboards and the setting in which the dashboard were implemented. Each article was scrutinised for characteristics that may influence the effectiveness of dashboards, which resulted in 188 found codes. The number of codes per article differed greatly, with 51 codes found in article 11 and 5 codes found in article 3 and 5. Out of the 188 codes, only 15 were mentioned in more than one article. This may indicate that there is no consensus on what characteristics an e-A&F dashboard should possess and that there is no golden set of characteristics that lead to an effective e-A&F dashboard.
The codes were organised into 16 overarching themes which were established using an iterative process where the complete list of codes was checked until every characteristic was in the right theme. The themes were based on the found terms in the articles. The level of detail in the themes was determined by usability. Any more than the 16 themes would give a level of depth at
27 which a visualisation would be unclear. This choice was based on the fact that both the functionality and properties of the dashboard are things that the dashboard should (be able to) do. This may affect the visualisation of the dependencies between the themes.
The visualisation of the dependencies showed a great number of dependencies between the themes. The theme “effective dashboard” was the most affected theme. The theme “users” was the theme that affected the most themes. The visualisation is complex and shows that there are many factors to consider when developing, implementing and using an e-A&F dashboard. This also may implicate that it is complicate to build an effective e-A&F dashboard.
This literature study has the following strengths. First, the scope was clear and narrow which led to a sharp focus. This sharp focus caused a thorough search and inclusion strategy. This strategy let to a great number of found characteristics. Besides that, the characteristics were extracted from the literature but a great effort was done to make sure that the definition of the characteristics were not lost during the refinement. This led to an easy to use list of characteristics which could be followed by future studies.
This study however also has a number of limitations. The first limitation is that the number of e-A&F dashboard studied is limited. A solution could be to include other dashboard studies, but this would limit the e-A&F dimension of this study.
The second limitation is that this study is performed by one researcher. The consequence of this is that one reviewer studied the articles, which may lead to missed characteristics of dashboards. Also, by having one researcher available, the interpretation of the literature may be affected. In addition to that, it is possible that having one researcher studying the articles affects which characteristics are found, because the reviewer may learn from each article he studies. On top of that, using one reviewer may cause a bias when categorising the characteristics. Therefore, for future studies we advise to have a thorough study design with multiple researchers to prevent bias.
The third limitation is that also two study protocols were included in the literature study. The effect of the characteristics described by these articles are unsure, however we believe that they add valuable characteristics that were otherwise missed by the other articles.
When defining effectiveness, this implicates the positive effect of a dashboard on care processes. The initial framework of this chapter is
limited to depicting characteristics and
interrelations of reported effective dashboards. To enhance the framework, a theoretical foundation on effective feedback is needed. Therefore, the found 188 codes, framework and model will be merged with the Feedback Intervention Theory into the advanced framework.
28
