Master Thesis O&SC
The Use of Indicators and Effect on Improvement in Health
Care
A Systematic Review
Supervisors: Dr. H. Broekhuis
Drs. M. Gort
Lipei Zhang (1401459)
MSc. of Operation and Supply Chain
Faculty of Economics & Management and Organization
University of Groningen
ABSTRACT
Health care indicators have been developed and frequently used by hospitals, teams, and professionals. The way indicators are used can have a substantial impact on quality improvement in health care. Researchers have been challenged to specify the efficient use of indicators and to link their usages to quality improvement in health care. This study explored the literature on the use of indicators as a means to improve the quality of health care, taking also into consideration that some indicators are only used for external accountability. We proposed a 6-stage framework to exam in steps the practical use of indicators by hospitals, teams, and professionals and the impact on improvement. We suggested that to come from indicators to suggestions development, health care providers will need to follow six steps. First of all, healthcare providers need to find out the specifications of indicators the ways there are defined. The second stage is to collect data based on determined health care indicators and thirdly to turn gathered data into information by analysis. The data then need to be interpreted by giving information a meaning. As the fifty step, emphasis must be placed on the importance of empowering investigated systems to utilize feedback and response from indicators in order to modify or change responsible processes. Finally, lessons learned in such process should be identified, shared with others, and documented for use in future improvements.
Table of Contents
1. Background………..….………...………. .3
2. Change, Improvement, and Measurement……..….….…….……….4
3. Indicators………..…….….………….…….……..6
4. From Indicators to Improvement………..….…….………..….….……….7
Background
Health care is an emotionally charged part of the economy and society. Providing it can be a difficult, demanding job, and not just for clinicians. Health care employees of all sorts, from managed care executives to hospital practitioners, face high expectations, deep personal commitment, and low tolerance for error. Attention to the quality of patient care has become an important health care issue in the last decade, not only among authorities, policymakers, and managers, but also among physicians and patients (Grol, 2001).
Quality of care is a multidimensional concept (Donabedian, 1988). On the one hand, health care requires technical quality that depends on the knowledge and judgment used in determining the appropriate strategies of care and on the skill in implementing those strategies. On the other hand, quality of care also relies on the management of interpersonal relationships between care providers and patients, the organization and structure of the hospital and the way processes are managed and controlled (Donabedian, 1988). Donabedian emphasized three major elements of quality of care, namely process, structure, and outcome. Process denotes what is actually done in giving and receiving care. This is a set of activities that go on within and between practitioners and patients (Donabedian 1980; Donabedian 1988). Structure refers to the relative stable characteristics of the providers of care, of the tools and resources they have at their disposal, and of the physical and organizational settings in which they work (Donabedian 1980). Outcome is a change in a patient’s current and future health status that can be attributed to antecedent health care (Donabedian 1980).
Over the past few years much time, effort, and publicly provided money have been devoted to the health care sector, in an effort to improve the quality, effectiveness, and efficiency in services and clinical treatments and at the same time reducing costs and maintaining patient satisfaction (Zgodavova et al., 2005). In reality, however, the quality of health care is believed not to be improved as much as has been expected by patients. A number of studies around the world suggest that approximately 10% of patients admitted to the hospital suffer some kind of harm, about half of which is preventable with current standards of treatment (Thomas and Brennan, 2001; Vincent et al., 2001).
spending per person but it reports the lowest level of health care system satisfaction in the world (Blendon et al., 1990). National and international surveys conducted in the early 1990s also suggested that Americans are less than enthusiastic about their health care system. In one survey, only one-quarter of Americans responded that the health care system works even reasonably well (Jajich- Toth and Roper, 1990). Another survey pointed to significant dissatisfaction, with over 70% of respondents rating the American health care system as fair or poor (DiMatteo et al., 1995).
Besides the United States, an evaluation in a large sample of general practitioners of the use of 29 national (evidence based) guidelines and 282 indicators for primary care in the Netherlands showed that on average 67% of the recommended care was provided to patients (Grol, 2002). Patients' evaluations of primary care collected in 16 countries in Europe with an internationally standardized questionnaire showed that 30-40% of the patients thought that the organization of services could be better (50% in the United Kingdom) (Grol, 2001). These researches suggest that there is a gap between patients’ expectation as well as recommended treatment and actual performance in the health care sector. This indicates that there is a room for change and improvement in health care.
Change, improvement and measurement
Health care organizations are undergoing fundamental changes. The rapid speed of change in the health care systems, changes in science and technology, new incentive structures and technologies, moral attitudes, environmental conditions and influence of rising costs present tremendous challenges for health care providers to continuously improve the provided care. Consumers and payers, on the other hand, also demand high quality services at reasonable and affordable costs. The aim of health care organizations therefore, given above changes and patients’ demand, should be to improve quality and to build up the confidence of patients, professionals and cost payers in the quality of the context, the structures, the processes, and the outcomes of health care.
cannot often lower mortality rates or cut costs or reduce error rates on their own. They need to work on the problem of improvement together with other providers and managers of the organization. They are a team, whether they know it or not (Nolan, 1998).
Health care managers and other leaders need to recognize and acknowledge the difference between where the organization is and where it is expected to be. Medical specialists are also important actors in health care given that the quality of health care is largely ensured by their expertise. Moreover, the gap between patients’ expectation and health care must be analyzed and communicated among relevant internal and external stakeholders. As one of the possible ways to find the gap, leaders and supporting staff should listen very carefully to the people they serve – patients and families. However, the process of gathering insight into performance of health care services from these external parties can be complicated given the complexity and intangibility of the process and output and the heterogeneity of the patients that influence the performance.
Indicators
Performance is often measured by developing indicators and then putting them into practical use by health care providers to evaluate whether the organization of services, patterns of care and outcomes are co nsistent with these standards that are related to those indicators. Indicators are ‘norms, criteria, standards, and other direct qualitative and quantitative measures used in determine the quality of health care. (Mainz, 2003)’ Indicators can be generic me asures relevant to all diseases, or disease-specific measures that describe the quality of care related to a specific diagnosis (Mainz, 2003).
There are several advantages of employing indicators in health care organizations. First, they are key tools for evaluating and improving all kinds of processes, structures and outcomes. Indicators can act as a catalyst for change within the organization (Tarr, 1995). They can for example be used to identify activities within a department that adversely affect the services provided and rectify poor work processes (Breedlove, 1994). Secondly, they can help identifying activities within a department or other system that may adversely affect the quality of services provided and customer/patient satisfaction (Ondategul-Parra et al., 2004). Thirdly, they can be used to provide information for relevant stakeholders of a system that the health care system assures to deliver an adequate level of health care quality.
These different functions of indicators are also reflected in the difference made between the external and internal use of indicators (Solberg et al., 1997; Freeman, 2002). According to these authors, external indicators are used by governments, patient organizations and payers to assess the quality of care of a health care provider, and to compare that quality to the performance of other health care providers. Internal indicators, on the other hand, are used by health care providers to monitor and improve the outcomes of their care processes. For internal purposes, indicators need to be relevant for the managers, teams and professionals involved: they have to be specific to the care process at stake, aimed at its particular peculiarities and problems, and sufficiently detailed to capture the impacts of (planned or unplanned) changes to that process (Solberg et al., 1997).
operationalized by using criteria and standards but they are not the same thing. According to Campbell and colleagues (Campbell, et al., 2003), indicators do not provide definitive answers but indicate potential problems or good quality of care. Care rarely meets absolute standards, and standards have to be set according to local context and patient circumstances (Lawrence and Olesen, 1997; Marshall, et al., 2002).
Another way to categorize indicators is to link them to the concepts of structure, process, and outcome in health care. According to Donabedian (1980), the structure of care refers to the providers of care; to the tools and resources they have at their disposal and to the physical and organizational settings in which they work. Process indicators, on the other hand, are a set of activities that go on within and between practitioners and patients. They are indicators which directly measure the performance of key processes that affect patient expectations. Specific actions can be taken to improve the performance of these indicators, which in turn should improve the performance of the result measurables. Outcome indicators in health care are intended to assess health care by informing stakeholders of the outcomes they can expect from the use of health care indicators. Outcome indicators reflect the functioning of the health care system and can help to improve quality and efficiency by identifying areas of poor performance (Schneider et al., 1999; Marshall et al., 2000). However, from information provided by indicators till improvement can take a long way. The main question that rises is to find out factors that can have impacts on this way, i.e. the use of indicators and the effect of this use on improvements in health care.
From indicators to improvement
from the use of indicators to the development of improvement suggestions in health care. By conducting a systematic literature review, this 6-stage framework is used to organize and summarize our findings from literature.
In the past decade, a huge amount of indicators have been developed and utilized. However, one single indicator can hardly be applied to all organizations or used by all practitioners. It is also possible that relevant stakeholders can force health care organizations to supply specific indicators and this implies that an external party can oblige them to make use of some specific indicators. To make the best use of indicators, criteria with respect to indicators should be identified and defined. The first step of our framework includes finding out the specifications of indicators, the way they are defined, and the person/group that is responsible to measure and interpret the indicator.
Indicators can provide useful data about areas worth exploring in order to explain variations and suggest potential approaches to improve performance. These data should be gathered. The second stage of our framework is the gathering of data or the actual measurement process. Data gathering should be managed, and utilized by relevant health care providers and other stakeholders. As suggested by Mullen (Mullen, 2004), performance indicators can be used to promote learning and exploration as a basis for performance improvement, in particular by encouraging analysis of data from indicators. Thanks to the fast development of IT technologies and computer science, information provided by indicators can be analyzed in a more efficient and cost-saving way.
After analyzing the information provided by indicators, the next step follows is to find out: what does this information say to us? Information created through analysis should be interpreted by giving information a meaning. During this interpretation process, norms are set or the information is related to previous norms. This step provides discussion about many questions, e.g. who is setting the norm? Is it set to improve accountability? Did we improve?
the investigated system, and the information/interpretation reflected as a response to the feedback. Before we move on to the next step, one has to keep in mind that gathered data could be interpreted either by the system or by the subject of investigation. So the last two steps – providing feedback and interpretation of information - can therefore be switched sometimes depending on who is interpreting the data.
As the final step, feedback and responses should be utilized and explored to turning them into useful suggestions towards improving quality of care. As mentioned earlier, previous studies have discussed numerous strategies and methods that might lead to improvements. The current study, however, focuses on literature on ways to turning feedback and responses into suggestions. During this process, it is important to identify the person(s) that is (are) responsible for coming up with suggestions and those are going to provide support and the techniques that have been used.
Figure 1. 6-stage framework
Research question
Although indicators hold promise for improved quality of care, stimulating and impeding factors concerning the way and processes with respect to the efficient use of indicators are still not clear. Several researchers have examined the structures, processes, and relationships common to designing, organizing, and implementing hospital quality improvement efforts (Barsness et al., 1993; Blumenthal and Edwards, 1995; Gilman and Lammers, 1995; Shortell, 1995; Weiner et al., 1996; Weiner et al., 1997; Westphal et al., 1997; Berlowitz et al., 2003;). Few, however, have investigated the actual quality impact in relation to the use of indicators, and as far as we know, none have considered the factors and circumstances under which indicator use would lead to improved quality outcomes.
1. Determine indicators
4. Interpreting information 3. Turning data into information
2. Collecting data
6. Developing suggestions 5. Providing feedback and response
In this study we focus on the use of performance indicators by professionals, teams and hospitals in the health care sector. By conducting a systematic literature review, we exam the steps and processes took by health care organizations, teams, and professional with regard to efficient use of indicators. Our study tried to address this gap by focusing on whether and how the use of indicators improves performance. The research question of our study is:
Which factors influence the relationship between the use of indicators and the development of suggestions for improvements in health care?
As specified earlier, we suggest a 6-stage framework for answering the main research question. Our study therefore formulates the following six sub questions to help us gain a better insight with respect to the use of indicators and to organize the results from our literature review:
1) What are general criteria/specifications of the indicators that users use for selecting indicators for improvement?
2) How can data on indicators be collected?
3) How can the data provided by indicators be analyzed?
4) How can the analyzed data be communicated and interpreted to health care employees?
5) In what ways can feedback and response be provided and who should be responsible for these?
6) In what ways can suggestions for improvement be developed?
Methods
the review (Brereton et al, 2007) (see Appendix 1). The main context for the current study is to investigate the use of indicators in health care organizations and the way they lead to suggestions for improvements.
Study identification
For this systematic review, we searched for research articles written in English language, peer-reviewed jour nals that are published from January 1, 1998, through June 30, 2007. Four databases namely Business Source Premier (BSP), PubMed, Embase, and IBSS were searched using combinations of nine key words/terms (health care, hospital, improv*, indicator*, outcome, performance, measur*, implement*, and innovat*). We searched for healthcare-related subjects and for indicator and innovation-related subjects. The search strategy was directed at finding articles with at least one healthcare-related subject combined with at least one indicator-related subject. The searching key words/terms and combinations used are further explained and listed in Appendix 2.
Inclusion/exclusion criteria
As mentioned above, the literature search was restricted to post-1998. Preconditions were set that only articles with abstract and full text available were extracted for review. The list with all the available standard key words/terms was scanned in order to find out useful subjects. We also included randomized controlled trials involving health care organizations, teams, and professionals. Nevertheless, we included only those studies that objectively investigated indicator use and performance improvement in a health care setting. The initial search therefore identified the title, authors, data, and journal of articles. These elements enabled us to gain a first impression about the articles’ relevance with regard to the use of health care indicators.
medical indicators and indicator development processes; concentrate on specific illness, e.g. HIV, mental illness; investigate indicators at national level; focus on nursing care and comparison between countries. Applying these criteria narrowed down our literature selection and made sure that the main content of each article was focused on the way indicators can be used and their effects.
Data abstraction
After performing our initial search with term combinations, 3,008 articles that potentially addressed quality indicator use and implementation were located. Three reviewers (MB, MG, and LZ) then started the selection processes, which can be presented as following procedures:
• Three reviewers worked in pairs (MB & MG, MB & LZ, and MG & LZ) to independently scan 3,008 titles to select promising articles for the current study. Selections at this stage were based on the titles of the articles.
• An overall quality rating (most relevant, interesting, and not relevant) was assigned to each article and articles cited by at least one of the reviewers as most relevant were further examined by the third reviewer.
• 2,694 articles that appeared from the title not to address the study question were excluded leaving 314 articles for further review.
• One reviewer (LZ) read the 314 abstracts. Given that these two reviewers (MB and MG) are more experienced in this topic, their suggestions regarding the relevance of each article are of great value. Their opinions were integrated into the selection process through discussion. This left 110 articles for full text review.
• Full text review was done by one reviewer (LZ) and 33 articles were suggested to be selected based on beforehand- formulated criteria.
• The three reviewers went through these articles independently and consensus was formed through discussion regarding the final 18 selected articles.
observations and critical for allowing researchers to effectively communicate about their findings. In this study, validity is defined as the level of agreement with respect to the relevance of literature between three reviewers viewing the list of articles. As mentioned above, each article was rated independently by two reviewers as most relevant, interesting, or not relevant. The rate of validity is calculated as the number of articles that were rated in the same way against the total amount of articles that were assessed. The calculation suggested that 10.4 per cent (314) of articles were rated by at least two reviewers as relevant to our research, 29.6 per cent (891) were rated as interesting, and 59.9 per cent (1803) were rated as not relevant.
Figure 2. Summation of search processes and results
BSP PubMed Embase IBSS
1,031 Total
Title review by 3 pairs of reviewers
Full text review by LZ 498 621 858 3,008 314 Abstract review by LZ 110 Suggestion & opinion
from other 2 reviewers (MB & MG) Final Selection 18 Articles 33 Inclusion criteria Source: authors. Results
issues concerning indicator characteristics, its development and measurements methods while the use of indicators was merely or only partly mentioned, and not empir ically investigated.
After careful reading of the selected articles we noticed that a large amount of the literature is practice-oriented, prescriptive and optimistic of the value, the use and the effect of indicators. We found only 1 study (Mullen, 2004) that looked integral how working with indicators can result in suggestions for improvement and focus on this process. This suggests that the amount of articles that investigate the actual use of indicators and the way suggestions can be generated from the ir use is limited. Nevertheless, it is also challenging to locate relevant articles as some of the key words/terms we used are to some extent very general and they are utilized under different contexts, for example, indicator and improvement. However, among those 18 articles, a general recognition that there is a gap between the use of indicators and suggestions for health care improvement is observed. Generally speaking, there is only a notion that efficient use of indicators can lead to improvement in the quality of health care, either directly or indirectly.
Criteria
As discussed earlier, the process of transferring indicator usage into the development of suggestions starts with finding out the specifications of indicators, e.g. the way they are defined and the person that is responsible for their measurement and interpretation. The selected articles show some agreements on what makes indicators desirable and productive and the difficulties involved in their use to improve quality of health care.
• Smith and Hillner (2002) indicate that the focus of the indicators is important. According to them, some indicators capture health care more in general, other capture a specific fragment of care, e.g. a specific disease or treatment.
• Ferlie and Shortell (2001), on the other hand, argued that before putting indicators into practice, it is important to determine whether improvements in quality can be made within the current set of rules and assumptions or whether these changes require a new set of rules and new assumptions.
• According to Perera and colleagues (Perera et al., 2007), the introduced performance indicators should possess the necessary attributes to provide a measure that reliably predicts health outcomes. These attributes include the supporting evidence of best practice and established validity pertaining to the indicator, and the likelihood of this evidence being generalisable to the setting and context within which the indicator will be implemented and from which data would be collected and analyzed.
• Furthermore, Perera and colleagues (Perera et al., 2007) propose that indicators should be identified according to the rationale for the choice of indicator, such as the stated purpose of the indicator and the relevance of the indicator to current policy. • Besides, Lemieux-Charles and colleagues (Lemieux-Charles et al., 2002) argued that
it is critical to determining the level at which an indicator will be used and defining the unit of analysis.
• In addition to recognize the interdependence of the various levels, it is also important to note that the effectiveness of the indicators will be situationally determined by the problem being addressed within the context of specific organizations and environments (Ferlie and Shortell., 2001).
• Having formed specific questions, indicators should be related to the questions one wants to have answered. As suggested by Mullen (Mullen, 2004), if the purpose is exploration, explanation and learning in order to improve performance, a large number of indicators can be an advantage if presented and used appropriately. For this purpose, an ideal system might be one which initially presents a limited range of important indicators and then allows users to drill down to more detailed indicators and data, to obtain more specific information about the performance being indicated and to explore, analyze and compare the information.
• Last but not least, another necessary specification of a good performance indicator is that it must be subject to influence by the system (provider, team, professionals etc.) whose performance is being measured (Perera et al., 2007).
Measurement
Data collection starts when indicators are defined and health care providers or others are seeking for information. After reading the selected articles, we found the following remarks about measurement.
• According to Shaller (Shaller, 2004), interviews are frequently used by the improvement team to gather data. Methods for conducting interview project consisted of 1) developing the interview guide; 2) identifying specific individuals to interview; 3) scheduling and conducting the interviews; and 4) compiling and synthesizing the information gathered by topic area (Shaller, 2004).
productivity and the team’s external communication through its leader or members to evaluate effectiveness.
• Nevertheless, Shaller (2004) believes that even when sufficient data of adequate quality is collected, health care providers should also collect comparable data that is important for interpretation from other teams or departments to use as benchmarks for their own performance.
• As proposed by Twaddle and colleagues (Twaddle et al., 2007), benchmarking is a recognized method of measuring performance against established standards or best practices to produce a target against which these activities can be compared and improved. Unlike randomized controlled trials, whic h include a priori hypotheses and rigorous study design to control for confounding variables, benchmarking uses unblinded retrospective medical record review to describe performance and measures it against a set of predetermined quality indicators.
Analysis
When indicators are been used, the next step is to conduct analyses with respect to the data that are collected. The literature we have reviewed suggested many different ways and tools for analyzing indicators in health care.
• Ayers and colleagues (Ayers et al., 2005) suggested that data analysts should be assigned to identify feasible projects, to keep group focused and to transform data into useful information.
• In their study, Lemieux-Charles and colleagues (Lemieux-Charles et al., 2002) conducted an exploratory maximum likelihood factor analysis. Two criteria were used to determine the number of factors to be retained: (1) the proportion of variance explained, and (2) the requirement that all extracted factors have at least three variables with loadings above 0.45.
lower than expected performance appear in the lower left-hand quadrant of the efficiency grid.
• Besides, Rad (2006) used the Statistical Package for the Social Sciences (SPSS 11) to analyze data. According to him, appropriate statistical procedures for description and inference need to be used to make sure that analysis are valid and correct.
Interpretation
When data is collected and analyzed, the next step is to give this information a meaning. In reality, this process can be cumbersome as some situations need sharing technical expertise by showing people how to do things and exercising relatively close supervision. The selected articles gave rise to an amount of different ways the analyzed information can be interpreted and the care providers that are responsible for this process.
• As a precondition for precise interpretation, Ayers and colleagues (Ayers et al., 2005) believe that the interpreters, no matter who they are, should understand the object that has been measured. They should also have sufficient knowledge about indicators and possess adequate research and analytical expertise (Ayers et al., 2005).
• Moreover, if people are to fulfill the interpretation of information, these roles must be embedded in treasures that support the requisite information flows and specify the appropriate scope of responsibility and authority for interdependent positions (Adler et al., 2003).
• To interpret information generated from data analysis, Alexander and colleagues (Alexander et al., 2006) propose that the information should be translated into policies and practices. These policies and practices can be referred to an array of organizational policies, practices, and characteristics that influence the use of indicators and facilitate the use by increasing employees’ capabilities, motivations, and opportunities to put the indicators into practical use (Alexander et al., 2006). And according to Kirkman-Liff (2004), these may take the form of flowcharts, spreadsheets, templates, or text documents such as policies or protocols.
distribution, education, and presentation of these materials to the various health care providers (Kirkman-Liff, 2004).
• As mentioned earlier, efficiency grids were used in the study conducted by Melick and colleagues (Melick et al., 2004). According to them, surgeons were presented with efficiency grids summarizing how efficient their practice patterns were in comparison to one another. These grids showed surgeons their position in relation to surgeons in competing surgical practices and to other surgeons within their own practices.
• In addition to the efficiency grids, Melick and colleagues (Melick et al., 2004) also proposed to supply each surgeon with a packet of graphical and tabular explicit performance information at an annual meeting. A presentation highlighting differences between all surgeons as a group and surgeons affiliated with benchmark hospitals was the focal point of the annual meeting (Melick et al., 2004).
• Similarly, Kirkman-Liff (Kirkman-Liff, 2004) also suggested that the interpretation of information involves a number of material distributions involving presentations to managers and relevant health care staff. According to him, teams/groups develop the final format of the information, as well as a plan for the use of indicators. The plan includes a specification of the target audience, the feedback process, and the specific roles of various care management staff in the dissemination. Nevertheless, the details of each project also include timelines, lists of those employees to be affected by the indicator usage (Kirkman-Liff, 2004).
Feedback
• According to Smith and Hillner (2001), generating feedback is an important step in the change and improvement process. It usually includes direct feedback on performance to physicians or general feedback on system performance.
• Persaud and Nestman (2006) propose that analyzed variances should be continuously monitored when they are within benchmarking ranges and showing good trends, otherwise processes responsible for such deficiency demonstrating gaps should be reassessed.
• Moreover, Ferlie and Shortell (2001) propose that when interpretation is received by organizational leaders, they are encouraged to establish a vision for quality improvement, provide a supportive environment with the necessary resource, and insist on accountability for results.
• Ostergern (2006), on the other hand, argued that the provision of regular feedback helps to determine gaps between expected and actual outcomes. This can improve awareness within the health system of the impact of health-care activities on outcomes of interest (Ostergern, 2006).
• Besides, Kirkman-Liff (2004) believes that the managing team feedback on identified problem area by establishing priorities for care management activities, creating organizational policy, and providing a forum for the sharing of ideas and resolution of issues. This team is composed of care management System Directors, leaders of care management teams as well as representatives of important organizational areas such as Information Technology, Finance, Operations and Risk Management (Kirkman-Liff, 2004).
Response
When feedback is generated and provided, the next step of our framework is to find out the responses of the investigated system.
• Cicek and colleagues (Cicek et al., 2005) argue that when feedback is received by teams, they are encouraged to diagnose and solve their own problems, and improve their processes via management support and constructive feedback.
• According to Ferlie and Shortell (2001), groups/teams respond to the feedback by implementing the new changes taking into account the varying needs, skills, and preferences of individual members and building on each person’s comparative advantage. Nevertheless, individual clinicians, managers, and policymakers will look within themselves and decide whether when and how they want to meet the difficult challenges of changing in practices (Ferlie and Shortell, 2001).
• Besides, Adler and colleagues (Adler et al., 2003) believe that the forward- looking leaders may respond by instituting regular meetings with representatives from all the hospital departments. These meetings served as forums for potential concerns and for discussing new policies and initiatives proposed by senior management (Adler et al., 2003).
• Similarly, Cicek and colleagues (Cicek et al., 2005) propose that health care teams should call a meeting periodically to discuss the implications of the feedback, areas of improvement, possible causes of poor performance, and courses of action for improvement.
• Kirkman-Liff (2004) also suggested that the investigated group meets face-to-face and when necessary, with participants from other groups/teams joining in by conference calls. The members conduct literature reviews and Internet searches; discuss experiences in their hospitals concerning the issue; and meet with other relevant employees whose work relates to the issue (Kirkman-Liff, 2004).
a coordinator accountable for managing the entire programme is consulted (Ostergern, 2006).
Suggestion development
As the last step of our framework, feedback and responses should be utilized and explored to turning them into useful suggestions towards improved quality of care. In order to bring the expected improvements into life, effective, actionable, and reliable suggestions for improvement should be developed. By conducting the literature review, we identified several ways the suggestions for improvement are developed.
• First of all, as suggested by Ferlie and Shortell (2001), developing a culture that emphasizes learning, teamwork, and customer focus may be a ‘core property’ that health care organizations need to adopt if significant progress in quality improvement is to be made.
• Ferlie and Shortell (2001) also raise some factors that are associated with making teams effective, such as determining the right team size given the problem or task at hand; clarifying the norms that will govern team performance; clearly establishing the roles of individual team members, as well as the overall role of the team within the larger organization; implementing timely, accurate.
• Adler and colleagues (Adler et al., 2003) argue that the way suggestions can be developed depends on how the horizontal networks and communication opportunities are built. For instance, the strategic investment in infor mation systems and the change in HR systems to reward involvement in performance improvement (Adler et al., 2003).
• With respect to care leaders, Adler and colleagues (Adler et al., 2003) believe that leaders need to articulate the vision, persuade their colleagues, invest time and resources, set goals and convince others that they can be achieved, provide encouragement and appreciation, focus on the current reality, and compare performance to others.
• Nevertheless, the processes of information and knowledge collection and document development may be repeated, resulting in increasingly refined sets of materials (Kirkman-Liff, 2004 ).
• Finally, Ayers and colleagues (Ayers et al., 2005) suggest that the ability to link differences in processes to variations identified through analysis requires a central database that captures elements under study. According to them, a well-supported database management system allows for identification of organizations with the best outcomes that can serve as benchmark sites.
Conclusions
Researchers have been challenged to specify the efficient use of indicators and to link their usages with quality improvement in health care. The present study explored the literature on the use of indicators as a means to improve the quality of health care, taking also into consideration that some indicators are only used for external accountability. Among the identified 18 articles, a general recognition that there is a gap between the use of indicators and suggestions for health care improvement is observed. However, our search for relevant literature returned only a limited number of articles that actually investigate the processes from indicator usages to suggestions. As mentioned earlier, we only found 1 study done by Mullen (Mullen, 2004) that coincides with our purpose of study. We proposed a 6-stage framework to exam in steps the practical use of indicators by hospitals, teams, and professionals. In summary, this process includes determining relevant indicators; collecting data; analyzing data; interpreting information; providing feedback and response; and developing suggestions for improvement.
proposed. While it is possible to achieve a small, limited improvement by focusing on only one of the three levels for change, we also believe that the greatest and longest-lasting impact will be achieved by considering all these levels simultaneously. Data on indicators can be collected by conducting interviews and benchmarking using different criteria. These data can then be analyzed using factor analysis, efficiency grids, and statistical procedures. To interpret the analyzed data it is required to specify the appropriate scope of responsibility and authority. It is also suggested that interpreters should have sufficient knowledge and adequate research and analytical expertise. The analyzed data can be communicated to health care employees by translating them into policies and practices using flowcharts, spreadsheets, and templates. As a feedback to the information interpreted, a supportive environment and priorities for care management activities are establish as a result of continuous monitoring or reassess of the applied processes. Health care providers are encourages to implement new changes and to have regular meetings to design new policies and initiatives for improvements. Finally, suggestions for improvement can be developed by building horizontal networks and communication opportunities, linking quality measures directly to specific activities, clarifying norms and reporting processes. It is also important for health care providers to determine the right team size and to have a well- supported database management system in place. Last but not least, a culture that emphasizes learning, teamwork, and customer focus is believed to play a vital role as well.
Reference
18 articles for the literature review
1
Ferlie E. B. and Shortell S. M. (2001), “Improving the quality of health care in the United Kingdom and the United States: A framework for change”, The
Milbank Quarterly; Vol. 79. No. 2. pp. 281-315.
2
Smith, T. J. and Hillner, B. E. (2001), “Ensuring quality cancer care by the use of clinical practice guidelines and critical pathways”, Journal of Clinical
Oncology; Vol. 19, No. 11, pp. 2886-2897.
3
Lemieux-Charles, L., Murray M, Baker G. R., Barnsley J., Tasa K. and Ibrahim S. A. (2002), “The effects of quality improvement practices on team effectiveness: a mediational model”, J. Organiz. Behav. 23, pp. 533-553.
4
Adler P. S., Riley P, Kwon S. W., Signer J., Lee B. and Satrasala R. (2003), “Performance improvement capability: keys to accelerating performance improvement in hospitals”, California Management Review; Vol. 45, No. 2. pp. 12-33.
5
Desai J., Solberg L., Clark C., Reger L., Pearson T., Bishop D., Roberts M., Sniegowski R. and O’Connor P. (2003), “Improving diabetes care and outcomes: the secondary benefits of a public health- managed care research collaboration”, J. Public Health Management Practice; S36-S43.
6
Shaller D. (2004), “Implementing and using quality measures for children’s health care: perspectives on the state of practice”, Pediatrics; Vol. 113 No. 1. pp. 217-227.
7
Kirkman-Liff B. (2004), “The structure, processes, and outcomes of Banner Health’s corporate-wide strategy to improve health care quality”, Q. Manage
Health Care; Vol. 13, No. 4, pp. 264-277.
8 Mullen P. M. (2004), “Using performance indicators to improve performance”, Health Services Management Research; Vol. 17, pp. 217-228.
9
pp. 31-40.
10
Cicek M. C., Koksal G. And Ozdemirel N. E. (2005), “A team performance measurement model for continuous improvement”, Total Quality Management; Vol. 16, No. 3, pp. 331-349.
11
Ayers L. R., Beyea S. C., Godfrey M. M., Harper D. C., Nelson E. C. and Batalden P. B. (2005), “Quality improvement learning collaboratives”, Q
Manage Health Care; Vol. 14, No. 4, pp. 234-247.
12
Persaud D. D. and Nestman L. (2006), “The utilization of systematic outcome mapping to improve performance management in health care”, Health Service
Management Research; Vol. 19, pp. 264-276.
13
Ostergern K. (2006), “The institutional construction of consumerism: a study of implementing quality indicators”, Financial Accountability and Management; Vol. 22, No. 2, pp. 179-205.
14
Rad A. M. M. (2006), “The impact of organizational culture on the successful implementation of total quality management”, The TQM Magazine; Vol. 18, No. 6, pp. 606-625.
15
Alexander J. A., Weiner B. J., Shortell S. M., Baker L. and Becker M. P. (2006), “The role of organizational infrastructure in implementation of hospitals’ quality improvement”, Hospital Topics; Vol. 84, No. 1, pp. 11-20.
16
Twaddle M. L., Maxwell T. L., Liao S., Usher B. M. and Cuny J. (2007), “Palliative care benchmarks from academic medical centers”, Journal of
Palliative Medicine; Vol. 10, No. 1, pp. 86-98.
17
Alexander J. A., Weiner B. J., Shortell S. M. And Baker L. C. (2007), “Does quality improvement implementation affect hospital quality of care?”, Hospital
Topics; Vol. 85, No. 2, pp. 3-12.
18
Additional references
• Adler, P. and Bryan B. (1996), “Two types of bureaucracy: enabling and coercive”,
Administrative science Quarterly; Vol. 41, No. 1. pp. 61-89.
• Aiken, L.H., J. Sochalski, and E.T. Lake. (1997), “Studying outcomes of organizational change in health services”, Medical Care ; 35: NS6-18.
• Balas, E.A.; Boren, S.A.; Brown, G.D.; Ewigman, B.G.; Mitchell, J.A. and Perkoff, G.T. (1996), “Effect of physician profiling on utilization. Meta-analysis of randomized clinical trials”, J Gen Intern Med; 11: 584-90.
• Bandura, A. (1986), “Social foundations of thought and action: a social cognitive
theory”, Prentice-Hall.
• Barsness, Z. I., Shortell, S. M. and Gillies, R. R. (1993), “The quality march: National survey profiles quality improvement activities”, Hospital Health Network; 67 (24): 40–42.
• Berlowitz D. R., G. J. Young, E. C. Hickey, D. Saliba, B. S. Mittman, E. Czarnowski, B. Simon, J. J. Anderson, A. S. Ash, L. V. Rubenstein, and M. A. Moskowitz. (2003), “Quality improvement implementation in the nursing home”, Quality improvement
implementation in Health Services Research; 38 (1): 65–83.
• Berwick, D. M., James, B. and Coye, M. J. (2003), “Connections between quality measurement and improvement”, Med Care; Vol. 41 (1 Suppl):I30-8.
• Bickell, N. A. and Young, G. J. (2001), “Coordination of care for early-stage breast cancer patients”, Journal of General Internal Medicine ; Vol. 16, pp. 737-42.
• Blendon, R.J., Leitman, R. Morrison, I. and Donela n, K. (1990), “Satisfaction with health systems in ten nations”, Health Aff; Vol. 9, pp. 185-192.
• Blumenthal, D., and J. N. Edwards. (1995), “Involving physicians in total quality
management: Results of a study”, In Improving clinical practice: Total quality
management and the physician, ed. D. Blumenthal and A. C. Scheck, 229–266. San Francisco: Jossey-Bass.
• Breedlove, T. H. (1994), “Measuring the impact of quality improvement efforts”,
• Brereton, P., Barbara A. Kitchenham, Budgen, D., Turner, M., and Khalil, M. (2007), “Lessons from applying the systematic literature review process within the software engineering domain”, The Journal of Systems and Software ; 80, 571-583.
• Camp, R.C. (1989), “Benchmarking: the search for industry best practices that lead
to superior performance”, Milwaukee, WI: Quality Press.
• Campbell, S.M.; Braspenning, B.; Hutchinson, A. and Marshall, M.N. (2003), “Improving the quality of health care: research methods used in developing and applying quality indicators in primary care”, BMJ; Vol. 326.
• Campbell, S. M., Roland, M. O. and Buetow, S. A.(2000), “Defining Quality of Care”. Soc Sci Med; 51 : 1611-25.
• Dahlgaard, J. J., Kristensen, K. and Kanji, C. K. (1998), “Fundamentals of Total
Quality Manafiement”, London: Chapman & Hall.
• DiMatteo, M. R., Shugars, D.A., McBride, C. A. and O'Neil, E. H. (1995), “Americans’ views of health professionals and the health care system”, Health
Values; Vol. 19, pp. 23-29.
• Fargason, C.A., and C.C. Haddock. (1992), “Cross- functional, integrative team decision-making: essentials for effective QI in health care”, Quality Review Bulletion; 7: 157-63.
• Ferlie, E.B., L. Fitzgerald, and M. Wood. (2000), “Achieving change in clinical practice”, Journal of Health Services Research and Policy; 2(April 5): 96-102. • Fox, R., Mazmanian, P. and Putnam, R. W. (1989), “Changing and learning in the
lives of physicians”, New York: Praeger.
• Freeman, T. (2002), “Using performance indicators to improve health care quality in the public sector: a reviewof the literature”, Health Services Management Research; Vol. 15, pp. 126–37.
• Fried, B., T. Rundall, and S. Topping. (2000), “Groups and teams in health services organization”, Health Care Management: Organization Design and Behavior (4t h ed.); Albany, N.Y.: Delmar.
• Gift, R.G. and Mosel, D. (1994), “Benchmarking in health care: A collaborative
approach” New York, NY: American Hospital Publishing.
• Gilman, S. C., and J. C. Lammers. (1995), “Tool use and team success in CQI: Are all tools created equal?”, Quality Management in Healthcare; 4 (1): 56–61.
• Green, L. W., Eriksen, M. P. and Schor, E. L. (1988), “Preventive practices by physicians: behavioural determinants and potential interventions”, Am J Prev Med; Vol. 4(suppl 4):101–7.
• Grimshaw, J.M.; Thomas, R.E.; MacLennan, G.; Fraser, C.; Ramsay, C.R.; Vale, L.; Whitty, P.; Eccles, M.P.; Matowe, L.; Shirran, L.; Wensing, M.; Dijkstra, R. and Donaldson, C. (2004), “Effectiveness and efficiency of guideline dissemination and implementation strategies”, Health Technology Assessment ; 8(6).
• Grol, R. (2001), “Improving the quality of medical care: building bridges among professional pride, payer profit, and patient satisfaction”, JAMA, Vol. 286, No.20. • Grol, R. (2002), “Improving the quality of medical care”, JAMA; Vol. 286, pp.
2578-85.
• Hackman, J. R. (1990), “Groups that work (and those that don’t)”, San Francisco: Jossey Bass.
• Hassan TB, Jagger C, and Barnett DB (2002). “A randomised trial to investigate the efficacy of magnesium sulphate for refractory ventricular fibrillation”. Emerg Med J. , 19:57–62.
• Institute of Medicine: Crossing the Quality Chasm: A New Health System for the 21st Century 2000.
• Jajich-Toth, C. and Roper, B. W. (1990), “Americans’ views on health care: a study in contradictions”, Health Aff; Vol. 9, pp.149-157.
• Janis, I. L. (1972). “Group think: A study of foreign policy decisions and fiascos”, Boston: Houghton Mifflin.
• Kaplan, R. S. and Norton, D.P. (1992), “The balanced scorecard – measures that drive performance”, Harvard Business Review; pp. 71-9.
• Kitchenham, B., (2004). “Procedures for Undertaking Systematic Reviews. Joint
Technical Report”, Computer Science Department, Keele University and National
• Lawler, E. E., Ledford, G., and Mohrman, S. (1992). “Employee involvement and
total quality”. San Francisco: Jossey Bass.
• Locke, E. A. (1968). “Toward a theory of task motivation and incentives”,
Organizational Behavior and Human Performance, 3, 157–189.
• Mainz, J. (2003), “Defining and classifying clinical indicators for quality improvement”, International Journal for Quality of Health Care; 15:523-30.
• McGlynn, E. A., Asch, S. M., Adams, J., Keesey, J., Hicks, J., DeCristofaro, A. and Kerr, E. A. (2003), “The quality of health care delivered to adults in the United States”, N Engl J Med; Vol. 348, pp. 2635-2645.
• McKinnon, S. M and Bruns, W. J. (1992), “The Information Mosaic”, Boston: HBS Press.
• Mitchell, L., S. Fife, A.A. Chochia, et al. (1996), “Three teams: improving thrombolytic therapy”, Joint Commission Journal of Quality Improvement; 22(6): 379-90.
• Mugford, M., Banfield, P. and O’Hanlon, M. (1991), “Effects of feedback of information on clinical practice: a riview”, BMJ; Vol. 303, pp. 398-402.
• Nolan, T. (1998), “Understanding medical systems”, Annals of Internal Medicine ; Vol. 128, No. 4, pp. 293-8.
• Ondategul-Parra, S., Bhagwat, J.G., Zou, K.H., Gogate, A., Intrlere, L.A., Kelly, P., Seltzer, S.E. and Ros, P.R. (2004), “Practice management performance indicators in academic radiology depoartments”, Health Policy and Practice; Vol. 233, No. 3, pp. 716-722.
• Pettigrew, A., E. Ferlie, and L. McKee. (1992), “Shaping strategic change”, London: Sage.
• Plsek, P. E. (1997). “Collaborating across organizational boundaries to improve the quality of care. ” Am J Infect Control; 25: 85-95.
• Prochaska, J. O., DiClemente, C. C. and Norcross, J. C. (1992), “In search of howpeople change. Applications to addictive behaviors”, Am Psychol; Vol. 47: 1102–14.
management and the physician, ed. D. Blumenthaland and A. C. Scheck, 205–228.
San Francisco: Jossey- Bass.
• Shortell, S. M., Bennett, C. L. and Byck, G. R. (1998), “Assessing the Impact of Continuous Quality Improvement on Clinical Practice: What It Will Take to Accelerate Progress?”, Milbank Q, 76:593–624.
• Shortell, S.M., J.E. Zimmerman, and D.M. Rousseau. (1994), “The performance of intensive care units: does good management make a difference?”, Medical Care; 32(5): 508-25.
• Simons, R. (1995), “Levers at Control”, Boston: HBS Press.
• Solberg, L. I., Mosser, G. and McDonald, S. (1997), “The three faces of performance measurement: improvement, accountability, and research”, Joint Commission
Journal on Quality Improvement; 23 : 135–47.
• Thomas, E. J. and Brennan, T. (2001), “Errors and adverse events in medicine: an overview”, In: Vincent CA, ed. Clinical Risk Management. Enhancing Patient Safety. London: BMJ Publications; 2001:31–44.
• Vincent, C., Neale, G. and Woloshynowych, M. (2001), “Adverse events in British hospitals: preliminary retrospective record review”, BMJ ;322:517–519.
• Weiner, B. J., J. A. Alexander, and S. M. Shortell. (1996), “Leadership for quality improvement in health care: Empirical evidence on hospital boards, managers and physicians”, Medical Care Research and Review; 53 (4): 397–416.
• Weiner, B. J., S. M. Shortell, and J. A. Alexander. (1997), “Promoting clinical involvement in hospital quality improvement efforts: The effects of top management, board, and physician leadership”. Health Services Research; 32, (4): 491–510.
• West, M. A. (2002), “Sparkling fountains or stagnant ponds: An integrative model of creativity and innovation implementation in work groups”, Applied Psychology: An
International Review ; 51, 355–387.
• Westphal, J. D., R. Gulati, and S. M. Shortell. (1997), “Customization or conformity? An institutional and network perspective on the content and consequences of TQM adoption:, Administrative Science Quarterly; 42 (2): 366–94.
Appendix 1
Systematic literature review processes
Appendix 2
Truncation is represented by an asterisk (*). It is used in a way that the root of a search term is entered and the ending is replaced with an *. The databases enable us to find all forms of that word. For instance, type improv* to find the words improve, improves,
improving or improvement. “AND” is used to combine search words/terms so that each
search result contains all of the terms. For instance, health care AND outcome finds articles that contain both terms.
Key words/terms used: • Health care • hospital • Improv* • Indicator* • Outcome • Performance • Measur* • Implement* • Innovat* Combinations used in BSP:
• Hospital AND improv* AND indicator* • Hospital AND improv* AND outcome • Hospital AND improv* AND performance • Hospital AND improv* AND measur* • Hospital AND implement* AND indicator* • Hospital AND implement* AND outcome • Hospital AND implement* AND performance • Hospital AND implement* AND measur* • Hospital AND innovat* AND indicator* • Hospital AND innovat* AND outcome • Hospital AND innovat* AND performance • Hospital AND innovat* AND measur* Combinations used in PubMed:
• Hospital AND indicator* • Hospital AND performance • Hospital AND measur* • Hospital AND outcome
• Hospital AND improv* AND indicator* • Hospital AND improv* AND measur* • Hospital AND improv* AND outcome • Hospital AND innovat* AND indicator* • Hospital AND innovat* AND measur* • Hospital AND innovat* AND outcome • Hospital AND implement* AND indicator* • Hospital AND implement* AND measur* • Hospital AND implement* AND outcome Combinations used in Embase:
• Hospital AND outcome
