Measuring the Results of the Implementation of an Operating Room Information System

(1)

Measuring the Results of the Implementation of an Operating Room Information System

MSc BA Business & ICT Martijn Huitema

August 2006

(2)

Title: Measuring the Results of the Implementation of an Operating Room Information System

Name: Martijn Huitema

Student nr. S1496859

Email: martijn_huitema@hotmail.com 1^st supervisor: Dr. E. Harison

2^nd supervisor: Prof. dr. E.W. Berghout University of Groningen

Faculty of Business Administration

August 2006

(3)

PREFACE

In a period of six months, I completed my Master thesis and thereby graduating in the Masters program in Business & ICT. The past six months have been an intense period of constructing a research that serves both scientific use and practical application. I would like to thank ChipSoft BV for giving me the opportunity to conduct research on their products and providing access to hospitals. I hope the results of this research will provide new value and knowledge for ChipSoft.

I would like to thank Karijn Marsman (ex-ChipSoft consultant) and Thierry van Groen (ChipSoft programmer) for their supervision, support and helpful comments during the research. Further, I would like to thank all the OR managers from the hospitals who dedicated time and effort to contribute to the research via interviews and questionnaires.

I would especially like to thank Dr. Elad Harison (Asst. Professor, University of Groningen) for his supervision during these six months. He helped me in writing and structuring the research. Further, he provided me improvements to my writings in the English language. He supported and inspired me during the difficult moments.

Further, I want to express my gratitude to my fellow students Tjeerd Bosma and Mark

Walstra for their support and the interest they have shown in my research. Last but not least, I would like to thank my parents and my sister for their support throughout my student years.

They have been great motivation for me.

Martijn Huitema Drachten, August 2006

(4)

ABSTRACT

The healthcare sector is an area that has a rich selection of IT systems and equipment. The development of these systems has made a big impact on the quality of care. In this kind of field, it is not straightforward to point out to which extent software contributes to the quality of care.

The purposes of OR-systems are realizing an optimal planning for the OR, and delivering relevant management-information. An optimal OR-planning involves patients, beds, OR’s, anaesthesia, surgeons, materials and more. On the other hand, management information is needed to gather more insight in OR productivity, efficiency and quality. At the moment, it is not clear to see to which extent OR-systems achieve these goals, or how to measure them.

This research aims at measuring the influence of information technology on the performance of a medical organization. This will be achieved by the construction of a model of weighed performance indicators that indicate improvements of quality, efficiency and performance.

The model has been used in a case study for the OR (Operating Room) information system of the Dutch company ChipSoft.

ChipSoft was founded in 1986, developing a small information system that supports the daily practice of a surgeon. Throughout the years ChipSoft has steadily grown into the biggest player in the Dutch healthcare information systems market. They have developed a hospital- wide information system (CS-ECIS), that consists of integrated modules for every possible hospital department. The modules are also separately available. The module developed for OR management is CS-OR.

CS-OR is an operating room management system that supports the total operative process;

from (waiting list) registration to planning and implementation, right up to the financial settlement. All data necessary for an efficient planning is accurately recorded and processed by CS-OR, so that optimum use can be made of the OR.

The methods being used in this research are literature survey and field research. The literature survey is performed in order to reveal methods and indicators for a) measuring the results of implementing IT applications in general, b) measuring hospital performance and c) measuring the effects of medical IT applications on hospital performance. The literature survey has provided a list of indicators, that can be used for measuring the influence of IT on OR

performance. During the field study, additional performance indicators have been revealed, by interviewing hospital experts.

A questionnaire that was sent to all Dutch hospitals provided insight in the use of OR-systems in Dutch hospitals and performance measurement. Further purpose was evaluating the relative importance of the indicators, found during the literature survey and field research. From the questionnaire can be concluded that the average occupation rate of the Dutch hospitals is 85%. Hospital experts believe that they can improve this to 90% by implementing an OR- system. Currently, most respondents are satisfied with their systems. On average, they give their system a 6,8. Further did the questionnaire point out that there is a relation between occupation rates and the level of satisfaction. Hospitals with a high occupation rate are highly

(5)

satisfied with their system, whereas low occupation rates are associated with low levels of satisfaction.

The hospital professionals have evaluated the relevance and importance of the performance indicators in the questionnaire. Based upon this information a model has been developed for comparing and selecting OR-systems. The model proposes different weights for each indicator, based upon the importance being stated by hospital professionals. Hospitals can grade the systems based on the indicators. By multiplying the weight, a total value can be calculated.

Grades Total

Indicator Weight A B A B

Productivity

Achieving production agreements 4,94 % Number of performed surgeries 4,60 %

Average time per operation 4,30 %

Variation of time per operation 4,16 % Cost savings

Costs of used materials 4,34 %

Personnel costs 4,07 %

Facility costs 3,83 %

IT costs 3,42 %

Quality

Number of cancelled operations 4,98 %

Patient satisfaction 4,84 %

Number of planning mistakes 4,71 %

Average patient waiting time 4,46 %

Total time “patient in hospital” 4,28 %

Mortality rate 4,01 %

Employee satisfaction

Sickness absence 4,28 %

Days requiring work in overtime 4,16 % Satisfaction about the planning 4,08%

Satisfaction about the OR system 3,96%

Efficiency

OR occupation rate 5,05 %

Number of surgery hours outside office hours. 4,74 % Emergency operations within the OR schedule 4,60 %

Bed occupation 4,37 %

Time needed to make an OR-planning 4,17 %

TOTAL 100%

Figure A: Model for assessing and selecting OR-systems

The performance indicators have been measured in a case study for ChipSoft. The objective indicators have been measured by collecting data from the CS-OR in two hospitals.

Subjective indicators were measured by conducting interviews with OR managers of 9 hospitals. The main results are the following.

(6)

CS-OR provides most advantages on the administration of data. Double administration is reduced to a minimum, paper flows are reduced, and a lot of time is saved by the secretaries.

Processing information and its distribution throughout the hospital is performed faster then before. Several managers have calculated the time reductions CS-OR has provided them.

Secretaries save about 10% of their workday by the reduction of manual input. All the hospitals are satisfied with CS-OR. Its usability and user-friendliness are highly regarded.

Employees see the advantages of the system and are willing to use it.

One of the issues to improve are the possibilities for gathering management information. This requires specific IT knowledge and is difficult for most OR managers.

The subjective measures implied that by implementing CS-OR, the productivity of the ORs increases with 10%. The objective measures have indicated an increase of the occupation rate by 10%. Further the number of treated patients increased by 5% and the length of transition times has been reduced to 5% of the total operating time.

(7)

1 INTRODUCTION AND RESEARCH

1.1 Introduction

In the past twenty years, information technology has changed the world dramatically. In different fields such as agriculture, industry and community services, IT has played a major role in improving products and services. The same occurs for healthcare.

The healthcare sector is an area that has a rich selection of IT systems and equipment. The development of these systems has made a big impact on the quality of care. In this kind of field, it is not straightforward to point out to which extent software contributes to the quality of care.

This research aims at measuring the influence of information technology on the performance of a medical organization. This will be achieved by the construction of a model of weighed performance indicators that indicate improvements of quality, efficiency and performance.

The model has been used in a case study for the OR (Operating Room) information system of the Dutch company ChipSoft.

This chapter provides an introduction of the research. In section 1.2 the organization ChipSoft and its products are described. Section 1.3 deals with the problem statement of the research, and section 1.4 explains the structure of the research.

1.2 Research Setting 1.2.1 ChipSoft

ChipSoft was founded in 1986, developing a small information system that supports the daily practice of a surgeon. Throughout the years ChipSoft has steadily grown into one of the biggest players in Dutch healthcare information systems. In 1994, ChipSoft introduced its first hospital information system based on DOS. In 2001, ChipSoft played a leading role with the launch of the first integrated Windows-based care information system: the Electronic Care Information System (CS-ECIS). In 2005, ChipSoft has become market leader in information technology for the Dutch healthcare sector, and opposite to their competitors, still manages to remain a purely Dutch company.

Currently ChipSoft is located at two offices; Amsterdam (120 employees) and Drachten (30 employees). The office in Drachten hosts mainly programmers, whereas most consultants are based in Amsterdam. The organization ChipSoft consists of four departments:

1) Research & Development: Development and maintenance of current modules and development of new modules based upon new trends in the market.

2) Implementation and Support: Implementation of soft- and hardware at hospitals and delivering support to the clients.

3) Marketing & Sales: Promoting and selling the products. Arranging presentations, seminars, demonstrations and keeping contacts informed.

4) Secretary / Administration: Supporting all departments with administrational tasks.

(10)

These four departments are displayed in the organization diagram of ChipSoft, as presented in figure 1.1.

Figure 1.1: Organization Diagram ChipSoft (Source: internal documents)

At the moment, over a hundred hospitals in the Netherlands are using one or more modules of ChipSoft software. Thirty of those hospitals have implemented CS-ECIS, the core hospital information system. The first international implementations have taken place at the

Netherlands Antilles (Aruba) and Belgium (Gent).

The enormous growth of ChipSoft over the past five years is characterized by the turnover, as displayed in figure 1.2.

Company Turnover

13 16,9

22

28,5

37

48,2

0 10 20 30 40 50 60

2000 2001 2002 2003 2004 2005

Millions (€)

Figure 1.2: Turnover ChipSoft 2000-2005 (Source: internal presentation)

(Sources: http://www.chipsoft.com, internal documents and presentations)

(11)

1.2.2 Software

Over the years, the ChipSoft modules have diversified intensely. From a small system that supports the daily practice, to 29 modules that support almost every practical aspect of hospital administration. The core product is CS-ECIS, a hospital-wide information system.

CS-ECIS consists of a set of basic modules, which are essential for a general hospital. In addition, every other ChipSoft module can be integrated with CS-ECIS. Further, CS-ECIS supports integration with software from other suppliers. Otherwise, ChipSoft modules can be integrated with hospital information systems from other suppliers.

Table 1.1 displays the most common modules from ChipSoft.

Module Name: Description:

CS-Patient Registration of all types of patient data like name, address, blood type, medication, medical history, insurance etc.

CS- Admission Registration of patients’ admission to hospital, and planning of hospital beds.

CS-Department Registration of patients on their departments, transferring patients.

CS-Agenda Registration of appointments and tasks for hospital employees.

CS-Episode Registration of Diagnosis Treatment Combination (DBC), a new Dutch method for billing treatments.

CS-Invoice Invoicing system for creating, controlling and accreditation of invoices.

CS-Order Order management system, an order is defined as a request for information like lab-results, röntgen, etc.

CS-OR The management of the operating room, planning of patients and staff.

CS-Preop Registration of pre-operative screening data, such as allergies, anaesthesia type etc.

CS-Document Creating standardized documents such as letters, reports and request forms.

CS-Sterile Registration of materials, nets, and articles for the central sterilization department.

CS-

Reportgenerator

Creating reports easily by selecting databases, data and queries.

CS-Röntgen Registration of röntgen data, integration with PACS (Picture Archiving Communication System).

Table 1.1 CS-modules

The strengths of the CS software lay in the integration of information of all hospital

departments. The information can be viewed hospital-wide, if permissions are granted, and has to be entered into the system only once. This saves a lot of administrative work and makes communication faster and easier. Even more, the ChipSoft software encourages a patient- oriented organization of hospital processes. The integrated software stimulates a fast routing of the patient through the hospital departments. Actions regarding a patient can be streamlined by using the integrated software.

(12)

1.2.3 Dutch OR-system Market

The OR-systems market is a rather dynamic and competitive market. At the moment, the biggest players on the market are ChipSoft and iSOFT. iSOFT has implemented their OR- system in 50 hospitals in the Netherlands. iSOFT became market leader in 2004 by taking over Torex Hiscom. ChipSoft has 45 implementations of the OR system CS-OR, of which 30 are included in CS-ECIS. McKesson is a medium player in the Dutch market, their hospital information system X/Care is implemented in 25 Dutch hospitals. BLOKplan is the OR planning module of Dutch company Lievestro, they are a small player in the market with six implementations in the Netherlands. Getronics PinkRoccade has implemented its system iMedOne in 170 German hospitals, but they are a small player in the Netherlands. iMedOne is not an OR-system but an EPD (Electronic Patient Dossier). Meierhofer is a German producer of medical software with 80 implementations in Germany, but only one in the Netherlands.

Another small player is the German SAP-based system i.s.h.med with seven Dutch implementations. This system is a large player on the German market with 149 implementations.

With CS-ECIS, ChipSoft plays an important role on the Dutch market, covering a market share of 35%. Figure 1.3 displays the growth in market share of some of their modules over the past five years.

Figure 1.3: Market share % per module (Source: internal presentation)

(Sources: http://www.chipsoft.com, http://www.isoft.nl, http://www.mckesson.nl, http://www.lievestro.nl, http://www.getronicspinkroccade.nl, http://www.meierhofer.de, http://www.ishmed.de, internal documents and presentations)

1.2.4 CS-OR

The module being the main subject of this thesis is CS-OR. CS-OR is an operating room management system that supports the total operative process; from (waiting list) registration to planning and implementation, right up to the financial settlement. All data necessary for an

Marketshare per module

0 10 20 30 40 50 60 70

ECIS Invoice Agenda OR Preop

market share %

2000 2005

(13)

efficient planning is accurately recorded and processed by CS-OR, so that optimum use can be made of the OR.

The operating room management system can be fine-tuned to fit the care institution’s workflow. Waiting list management, planning support, preoperative, real time and

postoperative input of performed operations, stock control and planning advice are the most important functionalities. The underlying thought of CS-OR is that operating rooms, materials and personnel are optimally used. CS-OR is supported by a conflict alarm that emerges when an operating room or a specialist is double booked. The Reportgenerator ensures clear insight into the average incision times, priority treatments and costs of an operation. This provides management with an important tool for fine-tuning policy and working practice.

The first, important supporting task for CS-OR regards the waiting list; the registration of the patient data, the expected procedure and the specialist carrying out the procedure. These data have to be registered one time only in the system, and are available to all other departments.

After the registration on the waiting list, the operation can be planned. CS-OR supports planning based on the session schedules defined by the care institution. A session schedule is a schedule that states the dedicated operation time for every specialist on each day. The operations are, depending on the specialty or the specialist, planned in one of these OR sessions. The average personal incision times of the specialists are taken into account when calculating planned operation times. This first planning can subsequently be altered. After agreement of the planning by the specialists, a definitive OR program is made. The following step is the recording of information during the operation. All the people present in the

operating room are registered as well as the articles and nets used, with the charge numbers, where necessary. In addition, an accurate time registration is kept. CS-OR includes several time registration moments as depicted in table 1.2.

Time Registration Moments Call patient

Patient on OR complex Patient in Holding Patient on OR Start Anaesthesia

End Anaesthesia (patient is asleep) First incision

Final stitch Start Anaesthesia

End Anaesthesia (patient awakes) Patient out OR

Patient in Recovery To Department

Table 1.2: Time registration moments of CS-OR (Source: www.chipsoft.com)

These registration moments come to life in the so-called airport-screen (as depicted on figure 1.4). On this screen, the OR’s can be viewed, with the operations on a timeline (like a

GANTT-chart). The colour of the operation depicts in which phase (registration moment) the operation is. The airport-screen also displays the planned times (grey) versus the actual times.

In some hospitals, these airport-screens can be viewed everywhere on the OR-complex.

(14)

Figure 1.4: Airport screen

Apart from a registration function, CS-OR also has a verifying function. Incorrectly entered data are immediately detected. After registration of all data has taken place, the financial details can be settled. Every type of operation has a financial code, the so-called CTG code.

By linking these codes to the financial system, the wages for the specialists can be calculated.

The major benefit of CS-OR is the way in which the operating room management system supports the user during registration, planning and data verification. In addition, exact insight into the average incision times for procedures for each surgical specialist is invaluable for facilitating efficient planning of OR space, personnel and material.

Afterwards, management reports can be constructed to view statistics of the OR-processes.

For example: OR-occupation percentages, the number of operations per specialist, and average surgery times per specialist.

(Sources: http://www.chipsoft.com, internal documents and presentations)

1.3 Problem Definition 1.3.1 Introduction

CS-OR is implemented in more then 40 hospitals in the Netherlands. The goals of CS-OR are realizing an optimal planning for the OR, and delivering relevant management-information.

An optimal OR-planning involves patients, beds, OR’s, anaesthesia, surgeons, materials and

(15)

quality. At the moment, it is not clear for ChipSoft to see to which extent CS-OR achieves these goals, or how to measure them.

In general, there is hardly any research conducted on the extent to which an IT system contributes to a higher performance. This research aims at measuring the influence of information technology on the performance of a medical organization. This will be achieved by the construction of a model of performance indicators that indicate measures of quality.

The model has been used in a case study for the operating room software of the company ChipSoft.

1.3.2 Problem Statement

From the description above, the following research question(s) can be defined:

With which methods and indicators can the results of medical IT applications be measured? How can those methods and indicators be applied to measure the contribution of quality, efficiency and productivity of medical organizations?

1.3.3 Research Questions

The research question can be divided in several sub-questions:

Which methods and indicators are used, to measure the results of IT applications?

Which methods and indicators are used, to measure hospital performance?

Which methods and indicators are used, to measure the results of medical IT applications?

Which performance indicators can be drawn from these methods?

How can these performance indicators be prioritized? (weighted) What are the benefits and limitations of this method?

What conclusions can be drawn for healthcare and science in general?

What conclusions can be drawn for ChipSoft?

1.4 Research Design 1.4.1 Methods

The methods being used in this research are literature survey and field research. The literature survey is performed in order to reveal methods and indicators for a) measuring the results of implementing IT applications in general, b) measuring hospital performance and c) measuring the effects of medical IT applications on hospital performance. The research has been

conducted by gathering relevant articles in scientific journals such as: Health Policy, Health Care Management Science, Journal of the American Medical Association, Journal of

Computer Information Systems and the International Journal for Quality in Health Care. The results of the literature survey can be found in Chapter 2. The outcomes provide a list of indicators that can be measured in hospitals.

The field study exists of several phases. First of all there have been meetings at several hospitals to identify the processes on the OR, their use of OR-systems, performance

(16)

measurement and management of the OR. Interviews have been conducted with hospital experts such as IT managers, OR managers, OR planners and anaesthesia managers. These interviews have resulted in a list of additional performance indicators.

Further several interviews have been conducted with programmers and software consultants of ChipSoft. The software itself has been explored thoroughly. Also an implementation of CS- OR at a hospital has been attained. This provided much insight in the motivation of hospital specialists for implementing an OR information system.

Based upon the gathered experiences, a questionnaire has been developed. The aim of the questionnaire is gaining insight in the use of OR-systems in Dutch hospitals and performance measurement. Further purpose is to evaluate the relative importance of the indicators, found during the literature survey and field research. The questionnaires have been sent to OR managers of all hospitals in the Netherlands. The development of the questionnaire is described in Chapter 3. The complete questionnaire can be found in Appendix A.

The results of the questionnaire provide a general view of Dutch operating rooms, a comparison of OR software suppliers and a model for assessing OR-systems.

After the indicators and their relative importance were defined, they have been measured in 9 Dutch hospitals. Some measures are subjective, they are derived from interviews with hospital professionals and include opinions and feelings about working with an OR-system. Other measures can be established objectively by collecting real information from the system such as performance data. Based on these measures can be concluded whether implementing ChipSoft improves hospital performance.

1.4.2 Structure

The structure of the paper follows the approach mentioned above. Chapter 2 describes the literature survey, based upon IT performance indicators, hospital performance indicators and OR performance indicators. The conclusion presents a list of indicators that can be measured in this research.

In Chapter 3 are the results of the field research described. They include a summary of the main issues, being stated in interviews with hospitals professionals. These have led to a list of indicators, additional to those found in literature. Also, the development of the questionnaire is described.

Chapter 4 describes the results from the questionnaire. These are presented in a structured way, support by graphs or tables if necessary. The results include information of Dutch ORs and their use of information systems, a comparison of software suppliers based upon OR satisfaction and the relative importance of the performance indicators.

Chapter 5 describes the result of the measurement of the performance indicators in 9 hospitals. The indicators are measured by conducting interviews with OR managers and collecting data from the hospital information system.

Chapter 6 presents a final conclusion to the thesis. The conclusion summarizes the answers to the research questions and presents a model of performance indicators for assessing and

(17)

Chapter 7 presents recommendations for ChipSoft, based upon the results of the performance indicators, measured during the data collection and interviews.

(18)

2 LITERATURE DISCUSSION

This chapter presents an in-depth discussion of relevant theories on performance measures.

The structure of the chapter is based upon the research questions. The first paragraph deals with measuring the performance of information technology. Next, the performance measures for hospitals and other medical instances are described. The third paragraph discusses

performance indicators for operating rooms. Finally, literature that reflects on the relation between medical IT applications and OR performance will be described.

2.1 IT Performance Indicators

In order to assess the contribution of an IT system to certain performance gains, indicators are needed which can identify the results of an IT project / IT system. This paragraph deals with such IT performance indicators. The paragraph is divided into three sections: development, classification, and discussion.

2.1.1 Developing Performance Indicators

Measuring IT performance involves much more then measuring the technical performance such as uptime, downtime and processing speed. IT performance should be measured in terms of a system’s contribution to core processes of the organization. As companies invest more in their IT systems, they are also interested in the results of their investments.

Cameron (2000) developed a method to establish IT performance indicators. This method consists of 8 steps, which are displayed in Table 2.1.

Table 2.1: Eight Steps to establish IT performance indicators (Cameron 2000)

Those steps are elaborated as follows:

1) By linking IT projects to organizational goals, it becomes easier to measure their contribution. It is helpful to describe general statements that explain the relation of the IT project to the organizational goals, and the desired outcomes. Four areas which should be addressed are:

1. Meeting the strategic needs of the enterprise.

2. Meeting the needs of individual customers.

3. Addressing IT internal business performance.

Nr. Description:

1 Link IT Projects to Organizational Goals 2 Develop Performance Measures

3 Establish a Baseline for Future Performance Comparison 4 Select IT Projects with the Greatest Value

5 Collect Data 6 Analyze the Results

7 Integrate into Management Processes 8 Communicate the Results

(19)

2) Managing a process is significantly difficult if results cannot be measured properly. When developing performance measures, the IT process should be looked at from a customer-based view. Cameron (2000) extracts four types of performance measures:

- Input measures: measure the input needed for the process.

- Output measures: measure the output delivered by the process.

- Outcome measures: reflect the actual output against the expected output.

- Impact measures: measure the effects of outcomes over a longer period of time.

3) A baseline indicates certain figures, to which the measures are being compared. This can be the first measurement, a company average, an industry average or a result of a competitor.

Once the baseline is set, new measurements can be executed and data can be compared to them.

4) As already stated, IT investments need to be connected to the business processes. It is important to select IT projects that provide the most value to the company. Investments that are made only to improve or upgrade the technology do not contribute directly to a higher value of existing business processes.

5) The collection of data has to be taken in account during the development of performance indicators. Issues involving data collection are the following:

- Availability of data and the ease of collecting them.

- Accuracy: the extent to which the data and the data-source are reliable and can be verified.

- Timeliness: how recent are the data, how long does it take to gather them, and how often should they be collected.

- Cost of data collection: the costs involved in collecting data and the extent to which they weigh up to the benefits of gathering them.

6) Outcomes of measures are not very meaningful on their own. After the measurement, both indicators and results need to be objectively reviewed in order to draw accurate conclusions.

7) Based on those conclusions, actions need to be taken in order to improve the IT processes.

If a company reacts slowly on the performance measurements, employees may regard the measurement as unimportant and are less willing to contribute.

8) Communication and distribution of the results throughout the company can gain more support for IT projects, and boost confidence of employees towards IT systems. External communication can result in gaining new customers and partners.

A critical success factor in IT performance measurement is top management support.

Management has to communicate throughout the company the importance of measuring IT performance and should act upon the results.

The article of Cameron (2000) explains that developing performance indicators is not a straightforward process. It needs involvement of the whole organization and there are several pitfalls which should be overcome. Cameron distinguishes between four types of measures:

input, output, outcome and impact measures. The following subsection reflects on different views on the classification of performance indicators.

(20)

2.1.2 Classifying Indicators

The previous subsection has shown that the amount of indicators is enormous and they are divided over several fields. In order to create a clear overview of these fields, the indicators are categorized. There are several ways of classification, such as Cameron’s’ distinction between input and output measures.

DeLone and McLean (1992) have investigated over 180 articles that focus on measuring information system success, in order to develop a well-defined measure that offers a relevant contribution to practice.

These articles have offered an enormous list of different indicators. This can be explained by the fact that information as output can be measured on three levels (Shannon & Weaver 1949):

- Technical level: accuracy and efficiency of the system.

- Semantic level: success of the information in transmitting the intended message.

- Effectiveness level: the effect of the information on the receiver.

Mason (1978) saw information as a hierarchical process, based upon these three levels. The technical level can be seen as the production of information. The semantic level measures the product itself. The effectiveness level is distinguished by the receipt of information and the influence information has on the recipient and the system.

DeLone and McLean have translated the theories of Shannon & Weaver (1949) and Mason (1978) into what they call the IT effectiveness model. Based on their study of 180 articles, they have renamed the definitions of Mason and added the dimension User Satisfaction.

Figure 2.1 displays the dimensions of the three theories and shows how they are related. As can be seen, the three categories of Shannon and Weaver are expanded by Mason and DeLone

& McLean.

Shannon, Weaver

(1949)

Technical Level

Semantic

Level Effectiveness Level

Mason

(1978) Production Product Receipt Influence on Recipient

Influence on System

DeLone, McLean (1992)

System Quality

Information

Quality Use User

Satisfaction

Individual Impact

Organizational Impact Figure 2.1: Categories of IS success

DeLone and McLean’s research adds new categorization and updates the results of earlier studies as follows:

1) System Quality – Measures of the processing system itself. These are mostly measures based on the technical aspects of an information system. They are relatively easy to measure,

(21)

2) Information Quality – Measures of information system output. These measures are focused on the quality of the information that the system produces. These are fairly subjective

measures, because users can value information differently. The description of those measures is fairly general, for example accuracy, timeliness and reliability.

3) Information Use – Recipient consumption of the output of an IS. This measure relates to the degree of which the receivers of the information make use to it. Obviously, the usage of information partly depends on its quality. Use is probably the most objective measure and it is relatively easy to measure.

4) User Satisfaction – Recipient response to the use of the output of an IS. If the use of an information system is necessary, the measure information use will be not applicable. In those cases user satisfaction can be a reliable measure. User satisfaction is easy to measure but it is a subjective measure.

5) Individual Impact – The effect of information on the behaviour of the recipient. Impact is a difficult measure to describe or to measure. Impact means that the recipient acts on the information in a positive way. For instance, improvements in performance or gaining new knowledge that can be attributed to the information suggests a strong individual impact.

6) Organizational Impact – The effect of information on organizational performance. This measure is the most interesting, but also the most difficult to measure. It is of no surprise that the least articles are found on this measure. The results of this measure can indicate relations between use of an information system and organizational performance, but often do not provide a solid conclusion.

All the identified measures found are displayed by category in paragraph 2.1.3. DeLone and McLean point out that measuring IT is a multidimensional process. However, only 25% of the researched studies include multiple dimensions. These dimensions interrelate and therefore influence each other. For example, both system quality and information quality have influence on use. If quality is low, users do not use the system and would find other ways to get

information. Obviously, use and satisfaction are related. If information is not useful, the recipient would not be satisfied. Therefore the system will have no influence, or a negative influence on individual impact. Finally, the organizational impact depends on the individual impact. If employees can gather more knowledge or gain higher productivity this would affect the organization in a positive way. The relations between the dimensions are presented in Figure 2.2.

(22)

Figure 2.2: I/S Success Model (DeLone & McLean 1992)

The interdependency between the factors is shown by arrows. Those arrows indicate a relation or influence between factors (Miles’ Ecology Model 1980).

To conclude the above, DeLone and McLean provide overview of the available literature on measuring IT performance. They propose a categorization based upon earlier studies and their complementary findings. Results are presented in six categories, proven to be interrelated.

The relations are presented in the I/S success model. Those measures are discussed in the following paragraph.

2.1.3 IT Performance Indicators

The research of DeLone and McLean provides overview of general indicators listed in different categories. This chapter provides a discussion on performance indicators for measuring the influence of IT systems on OR performance. Table 2.2 presents the overview of all the indicators, as constructed by DeLone and McLean.

Information Quality System

Quality Information

Use

User Satisfaction

Individual

Impact Organizational

Impact

(23)

Organizational Impact Range and scope of application Number of critical applications Operating cost reductions Staff reduction Overall productivity gains Increased revenues Increased sales Increased market share Increased profits Return on investment Return on assets Ratio of net income to operating expenses Cost / benefit ratio Stock price Increased work volume Product quality Contribution to achieving goals Service effectiveness

Individual Impact Information understanding Learning Accurate interpretation Information awareness Information recall Problem identification Decision quality Improved decision analysis Correctness of decision Time to make decision Confidence in decision Decision making participation Improved individual productivity Change in decision Task performance Quality of plans Individual power/influence Willingness to pay for information Personal valuation of IS

User Satisfaction Satisfaction with specifics Overall satisfaction Single item measure Multi item measure Information satisfaction Difference between information needed and received Enjoyment Software satisfaction Decision making satisfaction

Information Use Number of inquiries Amount of connect time Number of functions used Number of records accessed Frequency of access Frequency of report requests Number of reports generated Charges for system use Regularity of use Direct vs. chauffeured use Use vs. non-use Actual vs. reported use Use for intended purpose Appropriate use Type of information used Purpose of use General vs. specific use Recurring use Routinization of use Report acceptance % Used vs. opportunity of use Voluntariness of use Motivation of use

Information Quality Importance Relevance Usefulness Informativeness Usableness Understandability Readability Clarity Format Appearance Content Accuracy Precision Conciseness Sufficiency Completeness Reliability Currency Timeliness Uniqueness Comparability Quantitativeness Freedom from bias

System Quality Data accuracy Data currency Database Contents Ease of use Ease of Learning Convenience of access Human factors Realization of user requirements Usefulness of system features and functions System accuracy System flexibility System reliability System sophistication Integration of systems System efficiency Resource utilization Response time Turnaround time Table 2.2: Summary of IS measures by category (DeLone McLean 1992)

(24)

The I/S success model (figure 2.2) indicates that all the different categories are interrelated and they all contribute to organizational impact. Therefore, the purpose of this paper is to study the organizational impact (defined by OR performance) by the implemented

information system (OR information system). Hence, the emphasis of this section lies on the indicators of organizational impact.

The range or scope of an application is not a quantifiable measure, and it does not tell

anything about performance. This also applies to the measure number of critical applications.

They are not applicable in this research.

Operating cost reductions is a useful indicator for the OR, but it will be complicated to get insight in the costs, and to address cost reductions to the IT system. Staff reductions are easy to measure, and staff reductions also imply cost reductions, therefore staff reductions seem to be a useful indicator. Overall productivity gains are obviously a good performance measure.

Again, complications are how to address these productivity gains to the IT system.

Furthermore the measure productivity needs to be defined for an OR setting. These issues need to be solved in order to provide good measurements. Further discussion of these measures will take place in chapter 2.5.

Table 2.2 mentions a series of financial indicators (from increased revenues to stock price).

Such indicators do not apply to an OR but to the hospital as a whole. These measures cannot be calculated for an OR, and more important, they cannot be related to an IT system.

Increased work volume is a suitable indicator for OR performance. Key-issue is to give a clear definition of work volume. The same applies for the measure product quality.

The measure contribution to achieving goals is difficult to quantify. The result of this measure is probably a subjective answer. It is not a figure that can be calculated or derived from the information system. Service effectiveness seems difficult to measure. What does the term

“service” mean in an OR setting and how is service effectiveness quantified.

This discussion has provided that five indicators drawn from the research of DeLone and McLean (1992) can be useful for measuring the influence of IT systems on OR performance.

These are:

• Operating cost reductions

• Staff reductions

• Overall productivity gains

• Increased work volume

• Product quality.

These indicators however, need to be further defined in terms of how to measure them and how the IT-system contributes to them. This further elaboration will take place in paragraph 2.5.

(25)

2.2 Hospital Performance

The measuring of hospital performance has become an important issue over the last years.

Transparency of healthcare and responsibility are big issues in current society. The market has liberated and patients can choose between hospitals. Therefore measuring performance has become very important for hospitals, insurers and patients.

There is a lot of literature available on measures of hospital performance. This chapter

presents an introduction of the issues that hospitals and insurers face nowadays. The statistical methods that are being used are described, and different measurement frameworks are being explored. Finally the current indicators in use at hospitals are being discussed.

2.2.1 Introduction

In 2004, the Dutch Ministry of Healthcare has asked the Dutch Postal Office (TPG) to research patient logistics in the Dutch healthcare sector. TPG has developed an excellent logistics process for the delivery of letters and packages. With this specific knowledge, they analyzed logistical processes in Dutch hospitals. The results have been published in the report Sneller, Beter (Faster, Better).

The report has revealed two important bottlenecks in patient logistics;

- the process a patient goes through is a push-system, - and there are no positive stimuli.

The patient is being pushed into the system, based on urgency and available capacity. This implies several disadvantages. The patient is not the main object in the process, and does not know what to expect of the process. This lack of knowledge causes that patients experience the process as a black box. Further, hospital employees have less insight in the process. They do not know what to expect of the workload. There is no clear view on throughput times; as a result it becomes more difficult to manage hospital processes. Furthermore, waiting times will increase.

Second issue described is the lack of positive stimuli. The current healthcare system does not support hospitals to improve their patient logistics. The budgets, maximum prizes, and the number of operations are set by the government. This implies that hospitals that perform fewer operations get less money. But hospitals that perform more operations then agreed, do not receive higher earnings.

TPG advices to introduce a pull-system. The patient should become the central object in the process. It needs to be clear what a patient can expect during the process. At what moment are the intake, the röntgen, the operation and the release from the hospital scheduled. If all the steps of the process are planned, both the patient and the personnel will get a better overview of the process.

It requires a lot of effort to transform a push-system into a pull-system. TPG proposes 6 logistical principles, which can be a guideline in this transforming process.

1) Cluster patients based on predictability of the treatment (from standardized to patient- specific): Certain treatments can be standardized, and they can be executed according to best practices. For these treatments, the logistical processes can be standardized as well.

(26)

2) Make a good planning, based on throughput times: The planning to meet the demand for health can be easier than it seems. Hospitals can foresee the demand, based on their

experience. It is important to make a different planning for the standardized and patient- specific treatments.

3) Put high weight on measuring, registration, and providing information. Based upon patient wishes and statistical information, a service level can be defined. After the treatment, this service level is measured and compared with a standardized scale.

4) Stop discussion, let instances provide information: In this healthcare process, there are a lot of stakeholders. (E.g. patients, insurers, hospital management, hospital employees and the government) TPG issues to end the endless discussion between stakeholders and take actions.

A small number of hospitals need to be analyzed to get a transparent view on hospital processes.

5) Make clear who is in charge: There are a lot of stakeholders with different interests.

Someone needs to be in charge to make a final decision. It has to be clear when a person or department is functioning or not, and rewards or penalties should be given.

6) Build positive stimulus in the process: Hospitals which provide above average results should be rewarded. This motivates hospitals in providing efficient treatments.

An important role in this process is allocated to information technology. Patient information databases, planning systems, and management information systems play an important role in modern hospitals. These kinds of systems can improve the efficiency of the processes. TPG proposes that the total efficiency gain, based upon standardization and optimisation, can be 25%. (Sneller, Beter 2004)

The report “Sneller, Beter” has indicated that much improvement can potentially be gained in the total patient-logistics. By dramatically redesigning the processes, based on a patient-view, the process can be made much more efficient and at the same time quality can significantly be improved. It indicates the importance of IT systems, but does not specify its role.

2.2.2 Statistical Analysis

A lot of current research on hospital performance includes statistical analysis of performance measures. This section provides a short description of the most used methods.

The analyses that are most used are Data Envelopment Analysis (DEA) and the Stochastic Frontier Approach (SFA). In DEA, the result of a measure is compared to the best possible result. This optimal result does not have to be an actual result, but can be a combination of best results. For example, a race-track is divided in three sections. The three fastest section- times combined, can be faster than the fastest lap time. In SFA, this ‘virtual’ result is calculated by using pre-defined mathematical formulae. (Algemene Rekenkamer 2003).

The comparison of the measure and the optimal result gives a figure that indicates how close the state of the indicator matches the best possible result. Hence, a score of 1.0 (or 100 percent) is the best possible match.

(27)

Hollingsworth (2003) analyzed over 180 articles using statistical analysis, and found that 50%

of them used the DEA approach, while only 20% used the SFA. In the eighties, on average five efficiency-studies per year were conducted. In the nineties, this figure has grown by 200%.

The average score of all the hospitals is 0.844. This implies that efficiency can be improved by 16%. Other remarkable findings from Hollingsworth are that European hospitals score better than American (0.892 to 0.834), where non EU/USA hospitals score only 0.799.

(Hollingsworth 2003)

None of these methods will be used in this research, because this research aims at obtaining qualitative results rather than mathematical figures. Nevertheless, those methods are worth mentioning, having proven to be both useful and popular in general research.

2.2.3 Measurement Frameworks

The development of performance indicators is mostly done by using a framework. Such a framework consists of several dimensions on which performance is measured. A framework proposes different angles of looking to an organization or system, and therefore different sets of indicators. This section describes different frameworks for hospital performance

measurement.

The National Committee for Quality Assurance (NCQA) has developed a method to

electronically report performance data (Schneider et al. 1999). These performance data will be captured in a so-called health information framework. The NCQA has used three dimensions that include measures from a patient-based view of performance. These are:

- Effectiveness of Care: the influence of factors (e.g. demographics, insurance, behaviour, etc.) on health outcomes.

- Access to Healthcare: the time needed to treat the patient, availability of services.

- Patient Satisfaction: experiences of patients.

The measures of the performance networks are all based on patients’ views. That is a risky approach, because there are more dimensions of hospital performance than the satisfaction of patients. A broader view of dimensions is presented in the research of Yap et al. (2005), which studies different performance measures used in Canadian hospitals. The System Level Scorecard (Baker 1999) serves as a basis for this research and includes four dimensions:

- Clinical Utilization: effectiveness of internal processes.

- Financial Performance: cost-efficiency.

- Patient Satisfaction.

- System and Information: data-integration, information-usefulness and value.

The four dimensions of the scorecard have been compared with the measures being used in hospitals. It turned out that only 59% of the hospitals measured their performance by all these dimensions. This means that a lot of hospitals measure performance on too little dimensions of the organization.

(28)

Efficiency Staff Safety

Patient-centeredness

Clinical Effectiveness Responsive governance

The European Office of the World Health Organization (WHO) developed an alternative measuring method named PATH – Performance Assessment Tool for quality improvements in Hospitals (Veillard et al. 2005). The purpose of PATH is to assist hospitals in judging their performance, and to act upon the results. Worldwide, performance measure projects have been researched in order to form a comprehensive framework. This resulted in a

multidimensional framework, which is presented in Figure 2.3. Safety and patient-

centeredness are displayed in horizontal columns because these dimensions interrelate to other attributes (such as patient safety, staff safety, and environmental safety).

The dimensions of the model are explained as follows:

1. Clinical effectiveness indicates whether a hospital produces quality results, in regard to the resources they posses.

2. Efficiency means using resources in an optimal way to maximize output.

3. Staff includes the morale, training and knowledge of employees.

4. Responsive governance relates to a broad sense of contribution to society, in being innovative, non-discriminative, and serving society.

5. Safety indicates the use of measurable processes or structures which minimize patient risk or harm.

6. Patient centeredness involves the service delivered to the patient. The patient should be the centre of care.

Figure 2.3: The PATH theoretical model for hospital performance (Veillard et al. 2005) These six dimensions include 24 performance indicators. The most relevant indicators for OR performance research are highlighted:

Intensity of surgical theatre use - this is measured by the following formula:

h theatres of

number

a anaesthesi under

hours patients of

number

24

*

Measuring the Results of the Implementation of an Operating Room Information System