Proactive safety management in health care : towards a broader view of risk analysis, error recovery, and safety culture

(1)

Proactive safety management in health care : towards a

broader view of risk analysis, error recovery, and safety

culture

Citation for published version (APA):

Habraken, M. M. P. (2010). Proactive safety management in health care : towards a broader view of risk analysis, error recovery, and safety culture. Technische Universiteit Eindhoven.

https://doi.org/10.6100/IR657709

DOI:

10.6100/IR657709

Document status and date: Published: 01/01/2010

Document Version:

Publisher’s PDF, also known as Version of Record (includes final page, issue and volume numbers)

Please check the document version of this publication:

• A submitted manuscript is the version of the article upon submission and before peer-review. There can be important differences between the submitted version and the official published version of record. People interested in the research are advised to contact the author for the final version of the publication, or visit the DOI to the publisher's website.

• The final author version and the galley proof are versions of the publication after peer review.

• The final published version features the final layout of the paper including the volume, issue and page numbers.

Link to publication

General rights

Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights. • Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain

• You may freely distribute the URL identifying the publication in the public portal.

If the publication is distributed under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license above, please follow below link for the End User Agreement:

www.tue.nl/taverne

Take down policy

If you believe that this document breaches copyright please contact us at:

openaccess@tue.nl

(2)

Proactive Safety Management in Health Care:

Towards a Broader View of Risk Analysis, Error Recovery, and Safety Culture

(3)

Proactive safety management in health care: Towards a broader view of risk analysis, error recovery, and safety culture /

by Marieke M.P. Kessels - Habraken

– Eindhoven: Technische Universiteit Eindhoven, 2009. – Proefschrift. –

A catalogue record is avalaible from the Eindhoven University of Technology Library ISBN 978-90-386-2095-4

NUR 801

Keywords: Patient safety / Safety Management / Prospective risk analysis / Incident reporting / Retrospective incident analysis / Error recovery / Near miss / Safety culture

Printed by Universiteitsdrukkerij Technische Universiteit Eindhoven Cover design: Oranje Vormgevers

All rights reserved. No part of this publication may be reproduced or utilised in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage and retrieval system, without permission in writing from the author.

(4)

Proactive Safety Management in Health Care:

Towards a Broader View of Risk Analysis, Error Recovery, and Safety Culture

PROEFSCHRIFT

ter verkrijging van de graad van doctor aan de Technische Universiteit Eindhoven, op gezag van de rector magnificus, prof.dr.ir. C.J. van Duijn, voor een commissie

aangewezen door het College voor Promoties in het openbaar te verdedigen op woensdag 20 januari 2010 om 16.00 uur

door

Marieke Maria Petronella Kessels - Habraken

(5)

prof.dr. J. de Jonge en

prof.dr. C.G. Rutte

Copromotor:

(6)

Acknowledgements

This is it. With this dissertation, my PhD research is completed. Before I started this project, I weighed up the pros and cons of doing a PhD. I did see the advantages of acquiring and enhancing academic skills, conducting field research, and writing a book. Nevertheless, I also realised that it would be a challenge to complete the project successfully and in time. Fortunately, the potential risk of an unsuccessful project was minimised thanks to a good many people.

First, I would like to thank my promotors Jan de Jonge and Christel Rutte, and my copromotor Tjerk van der Schaaf. Their knowledge, constant support, and confidence has enabled me to design, conduct, and complete this PhD research. Our co-authorship was a very valuable and pleasant experience. I have learned a lot from you. Thank you.

I owe great gratitude to Alysis Zorggroep, Infoland, and MERS International for giving me the opportunity to do this research. In particular, I thank Gert de Bey and Gerard Gerritsen from Alysis Zorggroep, Jan Stege, Frank Stege and Piet Baudoin from Infoland, and Rinus Gelijns and Annemarie Eras from MERS International for their willingness to collaborate. The combination of theory, practice, and software solutions appeared to be a big success. Moreover, I would like to thank Infoland for the belief in my potential, as demonstrated by

(7)

the decision to hire me. I feel very honoured to work for such an innovative and ambitious organisation. Thank you.

Further, I am grateful to all employees and managers in Alysis Zorggroep who, despite their pressure of work, participated in the studies, which demonstrates that patient safety is absolutely a top priority in Alysis Zorggroep. A special thanks to Gerard Gerritsen, Paulien Ogink, Katja Kerkvliet, Gonda Nienhuis, Hanneke Stoffels, Adriaan van Sorge, the members from the ―Van MIP naar VIM‖ project group, and my esteemed colleagues from the quality department. I also thank Ian Leistikow and Petra Reijnders-Thijssen for our collaboration and co-authorship, Carolien Plaisier and Dorien Zwart for their help in data collection, and all people of University Medical Centre Utrecht and MAASTRO clinic who participated in the HFMEA™ analyses. Thanks to the Netherlands Health Care Inspectorate for introducing me to the field of patient safety and providing access to their incidents database and case files. In addition, I would like to thank the undergraduate students who assisted me in data collection and analysis: Jeroen Rutteman, Hanneke Wijers, Frank Rinkens, Onno Kuip, and Zeno Korsmit. Thank you.

I would like to thank my co-workers (and former co-workers) at the Human Performance Management Group of Eindhoven University of Technology for their support and friendship. In particular, I owe gratitude to Anniek van Bemmelen for correcting my manuscripts and for taking care of all kinds of administrative matters. Further, I am grateful to Ad Kleingeld for his methodological advices. Thanks to Eric van der Geer for his support and advice, especially during the final months of my project. Last but not least, I would like to thank my room-mate, Marieke van den Tooren. I think we perfectly Matched. Thank you.

During my PhD research, I sometimes really needed to take my mind of it. In that case, exercising or going out appeared to be the best medicine. My Borrel friends and korfball teammates always helped me to unbend my mind. Your friendship means a lot to me. Thank you.

In addition, I would like to thank my family and in-laws. Your moral support and belief in my capacity made me realise that I would make it. A special thanks to my parents and sister. I am proud of you. Thank you.

(8)

Finally, a thousand thanks and lots of love to Maikel. During the last few years, you mitigated all stressful moments, just by loving me. So Incredible. Thank you.

(9)

(10)

Chapter 1 Introduction

Risk is part of everyday life. People face risks in their working environments, private lives, and leisure activities. Obviously, particular industries and environments or certain kinds of sports and hobbies are considered more hazardous than others. Risk management is core business for managers and operators in chemical plants and nuclear power stations. People think about risks before setting up a company, taking out a mortgage, or making a parachute jump. However, do they also consider risks prior to a hospital visit or when consulting a family doctor? Do they realise that they run the risk of being harmed by medical errors?

Errors are made in health care organisations, just like in any other organisation. However, in contrast with most other industries, in health care human lives are at risk rather than products or processes. Unfortunately, fatal medical errors happen frequently. In fact, fewer people die from airplane crashes, road traffic accidents, or natural disasters, such as earthquakes and tsunamis, than from medical errors in acute care (Runciman, Merry, & Walton, 2007). In the United States for instance, annually tens of thousands of people die in hospitals due to medical errors (Kohn, Corrigan, & Donaldson, 2000). Record review showed that medical errors cause about 1,700 deaths in Dutch hospitals every year (Wagner & De Bruijne, 2007). A systematic review revealed that nearly 1 out of 10 patients (9.2%) experiences an unintended injury or complication during hospital admission (De Vries, Ramrattan, Smorenburg, Gouma, & Boermeester, 2008). In their evaluation of the frequency

(13)

and nature of medical errors in primary care, Sandars and Esmail (2003) found an incidence of 5 to 80 errors per 100,000 consultations. Leape (1994) even argued that in the United States the annual number of deaths caused by health care itself, instead of the injury or disease (i.e. iatrogenic harm), equates to three jumbo jet crashes every two days.

Errors in health care are not always related to complex treatments or sophisticated surgeries. Instead, many medical errors are related to routine acts and caused by a catalogue of failures. Think about Wayne Jowett, an 18 years old cancer patient, who died after a cytotoxic drug mistakenly had been injected into his spine (Dyer, 2001). Or consider the 18-month old Josie King, who died because of dehydration. A lack of communication between doctors and nurses and their failure to respond to the parents‘ concerns caused the little girl‘s death (King, 2006).

Medical errors can result in various negative consequences. First, patients, their relatives, and even health care employees themselves can be harmed physically, mentally, and emotionally. Often, additional care and extended length of stay are necessary to mitigate those adverse effects. Diminished satisfaction of patients and their relatives could damage health care organisations‘ reputations and even result in liability claims. Moreover, health care employees themselves could get frustrated, which might negatively affect their performance. Further, medical errors can cause damage to medical equipment, devices, and buildings. In the end, those negative consequences could all result in financial losses for health care organisations, households, and even for society in terms of decreased productivity and diminished population health status (Cohen, 2001; Kohn et al., 2000). For adverse drug events only, Bates et al. (1997) and Classen, Pestotnik, Evans, Lloyd, & Burke (1997) calculated an additional length of hospital stay of 2.2 and 1.9 days on average, which resulted in increased costs of at least $2595 (i.e. €1851) and $2262 (i.e. €1613) per incident, respectively. Total costs would probably even exceed those figures because costs of malpractice claims were not taken into account in those studies. In the United States, it was estimated that the total amount of additional costs caused by medical errors that resulted in harm would be $37.6 (i.e. €26.8) billion, annually (Thomas et al., 1999). Before elaborating on the efforts that health care organisations could make to reduce the number of medical errors, we first introduce some important terms and definitions.

1.1 Definitions

(14)

Introduction

with their working definitions as used in this dissertation: incident, accident, and near miss. An incident can be defined as ―an event where a failure or combination of failures has occurred with the potential to lead to negative … consequences, irrespective of whether in the end these negative consequences became manifest, at least to some extent, or were avoided completely‖ (Kanse, 2004, p. 193). In this dissertation, the terms incident and (medical) error are used interchangeably. The foregoing definition of incidents encompasses both accidents and near misses. So-called accidents1 did have negative consequences for patients, whereas in case of so-called near misses adverse consequences were prevented (Kanse, 2004; Van der Schaaf, 1992).

1.2 Proactive Safety Management

The large number of medical errors and the harm, costs, and other negative consequences involved express the need for effective safety management. However, in spite of media coverage, which largely resulted from the Institute of Medicine report ―To Err Is Human” (Kohn et al., 2000) progress in improving patient safety appears to be slow (Coiera & Braithwaite, 2009; Leape & Berwick, 2005; Patel & Cohen, 2008). This might be related to the facts that, traditionally, medical culture considers errors unavoidable and an evident feature of medical care, and that particularly doctors tend to normalise errors (Quick, 2006; Waring, 2005). Moreover, until recently health care organisations in particular used band-aid approaches to deal with medical errors after they occurred (Karsh, Escoto, Beasley, & Holden, 2006; Pronovost et al., 2003). An example of such a reactive approach towards safety management is the Radboud hospital affair. In the Dutch Radboud hospital in 2006, the cardiac surgery unit was closed for several months due to unusually high mortality and morbidity rates (Netherlands Health Care Inspectorate, 2006). Early warning signals had been ignored, and not until a whistle-blower openly brought the quality and safety of the cardiac surgeries up for discussion, did the Netherlands Health Care Inspectorate investigate the problems.

Apparently, such a reactive safety management approach is not sufficient. The vision of safety management efforts in health care should be zero patient harm and therefore, the

—————————————

1_{In USA-based patient safety literature, incidents that resulted in patient harm (i.e. accidents) are commonly} referred to as adverse events. However, we decided to use the term accident to be more consistent with safety literature in other industries.

(15)

objective is minimal patient harm (Battles & Lilford, 2003). In line with this objective, a

proactive approach towards safety management is essential. It is necessary to foresee risks,

and to eliminate or at least minimise them before harm is done (Battles, Dixon, Borotkanics, Rabin-Fastman, & Kaplan, 2006; Hollnagel, 2008; Rath, 2008). The question arising from this objective, which is central to the present dissertation, is:

How could health care organisations apply proactive safety management to prevent patient harm and minimise costs of poor safety?

This dissertation proposes that proactive safety management could be implemented via three distinct but complementary approaches (see Figure 1.1).

Proactive Safety Management Organisational Context: Safety Culture Methods: Risk Analysis Data: Error Recovery

Figure 1.1: Three approaches towards proactive safety management.

First, health care organisations can use more prospective methods to identify and assess risks before errors may occur (Hollnagel, 2008). Health care organisations can use prospective and/or retrospective methods to identify risks. Prospective methods aim to foresee risks, while retrospective methods attempt to derive lessons from medical errors that have actually happened. In a prospective analysis, multiple health care employees together determine and assess potential risks and propose actions to eliminate or reduce those risks. Prospectively developed failure scenarios can be used to reveal and solve latent problems that could some day have resulted in incidents with severe consequences for patients (Reason,

(16)

Introduction

2004). This in contrast with retrospective methods, which are used to identify and analyse medical errors that have actually occurred. Retrospective methods are applied to facilitate learning, and measures are taken to prevent recurrence of the errors. Logically, prospective methods are most appropriate for proactive safety management because they concentrate on potential risks and enable health care organisations to come into action before harm is done.

Second, certain data can be used to improve patient safety in a more proactive way. Near misses, which by definition did not result in patient harm, can yield information about error recovery, that is, the way errors are detected and corrected. This information could be used to promote effective error recovery strategies, which is important since errors cannot be completely prevented (Aspden, Corrigan, Wolcott, & Erickson, 2004; Hollnagel, 2008; Kanse, Van der Schaaf, Vrijland, & Van Mierlo, 2006). Moreover, reporting and analysis of near misses offers opportunities to eliminate risks before they may result in actual accidents with adverse consequences for patients (Aspden et al., 2004; Barach & Small, 2000; Kaplan & Rabin Fastman, 2003; Van der Schaaf & Wright, 2005).

Third, besides those analytic approaches, advances in organisational context (i.e. safety culture) can be important for proactive safety management in health care (Aspden et al., 2004; Hudson, 2001; Nieva & Sorra, 2003; Pronovost & Sexton, 2005). Safety culture can be defined as ―the product of individual and group values, attitudes, perceptions, competencies, and patterns of behaviour that determine the commitment to, and the style and proficiency of, an organisation‘s health and safety management.‖ (Advisory Committee on the Safety of Nuclear Installations, 1993, p. 23). In an advanced safety culture, health care employees at all levels constantly consider safety a top priority (Hale, 2003; Nieva & Sorra, 2003; Pronovost et al., 2003) and aim to minimise patient harm. A positive safety culture in which safety is an important goal, could enhance safety behaviour and performance (Aspden et al., 2004; Clarke, 2006b; Flin, 2007; Flin, Burns, Mearns, Yule, & Robertson, 2006; Neal, Griffin, & Hart, 2000). Besides, safety culture could be considered ―the motor that makes the structure of the SMS [safety management system] work‖ (Hale, 2003, p. 194).

The three approaches (i.e. risk analysis, error recovery, and safety culture) could enable health care organisations to improve patient safety more proactively (see Figure 1.1). Consider, for instance, the implementation of a bar coding system in a health care organisation. A bar coding system could assist nurses in making sure that the right drug is administered to the right patient, at the right dose, and at the right time (Bates, 2000; Hampton, 2004). However, such a system might also induce problems, like degraded coordination (Patterson, Cook, & Render, 2002). In a paper-based system, doctors and nurses

(17)

have quick access to current medication orders at the patient‘s bedside and discuss those. In an electronic bar coding system, such as the one evaluated by Patterson et al., it is often impossible to gain a clear and instant view of the pending medication orders, which might, for instance, prevent doctors and nurses from recognising errors. True proactive safety management would imply that the health care organisation conducts a prospective analysis prior to the implementation of the bar coding system in order to identify and reduce possible risks and to raise risk awareness among the people involved. Afterwards, near miss reporting and analysis could facilitate learning. Through a dual approach of error reduction and error recovery promotion strategies, patient harm could be averted, whereby safety-related costs could be decreased. Unfortunately, such proactive safety management in health care is still in its infancy.

1.3 Risk Analysis

Although both prospective and retrospective methods can be used to improve patient safety, health care so far has particularly used retrospective methods, such as incident reporting (Karsh et al., 2006; Pronovost et al., 2003). However, since retrospective methods focus on actual errors, which might already have caused harm to patients, this focus seems not to be adequate enough. It is also important to foresee risks by identifying and assessing risks before incidents may occur (Battles et al., 2006; Hollnagel, 2008; Rath, 2008). Unfortunately, prospective methods, such as Healthcare Failure Mode and Effect Analysis (HFMEA™), have been applied only limitedly in health care. In an HFMEA™ analysis, a multidisciplinary team identifies and prioritises potential risks in a selected health care process, and subsequently identifies actions to eliminate or reduce those risks (DeRosier, Stalhandske, Bagian, & Nudell, 2002). Though several studies report about the application and evaluation of such prospective methods in health care (e.g., Jeon, Hyland, Burns, & Momtahan, 2007; Kunac & Reith, 2005; Wetterneck et al., 2006), hardly any systematic research has yet been conducted that evaluates and discusses the benefits and drawbacks of the use of those methods in health care.

Actually, complete and reliable prospective analyses would anticipate all risks and consequently render retrospective analyses superfluous (Senders, 2004). However, both prospective and retrospective methods are subject to biases, such as inaccurate risk assessment, incomplete data, and hindsight and recall bias. Therefore, triangulation of those methods seems to be necessary to obtain a more complete and reliable overview of

(18)

patient-Introduction

Lundquist, 2008; Runciman et al., 2006; Senders, 2004). In addition, integration of prospective and retrospective methods enables direct comparison of the results of the analyses. This might limit the additional resources required and support health care management in making sense of patient safety data and setting priorities for appropriate interventions (Battles et al., 2006; Hogan et al., 2008). Although several studies have explored possibilities for the integration of prospective and retrospective methods (e.g., Trucco & Cavallin, 2006; Van der Hoeff, 2003; Wetterneck et al., 2006), until now no research has concentrated on the perceived usefulness of this integration. Moreover, due to limited resources, like available funds or staff, it could be impossible for health care organisations to implement prospective and retrospective methods simultaneously (Akins & Cole, 2005; Devers, Pham, & Liu, 2004). However, it still remains to be explored which order of implementation is most preferable (Hale, 2003).

1.4 Error Recovery

Although safety can be defined as the absence of risk (Hollnagel, 2008), errors will always occur. Therefore, the ultimate objective of safety management in health care should not be zero risk or zero errors; instead, one should strive for zero or at least minimal patient harm (Battles & Lilford, 2003). Hence, health care organisations can focus on error reduction as well as error recovery promotion (Aspden et al., 2004; Hollnagel, 2008; Kanse et al., 2006). While error reduction strategies intervene between contributing factors and the error, error recovery promotion strategies intervene between the error and negative consequences (Kontogiannis, 1997). Near misses enable health care organisations to acquire insight into error recovery. Moreover, since the causal pattern of near misses and accidents is likely to be similar, analysis of near misses might prevent actual accidents from occurring, thereby proactively averting patient harm (Aspden et al., 2004; Barach & Small, 2000; Kaplan & Rabin Fastman, 2003; Van der Schaaf & Wright, 2005). Nevertheless, both in theory and in practice, there is a lack of a clear and consistent definition of near misses (Affonso & Jeffs, 2004; Aspden et al., 2004; Yu et al., 2005). This causes underreporting of near misses and analysis problems (Affonso & Jeffs, 2004; Etchegaray, Thomas, Geraci, Simmons, & Martin, 2005; Tamuz, Thomas, & Franchois, 2004). Because of this, health care has still failed to make the most of near misses and information about error recovery (Aspden et al., 2004; Parnes et al., 2007; Patel & Cohen, 2008).

(19)

1.5 Safety Culture

By triangulation of prospective and retrospective methods and by obtaining information about error recovery, health care organisations can make progress on the analytical pathway to improve patient safety. In addition, health care organisations can advance on the cultural pathway (Aspden et al., 2004; Hudson, 2001; Nieva & Sorra, 2003; Pronovost & Sexton, 2005). Hudson (2003) and Reason (1998) stated that in an advanced safety culture, health care employees and management are (1) informed about quality, safety, and risks, (2) trust each other; that is, they openly speak about errors without being blamed or punished, (3) are

adaptable to change through learning, and (4) worry about safety, that is, they are

preoccupied with risks. Advances in safety culture could change health care employees‘ behaviour, thereby indirectly reducing the number of medical errors (Aspden et al., 2004; Clarke, 2006b; Flin, 2007; Flin et al., 2006; Neal et al., 2000). In a positive safety culture, health care employees could probably better observe safety regulations and procedures (Neal et al., 2000), which could reduce the number of errors that happen. Further, a safety culture of alertness and vigilance might enhance error recognition and correction (Kontogiannis & Malakis, 2009), as a result of which incidents could be prevented from developing into accidents with actual patient harm.

The two other approaches towards proactive safety management (i.e. risk analysis and error recovery) and safety culture are interrelated. On the one hand, a safety culture in which health care employees are aware of risks and openly discuss errors is essential for prospective and retrospective methods to be applied successfully (Cannon & Edmondson, 2005; Hudson, 2001; Nieva & Sorra, 2003). On the other hand, conducting a prospective analysis or introducing an incident reporting and analysis system that facilitates learning might, in turn, positively influence safety culture (Aspden et al., 2004; Carroll, Rudolph, & Hatakenaka, 2002; Kaplan & Barach, 2002; Pronovost et al., 2007). In line with the latter assumption, Nieva and Sorra (2003) claimed that safety culture change could be viewed as an indirect outcome measure of patient safety interventions.

1.6 Dissertation Outline

This dissertation deals with three approaches towards proactive safety management as depicted in Figure 1.1: risk analysis (methods), error recovery (data), and safety culture (organisational context). Six studies were carried out, which are presented in Chapters 2 to 7 (see Figure 1.2). Together, those studies address important gaps in current knowledge

(20)

Introduction

regarding safety management. Although the six studies are related, each chapter can be read independently from the others.

Proactive Safety Management Organisational Context: Safety Culture Ch. 4 / 5 / 7 Methods: Risk Analysis Ch. 2 / 3 / 4 / 7 Data: Error Recovery Ch. 5 / 6

Figure 1.2: Dissertation outline: Overview of chapters.

Chapters 2 to 4 mainly concentrate on appropriate methods for proactive safety management. More specifically, Chapter 2 presents a qualitative field study that evaluates the application of a prospective risk analysis method (HFMEA™) in Dutch health care by means of user feedback. The qualitative field study presented in Chapter 3 addresses any biases underlying prospective and retrospective methods and deals with the triangulation and integration of both methods on two units of a Dutch general hospital. The quasi-experimental field study reported in Chapter 4 concentrates on the relation between the order of implementation of prospective and retrospective methods and incident reporting behaviour on 12 units of two Dutch general hospitals.

The qualitative field studies presented in Chapters 5 and 6 both focus on proactive

data, that is, information about error recovery. In Chapter 5, empirical data from four units of

two Dutch general hospitals are used to sharpen the definition of near misses in order to stimulate their reporting and to gather information about effective error recovery strategies. In Chapter 6, accidents are used as a supplementary source of information about error recovery.

Though organisational context also comes up in Chapters 2 to 6, Chapter 7 lists some important findings regarding safety culture, and presents a longitudinal panel survey on

(21)

trends in safety culture in three Dutch hospitals after an extensive safety management programme had been implemented. Chapter 7 furthermore explores which safety culture dimensions predict incident reporting behaviour.

The studies presented in Chapters 3, 4, 5, and 7 were all conducted in the same health care foundation, which comprises three hospitals: a teaching hospital that offers basic and specialised care (750 beds), a hospital that offers basic care (250 beds), and a hospital for outpatient treatment (50 beds). For each of the studies described in Chapters 3 to 5, we selected different units from those hospitals; in the panel survey on safety culture (Chapter 7), we included all units from the three hospitals (see Figure 1.3).

In Chapter 8, the main findings of the six studies are summarised and reflected upon, the strengths and limitations inclusive. Theoretical and practical implications are discussed, and suggestions for future research are put forward.

Health care foundation

Hospital A: Teaching hospital Basic and specialised care

Hospital B: Basic care

Hospital C: Outpatient treatment

Hospital units Hospital units Hospital units

Ch. 4

Ch. 5 Ch. 3

. . . Ch. 7 . . . Ch. 7 . . .

Figure 1.3: Sub samples to be used in Chapters 3, 4, 5, and 7. The health care foundation

comprises three hospitals, which, in turn, each consist of multiple units. Chapters 3, 4, and 5 each concerned different units; Chapter 7 concerned all units from the three hospitals.

(22)

Chapter 2 Prospective Risk Analysis of Health Care Processes:

A Systematic Evaluation of the Use of HFMEA™

in Dutch Health Care

*

This chapter evaluates the use of the prospective risk analysis method Healthcare Failure Mode and Effect Analysis (HFMEA™) in Dutch health care. Thirteen HFMEA™ analyses of various health care processes were carried out. User feedback uncovered perceived benefits and drawbacks regarding HFMEA™ and showed there is room for improvement. Several suggestions are put forward to improve the perceived utility and acceptance of this prospective method.

Safety management in health care is still in its infancy compared to other sectors, such as the chemical and nuclear industries, and civil aviation. Health care organisations so far have particularly concentrated on retrospective incident reporting and analysis, while prospective risk analysis has been applied less frequently. However, when one considers the objective of safety management, this retrospective focus does not seem to be sufficient enough. According to the definition of patient safety, the objective of safety management should be to prevent

—————————————

*_{This chapter is largely based on: Habraken, M. M. P., Van der Schaaf, T. W., Leistikow, I. P., &} Reijnders-Thijssen, P. M. J. (2009). Prospective risk analysis of health care processes: A systematic evaluation of the use of HFMEA™ in Dutch health care. Ergonomics, 52, 809-819.

(23)

patient harm (Battles & Lilford, 2003). Hence, one should foresee risks in health care processes instead of reactively taking action after incidents have occurred (Battles et al., 2006; Hollnagel, 2008; Rath, 2008).

Failure Mode and Effect Analysis (FMEA) is a systematic method for prospective analysis that can be used to identify and assess potential failure modes in products, processes, and systems. FMEA has a long history in the technical design of work settings. In subsequent applications, the human and organisational components of work settings have also been taken into account. FMEA is mainly used in manufacturing. However, it has also been applied in health care to improve patient safety in processes such as drug administration and blood transfusion (e.g., Adachi & Lodolce, 2005; Apkon, Leonard, Probst, DeLizio, & Vitale, 2004; Burgmeier, 2002; Day, Dalto, Fox, Allen, & Ilstrup, 2007; Dhillon, 2003; Jeon et al., 2007; Kunac & Reith, 2005; Paparella, 2007). In 2002, Healthcare Failure Mode and Effect Analysis (HFMEA™) was developed by the United States Department of Veterans Affairs' National Center for Patient Safety (NCPS) by combining concepts, components, and definitions from Failure Mode and Effect Analysis (FMEA), Hazard Analysis and Critical Control Points (HACCP), and Root Cause Analysis (RCA) (DeRosier et al., 2002). This method was designed to enable health care organisations to evaluate and improve health care processes before actual incidents may occur.

In both FMEA and HFMEA™, a multidisciplinary team graphically describes a selected process and subsequently identifies and assesses all potential failure modes. In FMEA, the team calculates a so-called risk priority number for each identified failure mode by multiplying its potential severity, frequency, and detectability. In HFMEA however, each identified failure mode is assessed with respect to its potential severity and frequency only, while a decision tree is used to consider the detectability of the failure mode and the availability of existing control measures. After having identified the failure mode causes, the FMEA or HFMEA™ team determines actions, barriers, and controls that either eliminate the failure mode causes or mitigate their effects.

Since its introduction in 2002, HFMEA™ has been applied on several health care processes, such as drug ordering and administration, and the sterilisation and use of surgical instruments (e.g., Esmail et al., 2005; Linkin et al., 2005; Van Tilburg, Leistikow, Rademaker, Bierings, & Van Dijk, 2006; Wetterneck, Skibinski, Schroeder, Roberts, & Carayon, 2004; Wetterneck et al., 2006). In the United States in 2004, the Joint Commission on Accreditation of Healthcare Organizations (JCAHO) began requiring accredited health

(24)

Prospective Risk Analysis of Health Care Processes

care organisations to conduct one prospective analysis every year (The Joint Commission: Standard PI.3.20).

Despite reported successful FMEA and HFMEA™ applications in several health care settings, the use of those prospective methods in health care still needs to be thoroughly evaluated and discussed. In some studies a single FMEA or HFMEA™ analysis has been conducted and critically evaluated (e.g., Jeon et al., 2007; Kunac & Reith, 2005; Wetterneck et al., 2006). Unfortunately, a systematic evaluation of a larger set of HFMEA™ analyses has, to our knowledge, not taken place yet. The need for a profound evaluation of HFMEA™ applications is endorsed by the fact that The Joint Commission has found that health care organisations are not always conducting their prospective analyses consistently or well (N. Kupka, The Joint Commission, personal communication, April 24, 2008). In this study, we carried out multiple HFMEA™ analyses at MAASTRO clinic, a radiotherapy institute in Maastricht, and at University Medical Center Utrecht (UMC Utrecht) to systematically evaluate HFMEA™ by means of user feedback. The clustered positive and negative comments resulted in several suggestions for change to improve the perceived utility and acceptance of HFMEA™.

2.1 Methods

Setting

A total of 13 HFMEA™ analyses were carried out to obtain insight into the perceived benefits and drawbacks of the application of HFMEA™ in Dutch health care. MAASTRO clinic provided us with a single focus, a high volume health care environment, while UMC Utrecht represented the general and academic hospitals.

Selection of Health Care Processes

At MAASTRO clinic, four HFMEA™ analyses were conducted on topics which were selected by the management team and the patient safety manager. At MAASTRO clinic, actual accidents, near misses, and process deviations are registered in a database and analysed in a systematic way. The processes that were selected for the HFMEA™ analyses of this study were high risk processes (according to the MAASTRO clinic incidents database) and/or new and innovative processes.

At UMC Utrecht all 12 divisions were asked to define three high risk processes. Subsequently, the patient safety coordinator and the division management involved jointly selected one of these processes. Criteria for this decision were: a direct connection with

(25)

patient care, high risk, availability of clear process boundaries, and feasibility. Finally, nine health care processes were selected to be included in this study. In three cases, two divisions both identified identical high risk processes. In those cases the divisions involved carried out a single HFMEA™ analysis collectively.

The 13 selected processes were quite diverse. Both acute and non-acute care were included, and scheduled as well as unscheduled tasks were considered. Moreover, technology played an important role in some selected processes, while it played a minor role in others. Finally, processes in both inpatient and outpatient settings were selected.

HFMEA™ Analysis

For each HFMEA™ analysis a multidisciplinary team was composed that consisted of at least two employees who were involved in the investigated health care process (e.g., nurses, doctors, technicians, or clerical staff) and a facilitator. In three teams, a patient or a patient's relative participated in the analysis. At MAASTRO clinic, in two teams, the patient safety manager (PR) was also present during the meetings; in one of those two teams, a student (JR) was present to learn more about how to facilitate an HFMEA™ analysis. In those two teams, the patient safety manager and the student were only indirectly involved in the HFMEA™ analysis. At MAASTRO clinic, the number of team members ranged from 4 to 8; on average a team consisted of 5.5 persons (SD = 1.7). At UMC Utrecht, the number of team members ranged from 6 to 13; on average a team consisted of 7.9 persons (SD = 2.1). This difference in the average number of team members can be explained by the fact that at UMC Utrecht sometimes multiple units were involved in a single HFMEA™ analysis, while in all HFMEA™ analyses at MAASTRO clinic only one unit was involved.

In all teams the facilitator concentrated on the correct use of HFMEA™ and the progress of the analysis. In 12 of the 13 teams the facilitator also took the minutes. In nine teams the facilitator was not involved in the selected process at all (MH and JR); in fact, those two facilitators are non-health care workers. In four teams the facilitator was either employed at the organisation and familiar with the investigated health care process (PR and CP) or directly involved in the investigated health care process (DZ). All facilitators gathered specific knowledge about HFMEA™ by means of the NCPS toolkit. One facilitator (PR) had conducted HFMEA™ analyses before; two facilitators (MH and JR) had been taught FMEA at university. All facilitators had experience in conducting incident analyses and were familiar with the system approach (Reason, 2000). The other team members received a

(26)

first meeting the facilitator gave a short presentation about the objective and the contents of HFMEA™.

Each multidisciplinary team met several times and each meeting took one and a half hours. The teams first reached an understanding about the exact definition of the selected process. Subsequently, the selected process was mapped. Then, the teams made a decision about the focus of the analysis. Sometimes, the complete process was analysed, while in other cases only a particular part of the selected process was analysed due to time constraints. Next, the team determined all possible ways in which the process could fail (i.e. not produce the anticipated result). Those identified failure modes were all assessed on their potential severity (i.e. catastrophic, major, moderate, or minor outcomes) and frequency (i.e. frequent, occasional, uncommon, or remote). For each failure mode a decision was made about the extent to which the risk was sufficiently covered in the health care system. In case the system did not take care of the failure mode effectively, the team identified the causes of the failure mode. After the team had assigned priorities to the failure mode causes, the team described actions, barriers, and controls to either reduce the chance of occurrence of the failure modes or to mitigate their effects. All information and decisions were (mostly) on site recorded in a worksheet. As an example, the results of a single HFMEA™ analysis are summarised in Box 2.1.

User Feedback on HFMEA™: Evaluation Forms

At the end of an HFMEA™ analysis all team members (apart from the facilitators) were asked to fill out an evaluation form about their experiences with HFMEA™. The evaluation forms consisted of both multiple choice questions and open-ended questions. At MAASTRO clinic, the patient safety manager (PR) and the student (JR) were not asked to fill out an evaluation form because they were only indirectly involved in the two HFMEA™ analyses in question. Hence, none of the facilitators and none of the members of the research group filled out an evaluation form. The evaluation form for patients or their relatives slightly differed from the evaluation form for employees. The evaluation forms were anonymous with respect to person, but not with respect to team.

(27)

Box 2.1

Example of an HFMEA™ analysis. Step 1. Define the HFMEA™ topic

Medication administration by means of infusion pumps at an Intensive Care Unit

Step 2. Assemble the team

- Two nurses

- A member of the quality department - An internal medicine specialist - An external facilitator (MH)

Step 3. Graphically describe the process

The selected process was divided into the following process steps: - Prescribing the medication

- Entering the prescription in the computer system - Dispensing the medication

- Conducting a double check - Adjusting the drip speed

Step 4. Conduct a hazard analysis

The team identified several potential failure modes such as:

- Wrong prescription or wrong entry of the medication or its concentration - Making use of the wrong fluid when dispensing the medication

- Not conducting a double check - Adjusting the drip speed wrongly

The causes underlying the failure modes were technical, organisational and human in nature. Examples of failure mode causes were:

- Wrong computation

- Lack of communication about a modified layout or the medication cupboard - Incorrect or incomplete protocols

- Health care employees being unfamiliar with certain types of infusion pumps

Step 5. Identify actions and outcome measures

The team proposed several actions to eliminate or control the failure modes. The most important actions were:

- Use of generic drug names when prescribing medication - Use of weighing beds

- Communication of modifications via e-mail and advice - Revision of the double check protocol

- Computation as part of Intensive Care Unit education - Specific instructions in case of new equipment

Data Coding

On the evaluation forms, the respondents were asked to write down in free text comments regarding HFMEA™ and its application. Subsequently, those comments were categorised. Two independent coders were involved in the coding process (MH and HW). The fact that one of the two coders (MH) was the facilitator of nine teams could have biased the results of

(28)

the coding process. However, this potential bias was minimised because the second coder was an Industrial Engineering student (HW), who, apart from the coding process, was not involved in the study at all. HW had been taught FMEA and HFMEA™ at university and as part of her master project she had used patient safety tools such as incident analysis techniques. Moreover, the potential bias was lessened because the two coders discussed until a consensus was reached. The two coders first independently classified the comments into four categories: positive (single, positive statements; e.g., "a constructive attitude of the participants"), negative (single, negative statements; e.g., "the analysis was time-consuming"), plural (multiple statements; e.g., "a thorough approach, enthusiastic guidance, cooperation"), and irrelevant (single statements without any relation to the contents of HFMEA™ and/or its application; e.g., "good luck!"). The percentage agreement between the two coders was 71.8%. The corresponding Cohen's kappa of .61 indicated substantial agreement (Landis & Koch, 1977). For the comments that were classified as plural, the coders also determined which separate statements could be distinguished and to which category (i.e. positive, negative, or irrelevant) those statements could be assigned. The percentage agreement between the two coders regarding the classification of the plural statements was 58.3%. During a consensus meeting the two coders reached an agreement about the categorisation of all statements.

Subsequently, the two coders jointly defined nine codes that referred to the separate steps and aspects of HFMEA™ (such as the multidisciplinary team, the facilitator, and the identification of failure modes and failure mode causes). In addition, the coders used open coding (Babbie, 2005) to develop codes for the exact opinion the respondents had on the various steps and aspects of HFMEA™ (e.g., "difficult" or "time-consuming"). While assigning the statements to the positive, negative, and irrelevant categories, the coders gained a first understanding of the exact opinion of the respondents. Together, the two coders decided upon six codes for type of opinion. Those codes completely emerged from the data, which is in accordance with the open coding principle. Each of the six codes for type of opinion was formulated in both positive and negative terms (e.g., "easy" and "difficult" or "clear" and "unclear"). Both coders then independently assigned all positive and negative statements to one of the nine codes for the steps and aspects of HFMEA™ and to one or more of the six codes for type of opinion. The percentage agreement between the two coders on the classification of the statements into both steps / aspects of HFMEA™ and type of opinion was 58.4%. Because the coders were allowed to classify one statement into multiple types of opinions, it was only possible to calculate kappa for the assignment of the statements to the

(29)

nine steps / aspects of HFMEA™. The percentage agreement between the two coders on the assignment of the statements to the steps / aspects of HFMEA™ was 77.3%. The corresponding Cohen's kappa of .72 again indicated substantial agreement. During a second meeting the two coders reached a consensus about the classification of all statements. Moreover, the coders decided to accentuate the definitions of some types of opinions and to add an additional code referring to HFMEA™ in general. The final classification scheme for positive and negative statements regarding HFMEA™ thus consisted of ten codes for steps / aspects of HFMEA™ and six codes for type of opinion (see Table 2.1).

Table 2.1

Classification scheme for positive and negative statements regarding HFMEA™.

Step / aspect of HFMEA™ Process selection and scope Multidisciplinary team Facilitator

Process description

Identification of failure modes and failure mode causes Risk assessment

Identification of actions and outcome measures Implementation of actions

HFMEA™ in general Other

Type of opinion (positively stated) Type of opinion (negatively stated)

Pleasant Unpleasant

Easy Difficult

Clear Unclear

High output Low output

Small time investment Large time investment

Other Other

Facilitator's Feedback on HFMEA™: Discussions

During the project, the research group (MH, TS, IL, and PR) also consulted the facilitators (MH, PR, JR, DZ, and CP) to evaluate the application of HFMEA™. The research group and facilitators met several times and exchanged experiences. After all HFMEA™ analyses had been finalised but before the quantitative and qualitative data analysis of the evaluation

(30)

forms, the research group and facilitators collectively drew conclusions regarding the application of HFMEA™ in Dutch health care.

2.2 Results

Descriptive Statistics

All 13 HFMEA™ analyses were successfully concluded. Table 2.2 presents some important descriptive statistics for each selected health care process and the accompanying health care setting: the initials of the facilitator, the team size, the number of meetings, the total number of person-hours needed for the analysis, the number of identified failure modes, and the number of proposed actions. Every meeting took one and a half hours, as scheduled beforehand. The number of meetings needed to carry out the analysis ranged from 4 to 8; on average the teams needed 6.3 meetings (SD = 1.3). The average number of meetings at MAASTRO clinic was lower than that at UMC Utrecht (5.8 and 6.6, respectively). This difference can partly be attributed to the fact that at MAASTRO clinic the processes had already been mapped before the formal start of the HFMEA™ analyses and the graphical process descriptions only needed to be verified by the team members involved. The number of person-hours needed to conduct the analysis ranged from 30.0 to 136.5 (excluding reporting on the meetings and reporting on the results of the HFMEA™ analysis); on average the HFMEA™ analyses took 69.1 person-hours excluding reporting (SD = 28.7) and 78.0 person-hours including reporting. The differences between the teams with respect to time investment can largely be attributed to differences in team size and scope. In earlier studies, HFMEA™ analyses on vincristine prescription and administration and the sterilisation and use of surgical instruments took a total of 140 and 250 person-hours, respectively (Linkin et al., 2005; Van Tilburg et al., 2006). The average number of identified failure modes was 51.8 (SD = 30.6) and the average number of proposed actions was 16.2 (SD = 8.8). Again, differences in scope contributed to team differences regarding the number of identified failure modes and the number of proposed actions.

(31)

Table 2.2

Selected health care processes, accompanying health care settings and descriptive statistics.

ID Health care process Health care setting Facilitatora Team sizeb No. of meetings No. of person-hoursc No. of failure modes No. of actions

1 Documentation of treatment Radiotherapy PR 5 4 30.0 32 17

2 Electronic Portal Imaging (EPI) Radiotherapy MH 8 6 72.0 109 33 3 Treatment on linear accelerator Radiotherapy JR 5 8 60.0 70 30 4 Release of accelerator after maintenance Radiotherapy PR 4 5 30.0 50 22 5 Communication of unexpected findings Radiology Cardiology MH 7 5 52.5 19 7

6 Diet food process Children's Hospital MH 13 7 136.5 39 18

7 Physically restraining patients Neurosurgery MH 7 7 73.5 31 17 8 Ordering repeat prescriptions Primary care DZ 8 8 96.0 50 12

9 Patients with hip fractures Emergency Room Radiology Nursing ward Operating Room MH 8 6 72.0 120 7 10 Medication administration (pumps)

(32)

Table 2.2 continued

Selected health care processes, accompanying health care settings and descriptive statistics.

ID Health care process Health care setting Facilitatora Team sizeb No. of meetings No. of person-hoursc No. of failure modes No. of actions 11 Admission of cardiac patients Emergency Room Cardiac Cath Room Coronary Care Unit

CP 6 6 54.0 44 6

12 Use of a PICC line (catheter) Neonatal Intensive Care Unit MH 8 8 96.0 37 8 13 Administration of blood products Laboratory Haematology ward MH 8 6 72.0 27 11 M (SD) 7.2 (2.2) 6.3 (1.3) 69.1 (28.7) 51.8 (30.6) 16.2 (8.8) a

MH and JR Eindhoven University of Technology; PR MAASTRO clinic; DZ and CP UMC Utrecht. bA patient was included in teams 1, 6, and 8. cReporting on the meetings and the results of the HFMEA™ analysis are excluded.

(33)

User Feedback on HFMEA™: Results from Evaluation Forms

All team members apart from the facilitators and team members (if any) who were only indirectly involved in the HFMEA™ analysis (i.e. 77 people) were asked to fill out the evaluation form. In total 62 evaluation forms were filled out and returned to the researchers; 59 by employees and 3 by patients or their relatives. The overall response rate was 80.5%. The response rates of MAASTRO clinic and UMC Utrecht were almost equal (80.0% and 80.6%, respectively). Table 2.3 presents the contents and results of the multiple choice questions of the evaluation forms.

About 90% of the employees and patients who filled out the evaluation form thought that the HFMEA™ analysis was meaningful (90.3%). The majority of the respondents (87.1%) expected the investigated health care process to become more safe as a result of the HFMEA™ analysis that had been carried out. Also about 90% of the respondents would recommend others to participate in an HFMEA™ analysis (90.3%). The evaluation form for the employees also included some questions about patient involvement in the HFMEA™ analysis. Of all respondents who participated in an HFMEA™ analysis in which a patient was involved, over 90% (93.3%) thought that this patient involvement was useful. Interestingly, only a minority of all respondents who participated in an HFMEA™ analysis in which no patient had been involved (9.1%) thought patient involvement would have been useful.

(34)

Table 2.3

Contents and results of evaluation forms: Multiple choice questions.

Health care employees (n = 59)

Patients (n = 3)

Health care employees + Patients (N = 62)

Question Yes No No answer Yes No No answer Yes No No answer Did the manual provide you

with sufficient information about conducting an HFMEA™ analysis?

94.9% 1.7% 3.4% 100.0% 0.0% 0.0% 95.2% 1.6% 3.2%

Were all relevant disciplines represented in the team?

88.1% 11.9% 0.0% 66.7% 33.3% 0.0% 87.1% 12.9% 0.0% Was a patient represented in

the team?

25.4% 74.6% 0.0%

- If yes, do you think this was useful?

93.3% 6.7% 0.0%

- If no, do you think this would have been useful?

9.1% 72.7% 18.2%

Were all meetings useful for you?

74.6% 20.3% 5.1% 100.0% 0.0% 0.0% 75.8% 19.4% 4.8% Do you think the HFMEA™

analysis was meaningful?

91.5% 1.7% 6.8% 66.7% 0.0% 33.3% 90.3% 1.6% 8.1% Do you think the investigated

process will be safer thanks to the HFMEA™ analysis?

(35)

Table 2.3 continued

Contents and results of evaluation forms: Multiple choice questions.

Health care employees (n = 59)

Patients (n = 3)

Health care employees + Patients (N = 62)

Question Yes No No answer Yes No No answer Yes No No answer Did you obtain another

insight into your own work process thanks to the HFMEA™ analysis?

45.8% 45.8% 8.5%

Would you recommend others to participate in an HFMEA™ analysis?

93.2% 1.7% 5.1% 33.3% 33.3% 33.3% 90.3% 3.2% 6.5%

Are you more willing to report incidents since you have conducted the HFMEA™ analysis?

23.7% 62.7% 13.6%

Are you more assured about safety in the institution since you have conducted the HFMEA™ analysis? 33.3% 0.0% 66.7% Fine Too long Too short No answer Fine Too long Too short No answer Fine Too long Too short No answer What did you think of the

duration of the meetings?

83.1% 10.2% 6.8% 0.0% 66.7% 0.0% 0.0% 33.3% 82.3% 9.7% 6.5% 1.6%

(36)

The classification of the comments of the respondents into steps / aspects of HFMEA™ and types of opinions shows the perceived benefits and drawbacks of HFMEA™. In addition to the percentage of respondents that made a particular comment, the results show the number of teams in which that particular comment was made by at least one team member. In both the results section and the tables the percentage of respondents is directly followed by the number of teams, which is presented between parentheses. Table 2.4 presents the resulting classification of the positive statements. Since the respondents were allowed to write down multiple (positive) comments and because some respondents did not answer the open-ended questions, the totals do not equal 100%. According to 36.4% of the respondents (10 teams) the HFMEA™ analysis resulted in high output in terms of the insight obtained into the health care process in general, in other employees' tasks, and in the possible risks (e.g., "HFMEA™ makes failure modes apparent" or "by means of HFMEA™ I gained a clear insight into processes and relations"). Positive remarks with respect to HFMEA™ in general, such as the fact that HFMEA™ is a systematic, stepwise approach, were made by 28.6% of the respondents (9 teams) (e.g., "HFMEA™ is a clear method" or "it is a structural approach"). Furthermore, 22.1% of the respondents (8 teams) thought the multidisciplinary nature of the analysis was pleasant and useful (e.g., "the multidisciplinary approach was useful").

(37)

Table 2.4

Positive user feedback on HFMEA™: Percentage of respondents (no. of teams) per combination of step / aspect of HFMEA™ and type of opinion. Type of opinion

Pleasant Easy Clear High output Small time

investment Other Total

Step / aspect of HFMEA™

Process selection and scope 1.3% (1) 1.3% (1)

Multidisciplinary team 2.6% (1) 3.9% (2) 16.9% (8) 22.1% (8)

Facilitator 5.2% (3) 2.6% (2) 1.3% (1) 6.5% (4) 13.0% (5)

Process description 10.4% (5) 1.3% (1) 11.7% (6)

Identification of failure mode (causes) 7.8% (5) 7.8% (5)

Risk assessment 2.6% (2) 1.3% (1) 3.9% (3) Identification of actions 0.0% (0) Implementation of actions 2.6% (2) 1.3% (1) 3.9% (2) HFMEA™ in general 1.3% (1) 1.3% (1) 2.6% (2) 13.0% (7) 2.6% (2) 15.6% (6) 28.6% (9) Other 1.3% (1) 1.3% (1) Total 9.1% (4) 1.3% (1) 5.2% (4) 36.4% (10) 2.6% (2) 37.7% (11)

Note. An empty cell indicates that no comment referred to that particular combination of step / aspect of HFMEA™ and type of opinion. Since the

respondents were allowed to write down multiple (positive) comments and because some respondents did not answer the open questions, the totals do not equal 100%.

(38)

Table 2.5 presents the resulting classification of the negative statements. Since the respondents were allowed to write down multiple (negative) comments and because some respondents did not answer the open questions, the totals do not equal 100%. Negative remarks of 20.8% of the respondents (9 teams) concerned the notion that the time investment necessary to conduct the HFMEA™ analysis was large (e.g., "it takes a lot of time"). Although the positive remarks indicated that the HFMEA™ analysis resulted in high output in terms of the (additional) insight into processes, tasks, and risks, 20.8% of the respondents (7 teams) felt that the analysis did not yield (significant) results, that is, that the output was low (e.g., "many aspects lead to useless discussions"). According to 13.0% of the respondents (6 teams) the HFMEA™ analysis was difficult to carry out. From the negative remarks of 7.8% of the respondents (5 teams), it can be concluded that especially the risk assessment part of HFMEA™ (i.e. determining the hazard score and using the decision tree) was perceived to be difficult (e.g., "the decision tree was difficult for me" or "it is difficult to score the risks"). In general, the risk assessment part of HFMEA™ was subject of the negative comments of 15.6% of the respondents (8 teams). Although the multidisciplinary nature of the team was perceived to be beneficial, 13.0% of the respondents (6 teams) faced problems within the team such as planning problems and problems regarding the frequent absence of certain team members (e.g., "often people were absent").

As can be concluded from the positive and negative remarks, the facilitator's role is perceived to be crucial. Respondents from 5 teams mentioned that the facilitator's presence had been of great value (e.g., "pleasant and clear guidance"), while respondents from 4 teams even claimed that the facilitator's role had been essential and that the analysis would not have been possible without the facilitator (e.g., "a good facilitator is necessary" or "we needed quite a lot of guidance").

(39)

Table 2.5

Negative user feedback on HFMEA™: Percentage of respondents (no. of teams) per combination of step / aspect of HFMEA™ and type of opinion. Type of opinion

Unpleasant Difficult Unclear Low output Large time

investment Other Total

Step / aspect of HFMEA™

Process selection and scope 1.3% (1) 1.3% (1)

Multidisciplinary team 5.2% (4) 7.8% (5) 13.0% (6)

Facilitator 6.5% (4) 6.5% (4)

Process description 1.3% (1) 5.2% (2) 5.2% (4) 11.7% (6)

Identification of failure mode (causes) 2.6% (2) 1.3% (1) 3.9% (2) Risk assessment 7.8% (5) 1.3% (1) 2.6% (1) 1.3% (1) 5.2% (4) 15.6% (8) Identification of actions 1.3% (1) 1.3% (1) 2.6% (2) Implementation of actions 6.5% (4) 6.5% (4) HFMEA™ in general 1.3% (1) 2.6% (2) 6.5% (3) 16.9% (8) 9.1% (5) 27.3% (10) Other 0.0% (0) Total 0.0% (0) 13.0% (6) 3.9% (3) 20.8% (7) 20.8% (9) 37.7% (11)

Note. An empty cell indicates that no comment referred to that particular combination of step / aspect of HFMEA™ and type of opinion. Since the

respondents were allowed to write down multiple (negative) comments and because some respondents did not answer the open questions, the totals do not equal 100%.