UvA-DARE is a service provided by the library of the University of Amsterdam (https://dare.uva.nl)
UvA-DARE (Digital Academic Repository)
Routine outcome monitoring & learning organizations in substance abuse
treatment
Oudejans, S.C.C.
Publication date
2009
Document Version
Final published version
Link to publication
Citation for published version (APA):
Oudejans, S. C. C. (2009). Routine outcome monitoring & learning organizations in substance
abuse treatment.
General rights
It is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s) and/or copyright holder(s), other than for strictly personal, individual use, unless the work is under an open content license (like Creative Commons).
Disclaimer/Complaints regulations
If you believe that digital publication of certain material infringes any of your rights or (privacy) interests, please let the Library know, stating your reasons. In case of a legitimate complaint, the Library will make the material inaccessible and/or remove it from the website. Please Ask the Library: https://uba.uva.nl/en/contact, or a letter to: Library of the University of Amsterdam, Secretariat, Singel 425, 1012 WP Amsterdam, The Netherlands. You will be contacted as soon as possible.
R O U TIN E O U TC O M E M O N IT O R IN G & L E A R N IN G O R G A N IZ A TIO N S I N S U B S TA N C E A B U S E T R E A T M E N T S U Z A N O U d E jA N S
—
ROUTINE OUTCOME
MONITORING &
LEARNING
ORGANIZATIONS IN
SUBSTANCE ABUSE
TREATMENT
—
—
SUZAN
OUdEjANS
—
—
—
R O U T IN E O U TC O M E M O N IT O R IN G & L E A R N IN G O R G A N IZ A T IO N S I N S U B S TA N C E A B U S E T R E A T M E N T S U Z A N O U d E jA N S—
ROUTINE OUTCOME
MONITORING &
LEARNING
ORGANIZATIONS IN
SUBSTANCE ABUSE
TREATMENT
—
—
—
Routine Outcome Monitoring and Learning Organizations in Substance Abuse Treatment. Thesis, University of Amsterdam, Faculty of Medicine – with summary in Dutch.
Copyright Suzan Oudejans, 2009. ISBN: 978-90-9024132-6
All rights reserved. No part of this book may be reproduced, stored in or introduced into a retrieval system, or transmitted, in any form or by any means: electronically, mechanically, photocopying, recording or otherwise, without the prior written permission of the author.
Editing: Clare McNally, Amsterdam
Design and print production: Clare McNally, Amsterdam Printed by: Ipskamp Drukkers, Enschede
The studies presented in this thesis were conducted at the Amsterdam Institute for Addiction Research, at the Jellinek (now the Jellinek division of Arkin), at ParnassiaBavo Groep division Brijder Verslavingszorg and at Novadic-Kentron. The studies were funded by the Netherlands Organization for Health research (ZON-MW grant no. 310000050).
Financial support for preparing and printing of this thesis was kindly provided by the Academic Medical Center, Department of Psychiatry, Amsterdam and Arkin, Amsterdam.
ACAdEMISCh PROEfSChRIfT
ROUTINE OUTCOME MONITORING ANd LEARNING ORGANIZATIONS IN SUBSTANCE ABUSE TREATMENT
ter verkrijging van de graad van doctor aan de Universiteit van Amsterdam op gezag van de Rector Magnificus
prof. dr. D.C. van den Boom ten overstaan van een door het college
voor promoties ingestelde commissie,
in het openbaar te verdedigen in de Aula der Universiteit op dinsdag 19 mei 2009, te 10:00 uur
dOOR
Suzanna ChriStina Catharina OudejanS
—
PrOmOtieCOmmiSSie:
PROMOTORES: PROf. dR. G.M. SChIPPERS PROf. dR. W. vAN dEN BRINk
CO-PROMOTOR: dR. M.W.j. kOETER
OvERIGE LEdEN: PROf. dR. k. STRONkS PROf. dR. h. vAN dE MhEEN dR. Y.A.M. NIjSSEN
PROf. dR. N.S. kLAZINGA PROf. dR. j. dEkkER
—
—
TABLE Of
CONTENTS
—
TABLE Of CONTENTS / PAGE 5
TA B LE O f C O N T E N T S 7 57 75 89 99 115 136 142 144 146 132 ChaPter 1 INTROdUCTION ChaPter 2
fEASIBILITY ANd vALIdITY Of LOW-BUdGET TELEPhONIC fOLLOW-UP INTERvIEWS IN ROUTINE OUTCOME MONITORING (ROM) Of SUBSTANCE ABUSE TREATMENT
ChaPter 3
fACILITATING ANd IMPEdING fACTORS fOR ROUTINE OUTCOME MONITORING (ROM) IN SUBSTANCE ABUSE TREATMENT
ChaPter 4
EffECTIvENESS Of MANUAL-BASEd COGNITIvE BEhAvIORAL ThERAPY IN ROUTINE OUTPATIENT ALCOhOL TREATMENT
ChaPter 5
MEASURING ThE LEARNING CAPACITY Of ORGANIZATIONS. dEvELOPMENT ANd fACTOR ANALYSIS Of ThE QUESTIONNAIRE fOR LEARNING ORGANIZATIONS (QLO)
ChaPter 6
EffECT Of TREATMENT OUTCOME fEEdBACk ON LEARNING CAPACITY IN SUBSTANCE ABUSE TREATMENT CENTERS
ChaPter 7 GENERAL dISCUSSION reFerenCeS SUMMARY SAMENvATTING dANkWOORd CURRICULUM vITAE PUBLICATIES 23 41
—
—
—
/ PAGE 6 C h A P T E R 1—
—
ChAPTER 1
—
INTROdUCTION
IN T RO d UC TI O N—
—
—
Extent and costs of substance abuse
There is a growing emphasis on quality and results of health care organizations, and this is also true for the substance abuse treatment sector. The abuse of alcohol and other drugs is associated with serious public health problems, pub-lic safety problems and economic damage. In the United States, 5% of people aged over 12 meet the criteria for substance dependence (Dennis & Scott, 2007). In Europe, 12 month prevalence ranges from 1.1% in women to 6.1% in men (Rehm, Room, van den Brink, & Jacobi, 2005; Rehm, Room, van den Brink, & Kraus, 2005), and according to the most recent estimations for the Netherlands, 8.2% of the adult population meets the criteria for alcohol abuse or dependence (3.7% meet the criteria for dependence) and about 1.3% for drug abuse or dependence (Verdurmen, Monshouwer, van Dorsselaar, & de Graaf, 2003; Vollebergh et al., 2003) , representing around 800.000 adults with alcohol abuse or dependency and 130.000 adults with drug abuse or dependency1.
Costs of substance abuse are high. The estimated tangible costs of harmful alcohol use to the European Union (EU) for 2003 are estimated at 125 billion euros. Spendings on alcohol-related problems like health care, the criminal justice system and traffic accidents accounted for 61 billion euros. The costs of potential production not realized due to absenteeism, unemployment, and pre-mature mortality were estimated at 59 billion euros. The remaining 5 billion euros were spent on treatment and prevention (Cnossen, 2006). Tangible costs of harmful alcohol use in the Netherlands were estimated at a minimum of 0.7% of the GDP. This is low in comparison with other EU member states such as Belgium (1.7%), but higher than Italy and Portugal (both 0.4%). Taking into account a GDP of 600 billion euros2, costs for the Netherlands are at least
4.2 billion euros. In all EU member states, except Finland, excise duty collec-tions fall short of the estimated tangible costs (Cnossen, 2006).
These costs, combined with serious public health and public safety problems, and the fact that most treatments for substance abuse are provided through public funding, mean there is a legitimate interest of stakeholders like governments, health insurers, and consumers of care to assure that substance abuse treatment centers are effective and efficient, i.e. deliver high quality care for a reasonable price. In addition, stakeholders put pressure on the sector to demonstrate pro-grammatic transparency and accountability regarding treatment outcomes.
ChAPTER 1 / INTROdUCTION / PAGE 9
1 2
Taking into account a population of 10 million adults (source: Statline, cbs (http://statline.cbs.nl/ StatWeb/publication/? dm=slnl&pa=37296ned&d1=0-51&d2=0,10,20,30,40,50,(l-1)-l&vw=t) http://www.imf.org/external/pubs/ft/weo/2008/01/weodata/weorept.aspx?sy=2006&ey=2013&scsm =1&ssd=1&sort=country&ds=.&br=1&c=138&s=ngdp%2cngdpd&grp=0&a=&pr1.x=43&pr1.y=5
—
Demonstrating, maintaining, and improving quality of health care requires performance measurement, measurement of treatment outcomes, and learning from feedback.
Quality of health care
The Institute of Medicine (IOM) defines quality of health care as: “The degree to which health services for individuals and populations increase the likelihood of desired outcomes and are consistent with current professional knowledge” (Institute Of Medicine, 2001). In 1966, Donabedian proposed a three-element model for quality measurement in health care, consisting of structure (char-acteristics of the health care setting), process (what is done), and outcome (the status of a patient after a series of interventions) (Donabedian, 2005; Loeb, 2004). With his model, he established the current field of performance manage-ment and outcomes research. However, measuring quality of care originates further back than Donabedian, and two historical names in that perspective are Florence Nightingale and Ernest Codman. During the Crimean war in the early 19th century, Florence Nightingale collected mortality data and infection rates for hospitals in England, resulting in a decrease of soldiers dying from bad living conditions in hospitals. It is said she was a pioneer in the visual presentation of information and statistical graphics at the military field hos-pital she managed (for instance to illustrate seasonal fluctuations of patient mortality)3. Decades later, in the early part of the 20th century, Ernest Codman
developed what is known today as outcomes management in health care, by proposing a measurement system for results of care of surgeons (Loeb, 2004). According to Codman’s “End Results Idea”, every hospital should “follow every patient it treats, long enough to determine whether or not the treatment has been successful, and then to inquire ‘if not, why not?’ with a view of preventing similar failures in the future” (Kaska & Weinstein, 1998). In Codman’s End Results Hospital each patient was provided with an “end result card” on which the surgeon filled out the details of the case before and after surgery, as well as the symptoms, the diagnosis, the treatment plan, complications that occurred, and the diagnosis at discharge. A year later the “end result card” was brought up, and based on the patient’s current condition the treatment was evaluated. Results were put on the end result card each year afterward, until a definitive determination of the result of treatment could be made. Codman also believed that the information should be public, which enabled patients and providers to compare results of various treatments among surgeons and different hospitals (Kaska & Weinstein, 1998).
ChAPTER 1 / INTROdUCTION / PAGE 10
http://en.wikipedia.org/wiki/Florence_Nightingale 3
—
Measuring quality of health careOver the years, many definitions of quality of health care have been suggested. In many health care sectors, patients as well as governments and health in-surers have become “consumers”, and doctors and treatment counselors have become “providers”, implying an important change in the roles of the players in the field (McLellan, Chalk, & Bartlett, 2007). McLellan proposes a practi-cal and usable definition of quality care as: “evidence-based treatments that are provided by licensed or credentialed practitioners who have demonstrated core competence in their practice areas and whose activities are monitored regularly by program- and system-level measurement of quality indicators” (McLellan et al., 2007, p.334). In this definition, performance or quality in-dicators establish the extent to which the care being delivered conforms to evidence-based practices. In addition, the definition can be extended to ad-herence to best practices, if no evidence-based practices are present. However, this definition does not state anything about the actual contents or aspects of care that are measured with the performance indicators. In that respect, the six core needs for health care as proposed by the IOM are very useful. In The Quality Chasm, a report in which aims for a new health system for the 21st century are laid out, the IOM presents six core needs for health care: it should be safe, effective, patient-centered, timely, efficient and equitable (Institute Of Medicine, 2001). It is on these core needs that organizations can provide performance indicators.
There is, however, debate about whether treatment outcome should be re-garded as a performance indicator. Perrin argues that outcome is only mean-ingful as a measure of performance if there is a causal relationship between the structure and process of care on the one hand and the outcome on the other hand (Perrin, 2002). The structure and the process are the performance itself and the outcome should then be the logical result of that performance. Therefore, McLellan puts measures of effectiveness and outcomes aside from performance indicators (McLellan et al., 2007). Sperry does the same in sepa-rating process evaluation from outcomes evaluation (Sperry, Brill, Howard, & Grissom, 1996). We prefer to use the model of Donabedian, where there are indicators on the way the treatment is delivered (structure and process indicators) as well as measurements of the results of the treatment (outcome). In this, the IOM core need of effectiveness is the result or the outcome, and the other five core needs are to be considered as process indicators. Measuring, maintaining and improving quality of health care through performance indicators is often described in systems for quality management.
—
Quality management in health care
During the last decades, most health care providers implemented some type of quality management system by which they can identify, measure, control, and improve the various core business processes that should lead to improved business performance. The most common term in this field is Total Quality Management (TQM). TQM is a management strategy aimed at embedding awareness of quality in all organizational processes. TQM was first promoted in Japan with the Deming prize, and was adopted and adapted in the USA as the Malcolm Baldrige National Quality Award and in Europe as the European Foundation for Quality Management award (EFQM). Both the Baldrige Award and the EFQM Model comprise of a set of standards against which the features of the organization are examined. The EFQM Model (Figure 1) consists of five areas on organizational structure: Leadership, People, Policy and Strategy, Partnerships and Resources, and Processes. These areas provide structure to describe the way an organization functions, they also function as a tool to guide organizations towards better performance. The model contains four further areas on results of the organization: People Results, Customer Results, Society Results and Key performance Results. For every result area the organization determines what measure is relevant, how the measures will be employed and what is done with the results of these measurements. The feedback loop of Innovation and Learning implies that the organization has to learn from previ-ous results to stay on track, to improve or to innovate. Empirical studies on the effectiveness of the EFQM approach are scarce (Nabitz, 2006), but single case studies suggest the approach can be successful in broadening quality awareness in teams and in the implementation of evidence-based treatments in organizations (Nabitz, Schaefer, & Walburg, 2006; Nabitz, Schramade, & Schippers, 2006).
Figure 1: The EFQM Model
—
Quality improvement by learning from feedback and benchmarkingIn management science it is assumed that quality improvement in organiza-tions profits from performance measurement and feedback of results. This is also recognized in health care management, by adopting quality systems such as TQM or the EFQM Model where innovation and learning from previous results are central concepts (Colton, 2000; Loeb, 2004; Nabitz, 2006; Thier & Gelijns, 1998). Generally, feedback of results is employed in two strategies: The first strategy is to feed back results during the treatment process, aim-ing at adjustaim-ing the concurrent process if necessary. This type of feedback is to support individual treatment professionals in their decision-making. The second strategy is the feedback of aggregated results at the end of the pro-cess. The aim is to formulate improvement projects for treatment programs or other processes in the organization, or to provide stakeholders with informa-tion on accountability. However, the effects of both strategies of feedback of performance on professional practice and health care outcomes are not clear-cut. The Cochrane Institute issued reviews on this topic, and they conclude that audit4 and feedback can sometimes be effective in improving the practice
of health care professionals. The effects seem to be small to moderate, but potentially worthwhile (Jamtvedt, Kristoffersen, O’Brien, & Oxman, 2006; Jamtvedt, Young, Kristoffersen, O’Brien, & Oxman, 2006; Thomson O’Brien et al., 2000, 2003).
A specific type of feedback provision is benchmarking. Benchmarking is a process used in management in which organizations compare various aspects of their processes in relation to that of the best practice, usually within their own sector. Established reference points are used to interpret data. These reference points can be best practices or external results, like results from randomized clinical trials (Mulder & de Loor, 2005; Walburg, 2001).
Routine outcome monitoring
A specific approach to improve clinical practice through feedback and bench-marking is routine outcome monitoring (ROM), where the focus is on measuring and reporting on health outcomes. ROM is a method to assess outcome of treatments by measuring the nature and severity of patients’ symptoms periodically, basically according to Codman’s End Result Card. This approach
ChAPTER 1 / INTROdUCTION / PAGE 13
Jamtvedt defines audit and feedback as “any summary of clinical performance of health care over a specific time period”, therefore audit and feedback will be treated as one entity, although both terms are also defined separately.
—
ChAPTER 1 / INTROdUCTION / PAGE 14
is strongly advocated by Ellwood. He proposed a “common patient-understood language of health outcomes, a national data-base containing information on clinical, financial, and health outcomes, that estimates as best as we can the relation between medical interventions and health outcomes, as well as the relation between health outcomes and money; and an opportunity for each decision-maker to have access to the analyses that are relevant to the choices they must make” (de Beurs & Zitman, 2007; Ellwood, 1988). Ellwood also noted that outcome management would be a “clinical trial machine”, i.e. a routine part of medical care that would never stop, and standards and measures would be constantly modified based on feedback. The idea was welcomed, and more publications emerged stating the importance of outcome measurement, the creation of routinely accessible “real-world” data sets, and the elimination of undesirable variations in clinical practice (Holloway, 2002). Outcomes that should be measured in ROM are the functioning and well-being of the patient, together with disease-specific outcomes. Sperry distinguishes clinical out-comes and functional outout-comes (Sperry et al., 1996). Clinical outout-comes are those that describe the physical and psychiatric signs and symptoms of a dis-ease or a disorder, whereas functional outcomes are those that describe levels of patient functioning in such areas as work, family, health and grooming, in-timate relations, self-management, and social relations. Others propose to ex-tend the set of measures to patient satisfaction, costs of treatment, or numbers of readmissions as part of the ROM system (Thier & Gelijns, 1998; Walburg, 2001).
Strategies in ROM are similar to the two strategies in the feedback of results. The first strategy in ROM is to feed back outcome during an indi-vidual treatment process, with the aim of adjusting indiindi-vidual treatments if necessary, as is illustrated with the grey lines in Figure 2. In this strategy, treatment professionals receive information about their individual patients, preferable related to existing norms. This type of feedback is to directly sup-port individual professionals in decision-making. For instance, several studies in mental health care showed that outcome of poorly responding patients in an early stage of treatment can improve with feedback on outcome (Lambert, Harmon, Slade, Whipple, & Hawkins, 2005; Sapyta, Riemer, & Bickman, 2005). The second strategy is the feedback of aggregated results over groups of patients afterwards, as is illustrated with the black lines in Figure 2. The aim of this strategy is to formulate improvement projects, not for individual patients, but for treatment modalities or programs. In addition, aggregated figures can serve as accountability figures to internal or external stakeholders (Lambert et al., 2005; McKenzie & Marks, 2003; McLellan et al., 2007; Nabitz & Walburg, 2002).
—
ChAPTER 1 / INTROdUCTION / PAGE 15
Figure 2: Feedback strategies in ROM
Defining ROM
Sperry provided definitions for outcome monitoring and outcome management (Sperry et al., 1996)5. He defined outcome monitoring as: “The use of periodic
measurement or assessment of treatment outcomes over time compared against expected outcomes in order to alter treatment, compare treatment interven-tions, or make attributions about what produced change. Typically, feedback of data to the provider serves as the basis for modifying the plan and the course of treatment”, thereby classifying the term “monitoring” as the appropriate term for the concurrent, individual feedback strategy. Outcome management is defined as: “The use of monitoring data in a way that allows learning from experience. Usually this results in reshaping or improving the administrative and clinical processes of services provided. Patient profiling and provider pro-filing are additional aspects of outcome[s] management systems”, using the word “management” for the aggregated feedback strategy.
Routine outcome monitoring (ROMon) therefore refers to the concurrent,
Sperry et al. use the plural form of “outcome” in their glossary, i.e. “outcomes monitoring” and “outcomes management”. For matters of consistency, we used the singular form in this text. 5
—
ChAPTER 1 / INTROdUCTION / PAGE 16
individual feedback strategy and routine outcome management (ROMan) to the aggregated feedback strategy. Although the terms ROMon and ROMan are attractive from an educational point of view, we will use the term routine outcome monitoring (ROM) for both strategies with reference to one of the two strategies – the concurrent, individual (monitoring) strategy or the aggregated (management) strategy. This choice is partly conceptual and partly practical. As Sperry’s definitions imply, data collected for outcome monitoring are frequently also used for outcome management and both activities are often intertwined, justifying the use of a term that encloses both strategies and activities. In our experience, “monitoring” covers both strategies best. In regard to the practical choice, the distinction between “monitoring” and “management” vanishes with the convenient use of the abbreviation ROM. Therefore; we chose to use one meaning for the letter “M” in ROM term, providing an addition to the term to clarify the used strategy when needed.
ROM in mental health care and substance abuse treatment
As early as 1990, the Institute of Medicine described the principles of outcome monitoring in the substance abuse treatment sector (Institute Of Medicine, 1990). Sperry contributed a volume on ROM in mental health care and substance abuse treatment, in which several ROM projects were presented as illustrative cases (Sperry et al., 1996). Gradually, ROM projects and publications started to appear in mental health care (Holloway, 2002; McKenzie & Marks, 2003; Sperry et al., 1996) and in the area of substance abuse treatment (Evans & Hser, 2004; Harrison & Asche, 2001a; Moos, Finney, Federman, & Suchinsky, 2000; Tiet, Byrnes, Barnett, & Finney, 2006). These studies were mainly focused on evaluating the service delivery system and enhancing treatment outcomes. In all cases, the assumption is that programs or organizations will profit from ROM through learning from the feedback of outcome data.
Learning organizations
A concept related to feedback and benchmarking of performance and outcomes is the “learning organization”. In Scotland and England, the National Health Services have been encouraged to become learning organizations to meet the demands of reform and change in health care (Kelly et al., 2007; Rushmer et al., 2007) and it is stated by Ellis that a learning organization is required for benchmarking to be effective (Ellis, 2006). Learning from experience was also mentioned in Sperry’s definition of outcome management. According to the organization model of Senge (Senge, 1992), learning capacity is associated
—
ChAPTER 1 / INTROdUCTION / PAGE 17
with organizational effectiveness, and the learning principles of shared vision and team learning are predictors for organizational effectiveness (Jeong, Lee, Kim, Lee, & Kim, 2007; Kelly et al., 2007). Since the theory of Senge incorporates the role of feedback, it is very applicable in the field of quality improvement through ROM.
In his book The Fifth Discipline, Senge defined a learning organization as one that possesses five core-learning disciplines: Personal Mastery, Mental Models, Shared Vision, Team Learning, and Systems Thinking (Senge, 1992). Systems Thinking is stressed as the core discipline that integrates the others into a unity of theory and practice. Systems Thinking is meant as “the fifth discipline”. Another essential discipline is Team Learning, because this is a requirement for organizational learning: only when teams have the ability to learn, is organizational learning attainable. The combination of these five disci-plines is a permanent course of practice for individuals, teams, and organizations. The five disciplines of a learning organization are (Senge, 1992):
1. Personal Mastery – the process of expanding personal capacity and con-tinually improving one’s level of expertise in order to achieve desired goals.
2. Mental Models – internal pictures, assumptions, and generalizations of the world that influence perceptions, reactions and decisions.
3. Shared Vision – the capacity to develop commitment to and in an orga-nization or a group, by establishing the principles and guiding practices to develop shared pictures of the future desired by members.
4. Team Learning – the capacity of groups to acquire collective thinking skills through dialog, in order to develop intelligence and results that wouldn’t be reached individually.
5. Systems Thinking – the ability to use knowledge and instruments in order to perceive the greater patterns in systems and organizations and to change them effectively.
The theory of Senge implies that learning organizations are expected to book better results than organizations with less learning capacity. Senge em-phasizes the role of feedback in the learning organization. ROM therefore plays an important role in the learning organization and quality improvement through feedback and benchmarking of performance and outcomes.
Aim of the thesis
Policymakers, managers, and clinicians in the sector of mental health care and substance abuse treatment endorse the merits of ROM. However, from multiple implemented projects and available reports and studies on ROM in scientific literature, some unsolved issues have emerged. It is the aim of this
—
ChAPTER 1 / INTROdUCTION / PAGE 18
thesis to elaborate on these issues and to suggest possible solutions. Firstly, there is the issue of the feasibility of data collection and the validity of the col-lected data for ROM. Data collection can be expensive and time consuming (McKenzie & Marks, 2003; Tiet et al., 2006). As a result, response rates can be very low and results may be of questionable validity for the target popula-tion. Up to 20 interviewer hours per patient are reported, inclusion of patients can be very difficult and response rates can be as low as 48% (Gerstein & Johnson, 2000; Harrison & Asche, 2001a; Rosenheck & Seibyl, 2005). This is especially problematic in the case of aggregated ROM data, but also for individual concurrent feedback, since low response rates imply that only few patients will profit from this quality improvement. Secondly, while researchers, policymakers, and top managers generally support ROM projects; this support is often much lower under treatment professionals and lower-management staff (Harrison & Asche, 2001a; Teruya, Hardy, Hser, & Evans, 2006), result-ing in low credibility of the ROM system. Thirdly, databases developed for ROM can be a rich source for add-on research on effectiveness of substance abuse treatment. But, in many cases, the methodology of studies in the field of quality lack precision in operationalization and standardized instruments (Nabitz, 2006). The question is whether it is possible to retrieve information from these databases to perform research on effectiveness on the database – we will provide an example of this. The fourth aim of this thesis is to elaborate on feedback and the learning organization. It is believed that the availability of information on treatment outcome, on the individual real-time level as well as on the aggregated level enhances the quality of treatment, although results of studies are not clear-cut and effects vary widely (Jamtvedt, Young et al., 2006; Thomson O’Brien et al., 2000, 2003). Therefore, an appropriate question is whether delivering feedback has an effect on the prerequisite for quality improvement, i.e. on the learning capacity of individuals, teams and organiza-tions. In other words, does ROM have an effect in terms of Senge’s concept of learning organizations? These are questions on the effectiveness of feedback and benchmarking.
The questions in this thesis are concentrated around implementing ROM in substance abuse treatment and an attempt to answer the question: Is it pos-sible to implement a kind of ROM that is feapos-sible, generates valid data, is sup-ported by key persons in services, can be used to generate conclusions about effectiveness and, finally, will contribute to the learning capacity of employees?
In answering the research questions, this thesis is limited to the ROM strategy of collecting data for feedback of aggregated results over groups of patients afterwards. This means that results and conclusions of this thesis are of re-stricted value for ROM projects aimed at the strategy of individual feedback
—
ChAPTER 1 / INTROdUCTION / PAGE 19
on the concurrent treatment process. However, since it is not uncommon that these data are also used to report on aggregated levels (de Beurs & Zitman, 2007; Zwanepol & De Groot, 2008), this thesis is also of value for that type of application. The current research is performed within the setting of the substance abuse treatment sector in the Netherlands. This sector is part of the mental health care sector and most organizations are members of the branch organization for Dutch Institutions for Mental health Care and Addiction Services (GGZ Nederland). Although research is carried out in the substance abuse treatment sector, we expect the findings to be relevant for the entire mental health sector.
Setting of the thesis
The Netherlands has an extensive network of medical and psychosocial treat-ment facilities for people with substance use disorders or pathological gam-bling. The centers are part of the mental health services. In 2006, a total of 60,976 patients were registered in Dutch outpatient and inpatient facilities. Almost 50% of them had a primary diagnosis of alcohol dependency, 21% had a diagnosis of opiate dependency, and 16% and 11% were dependent on cocaine and cannabis, respectively (van Laar, Cruts, Verdurmen, van Ooyen-Houben, & Meijer, 2008). Funding of the alcohol and drug treatment services is public. The costs for substance abuse treatment in 2000 amounted to around 125 million euro’s (ca. US$180 million), 51% of which was spent on outpatient services and 49% on inpatient services.
The definition of quality of care of the IOM implies that desired health outcomes should be achieved and current knowledge should be applied: this was the exact reason for a major reorganization that took place in Dutch sub-stance abuse treatment services during the last decade. It was partly in reaction to public criticism that substance abuse treatment was not effective and not contributing to solving social problems and public nuisance. As a consequence, the sector had a serious image problem and treatment services suffered some serious drawbacks (Nabitz, Oudejans, Brink van den, & Vis, 2006; Nabitz, Vis, & Brink van den, 2001; Schippers, Schramade, & Walburg, 2002; Schippers, van Es, Mulder, & Dijk van, 2005). Therefore, the sector started a nation-wide government supported reform called “To Score Results”. The objective was to redesign existing treatment services on the basis of scientific evidence regarding efficacy and effectiveness and to improve treatment practices on the basis of feedback on clinical and societal outcomes (Schippers et al., 2002). The existing treatment services were reorganized into services with a central intake location where protocollized treatment-allocation took place to evidence-based
—
ChAPTER 1 / INTROdUCTION / PAGE 20
interventions. However, the implementation of a system for the improvement of treatment practices based on feedback of outcomes, lagged behind. To boost this part of the project, the ROM project Benchmark Lifestyle Training in sub-stance abuse treatment was started. In this project, outcomes of an outpatient treatment for substance dependence at four treatment centers are measured routinely and retrospectively fed back and benchmarked at the aggregated level to treatment professionals, managers and Boards of Directors. Next, we formu-lated research questions concerning implementation, feasibility and effective-ness of the ROM project. This thesis is a result of this research.
Overview of the thesis
Chapter 2 deals with the topic of feasibility and validity of data collected in
ROM. It primarily describes the feasibility of a telephonic follow-up interview for ROM. Post-treatment follow-up interviews, designated to monitor and evaluate outcomes after treatment has ended are an important part of ROM. However, follow-up interviews are costly and time consuming. Investments of up to 20 interviewer hours per patient are reported in the literature, and re-sponse rates are generally (much) lower than 70% (Gerstein & Johnson, 2000). Therefore, we implemented a telephonic follow-up interview to conduct these interviews and studied the feasibility in terms of response rate, time invest-ment and costs. Next, we investigated whether the collected data showed va-lidity in terms of tracking difficulty and representativeness. We also studied the relationship between response-related baseline characteristics and treat-ment outcome. This was in order to assess whether the data suffered from selection bias, which would limit the value of generalizing outcomes from the follow-up sample to the total treated population.
Chapter 3 describes facilitating and impeding factors and the observed
sup-port for routine outcome monitoring (ROM) in substance abuse treatment centers in the Netherlands. During implementation of ROM we encountered problems in data collection and in the participation in feedback sessions. In order to identify facilitating and impeding factors and support for the system, key persons and professionals from four substance abuse treatment centers in the Netherlands were interviewed and filled out questionnaires to find out their opinions on the ROM project.
Chapter 4 is a naturalistic outcome study. It is an example of how databases
for ROM can be utilized to conduct effectiveness studies. Such studies are a necessary addition to randomized controlled trials (RCTs) with their highly-selective patient samples and their overly-structured treatments. In RCTs, ef-ficacy is tested, i.e. what are the results of a treatment under ideal conditions?
ChAPTER 1 / INTROdUCTION / PAGE 21
—
As a consequence, most if not all RCTs have low external validity and do not offer a realistic estimate of effectiveness, i.e. how well a treatment works in the real world of day-to-day clinical practice. Therefore, naturalistic outcome studies are performed in order to evaluate the merits of interventions in a rou-tine treatment environment. In this chapter, we examined the feasibility and effectiveness of two evidence-based outpatient psychotherapeutic interven-tions for patients with alcohol use disorders. Furthermore we assessed whether pre-treatment patient characteristics are predictive of treatment outcomes for these interventions.
Chapter 5 describes the development and psychometric evaluation of the
Questionnaire for Learning Organizations (QLO). The QLO is based on the theory of Learning Organizations of Senge. Psychometric properties (reliability and factor structure) of the QLO are assessed for a group of treat-ment professionals and a group of supporting employees in substance abuse treatment centers, in order to evaluate the quality of the questionnaire.
Chapter 6 investigates whether ROM plays a role in enhancing learning
capacity in terms of Senge’s theory of Learning Organizations. In a quasi-experimental pre-post design, we investigate whether treatment professionals who engaged in the ROM system show an increase in learning potential.
This thesis ends with a general discussion and a summary, where the conclusions from all chapters are compiled and discussed and suggestions for future research will be given.
—
—
—
/ PAGE 22 C h A P T E R 2—
—
ChAPTER 2
—
fEASIBILITY
& vALIdITY Of
LOW-BUdGET
TELEPhONIC
fOLLOW-UP
INTERvIEWS IN
ROUTINE OUTCOME
MONITORING (ROM)
Of SUBSTANCE
ABUSE TREATMENT
ACCEPTEd fOR PUBLICATION / AddICTION
fE A S IB IL IT Y & v A LId IT Y
ChAPTER 2 / fEASIBILTY & vALIdITY / PAGE 23
OUdEjANS, S.C.C. SChIPPERS, G.M. MERkx, M.j.M. SChRAMAdE, M.h. kOETER, M.W.j. vAN dEN BRINk, W.
—
ChAPTER 2 / ABSTRACT / PAGE 24
Abstract
Aims: Routine outcome monitoring (ROM) is receiving growing attention.
However, follow-up interviews are time consuming and costly. This study ex-amines the feasibility and validity of low-budget telephonic follow-up interviews for ROM in a substance abuse treatment center (SATC).
Design: Observational study using data collected for routine outcome monitoring. Setting: The study was performed in a SATC in an urban area in the Netherlands. Participants: Feasibility and validity were assessed on data of 2,325 patients. Measurements: Data on pre-treatment socio-demographic and clinical
charac-teristics was collected using electronic patient records (EPRs) and the European version of the Addiction Severity Index (EuropASI). Data on intensity of treatment was also collected through the EPRs. Telephonic follow-up interviews were conducted between 9 to 10 months after intake.
Findings: A 53% follow-up rate was achieved, 32% of the patients could not be
contacted, 3% explicitly refused, and in 9% other reasons accounted for non-participation. About 50% of the interviews took place in the intended time frame. Costs were €40 ($57) per completed interview. There were indications of selection bias because patients with cocaine as their primary problem and patients with polysubstance abuse were under-represented in the follow-up sample and because the presence of these disorders is associated with negative treatment outcome.
Conclusions: Implementing telephonic low-budget follow-up interviews for
ROM is feasible, but selection bias threatens internal validity of data, limiting generalization to the total treatment population. Increased efforts to track patients for follow-up may improve generalization.
—
—
IntroductionSubstance abuse treatment centers (SATCs) currently face two challenges: (1) demonstrating programmatic transparency and accountability regarding treat-ment outcomes and (2) improving practices based on assesstreat-ment and feedback of these outcomes. In the Netherlands, SATCs are supported in this endeavor in a nationwide quality-enhancing reform program targeting an evidence-based, transparent and accountable treatment system (Schippers et al., 2002). As a result, most SATCs recently redesigned their treatment programs into evidence-based treatment systems and most SATCs are currently in the pro-cess of implementing routine outcome monitoring (ROM) as an instrument for quality improvement. Post-treatment follow-up interviews, designated to monitor and evaluate outcomes after treatment has ended are an important part of ROM (Evans & Hser, 2004; Gossop, Marsden, & Stewart, 2000; Harrison & Asche, 2001; Meijerink, 2003; Moos, Humphreys, Ouimette, & Finney, 1999; Project Match Research Group, 1997). However, follow-up interviews are costly and time consuming. Investments of up to 20 interviewer hours per patient are reported in follow-up studies like the Drug Abuse Treatment Outcome Studies (DATOS), the National Treatment Improvement Evaluation Study (NTIES), the Services Research Outcome Study (SROS) and California Drug and Alcohol Treatment Assessment (CALDATA) reaching response rates between 48% and 70% (Flynn, Simpson, Anglin, & Hubbard, 2001; Gerstein & Johnson, 2000; Rosenheck & Seibyl, 2005). Although some ROM projects report response rates above 80% of included patients, inclusion procedures were not transparent, resulting in outcome samples with questionable repre-sentativeness (Evans & Hser, 2004; Moos et al., 1999; Rohrer, Vaughan, Cadoret, & Zwick, 1999). Moreover, Tiet et al. (Tiet et al., 2006) report costs up to $186 per patient to obtain a 67% follow-up rate with a mailed questionnaire in the Veterans Affair (VA) Outcome Monitoring Project (further referred to as OMP). These are personnel costs involving the actual up efforts only. Telephonic follow-up interviews carried out by an independent survey vendor in the Minnesota Outcome Monitoring System (OMS) were much cheaper at $30 per completed interview reaching a follow-up rate of almost 65% (Harrison & Asche, 2001).
Considering this, several SATCs in the Netherlands have implemented a system to conduct telephonic follow-up interviews for ROM, in the hope of collecting valid data and keeping costs and interviewer hours under control. In collaboration with the Amsterdam Institute for Addiction Research (AIAR), some of these SATCs installed an independent call center, staffed by inter-viewers conducting telephonic follow-up interviews.
Low response rates in the field of substance abuse treatment evaluation or
—
ChAPTER 2 / fEASIBILTY & vALIdITY / PAGE 26
monitoring are mainly caused by attrition, because many patients are difficult to reach. Non-response due to refusal, as is often the case in general popula-tion surveys, is usually a less important cause, although this varies between 2% in DATOS and 29% in CALDATA (Gerstein & Johnson, 2000; Grella, Hser, Joshi, & Anglin, 1999). Low response-rates are problematic because they may result in selection bias that may affect internal validity, i.e. patients with poor outcome are more difficult to find for follow-up or more likely to refuse. Several studies show this threat to be real (Lahaut, Jansen, van de Mheen, & Garretsen, 2002; Marcus & Schutz, 2005; Mond, Rodgers,
Hay, Owen, & Beumont, 2004; Rohrer et al., 1999; Rosenheck & Seibyl, 2005; Scott, 2004). A number of studies reported associations between tracking difficulty or follow-up time and negative outcome, but these finding are not consistent (Digiusto, Panjari, Gibson, & Rea, 2006; Hansten, Downey, Rosengren, & Donovan, 2000; Stinchfield, Niforopulos, & Feder, 1994; Walton, Ramanathan, & Reischl, 1998). As a consequence, accountability figures are potentially overestimating the positive outcome of a treatment program.
This study evaluates the feasibility of ROM with a call center and establishes the internal validity of the collected data with a central focus on selection bias.
Methods Setting
The call center was operational five days a week, calling patients between 4pm and 8pm. The five call center interviewers were specially-trained psychology students. A psychologist supervised the interviewers and a research assistant managed the call center. Follow-up time was set at 9 months after intake. Most treatment programs in participating SATCs last between 3 and 6 months. Measuring outcome at 9 months would reveal results from patients 3 to 6 months after treatment completion.
In this study we used data of one participating SATC covering an urban area of approximately 1 million inhabitants. Treatment in this SATC is organized into five levels of care: (1) brief outpatient treatment, (2) standard outpatient treatment, (3) day treatment, (4) inpatient treatment, and (5) harm-reduction treatments. Intake counselors allocate patients to an intensity of treatment following a structured matching and allocation protocol (Merkx et al., 2007; Schippers et al., 2002).
Patients
Patients were eligible for follow-up if a) their last treatment intake assessment at the SATC took place 9 months earlier and b) they were no longer in treatment.
—
ChAPTER 2 / fEASIBILTY & vALIdITY / PAGE 27
In this study we focus on a cohort of 2,572 patients who had their intake as-sessment in the year 2004. Of these, 233 were still in treatment 9 months after intake, and 14 had a subsequent intake in the 3 months following their initial intake assessment date. The follow-up date of these 14 patients was “pushed” into the subsequent cohort, just as the follow-up dates of some patients of the former cohort were pushed into the cohort of this study. This resulted in 2,325 patients eligible for analysis.
Procedure and assessments
Data on pre-treatment socio-demographic and clinical characteristics were collected using the electronic patient records (EPR) where administrators and intake counselors entered information following admission to the SATC. Administrators collected socio-demographic characteristics during the first admission session. Next, intake counselors assessed clinical characteristics using the European version of the Addiction Severity Index (EuropASI) 5th edition (Kokkevi & Hartgers, 1995). The EuropASI is a semi-structured inter-view providing information on substance use and substance-related problems in different problem areas: medical, employment/education, alcohol, drugs, legal, family/social support, psychiatry and gambling. Items from each area are used to generate interviewer severity ratings, which provide an assessment of the overall problem severity in that area. Data on intensity of treatment were also collected from the EPR.
The call center collected follow-up data with a structured interview, in-cluding a selection of 20 questions about the 30 days preceding the follow-up interview from the seven areas of the EuropASI. Lists of patients eligible for follow-up were printed from the EPR on a monthly basis, displaying contact details about the patient and his or her collaterals. Collaterals were contacted when repeated direct contact attempts had failed. A maximum of four attempts to contact a patient were made, the attempt with the collateral excluded.
Interviewers confirmed name and date of birth before starting the inter-view. To assess whether the follow-up interview was valid, interviews ended with two questions. The first is whether the interviewer had any indication that the interview was contaminated by an incorrect reproduction of reality by the patient. The second question is whether the interviewer had any indica-tion of the patient having difficulties understanding the quesindica-tions. A “yes” on either of these questions deemed the interview invalid.
Feasibility in this study is defined as: a) a response rate that is consistent with response rates known from DATOS, NTIES, SROS, CALDATA and the Minnesota OMS, i.e. above 48%; b) low refusal rates, i.e. below 10%; c) the
—
ChAPTER 2 / fEASIBILTY & vALIdITY / PAGE 28
majority of the interviews are conducted within the intended time frame, i.e. 9 to 10 months after intake; d) time investment per interview – including tracking time – is limited to one hour per interview; and finally e) costs per interview are similar or lower than the costs reported by the Minnesota OMS, i.e. $30.
Internal validity of collected data is defined as: a) no association between tracking difficulty (i.e. time to follow-up and number of contact attempts) and outcome; b) no differences between respondents and non-respondents on socio-demographic, pre-treatment clinical characteristics and process variables that have an association with treatment outcome.
Data-analysis
Response rate was defined as the number of patients with a valid follow-up interview (“respondents”) divided by the number of patients eligible for follow-up. Reasons for non-response were collected from the records of the call center. Time between intake and follow-up interview was defined as the number of days between date of assessment at intake with the EuropASI and date of the follow-up interview. A follow-up time between 9 and 10 months (275 and 305 days) after intake was considered satisfactory. Time investment of the inter-views was estimated by dividing the total time investments by the interviewers by the number of interviews being held. Costs were defined as total costs of the call center divided by the total number of valid interviews. Not included were costs for analyzing, interpreting, and disseminating findings from the data. Cost calculations were based on figures from the call center year report of 2004.
Tracking difficulty was operationalized in follow-up time and number of contact attempts to reach a respondent. The relation between follow-up time and treatment outcome was defined as the difference in mean follow-up time between respondents with and without positive treatment outcome. Association between number of contact attempts and treatment outcome was assessed by calculating the percentage of respondents with positive treatment outcome for each group of respondents that was reached after one, two, three, four or more attempts. Up to four contact attempts were recorded. In reality more than four attempts were made when interviewers had the opportunity, implying that a recorded amount of four attempts means that at least four attempts were made.
Treatment outcome was based on information about primary and second-ary substance use reported by the patient during intake assessment and items in the alcohol, drugs, and gambling areas from the follow-up interview. Positive treatment outcome was defined as abstinence or no excessive use of primary and secondary substance in the 30 days preceding the follow-up interview.
—
ChAPTER 2 / fEASIBILTY & vALIdITY / PAGE 29
No excessive use of alcohol was defined as a maximum of 21 days of alcohol use (14 days for women) and no days with 5 or more glasses. The first criterion requires alcohol-free days – preferably 2 days a week – as is advised by Dutch informational institutes and their public websites6 to prevent habituation. The
second criterion is modeled after the criterion for binge drinking from the ASI manual, and is in line with the advice of the Health Council of the Netherlands to drink moderately on drinking days (Health Council of the Netherlands, 2006; Kokkevi & Hartgers, 1995). No excessive use of other substances and gambling was defined as a maximum of 6 days of use or gambling. Negative treatment outcome was defined as excessive use of primary and/or secondary substance, e.g. more than 21 days of alcohol and/or days with 5 or more glasses; or more than 6 days of gambling or use of other substances.
To test for differences between respondents and non-respondents in terms of socio-demographic data and pre-treatment clinical characteristics and to test whether these characteristics were associated with treatment outcome, we used a two step procedure. First, we applied multiple logistic regression analysis to select the strongest independent predictors for becoming a respondent. Next, we tested – again with multiple logistic regression analysis – which characteristics served as independent predictors for positive treatment outcome. In both cases a backward stepwise elimination procedure was employed. Because the allocation and matching guideline used clinical variables to allocate patients to intensity of treatment, resulting in a strong association between these variables and intensity of treatment, the latter was not included in the regression analyses.
Results Subjects
Table 1 presents the pre-treatment socio-demographic, clinical characteristics, and process variables of the 2,325 patients, showing a mixed inpatient and out-patient population, with the largest group primarily seeking help for alcohol problems (49%). A substantial portion of the patients reported polysubstance abuse (36%). These findings are corroborated by the ASI interviewer severity ratings (ISRs), indicating that almost half of the patients were in need of treat ment for alcohol and the remaining for drugs. Additional problem areas were psychiatry, family and social support, and employment and education, with large numbers of patients in need of treatment as indicated by ISRs above 3.
http://www.alcoholdebaas.nl/Stoppen,-minderen-/Hoeveel-is-veel-.aspx This website is maintained by Tactive, a commercial branch of the Tactus, regional public SATC in the Netherlands
—
ChAPTER 2 / fEASIBILTY & vALIdITY / PAGE 30
Table 1:
Process variables, pre-treatment socio-demographic and clinical characteristics of patients eligible for follow-up
Total sample (n = 2,325) Respondents (n = 1,237)* Positive treatment outcome# (n = 499)** % of total n/Mean (sd) Treatment intensity χ2 (6) = 92.2; p = 0.00 Brief Outpatient 13 18 18 Standard Outpatient 23 25 25 Day Treatment 5 6 5 Inpatient 13 11 10 Harm reduction 14 11 10 No intensity/unknown 4 5 5
No treatment after intake 29 26 28
t(2290) = 6.35; p = 0.00 Age 41.7 (11.2) 43.0 (11.7) 42.7 (12.1) Sex χ2 (1) = 9.02; p = 0.00 Male 72 69 70 Ethnic Origin χ2 (1) = 65.8; p = 0.00 χ2(1) = 16.3; p = 0.00 Dutch 77 84 79 Civil Status χ2 (1) = 14.3; p = 0.00 Living alone/single 84 81 79 Educational Level χ2 (2) = 32.4; p = 0.00 No education/low 31 26 27 Middle 40 41 41 High 29 33 32
Primary substance use χ2
(6) = 72.1; p = 0.00 χ2(6) = 84.4; p = 0.00 Alcohol 49 53 43 Heroin 7 6 6 Cocaine 19 14 19 Cannabis 10 11 12 Gambling 6 5 11 Nicotine 6 8 5 Other 4 4 4 Polysubstance abuse χ2 (1) = 18.2; p = 0.00 χ2(1) = 27.8; p = 0.00 Yes 36 32 24
ASI severity scores > 3
Medical 20 20 18 Employment/education 31 25@ 25 Alcohol 49 52$ 44! Drugs 39 34% 35 Legal 8 5& 5 Family/social support 36 31^ 30 Psychiatry 40 39 36 Gambling 5 5 10
# positive treatment outcome – abstinence or no excessive use of primary and secondary substance in the 30 days preceding the follow-up interview. No excessive use of alcohol: maximum 21 days (14 days for women) of alcohol use and no days with 5 or more glasses.
No excessive use of other substances and gambling: maximum 6 days of use or gambling. * test statistics are given for respondents vs. non-respondents
** test statistics are given for positive vs. negative treatment outcome @ χ2
—
ChAPTER 2 / fEASIBILTY & vALIdITY / PAGE 31
Feasibility
Table 2 shows that in 1,237 of the 2,325 eligible patients (53%) a valid follow-up interview could be obtained.
About 31% of patients were interviewed less than 275 days (9 months) after in-take, and 18% were interviewed more than 305 days (10 months) after intake. Subsequently, 50% of patients were interviewed between 9 and 10 months after intake. Mean time between intake and follow up was 9.5 months (sd = 0.86).
Table 3 shows that the main reason for non-response was lack of a valid phone number or no contact at all (35%). In only 3% of cases, the patient explicitly refused the interview.
Table 2:
Total and yearly response rate, proportion interviews in time*, mean follow-up time and number of attempts per interview
Total N eligible for follow-up 2,325 Valid interviews
N 1,237
% 53
Time to follow-up interview
< 9 months 31
9 – 10 months 50
> 10 months 18
Follow-up time in months
Mean 9.5
SD 0.86
Number of contact attempts
1 35 [19]#
2 25 [32]
3 13 [39]
4 or more 27 [53]
* in time: between 275 and 305 days (9 and 10 months) after intake; # percentages in brackets express cumulative response rate
Table 3:
Results for response
Reasons for non-response n % Invalid phone number/no contact 817 35.1 Refusal client (angry, frustrated, no time) 77 3.3
Other* 194 8.3
Valid interview 1,237 53.2
Total eligible 2,325 100
* including 29 interviews scored as “invalid”, other reasons include language problems; patients that turned out to be deceased; deaf or hearing impaired patients, and intoxicated patients.
—
ChAPTER 2 / fEASIBILTY & vALIdITY / PAGE 32
On average a completed interview took 46 minutes, including contacting and interviewing. Not included was time for supervisors and management. The total costs for one completed interview was €40 ($57)7, including personnel costs
for interviewers, manager, supervision, and overhead costs, such as use of office space, phone contracts, computers and software licenses (estimated at 10% of the total costs). The total completed interview costs do not include analyzing, interpreting and disseminating the findings from the follow-up data.
Validity
In total, 42.2% of the 1,237 respondents reported a positive treatment outcome. Mean follow-up time was similar for respondents with positive treatment out-come and respondents with a negative treatment outout-come: 9.4 months (sd = 0.90) versus 9.5 months (sd = 0.85) respectively (t1176 = 1.02; p = 0.31). Treatment
out-come was not significantly associated with tracking difficulty: 46% of respon-dents reached after one attempt had a positive outcome and 38% reached after four or more attempts had a positive treatment outcome (χ2
(3) = 5.8; p = 0.12).
Table 1 shows percentages on process, socio-demographic, and clinical variables within the group of respondents (n = 1,237). Patients who received more intensive treatment were under-represented, just as those who received no treatment after intake. In addition, under-representation was found for younger, male, and non-Dutch patients, patients who did not live alone, patients with low education, patients with cocaine as their primary problem drug and patients who reported polysubstance abuse. In addition, patients who were more in need of interventions (as indicated by an ISR higher than 3 on em-ployment/education, legal problems, drugs, and family/social support) were under-represented. Patients who just needed help for their alcohol problem were over-represented in the group of respondents. Table 1 also shows percentages on process, socio-demographic and clinical variables within the group of patients with positive treatment outcome (n = 499). Dutch patients were under-represented, as well as patients with alcohol as their primary substance, patients who reported polysubstance abuse and patients with an alcohol ISR higher than 3. Patients with cocaine as their primary substance were over-represented amongst those with positive treatment outcome.
Although statistically significant, differences on baseline characteristics between respondents and non-respondents and between those with positive
Using a euro-dollar rate of 1.4369, level of September 2008 as reported by the European Central Bank at http://sdw.ecb.europa.eu/browseSelection.do?dataset=0&freq=m¤cy=usd&node =2018794
—
ChAPTER 2 / fEASIBILTY & vALIdITY / PAGE 33
and negative treatment were not large. In terms of Cohens’ h, effect sizes for ethnic origin, primary substance use and polysubstance abuse (vari-ables that showed both significant differences in the 3rd and 4th column
of Table 1) are small to moderate (Cohen, 1988). The largest effect size between respondents and non-respondents is found for ethnic origin (h = 0.36), and the largest one between patients with and without positive treat-ment outcome for patients who have gambling as their primary problem (h = 0.42).
Table 4 shows the results of the multiple logistic regression model with respondent status as the dependent variable. Compared to patients primarily seeking help for alcohol problems, patients primarily seeking help for cocaine were less likely to become a respondent. Older patients were more likely to be-come respondents. Non-Dutch patients were up to twice less likely to bebe-come respondents than Dutch patients. Patients abusing only one substance were 21% less likely to become respondents than patients without polysubstance abuse. Finally, chances to become a respondent decreased with increasing ISRs for employment/education, legal and family/social support.
Table 4:
Multiple logistic prediction for becoming a respondent#, n = 2,325
OR P value 95% CI
Age 1.01 0.02 1.00, 1.03
Ethnic Origin
Non-Dutch vs. Dutch 0.54 0.00 0.41, 0.72 Clinical variables
Primary substance use
Alcoholref 1.00 (0.09) Heroin 0.84 0.48 0.52, 1.36 Cocaine 0.67 0.02 0.48, 0.94 Cannabis 1.23 0.33 0.81, 1.85 Gambling 0.82 0.43 0.49, 1.35 Nicotine 1.11 0.71 0.64, 1.90 Other 1.40 0.30 0.75, 2.62 Polysubstance abuse Yes vs. No 0.79 0.06 0.62, 1.01
ASI severity scores
Employment/education 0.94 0.04 0.88, 1.00
Legal 0.84 0.00 0.77, 0.92
Family/social support 0.94 0.05 0.88, 1.00
Constant 1.69 0.08
ref = reference category
(for primary substance use, p-value between parentheses is the p-value for the omnibus test for all levels of the variable) # Backward Stepwise elimination (likelihood ratio; p in = 0.05, p out = 0.10) method from SPSS 13.0 for MacOSx; variables not in the equation: sex, age, ethnic origin, educational level, severity scores medical, alcohol, drugs, family/ social support, psychiatry, and gambling.
—
ChAPTER 2 / fEASIBILTY & vALIdITY / PAGE 34
Table 5 presents the predictors in the multiple logistic regression model with positive treatment outcome as dependent variable. Patients living alone, seeking primary help for alcohol or nicotine problems, and those reporting polysubstance abuse were less likely to report positive treatment outcome. Cocaine abusers and gamblers were most likely and smokers were least likely to have a positive treatment outcome.
Common predictors for becoming a respondent and reporting positive treat-ment outcome in this sample are substance of abuse and polysubstance abuse. Patients primarily seeking help for cocaine problems are under-represented and more likely to report positive treatment outcome, and patients abusing more than one substance are under-represented and are less likely to have a positive treatment outcome.
Conclusions and discussion
Results of this study show that it is feasible to implement a telephonic low-budget follow-up system for ROM with a call center, but that this method does not guarantee a representative follow-up sample of the treated population, thereby limiting a valid interpretation of the collected data. Patient tracking problems turned out to be the major reason for non-response.
Table 5:
Multiple logistic prediction# for positive treatment outcome@; n = 1,237
OR P value 95% CI
Civil Status
Living alone vs. Not living alone 0.52 0.00 0.35, 0.77 Primary substance use
Alcoholref 1.00 (0.00) Heroin 1.66 0.17 0.83, 3.32 Cocaine 3.69 0.00 2.27, 6.00 Cannabis 2.00 0.36 1.24, 3.24 Gambling 8.95 0.00 3.98, 20.16 Nicotine 0.42 0.00 0.22, 0.80 Other 1.37 0.99 0.66, 2.84 Polysubstance abuse Yes vs. No 0.38 0.00 0.26, 0.54 Constant 1.14 0.47
ref = Reference category
(for primary substance use, p-value between parentheses is the p-value for the omnibus test for all levels of the variable) # Backward Stepwise elimination (likelihood ratio; p in = 0.05, p out = 0.10) method from SPSS 13.0 for MacOSx; variables not in the equation: sex, age, ethnic origin, educational level, severity scores medical, employment/education, alcohol, drugs, legal, family/social support, psychiatry, and gambling
@ Positive treatment outcome: abstinence or no excessive use of primary and secondary substance in the thirty days preceding the follow-up interview. No excessive use of alcohol: a maximum of 21 days of alcohol use (14 days for women) and no days with 5 or more glasses. No excessive use of other substances and gambling: a maximum of 6 days of use or gambling. Negative treatment outcome was defined as excessive use of primary and/or secondary substance.
—
ChAPTER 2 / fEASIBILTY & vALIdITY / PAGE 35
Feasibility results were fairly positive. The 53% response is modest, but similar to the 48% reported for DATOS. Higher response rates are reported (DATOS: 70%), but this was after exclusion of patients deemed inaccessible by virtue of incarceration or hospitalization (Flynn et al., 2001). In the current study, these patients were not excluded and therefore the response rate is com-parable with the lower rate of 48% reported by Gerstein (Gerstein & Johnson, 2000). Refusal rates were low, although refusal rate might increase when more patients are contacted. Non-respondents to surveys tend to be less agreeable and less open to experience than respondents (Marcus & Schutz, 2005). These characteristics might be associated with telephonic traceability (e.g. making less effort to be contacted, or inability to maintain a contract with a provider), and therefore it is conceivable that the chances of refusal are bigger amongst non-respondents. Time investment was less than one hour. The cost of €40 ($57) is far below the $186 reported by the OMP (Tiet et al., 2006), but almost double the $30 reported by the Minnesota OMS conducted by an independent survey vendor. However, €40 in 2004 is similar to almost $33 in 20018 (the
year of Harrison’s Minnesota OMS paper), taking into account inflation and currency rates. In conclusion, response rates are moderate but similar to other studies, and costs are low and comparable with costs of interviews conducted by independent survey vendors.
It turned out to be difficult to interview patients at the intended follow-up time. Only 50% of the interviews were held in the 9 to 10 months time frame. This needs to be improved, because the presence of excessive follow-up times can induce bias, comparisons between patients become difficult and the relationship between treatment and outcome becomes tenuous. On the other hand, follow-up time and tracking difficulty were not associated with out-come, indicating that outcomes were stable in the follow-up window and that difficult-to-reach respondents had outcomes similar to those easier to reach. In this, we are aware of the restricted number of attempts to contact patients in this study. Other studies that made more attempts to contact patients, did find an association between tracking difficulty or follow-up and outcome, but in general, findings are equivocal (Digiusto et al., 2006; Hansten et al., 2000; Stinchfield et al., 1994; Walton et al., 1998).
Given the response rate and differences between responders and non-re-sponders a valid interpretation of the data in our study remains difficult. Some studies with response rates higher than 60% suggest that it is possible to
Taking into account an inflation of 3% each year, €40 in 2004 was €36,61 in 2001. Using the exchange rate of 0.8956 as reported by the European Central Bank for 2001 this was $32,78. Source: http://sdw. ecb.europa.eu/browseSelection.do?dataset=0&freq=a¤cy=usd&node=2018794
