UvA-DARE (Digital Academic Repository)
Routine outcome monitoring & learning organizations in substance abuse
treatment
Oudejans, S.C.C.
Publication date
2009
Link to publication
Citation for published version (APA):
Oudejans, S. C. C. (2009). Routine outcome monitoring & learning organizations in substance
abuse treatment.
General rights
It is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s) and/or copyright holder(s), other than for strictly personal, individual use, unless the work is under an open content license (like Creative Commons).
Disclaimer/Complaints regulations
If you believe that digital publication of certain material infringes any of your rights or (privacy) interests, please let the Library know, stating your reasons. In case of a legitimate complaint, the Library will make the material inaccessible and/or remove it from the website. Please Ask the Library: https://uba.uva.nl/en/contact, or a letter to: Library of the University of Amsterdam, Secretariat, Singel 425, 1012 WP Amsterdam, The Netherlands. You will be contacted as soon as possible.
—
—
ChAPTER 2
—
fEASIBILITY
& vALIdITY Of
LOW-BUdGET
TELEPhONIC
fOLLOW-UP
INTERvIEWS IN
ROUTINE OUTCOME
MONITORING (ROM)
Of SUBSTANCE
ABUSE TREATMENT
ACCEPTEd fOR PUBLICATION / AddICTION
fE A S IB IL IT Y & v A LId IT Y OUdEjANS, S.C.C. SChIPPERS, G.M. MERkx, M.j.M. SChRAMAdE, M.h. kOETER, M.W.j. vAN dEN BRINk, W.
—
Abstract
Aims: Routine outcome monitoring (ROM) is receiving growing attention.
However, follow-up interviews are time consuming and costly. This study ex-amines the feasibility and validity of low-budget telephonic follow-up interviews for ROM in a substance abuse treatment center (SATC).
Design: Observational study using data collected for routine outcome monitoring. Setting: The study was performed in a SATC in an urban area in the Netherlands. Participants: Feasibility and validity were assessed on data of 2,325 patients. Measurements: Data on pre-treatment socio-demographic and clinical
charac-teristics was collected using electronic patient records (EPRs) and the European version of the Addiction Severity Index (EuropASI). Data on intensity of treatment was also collected through the EPRs. Telephonic follow-up interviews were conducted between 9 to 10 months after intake.
Findings: A 53% follow-up rate was achieved, 32% of the patients could not be
contacted, 3% explicitly refused, and in 9% other reasons accounted for non-participation. About 50% of the interviews took place in the intended time frame. Costs were €40 ($57) per completed interview. There were indications of selection bias because patients with cocaine as their primary problem and patients with polysubstance abuse were under-represented in the follow-up sample and because the presence of these disorders is associated with negative treatment outcome.
Conclusions: Implementing telephonic low-budget follow-up interviews for
ROM is feasible, but selection bias threatens internal validity of data, limiting generalization to the total treatment population. Increased efforts to track patients for follow-up may improve generalization.
—
—
Introduction
Substance abuse treatment centers (SATCs) currently face two challenges: (1) demonstrating programmatic transparency and accountability regarding treat-ment outcomes and (2) improving practices based on assesstreat-ment and feedback of these outcomes. In the Netherlands, SATCs are supported in this endeavor in a nationwide quality-enhancing reform program targeting an evidence-based, transparent and accountable treatment system (Schippers et al., 2002). As a result, most SATCs recently redesigned their treatment programs into evidence-based treatment systems and most SATCs are currently in the pro-cess of implementing routine outcome monitoring (ROM) as an instrument for quality improvement. Post-treatment follow-up interviews, designated to monitor and evaluate outcomes after treatment has ended are an important part of ROM (Evans & Hser, 2004; Gossop, Marsden, & Stewart, 2000; Harrison & Asche, 2001; Meijerink, 2003; Moos, Humphreys, Ouimette, & Finney, 1999; Project Match Research Group, 1997). However, follow-up interviews are costly and time consuming. Investments of up to 20 interviewer hours per patient are reported in follow-up studies like the Drug Abuse Treatment Outcome Studies (DATOS), the National Treatment Improvement Evaluation Study (NTIES), the Services Research Outcome Study (SROS) and California Drug and Alcohol Treatment Assessment (CALDATA) reaching response rates between 48% and 70% (Flynn, Simpson, Anglin, & Hubbard, 2001; Gerstein & Johnson, 2000; Rosenheck & Seibyl, 2005). Although some ROM projects report response rates above 80% of included patients, inclusion procedures were not transparent, resulting in outcome samples with questionable repre-sentativeness (Evans & Hser, 2004; Moos et al., 1999; Rohrer, Vaughan, Cadoret, & Zwick, 1999). Moreover, Tiet et al. (Tiet et al., 2006) report costs up to $186 per patient to obtain a 67% follow-up rate with a mailed questionnaire in the Veterans Affair (VA) Outcome Monitoring Project (further referred to as OMP). These are personnel costs involving the actual up efforts only. Telephonic follow-up interviews carried out by an independent survey vendor in the Minnesota Outcome Monitoring System (OMS) were much cheaper at $30 per completed interview reaching a follow-up rate of almost 65% (Harrison & Asche, 2001).
Considering this, several SATCs in the Netherlands have implemented a system to conduct telephonic follow-up interviews for ROM, in the hope of collecting valid data and keeping costs and interviewer hours under control. In collaboration with the Amsterdam Institute for Addiction Research (AIAR), some of these SATCs installed an independent call center, staffed by inter-viewers conducting telephonic follow-up interviews.
—
monitoring are mainly caused by attrition, because many patients are difficult to reach. Non-response due to refusal, as is often the case in general popula-tion surveys, is usually a less important cause, although this varies between 2% in DATOS and 29% in CALDATA (Gerstein & Johnson, 2000; Grella, Hser, Joshi, & Anglin, 1999). Low response-rates are problematic because they may result in selection bias that may affect internal validity, i.e. patients with poor outcome are more difficult to find for follow-up or more likely to refuse. Several studies show this threat to be real (Lahaut, Jansen, van de Mheen, & Garretsen, 2002; Marcus & Schutz, 2005; Mond, Rodgers,
Hay, Owen, & Beumont, 2004; Rohrer et al., 1999; Rosenheck & Seibyl, 2005; Scott, 2004). A number of studies reported associations between tracking difficulty or follow-up time and negative outcome, but these finding are not consistent (Digiusto, Panjari, Gibson, & Rea, 2006; Hansten, Downey, Rosengren, & Donovan, 2000; Stinchfield, Niforopulos, & Feder, 1994; Walton, Ramanathan, & Reischl, 1998). As a consequence, accountability figures are potentially overestimating the positive outcome of a treatment program.
This study evaluates the feasibility of ROM with a call center and establishes the internal validity of the collected data with a central focus on selection bias.
Methods
Setting
The call center was operational five days a week, calling patients between 4pm and 8pm. The five call center interviewers were specially-trained psychology students. A psychologist supervised the interviewers and a research assistant managed the call center. Follow-up time was set at 9 months after intake. Most treatment programs in participating SATCs last between 3 and 6 months. Measuring outcome at 9 months would reveal results from patients 3 to 6 months after treatment completion.
In this study we used data of one participating SATC covering an urban area of approximately 1 million inhabitants. Treatment in this SATC is organized into five levels of care: (1) brief outpatient treatment, (2) standard outpatient treatment, (3) day treatment, (4) inpatient treatment, and (5) harm-reduction treatments. Intake counselors allocate patients to an intensity of treatment following a structured matching and allocation protocol (Merkx et al., 2007; Schippers et al., 2002).
Patients
Patients were eligible for follow-up if a) their last treatment intake assessment at the SATC took place 9 months earlier and b) they were no longer in treatment.
—
In this study we focus on a cohort of 2,572 patients who had their intake as-sessment in the year 2004. Of these, 233 were still in treatment 9 months after intake, and 14 had a subsequent intake in the 3 months following their initial intake assessment date. The follow-up date of these 14 patients was “pushed” into the subsequent cohort, just as the follow-up dates of some patients of the former cohort were pushed into the cohort of this study. This resulted in 2,325 patients eligible for analysis.
Procedure and assessments
Data on pre-treatment socio-demographic and clinical characteristics were collected using the electronic patient records (EPR) where administrators and intake counselors entered information following admission to the SATC. Administrators collected socio-demographic characteristics during the first admission session. Next, intake counselors assessed clinical characteristics using the European version of the Addiction Severity Index (EuropASI) 5th edition (Kokkevi & Hartgers, 1995). The EuropASI is a semi-structured inter-view providing information on substance use and substance-related problems in different problem areas: medical, employment/education, alcohol, drugs, legal, family/social support, psychiatry and gambling. Items from each area are used to generate interviewer severity ratings, which provide an assessment of the overall problem severity in that area. Data on intensity of treatment were also collected from the EPR.
The call center collected follow-up data with a structured interview, in-cluding a selection of 20 questions about the 30 days preceding the follow-up interview from the seven areas of the EuropASI. Lists of patients eligible for follow-up were printed from the EPR on a monthly basis, displaying contact details about the patient and his or her collaterals. Collaterals were contacted when repeated direct contact attempts had failed. A maximum of four attempts to contact a patient were made, the attempt with the collateral excluded.
Interviewers confirmed name and date of birth before starting the inter-view. To assess whether the follow-up interview was valid, interviews ended with two questions. The first is whether the interviewer had any indication that the interview was contaminated by an incorrect reproduction of reality by the patient. The second question is whether the interviewer had any indica-tion of the patient having difficulties understanding the quesindica-tions. A “yes” on either of these questions deemed the interview invalid.
Feasibility in this study is defined as: a) a response rate that is consistent with response rates known from DATOS, NTIES, SROS, CALDATA and the Minnesota OMS, i.e. above 48%; b) low refusal rates, i.e. below 10%; c) the
—
majority of the interviews are conducted within the intended time frame, i.e. 9 to 10 months after intake; d) time investment per interview – including tracking time – is limited to one hour per interview; and finally e) costs per interview are similar or lower than the costs reported by the Minnesota OMS, i.e. $30.
Internal validity of collected data is defined as: a) no association between tracking difficulty (i.e. time to follow-up and number of contact attempts) and outcome; b) no differences between respondents and non-respondents on socio-demographic, pre-treatment clinical characteristics and process variables that have an association with treatment outcome.
Data-analysis
Response rate was defined as the number of patients with a valid follow-up interview (“respondents”) divided by the number of patients eligible for follow-up. Reasons for non-response were collected from the records of the call center. Time between intake and follow-up interview was defined as the number of days between date of assessment at intake with the EuropASI and date of the follow-up interview. A follow-up time between 9 and 10 months (275 and 305 days) after intake was considered satisfactory. Time investment of the inter-views was estimated by dividing the total time investments by the interviewers by the number of interviews being held. Costs were defined as total costs of the call center divided by the total number of valid interviews. Not included were costs for analyzing, interpreting, and disseminating findings from the data. Cost calculations were based on figures from the call center year report of 2004.
Tracking difficulty was operationalized in follow-up time and number of contact attempts to reach a respondent. The relation between follow-up time and treatment outcome was defined as the difference in mean follow-up time between respondents with and without positive treatment outcome. Association between number of contact attempts and treatment outcome was assessed by calculating the percentage of respondents with positive treatment outcome for each group of respondents that was reached after one, two, three, four or more attempts. Up to four contact attempts were recorded. In reality more than four attempts were made when interviewers had the opportunity, implying that a recorded amount of four attempts means that at least four attempts were made.
Treatment outcome was based on information about primary and second-ary substance use reported by the patient during intake assessment and items in the alcohol, drugs, and gambling areas from the follow-up interview. Positive treatment outcome was defined as abstinence or no excessive use of primary and secondary substance in the 30 days preceding the follow-up interview.
—
No excessive use of alcohol was defined as a maximum of 21 days of alcohol use (14 days for women) and no days with 5 or more glasses. The first criterion requires alcohol-free days – preferably 2 days a week – as is advised by Dutch informational institutes and their public websites6 to prevent habituation. The
second criterion is modeled after the criterion for binge drinking from the ASI manual, and is in line with the advice of the Health Council of the Netherlands to drink moderately on drinking days (Health Council of the Netherlands, 2006; Kokkevi & Hartgers, 1995). No excessive use of other substances and gambling was defined as a maximum of 6 days of use or gambling. Negative treatment outcome was defined as excessive use of primary and/or secondary substance, e.g. more than 21 days of alcohol and/or days with 5 or more glasses; or more than 6 days of gambling or use of other substances.
To test for differences between respondents and non-respondents in terms of socio-demographic data and pre-treatment clinical characteristics and to test whether these characteristics were associated with treatment outcome, we used a two step procedure. First, we applied multiple logistic regression analysis to select the strongest independent predictors for becoming a respondent. Next, we tested – again with multiple logistic regression analysis – which characteristics served as independent predictors for positive treatment outcome. In both cases a backward stepwise elimination procedure was employed. Because the allocation and matching guideline used clinical variables to allocate patients to intensity of treatment, resulting in a strong association between these variables and intensity of treatment, the latter was not included in the regression analyses.
Results
Subjects
Table 1 presents the pre-treatment socio-demographic, clinical characteristics, and process variables of the 2,325 patients, showing a mixed inpatient and out-patient population, with the largest group primarily seeking help for alcohol problems (49%). A substantial portion of the patients reported polysubstance abuse (36%). These findings are corroborated by the ASI interviewer severity ratings (ISRs), indicating that almost half of the patients were in need of treat ment for alcohol and the remaining for drugs. Additional problem areas were psychiatry, family and social support, and employment and education, with large numbers of patients in need of treatment as indicated by ISRs above 3.
http://www.alcoholdebaas.nl/Stoppen,-minderen-/Hoeveel-is-veel-.aspx This website is maintained by Tactive, a commercial branch of the Tactus, regional public SATC in the Netherlands
—
Table 1:
Process variables, pre-treatment socio-demographic and clinical characteristics of patients eligible for follow-up
Total sample (n = 2,325) Respondents (n = 1,237)* Positive treatment outcome# (n = 499)** % of total n/Mean (sd) Treatment intensity χ2 (6) = 92.2; p = 0.00 Brief Outpatient 13 18 18 Standard Outpatient 23 25 25 Day Treatment 5 6 5 Inpatient 13 11 10 Harm reduction 14 11 10 No intensity/unknown 4 5 5
No treatment after intake 29 26 28
t(2290) = 6.35; p = 0.00 Age 41.7 (11.2) 43.0 (11.7) 42.7 (12.1) Sex χ2 (1) = 9.02; p = 0.00 Male 72 69 70 Ethnic Origin χ2 (1) = 65.8; p = 0.00 χ2(1) = 16.3; p = 0.00 Dutch 77 84 79 Civil Status χ2 (1) = 14.3; p = 0.00 Living alone/single 84 81 79 Educational Level χ2 (2) = 32.4; p = 0.00 No education/low 31 26 27 Middle 40 41 41 High 29 33 32
Primary substance use χ2
(6) = 72.1; p = 0.00 χ2(6) = 84.4; p = 0.00 Alcohol 49 53 43 Heroin 7 6 6 Cocaine 19 14 19 Cannabis 10 11 12 Gambling 6 5 11 Nicotine 6 8 5 Other 4 4 4 Polysubstance abuse χ2 (1) = 18.2; p = 0.00 χ2(1) = 27.8; p = 0.00 Yes 36 32 24
ASI severity scores > 3
Medical 20 20 18 Employment/education 31 25@ 25 Alcohol 49 52$ 44! Drugs 39 34% 35 Legal 8 5& 5 Family/social support 36 31^ 30 Psychiatry 40 39 36 Gambling 5 5 10
# positive treatment outcome – abstinence or no excessive use of primary and secondary substance in the 30 days preceding the follow-up interview. No excessive use of alcohol: maximum 21 days (14 days for women) of alcohol use and no days with 5 or more glasses.
No excessive use of other substances and gambling: maximum 6 days of use or gambling. * test statistics are given for respondents vs. non-respondents
** test statistics are given for positive vs. negative treatment outcome @ χ2
—
Feasibility
Table 2 shows that in 1,237 of the 2,325 eligible patients (53%) a valid follow-up interview could be obtained.
About 31% of patients were interviewed less than 275 days (9 months) after in-take, and 18% were interviewed more than 305 days (10 months) after intake. Subsequently, 50% of patients were interviewed between 9 and 10 months after intake. Mean time between intake and follow up was 9.5 months (sd = 0.86).
Table 3 shows that the main reason for non-response was lack of a valid phone number or no contact at all (35%). In only 3% of cases, the patient explicitly refused the interview.
Table 2:
Total and yearly response rate, proportion interviews in time*, mean follow-up time and number of attempts per interview
Total N eligible for follow-up 2,325 Valid interviews
N 1,237
% 53
Time to follow-up interview
< 9 months 31
9 – 10 months 50
> 10 months 18
Follow-up time in months
Mean 9.5
SD 0.86
Number of contact attempts
1 35 [19]#
2 25 [32]
3 13 [39]
4 or more 27 [53]
* in time: between 275 and 305 days (9 and 10 months) after intake; # percentages in brackets express cumulative response rate
Table 3:
Results for response
Reasons for non-response n % Invalid phone number/no contact 817 35.1 Refusal client (angry, frustrated, no time) 77 3.3
Other* 194 8.3
Valid interview 1,237 53.2 Total eligible 2,325 100
* including 29 interviews scored as “invalid”, other reasons include language problems; patients that turned out to be deceased; deaf or hearing impaired patients, and intoxicated patients.
—
On average a completed interview took 46 minutes, including contacting and interviewing. Not included was time for supervisors and management. The total costs for one completed interview was €40 ($57)7, including personnel costs
for interviewers, manager, supervision, and overhead costs, such as use of office space, phone contracts, computers and software licenses (estimated at 10% of the total costs). The total completed interview costs do not include analyzing, interpreting and disseminating the findings from the follow-up data.
Validity
In total, 42.2% of the 1,237 respondents reported a positive treatment outcome. Mean follow-up time was similar for respondents with positive treatment out-come and respondents with a negative treatment outout-come: 9.4 months (sd = 0.90) versus 9.5 months (sd = 0.85) respectively (t1176 = 1.02; p = 0.31). Treatment
out-come was not significantly associated with tracking difficulty: 46% of respon-dents reached after one attempt had a positive outcome and 38% reached after four or more attempts had a positive treatment outcome (χ2
(3) = 5.8; p = 0.12).
Table 1 shows percentages on process, socio-demographic, and clinical variables within the group of respondents (n = 1,237). Patients who received more intensive treatment were under-represented, just as those who received no treatment after intake. In addition, under-representation was found for younger, male, and non-Dutch patients, patients who did not live alone, patients with low education, patients with cocaine as their primary problem drug and patients who reported polysubstance abuse. In addition, patients who were more in need of interventions (as indicated by an ISR higher than 3 on em-ployment/education, legal problems, drugs, and family/social support) were under-represented. Patients who just needed help for their alcohol problem were over-represented in the group of respondents. Table 1 also shows percentages on process, socio-demographic and clinical variables within the group of patients with positive treatment outcome (n = 499). Dutch patients were under-represented, as well as patients with alcohol as their primary substance, patients who reported polysubstance abuse and patients with an alcohol ISR higher than 3. Patients with cocaine as their primary substance were over-represented amongst those with positive treatment outcome.
Although statistically significant, differences on baseline characteristics between respondents and non-respondents and between those with positive
Using a euro-dollar rate of 1.4369, level of September 2008 as reported by the European Central Bank at http://sdw.ecb.europa.eu/browseSelection.do?dataset=0&freq=m¤cy=usd&node =2018794
—
and negative treatment were not large. In terms of Cohens’ h, effect sizes for ethnic origin, primary substance use and polysubstance abuse (vari-ables that showed both significant differences in the 3rd and 4th column
of Table 1) are small to moderate (Cohen, 1988). The largest effect size between respondents and non-respondents is found for ethnic origin (h = 0.36), and the largest one between patients with and without positive treat-ment outcome for patients who have gambling as their primary problem (h = 0.42).
Table 4 shows the results of the multiple logistic regression model with respondent status as the dependent variable. Compared to patients primarily seeking help for alcohol problems, patients primarily seeking help for cocaine were less likely to become a respondent. Older patients were more likely to be-come respondents. Non-Dutch patients were up to twice less likely to bebe-come respondents than Dutch patients. Patients abusing only one substance were 21% less likely to become respondents than patients without polysubstance abuse. Finally, chances to become a respondent decreased with increasing ISRs for employment/education, legal and family/social support.
Table 4:
Multiple logistic prediction for becoming a respondent#, n = 2,325
OR P value 95% CI
Age 1.01 0.02 1.00, 1.03
Ethnic Origin
Non-Dutch vs. Dutch 0.54 0.00 0.41, 0.72 Clinical variables
Primary substance use
Alcoholref 1.00 (0.09) Heroin 0.84 0.48 0.52, 1.36 Cocaine 0.67 0.02 0.48, 0.94 Cannabis 1.23 0.33 0.81, 1.85 Gambling 0.82 0.43 0.49, 1.35 Nicotine 1.11 0.71 0.64, 1.90 Other 1.40 0.30 0.75, 2.62 Polysubstance abuse Yes vs. No 0.79 0.06 0.62, 1.01 ASI severity scores
Employment/education 0.94 0.04 0.88, 1.00
Legal 0.84 0.00 0.77, 0.92
Family/social support 0.94 0.05 0.88, 1.00
Constant 1.69 0.08
ref = reference category
(for primary substance use, p-value between parentheses is the p-value for the omnibus test for all levels of the variable)
# Backward Stepwise elimination (likelihood ratio; p in = 0.05, p out = 0.10) method from SPSS 13.0 for MacOSx; variables not in the equation: sex, age, ethnic origin, educational level, severity scores medical, alcohol, drugs, family/ social support, psychiatry, and gambling.
—
Table 5 presents the predictors in the multiple logistic regression model with positive treatment outcome as dependent variable. Patients living alone, seeking primary help for alcohol or nicotine problems, and those reporting polysubstance abuse were less likely to report positive treatment outcome. Cocaine abusers and gamblers were most likely and smokers were least likely to have a positive treatment outcome.
Common predictors for becoming a respondent and reporting positive treat-ment outcome in this sample are substance of abuse and polysubstance abuse. Patients primarily seeking help for cocaine problems are under-represented and more likely to report positive treatment outcome, and patients abusing more than one substance are under-represented and are less likely to have a positive treatment outcome.
Conclusions and discussion
Results of this study show that it is feasible to implement a telephonic low-budget follow-up system for ROM with a call center, but that this method does not guarantee a representative follow-up sample of the treated population, thereby limiting a valid interpretation of the collected data. Patient tracking problems turned out to be the major reason for non-response.
Table 5:
Multiple logistic prediction# for positive treatment outcome@; n = 1,237
OR P value 95% CI Civil Status
Living alone vs. Not living alone 0.52 0.00 0.35, 0.77 Primary substance use
Alcoholref 1.00 (0.00) Heroin 1.66 0.17 0.83, 3.32 Cocaine 3.69 0.00 2.27, 6.00 Cannabis 2.00 0.36 1.24, 3.24 Gambling 8.95 0.00 3.98, 20.16 Nicotine 0.42 0.00 0.22, 0.80 Other 1.37 0.99 0.66, 2.84 Polysubstance abuse Yes vs. No 0.38 0.00 0.26, 0.54 Constant 1.14 0.47
ref = Reference category
(for primary substance use, p-value between parentheses is the p-value for the omnibus test for all levels of the variable)
# Backward Stepwise elimination (likelihood ratio; p in = 0.05, p out = 0.10) method from SPSS 13.0 for MacOSx; variables not in the equation: sex, age, ethnic origin, educational level, severity scores medical, employment/education, alcohol, drugs, legal, family/social support, psychiatry, and gambling
@ Positive treatment outcome: abstinence or no excessive use of primary and secondary substance in the thirty days preceding the follow-up interview. No excessive use of alcohol: a maximum of 21 days of alcohol use (14 days for women) and no days with 5 or more glasses. No excessive use of other substances and gambling: a maximum of 6 days of use or gambling. Negative treatment outcome was defined as excessive use of primary and/or secondary substance.
—
Feasibility results were fairly positive. The 53% response is modest, but similar to the 48% reported for DATOS. Higher response rates are reported (DATOS: 70%), but this was after exclusion of patients deemed inaccessible by virtue of incarceration or hospitalization (Flynn et al., 2001). In the current study, these patients were not excluded and therefore the response rate is com-parable with the lower rate of 48% reported by Gerstein (Gerstein & Johnson, 2000). Refusal rates were low, although refusal rate might increase when more patients are contacted. Non-respondents to surveys tend to be less agreeable and less open to experience than respondents (Marcus & Schutz, 2005). These characteristics might be associated with telephonic traceability (e.g. making less effort to be contacted, or inability to maintain a contract with a provider), and therefore it is conceivable that the chances of refusal are bigger amongst non-respondents. Time investment was less than one hour. The cost of €40 ($57) is far below the $186 reported by the OMP (Tiet et al., 2006), but almost double the $30 reported by the Minnesota OMS conducted by an independent survey vendor. However, €40 in 2004 is similar to almost $33 in 20018 (the
year of Harrison’s Minnesota OMS paper), taking into account inflation and currency rates. In conclusion, response rates are moderate but similar to other studies, and costs are low and comparable with costs of interviews conducted by independent survey vendors.
It turned out to be difficult to interview patients at the intended follow-up time. Only 50% of the interviews were held in the 9 to 10 months time frame. This needs to be improved, because the presence of excessive follow-up times can induce bias, comparisons between patients become difficult and the relationship between treatment and outcome becomes tenuous. On the other hand, follow-up time and tracking difficulty were not associated with out-come, indicating that outcomes were stable in the follow-up window and that difficult-to-reach respondents had outcomes similar to those easier to reach. In this, we are aware of the restricted number of attempts to contact patients in this study. Other studies that made more attempts to contact patients, did find an association between tracking difficulty or follow-up and outcome, but in general, findings are equivocal (Digiusto et al., 2006; Hansten et al., 2000; Stinchfield et al., 1994; Walton et al., 1998).
Given the response rate and differences between responders and non-re-sponders a valid interpretation of the data in our study remains difficult. Some studies with response rates higher than 60% suggest that it is possible to
Taking into account an inflation of 3% each year, €40 in 2004 was €36,61 in 2001. Using the exchange rate of 0.8956 as reported by the European Central Bank for 2001 this was $32,78. Source: http://sdw. ecb.europa.eu/browseSelection.do?dataset=0&freq=a¤cy=usd&node=2018794
—
generalize their findings to the total treated population (Digiusto et al., 2006; Hansten et al., 2000). We have reasons to suspect differential non-response in our sample so that non-respondents and respondents may differ in treat-ment success. Patients with features probably related to negative treattreat-ment outcomes (e.g. non-Dutch, using cocaine or opiates) were under-represented and unexpectedly showed good outcome – although effect sizes were not large. Multiple regression showed selection bias to be most prominent for patients with primary cocaine problems and patients with polysubstance abuse, resulting in an over-estimation of positive outcomes. It is highly likely that patients with good outcomes are more likely to participate in the follow-up assess-ment then those with negative outcomes. Therefore, we should be very careful to generalize outcome results from the follow-up sample to the total treated population.
Recently, Hansten (2000) presented two correction procedures for this problem (Hansten et al., 2000). The first is imputation: a substitution method that replaces missing outcome data with data from respondents that resemble missing cases on important characteristics. This method is also advocated by Digiusto at al. in the study of the outcome of heroine dependence treatment (Digiusto et al., 2006). The other method consists of weighing outcomes based on characteristics that are associated with follow-up and outcome. These methods could be helpful in ROM projects. However, both strategies assume that for each stratum the respondents pertaining to that stratum are a random sample of all patients in that stratum, i.e. no selection bias, which we suspect is not the case in our study. Non-respondents do possess certain characteristics that do have an association with the outcome and it is highly likely that the reason for attrition is related to the outcome being measured, as it is conceivable that patients who have negative treatment outcome are not able to answer the phone for several reasons, e.g. being intoxicated or discontinuation of the contract being a few of them. There are imputation procedures available for the situation where cases are “missing not at random” (MNAR), as is the case in this study. MNAR imputation procedures can be employed as long as the “cause” for at-trition is only moderately associated with the outcome, not when the cause is the response (Schafer & Graham, 2002). It is believed that the first situation “[...] is the exception and the latter is the rule” (Schafer & Graham, 2002), so available procedures like “selection models” and “pattern-mixture models” can be explored and possibly applied to follow-up data for ROM.
Patient tracking problems were the major reason for non-response. Therefore, keeping contact details of patients up to date might raise response rates substantially. An alternative strategy is drawing a random sample of the treated population and spending all available resources in order to reach better
—
follow-up rates. Scott (Scott, 2004) presents the Engagement, Verification, Maintenance and Confirmation (EVMC) protocol, which could raise response rates up to 90%. The four stages of the EVMC protocol are: education and motivation for participation, securing consent, verifying and maintaining con-tact details by mailings and phone concon-tacts that refer to the scheduled follow-up interview, and finally confirming the final contact details a few weeks before the actual follow-up date. However, these repeated contacts introduce another threat to validity. They might function as an intervention in itself and can result in altered reporting of substance use (Hansten et al., 2000; Scott, 2004).
The follow-up time of 9 months limited the study population to patients in shorter, mostly cure-oriented treatments, although in our sample patients from harm reduction programs were present. Including patients from longer, care-oriented programs like methadone maintenance would yield similar but pronounced selection bias in assessing outcome with telephonic interviews, since the lifestyle of these patients is expected to be more chaotic and unstable than the population in this study. Including these patients would also make other or additional measures conceivable, like quality of life.
The results of this study rely on the allocation of patients to one of the two outcome categories. It is possible that the results of this study would have been different if other criteria to assign patients to outcome categories had been used, or if continuous measures like increase or decrease of the number of days using substances were used. For the classification of alcohol use, we adhered to an external criterion like the one provided by the Health Council of the Netherlands (Health Council of the Netherlands, 2006), but in the case of other substances the decision to assign patients to an outcome category was more arbitrary and other classifications would have been plausible as well. Our criterion for cannabis use was quite strict in comparison with a study of Lozano et al., where use up to 4 days a week in the last 3 months was considered as “moderate use” (Lozano, Stephens, & Roffman, 2006). Our criterion for cocaine, on the other hand, was quite liberal in comparison with the study of Crits-Christoph et al., who distinguished only between abstinence and no ab-stinence in the recall period of 30 days (Crits-Christoph et al., 1999). In future re-search, it is desirable to optimize outcome measurement, with measures like drinks per drinking day or amount of substance use per day, information that is not recorded with the ASI. The maximum of 4 glasses per drinking day for moderate use, as we used in this study, is quite liberal in comparison with the one glass (for women) and two glasses (for men) of alcohol a day as advised by the Health Council of the Netherlands (Health Council of the Netherlands, 2006). A newly developed instrument in the Netherlands and Germany, aim-ing at a more reliable and valid assessment for patients with substance use
disorders (Koeter, Broekman, Schippers, & van den Brink, 2003; Schippers, Broekman, & Buchholz, 2007), contains more detailed items. The first imple-mentation phase of this instrument has been accomplished, and availability of research data is expected in the near future. A follow-up version is under construction and in that the process of synchronization with other recently developed instruments for outcome monitoring, like the Treatment Outcomes Profile (TOP) is important (Marsden et al., 2008).
Next, our assessment of the internal validity was restricted to selection bias. Other aspects of internal validity like interviewer variability, protocol integrity, cognitive problems of respondents due to substance use or language barriers, over- or under-reporting of substance abuse and inter-correlations between key variables were not studied. These steps should be the focus of subsequent study projects.
Acknowledgements
This study was funded by the Netherlands Organization for Health Research and Development (ZON-MW grant no. 310000050). The Jellinek (now the Jellinek division of Arkin) made it possible for us to conduct the study. We thank Hans Kronemeijer for his help with data management and Gijs Visser for managing the call center.
