Impact of Agency Receipt of Incentives and Reminders on Engagement and Continuity of Care for Clients With Co-Occurring Disorders

Many adults who report having a substance use disorder also report having a mental health problem in the past year (1). Individuals with co-occurring disorders often have more complex treatment needs (2), tend to have shorter stays in treatment and tend not to complete treatment (3,4), tend to be readmitted within a short period of time (5), and have less favorable outcomes compared with individuals who have a substance use disorder alone (6).

Research indicates that substance use disorder treatment can be improved through the use of performance measures to monitor treatment processes, such as treatment engagement and continuity of care. Meeting these measures is associated with improved outcomes for clients with substance use disorders (7–14). Treatment engagement occurs when ongoing treatment begins a short time after admission to outpatient treatment (15,16). Continuity of care occurs when clients from residential treatment or detoxification services receive at least one follow-up service shortly after discharge (15,17). The concept captured by these measures is that follow-up treatment should be timely and occur soon after the client enters outpatient treatment or leaves detoxification or residential care.

Lessons learned from research on substance use disorders may be used to improve treatment for those with co-occurring disorders. In fact, timely follow-up services (18,19) and continuity of care have been found to improve outcomes among individuals with co-occurring disorders, for example, by lowering mortality and reducing risk of early readmission (5,20).

Financial incentives and electronic reminders, interventions that have been used to improve treatment in general medicine (21–26) and substance use disorder treatment (27,28), may also be promising for individuals with co-occurring disorders. One of the few studies that used financial incentives to improve quality of mental health treatment showed that agency-level incentives increased the percentage of clients who received follow-up care shortly after assessment (29).

The main goal of this study was to examine the effects of treatment agency financial incentives and electronic reminders, referred to as “alerts” in our study, on treatment of clients with co-occurring substance use and mental disorders. We addressed three questions: whether having co-occurring disorders influences clients’ engagement in outpatient treatment or continuity of care after discharge from residential treatment or detoxification, whether agencies’ receipt of incentives or client-specific alerts influence treatment engagement or continuity of care, and whether combined receipt of client-specific alerts and incentives leads to additional improvement in performance beyond that of incentives only or electronic reminders only.

Methods

Agencies providing publicly funded outpatient or residential substance use disorder services to 25 or more clients between May 1, 2012, and April 30, 2013, in Washington State were assigned to four research arms (incentives only, alerts only, incentives and alerts, and a control group) by using a randomized cluster design. Because of their smaller number, detoxification agencies were randomly assigned to only three arms, omitting the incentives and alerts arm. The sampling frame comprised 151 outpatient, 12 detoxification, and 33 residential agencies. We stratified the randomization process to take into account the agencies’ baseline level of engagement in outpatient treatment or continuity of care after residential treatment or detoxification and number of admissions, an indicator of agency size. The study was approved by the Brandeis University and Washington State institutional review boards.

Data Sources

All data were from administrative data sources and included all publicly funded specialty treatment for substance use and mental disorders in the state, but they did not include primary care treatment. Client treatment data were obtained for individuals receiving state-funded substance use disorder treatment during the preintervention period (July 2012–September 2013) and the intervention period (October 2013–December 2015). Data from Washington’s Treatment Activity Report Generation Tool (TARGET) provided information on dates and types of services, information about clients’ demographic characteristics, employment, housing status, substance use history, and current substance use. Because the analytic file consists of client admissions, we use the term “clients” to refer to individual admissions. Mental health service data were merged with substance use disorder treatment data. The mental health file included data from Washington’s state hospital, community hospital, and outpatient databases, and, therefore, the assessments and service records contained in the file were from mental health facilities and not from substance use disorder treatment agencies.

Interventions

Incentives.

Treatment agencies in the incentives arm were eligible to receive financial incentives during each quarter on the basis of earned points. The incentives method was initially developed by researchers at the Brandeis/Harvard NIDA Center (30) and was implemented by the Centers for Medicare and Medicaid Services to reward hospitals for performance on a set of specified measures (31,32). Points could be earned based on quarterly performance rates in one of two ways.

Achievement and improvement points.

Agencies in the incentives arms could earn achievement and improvement points during each quarter on the basis of their quarterly performance relative to a minimum threshold and benchmark. The threshold and benchmark values were determined from the percentiles of agencies providing engagement or continuity of care during the baseline year. The achievement threshold was set at the 50th percentile, a reasonable lower bound for performance to be considered meritorious, and the benchmark level was set at the 90th percentile, a performance level that could reasonably be considered excellent. On the basis of baseline data, the achievement threshold and benchmark levels of continuity of care were set at 29% and 37%, respectively, for detoxification agencies and at 40% and 56%, respectively, for residential agencies. For outpatient agencies, the achievement threshold and benchmark levels of engagement were set at 76% and 90%, respectively.

Achievement points were awarded based on the extent to which an agency’s performance in a quarter exceeded the achievement threshold using equal intervals between the threshold and benchmark, with 1 point awarded for meeting the minimum achievement threshold and 10 points awarded for meeting or exceeding the benchmark level.

Treatment agencies could earn up to 10 improvement points each quarter for raising the rates of engagement or continuity of care above their own baseline rate, using equal intervals between their baseline value and the benchmark. An agency earned an improvement score of 0 if its rate was below its own baseline value (no improvement) and a score of 1 to 9 if its rate was in between (shows some improvement, but not yet up to benchmark).

Agencies were rewarded based on their achievement or improvement points, whichever were higher. For agencies with under 20 clients in a quarter, points were awarded in subsequent quarters when the count reached 20.

Alerts.

Alerts were delivered to treatment agencies in the form of an Excel workbook sent weekly through secure e-mail by Washington’s Behavioral Health Administration. The alerts showed the names of outpatient clients who were admitted in the past 45 days and clients who had been discharged from residential treatment and detoxification in the previous two weeks, the deadline for achieving engagement or continuity of care for each client, and the number of days until the deadline. The information was intended to “alert” agencies to clients who may be at risk of not meeting the engagement (outpatient treatment) or continuity of care (residential treatment and detoxification) criteria.

Variables

Dependent variables.

Treatment engagement was defined as receiving at least one service within 14 days of outpatient treatment admission (treatment initiation) and two or more services within 30 days of initiation (15,16).

Continuity of care was defined as receiving a treatment service within 14 days of being discharged from detoxification or residential treatment (15).

Co-occurring disorders.

A client was considered to have co-occurring substance use and mental disorders if he or she received one or more mental health services, reimbursed by state mental health funding, in the year prior to treatment admission.

Client-level covariates.

Outpatient and residential treatment analyses were controlled for client’s age, gender, race-ethnicity, education, marital status, employment status, past-month use of substances (alcohol, marijuana, cocaine, opioids, methamphetamines, and other drugs), age at first use of any substance, and homelessness status, based on clients’ self-report at admission. Other covariates included criminal justice involvement or referral, indicator variables for whether the client lived in a rural area, and indicator variables for whether clients had received substance use services in the year prior to admission. Detoxification models did not include education, marital status, employment status, and homelessness status, given that detoxification agencies collected less information.

Analyses

Analyses were conducted separately for each level of care by using individual, client-level data. For each level of care, two analyses were performed. First, a bivariate (unadjusted) analysis of the preintervention sample compared engagement and continuity of care rates for clients with co-occurring disorders and clients with a substance use disorder only. Second, a multilevel logistic regression (with clients nested within agencies) examined the effect of the interventions while controlling for client and agency characteristics. The multilevel regressions were performed separately by co-occurring status and used a difference-in-difference specification, which included indicators for the intervention arms, the intervention period, and their interaction. Significant interactions between the intervention arm and the intervention period indicated interventions with a significant effect on the likelihood of engagement or continuity of care. All analyses were conducted by using Stata, version 12.1.

Results

Table 1 shows characteristics of clients with substance use or co-occurring disorders by level of care. Among the 76,044 admitted outpatient clients, 29% received at least one mental health service in the year prior to admission, whereas 71% had a substance use disorder only. Of the 32,797 admitted residential clients, 36% had co-occurring disorders and 64% had a substance use disorder only. Of the 39,006 admitted detoxification clients, 34% had co-occurring disorders compared with 66% with a substance use disorder only.

Table 1 also indicates differences between clients with co-occurring disorders or a substance use disorder only. For example, compared with clients with a substance use disorder only, a higher proportion of clients with co-occurring disorders were women and were older (ages 31–44 and ≥45). This was true in all levels of care. Pairwise comparisons within each level of care showed that clients with co-occurring disorders differed significantly from clients with a substance use disorder on nearly every demographic variable. Only a few of the pairwise comparisons of clients in the same level of care were not significantly different (noted in Table 1). However, the preponderance of significant differences was due to the large sample sizes for each level of care.

Outcome Differences by Co-Occurring Status

We compared unadjusted outcomes for preintervention samples of clients with co-occurring disorders and clients with a substance use disorder only. Among outpatient clients, those with co-occurring disorders demonstrated a significantly lower rate of engagement (N=5,679, 27.3%) than did those who had a substance use disorder only (N=15,126, 72.7%) (p<.001). Among residential clients, the rate of continuity of care was significantly lower among clients with co-occurring disorders (N=1,749, 30.7%) than among clients with a substance use disorder only (N=3,948, 69.3%) (p<.001). Among detoxification clients, the rate of continuity of care was significantly lower among clients with co-occurring disorders (N=1,527, 33.6%) than among clients with a substance use disorder only (N=3,021, 66.4%) (p<.001).

For the same preintervention samples, we ran multilevel logistic regressions to estimate the effects of co-occurring mental disorders by level of care. The analyses were adjusted for client and agency characteristics (data not shown). These models provided similar estimates of the effects of co-occurring disorders on our outcomes, except that detoxification clients with co-occurring disorders tended to have higher rates of continuity of care than those who had a substance use disorder only.

Impact of Interventions on Clients With Co-Occurring Disorders

To examine whether the interventions had different effects on clients with co-occurring disorders, regressions were conducted separately for clients with co-occurring disorders and those who had a substance use disorder only. Table 2 shows the impact of interventions on each group by level of care. Incentives or alerts had no significant effect on engagement in outpatient treatment or on continuity of care after discharge from detoxification for any group of clients, regardless of whether they had a co-occurring disorder.

Analysis of continuity after residential care, however, showed a significant positive interaction between the alerts-only research arm and the intervention time period (estimate=.32; 95% confidence interval [CI]=.05 to .60, p<.05). This result implied that the difference in likelihood between receiving continuity of care during the intervention compared with before the intervention was significantly larger among the alerts-only group than among the comparison group. Because the regression model also showed a significant negative main effect for the intervention period (estimate=–.19; CI=–.35 to –.02, p<.05), this significant positive interaction indicated that the alerts counteracted what was otherwise a decreasing likelihood of continuity of care over time. That is to say, providing residential agencies with alerts appeared to counteract an overall decline in continuity of care during the intervention period. Specifically, the alerts informed residential agencies regarding which clients were approaching 14 days since discharge without having received a follow-up treatment service and, therefore, were at risk of not meeting the continuity-of-care measure. However, the alerts were helpful in achieving continuity of care only for residential clients with co-occurring disorders. No similar impact was found for residential clients who had a substance use disorder only, even though they experienced an overall decline in continuity of care during the intervention period that was almost as large as that of the clients with co-occurring disorders.

Discussion

Clients with co-occurring substance use and mental disorders were found to be less likely than clients with a substance use disorder only to engage in outpatient treatment or to experience continuity of care after discharge from residential treatment. Detoxification clients who had co-occurring disorders were more likely than those with a substance use disorder only to have continuity of care. Individuals with mental health problems tend to have multiple detoxifications and utilize more services (33,34), which results in higher rates of continuity of care.

Our interventions generally had no impact on outpatient treatment engagement or on continuity of care after detoxification and residential discharge. However, an impact was found for the use of alerts in one group of clients—those with co-occurring disorders who were treated at residential agencies in the alerts-only arm. Because residential treatment is a more intense level of care (35), it tends to admit fewer clients than other levels of care, and staff may be better able to attend to client information, such as the alerts.

This finding somewhat duplicates previous research that found that use of electronic reminders for a recommended brief intervention was more likely to occur with clients who had more severe unhealthy alcohol use than with those who had mild or moderate alcohol problems or clients with mental health conditions (36,37). It appears that treatment staff focus more on clients with more complex needs to ensure that they receive needed follow-up services.

In addition, previous research found that certain types of practitioners may be more inclined than others to use the information from alerts. For example, nurse practitioners tend to use electronic reminders more than other staff (36). The tendency of residential treatment agencies to have more nurses on staff compared with outpatient agencies (38) may explain why implementation of alerts demonstrated an interaction effect for the former level of care.

Research has been somewhat mixed regarding the effectiveness of alerts and incentives in the areas of general medical health and behavioral health (23,27,28). Our results do little to clarify the influence of these interventions, given that our regression models found only limited situations in which alerts or incentives had a significant impact on treatment engagement and continuity of care. These findings suggest that generally no single intervention alone, such as alerts or incentives, is sufficient to bring about desired improvements. Additional interventions may often be needed to improve agency performance across the board. It has been found that information delivered in electronic reminders is used more readily by clinicians when the reminders are aligned with performance measures supported by the leadership (36). In Washington, treatment retention, not continuity of care, is the measure supported by state leadership. Although treatment agencies agree that continuity of care is important, the fact that Washington focuses on a retention measure may well explain why our study results were so modest.

There were several limitations of the study. First, the data were from one state, and, therefore, the ability to generalize to other states is limited. Second, although administrative data are a rich resource that include information for all clients who receive treatment in the state, the TARGET database captures only publicly treated clients. To the extent that agencies might treat state-funded clients differently than other clients, the results are again limited with regard to generalizability. Third, our definition of co-occurring substance use and mental disorders is based on clients’ service claims in the prior year. We used receipt of mental health services in the year prior to admission as a marker for having a mental health problem because the diagnosis data had a very high percentage of missing values. Because of the diagnosis data quality, we were not able to use that data to distinguish the various types and level of severity of mental health problems among clients who had a co-occurring disorder. Although everyone who received a mental health service in the prior year may not have had a mental health problem during the study period, we found that the great majority of clients who received a mental health service in the prior year continued to receive additional mental health services during the study period. Data completeness, however, may have resulted in an underestimation of the number of clients who had co-occurring disorders.

As a fourth limitation, our study was conducted while Washington State was transitioning to a system of behavioral health organizations. As a result, the agencies’ efforts were focused to a large extent on adapting to the new system, and they had reduced time and resources to focus on improving engagement and continuity of care. Finally, the use of electronic reminders could result in performance improvement when the desired outcome, such as conducting screening, is under the control of the agency. However, reminders may be less helpful when the outcome is not under the control of the agency, such as getting clients into follow-up care when the system is at capacity.

Conclusions

The implementation of electronic reminders, or alerts, was associated with a greater likelihood of continuity of care after residential treatment for clients with co-occurring disorders but not for those with a substance use disorder only. Financial incentives had no influence on continuity of care after residential treatment. Neither incentives nor alerts had any impact on engagement among outpatients or continuity care among detoxification clients. Limitations notwithstanding, to our knowledge, there has not been a randomized controlled study that examined the influence of incentives and alerts, together and separately, on treatment agency performance measures among clients with co-occurring disorders. The research on the impact of incentives on performance has been mixed and suggests that multiple interventions may be needed to bring about improvements and that the impact of these interventions may vary with different populations.