Volume 44 Number 6, 2007
Pages 867 — 878
Dissemination of supported employment in Department of Veterans Affairs
Sandra G . Resnick, PhD;* Robert Rosenheck, MDVeterans Integrated Service Network 1, Mental Illness Research, Education, and Clinical Center, West Haven, CT; Department of Psychiatry, Yale University School of Medicine, New Haven, CT
Abstract — The dissemination of complex innovative practices is one of the major challenges of mental health service organizations. Although substantial progress has been made in the development of evidence-based practices for people with severe mental illness, development of approaches for the large-scale dissemination of such practices has lagged. In 2004, the Veterans Health Administration began a national dissemination effort of supported employment (SE). Concomitant with the clinical initiative, a research project was funded to study the factors that promote successful program implementation through an evaluation of the effectiveness and cost-effectiveness of two levels of training in the evidence-based practice of SE. This article reviews the dissemination effort and associated research project. This initiative represents the largest dissemination effort of any psychosocial rehabilitation model to date in any single healthcare system in the United States. We review the dissemination plan, including development of a mentor-trainer system at two intensity levels, regular on-site and telephone training and supervision, ongoing fidelity evaluation, and national outcomes monitoring with a Web-based data collection system.
Key words: dissemination, implementation, mentor-trainer system, psychosocial rehabilitation, schizophrenia, severe mental illness, supported employment, train the trainer, VA, vocational rehabilitation.
Abbreviations: CWT = Compensated Work Therapy, FY = fiscal year, HSR&D = Health Services Research and Development Service, NEPEC = Northeast Program Evaluation Center, PSR = psychosocial rehabilitation, SE = supported employment, TWE = transitional work experience, VA = Department of Veterans Affairs, VACO = VA Central Office, VAMC = VA medical center, VHA = Veterans Health Administration, VISN = Veterans Integrated Service Network.*Address all correspondence to Sandra G . Resnick, PhD; VA Connecticut Healthcare System, NEPEC, 950 Campbell Avenue, 182, West Haven, CT 06516; 203-932-5711, ext 5106; fax: 203-937-3433. Email: firstname.lastname@example.orgDOI: 10.1682/JRRD.2007.02.0043INTRODUCTION
The dissemination of complex innovative practices is one of the major challenges of mental health service organizations. Although substantial progress has been made in the development of evidence-based practices for people with severe mental illness , development of approaches for the large-scale dissemination of such practices has lagged, creating a gap between research and practice . Studies of training methods in new practices suggest that multifaceted training initiatives addressed to multiple levels of the organization over an extended period are superior to single narrowly focused interventions . However, innovative programs likely fall on a continuum of training requirements, with simpler and more culture-congruent practices requiring less-intensive training than those with greater complexity and more innovative features . The intensity of the specialized training needed to successfully implement of complex interventions has not been well evaluated . In this article, we describe a current Department of Veterans Affairs (VA), Veterans Health Administration (VHA), initiative to disseminate supported employment (SE).HISTORY OF VHA SUPPORTED EMPLOYMENT INITIATIVE
Since the early 1970s, the VHA has provided vocational rehabilitation services through the Compensated Work Therapy (CWT) program [6-8]. This program includes sheltered workshops and transitional work experiences (TWEs), and veterans with diverse health-related functional impairments are offered paid work opportunities with businesses that contract with the VHA for services. The CWT program operates as an intermediary between employers and veterans for TWE placements. The CWT program establishes contracts with employers, who pay the program directly, and then pays the veterans an hourly wage. No contractual employer-employee relationship is established.
A major expansion of program authority occurred in December 2003, when S-1156, the "Veterans Health Care, Capital Asset, and Business Improvement Act of 2003," was signed into law, permitting the provision of SE services within VHA CWT programs as routine care. The SE model was developed to assist individuals with severe mental illnesses obtain employment beyond transitional work, in permanent jobs in the competitive economy. Once individuals have expressed an interest in working, employment specialists assist them in identifying and obtaining jobs based on individualized preferences and skills and then provide ongoing support and vocational assistance . Unlike traditional CWT, in CWT/SE the worker "owns" the job and is paid directly by the employer. SE is widely considered to be an evidence-based practice, a position supported by both quasi-experimental studies and 15 randomized controlled trials. In one recent review, the rate of competitive employment across nine trials averaged 59 percent for participants in SE compared with 19 percent for those in traditional employment programs .
In 2004, the Secretary of Veterans Affairs, as part of his commitment to implementing the recommendations of the President's New Freedom Commission on Mental Health , allocated $6 million to SE implementation, which was enhanced by an additional $10 million from the VA Strategic Mental Health Plan. Accordingly, a national dissemination effort began under the auspices of the Director for Therapeutic and Supported Work Services in the VA Central Office (VACO) Office of Mental Health Services in partnership with the VA's Northeast Program Evaluation Center (NEPEC).
A previous effort on a smaller scale found that a low-cost, <$10,000 per site, dissemination intervention that provided SE to homeless veterans yielded uneven fidelity overall and modest improvement in employment outcomes , and some have suggested that a more intense training effort would have been more effective . However, little research is available to help mental health service organizations determine the appropriate type and intensity of training to foster successful implementation, while simultaneously minimizing costs.
Concomitant with the clinical initiative, a research project has been funded by the VA Health Services Research and Development Service (HSR&D) on the factors that promote successful program implementation through an evaluation of the effectiveness and cost-effectiveness of two levels of SE training. Examining differences in implementation across sites that receive different levels and types of training will help us identify the training components that are most effective at changing provider behavior. The large number of sites involved in the dissemination provides us a unique opportunity to examine a very large-scale system transformation and will add to our understanding of organizational and provider factors that contribute to successful implementation.
This article reviews of the dissemination effort and associated research project. This initiative represents the largest dissemination effort of any psychosocial rehabilitation (PSR) model to date in any single healthcare system in the United States. We review the dissemination plan, including development of a mentor-trainer system at two intensity levels, use of regular on-site and telephone training and supervision, ongoing fidelity evaluation, and national outcomes monitoring with a Web-based data collection system.DISSEMINATION OVERVIEW
The dissemination effort was based on Rogers' theory of the diffusion of innovation, which identifies five key factors that influence the likelihood and speed at which an innovation will be adopted: (1) the congruence, compatibility, and acceptability of the innovation with providers; (2) the relative advantage of the innovation; (3) the complexity of the innovation; (4) the ease of piloting ("trialability") of the innovation; and (5) the observability of the results of the innovation . All but the first of these factors were theorized to be invariant in the dissemination of SE across VA medical centers (VAMCs). However, the first factor, congruence of existing provider attitudes and behaviors with SE, was hypothesized to be influenced by prior attitudes toward PSR and manualized therapies, while the compatibility and acceptability of SE, especially as they meet the needs of potential adopters, were hypothesized to be influenced by administrative support, overall work environment, and intensity and frequency of training efforts. We hypothesized that the influence of these factors on client employment outcomes would be mediated through the common pathway of fidelity to the evidence-based SE model (Figure 1). A growing body of literature shows that higher SE program fidelity is related to higher rates of competitive employment . We thus hypothesized that congruence, compatibility, and acceptability of SE at each VAMC would directly and jointly influence program effectiveness through their impact on model fidelity.
A total of 162 CWT programs exist nationwide, with operations in each of the VA's 21 regions or Veterans Integrated Service Networks (VISNs). Capitalizing on this structure, we selected 21 CWT programs, one from each VISN, to become "mentor-trainer" sites. The mentor-trainer sites became the focus of the national dissemination initiative, with the further expectation that they would train the remaining SE programs within their VISNs. The 21 sites were selected by national program leadership on the basis of previous success in working with community employers and veterans with severe mental illness, the target group for this initiative. Each site received funding for staff and equipment, including laptop computers with wireless technology and vehicle leases. Sites were also provided funds to hire a minimum of three employment specialists, one of whom would act as the VISN mentor-trainer and assume primary responsibility for training staff at the other CWT programs in the VISN.METHODS: SUPPORTED EMPLOYMENT TRAINING AT MENTOR-TRAINER SITES
The goal of the national dissemination initiative was to train employment specialists at the 21 mentor-trainer sites to both practice and teach SE with a "train-the-trainer" model. The training was designed to be comprehensive and included a series of on-site orientation meetings, with multiple levels of leadership and service providers at each facility; periodic, ongoing, on-site performance monitoring and the interactive review of standardized fidelity scores; regular conference calls with all sites; data-based performance feedback; and telephone supervision. The training time frame was 3 years.
Because of limited training resources, two levels of training were implemented: basic and intensive. All 21 sites received the basic training, which consisted of an initial site visit, follow-up site visits with fidelity ratings and feedback every 6 months thereafter, provision of manuals and other resources, data-based performance feedback, and access to national telephone conference calls. In addition, the intensive sites received more frequent individualized feedback via telephone conference calls. Thus, at the end of the dissemination period, differences in program fidelity and veteran outcomes will be examined by intensity. We designated two national trainers, both of whom had prior national experience training VHA SE vocational programs .Initial On-Site Training (21 Mentor-Trainer Sites)
The training program was initiated at a site when at least one employment specialist had been hired, at which point the initial visit was scheduled. This initial site visit was designed to create a foundation for successful implementation by targeting all levels of the facility from facility leadership through frontline staff as well as training the SE staff on the practice itself. The initial visit included four elements that were designed to follow chronologically, although occasionally they occurred out of order because of scheduling conflicts. We present here the most common schedule for the initial site visit.
Because senior facility leadership can have a strong influence on the acceptance of new programs [15-16], the first element of the initial site visit was an orientation meeting for the top VAMC leaders. We invited the Director, Associate Director(s), Chief of Staff, and Chief of the Mental Health Service Line. A member of VACO PSR attended this meeting in person to underscore the importance of the initiative. The orientation meeting provided an overview of the initiative, its relationship to the President's New Freedom Commission, the ongoing evaluation and monitoring efforts, and an overview of the research and operating principles of SE. Thus, the program goals and evaluation objectives were defined and introduced from the very beginning, with a demonstration of national support.
The second element of the initial site visit was a 1-2 hour training session for frontline clinicians from the entire mental health service line in which the national trainer provided a detailed presentation about SE. An essential element in the evidence-based SE model is close integration between the employment specialists and the mental health treatment teams . While a small number of VA programs regularly use an interdisciplinary team approach [17-18], most mental health services are provided by individual clinicians working independently. As a result, the employment specialists were advised to actively engage individual mental health clinicians to obtain SE referrals and establish partnerships between clinical staff and the SE team to best serve each veteran. The in-service training session was thus an opportunity to begin involving the clinicians with SE education and provide an open forum for them to voice doubts and concerns about the initiative. An important part of this session was the description of "success stories" from a previous demonstration program of SE in VA , which helped frontline staff better understand how SE could potentially improve the lives of the veterans with whom they work. These success stories were designed to influence providers' attitudes, because the extent to which the current organizational culture supports change and innovation may be a key to the success of the intervention . Another potential outgrowth of this session was the encouragement of local stakeholders to provide their support for the initiative and the corresponding change to a more recovery-oriented organizational culture .
The third element was an orientation meeting between the national trainer and the entire CWT program staff. Discussion centered on specific implementation plans, including the referral process, administrative oversight, and strategies for fostering support from clinicians and facility leaders. It was an opportunity for the staff to discuss how the various components of the CWT program would work together, including plans for supervision and staff education.
The last and lengthiest element of the initial site visit was a meeting between the national trainer and the SE employment specialists themselves. In this meeting, the "nuts and bolts" of the SE program were explained and demonstrated, including the assessment process, job development, and ongoing support. Additionally, each site was asked to identify a veteran interested in SE who would volunteer to participate in a demonstration of an initial vocational assessment, which was conducted by the national trainer. The trainer was thus able to demonstrate a detailed vocational assessment, which provides the foundation for job development. The last goal of this element of the initial site visit was to work with the individual designated as the VISN mentor-trainer to discuss strategies for training the other facilities in the region. The mentor-trainer was offered the materials (slides, handouts, manuals) used by the national trainer and encouraged to follow the same training plan at the other sites in the VISN.Ongoing Program Monitoring (21 Mentor-Trainer Sites)
After the initial site visit, one of the two national trainers visited each of the 21 sites for 2 days every 6 months for up to 3 years to consult, meet with leadership, and complete the SE fidelity scale . The SE fidelity scale is the "gold standard" of fidelity assessment, extensively validated, and used in all recent randomized controlled trials of SE. The 15 items are rated on a 5-point scale and organized into three subscales: staffing, organization, and services. The SE fidelity scale successfully discriminates between types of vocational programs with empirically validated cutoff scores . Scores >66 indicated good SE program implementation, from 56-65 fair SE program implementation, and <55 not SE program implementation.
To complete the fidelity scale, the national trainer performed an in-depth program review. Approximately 2 weeks prior to each follow-up visit, each site provided a list of veterans enrolled in SE who had been assigned one of three categories: (1) currently working, (2) had initial assessment and are job seeking, and (3) still in process of initial assessment. A random number generator was used to select one veteran from each category at each of the sites, and sites were notified of the selection.
During the follow-up site visit, the program review focused on these selected veterans and included direct interviews and chart reviews. Each site arranged for the national trainer to meet and interview each of the selected veterans, the mental health clinicians involved with their care, and the employer of the veteran who was currently working. The national trainer also met with all of the SE staff to review caseloads, job placement strategies, program structure, and other features of the SE model. To the extent that time allowed, the national trainer reviewed medical records of the other veterans participating in SE and met with other clinicians and relevant stakeholders. After gathering information for the fidelity assessment, the trainer met with CWT program staff and facility leadership to provide feedback about each site's progress with SE implementation.
In parallel to the initial site visit, each follow-up site visit included a progress review that involved facility leadership, a representative from VACO PSR, and local staff. This meeting provided feedback about the team's progress and reemphasized the importance of support from all levels of the organization. Additionally, providing a realistic time frame for successful implementation was important. Facility leaders were likely to expect full implementation within the first 6 months of the program, and many were concerned when this timetable was not met. Thus, the meeting emphasized that SE implementation is often a lengthy process, dependent on a larger cultural and structural transformation. Without the support of frontline clinicians, even an exceptional SE program cannot achieve full implementation in a brief period of time. This meeting was also an opportunity for the national trainer and the CWT coordinator to solicit assistance from the facility leadership in obtaining additional resources necessary for achieving full fidelity of the SE model.
The last part of each follow-up site visit was a meeting with the SE employment specialists and the national trainer, who provided feedback, positive reinforcement for successes, and suggestions for further development. Often these meetings included brainstorming on ways in which the sites could respond to the national trainer's recommendations. The site mentor-trainers also had the opportunity to ask questions specific to their role in training employment specialists at the other sites in the VISN.
After each follow-up site visit, the national trainer completed the fidelity scale and a narrative that summarized the site's progress and recommendations for improvements. A summary report, including the site's fidelity score and charts that compared the site's progress to national averages, was sent via e-mail to the program director and the VACO Director of Therapeutic and Supported Work Services. After site personnel had time to review the report, it was sent to the facility leadership for review, again emphasizing the importance of leadership in supporting the SE program, such as vehicles, laptop computers, and cellular telephones.
One of the central features of this initiative was open data sharing on a system-wide scale. Approximately every 2 months, a table of all fidelity scores by site and by time (e.g., 6 months, 12 months) was circulated to a national e-mail group that included all 21 mentor-trainer sites. This practice allowed all sites to monitor their progress in comparison with other sites and established a climate of candor, transparency, and mutual learning.Telephone Supervision (Intensive Sites Only)
The intensive training provided to half of the mentor-trainer sites provided an increased amount of individualized training throughout the implementation initiative. These sites were offered regularly scheduled biweekly case supervision conference calls and encouraged to contact their national trainer as needed between calls. Most site personnel took advantage of this supervision; however, some chose not to have regular calls but instead checked in on an as-needed basis. This variation in training intensity was a central feature of the dissemination research project because it allowed for outcomes comparisons across conditions as well as by "training dose." This information may allow clarification of the training components and intensity required for optimal SE dissemination. Both national trainers kept logs of all in-person and telephone contact with their mentor-trainer sites.DISSEMINATION TO ALL SITESOn-Site Training
The ultimate goal and principal challenge of this initiative was the dissemination of SE to all 162 VA CWT programs, not just the 21 mentor-trainer sites. As such, a second level of the train-the-trainer approach was also planned. In this second level, the remaining non-mentor-trainer sites received funds to hire SE staff. The primary responsibility for training these SE staff at the additional sites was assigned to the 21 mentor-trainers; the training was supplemented by national conference calls and periodic training conferences, which created a national learning community for SE implementation.
General guidelines were developed for the designated mentor-trainer at each mentor-trainer site. The designated mentor-trainer was required to have a small caseload, which was expected to increase as training responsibilities decreased. The mentor-trainer was asked to contact the other non-mentor-trainer sites in the VISN to assist in start-up, training, and ongoing monitoring. The mentor-trainer was encouraged to use the same structure and didactic tools as the national trainer, including slides for meetings with facility leadership and frontline clinical staff and regular on-site visits to provide feedback. This second level of training proved challenging because in most cases the mentor-trainers were simultaneously learning the approach themselves and teaching the SE model to staff at several other sites spread over extensive geographical regions. The number of sites in each VISN varied from 5 to 10. Diverse approaches for creating learning communities were thus required.National Conference Calls
All sites had access to biweekly national conference calls, which provided structure for a national learning community, a forum for sites to learn from each other, and an additional opportunity for expert consultation. An active national e-mail group also provided a forum in which various issues could be raised. This e-mail group had weekly, if not daily, traffic and more than 600 CWT clinician subscribers. The CWT staff used this forum to ask questions, share successes, and raise concerns on a range of issues, including those about VHA regulations, employment and receipt of public support and VA benefits, and job development strategies. Staff at the different sites shared examples of brochures that described their programs, relevant articles, and Web sites related to employment and veterans issues.Training Meetings
Several national conferences were held that provided an opportunity for SE staff from all sites to hear a presentation by the national trainer and to participate in discussions amongst themselves about the challenges of SE implementation. The goal of these meetings was to introduce SE to CWT programs in VISNs where the mentor-trainer may have not yet been hired as well as provide a forum for staff to meet each other face-to-face, thus reinforcing the creation of the national learning community.DATA MONITORING
Ongoing program monitoring with regular performance feedback is widely believed to be important for program development and sustainability . Three aspects of program performance were measured following the framework of Donabedian : program structure, program process, and program outcomes. Quality data reports included both national summary data and site-specific data. Structural data were collected via a yearly staffing report; fidelity ratings were performed by the national trainer (for the 21 mentor-trainer sites); and client-level process and outcome data were gathered continuously by CWT staff through an electronic data collection system.
The national electronic data collection system is administered by NEPEC, an evaluation component of the VHA Mental Health Strategic Healthcare Group, and was a central feature of the dissemination and implementation effort. The system is located on the VA's intranet, which ensures that only those with a VA network connection and valid login have access to the system. Multiple security features have been implemented, such as disabling the "Back" button, which ensures that the data flow is one-way-data can be entered into the system but not retrieved by the user. Data are downloaded by NEPEC and then run through a series of range and internal consistency checks. NEPEC program analysts then work with national CWT staff to ensure the validity of the data. The system comprises three electronic data forms: (1) admission, (2) quarterly update, and (3) discharge.
The admission form, as the name implies, is administered by CWT staff when veterans are admitted to the CWT program and documents the characteristics of the veterans participating in the program, such as age, education, and marital status. Information is obtained about the veteran's military service era, theatre of operations, and exposure to hostile or friendly fire. Historical information is also obtained, including income and income sources (benefits and compensation), housing status, clinical diagnoses, recent and past work history, level of functional impairment, and whether or not the veteran desires competitive employment.
Quarterly reports are completed by CWT staff on each veteran every 90 days during CWT participation and document services delivered and outcomes at the individual veteran level. The quarterly updates document participation across the entire CWT continuum, including competitive employment as part of and outside of SE, TWE, sheltered workshops, National Industries for the Severely Handicapped (NISH) jobs, volunteer employment, and vocational training. For each type of activity, the duration of employment, average earnings, average hours worked a week, and number of job changes are documented. The quarterly updates also capture where contact between the CWT staff and the veteran occurs place, such as at a community work site or in the CWT staff offices. An e-mail notification is sent to the staff when a quarterly update is due.
The discharge summary is completed by the CWT staff at the point of discharge and documents veteran outcomes from the time of the last quarterly report to discharge, including the date and reasons for discharge; work activity since the most recent quarterly report; current income, including changes in benefits and compensation since program admission; housing status; employment status at discharge; clinical improvement; and plans for clinical follow-up.
In addition to these client-level data, data on the staffing structure of each program are recorded with an annual staffing survey. During each project year, the CWT program coordinators complete a survey about the total number of staff, hours worked (e.g., full-time equivalent employee), and staffing changes during the reporting period. These surveys are verified and signed by the local facility director. This survey also provides information on each staff member's professional discipline and salary.RESULTS: YEAR 1 DATA
The dissemination initiative is currently in its second year, and preliminary data are available for year 1 only. SE fidelity scores have been gradually increasing over time (Figure 2). On average, programs are demonstrating fair SE implementation by year 1, with a trend toward further improvement at 18 and 24 months. Figure 3 shows the average scores by fidelity subscale. By 18 months, the average for all three subscales was >4 on a 5-point scale. These data trends are encouraging because they suggest continued learning and improvement over time.
We examined the characteristics of those veterans admitted to the CWT program by dividing the sample into two groups, the mentor-trainer sites and the non-mentor-trainer sites, and comparing client characteristics and treatment processes before the dissemination effort was initiated (first half of fiscal year [FY] 2005) and after the second half of the first year of the initiative (second half of FY2006; Table 1). The demographics of those admitted to the program were broadly consistent over time and by type of site. However, between FY2005 and FY2006, both the mentor-trainer and the non-mentor-trainer sites showed increased numbers and percentages of individuals with psychotic disorders, with an ~10 percent increase for both groups. One explanation for this increase is that the target population for SE is individuals with a psychotic disorder, a group previously underserved by the CWT program. Another increase was observed in both the proportion of individuals receiving public financial support, such as disability benefits and other entitlements, and the overall dollar amount of financial support received.
Table 1.Admission characteristics of compensated work therapy (CWT) participants. Data presented as n (%) unless otherwise noted. Variable Mentor-Trainer Sites Non-Mentor-Trainer Sites 1st Half FY2005 2nd Half FY2006 1st Half FY2005 2nd Half FY2006 (n = 890) (n = 1,331) (n = 2,556) (n = 4,181) Male 851 (95.7) 1,246 (93.6) 2,438 (95.4) 3,896 (93.2) Marital Status Currently Married 69 (7.8) 184 (13.8) 181 (7.1) 576 (13.8) Once Married 540 (61.0) 738 (55.5) 1,662 (65.2) 2,457 (58.8) Never Married 276 (31.2) 409 (30.7) 706 (27.7) 1,148 (27.5) Race/Ethnicity White 312 (35.2) 567 (42.6) 1,142 (44.8) 2,158 (51.6) Black 530 (59.8) 728 (54.7) 1,228 (48.1) 1,852 (44.3) Hispanic 29 (3.3) 16 (1.2) 104 (4.1) 39 (0.9) Other 15 (1.7) 20 (1.5) 78 (3.1) 132 (3.2) Psychotic Disorder 157 (17.6) 365 (27.4) 394 (15.4) 1,035 (24.8) Receiving Financial Support (e.g., benefits) 203 (22.8) 646 (48.5) 581 (22.7) 1,929 (46.1) Employment Pattern Past 3 Years Paid Employment 670 (75.4) 877 (65.9) 1,861 (73.1) 2,847 (68.1) Student/Trainee/Volunteer 4 (0.5) 18 (1.4) 29 (1.1) 70 (1.7) Retired/Disabled/Unemployed 215 (24.2) 436 (32.8) 657 (25.8) 1,264 (30.2) Service Era Pre-Vietnam (1918-1964) 16 (1.8) 54 (4.1) 38 (1.5) 142 (3.4) Vietnam (1964-1975) 321 (36.1) 493 (37.0) 961 (37.7) 1,457 (34.9) Post-Vietnam (1975-present) 552 (62.1) 784 (58.9) 1,553 (60.9) 2,582 (61.8) Received Fire in Combat 129 (14.6) 185 (13.9) 462 (18.3) 578 (13.8) Prior Admission to CWT 307 (34.7) 425 (33.9) 816 (32.1) 1,356 (33.6) Residence Past 30 Days House/Apartment 320 (36.0) 608 (45.8) 861 (33.7) 1,872 (44.8) Transitional Housing/Institution 352 (39.6) 559 (42.1) 1,214 (47.5) 1,677 (40.2) No Residence/Other 218 (24.5) 161 (12.1) 481 (18.8) 627 (15.0) Age (yr) (mean ± SD) 47.8 ± 7.1 49.5 ± 7.6 48.0 ± 7.0 49.0 ± 10.9 Education (yr) (mean ± SD) 12.7 ± 1.4 13.0 ± 1.6 12.7 ± 1.5 12.9 ± 1.5 Benefits/Public Support Earnings Past
30 Days ($) (mean ± SD)
202.98 ± 454.16 372.27 ± 774.82 181.90 ± 377.19 345.94 ± 672.70 Note: Data are based on actual number of observations, which may not match total because of missing data.
FY = fiscal year, SD = standard deviation.
Quarterly progress data on work activity during CWT participation is presented in Table 2. Evidence of SE implementation at mentor-trainer sites is apparent, with 29 percent of veterans working in a competitive job as part of SE compared with only 16 percent at the non-mentor-trainer sites. Other trends support the hypothesis that the mentor-trainer sites, the primary target of the national training, have implemented SE quicker than the non-mentor-trainer sites. For example, only 3 percent of veterans at mentor-trainer sites participated in unpaid vocational assessment versus 11 percent at non-mentor-trainer sites, a result consistent with the SE principle of fostering rapid placement without preliminary unpaid vocational assessment. Similarly, rates of TWE are lower at the mentor-trainer sites than at the non-mentor-trainer sites, with 40 percent of individuals in a TWE placement at the mentor-trainer sites compared with 63 percent at the non-mentor-trainer sites. Participation in sheltered workshops is minimal at mentor-trainer sites (2%), while somewhat more common at non-mentor-trainer sites (10%). Interestingly, rates of overall employment are higher at the non-mentor-trainer sites, with 89 percent of those at the non-mentor-trainer sites working Š1 day for each quarterly progress report compared with 76 percent at the mentor-trainer sites. This result is consistent with the findings from the SE controlled trials, in which rates of competitive employment are much higher for experimental sites but equal and sometimes lower rates of overall employment are seen in control sites, because transitional models are also included in the calculation of employment rates [25-27].
Table 2.Mentor-trainer (n = 1,835) versus all other sites (n = 2,908) on vocational outcomes from quarterly updates. Data presented as n (%) unless otherwise noted. Variable Mentor-Trainer Sites Non-Mentor-Trainer Sites Currently Working (any job) 1,061 (57.9) 1,966 (67.6) Past 90 Days, Worked ≥1 Day Competitive Employment, Part of SE 528 (28.8) 457 (15.7) TWE in Community 94 (5.1) 305 (10.5) TWE on VA Grounds 675 (36.8) 1,533 (52.7) Sheltered Workshop 30 (1.6) 289 (9.9) Volunteer Work 36 (2.0) 33 (1.1) Competitive Employment, Not Part of SE 49 (2.7) 94 (3.2) Unpaid Vocational Assessment 59 (3.2) 315 (10.8) Vocational Support Group 430 (23.4) 871 (30.0) Any Employment Past 90 Days 1,390 (75.8) 2,582 (88.8) Days Worked (any job) (mean ± SD) 50.2 ± 25.0 53.4 ± 23.6 Hourly Wage ($) (mean ± SD) 6.93 ± 1.69 6.84 ± 1.73 SD = standard deviation, SE = supported employment, TWE = transitional work experience, VA = Department of Veterans Affairs.DISCUSSION AND CONCLUSIONS
The delivery of verifiable evidence-based practices is one of the major challenges facing contemporary healthcare systems, and the implementation of PSR programs such as SE must be a central objective of mental health systems that seek to encourage a recovery-oriented culture. The national VA initiative described here illustrates key elements required by such initiatives: attention to all levels of the organization, especially top leadership; clear program objectives; investment in training by experienced experts; creation of an open learning community through multiple media; and performance measurement with regular feedback at both the program and client level.
All mentor-trainer sites will continue to be followed for a total of 3 years, at which point we will test our hypothesized model (Figure 1). These data will significantly contribute to the research base on translating efficacious interventions into routine practice. The size of the VHA system provides a tremendous opportunity for examining a large-scale system transformation, and these data will add to our understanding of organizational and provider factors that contribute to successful implementation. Further, examining differences in implementation across sites that receive different levels and types of training will help disentangle which training components are most effective in changing provider behavior.ACKNOWLEDGMENTS
This material was based on work supported by the VA HSR&D (grant MNT 05-098).
The authors have declared that no competing interests exist.REFERENCES1. Drake RE, Goldman HH, Leff HS, Lehman AF, Dixon L, Mueser KT, Torrey WC. Implementing evidence-based practices in routine mental health service settings. Psychiatr Serv. 2001;52(2):179-82. [PMID: 11157115]2. Corrigan PW, Steiner L, McCracken SG, Blaser B, Barr M. Strategies for disseminating evidence-based practices to staff who treat people with severe mental illness. Psychiatr Serv. 2001;52(12):1598-1606. [PMID: 11726749]3. Grimshaw JM, Shirran L, Thomas R, Mowatt G , Fraser C, Bero L, Grilli R, Harvey E, Oxman A, O'Brien MA. Changing provider behavior: An overview of systematic reviews of interventions. Med Care. 2001;39(8 Suppl 2):II2-45.
[PMID: 11583120]4. Simpson DD. A conceptual framework for transferring research to practice. J Subst Abuse Treat. 2002;22(4):171-82.
[PMID: 12072162]5. Schoenwald SK, Hoagwood K. Effectiveness, transportability, and dissemination of interventions: What matters when? Psychiatr Serv. 2001;52(9):1190-97. [PMID: 11533392]6. Resnick SG , Medak S, Baldino R, Corwel L, Rosenheck RA. Ninth Progress Report on the Compensated Work Therapy (CWT)/Veterans Industries (VI) Program: Fiscal Year 2005. West Haven (CT): Northeast Program Evaluation Center; 2006.7. Resnick SG, Rosenheck RA, Drebing CE. What makes vocational rehabilitation effective? Program characteristics versus employment outcomes nationally in VA. Psychol Serv. 2006;3(4):239-48.8. Kashner TM, Rosenheck R, Campinell AB, Surís A, Crandall R, Garfield NJ, Lapuc P, Pyrcz K, Soyka T, Wicker A. Impact of work therapy on health status among homeless, substance-dependent veterans: A randomized controlled trial. Arch Gen Psychiatry. 2002;59(10):938-44. [PMID: 12365881]9. Becker DR, Drake RE. A working life for people with severe mental illness. New York (NY): Oxford University Press; 2003.10. Bond GR. Supported employment: Evidence for an evidence-based practice. Psychiatr Rehabil J. 2004;27(4):345-59.
[PMID: 15222147]11. Hogan MF. The President's New Freedom Commission on Mental Health: Recommendations to transform mental health care in America. Psychiatr Serv. 2003;54(11):1467-74.
[PMID: 14600303]12. Rosenheck RA, Mares AS. Implementation of supported employment for homeless veterans with psychiatric or addiction disorders: Two-year outcomes. Psychiatr Serv. 2007; 58(3):325-33. [PMID: 17325104]13. Bond GR. Modest implementation efforts, modest fidelity, and modest outcomes. Psychiatr Serv. 2007;58(3):334.
[PMID: 17325105]14. Rogers EM. Diffusion of innovations. New York (NY): Free Press; 1995.15. Hirschhorn L. Campaigning for change. Harv Bus Rev. 2002;80(7):98-104,126. [PMID: 12140858]16. Bloch RM, Saeed SA, Rivard JC, Rausch C. Lessons learned in implementing evidence-based practices: Implications for psychiatric administrators. Psychiatr Q. 2006; 77(4):309-18. [PMID: 16927164]17. Neale M, Rosenheck R, Castrodonatti J, Martin A, Morreissey J, D'Amico M. Mental health intensive case management (MHICM): The Ninth National Performance Monitoring Report FY2005. West Haven (CT): Northeast Program Evaluation Center; 2006.18. Rosenheck RA, Neale MA. Development, implementation, and monitoring of intensive psychiatric community care in the Department of Veterans Affairs. In: Dickey B, Sederer L, editors. Achieving quality in psychiatric and substance abuse practice: Concepts and case reports. Washington (DC): American Psychiatric Publishing; 2001. p. 183-92.19. Zammuto RF, Krakower JY. Quantitative and qualitative studies of organizational culture. In: Woodman RW, Pasmore WA, editors. Research in organizational change and development. 5th ed. Greenwich (CT): JAI Press; 1991. p. 83-114.20. Doumit G , Gattellari M, Grimshaw J, O'Brien MA. Local opinion leaders: Effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2007;(1): CD000125. [PMID: 17253445]21. Bond GR, Becker DR, Drake RE, Vogler KM. A fidelity scale for the individual placement and support model of supported employment. Rehabil Couns Bull. 1997;40(4): 265-84.22. Bond GR, Vogler KM, Resnick SG , Evans LJ, Drake RE, Becker DR. Dimensions of supported employment: Factor structure of the IPS fidelity scale. J Ment Health. 2001; 10(4):383-94.23. Rosenheck R. Stages in the implementation of innovative clinical programs in complex organizations. J Nerv Ment Dis. 2001;189(12):812-21. [PMID: 11794574]24. Donabedian A. The definition of quality and approaches to its assessment. Ann Arbor (MI): Health Administration Press; 1980.25. Drake RE, McHugo GJ, Bebout RR, Becker DR, Harris M, Bond GR, Quimby E. A randomized clinical trial of supported employment for inner-city patients with severe mental disorders. Arch Gen Psychiatry. 1999;56(7):627-33.
[PMID: 10401508]26. Lehman AF, Goldberg R, Dixon LB, McNary SW, Postrado L, Hackman A, McDonnell K. Improving employment outcomes for persons with severe mental illness. Arch Gen Psychiatry. 2002;59(2):165-72. [PMID: 11825138]27. Mueser KT, Clark RE, Haines M, Drake RE, McHugo GJ, Bond GR, Essock SM, Becker DR, Wolfe R, Swain K. The Hartford study of supported employment for persons with severe mental illness. J Consult Clin Psychol. 2004;72(3): 479-90. [PMID: 15279531]Submitted for publication February 27, 2007. Accepted in revised form June 25, 2007.
Go to TOP
Last Reviewed or Updated Tuesday, December 4, 2007 2:09 PM