Volume 44 Number 6, 2007
Pages 879 — 892
Bridging science to service: Using Rehabilitation Research and Training Center program to ensure that research-based knowledge makes a difference
Marianne Farkas, ScD;* William A. Anthony, PhDCenter for Psychiatric Rehabilitation, Sargent College of Health and Rehabilitation Sciences, Boston University, Boston, MA
Abstract — The challenge of bridging science to service is increasingly visible in the healthcare field, with emphasis on the influence of evidence-based knowledge on both policy and practice. Since its inception more than 40 years ago, the Rehabilitation Research and Training Center (RRTC) program has provided grants for both research and training activities designed to ensure that research knowledge is translated into practice. The RRTC program is unique in that its mission and funding have always required that both time and money be invested in the translation and dissemination of research-generated knowledge to users in the field, i.e., decision makers and practitioners. Boston University's Center for Psychiatric Rehabilitation has been an RRTC for more than 25 years and provides an example of the effect of the RRTC program in bridging science to service. The Center's mission as an RRTC has been to develop and transfer research knowledge to decision makers and practitioners who can then inform change and promote progress in mental health disability policy and practice. This article reviews five basic dissemination and utilization principles for overcoming the most common barriers to effective dissemination of evidence-based knowledge and provides examples of the Center's activities related to each principle. In addition, a knowledge-transfer framework developed by the Center to organize dissemination and utilization efforts is described.
Key words: evidence-based messages, knowledge dissemination, knowledge-transfer framework, knowledge utilization, policy maker, psychiatric disabilities, psychosocial, recovery, rehabilitation, research.
Abbreviations: CMHS = Center for Mental Health Services, EBM = evidence-based message, KT = knowledge transfer, NIDRR = National Institute on Disability and Rehabilitation Research, NIMH = National Institute of Mental Health, RRTC = Rehabilitation Research and Training Center, SSA = Social Security Administration.*Address all correspondence to Marianne Farkas, ScD; Center for Psychiatric Rehabilitation, Boston University, 940 Commonwealth Avenue West, Boston, MA 02215; 617-353-3549. Email: firstname.lastname@example.orgDOI: 10.1682/JRRD.2006.08.0101INTRODUCTION
The challenge of getting research-generated knowledge disseminated to decision makers and practitioners has been increasingly noted in the healthcare field in general [1-3] and the mental health field in particular [4-7]. Given the overuse, underuse, and misuse of research information by healthcare providers, managers, and decision makers , many groups in the United States, such as the New Freedom Commission  and the Institute of Medicine , have called for the revision of common strategies for bridging the gap between "what is known" and "what is done" [1,10-11]. Increasingly, experts in the field are recognizing that overcoming this gap involves a complex and dynamic process of knowledge transfer (KT) [12-16]. KT has been defined as a "process of exchange, synthesis, and ethically sound application of knowledge within a complex system of relationships among researchers and users" . The components of KT have typically been thought to include a method of identifying quality information (such as systematic reviews, evidence grading, syntheses), program development or adaptation of the content, program implementation, evaluation of knowledge utilization, and implementation of strategies for sustainability [16-19]. Concerted efforts to transfer research knowledge to the field have recently become more widespread. The activities have ranged widely, from evaluating a body of evidence in a particular domain (e.g., supported housing , grading mental health program evidence , developing tool kits for promoting the sustainability of a specific group of evidence-based practices , establishing state funded centers) to specifically promoting the implementation of evidence-based practices (e.g., Ohio's eight Coordinating Centers of Excellence and Michigan's Practice Improvement Steering Committee ).
Despite these efforts, many of the obstacles (e.g., lack of congruence between dissemination and utilization strategies and their intended goals, lack of planning and resources allocated to overcoming the gaps, existence of "two [separate] communities": researchers and users) that created the gap between science and service and were identified more than 30 years ago persist in current practice today [6,23-27]. For example, Landry et al. recently found that 53 percent of Canadian government officials surveyed indicated that research results rarely or never influenced their policy-making decisions . Similar findings have been identified in studies in other countries [28-29].
In attempting to develop evidence-based or "evidence-informed" decision making , policy makers and administrators have turned to various researchers and academics . At least 33 states have recently initiated partnerships with their major universities and private colleges to support the implementation of specific evidence-based practices, primarily through evaluation and training activities . The National Institute of Mental Health (NIMH) instituted the Research Infrastructure Program to expand the number of partnerships between community-based programs, clinical settings, and academic institutions to enhance our national capacity to provide evidence-based mental healthcare in community settings . However, many universities and research institutions still do not place a high priority on planned KT , from the generation of research to the implementation of the innovation or research findings. KT activities still tend to be the least-funded aspect of research grants, and they are often an afterthought rather than a planned component of research studies [33-34]. Since most researchers are professionally rewarded (i.e., promotions and tenure) based on publication in peer-reviewed journals [32,35], the transfer of research findings may be targeted to a narrow audience of professionals, which can result in inadequate communication of the findings to the practice field (e.g., publication in journals read only by researchers or use of language poorly understood by constituents beyond the research community) [35-36]. In addition, the lack of incentives results in many researchers having little training or experience with KT strategies and techniques . This lack of planning is often compounded by the large lag in time between identification of findings, publication of the research, and translation into usable products. These limitations make it difficult for the published information to be responsive to current circumstances [37-38] and, therefore, reduce the credibility of researchers in the eyes of decision makers . Effectively bridging science to service is facilitated by some level of understanding between the communicator-researcher and the specific stakeholder-user. The knowledge itself must be a fluid interactive set of understandings that includes stakeholder motivation and sociocultural differences affecting implementation of an innovation by decision makers and be shaped by both those who originate it and those who receive it [40-42].
The challenge of ensuring that research-based knowledge makes a difference, therefore, involves effectively disseminating critical evidence-based information to decision makers and others. A very early and ongoing response to this science to service or KT initiative in the field of disability research has been the university-based Rehabilitation Research and Training Center (RRTC) program.COMBINING RESEARCH AND TRAINING EXPERTISE IN ONE ORGANIZATION
The Center for Psychiatric Rehabilitation at Boston University (the Center), an RRTC, is unique as a research organization in mental health in that its mission and funding have always required that both time and money be invested in the translation and dissemination of research-based knowledge to users in the field, e.g., decision makers and practitioners. The Center's development as an RRTC in psychiatric rehabilitation was originally funded through the cooperation of NIMH and the Rehabilitation Services Administration and now exists as a collaborative effort between the Center for Mental Health Services (CMHS) and the National Institute on Disability and Rehabilitation Research (NIDRR). Center researchers and experts work together at dissemination and utilization of research knowledge to bridge science to service.
The RRTC program was initiated in 1962 to conduct rehabilitation research, train personnel in rehabilitation, and support dissemination and utilization of research findings and products . Currently, the NIDRR program funds 34 RRTCs in a wide variety of disability areas (http://www.ncddr.org).PRINCIPLES OF EVIDENCE-BASED MESSAGING
The RRTC at Boston University has coined the term "evidence-based messaging" to characterize attempts to ensure that research results become take-home messages for the decision makers, administrators, and practitioners who have the power to act on them. The Center's efforts have been guided by the early and ongoing dissemination and utilization literature. This article describes five principles of evidence-based messaging, with examples from the Center's more than 25 years of experience as an RRTC in using these principles to transfer evidence-based messages (EBMs) into practice:1. Develop EBMs that are based on a body of research rather than single data sets.2. Build credibility with decision makers as legitimate developers of EBMs.3. Build KT expertise and infrastructure within the research organization.4. Convey EBMs using an organized approach to achieve targeted outcomes.5. Routinely evaluate evidence-based messaging efforts.Principle 1. Develop Evidence-Based Messages from Body of Research not Single Data Sets
In contrast to dissemination of data from a single research study, EBMs are the critical ideas that emerge out of a body of research. Furthermore, these messages are framed to influence decision makers. Decision makers are not apt to be influenced by the size of regression coefficients or the significance levels in individual studies but by critical ideas presented as simple messages that can inform their decision making . The Figure provides examples of EBMs from research performed by the Center and other researchers in the field. Center researchers reviewed studies on a particular topic or question to determine if a simple theme or message emerged. Next, the Center's technical assistance experts and the Center's researchers framed each message in a single sentence.
1. Mental health rehabilitation is plagued by poor outcomes, distracted by irrelevant diagnostic system, saddled with inefficient treatment approaches, and burdened by disorganized community support system. 2. Helping people choose their own goals can positively affect rehabilitation outcome. 3. Psychiatric diagnostic system is not predictive, prescriptive, or descriptive of psychiatric rehabilitation practices. 4. No strong correlation exists between peoples' symptoms and skills. 5. Psychiatric rehabilitation, including its processes, outcomes, values, and principles, can be successfully described and taught. 6. People with psychiatric disabilities are positively affected by psychiatric rehabilitation services. 7. People with psychiatric disabilities can and want to go to school. 8. People with psychiatric disabilities can and want to work. 9. People with psychiatric disabilities can and do recover. 10. Supports can improve people's functioning in living, learning, and working environments (i.e., supported housing, supported education, and supported employment). Figure.Examples of evidence-based messages developed and transferred to decision makers by Boston University's Center for Psychiatric Rehabilitation.
Each of the examples in the Figure can be further elaborated by a review of the underlying data. For example, the EBM on the failures of the mental health system with respect to rehabilitation outcomes (EBM 1, Figure) was regularly detailed over the last 15-30 years in several articles and book chapters [44-47]. The evidence specified that the base rehabilitation outcome rates for people with severe mental illnesses were very poor (i.e., less than 25% have competitive employment, and in the first year, 40%-50% exhibit state hospital recidivism).
While decision makers who are transforming the mental health system to include more of a rehabilitation perspective may be comforted by knowing about numerous research studies, the essence of the body of research must be framed in a straightforward succinct way for it to be effective. In EBM 1, the implication for system leadership was that the system clearly needed to be redesigned if rehabilitation outcomes were to be improved. Decision makers need the ideas and knowledge that emerge from the data rather than the empirical details [28,40]. Even then, however, the EBM will only be useful to the extent to which it can be applied within the constraints of a decision maker's environment .
Researchers and technical assistance experts who develop EBMs from research data must examine the data they themselves generate, data from other mental health studies, and data from studies in related fields. An EBM describes a theme that is generated from a variety of data sources. Systematic reviews, meta-analyses, and registries of evidence-based guidelines are all current methods of developing information syntheses [1,10,48]. Such syntheses are often conducted by research utilization committees and interdisciplinary consensus panels (e.g., http://www.campbellcollaboration.org or http://www.cochranecollaboration.com). An RRTC has the infrastructure and internal resources needed to conduct reviews and syntheses . For example, another important EBM the Center promoted to decision makers is that people with psychiatric disabilities must have the opportunity to choose their own rehabilitation goals (EBM 2, Figure). In formulating this EBM, Center interdisciplinary staff were guided not only by the experimental laboratory research on goal setting of Locke and Latham  but also by existing mental health research [44,51-55]. When all the goal-setting studies were examined, a theme emerged that setting one's own goals can improve satisfaction, performance, and perseverance.
The current NIMH initiative on translational research presupposes that behavioral science research conducted on non-mental health participants is significant to mental health services research . The previously cited research of behavioral scientists on goal setting is but one confirmatory example of the wisdom of this translational research initiative. Take-home messages must be based on conclusions from research across the available literature in both the primary field of study and related fields.
EBMs are succinct statements that are substantiated with research data and designed to imply a direction but not specific polices and procedures for policy makers and practitioners. Examples of EBMs that suggest direction but not precise action are EBM 3, about the limitations of the psychiatric diagnostic system, and EBM 4, about poor correlation between a person's skills and symptoms (Figure). These particular EBMs infer the necessity of change for those state vocational rehabilitation directors, who often spend considerable amounts on psychiatric assessments and psychological testing for the purpose of diagnostic clarity and rehabilitation planning. However, these EBMs do not indicate a particular psychiatric assessment policy or procedure.
Standards that can rate the research rigor and the meaning of the studies that contribute to an EBM are still to be developed. Numerous attempts at developing standards for research on various interventions in healthcare, education, and mental health have been made (e.g., the Cochrane Collaboration, the Campbell Collaboration, the Department of Education's What Works Clearinghouse, the National Registry of Evidence-Based Programs and Practices). Existing standards seem to be in a constant state of refinement . For example, current standards do not usually include an indication of the meaningfulness of the research to various stakeholders , despite the fact that the meaningfulness of the information is clearly important to the likelihood of the information being used [57-58]. The Center is in the process of developing such standards for disability research through a 5-year NIDRR grant on knowledge dissemination. The project also includes identification of standards for observational and correlational studies that are more frequent in the disability and rehabilitation fields than randomized clinical trials . In addition to the emergence of standards for observational and correlational studies, standards for rating intervention research may be giving way to current suggestions for research on change processes and treatment principles [59-62], which is more in line with the notion of EBMs.Principle 2. Build Credibility with Decision Makers as Legitimate Developers of Evidence-Based Messages
Whether we as scientists like it or not, the credibility of the messenger affects the impact of the message. If individuals or organizations are to play a role in delivering EBMs, then they must be viewed as credible sources by decision makers. Credibility is one of the hallmarks of innovation [40,63]. Credibility is earned over a period of time, however, which partly explains why innovations take a significant time to be accepted [40,64]. Furthermore, credibility is constantly assessed by the field so that it is not a static characteristic but one that must be achieved on an ongoing basis .
To build knowledge development and KT credibility, an organization must demonstrate organizational capability [14,16]. In the mental health research field, the recognized method for meeting the criteria of capability and credibility is successful peer review of submitted articles and grant proposals. In addition to publishing data about individual research projects, an individual or organization must also perform seminal reviews of selected topic areas to establish credibility. For example, EBMs 5 and 6 (Figure) state that the processes, outcomes, values, and principles of psychiatric rehabilitation can be successfully described, taught, and delivered and have been periodically reviewed for more than 2 decades by Center researchers in textbooks [44,65], book chapters , and journal articles [66-67].
While peer review has no substitute for establishing research credibility, the Center realized early on in its existence that it must also earn credibility in the KT process if it wished to significantly influence decision makers. As an academic research organization, it was in danger of being perceived as simply an ivory tower research organization, out of touch with the clientele the research knowledge was designed to help (a common perception of research organizations [3,17]), the cultural needs of the patients and providers, and the feasibility concerns of the community organizations involved.
The Center decided to develop two experiential components with which to counteract this perception. First, training and technical assistance experts on staff worked with mental health agencies to implement new research knowledge in the context of the normal daily struggles encountered in mental health practice. As Jacobson et al. point out , KT-focused consulting is a powerful tool for effective transfer of research information to decision makers. Second, innovative services, such as supported education (EBM 7, Figure), were developed, conducted, and evaluated at the Center's research site [68-69]. By providing these two experiential components, our researchers and technical assistance experts had the opportunity to interact regularly with service recipients, providers, and administrators and expand our staff with recruits from these same groups.
To build credibility among policy makers, an individual or organization must also identify those persons who influence policy. In the mental health field, the policy-level decision makers are primarily the state leaders in mental health and rehabilitation, local exemplary organizations, government-funding bodies, elected officials and their staff, other researchers with a publishing and funding track record, and now more than ever, advocacy organizations, including groups of people with psychiatric disabilities and their families, all of whom may have very different information needs. Once the decision makers are identified, credibility is built by listening to the decision makers, understanding their specific concerns, building research projects relevant to these concerns, and following their lead [16,65]. An organization attempting to build credibility must attend decision makers' conferences and engage them in ongoing communication through in-person visits, telephone calls, and electronic means. Leaders of these constituencies must be invited to sit on advisory boards and participate as consultants or staff on individual research projects .
Decision makers from one constituency will be able to advise the research organization about issues faced by other constituencies and facilitate access to other decision makers. For example, the Center's early research on the relationship between vocational outcome and people's skills, symptoms, diagnoses, and demographics (EBMs 3, 4, and 8; Figure) was introduced to the Social Security Administration (SSA) and the Mental Health Law Project (now the Bazelon Center) through the intervention of a Center Advisory Board member who represented NIMH. As a result, Center researchers testified in several class action lawsuits about the relevance of this body of research to SSA's disability determination revisions , influencing a revision in their disability determination procedures for people with severe mental illnesses .Principle 3. Develop Knowledge Transfer Expertise and Infrastructure Within Research Organization
To provide effective ongoing KT activities, organizations must plan their activities with the requisite KT expertise and an infrastructure of technology, support, and resources. As the Center's researchers are housed within a university setting, KT is a compatible and expected activity. However, we have found that most researchers in academia do not have the time, skills, or interest for this activity, nor should they, since KT is not their defined role or training. As indicated previously, as an RRTC, most of the Center's grant-supported research projects include funding for dissemination of the research. This funding has allowed us to hire experts in KT. As the Center developed expertise in KT, we published both research and conceptual articles on issues related to the application of research knowledge [12,73-74], which in turn increased the Center's credibility in the KT field.
An important part of the infrastructure needed for KT is electronic media. An interactive Web site is almost a requirement for an organization attempting to influence decision makers. As a research organization attempting to bridge science to service, the Center has invested in a Web expert whose major responsibility is Web design and maintenance. The Center's Web site, http://www.bu.edu/cpr, currently hosts an average of 9,000 unique visitors each month. This Web site was recognized as a "Web site worth watching" in Psychiatric Services . A monthly Mental Health and Rehabilitation "e-Cast" that contains information on current research projects and KT activities is e-mailed to more than 5,500 subscribers.
Effective KT, however, must appeal not only to formal decision makers but also to a variety of influential constituencies who need to know, understand, and use the research information (e.g., consumers, families, educators, providers) . Not all of these audiences regularly use the Web as a primary means of information. Consequently, expertise and infrastructure must also support traditional print media (e.g., fact sheets, brochures) and face-to-face interactions (e.g., consulting, training seminars, workshops). When a research organization that engages in evidence-based messaging hires an in-house expert in materials design and production, as the Center has, the specific materials needed to facilitate KT (books, tapes, videos, etc.) can be produced quickly and inexpensively, with direct input from the researchers who study the topic area and the constituents who will be using the information.Principle 4. Convey Evidence-Based Messages Using Organized Approach to Achieve Targeted Outcomes
In addition to creating the message itself, the research organization must use an organized planned dissemination approach that specifies the outcome goals, strategies, and target audiences of the EBM. A lack of clarity about the congruence between the methods used and the expected outcome has been one of the reasons cited for the gap between health services research and community-based practice [38,76]. Research has demonstrated that simply making information available to decision makers will not necessarily result in a change in practice . Difficulties in changing practice stem rather from issues such as legislative, funding, and sociopolitical structures that do not support innovation [16,65,77-78].
To implement this principle, the Center has developed a framework for the types of goals that organizations can expect and the KT strategies that stem from them. The "4E" framework of dissemination and utilization clarifies what effects an organization might expect from the common KT strategies: exposure, experience, expertise, and embedding . In other words, the KT strategy employed flows out of the type of goal or outcome the organization is attempting to achieve. The 4E framework, presented in the Table, was developed to overcome the difficulties of KT noted earlier, i.e., the confusion between goals and strategies, the lack of a planned approach to the transfer of research information, and the lack of interaction in KT methodologies between users and researchers.
Table.Summary of knowledge dissemination and utilization framework. 4E framework: exposure (dissemination), experience (dissemination), expertise (utilization), and embedding (utilization). Strategy Exposure Experience Expertise Embedding Goal Increase knowledge. Increase knowledge/ positive attitudes. Increase competence. Increase use in practice. Example by Target
Provider Hold conferences, publish in popular media, develop group-specific Web site. Create internships. Write/deliver training programs/manuals with supervised skill practice. Provide long-term clinical/program supervision. Person Served,
Publish in popular
media, develop group-specific Web site.
program visits/discussion groups led by role models with positive outcomes from
programs in how to
use or support use of
assistance in advocacy
Present position papers, testify at hearings, form task forces. Meet with policy makers who have used EBM. Visit successful policy sites for EBM. Provide on-site systems-level technical assistance for EBM. Administrator Hold conferences, publish in e-journals/popular media, create e-mail bulletins, develop group-specific Web site. - Visit successful
Provide on-site program-level technical assistance/organizational
Researcher Publish in professional journals, give seminars, create e-mail bulletins/digital libraries. Create research mentorships/brief internships. Create supervised research fellowships/courses. Provide technical assistance/advocacy to create ongoing availability of experts, fund ongoing research. Educator Publish in professional journals/association newsletters, write
- Create summer institutes to teach educators how to teach new EBM material develop scripted lesson plans/curricula. Create faculty incentives for implementation/mentorship/peer
relationships to support implementation.
General Public Create Web sites/fact sheets/public service spots for media, designate neighborhood spokesperson. Take positions on community boards, disseminate EBM information. Develop brief lay EBM training programs for persons likely to be affected. Implement methods for other specific target populations (e.g., change in
Source: Adapted from Farkas M, Jette AM, Tennstedt S, Haley SM, Quinn V. Knowledge dissemination and utilization in gerontology: An organizing framework. Gerontologist. 2003;43(Special No. 1):47-56 [PMID: 12637689].EBM = evidence-based message.
The 4Es reflect the complexity of KT. Exposure and experience (i.e., dissemination) strategies are designed to spread or promulgate new information and ensure that individuals are knowledgeable and have positive attitudes toward the information. These strategies include exposure-related activities, such as traditional passive methods of dissemination (journal or popular articles, conferences, lectures) , as well as newer methods (Web strategies) based on an active information-seeking paradigm . The information-seeking paradigm sees users as exploring, learning, analyzing, and confirming information, rather than just reading it. Experience-related activities are aimed at changing attitudes that limit the implementation of innovations [5,80-81] and include the use of role models who provide the opportunity to discuss proposed innovations with peers. An experienced mentor can inculcate a sense of hopefulness about the possibility of change that is not present when users are simply provided with new information [82-83].
Expertise and embedding (i.e., utilization) strategies are designed to translate the new information into organizational and personnel structures that ensure the expert application and embedding of the information in daily practice. Expertise activities, such as intensive training programs with supervised practice to build competence, are a step toward precise performance and predictable outcomes . Supervised practice is often forgotten in attempts to create expertise. For example, tool kits or curricula that describe competencies and implementation strategies do not produce expertise without the input of a supervisor or mentor to provide feedback on the implementation effort . The Center has developed a series of practitioner training curricula to support such supervisors or mentors in developing expertise in the psychiatric rehabilitation approach [83,85-86], a promising practice developed by the Center KT specialists and researched over a 30-year period . Embedding strategies require complex knowledge utilization methods to increase the implementation of the new findings or innovation. This need to institutionalize knowledge in daily practice has long been recognized as one of the most difficult aspects of KT and is indeed the linchpin of bridging science to service [14,16,30,76]. Embedding includes activities such as technical assistance or consultation, typically over a lengthy time period. These activities can help organizations develop (1) structures that support the EBM [14,30,74], (2) rules and legislation that support the EBM, (3) public policy strategies that strengthen social norms and expectations related to the EBM , and (4) resources to sustain the changes over time, such as funding for the EBM . The 4E framework can be used categorically, to generate strategies for one KT goal only (e.g., expertise), or comprehensively, to develop strategies across two or more goals (e.g., expertise and embedding).
The strategies to achieve each goal are designed for specific groups of decision makers (Table). Strategies can be developed to reach each group of decision makers and increase that group's knowledge, positive attitudes, skills, or institutionalized use of the EBM. For example, to transfer the messages that people with severe mental illnesses can recover (EBMs 7, 8, and 9; Figure), the Center has used multiple strategies directed toward multiple outcomes. In terms of a simple exposure strategy for this message, the Center has used the interaction provided by the Web, presentations, and workshops to repeatedly communicate to researchers, family members, and practitioners that long-term follow-up studies demonstrate that recovery from severe mental illnesses occurs often . The Center uses the collection and sharing of real stories of recovery as an experience method to reinforce the attitudes of people with psychiatric disabilities about the possibilities of their own recovery . At the expertise level, the Center has developed training materials to help people with mental illnesses and their service providers be more skilled at facilitating the recovery process and program administrators incorporate recovery-oriented programming into their sites more easily [82-83,86,90-91]. At the embedding level, the Center has provided on-site technical assistance to policy makers who design mental health programs and systems to become more recovery oriented [60,92].
Such experiences have underlined the need for KT experts to spend significant time on-site in personal contact with those involved in the implementation of an innovation to ensure that the EBM's become embedded. Demonstrations, individualized feedback, and modifications must be made in real time and face-to-face if embedding is to be successful. For example, based on the evidence underlying EBMs 5, 6, and 10 (Figure), the Center's technical assistance experts used expertise and embedding strategies with a local agency who had received Federal-supported housing funds to install a supported housing program with the Center's psychiatric rehabilitation approach [83,85-86,93]. Evaluation of the program indicated that the service recipients, a majority of whom had at one time been homeless and abused drugs, significantly increased their community living days and their homelessness was eliminated . However, work outcomes were minimally affected, and a new research-based program of supported employment was added to the supported housing program, with combined support from the state Division of Vocational Rehabilitation and the state Office of Mental Health Services. Evaluation of this project evidenced an employment rate of 47 percent for the service recipients of this combined program .Principle 5. Routinely Evaluate Evidence-Based Messaging Efforts
Ensuring ongoing relevance requires various forms of interaction with information users. One such type of interaction is the evaluation of changes brought about by knowledge generation and KT, something that is not regularly evaluated in most research projects. Because the Center has been funded, however, to both generate and transfer knowledge, we have been able to attempt this task through the use of two major strategies: collecting direct and indirect evidence of impact.Direct Evidence
For some KT outcomes, we have been able to track the effect of our research and related technical assistance through follow-up surveys of information users and unsolicited feedback. We have evaluated our efforts in terms of tracking outputs, utility, and outcomes using a framework developed by NIDRR, one of our main Federal funding sources . Outputs are the direct products or results of our 4E strategies. Utility is the perceived usefulness of these outputs. Outcomes are the effects of these outputs. For example, over a 5-year period (1999 to 2004), the Center tracked and reported on these evaluation components . Center outputs included training more than 45,000 personnel in recovery, community support, and rehabilitation topics, including research results and implications of research results supported by Federal funds. In another example of tracking outputs over the same period, the Center tracked use of the Web site. Data revealed that more than 839,193 pages were viewed between October 2002 and September 2003. An average of 7,190 unique individuals visited the Web site each month, viewing six to seven pages per visit between 1999 and 2004.
These output data, however, only provide a gauge of the degree to which the target audiences are exposed to EBMs. The extent to which the target audiences view the output as useful is a key link between communication of the information and the eventual embedding of the information [40,78]. In addition to training evaluations conducted immediately after each training event, we sent follow-up surveys between 3 and 6 months after participation in a short-term dissemination or utilization strategy. One such survey was sent to a sample of 1,035 individuals randomly selected from lists of those who had participated in a training workshop, conference, Webcast, or seminar at the end of one funding cycle . In addition to the initial message, two reminders were sent over a 2-week period; 168 individuals responded (16%). In addition, 500 individuals who had purchased some form of Center material (e.g., article, book, manual, multimedia package) were likewise surveyed; 85 individuals responded (17%). The survey asked questions about the utility of the event or materials to the participant's daily work. Each question was answered on a 3-point scale (i.e., very useful, somewhat useful, or not at all useful). The information was collected on a Web form, with tab-delimited data files. Alternative formats were provided for those who had difficulty using the Web form. Only three individuals expressed difficulty (0.02%). Seventy-one percent of the respondents who had participated in Center workshops, Webcasts, and conferences reported that the experience proved to be very useful. The participants were also asked about identifiable ways in which the information was helpful. For example, 32 percent of respondents reported using the information in subsequent meetings projects, grants, and papers; 18 percent used it in program development; 20 percent used it to teach other staff, students, and peers; and 30 percent used it to help reevaluate their own philosophy or as a way of interacting with others in their direct service provision.
Outputs with respect to a specific EBM can also be evaluated. For example, a supported education program implementation manual related to EBMs 7 and 10 (Figure) was written to help organizations embed supported education. Within 2 years, 15 programs reported using the manual to implement their own programs.
When directly surveying decision makers about the Center's influence on their actions, we find that the actual behavior change depends on the individual's role. For clinicians, the change in behavior may be use of different practice guidelines. For an advocacy group, it may be communicating the message in an action format to their constituencies. For a legislator, it may be crafting new laws. For policy makers, it may be creating new funding streams. For program administrators, it may be rewriting their program policies and procedures. For researchers, it may be investigating additional hypotheses.Indirect Evidence
EBMs are relatively easy to track through indirect evidence as well. For example, the value placed on Center information can be evidenced by looking at the number of times Center researchers are cited by other authors as well as more widespread changes in the field and the Center's inferred role in helping to bring about these changes. Other organizations in the field acknowledge the Center's Web site as a resource and confirm the value of the Center's information by linking to the site. A search conducted in November 2005 found more than 100 organizations, associations, and research centers that link their Web sites to the Center's Web site.
As an example of tracking the role a research organization plays in more widespread change, Center researchers demonstrated the need for supported education  and its positive effect on individuals who receive such services . The Center is generally attributed as one of the main founders of the supported education service model . In Massachusetts, the State Department of Mental Health contracted with Center technical assistance experts to demonstrate the service around the state  and later embedded supported education in state regulations and funding . More recently, CMHS came to believe that supported education was important enough to its efforts at transforming the U.S. mental health system that it funded the development of implementation tool kits on supported education . Another example on both the state and national level is the Center's repeated communication about recovery from severe mental illnesses (EBM 9, Figure). Currently, at least 39 states have adopted a mission statement or policy about the potential of consumers to recover . In another example, the Center developed and evaluated a structured psychiatric rehabilitation process for helping consumers choose, get, and keep the roles they prefer [44,65,87]. In addition to being implemented in numerous programs in more than 20 countries around the world, this process was recently nominated as one of Oregon's approved evidence-based practices. Furthermore, it was selected as the process used to train Fellows in the VA National Psychosocial Fellowship program. Lastly, working collaboratively with others communicating the same message, the Center's evidence-based messaging seems to have played a part in variety of states' adoption of recovery as their guiding vision  as well as that of entire countries, such as the United States  and New Zealand . However, the notion of indirect impact remains difficult to empirically evaluate and may be seen as more of a yearning than a reality. In the future, research needs to be conducted to further understand the role of evidence-based messaging in complex system-level transformation.DISCUSSION AND CONCLUSIONS
The reason that bridging science to service does not routinely occur is that it is a difficult process and cannot be an underfunded afterthought to the research endeavor. Furthermore, the KT process requires constant innovation to remain relevant in the field. The new knowledge developed may well pose direct challenges to accepted ideas and practices (e.g., the recovery vision) and require new EBMs to reach new stakeholders. Thus, effective evidence-based messaging requires ongoing dialogue among researchers, KT experts, and varying groups of decision makers as well as multiple evaluations of existing information to continually develop relevant new messages and overcome shifting challenges to the implementation of new findings. This dynamic process can be best delivered by organizations mandated to conduct both research and KT.
Such organizations need to have the capacity to develop long-term relationships of trust with decision makers to identify new areas of needed knowledge, overcome potential resistance, and influence practice. In addition to longevity, these organizations also need flexibility to make the kinds of changes in research and KT strategies that may be required as the implications of new findings become clear. Using the five principles of KT, organizations must plan dissemination and utilization activities as part of the research process, carried out by credible experts in KT, separately funded, and evaluated. Through more than 25 years of funding as an RRTC, the Center has been one of the few university-based mental health research centers specifically funded to bridge science to service. The Center's experience and knowledge gained from the use of dynamic and constantly adapting bridging activities have resulted in the development of guiding principles and suggestions for different ways research organizations may contribute to bridging science to service.ACKNOWLEDGMENTS
This material was based on work partially supported by NIDRR/CMHS grants H133B990023 and H133B040026 and NIDRR grant H133A050006.
The authors have declared that no competing interests exist.References1. Grol R, Grimshaw J. From best evidence to best practice: Effective implementation of change in patients' care. Lancet. 2003;362(9391):1225-30. [PMID: 14568747]2. Innvaer S, Vist G , Trommald M, Oxman A. Health policy-makers' perceptions of their use of evidence: A systematic review. J Health Serv Res Policy. 2002;7(4):239-44.
[PMID: 12425783]3. Moynihan R. Using health research in policy and practice: Case studies from nine countries. New York (NY): Millbank Memorial Fund; 2004.4. Amador XF, Fitzpatrick M. Science to services: Consumers need "real-world" science. Schizophr Bull. 2003;29(1): 133-37. [PMID: 12908668]5. Corrigan PW, Steiner L, McCracken SG , Blaser B, Barr M. Strategies for disseminating evidence-based practices to staff who treat people with serious mental illness. Psychiatr Serv. 2001;52(12):1598-1606. [PMID: 11726749]6. Institute of Medicine. Improving the quality of health care for mental and substance-use conditions. Washington (DC): National Academies Press; 2006.7. Hogan MF. The President's New Freedom Commission: Recommendations to transform mental health care in America. Psychiatr Serv. 2003;54(11):1467-74. [PMID: 14600303]8. Walshe K, Rundall TG . Evidence-based management: From theory to practice in health care. Millbank Q. 2001; 79(3):429-57. [PMID: 11565163]9. What Works Clearinghouse [homepage on the Internet]. Princeton (NJ): U.S. Department of Education, Institute of Science; c2007 [updated 2007 Aug 13; cited 2005 Jun 2]. Available from: http://ies.ed.gov/ncee/wwc/10. Choi BC. Understanding the basic principles of knowledge translation. J Epidemiol Community Health. 2005;59(2):93.
[PMID: 15650137]11. Davis DA, Thomson MA, Oxman AD, Haynes RB. Evidence for the effectiveness of CME. A review of 50 randomized controlled trials. JAMA. 2003;268(9):1111-17.
[PMID: 1501333]12. Farkas M, Jette AM, Tennstedt S, Haley SM, Quinn V. Knowledge dissemination and utilization in gerontology: An organizing framework. Gerontologist. 2003;43 Spec No 1:47-56. [PMID: 12637689]13. Hoge MA, Paris M Jr, Adger H Jr, Collins FL Jr, Finn CV, Fricks L, Gill KJ, Haber J, Hansen M, Ida DJ, Kaplan L, Northey WJ Jr, O'Connell MJ, Rosen AL, Taintor Z, Tondora J, Young AS. Workforce competencies in behavioral health: An overview. Adm Policy Ment Health. 2005; 32(5-6):593-631. [PMID: 16082798]14. Jacobson N, Butterill N, Goering P. Consulting as a strategy for knowledge transfer. Milbank Q. 2005;83(2):299-321.
[PMID: 15960773]15. Landry R, Lamari M, Amara N. Extent and determinants of utilization of university research in government agencies. Pub Adm Rev. 2003;63(2):192-205.16. Lavis JN, Robertson D, Woodside JM, McLeod CB, Abelson J; Knowledge Transfer Study Group. How can research organizations more effectively transfer research knowledge to decision makers? Milbank Q. 2003;81(2): 221-48. [PMID: 12841049]17. Canadian Institutes of Health Research. Canadian Institutes of Health Research Knowledge Translation Strategy: Niche and Focus 2005-2009. Ottawa, Canada; 2004.18. Jacobson N, Butterill D, Goering P. Development of a framework for knowledge translation: Understanding user context. J Health Serv Res Policy. 2003;8(2):94-99.
[PMID: 12820671]19. National Center for the Dissemination of Disability Research. Overview of international literature on knowledge translation. Focus. 2006;14:1-6.20. Rog DJ. The evidence on supported housing. Psychiatr Rehabil J. 2004;27(4):334-44. [PMID: 15222146]21. National Association of State Mental Health Program Directors. State mental health agency implementation of the six new freedom commission goals: 2006: State mental health agencies (SMHAs) are making substantial progress toward achieving the major goals of the commission. NASMHPD Research Institute. 2006;6(1):1-6.22. Substance Abuse and Mental Health Services Administration (SAMHSA) National Mental Health Information Center [homepage on the Internet]. Washington (DC): SAMHSA; c2007 [updated 2007 Sep 10; cited 2006 Nov 1]. Available from: http://mentalhealth.samhsa.gov23. Caplan N. The two-communities theory and knowledge utilization. Am Behav Sci. 1979;22(3):459-70.24. Hamilton LS, Muthard JE. Research utilization specialists in vocational rehabilitation: Five years of experience. Gainesville (FL): Rehabilitation Research Institute, College of Health Related Professions, University of Florida; 1975.25. Switzer ME. Research and demonstration grant program: Description of program and procedures for applying for and administering grants. Washington (DC): Vocational Rehabilitation Administration, U.S. Department of Health, Education, and Welfare; 1965.26. Backer TE. The failure of success: Challenges of disseminating effective substance abuse prevention programs. J Comm Psychol. 2000;28(3):363-73.27. National Advisory Mental Health Council. Bridging science and service: A report. Bethesda (MD): National Institute of Mental Health; 1999.28. Whiteford H. Can research influence mental health policy? Aust N Z J Psychiatry. 2001;35(4):428-34.
[PMID: 11531721]29. Selby Smith C. The impact of research on decision-making by practitioners and managers. In: Proceedings of the Australian Vocational Education and Training Research Association Conference; 2001; Adelaide (Australia).30. Rosenheck RA. Organizational process: A missing link between research and practice. Psychiatr Serv. 2001;52(12): 1607-12. [PMID: 11726750]31. National Institute of Mental Health [homepage on the Internet]. Bethesda (MD): National Institute of Mental Health; c2007 [updated 2007 Mar 13; cited 2006 Nov 1]. R24 programs supported by NIMH; [about 4 screens]. Available from: http://www.nimh.nih.gov/grants/r24.cfm32. Jacobson N, Butterill D, Goering P. Organizational factors that influence university-based researchers' engagement in knowledge transfer activities. Sci Commun. 2004; 25(3):246-59.33. Duffy M. The Internet as a research and dissemination resource. Health Promot Int. 2000;15(4):349-53.34. Scullion PA. Effective dissemination strategies. Nurse Res. 2002;10(1):65-77. [PMID: 12405007]35. Prohaska TR, Peters KE, Warren JS. Health behavior: From research to community practice. In: Albrecht GA, Fitzpatrick R, Scrimshaw SC, editors. Handbook of social studies in health and medicine. London (England): Sage Publications; 2000. p. 359-73.36. Dobbins M, Cockerill R, Barnsley J. Factors affecting the utilization of systematic reviews. A study of public health decision makers. Int J Technol Assess Health Care. 2001; 17(2):203-14.37. Bradley EH, Webster TR, Baker D, Schlesinger M, Inouye SK, Barth MC, Lapane KL, Lipson D, Stone R, Koren MJ. Translating research into practice: Speeding the adoption of innovative health care programs. Issue Brief (Commonw Fund). 2004;(724):1-12.
[PMID: 15270051]38. Morrissey E, Wandersman A, Seybolt D, Nation M, Crusto C, Davino K. Toward a framework for bridging the gap between science and practice in preventions: A focus on evaluator and practitioner perspectives. Eval Program Plann. 1997;20(3):367-77.39. Jacobson N, Goering P. Credibility and credibility work in knowledge transfer. Evid Pol J Res Debate Pract. 2006; 2(2):151-65.40. Greenhalgh T, Robert G , MacFarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: Systematic review and recommendations. Milbank Q. 2004; 82(4):581-629. [PMID: 15595944]41. King L, Hawe P, Wise M. Making dissemination a two-way process. Health Promot Int. 1998;13(3):237-44.42. Rogers EM. Lessons for guidelines from the diffusion of innovations. Jt Comm J Qual Improv. 1995;21(7):324-28.
[PMID: 7581733]43. Hayward B, Adelman N, Tashjian M. Assessment of the rehabilitation research and training centers program. Report for Rehabilitation Services Administration. Washington (DC): Policy Studies Associates; 1988.44. Anthony WA, Cohen MR, Farkas M. Psychiatric rehabilitation. Boston (MA): Boston University Center for Psychiatric Rehabilitation; 1990.45. Anthony WA, Cohen MR, Cohen BF. Psychiatric rehabilitation. In: Talbot JA, editor. The chronic mental patient: Five years later. Orlando (FL): Grune & Stratton; 1984. p. 137-57.46. Anthony WA, Cohen MR, Vitalo R. The measurement of rehabilitation outcome. Schizophr Bull. 1978;4(3):365-83.47. Anthony WA, Nemec PB. Psychiatric rehabilitation. In: Bellack AS, editor. Schizophrenic treatment, management, and rehabilitation. Orlando (FL): Grune & Stratton; 1984. p. 375-413.48. Graham ID, Logan J, Harrison MB, Straus SE, Tetroe J, Caswell W, Robinson N. Lost in knowledge translation: Time for a map? J Contin Educ Health Prof. 2006;26(1): 13-24. [PMID: 16557505]49. Farkas M, Rogers ES. Rehabilitation for persons with long-term mental illness: Final report. Grant H133B90023-00. Washington (DC): U.S. Department of Education, National Institute on Disability Rehabilitation Research; 2005.50. Locke EA, Latham GP. Building a practically useful theory of goal setting and task motivation. A 35-year odyssey. Am Psychol. 2002;57(9):705-17. [PMID: 12237980]51. Becker DR, Drake RE, Farabaugh A, Bond GR. Job preferences of clients with severe psychiatric disorders participating in supported employment programs. Psychiatr Serv. 1996;47(11):1223-26. [PMID: 8916240]52. Bell M, Lysaker P. Levels of expectation for work activity in schizophrenia: Clinical and rehabilitation outcomes. Psychiatr Rehabil J. 1996;19(3):71-76.53. Farley RC, Bolton B, Taylor JK. The effects on vocation development of a strategy for empowering and involving consumers in the vocational evaluation process. Vocat Eval Work Adjust Bull. 1993;26(3):97-100.54. Kissen B, Platz A, Su WH. Selective factors in treatment choice and outcome in alcoholics. In: Mellow NK, Mendelson JH, editors. Recent advances in studies of alcoholism; An interdisciplinary symposium. Rockville (MD): National Institute on Alcohol Abuse and Alcoholism; 1971.55. Lovell AM, Cohn S. The elaboration of "choice" in a program for homeless persons labeled psychiatrically disabled. Hum Organ. 1998;57(1):8-20.56. Farkas M, Rogers ES. Innovative knowledge dissemination and utilization for disability and professional organizations and stakeholders; Grant H133A050006. Washington (DC): U.S. Department of Education, National Institute on Disability Rehabilitation Research; 2005.57. National Institute on Disability and Rehabilitation Research (NIDRR). NIDRR's long range plan for fiscal years 1999-2003. Washington (DC): Department of Education; 1999.58. Rose D, Thornicroft G , Slade M. Who decides what evidence is? Developing a multiple perspectives paradigm in mental health. Acta Psychiatr Scand Suppl. 2006;(429): 109-14. [PMID: 16445492]59. Ablon JS, Marci C. Psychotherapy process: The missing link: Comment on Westen, Novotny, and Thompson-Brenner (2004). Psychol Bull. 2004;130(4):664-68.
[PMID: 15250816]60. Anthony WA. Studying evidence-based processes, not practices. Psychiatr Serv. 2003;54(1):7-11. [PMID: 12509657]61. Anthony W, Rogers ES, Farkas M. Research on evidence-based practices: Future directions in an era of recovery. Community Ment Health J. 2003;39(2):101-14.
[PMID: 12723845]62. Westen D, Novotny CM, Thompson-Brenner H. The empirical status of empirically supported psychotherapies: Assumptions, findings, and reporting in controlled clinical trials. Psychol Bull. 2004;130(4):631-63.
[PMID: 15250817]63. Locock L, Dopson S, Chambers D, Gabbay J. Understanding the role of opinion leaders in improving clinical effectiveness. Soc Sci Med. 2001;53(6):745-57. [PMID: 11511050]64. Stein LI. Innovating against the current. In: Stein LI, editor. Innovative community mental health programs. San Francisco (CA): Jossey Bass, Inc; 1992.65. Anthony WA. Psychiatric rehabilitation. Boston (MA): Boston University, Center for Psychiatric Rehabilitation; 2002.66. Anthony WA, Cohen M, Farkas M. The future of psychiatric rehabilitation. Int J Ment Health. 1999;28(1):48-68.67. Dion GL, Anthony WA. Research in psychiatric rehabilitation: A review of experimental and quasi-experimental studies. Rehabil Couns Bull. 1987;30(3):177-203.68. Danley KS, Rogers ES, MacDonald-Wilson K, Anthony W. Supported employment for adults with psychiatric disability: Results of an innovative demonstration project. Rehabil Psychol. 1994;39(4):269-76.69. Unger KV, Anthony WA, Sciarappa K, Rogers ES. A supported education program for young adults with long-term mental illness. Hosp Community Psychiatry. 1991;42(8): 838-42. [PMID: 1894260]70. Rogers ES, Palmer-Erbs V. Participatory action research: Implications for research and evaluation in psychiatric rehabilitation. Psychosoc Rehabil J. 1994;18(2):3-12.71. Anthony WA, Jansen MA. Predicting the vocational capacity of the chronically mentally ill. Research and policy implications. Am Psychol. 1984;39(5):537-44.
[PMID: 6742594]72. Meyerson AT, Fine T. Psychiatric disability: Clinical, legal and administrative dimensions. Washington (DC): American Psychiatric Press; 1987.73. Farkas MD, Anthony WA, Cohen MR. An overview of psychiatric rehabilitation: The approach and its programs. In: Farkas MD, Anthony WA, editors. Psychiatric rehabilitation programs: Putting theory into practice. Baltimore (MD): Johns Hopkins University Press; 1989.74. Nemec PB, Forbess R, Cohen MR, Farkas MD, Rogers ES, Anthony W. Effectiveness of technical assistance in the development of psychiatric rehabilitation programs. J Ment Health Admin. 1991;18(1):1-11. [PMID: 10110686]75. Web sites worth watching. Psychiatr Serv. 2000;51(11):1382.
[PMID: 11058183]76. Altman DG. Sustaining interventions in community systems: On the relationship between researchers and communities. Health Psychol. 1995;14(6):526-36.
[PMID: 8565927]77. Farkas MD, Cohen MR, Nemec PB. Psychiatric rehabilitation programs: Putting concepts into practice? Community Ment Health J. 1988;24(1):7-21. [PMID: 3370939]78. Kilbourne AM, Schulberg HC, Post EP, Rollman BL, Belnap BH, Pincus HA. Translating evidence-based depression management services to community-based primary care practices. Milbank Q. 2004;82(4):631-59.
[PMID: 15595945]79. Paisley W. Knowledge utilization: The role of new communication technologies. J Am Soc Inform Sci. 1993;44(4): 222-34.80. Backer TE, Liberman RP, Kuehnel TG . Dissemination and adoption of innovative psychosocial interventions. J Consult Clin Psychol. 1986;54(1):111-18. [PMID: 3958295]81. Good TL, Berenbaum H, Nisenson LG . Residential caregiver attitudes toward seriously mentally ill persons. Psychiatry. 2000;63(1):23-33. [PMID: 10855757]82. Cohen M, Forbess R, Farkas M. Psychiatric rehabilitation training technology: Developing readiness for rehabilitation. Boston (MA): Boston University, Center for Psychiatric Rehabilitation; 2000.83. Farkas M, Cohen M, McNamara S, Nemec P, Cohen B. Assessing readiness for rehabilitation (Rehabilitation readiness training technology). Boston (MA): Boston University, Center for Psychiatric Rehabilitation; 2001.84. Farkas M, Anthony WA. Overview of psychiatric rehabilitation education: Concepts of training and skill development. Rehabil Educ. 2001;15(2):119-32.85. Cohen J. A power primer. Psychol Bull. 1992;112(1):155-59.86. Cohen MR, Farkas MD, Cohen BF, Unger KV. Psychiatric rehabilitation training technology: Setting an overall rehabilitation goal (Trainer package). Boston (MA): Boston University, Center for Psychiatric Rehabilitation; 1990.87. Rogers ES, Anthony W, Farkas M. The choose-get-keep approach to psychiatric rehabilitation: A synopsis of recent studies. Rehabil Psychol. 2006;51(3):247-56.88. Orleans CT. Promoting the maintenance of health behavior change: Recommendations for the next generation of research and practice. Health Psychol. 2000;19(1 Suppl):76-83.
[PMID: 10709951]89. Harding CM, Zahniser JH. Empirical correction of seven myths about schizophrenia with implications for treatment. Acta Psychiatr Scand Suppl. 1994;384:140-46.
[PMID: 7879636]90. Spaniol L, Koehler M, Hutchinson D. The recovery workbook: Practical coping and empowerment strategies for people with psychiatric disability. Boston (MA): Boston University, Center for Psychiatric Rehabilitation; 1994.91. Spaniol LJ, Koehler M, Hutchinson D. Leaders' guide: The recovery workbook: Practical coping and empowerment strategies for people with psychiatric disability. Boston (MA): Boston University, Center for Psychiatric Rehabilitation; 2000.92. Farkas M, Gagne C, Anthony W, Chamberlin J. Implementing recovery oriented evidence based programs: Identifying the critical dimensions. Community Ment Health J. 2005;41(2):141-58. [PMID: 15974495]93. Anthony WA, Furlong-Norman K, Koehler M. Shifting paradigm in mental health services: Supported education within the context of rehabilitation and recovery. In: Mowbray C, editor. Supported education and psychiatric rehabilitation: Models and methods. Columbia (MD): International Association of Psychosocial Rehabilitation Services; 2002. p. 287-94.94. Anthony WA, Brown MA, Rogers ES, Derringer S. A supported living/supported employment program for reducing the number of people in institutions. Psychiatr Rehabil J. 1999;23(1):57-61.95. McLaughlin JA, Jordan GB. Logic models: A tool for describing program theory and performance. In: Wholey, editor. Handbook of practical evaluation. New York (NY): Jossey-Bass; 2003.96. Anthony WA, Unger KV. Supported education: An additional program resource for young adults with long term mental illness. Community Ment Health J. 1991;27(2): 145-56. [PMID: 2044354]97. Mowbray C, editor. Supported education and psychiatric rehabilitation: Models and methods. Columbia (MD): International Association of Psychosocial Rehabilitation Services; 2002. p. 181.98. Lim SY, Nicolellis DL, Fahey H, Jaggi C. Massachusetts mobile supported education programs. In: Mowbray C, editor. Supported education and psychiatric rehabilitation: Models and methods. Columbia (MD): International Association of Psychosocial Rehabilitation Services; 2002. p. 75-86.99. Jacobson N, Curtis L. Recovery as policy in mental health services: Strategies emerging from the states. Psychiatr Rehabil J. 2000;23(4):333-41.100. Lapsley H, Nikora LW, Black R. "Kia Mauri Tau!" Narratives of recovery from disabling mental health problems. Wellington (New Zealand): Mental Health Commission; 2002.Submitted for publication August 28, 2006. Accepted in revised form March 29, 2007.
Go to TOP
Last Reviewed or Updated Friday, December 7, 2007 9:53 AM