Journal of Rehabilitation Research and
Development Vol. 36 No. 4, October 1999
Outcome assessment of the rehabilitation of the visually impaired
William R. De l'Aune, PhD; Michael D. Williams, MS; Richard L. Welsh, PhD
VA Rehabilitation R&D Center, Atlanta, GA 30033; Pittsburgh Vision Services, Pittsburgh, PA 15213
Abstract - The purposes of this project are to establish the psychometric properties of instruments used to gather data relevant to blind rehabilitation outcomes, to refine the scaling and scoring protocols for the instruments, and to revise and refine the outcome instruments. This 3-year project will gather outcome and demographic data from an estimated 1,200 visually impaired veterans per year, along with a companion sample an estimated 1,200 visually impaired nonveterans per year, using the following core measures: Blind Rehabilitation Service Follow-up Outcome Survey (BRSFOutSur) measuring functional performance, Blind Rehabilitation Service Data Base (BRSDBase) recording subject characteristics, and Blind Rehabilitation Service Satisfaction Survey (BRSSatSur) measuring satisfaction with rehabilitation. As of July 1999, data from 2,624 veterans have been collected for the demographic instrument, from 1,630 veterans for the functional outcomes instrument, and from 1,655 veterans for the satisfaction instrument. Data collection and analysis are currently ongoing. These findings and the further development of outcome instruments in this area will contribute to greater efficiency and effectiveness of the delivery of blind rehabilitation services by the Department of Veterans Affairs.
Key words: aged, outcome assessment (health care), vision subnormal.
This material is based upon work supported by the Department of Veterans Affairs Office of Rehabilitation Research and Development, Washington, DC 20204
Address all correspondence to William R. De l'Aune, PhD, VA Rehabilitation R&D Center, Atlanta, GA 30033; email: email@example.com
This report is of a project in progress. Approximately half of the data acquisition and most of the data analysis are yet to be accomplished. As such, the report will focus on the methodology and provide only a cursory view of the results to date.
All rehabilitation service providers are under increasing pressure to demonstrate that the services they provide result in the people they serve enjoying improved functional abilities, improved quality of life, or both. In spite of these growing expectations, there has been relatively little research in outcome measurement in the area of rehabilitation services for people with blindness and visual impairment. Typically, practitioners in blind rehabilitation have used clinical assessments and subjective checklists to record changes in their clients' ability to perform functional tasks. A review of the literature by Barrett on behalf of the National Accreditation Council for Agencies Serving the Blind and Visually Handicapped (NAC) indicated very few studies that focused directly on the measurement of rehabilitation outcomes for people with visual impairments (1).
A national project is under way in the United States to develop, test, and validate functional outcome measures that will be of value in rehabilitation programs serving blind or visually impaired adults. This project is also developing a unified database of demographic, program, and outcome information that may begin to provide "benchmarks" for comparing the effectiveness of rehabilitation agencies, programs, and delivery systems. These tools will be of value both in research and in guiding future funding decisions.
This research project, conducted by the Rehabilitation Research and Development (RR&D) Center of the Atlanta VA Medical Center in Decatur, Georgia, in cooperation with the U.S. Department of Veterans Affairs (VA) Blind Rehabilitation Service (BRS), the NAC, and more than 20 state and private rehabilitation agencies in over 13 states, has two specific research questions: 1) are the outcome measures currently in use by VA BRS sensitive to long-term changes associated with blind rehabilitation? 2) can these outcome measures be adequately utilized to predict future service demands for blind rehabilitation? The ultimate goal of this research is to provide rehabilitation professionals with tools for program planning, program evaluation, and outcomes research that will result in the improvement of function and quality of life of veterans aging with a visual disability.
Outcomes Measurement Construction
The indisputable goal of rehabilitation is improving patient outcomes. Consequently, there has been much discussion in the literature about what patient outcomes should be measured and what is the best way to measure them (2-4). From these discussions has come a consensus that the primary rehabilitation outcome is functional status (2,3,5-7), and that "quality of daily life" that goes beyond physical functioning, discharge status (destination), resource use (personal care and medical care), productive activity, and patient satisfaction is also an important outcome (2,4,7-9). Several authors have critiqued instruments proporting to assess these outcomes quantitatively and have provided recommendations for further research into rehabilitation outcome assessment (2-7).
The discussion of rehabilitation outcomes has also given attention to what factors contribute to desired outcomes. The changing health care environment, with a focus on accountability and cost-effectiveness, necessitates more than ever the clear identification of those factors (aside from patient characteristics) that can be linked to outcomes. The implication is that those factors that demonstrate a positive relationship will find a place in the health care environment and those that do not may be eliminated regardless of the previous role they have played in health care delivery.
Qualitative data can provide valuable information that can be used to develop hypotheses regarding rehabilitation outcomes. However, the hypothesis testing needed to quantify the benefits patients derive from rehabilitation requires quantitative assessment. To accomplish the task of developing and evaluating quantitative assessment instruments, this project is focused on the fundamental psychometric properties that all quantitative assessments should possess. The most fundamental of these are reliability and validity. For measurement applications involving change (such as pre/post rehabilitation change), it is also important that an instrument be responsive to change and yield scores meaningful at the group level. Finally, a measure should be feasible to administer.
When constructing an instrument, the first task is to develop a set of items that could potentially be included in the final version of the instrument. A set of items is typically based on a theory or framework that comes from literature review, professional expertise, input from potential respondents, and/or personal experience. Evidence for face validity comes from professional review and that of potential respondents on an informal basis or through pilot testing, while evidence of content validity comes from a formal review of the theory or framework on which the measure is based. Once content validity is established, empirical evidence of construct validity and other types of validity, including the validity of scoring, can be conducted. An investigator can derive information about the feasibility of an instrument's administration through pilot testing and continued administration. Despite a rigorous development process, it should be noted that the demonstration of an instrument's reliability is situation- and population-specific. For example, a quantitative assessment of blind rehabilitation outcomes that appears to be reliable in the VA setting may not necessarily demonstrate the same level of reliability in a setting of private blind rehabilitation agencies. Another important consideration is that validity can never be conclusively proven. To support a claim of reliability and validity requires a series of empirical demonstrations in the context of a well-defined theory and framework.
The Measurement of Outcomes in Rehabilitation of People with Visual Impairments
The current project builds on a sequence of related research that began with the work of John Crews at the Michigan Commission for the Blind in the early 1980s. This research studied changes in the performance of 57 functional items on a two-dimensional scale: independence and difficulty (10). In 1988, a study by the VA Southeastern Blind Rehabilitation Center in Birmingham attempted to modify for use with individuals with visual impairment the Functional Independence Measure (FIM), the most widely used inventory for persons with physical impairments.
The Birmingham staff attempted to modify the FIM to measure visual abilities, orientation and mobility, living skills, and manual skills. The two previously reported studies were blended together in another VA-sponsored study titled "Functional Independence Measure for Blind Adults" (FIMBA; 11). The FIMBA used the two scales of independence and difficulty from the Michigan study and added a satisfaction scale as well. This addition made it possible to assess a client's level of satisfaction with the performance of certain items, independent of the levels of independence and difficulty.
In 1996, the VA BRS Task Group on Outcomes (BRSTGO) developed the BRS Functional Outcomes Survey (BRSFOutSur) as the centerpiece of the current study. This endeavor has emerged as a broad-based and comprehensive national outcomes project studying services for both veterans and nonveterans.
The National Outcomes Project
This project, as conceptualized by BRSTGO and the RR&D Center of the Atlanta VA Medical Center, was designed to gather outcomes data from an estimated 1,200 visually impaired veterans per year over the 3 years of the project. These data are combined with the demographic, diagnostic, and program data of the unified database in an effort to identify those factors that seem most likely to influence or predict outcomes for veterans with certain characteristics and receiving certain types of services. While this research database is currently being maintained at the Rehab R&D Center, the transition to an ongoing clinical and administrative database housed in the VA Computer Center in Austin, Texas, is underway and should be completed in 1999.
Concurrent with the development of the VA's Blind Rehabilitation Outcome Measurement Project, the NAC was exploring ways to encourage the systematic development of outcome measures that could be used by schools and other agencies that avail themselves of NAC's voluntary accreditation process. The VA's Rehab R&D Center saw the value of adding a sample of nonVA subjects and programs to the VA's project in order to further test the assessment instruments, to validate the project's findings against a nonVA group, and to develop a more comprehensive set of benchmarks for blind rehabilitation services.
Twenty nonVA agencies indicated a willingness to submit a total of approximately 1,200 subjects to participate in the study each year of the project. These data from the nonVA agencies are combined in the central research database with the demographic, diagnostic, and program data gathered by the VA blind rehabilitation facilities. The analyses of this rich pool of complex data will be used to validate and test the reliability of the instruments and to begin to propose benchmarks for blind rehabilitation services. The combined database will be large enough to allow for statistical control of the many different variables that have an impact on rehabilitation outcomes and to develop models of rehabilitation outcomes that are "risk adjusted" for these client characteristics.
Approximately 3,600 visually impaired veterans are expected to take part in VA Blind Rehabilitation programs during the 3-year duration of this project. Attempts will be made to enroll this entire group, but based on experience with telephone interviews of this population to date, project staff anticipate successful involvement of only 80 percent (3,000). Videotapes of 5 percent of the sample performing domain-specific functional tasks in their home environments will be made. Additionally, a quasi-random sample of as many as 3,000 nonveteran visually impaired subjects are being recruited from the nonVA blind rehabilitation programs taking part in a parallel study. Sampling specifics were determined in May 1997. Sampling was based on the portion of clients that an agency felt it could enroll in the project. Given this number and the estimated clients served per year, a sampling protocol of enrolling every nth client was established. (For example, if an agency were willing to recruit 25 percent of its clients to participate in this study, it would select every fourth client enrolled in its program). Participating VA and nonVA centers are listed in Table 1.
Table 1. Collaborating institutions.
VA Blind Rehabilitation Centers: American Lake Blind Rehabilitation Center, Tacoma, WA Augusta Blind Rehabilitation Center, Augusta, GA Birmingham Blind Rehabilitation Center, Birmingham, AL Hines Blind Rehabilitation Center, Hines, IL Palo Alto Blind Rehabilitation Center, Palo Alto, CA Puerto Rico Blind Rehabilitation Center, San Juan, PR Tucson Blind Rehabilitation Center, Tucson, AZ Waco Blind Rehabilitation Center, Waco, TX West Haven Blind Rehabilitation Center, West Haven, CT Non-VA Blind Rehabilitation Agencies (as of July 15, 1998): Association for the Blind and Visually Impaired, Grand Rapids, MI The Carroll Center for the Blind, Newton, MA Cincinnati Association for the Blind, Cincinnati, OH Cleveland Sight Center, Cleveland, OH The Clovernook Center Opportunities for the Blind, Cincinnati, OH Community Services for the Blind and Partially Sighted, Seattle, WA Florida Division of Services for the Blind, Daytona Beach, FL Indiana Bureau for the Blind, Indianapolis, IN The Lighthouse, Duluth, MN Low Vision Service, The Department of Ophthalmology and
Visual Sciences, The University of Iowa Hospitals and Clinics,
Iowa City, IA
Metrolina Association for the Blind, Charlotte, NC Pinellas Center for the Visually Impaired, Largo, FL Pittsburgh Vision Services, Pittsburgh, PA St. Louis Society for the Blind and Visually Impaired,
Webster Groves, MO
Tampa Lighthouse for the Blind, Tampa, FL Vermont Association for the Blind and Visually Impaired,
The core instruments used in this study are:
- BRSFOutSur--Blind Rehabilitation Service Functional Outcome Survey
- BRSSatSur--Blind Rehabilitation Service Satisfaction Survey
- BRSDBase--Blind Rehabilitation Service Database
The primary measure used in this project is the BRSFOutSur, attached as Appendix A. It samples self-reported behaviors and perceptions associated with tasks linked to each of the four major Blind Rehabilitation skill area domains (Orientation and Mobility, Communication and Activities of Daily Living, Manual Skills, and Visual Skills) as well as other behaviors associated with general adjustment to blindness. The individual is asked about each task in terms of how often it is undertaken, how important its accomplishment is, how independently it can be performed, and how satisfied the respondent feels with his/her ability to perform the task.
The research team also administers the BRSSatSurv, an instrument designed to measure satisfaction with the programatic aspects of the rehabilitation program. This instrument is attached as Appendix B.
In addition to the BRSFOutSur and BRSSatSurv, the project team collects extensive information about the subject's demographic, rehabilitation, and medical history contained in the BRSDBase. This includes many factors that can be used for risk adjustment in outcomes studies resulting from this project and is completed by the rehabilitation facility at the time of the client's discharge. The BRSDBase data collection form is attached as Appendix C.
The research protocol and study design are shown in Table 2. The visually impaired subjects participating in this study are provided information about the project and asked to participate during their enrollment in rehabilitation. Each rehabilitation facility takes responsibility for compiling the data required for the BRSDBase on each client enrolled in the project. After completion of the rehabilitation program, the client's relevant data are transmitted to the Decatur Rehabilitation R&D Center. Information Resource Management Development and Support for VA Head Quarters (VAHQ) has been working with the BRSTGO on implementing the BRSDBase and portions of the BRSFOutSur as Class I software used on all VA computer systems by Fall 1999. This will allow direct input of this data from the Blind Rehabilitation Centers and Visual Impairment Service Team (VIST) case managers.
Table 2. Study design.
Task Time Responsibility Data Gathering Months 1-36 RA and BRC CS BRSDBase Discharge BRC CS BRSFOutSur-Post < 1 month after Discharge RA BRSSatSur < 1 month after Discharge RA BRSFOutSur-ProjPre < 1 month after Discharge RA BRSFOutSur-FU < 6 month after Discharge RA Test-Retest Reliability (5%) BRSFOutSur-Retest < 1 week after BRSFOutSur-Post RA Home Visit Video Taping for Validity (5%) Video Tape < 6 month after Discharge RA Video Analysis Months 1-36 RA Descriptive Analysis Quarterly PI and RA Analysis of Reliability, Validity, and Responsivity Annual PI and Stat Team Review and Revisions Annual Advisory Group
RA = Research Associate; BRC CS = Blind Rehabilitation Center Clinical Staff; PI = Principal Investigator.
A Research associate (RA) from the Decatur Rehabilitation R&D Center phones the client at his or her home within a month of discharge to administer the BRSSatSur, the BRSFOutSur post-rehabilitation, and a repeat of the BRSFOutSur, in which the client is asked to respond as if the survey were taking place prior to his or her recent rehabilitation experience. This second "projected pre-rehab" administration is undertaken to ascertain change in function, while controlling for a shift in response bias brought about by the rehabilitation experience. A review of the literature in the evaluation of training programs revealed that the retrospective pretest was necessary in order to account for this response shift (11). The Decatur Rehab R&D Center will also administer the 6-mo follow-up BRSFOutSur via telephone to approximately 5 percent of the clients. All interviews are conducted in English or Spanish as appropriate, and information obtained will be coded directly into computers during the interview process.
These data are then analyzed and the results distributed to VAHQ and the individual VA facilities in summary statistical form on a quarterly basis for ongoing quality assurance applications. Nonveteran participants will follow an identical protocol, with NAC functioning in the role of VAHQ, and the participating nonVA agencies operating in a fashion similar to that of the individual VA facilities.
In addition to data obtained from participants, home visits are being paid to approximately 5 percent (50 per year or 150 total) of the subjects to videotape them doing tasks that they indicated on the BRSFOutSur they could do independently. Subjects are selected using a convenience sample from location clusters in order to increase the efficiency of a limited travel budget. The resultant tapes are rated by rehabilitation professionals to assess the functional independence of the participant in the performance of the specified tasks.
Descriptive summaries of the data are provided to participating centers on a quarterly basis. All psychometric analyses will be undertaken on an annual basis in an iterative process. Refinements suggested by the analysis of the prior year's data being subjected to psychometric analysis will be implemented in the subsequent year.
Question 1: What is the reliability and validity of the BRSFOutSur?
A reliable test is an accurate test, one that is free of error and provides repeatable, consistent results. Reliability will be measured in two ways. First, the internal consistency of the instrument will be derived from the computation of coefficient alpha. This is a measure of the extent to which the various test items tend to be related to one another. Coefficient alpha can be shown to be equal to the mean split-half reliability coefficient that would result from all possible "splits" of an instrument and is a conservative estimate of the reliability of an instrument. Second, the level of repeatability of the instrument will be derived through the assessment of test-retest reliability. On a sample of 5 percent of the VA participants, project staff will compare the postrehabilitation administration and readministration of the BRSFOutSur 1 week later and derive the I-Class correlation for test-retest. To assess inter-observer reliability, 5 percent of the telephone BRSFOutSur-Post interviews will be audiotaped and scored by an alternate RA. In order to assess intra-observer reliability, 5 percent of the taped telephone BRSOutSur-Post interviews will be scored by the same RA who did the original interview. Once again, I-Class correlations will be used as the coefficient of equivalence.
A valid test is a useful test, one that fulfills the purposes that prompted its administration. The content validity of a test reflects the extent to which it adequately samples the domains of information that it purports to measure. Content validity is not statistically determined, but rather is built into a test as it is constructed through a plan that specifies content and methods for selecting test items. The BRSFOutSur was developed by the BRSTGO in association with all of the skill areas in blind rehabilitation. Individuals with visual impairments were involved during all stages of its development. It was based on 8 years of work in this area by associated professionals and has undergone intensive scrutiny and has been thoroughly critiqued throughout its development. This provides evidence of good content validity.
Construct validity of the BRSFOutSur will be examined through a multitrait/multimethod matrix (13). This involves the assessment of empirical relationships between the various BRSFOutSur items to test for high correlation between assessments measuring the same or similar constructs and low correlation between tests that measure different constructs (i.e., discriminant validity). It is expected, for example, that measures of domain function among subjects (such as frequency or difficulty of travel in different settings or frequency or difficulty of different visual tasks) will be at least moderately correlated. Additional relationships among the variables being evaluated will be determined by the advisory group and tested for correlation as a means of establishing construct validity. In addition, validity of the instrument will also be explored by the use of factor analytic techniques, which permit investigation of the factor structure of the instruments as a whole. It is expected that items that cluster logically, such as those related to orientation and mobility, will cluster on the confirmatory factor analysis as well.
Criterion validity reflects the relationship between test scores and performance on a criterion measure, a "gold standard." No such measure exists. However, project staff will compare the functional assessments obtained through use of the BRSFOutSur with the results obtained from the functional assessments made by rehabilitation professionals observing videotapes of the subjects at home demonstrating tasks (listed as frequently and successfully done in the BRSFOutSur) in a sample of 5 percent of the cases. These subjects used in this analysis are selected through a convenience sampling procedure in a quasi-random fashion, based on geographical clustering. Once again, an I-Class correlation will be used as a coefficient of equivalence to assess agreement between the two data sources. (This methodology resulted in 98.7 percent agreement between survey report and expert assessment in an earlier project, "A national survey of the impact of low vision prosthetic device use among veterans," conducted at this center.)
All measures of reliability and validity of the instrument will be assessed on an annual basis. For the correlation-based measures, a sample size of 1,000 with an alpha level of 0.05 will provide a measure sensitive to a small effect size of 0.09 with a power of 0.81. The subsamples of 50 used for test-retest, inter-observer, and intra-observer reliability and criterion validity will provide sensitivity-to-effect sizes of 0.28 at a power of 0.80. The information gained from these assessments will be shared with the advisory group. Analogous analyses will be used for the nonVA participants.
Question 2: What is the responsiveness of the BRSFOutSur to changes associated with blind rehabilitation?
The usefulness of any instrument is gauged by its demonstrated reliability and validity. However, in the case of an evaluative instrument, reliability and validity are necessary but not sufficient. An instrument used to compare pre/post intervention change scores must also have demonstrated responsiveness, that is, the ability to detect a difference when one exists.
A repeated measures analysis of co-variance (ANCOVA) with pre/post change scores on the BRSFOutSur (Post-Rehab and Projected Pre-Rehab) as the dependent variable and pre-rehabilitation score as the primary covariant will be employed to assess change in the domains surveyed by BRSFOutSur. Elements representing significant change associated with blind rehabilitation will be considered to have responsiveness to these changes. Risk correction factors from the NBRPDB will also be used as covariants in exploratory analyses of the data. With a sample size of 1,000 and an alpha level of 0.05 (Bonferroni adjusted to 0.001 for multiple tests), this analysis will be sensitive to a small effect size of 0.14 with a power of 0.862. These analyses will be undertaken on an annual basis with the results being shared with the advisory group.
Analysis of variance (ANOVA) comparing the veteran and nonveteran samples will be employed to examine differences in responsiveness to types of rehabilitation programs. With a sample size of 1,000 in each group per year and an alpha level of 0.05 (Bonferroni adjusted to 0.001 for multiple tests), this analysis will be sensitive to a small effect size of 0.08 with a power of 0.848. An analysis of samples of up to three subgroup service delivery types, such as 1) intensive, in-house residential rehabilitation taking 2 more months similar to VA BRC programs; 2) intensive, center-based outpatient rehabilitation similar to VA VICTORS programs; and 3) out-patient home based rehabilitation similar to VA BROS programs, with 100 clients balanced for risk factors per site will allow sensitivity to a medium effect size of 0.18 with a power of 0.809.
Question 3: What is the relationship of client characteristics to rehabilitation outcomes?
Subject descriptors such as clinical measures of sensory performance and existence of comorbidities contained in the BRSDBase will be assessed for relationship to change in function as measured by the BRSFOutSur. Multivariate models will be evaluated and used as the basis of developing risk adjustment equations. Changes in client status condition between administrations are monitored and assessed for impact on function.
Question 4: How should the BRSFOutSur and associated scaling and scoring protocols be refined to improve reliability, validity, and responsiveness?
Problems uncovered in the reliability, validity, and responsiveness assessment are addressed through test revisions. Item analysis and exploratory factor analysis is undertaken to determine the relative contribution of individual items to the overall test result. Items with high intercorrelations are evaluated for elimination in order to shorten the test. The project statistical staff will perform Rasch analysis, scoring and scaling, (including trans-response interactions), at the culmination of the project. The BRSTGO will then evaluate the skill domains for adequate breadth and depth of item sampling, especially in terms of responsiveness to change resulting from the rehabilitation process. Representatives from NAC will assist in this process in order to assure a consistent set of items for the nonVA version of the instrument. A representative from the Blinded Veterans of America (BVA) will also serve in this advisory group to provide consumer input to the instrument development. Revisions of the instrument, scaling, and scoring protocols will take place on an annual basis if determined necessary by the advisory group.
Because of the modifications required to the VA data collection instruments, the collection of data from the nonVA participants started approximately 12 mo after the start of VA data collection. Therefore, the results reported at this time are based primarily on data from the VA participants. Overall results based on the VA portion of the data set are shown in Appendices A, B, and C.
The reliability of the primary survey inventory BRSFOutSur is being measured in two ways. First, the internal consistency of the instrument is periodically tested from the computation of coefficient alpha. Reliability measures using Cronbach's alpha revealed an extremely high level of internal consistency for BRSFOurSur items' frequency, independence, and satisfaction.
Reliability Analysis: Cronbach's
Frequency 0.864 Independence 0.933 Satisfaction 0.936
Second, reliability measures examining the level of repeatability of the instrument were derived through the assessment of test reliability based upon inter-rater coefficient of equivalence. This measure also revealed an extremely high level of internal consistency for BRSFOutSur items' frequency, independence, and satisfaction.
Reliability Analysis: Inter Rater
Frequency 0.956 Independence 0.875 Satisfaction 0.800
The agreement between BRSFOutSur results and expert rated assessment of function through videotapes of client's task performance is being assessed. Although preliminary, the results are very positive.
Content Results to Date
BRSDB--Blind Rehabilitation Service Database
Project staff have obtained demographic data on 2,624 visually impaired veterans who have taken part in VA blind rehabilitation programs during the 18 mo of this project. These data form the largest and most detailed data set currently available on veterans served by the BRS of the VA. The average age of these veterans is 67.2 (SD=12.6) years. As expected, gender is heavily biased towards male (96 percent). Race is predominantly Caucasian (71.1 percent), with 19 percent African American, 8 percent Hispanic, 1.1 percent Native American, and 0.2 percent Asian. Subject living situation is shown to primarily be residing with a spouse (42 percent), alone (28 percent) or other family member (24 percent). Similarly, place of residence tends to be a private residence such as a house, apartment, condominium (93 percent), with only 3 percent residing in an institutional setting such as nursing home, domicillary or state veterans Home.
Following 8 mo of data collection, demographic information has been obtained for 245 nonveteran visually impaired subjects. Of this number, the average age is 70.1 (SD=18.8). Unlike the veteran population, the majority is 66 percent female. Nearly 91 percent are Caucasian, with 7.3 percent African-American, 0.5 percent Hispanic or Latino, 1 percent Native American, with no Asian subjects. A review of their living situations indicates that 28 percent reside with a spouse, 51.2 percent live alone, 17 percent live with family, and 1 percent live with a friend or roommate. For 81.2 percent, the place of residence is a private residence; 10.6 percent live in a retirement facility or similar institutional setting.
Findings examining mobility environment, eye diagnosis, visual acuity and visual field, physical and cognitive involvement, as well as a host of other diagnostic information, are gathered. There is insufficient space in this article to provide comprehensive details of this demographic data set.
BRSSS--Blind Rehabilitation Service Satisfaction Survey
Through the use of this instrument, data have been gathered from 1,655 veterans about satisfaction with programmatic elements of the blind rehabilitation process. As mentioned above, the information derived from this instrument is currently being used at both Veterans Integrated Service Network (VISN), and VAHQ levels to evaluate satisfaction levels of veterans attending blind rehabilitation programs. To date, 1,552 of these veterans (93.7 percent) express either "satisfaction" or "complete satisfaction" in response to this question.
BRSFOutSur--Blind Rehabilitation Service Functional Outcomes Survey
To date, outcomes data (post-rehab) on 1,637 veterans, outcomes data (retrospective pre-rehab) on 511 veterans, and outcomes data (6-mo follow-up) on 198 veterans have been developed, revealing several very important preliminary findings. First, investigators have noted that virtually all average changes associated with the rehabilitation experience are in a positive direction. Specifically, other than a slight decrease in the frequency of task 10 (Since discharge from the blind rehabilitation program, have you walked in your house without difficulty?), all tasks have shown improvement in terms of frequency, independence, and satisfaction. Second, although the most objective item of the multidimensional response set, frequency of accomplishing a specific task, does not always show a major positive change, two other items, self-perceived level of independence and satisfaction with performance of task, generally show larger positive changes (see Figure 1). For example, given the relatively advanced age of this population, that a task such as performing outdoor tasks remains a low frequency event with only a 5.6 percent increase is not surprising. However, the self-reported assessment that the task could be accomplished in a more independent (18.2 percent) and satisfactory (20.5 percent) manner after rehabilitation is very pronounced.
Frequency and self-perceived levels of independence and satisfaction with performance of task.
Due to the large amount of data available and the preliminary nature of the analyses, representative findings from two functional skill area domains, low vision and orientation and mobility, will be examined using data from the VA portion of the data set. It is important to point out that none of these findings are final results, but rather indications of emerging patterns developing in the data set.
Item 3. Since discharge from the blind rehabilitation program, have you performed outdoor household chores such as taking out the trash, or raking the lawn?
With the VA subjects, modest gains in frequency for item 3 were shown from pre- to post-rehabilitation (76.8 to 83.3 percent performing task daily or weekly). However, 18.2 percent of respondents indicated higher levels of perceived independence post rehabilitation (mean=0.25), and 20.5 percent showed higher levels of satisfaction with their ability to perform this task post-rehabilitation (mean=0.22).
Data from the BRSFOutSurN for nonVA subjects indicate that of the 41 people interviewed, gains in frequency for item 3 were shown from pre- to post-rehabilitation (81.4 to 88.3 percent performing task daily or weekly). Furthermore, 9.3 percent indicated higher levels of satisfaction with their ability to perform this task post-rehabilitation (mean=0.16).
Item 7. Since discharge from the blind rehabilitation program, have you read mail such as letter or bills?
Among VA respondents, the number doing this on a daily basis increased from 43.3 to 70.1 percent pre- and post-rehabilitation (mean=1.31). Perceived levels of independence increased after rehabilitation in 48.1 percent (mean=1.05); levels of satisfaction increased in 39.4 percent (mean=0.77). Furthermore, levels of frequency, independence, and satisfaction with this item remained very stable 6 mo after rehabilitation.
Findings for nonVA subjects indicate that while frequency of performing item 7 declined somewhat from pre- to post-rehabilitation, perceived levels of independence increased after rehabilitation in 9.9 percent of respondents (mean=0.09), and levels of satisfaction increased after rehabilitation in 28 percent of respondents (mean=0.21).
Item 13. Since discharge from the blind rehabilitation program, have you cut food using a knife and fork?
Prior to rehabilitation, 86.4 percent of VA respondents reported doing this either daily or weekly; after it, 92.5 percent did (mean=0.25). While the gains in frequency of this task improved measurably from pre to post-rehab, even greater gains in perceived levels of independence were noted, from 70.3 to 86.7 percent (mean=0.24), a gain of 16.3 percent. Even more pronounced was the change in level of perceived satisfaction with performing this task, from 71.3 to 96.4 percent saying they felt satisfied or very satisfied with their abilities to accomplish this task.
Findings for the nonVA sample reveal a very modest (92.8 to 92.9 percent) change from pre to post-rehab in frequency of task, (mean=0.02). Some improvement in the perceived level of independence was noted, from 90.5 to 95.2 percent felt they could accomplish this task independently. The level of perceived satisfaction in this group showed a clear impact from rehabilitation, from 85.7 percent reporting satisfaction pre- to 100 percent reporting satisfaction post-rehabilitation.
Item 14. Since discharge from the blind rehabilitation program, have you familiarized yourself to a new area?
The rehabilitation increased the number of VA respondents familiarizing themselves to a new area either weekly or daily from 12 to 18.7 percent (mean=0.25). Moreover, significantly higher levels of satisfaction (from 54.7 to 86.2 percent) with abilities to accomplish this task were shown after rehabilitation.
The nonVA group experienced a decrease from 21.4 to 14.3 percent in the level of frequency of task accomplishment for this item daily or weekly. However, findings reveal a gain of 11.9 percent in independence levels from pre- to post-rehab, as well as an 11 percent gain in satisfaction with abilities to carry out this task.
Item 17. Since discharge from the blind rehabilitation program, have you read a magazine, newspaper article or book?
Prior to rehabilitation, 44 percent of VA respondents said that they performed this task daily or weekly; afterward, 72.2 percent said they did, a gain of 28.2 percent in frequency of task performance (mean=1.00). Independence levels with task completion increased as well, from 51.9 to 81.8 percent, along with levels of satisfaction (49.1 to 82.9 percent).
Findings for non-VA subjects indicate that frequency of task accomplishment declined 23.9 percent following rehabilitation from 38.1 percent to 14.2 percent, as did perceived levels of independence from 66.7 to 59.5 percent feeling they could accomplish this task independently. Despite these losses, these subjects reported higher levels of satisfaction with abilities to accomplish this task, from 54.7 to 86.2 percent feeling satisfied or very satisfied with their abilities to read a magazine, newspaper article or book after rehabilitation.
PRELIMINARY CONCLUSIONSWhile it is premature to discuss conclusions about a study in progress, we can posit a few observations.
- The current revision of the BRSFoutSur instrument (28 item) can be administered in approximately 15 minutes. This can be taken to indicate that it meets the basic criterion of feasibility of administration.
- The internal consistency and inter-rater reliability of the BRSFOutSur have been shown to exist at acceptable levels.
- Clinical involvement in test construction, preliminary home visits and confirmatory factor analytic results all support the validity of the BRSFOutSur survey.
- Although some of the gains are modest, all but one of the frequency items and all of the independence and satisfaction items show positive change associated with the rehabilitation experience. This indicates that the instrument is responsive to rehabilitation-related change.
- Information gained from the BRSSatSur indicates that clients express a very high level of satisfaction with the overall programatic quality of the services that they have received.
- Barrett, Stephen S. Measuring outcomes of services to people with disabilities: an annotated bibliography. New York: The National Accreditation Council for Agencies Serving the Blind and Visually Handicapped; 1996.
- Keith RA. Conceptual basis of outcome measures. Am J Phys Med Rehabil 1995;74:73-80.
- Studenski S, Duncan PW. Measuring rehabilitation outcomes. Clin Geriatr Med 1993;9:823-9.
- Johnston MV, Granger CV. Outcomes research in medical rehabilitation: a primer and introduction to a series. Am J Phys Med Rehabil 1994;73:296-303.
- Kilgore KM. Measuring outcomes in post-acute continuum. Arch Phys Med Rehabil 1995;76:SC21-6.
- Bummel-Smith K. Research in rehabilitation. Clin Geriatr Med 1993;9:895-904.
- Keith RA. Functional status and health status. Arch Phys Med Rehabil 1994;75:478-83.
- Fuhrer MJ. Conference report: an agenda for medical rehabilitation outcomes research. Am J Phys Med Rehabil 1995;74:243-8.
- Granger CJ. A conceptual model for functional assessment. In: Granger CV, Gresham GE, editors. Functional assessment in rehabilitation medicine. Baltimore: Williams & Wilkins; 1984.
- Nieuwenhuijsen ER, Frey WD, Crews JE. Measuring small gains using the ICIDH Severity of Disability Scale: assessment practices among older people who are blind. Int Disabil Stud 1991;13:29-33.
- Long RG, Crews JE. Final report: functional independence measure for blind adults. Decatur, GA: Rehabilitation Research and Development Center, VA Medical Center, Atlanta, GA; 1995.
- Babcock JL. Assessing the utility of retrospective pretests. Presented at the American Evaluation Association Annual Meeting. Boston, MA; 1994.
- Campbell DT, Fiske DW. Convergent and discriminant validation by the multitrait-multimethod matrix. Psych Bull 1959;56:81-105.
Go to TOP
Last Updated Tuesday, May 29, 2007 10:26 AM