Journal of Rehabilitation Research & Development (JRRD)

Quick Links

  • Health Programs
  • Protect your health
  • Learn more: A-Z Health
Veterans Crisis Line Badge
 

Volume 48 Number 1, 2011
   Pages 21 — 30

Validity of PTSD diagnoses in VA administrative data: Comparison of VA administrative PTSD diagnoses to self-reported PTSD Checklist scores

Amy A. Gravely, MA;1* Andrea Cutting, MA;1 Sean Nugent, BA;1 Joseph Grill, MS;1 Kathleen Carlson, PhD;1-2 Michele Spoont, PhD1,3-4

1Minneapolis Department of Veterans Affairs Medical Center, Center for Chronic Disease Outcomes Research, Minneapolis, MN; Departments of 2Medicine and 3Psychology, University of Minnesota, Twin Cities Campus, Minneapolis, MN; 4Department of Psychiatry, University of Minnesota, Minneapolis, MN

Abstract — Little research has been done on the validity of posttraumatic stress disorder (PTSD) diagnoses that are found in Department of Veterans Affairs (VA) administrative data, even though they are often used in VA research. We compared PTSD diagnoses found in VA administrative data with PTSD Checklist (PCL) scores self-reported by 4,777 newly diagnosed participants in a national postal survey study. Using PCL scores of at least 50 as the gold standard, we compared positive predictive values (PPVs) for at least one versus at least two PTSD diagnoses (found within 4 months of the first) in VA administrative data overall and by subgroups of interest: age, sex, and clinic where first diagnosed. The overall PPV was 75% for at least one PTSD diagnosis and 82% for at least two PTSD diagnoses. Similarly, the PPV significantly increased for all subgroup analyses when at least two PTSD diagnoses were used. The increase in PPV was greatest for those first diagnosed in primary care and for those older than 65. To select a sample of veterans with more definitive PTSD from administrative data, researchers should select those veterans with at least two PTSD diagnoses as opposed to at least one.

Key words: administrative data, algorithm, electronic medical record, new episode, PCL, PTSD, PTSD Checklist, PTSD diagnosis, PPV, sample, sample inclusion criteria, validity, Veterans Health Administration.

Abbreviations: CAPS = Clinician-Administered PTSD Scale, CI = confidence interval, DSM = Diagnostic and Statistical Manual of Mental Disorders, HSR&D = Health Services Research and Development, ICD-9 = International Classification of Diseases-9th Revision, NPCD = National Patient Care Database, OIF/OEF = Operation Iraqi Freedom/Operation Enduring Freedom, PCL = PTSD Checklist, PPV = positive predictive value, PTSD = posttraumatic stress disorder, SCID = Structured Clinical Interview for DSM, VA = Department of Veterans Affairs.
*Address all correspondence to Amy A. Gravely, MA; Minneapolis VAMC, Center for Chronic Disease Outcomes Research, 1 Veterans Drive, Minneapolis, MN 55417; 612-467-5208; fax: 612-727-5699. Email: amy.gravely@va.gov
DOI:10.1682/JRRD.2009.08.0116
INTRODUCTION

The Department of Veterans Affairs (VA) has one of the largest repositories of electronically collected, administrative, national healthcare data. Researchers often use this administrative data to identify patient populations for research studies. International Classification of Diseases-9th Revision (ICD-9) diagnosis codes are often used for this purpose. ICD-9 codes are entered into the local computerized patient record system to indicate a problem treated at either an outpatient encounter or during an inpatient episode of care. These data are compiled nationally and made available to researchers by the Austin Information Technology Center. Using these codes from administrative data offers many advantages to researchers, including cost-effectiveness and placing minimal or no burden on potential research participants.

Posttraumatic stress disorder (PTSD) is an important research focus within the VA. Researchers use ICD-9 codes from VA administrative data to identify populations of veterans with PTSD for inclusion in research studies. To date, little investigation has been done on the validity of PTSD diagnosis codes that are found in VA administrative data, despite the fact that they are often used for research purposes. The validity of these codes could be diminished by data entry errors, diagnostic inaccuracy, or various other factors that lead to providers incorrectly entering or not entering PTSD diagnoses into patients' records. Our overall goal is to add to the sparse, almost nonexistent, literature on the validity of PTSD diagnoses in VA administrative data. We will evaluate the validity of two PTSD administrative data algorithms by comparing them with self-reported PTSD Checklist (PCL) scores collected through a national survey. Additionally, we will look at the decrease in sample size that results from using the more restrictive algorithm and the variations in validity and sample size by important subgroups of interest: age, sex, and clinic where first diagnosed with PTSD. This information will aid researchers in identifying study samples in which a greater percentage of participants are likely to actually have PTSD. In addition, researchers will have more information with which to make decisions about using administrative data to select PTSD samples for research studies in that they will know with more accuracy the actual composition of their sample. Ultimately, these contributions will lead to more accurate inferences from VA PTSD research studies that use administrative data to select samples, which could then lead to more effective interventions for PTSD. The investigation of the validity of PTSD diagnosis codes is essential to using these codes in VA research.

BACKGROUND

Many examples exist of studies in which researchers use PTSD diagnoses from VA administrative data to obtain study samples. In one study, Mohamed and Rosenheck used PTSD diagnoses from VA administrative data to identify a population of patients with PTSD to examine prescription behavior [1]. As their sample inclusion criteria, they included all VA patients who had at least one PTSD diagnosis in administrative data in fiscal year 2004. Rosenheck and Fontana used PTSD diagnoses from VA administrative data from 180 days before and 180 days after September 11, 2001, to examine whether the use of mental health services changed during that time frame for people diagnosed with PTSD [2]. They also used the criterion of finding at least one PTSD diagnosis in administrative records. These are just two examples.

Although several studies have used VA administrative data to select veteran samples with PTSD, to our knowledge, only two studies have examined the validity of PTSD diagnosis codes in administrative data. The first, by Magruder and Yeager, used a sample of VA primary care patients to compare ICD-9 codes in administrative data against diagnostic interview data from the Clinician-Administered PTSD Scale (CAPS) [3]. They found that only 43 percent of those diagnosed with PTSD based on the CAPS carried a PTSD ICD-9 code in their medical record within 2 years of CAPS administration, suggesting that among primary care patients, PTSD would be under-identified on the basis of administrative data alone. Examination of subgroups revealed that providers were more apt to recognize PTSD and thus enter a PTSD diagnosis code into administrative data for middle aged (potentially Vietnam) veterans. The second study, by Frayne et al., compared various administrative data algorithms to self-reported lifetime PTSD status in a national cohort of veterans with diabetes [4]. The algorithms included using at least one versus at least two PTSD diagnoses codes in any VA record (outpatient or inpatient) within a 2-year period and comparing at least one diagnosis from mental health outpatient visits versus primary care outpatient visits versus any outpatient visit within that same 2-year period. They found that the positive predictive values (PPVs) were higher when using at least two instances of PTSD in either outpatient or inpatient administrative databases (90%) versus using at least one instance (82%). When only outpatient databases and only one instance of PTSD were used as the criteria, the PPV was higher for diagnoses associated with primary care visits (88%) than for mental health care visits (85%) or any ambulatory care visit (82%).

This study is a secondary analysis of data collected for a large national survey study.1 This large survey study's main purpose was to identify factors associated with participation in mental health treatment among veterans who were newly diagnosed with PTSD. A national sample of veterans with at least one PTSD diagnosis (ICD-9 code 309.81) during the recruitment period was identified through VA administrative data. Veterans were excluded if they had severe cognitive impairments, received antidepressants, had any mental health care (except chemical dependency), or had any PTSD-related visits in the prior year. The sample was identified through VA administrative data, specifically through the outpatient encounter files from the National Patient Care Database (NPCD). These files are updated every 2 weeks and made available to researchers; therefore, new PTSD cases were identified every 2 weeks. For every new PTSD case that was identified, a mailed survey was administered immediately. Service utilization data and prescription data were then extracted for the following 6 months in order to study participation in mental health treatment. From this larger survey study of people with at least one new PTSD diagnosis, we used administrative data to look for a second PTSD diagnosis; we also took PCL scores from the mailed survey and various demographic variables from both the survey and administrative data. The large survey study was not fully complete at the time of publication of this article.

Studies published to date have not compared the use of PTSD ICD-9 diagnosis codes from VA administrative data against the PCL. Although the PCL is not the gold standard in diagnosing PTSD, it has excellent psychometric properties and diagnostic utility [5-7].

Only two studies have been done on the validity of PTSD diagnoses found in VA administrative data. Although one study did compare different algorithms and included various subgroups, it was potentially limited in its use of self-reported lifetime status of PTSD as its gold standard. The other study included primary care patients only and did not compare different algorithms. Neither of these studies statistically compared the validity of different algorithms. Because both studies found variation in diagnostic validity among subgroups of patients [3-4], we also examined validity within subgroups of interest: age, sex, and clinic where first diagnosed. These results may be particularly helpful to researchers wanting to study PTSD in specific care settings, such as primary care [3,8] or mental health [9]. In addition, high priority areas for the VA in 2009 included studying the needs of female veterans [10] and Operation Iraqi Freedom/Operation Enduring Freedom (OIF/OEF) veterans [11]. Although age is not a direct proxy for service era, it can potentially offer some insight into the validity of PTSD diagnoses from administrative data for this group.

METHODS
Study Design

The first administrative data algorithm (and less restrictive criterion), which involved selecting an individual with at least one PTSD diagnosis in administrative data, was satisfied by all 4,777 veterans in our sample because this was the condition for participating in the national survey study. The second, more restrictive, criterion was used to select individuals among the sample of 4,777 veterans for whom at least two PTSD diagnoses were found in administrative data within a 4-month period of the first diagnosis. This was the only time frame possible given that the study was recruiting patients every 2 weeks; the most recently recruited group had 4 months of data available at the time these analyses were conducted. To check the validity of this time frame, we examined participants who were recruited first for whom we had 6 months worth of follow-up data and found that in 100 percent of cases in which a second diagnosis had been assigned, this diagnosis was given within 2 months of the first one. Therefore, this 4-month window appears to be adequate to determine rates of subsequent PTSD diagnoses. The 4-month window was used for all participants.

Study Sample

This study includes only veterans who had at least one new PTSD diagnosis (ICD-9 code 309.81) in VA administrative data between January 14, 2008, and January 24, 2008 (pilot study) or between May 12, 2008, and January 22, 2009 (main study). Together, there were approximately 30 weeks of total recruitment. Every 2 weeks an average of 1,822 individuals were identified; thus, a total of 27,330 patients were found to be eligible based on the study inclusion/exclusion criteria. Of the people identified every 2 weeks, an average of 571 people were sampled or invited to participate in the survey; the total number of invitations sent was 8,562. Of those, 6 people were deceased and 599 surveys were undeliverable, leaving 7,957 people who received a survey. Questionnaires were sent to each of these veterans, of which 5,207 (65%) completed and returned them. Of our responders, 4,777 (92%) completed all 17 items of the PCL. Our analyses were therefore based on survey and administrative data for these 4,777 veterans.

Data Sources
Administrative Data

The outpatient encounter data from the NPCD was used for sample selection as well as to search for a second PTSD diagnosis code within 4 months of the first. Additionally, clinic type, sex, and age were also taken from this administrative data source. All demographic variables used for responder-nonresponder analysis were taken from administrative data; they include race/ethnicity, sex, marital status, age, OIF/OEF status, service connection for PTSD status, general service connection status, and census region. All administrative variables were extracted at the time that the patient was determined eligible for the study. Clinic type where first diagnosed was derived from stop code information and was categorized into three categories: primary care, non-PTSD mental health clinic, and PTSD clinics. New PTSD diagnoses made in other clinics were excluded from this study. Age was categorized into three categories: <45, 45-64, and Š65.

Survey Data

PCL scores and education level were taken from the survey. Additionally, marital status and race/ethnicity were taken from the survey and supplemented with administrative data where missing. Multiple mailings were used in accordance with Dillman's methods [12].

PCL

The PCL is a self-report questionnaire consisting of 17 Diagnostic and Statistical Manual of Mental Disorders (DSM)-Fourth Edition PTSD symptoms. They are rated on a five-point scale ranging from "not at all" to "extremely." Items are added to obtain a total score. The higher the score, the more symptoms are present. The PCL is useful in a variety of screening and clinical contexts, especially when administration of a structured interview is not feasible. It can be used as a continuous measure of PTSD symptoms or it can be used to derive a PTSD diagnosis [13-14]. A cutoff score of 50 was used for this analysis to indicate PTSD status because it is the most commonly used cut-off score in military and vet-eran populations [15] and because it has been shown to be a good indicator of PTSD status when compared with both the CAPS [13] and the Structured Clinical Interview for DSM (SCID) [14], which are two gold standards for diagnosing PTSD in the field.

Statistical Analysis

To measure the validity of PTSD administrative data algorithms, we calculated the proportion of veterans with PCL scores of Š50 for those who had at least one diagnosis and for those who had at least two for the entire sample and by subgroups of interest: age, sex, and clinic where first diagnosed with PTSD. These proportions are also the PPVs of having a PCL score of Š50. We were unable to calculate negative predictive values, sensitivity, and specificity because all of the people in the sample had at least one PTSD diagnosis in VA administrative data.

Traditional statistical methods are not appropriate when comparing the PPV for at least one PTSD diagnosis with the PPV of at least two PTSD diagnoses because the latter are a subset of the former, creating data dependency. Therefore, in order to compare these data, we used bootstrapping methods to assess the statistical significance of the difference between the two PPVs [16-18]. We also bootstrapped the estimated decrease in sample size that resulted from using the more restrictive algorithm (at least two diagnoses) as opposed to the less restrictive algorithm (at least one diagnosis). We did this for the entire sample and for all subgroups of interest.

Finally, we used bootstrapping to determine whether sex, age, or clinic where first diagnosed separately modified the effect of the number of administrative diagnoses on the PPVs. For sex, for example, we took the ratio of the PPV for at least two diagnoses and divided it by the PPV for at least one diagnosis. We then compared the ratio for females to the ratio for males using bootstrapping techniques. Similar techniques are described by Cole [19]. If the confidence interval (CI) for these ratios of ratios did not include one, then a significant moderating effect was present; in other words, the effect of number of diagnoses on the PPV for PCL scores of Š50 was significantly different between the groups for the variable in question. We used SAS version 9.1 (SAS Institute, Inc; Cary, North Carolina) for all analyses. For all bootstrap resampling, we took 2,000 resamples, each of size 4,777, that were randomly selected with replacement from the original sample. For all analyses, the statistical significance was defined as 0.05.

To determine whether nonresponse bias affected the results, we also calculated all PPVs after adjusting for potential nonresponse bias (results not shown). Responder- nonresponder analysis compared those who answered all of the PCL questions with those who did not. To adjust for potential nonresponse bias, we used propensity scores. Specifically, we constructed response propensities and produced a weighted combination of within-propensity class estimates [20]. All variables that we had for both responders and nonresponders were used. The responders who had one or more missing items on the PCL were a very small group; they were 5 percent of all participants. As a comparison, nonresponders were 35 percent of all participants and responders who had no missing items on the PCL were 60 percent of all participants. Therefore, it is reasonable to assume that adjusting for nonresponse on the PCL specifically using multiple imputation would make little difference in the validity of our results.

RESULTS
Responder-Nonresponder

Responders and nonresponders differed on marital status (55% of responders were married vs 47% of nonresponders), OIF/OEF status (27% of responders were OIF/OEF vs 40% of nonresponders), and age (36% of responders were < 45 vs 50% of nonresponders). Of note, responders and nonresponders were not different in terms of having a second PTSD diagnosis found in administrative data after the initial, new diagnosis (60% of responders vs 58% of nonresponders). Most of the differences between the adjusted and unadjusted (observed) PPVs were 1 percentage point, and the largest difference was 3 percentage points. The results were not appreciably altered by adjustment; therefore, the nonadjusted results are shown.

Main Results

Demographic characteristics for at least one and at least two PTSD diagnoses in administrative data are shown in Table 1. Of particular interest is the distribution of PCL scores for at least one versus at least two PTSD diagnoses. More veterans are in the higher ranges for at least two diagnoses. As can be seen in Table 2, for both the entire sample and for all subgroups, using at least two diagnoses instead of at least one increased the PPV of PCL scores of Š50 and decreased the available sample size. Notice also that when using at least one diagnosis as the selection criterion, 75 percent of the overall sample will have PCL scores of Š50, with those first diagnosed in primary care having a lower PPV (69%) and those aged 65 and older having an even lower PPV (61%). Those first diagnosed in non-PTSD mental health clinics have the highest PPV (80%).


Table 1.
Demographic characteristics for those with ≥1 and ≥2 posttraumatic stress disorder (PTSD) diagnoses in Department of Veterans Affairs administrative data. Data presented as percent.

Characteristic
≥1 PTSD Diagnosis
(n = 4,777)
≥2 PTSD Diagnoses
(n = 2,866)

Male
85
83
Some College or More
69
69
Marital Status
 
Married
61
61
 
Separated/Divorced
21
22
 
Single
13
14
 
Widowed
3
2
Race/Ethnicity
 
Caucasian
42
42
 
African American
17
19
 
Asian/Pacific Islander
5
5
 
Native American/Native Alaskan
5
4
 
Latino/Hispanic
18
18
 
>One Race
13
13
Age (yr)
 
< 45
36
41
 
45-64
52
50
 
Š65
12
9
Clinic Where First Diagnosed with PTSD
   
 
Primary Care
44
32
 
Non-PTSD Mental Health Clinic
44
54
 
PTSD Clinic
11
14
PTSD Checklist Score
   
 
<30
4
2
 
30-39
7
4
 
40-49
13
12
 
50-59
22
22
 
60-69
28
31
 
Š70
25
29






Table 2.
Positive predictive values (PPVs) for ≥1 and ≥2 posttraumatic stress disorder (PTSD) diagnoses in Department of Veterans Affairs administrative data for PTSD Checklist scores of ≥50.

 
Variable
≥1 PTSD Diagnosis
≥2 PTSD Diagnoses

All
Total n
PPV
4,777
2,866
74.8
81.8
Female
Total n
PPV
731
487
72.6
80.3
Male
Total n
PPV
4,046
2,379
75.2
82.1
Age <45
Total n
PPV
1,724
1,186
75.2
81.2
Age 45-64
Total n
PPV
2,478
1,421
77.7
83.4
Age ≥65
Total n
PPV
575
259
61.4
76.1
Primary Care Clinic
Total n
PPV
2,071
914
69.3
78.7
Non-PTSD Mental Health Clinic
Total n
PPV
2,122
1,537
79.9
84.1
PTSD Clinic
Total n
PPV
540
399
77.6
81.0


As can be seen in Table 3, all of the increases in PPV values that were found when at least two PTSD diagnoses in VA administrative data as opposed to at least one were used were statistically significant for the entire sample and for all subgroups. The largest increases in PPV were for those Š65 and those diagnosed in primary care clinics. Not surprisingly, using at least two PTSD diagnoses as opposed to at least one also significantly decreased the sample sizes, with the largest decreases in those veterans Š65 and those first diagnosed in a primary care clinic.


Table 3.
Bootstrapped statistical comparisons between ≥1 and ≥2 posttraumatic stress disorder (PTSD) diagnoses in Department of Veterans Affairs administrative data.

Characteristic
PPV Difference: ≥1 vs ≥2 PTSD Diagnoses
95% CI
% Decrease in Sample Size from ≥1 to ≥2
95% CI

All
7.0
6.0-8.0
40.0
38.7-41.5
Female
7.6
5.1-10.3
33.4
29.9-36.9
Male
6.9
5.7-8.0
41.2
39.7-42.8
Age <45
6.0
4.6-7.5
31.2
29.0-33.4
Age 45-64
5.7
4.2-7.1
42.7
40.8-44.6
Age Š65
14.6
10.5-19.0
54.9
50.7-58.7
Primary Care Clinic
9.4
7.2-11.6
55.9
53.8-58.0
Non-PTSD Mental Health Clinic
4.2
3.0-5.3
27.6
25.6-29.6
PTSD Clinic
3.3
1.0-5.7
26.1
22.4-30.0

CI = confidence interval, PPV = positive predictive value.


Finally, as can be seen in Table 4, the difference in the ratio of PPV ratios between males and females was not significant, whereas clinic where first diagnosed and age did show significant differences in PPV ratios. The effect of the number of PTSD diagnoses on the PPV of PCL scores of Š50 across age groups was significant only when comparing the group of older veterans with the two younger age groups. Also, a significantly greater effect was noted for at least two diagnoses as opposed to at least one on the PPV for people first diagnosed in primary care as opposed to mental health or PTSD clinics.


Table 4.
Ratios of positive predictive value (PPV) ratios for designated subgroups with 95% confidence intervals (CIs) computed via bootstrap resampling.

Characteristic
Ratio of PPV Ratios
95%
CI

Sex
 
Male vs Female
1.01
0.98-1.05
Age Group
 
<45 vs 45-64
1.01
0.98-1.03
 
<45 vs Š65
0.87
0.82-0.93
 
45-64 vs Š65
0.87
0.81-0.92
Clinic Where First Diagnosed with PTSD
 
PTDS Clinic vs Non-PTSD Mental Health
0.99
0.96-1.02
 
PTSD Clinic vs Primary Care
0.92
0.88-0.96
 
Non-PTSD Mental Health vs Primary Care
0.93
0.90-0.96

Note: CIs not including value 1 indicate that effect of number of PTSD diagnoses (Š1 vs Š2) on PPV for PTSD Checklist scores of Š50 is significantly different between two subgroups (i.e., evidence of interaction).
PTSD = posttraumatic stress disorder.

Of note, the estimates of the differences in PPVs, decreases in sample size (Table 3), and the ratios of ratios (Table 4) are unbiased because the mean of the bootstrapped distributions are within 1 to 3 percent of the observed values [18].

We can also be assured that our 95 percent CIs are accurate for two reasons. We followed the suggestion of Efron and Tibshirani in doing at least 1,000 bootstrapped samples to get accurate CIs [17], and the bootstrapped distributions of the sampling distributions of the PPV differences, the sample size differences, and the ratios of ratios were symmetrical in shape. The more skewed the sampling distribution, the less likely that the 2.5 and 97.5 percentiles of the bootstrap sampling distributions reflect a true 95 percent CI.

DISCUSSION

In summary, we found significantly greater validity associated with using the more restrictive selection criterion (at least two PTSD diagnoses) than the less restrictive criterion (at least one diagnosis). Using at least two diagnoses instead of at least one also had the effect of significantly reducing the available sample size, with an overall reduction of 40 percent. This result persisted by subgroups, and significant moderating effects were also found.

Our data are consistent with those of Frayne et al. who also found a higher PPV when using at least two as opposed to at least one administrative PTSD diagnoses [4]. A contrasting result is that we found a PPV of 75 percent and Frayne et al. found a PPV of 82 percent for any outpatient visit. In addition, we found that the PPV of primary care visits was lower than the PPV for mental health care visits (69% vs 80%), whereas Frayne et al. found the opposite (88% vs 85%) when using at least one diagnosis as the criterion; however, both studies suggest a variation by clinic type. Differences are possibly due to study design, particularly our dependent variables and the time frame used to search for PTSD diagnoses in administrative data. Our study is potentially more accurate because we were using contemporaneous measures of PTSD diagnosis and PCL score, whereas Frayne et al. used lifetime self-reported PTSD diagnosis as their dependent variable. Additionally, our difference was larger and statistically significant.

Although Magruder and Yeager used a drastically different study design [3], some comparisons can be made. Their study gave evidence that PTSD diagnoses are under-recognized in primary care. Our study suggests that PTSD is potentially over-diagnosed in primary care compared with other settings because based on our comparison with PCL scores, many of the initial diagnoses are likely to be in error. Magruder and Yeager looked at subgroups and, similar to our study, found no significant differences in provider recognition of PTSD (finding at least one PTSD diagnosis in administrative data) based on sex, but did find a difference based on age, suggesting possible differences in clinical severity or presentation by age cohort.

Limitations

There are several limitations to this administrative data analysis. One limitation was that it is a secondary analysis of a larger study. Since the PCL was only given to those people who were identified in VA administrative data as having at least one PTSD diagnosis, we were unable to calculate sensitivity, specificity, or negative predictive value. However, we were able to look at the correspondence between administrative data and PCL scores by looking at PPVs. The data analyzed were restricted to newly diagnosed PTSD patients (defined as not having had a PTSD diagnosis in the previous year); however, if participants were seen outside the VA or if they had another PTSD diagnosis before that year, it is possible that some of these were not new cases. In addition, we only used a 4-month period to look for a second diagnosis. Using more than 4 months or using any alternate time frame could create different results. This study does however provide a large national sample that allows for comparison of administrative data with PCL scores across different care settings and types of patients.

A second limitation of this study was that the survey response rate was average to low (65%). To potentially combat this, we examined differences between responders and nonresponders and used propensity methods to adjust for these differences when estimating the PPVs. The observed PPVs were very similar to the PPVs that were adjusted for potential nonresponse bias.

A third limitation was that this secondary analysis did not allow for conclusions to be made regarding the reasons for the findings. It is possible that veterans who have more severe cases of PTSD are preferentially seen in mental health clinics or that primary care doctors refer more severe cases to mental health clinics. Additionally, older veterans may have milder symptoms or symptoms that diminish over time but may still be considered by a VA clinician as having PTSD. We also do not know how providers decide to enter diagnosis codes into administrative records. One study looked at self-reported behaviors for screening and treating PTSD in the VA [21]. The authors compared PTSD specialists and mental health generalists. They found that generalists screened for PTSD much less consistently than they screened for depression or substance abuse and that PTSD specialists were more likely to use validated assessment measures. However, that study was conducted before the implementation of any type of universal PTSD screening. There is evidence to suggest that as of June 2004, universal PTSD screening has been implemented for OIF/OEF veterans [22] through the 4-item primary care PTSD Screen [23]. However, Seal et al. found that after implementing this directive, only 45 percent of OIF/OEF veterans that presented at VA medical centers postdeployment were screened [24]. It is unclear whether methods exist that would allow for more consistency or accuracy in entering PTSD diagnoses into VA administrative databases.

More research is needed in three areas. First, more studies are needed that compare PTSD diagnoses in VA administrative data with one of the gold standards for diagnosing PTSD: the SCID or the CAPS. Second, various algorithms should be tested on VA administrative data so that researchers can use administrative data and be as accurate as possible when using it. Third, future studies need to include a more complete look at accuracy, including sensitivity, specificity, PPV, and negative predictive value.

Sample Selection Recommendations

To select a sample of veterans who have a more definitive PTSD diagnosis using administrative data, researchers should select those veterans with at least two PTSD diagnoses as opposed to at least one. This is particularly true for those first diagnosed in primary care and for older veterans. Using at least two diagnoses might be particularly useful in situations in which a researcher wants to examine the effects of an intervention for those with chronic illness or wants a pure sample to test a PTSD medication. Another consideration might be cost. If a researcher is doing expensive face-to-face interviews, it may be important to have more confidence in the PTSD diagnosis. However, if a researcher is conducting a postal survey, it may not be as expensive to include people with fewer symptoms who may not have PTSD. If doing a study similar to this one, researchers will have to allow for longer recruitment periods to get the sample size needed.

Researchers may also choose to continue to use at least one diagnosis as their criteria. If choosing to do this, 75 percent of the sample will have PCL scores of Š50. This suggests that for 75 percent of the sample, researchers can have a relatively high degree of confidence in the accuracy of the PTSD diagnoses but will be more likely to include some people who probably do not have PTSD. Less confidence can be given to PTSD when it is first diagnosed in primary care and in those Š65. Using at least two diagnoses may not be as important if a researcher is only studying those first diagnosed in non-PTSD mental health or PTSD clinics or veterans under the age of 65. Additionally, researchers might consider using at least one diagnosis if they are doing a study focused on the utility of a new diagnostic tool or on interventions to engage veterans in mental health treatment. In these cases, it might be reasonable to include people with less severe PSTD symptoms. Another consideration is if one is doing a local study as opposed to a national study; in a local study, a severe reduction in sample size may not be practical. Knowledge of the PPV of different sampling algorithms can help researchers know with more accuracy the actual composition of their samples.

CONCLUSIONS

The investigation of the validity of PTSD diagnosis codes found in VA administrative data is essential to using these codes appropriately in VA research. The results of this study indicate that to select a sample of veterans with more definitive PTSD from VA administrative data, researchers should select those veterans with at least two diagnoses as opposed to those with at least one. This is particularly true for those first diagnosed in primary care and for older veterans but statistically significant for the entire sample and all subgroups of interest: age, sex and clinic where first diagnosed. Researchers should also consider the consequences of using one algorithm over the other in terms of the desired homogeneity and sample size. It is important for researchers to know with more accuracy the actual composition of their samples when using VA administrative data to select samples for PTSD studies so that accurate inferences can be made and, ultimately, more effective interventions for PTSD can be developed. More research is needed on the validity of PTSD diagnoses found in VA administrative data.

ACKNOWLEDGMENTS
Author Contributions:
Study concept and design: A. A. Gravely, M. Spoont, S. Nugent.
Acquisition of data (generating and gathering): S. Nugent, A. Cutting, M. Spoont.
Analysis and interpretation of data: A. A. Gravely, J. Grill, A. Cutting, S. Nugent, M. Spoont.
Drafting of manuscript: A. A. Gravely.
Critical revision of manuscript for important intellectual content: A. A. Gravely, M. Spoont, A. Cutting, S. Nugent, J. Grill, K. Carlson.
Statistical analysis: J. Grill, A. A. Gravely.
Obtained funding: M. Spoont.
Administrative, technical, or material support: A. Cutting, A. A. Gravely, S. Nugent, K. Carlson.
Study supervision: M. Spoont.
Financial Disclosures: The authors have declared that no competing interests exist.
Funding/Support: This material was based on work supported by VA HSR&D Service through the Center for Chronic Disease Outcomes Research (grant IAC 06-266 and training grant CDA 08-025).
Additional Contributions: Dr. Carlson is now with Portland VA Medical Center, Portland Center for the Study of Chronic, Comorbid Mental and Physical Disorders, Portland, Oregon.
The views expressed in this article are those of the author(s) and do not necessarily represent the position or policy views of the VA or any other agency.
Institutional Review: The Minneapolis VA Institutional Review Board approved the protocol for this study.
REFERENCES
1. Mohamed S, Rosenheck RA. Pharmacotherapy of PTSD in the U.S. Department of Veterans Affairs: Diagnostic- and symptom-guided drug selection. J Clin Psychiatry. 2008; 69(6):959-65. [PMID: 18588361]
DOI:10.4088/JCP.v69n0611
2. Rosenheck R, Fontana A. Use of mental health services by veterans with PTSD after the terrorist attacks of September 11. Am J Psychiatry. 2003;160(9):1684-90. [PMID: 12944346]
DOI:10.1176/appi.ajp.160.9.1684
3. Magruder KM, Yeager DE. Patient factors relating to detection of posttraumatic stress disorder in Department of Veteran Affairs primary care settings. J Rehabil Res Dev. 2008;45(3):371-81. [PMID: 18629746]
DOI:10.1682/JRRD.2007.06.0091
4. Frayne S, Miller D, Halanych J, Wang F, Sharkansky E, Kader B, Keane T, Pogach L, Rosen C, Berlowitz D. Using administrative data to identify diagnosed posttraumatic stress disorder. Talk presented at: Society of General Internal Medicine Regional Meeting; 2006 Mar 17; Stanford, CA.
5. Blanchard EB, Jones-Alexander J, Buckley TC, Forneris CA. Psychometric properties of the PTSD Checklist (PCL). Behav Res Ther. 1996;34(8):669-73. [PMID: 8870294]
DOI:10.1016/0005-7967(96)00033-2
6. Weathers FW, Huska JA, Keane TM. PTSD Checklist-Military Version (PCL-M). Boston (MA): National Center for PTSD; 1991.
7. Ruggerio KJ, Del Ben K, Scotti JR, Rabalais AE. Psychometric properties of the PTSD Checklist-Civilian version. J Trauma Stress. 2003;16(5):495-502. [PMID: 14584634]
DOI:10.1023/A:1025714729117
8. Magruder KM, Frueh BC, Knapp RG , Johnson MR, Vaughan JA 3rd, Carson TC, Powell DA, Hebert R. PTSD symptoms, demographic characteristics, and functional status among veterans treated in VA primary care clinics. J Trauma Stress. 2004;17(4):293-301. [PMID: 15462536]
DOI:10.1023/B:JOTS.0000038477.47249.c8
9. Rosenheck RA, Fontana AF. Recent trends in VA treatment of post-traumatic stress disorder and other mental disorders. Health Aff (Millwood). 2007;26(6):1720-27.
[PMID: 17978391]
DOI:10.1377/hlthaff.26.6.1720
10. HSR&D priorities for investigator-initiated research [Internet]. Washington (DC): Department of Veterans Affairs; 2008 [updated 2010 Sep 9; cited 2009 Dec 10]. Available from:
http://www.research.va.gov/funding/solicitations/docs/HSRD-IIR-Priorities-2011.pdf.
11. Afghanistan & Iraq [Internet]. Washington (DC): Veterans Health Administration Research and Development; 2009 [cited 2009 Dec 10]. Available from:
http://www.research.va.gov/resources/pubs/docs/OIF-OEF-brochure.pdf.
12. Dillman DA. Mail and internet surveys: The tailored design method. New York (NY): Wiley; 2007.
13. Forbes D, Creamer M, Biddle D. The validity of the PTSD checklist as a measure of symptomatic change in combat-related PTSD. Behav Res Ther. 2001;39(8):977-86.
[PMID: 11480838]
DOI:10.1016/S0005-7967(00)00084-X
14. Weathers FW, Litz BT, Herman DS, Huska JA, Keane TM. The PTSD Checklist (PCL): Reliability, validity, and diagnostic utility. Proceedings of the Annual Conference of the International Society for Traumatic Stress Studies; 1993 Oct 25; San Antonio, TX. Deerfield (IL): International Society for Traumatic Stress Studies; 1993.
15. Terhakopian A, Sinaii N, Engel CC, Schnurr PP, Hoge CW. Estimating population prevalence of posttraumatic stress disorder: An example using the PTSD checklist. J Trauma Stress. 2008;21(3):290-300. [PMID: 18553416]
DOI:10.1002/jts.20341
16. Chernick MR. Bootstrap methods: A practitioner's guide. New York (NY): Wiley; 1999.
17. Efron B, Tibshirani R. An introduction to the bootstrap. New York (NY): Chapman & Hall; 1994.
18. Davison AC, Hinkley DV. Bootstrap methods and their application. Cambridge (UK): Cambridge University Press; 2007.
19. Cole SR. Simple bootstrap statistical inference using the SAS system. Comput Methods Programs Biomed. 1999;60(1): 79-82. [PMID: 10430465]
DOI:10.1016/S0169-2607(99)00016-4
20. Little RJ. Survey nonresponse adjustments for estimates of means. Int Stat Rev. 1986;54(2):139-57. DOI:10.2307/1403140
21. Rosen CS, Chow HC, Finney JF, Greenbaum MA, Moos RH, Sheikh JI, Yesavage JA. VA practice patterns and practice guidelines for treating posttraumatic stress disorder. J Trauma Stress. 2004;17(3):213-22. [PMID: 15253093]
DOI:10.1023/B:JOTS.0000029264.23878.53
22. VHA Directive 2001-015: Implementation of the new national clinical reminder, the "Afghan and Iraq post-deployment screen." Washington (DC): Department of Veterans Affairs, Veterans Health Administration; 2004.
23. Prins A, Ouimette P, Kimerling R, Camerond RP, Hugelshofer DS, Shaw-Hegwer J, Thrailkill A, Gusman FD, Sheikh JI. The primary care PTSD screen (PC-PTSD): Development and operating characteristics. Int J Psychiatry Clin Pract. 2004; 9(1):9-14.
24. Seal KH, Bertenthal D, Maguen S, Gima K, Chu A, Marmar CR. Getting beyond "Don't ask; don't tell": An evaluation of US Veterans Administration postdeployment mental health screening of veterans returning from Iraq and Afghanistan. Am J Public Health. 2008;98(4):714-20.
[PMID: 18309130]
DOI:10.2105/AJPH.2007.115519
Submitted for publication August 10, 2009. Accepted in revised form July 20, 2010.
This article and any supplementary material should be cited as follows:
Gravely AA, Cutting A, Nugent S, Grill J, Carlson K, Spoont M. Validity of PTSD diagnoses in VA administrative data: Comparison of VA administrative PTSD diagnoses to self-reported PTSD Checklist scores.
J Rehabil Res Dev. 2011;48(1):21-30.
DOI:10.1682/JRRD.2009.08.0116
Crossref
1Spoont M. VA Health Services Research and Development (HSR&D) Service through the Center for Chronic Disease Outcomes Research, grant IAC 06-266, "Participation in PTSD Treatment, Who Starts, Who Stays and Who Drops Out?"

Go to TOP

Go to the Table of Contents of Vol. 48 No. 1

Last Reviewed or Updated  Friday, January 14, 2011 9:32 AM

Valid HTML 4.01 Transitional