Journal of Rehabilitation Research & Development (JRRD)

Quick Links

  • Health Programs
  • Protect your health
  • Learn more: A-Z Health
Veterans Crisis Line Badge
 

Volume 51 Number 3, 2014
   Pages 363 — 376

Concordance of clinician judgment of mild traumatic brain injury history with a diagnostic standard

Terri K. Pogoda, PhD;1–2* Katherine M. Iverson, PhD;3 Mark Meterko, PhD;1–2 Errol Baker, PhD;1 Ann M. Hendricks, PhD;2,4 Kelly L. Stolzmann, MS;1 Maxine Krengel, PhD;5 Martin P. Charns, DBA;1–2 Jomana Amara, PhD;6 Rachel Kimerling, PhD;7 Henry L. Lew, MD, PhD8

1Center for Healthcare Organization and Implementation Research, Department of Veterans Affairs (VA) Boston Healthcare System, Boston, MA; 2Department of Health Policy and Management, Boston University School of Public Health, Boston, MA; 3National Center for Posttraumatic Stress Disorder, VA Boston Healthcare System, Boston, MA; and Department of Psychiatry, Boston University School of Medicine, Boston, MA; 4Health Care Financing and Economics, VA Boston Healthcare System, Boston, MA; 5Research and Development Service, VA Boston Healthcare System, Boston, MA; and Department of Neurology, Boston University School of Medicine, Boston, MA; 6Defense Resource Management Institute, Naval Postgraduate School, Monterey, CA; 7National Center for Posttraumatic Stress Disorder and Center for Health Care Evaluation, VA Palo Alto Healthcare System, Palo Alto, CA; 8Department of Physical Medicine and Rehabilitation, Defense and Veterans Brain Injury Center, Virginia Commonwealth University, Richmond, VA; and Department of Communication Sciences and Disorders, John A. Burns School of Medicine, University of Hawai’i at Mānoa, Honolulu, HI

Abstract — The concordance of Department of Veterans Affairs (VA) clinician judgment of mild traumatic brain injury (mTBI) history with American Congress of Rehabilitation Medicine (ACRM)-based criteria was examined for Operation Iraqi Freedom (OIF) and Operation Enduring Freedom (OEF) Veterans. In order to understand inconsistencies in agreement, we also examined the associations between evaluation outcomes and conceptually relevant patient characteristics, deployment-related events, current self-reported health symptoms, and suspected psychiatric conditions. The Veteran sample comprised 14,026 OIF/OEF VA patients with deployment-related mTBI history (n = 9,858) or no history of mTBI (n = 4,168) as defined by ACRM-based criteria. In the majority of cases (76.0%), clinician judgment was in agreement with the ACRM-based criteria. The most common inconsistency was between clinician judgment (no) and ACRM-based criteria (yes) for 21.3% of the patients. Injury etiology, current self-reported health symptoms, and suspected psychiatric conditions were additional factors associated with clinician diagnosis and ACRM-based criteria disagreement. Adherence to established diagnostic guidelines is essential for accurate determination of mTBI history and for understanding the extent to which mTBI symptoms resolve or persist over time in OIF/OEF Veterans.

Key words: American Congress of Rehabilitation Medicine guidelines, blast injuries, comprehensive traumatic brain injury evaluation, diagnosis, mild traumatic brain injury, neurobehavioral manifestations, non-blast injuries, posttraumatic stress disorder, psychological factors, Veterans.

Abbreviations: ACRM = American Congress of Rehabilitation Medicine, AOC = alteration of consciousness, aOR = adjusted odds ratio, CI = confidence interval, CTBIE = comprehensive traumatic brain injury evaluation, DOD = Department of Defense, GCS = Glasgow Coma Scale, IED = improvised explosive device, LOC = loss of consciousness, mTBI = mild traumatic brain injury, N = no, NSI = Neurobehavioral Symptom Inventory, OEF = Operation Enduring Freedom, OIF = Operation Iraqi Freedom, PTA = posttraumatic amnesia, PTSD = posttraumatic stress disorder, TBI = traumatic brain injury, VA = Department of Veterans Affairs, Y = yes.
*Address all correspondence to Terri K. Pogoda, PhD; VA Boston Healthcare System, 150 S Huntington Ave (152M), Boston, MA 02130; 857-364-2626; fax: 857-364-6140. Email: terri.pogoda@va.gov
http://dx.doi.org/10.1682/JRRD.2013.05.0115
INTRODUCTION

Mild traumatic brain injury (mTBI), or concussion, is a defining injury of U.S. servicemembers who have served in the Iraq and Afghanistan wars. Recent estimates indicate self-reported or clinician-confirmed traumatic brain injury (TBI) ranging from 6.8 to 22.8 percent of Operation Enduring Freedom (OEF) and Operation Iraqi Freedom (OIF) servicemembers and Veterans [1–6]. Blast (e.g., from bombs or improvised explosive devices [IEDs]) is the most prevalent mechanism of mTBI for OIF/OEF servicemembers [7–9], and vehicular accidents and falls [4,6] are other common etiologies of TBI among OIF/OEF servicemembers. Symptoms following mTBI typically resolve within hours or days, and usually last less than 12 mo, but can emerge or persist over time and interfere with daily functioning [10–11]. However, there is uncertainty as to whether symptoms that appear to be persistent are directly related to an mTBI incident [6,11–13]. Due to the prevalence and health consequences of mTBI, accurate determination of mTBI is important so that appropriate treatment is delivered and progress is evaluated.

Identifying an mTBI event can be challenging. On the battlefield, moderate to severe forms of TBI can be recognized by obvious physical signs or functional limitations [14–15]. However, mTBI symptoms in their acute stage may go undetected for many reasons, including confounding mental (e.g., posttraumatic stress disorder [PTSD]) or physical (e.g., amputation) health conditions requiring more immediate attention [16–18], fast-paced battlefield situations in which mTBI and resulting change in consciousness or memory is unnoticed or undocumented, or servicemembers’ reluctance to report injuries while deployed [16]. Imaging and neurological examinations for mTBI, if performed, may appear normal [11,17]. Because of these contextual factors, Veteran self-report, in conjunction with a medical examination [1], has become standard for identifying mTBI history in the Department of Veterans Affairs (VA) [11].

TRAUMATIC BRAIN INJURY SCREENING AND EVALUATION

In April 2007, the VA mandated a TBI screening clinical reminder system for all OIF/OEF Veterans seeking VA healthcare [4,19–20]. Patients who report a prior TBI diagnosis upon screening are offered a referral for follow-up treatment. Patients without a prior TBI diagnosis are asked four sets of TBI screening questions regarding (1) events that increase TBI risk, (2) immediate symptoms following the event, (3) new or worsening symptoms postevent, and (4) current symptoms. Patients endorsing one or more questions in each of the four sections screen positive and are offered a referral for a VA comprehensive TBI evaluation (CTBIE) [11,19,21–22].

VA’s standard for identifying TBI is a medical examination and in-depth clinical interview by a clinician assessing possible TBI events and current symptoms. The VA and Department of Defense (DOD) define TBI as a structural injury and/or disruption in brain function caused by an external force resulting in the onset or worsening of clinical signs immediately postevent [11]. These signs include loss of consciousness (LOC) or decreased level of consciousness, loss of memory for events immediately prior to or following the injury (posttraumatic amnesia [PTA]), and altered mental state (alteration of consciousness [AOC]) [11]. TBI is further categorized as mild, moderate, or severe based on the duration of these sequelae [11]. The VA/DOD clinical practice guidelines adopted criteria based on those of the American Congress of Rehabilitation Medicine (ACRM) for classifying mTBI. According to ACRM criteria, mTBI is defined as a physiological disruption in brain function that is manifested by at least one of the following: LOC ≤30 min, AOC ≤24 h, PTA ≤1 d, any focal neurological deficit(s) that may or may not be transient, and a Glasgow Coma Scale (GCS) score of 13 to 15 after the first 30 min postinjury. The CTBIE template captures information related to LOC, AOC, and PTA.

The extent to which VA clinicians identify mTBI history in adherence to ACRM-based criteria is unknown. Following established guidelines is essential for identifying patients needing treatment for complaints that are potentially related to mTBI. Accurate identification also allows for recognition that mTBI may be comorbid with other health conditions, thereby informing different types of targeted treatments. An incorrect diagnosis can delay or prevent follow-up care or result in iatrogenic effects, unnecessary treatment, and/or a lack of treatment for conditions that share mTBI-related symptoms (e.g., PTSD) [23–24]. The first aim of this study was to identify the concordance between the CTBIE clinician judgment of mTBI history and ACRM-based criteria. The second aim was to examine the associations between discordant ­clinician judgment and ACRM-based criteria outcomes and conceptually relevant patient factors, including demographic characteristics, injury etiology, self-reported current health symptoms, and suspected psychiatric conditions.

METHODS
Design

This was a retrospective cross-sectional database review of Veterans who completed a CTBIE between October 1, 2007, and June 30, 2009. The VA Boston Healthcare System Institutional Review Board approved this study.

Data Sources

CTBIE and patient demographic data were obtained from the national VA Patient Care Services database. Clinicians performing the CTBIE used an electronic evaluation template to facilitate discussion and capture information on predeployment, deployment, and postdeployment experiences. The deployment-related self-report questions regarding injury etiology include items assessing frequency and type of non-blast (i.e., vehicular accident, bullet wound, fall, other blunt trauma) and blast (i.e., IED, rocket-propelled grenade, mortar, grenade, bomb, other) injuries and postinjury sequelae (i.e., experiencing at least one of the following ACRM-based criteria: LOC, PTA, AOC) potentially indicative of TBI. CTBIE records did not include data related to other ACRM criteria (i.e., GCS score, neuroimaging results, evidence of focal neurological deficits) [11]. However, the former two measures typically appear normal for mTBI, and VA/DOD guidelines indicate that the latter cannot be used for diagnostic purposes if there is no disrupted consciousness [11]. Patients also reported on the severity of current symptoms, and clinicians indicated their judgment regarding any suspected behavioral or psychiatric symptoms. Finally, the CTBIE clinician made two sequential TBI-related judgments: (1) whether the history and clinical course was consistent with a diagnosis of TBI (TBI history), and (2) whether current clinical symptom presentation was consistent with TBI residual problems or another condition. The focus of this study was on mTBI history.

Participants

Among 36,919 electronic CTBIE records, 14,026 were included in the final sample. Cases were initially excluded if they were known test cases or duplicate records. We identified pathways for categorizing deployment-related mTBI history as determined by ACRM-based criteria or clinician judgment (Figure 1). The inclusion criteria for the present study were (1) not reporting pre- or postdeployment TBI; (2) responding to all three ACRM-based criteria questions concerning LOC, AOC, and PTA and their respective durations; and (3) having an electronic record of a clinician’s judgment (yes [Y] or no [N]) that the “history and clinical course was consistent with a diagnosis of TBI.” Among patients who endorsed criteria indicative of a TBI history, only those who reported LOC, AOC, and/or PTA within the duration parameters [11] of mTBI were included. Patients were excluded from all analyses if their CTBIE indicated that they were “uncertain” of any LOC, AOC, or PTA experiences or if there were inconsistent responses (e.g., a response of “no” for LOC but an LOC duration of “less than 30 min”). Patients were also excluded from specific analyses that included injury etiology if there were inconsistent responses for this variable (e.g., a response of no blast exposure, but another response of experiencing 4 blast episodes).


Figure 1. Final sample of patients with deployment-related mild traumatic brain injury (mTBI) or no mTBI history, as identified by two pathways of clinician judgment and American Congress of Rehabilitation Medicine (ACRM)-based criteria. Note: ACRM-based criteria include loss of consciousness (LOC), alteration of consciousness (AOC), and posttraumatic amnesia (PTA). At least one of these features must be present for positive traumatic brain injury (TBI) judgment. N reported in each of the last four cells indicate number of patients who reported experiencing LOC, AOC, and/or PTA. *Of 36,919 cases in original data set, test cases or repeat comprehensive TBI evaluations (CTBIEs) (n = 281) were excluded. Other exclusions include pre- or post-deployment TBI history (n = 11,385); records in which LOC, PTA, or AOC did not fall within parameters of ACRM-based mTBI criteria (807 cases were moderate TBI and 1,393 were severe TBI); or insufficient information to make TBI judgment, including patient self-report of uncertainty of TBI sequelae or missing data (n = 9,027).

Figure 1.

Final sample of patients with deployment-related mild traumatic brain injury (mTBI) or no mTBI history, as identified by two pathways of clinician judgment and American Congress of Rehabilitation Medicine (ACRM)-based criteria. Note: ACRM-based criteria include loss of consciousness (LOC), alteration of consciousness (AOC), and posttraumatic amnesia (PTA). At least one of these features must be present for positive traumatic brain injury (TBI) judgment. N reported in each of the last four cells indicate number of patients who reported experiencing LOC, AOC, and/or PTA. *Of 36,919 cases in original data set, test cases or repeat comprehensive TBI evaluations (CTBIEs) (n = 281) were excluded. Other exclusions include pre- or post-deployment TBI history (n = 11,385); records in which LOC, PTA, or AOC did not fall within parameters of ACRM-based mTBI criteria (807 cases were moderate TBI and 1,393 were severe TBI); or insufficient information to make TBI judgment, including patient self-report of uncertainty of TBI sequelae or missing data (n = 9,027).

Click Image to Enlarge. View as PowerPoint Slide

Neurobehavioral Symptoms

The CTBIE included the 22-item Neurobehavioral Symptom Inventory (NSI) [25], which asked patients to rate the extent they have been affected by various health symptoms within the last 30 d on a 0 (none) to 4 (very severe) Likert-type scale. Psychometric assessment by Meterko et al. indicated that scales representing four health symptom domains can be scored from the NSI: affective (e.g., irritability), cognitive (e.g., forgetfulness), somatosensory (e.g., headaches), and vestibular (e.g., feeling dizzy) [26]. The CTBIE included an additional question about pain, measured on a 5-point scale similar to that of the NSI items. Pain loaded on the somatosensory factor and was included in that scale [26]. The NSI is meant to identify health symptoms so that patients can be referred for appropriate treatment, as needed, regardless of symptom etiology.

Suspected Psychiatric Conditions

Using a checklist in the CTBIE, clinicians can indicate their clinical judgment about patients’ suspected psychiatric conditions. We examined the association of mTBI history evaluation outcome with suspected PTSD, non-PTSD anxiety disorders, depression, and drug and alcohol abuse and/or dependence, since they are among the most prevalent psychiatric conditions documented in OIF/OEF Veterans [2,5,27–36].

Statistical Analysis

Using the three ACRM-based criteria (LOC, AOC, PTA) as the reference standard, patients were stratified to examine concordance between clinician judgment of mTBI history and ACRM-based criteria (Y/N). This yielded a 2 × 2 matrix of outcomes. Consistent responses included agreement between the clinician and ACRM-based criteria on either the presence (Clinician Y/ACRM Y) or absence (Clinician N/ACRM N) of mTBI history. Inconsistent responses were said to have occurred when clinicians and ACRM-based criteria were not in agreement: clinicians did not indicate mTBI history but ACRM-based criteria did (Clinician N/ACRM Y), or clinicians indicated mTBI history but the ACRM-based criteria did not (Clinician Y/ACRM N). The consistent groups (Clinician Y/ACRM Y and Clinician N/ACRM N) were compared on demographics, deployment-related injury experiences, suspected psychiatric conditions, and current NSI health symptoms. Chi-square analyses and independent t-tests were computed to examine group differences for categorical and quantitative data, respectively. Between-group differences in NSI health symptoms among all four outcome groups were analyzed with multivariate analysis of variance.

Logistic regression analyses modeled the odds of inconsistent outcomes between clinicians and ACRM-based criteria for mTBI history diagnosis. For each model, demographics, deployment-related injury experiences, suspected psychiatric conditions, and NSI health symptoms were entered simultaneously. Adjusted odds ratios (aORs) and 95 percent confidence intervals (CIs) are reported. Analyses were conducted using SPSS version 18.0 (IBM Corporation; Armonk, New York).

RESULTS
Patient and Deployment-Related Characteristics of Study Sample

The sample of Veterans who completed a CTBIE consisted of patients with deployment-related mTBI history (n = 9,858) or no mTBI history (n = 4,168) (Figure 1) using ACRM-based criteria as the reference standard. Of the 14,026 Veterans included in the final sample, the average age was 31.6 ± 8.8 yr old, 93.8 percent were male, and more than three-quarters of reported injury etiologies were by blast (Table 1). Among those who reported post head injury event sequelae, LOC was indicated by approximately one-third, AOC by two-thirds, and PTA by one-quarter of the sample. Clinicians suspected PTSD in two-thirds, non-PTSD anxiety in one-quarter, depression in approximately 40 percent, alcohol abuse and/or dependence in 7.3 percent, and drug abuse and/or dependence in less than 2 percent of the sample. On average, current NSI self-reported symptoms ranged between mild and moderate in severity for the affective, somatosensory, cognitive, and vestibular domains.


Table 1. 
Total
(
n = 14,026)
p-Value
40
Comparisons Between Veterans With and Without Deployment-Related Mild Traumatic Brain Injury History Based on Concordance Between Clinician Judgment and American Congress of Rehabilitation Medicine-Based Criteria

Veterans with deployment-related mTBI history were younger (χ2 (3) = 259.72, p < 0.001) and more likely to have experienced a combination of non-blast and blast injuries (χ2 (2) = 331.62, p < 0.001) than Veterans without mTBI history. They were also more likely to have suspected PTSD (χ2 (1) = 495.86, p < 0.001), depression (χ2 (1) = 25.64, p < 0.001), and drug abuse and/or dependence (χ2 (1) = 4.98, p < 0.03), but not suspected non-PTSD anxiety (χ2(1) = 0.24, p = 0.63) or alcohol abuse and/or dependence (χ2 (1) = 3.80, p < 0.06), though the latter approached statistical significance. More than half of those with deployment-related mTBI history reported experiencing LOC, approximately 42 percent reported PTA, and nearly 96 percent reported AOC following head injury. For all NSI health symptom domains, Veterans with deployment-related mTBI history reported higher current symptom severity, which generally ranged from mild to moderate, than Veterans with no mTBI history (all F (1, 10,601) > 434.40, p < 0.001).

Concordance Between Clinician Judgment of Mild Traumatic Brain Injury History and American
Congress of Rehabilitation Medicine-Based Criteria

Clinicians indicated that 51.7 percent (7,252 of 14,026) of patients met criteria for mTBI history, whereas ACRM-based standards indicated 70.3 percent (9,858 of 14,026) (Table 2). We observed the following concordance rates: 49.0 percent Clinician Y/ACRM Y, 21.3 percent Clinician N/ACRM Y, 2.7 percent Clinician Y/ACRM N, and 27.0 percent Clinician N/ACRM N. Kappa for this distribution was 51.3 percent, suggesting moderate agreement between clinician judgment and ACRM-based criteria. Assuming ACRM-based criteria as the reference standard, the sensitivity and specificity rates for the clinician judgment were 69.7 and 90.9 percent, respectively.


Table 2. 
Concordance (n [%]) between clinician judgment of mild traumatic brain injury history with American Congress of Rehabilitation Medicine (ACRM)-based criteria.
Factors Associated with Clinician and American Congress of Rehabilitation Medicine Concordance Outcomes
Clinician (No) and American Congress of Rehabilitation Medicine (Yes) Versus Clinician (Yes) and American Congress of Rehabilitation Medicine (Yes)

The logistic regression model predicting the odds of a Clinician N/ACRM Y (1 = target group) versus a Clinician Y/ACRM Y (0 = reference group) outcome, using conceptually relevant patient characteristics and deployment-related experiences, was significant (χ2 (18) = 1,449.11, p < 0.001). Adjusting for all variables in the model (Table 3), a Clinician N/ACRM Y outcome was more likely for patients who were 30 to 39 yr old (aOR = 1.15, 95% CI: 1.02–1.31) and ≥40 yr old (aOR = 1.30, 95% CI: 1.13–1.49) than for patients who were 25 to 29 yr old. Sex was not significantly associated with outcome. Veterans who reported an injury etiology of blast only compared with a non-blast etiology only were significantly more likely (aOR = 1.32, 95% CI: 1.16–1.52) to have a Clinician N/ACRM Y outcome. A combination of blast and non-blast injuries, however, did not significantly differ from a non-blast only mechanism of injury. Reports of LOC, AOC, and PTA were each associated with a decreased likelihood (aOR = 0.30–0.69, 95% CI: 0.27–0.88) of a Clinician N/ACRM Y outcome.


Table 3. 
40
*p < 0.05.
p < 0.001.
p < 0.01.

Patients with suspected PTSD (aOR = 0.73, 95% CI: 0.65–0.82) or a non-PTSD anxiety (aOR = 0.84, 95% CI: 0.75–0.95) condition were significantly less likely to be categorized as Clinician N/ACRM Y than patients with neither condition. Suspected depression and drug and alcohol abuse and/or dependence were not associated with this outcome.

Regarding NSI health symptoms, patients reporting more severe affective symptoms were about 25 percent more likely (aOR = 1.24, 95% CI: 1.14–1.34) to have a Clinician N/ACRM Y outcome than patients reporting less severe affective symptoms; those reporting more severe somatosensory, cognitive, and vestibular symptoms were, by contrast, significantly less likely (aOR = 0.79–0.85, 95% CI: 0.73–0.93) to have this outcome.

Clinician (Yes) and American Congress of Rehabilitation Medicine (No) Versus Clinician (No) and American Congress of Rehabilitation Medicine (No)

The logistic regression model predicting the odds of a Clinician Y/ACRM N (1 = target group) and Clinician N/ACRM N (0 = reference group) outcome, using conceptually relevant patient characteristics and deployment-related experiences, was also significant (χ2 (15) = 100.51, p < 0.001). Neither age nor sex were associated with a Clinician Y/ACRM N outcome. Patients who reported blast only (aOR = 0.51, 95% CI: 0.38–0.69) or a combination of blast and non-blast injuries (aOR = 0.57, 95% CI: 0.44–0.74) were less likely to have a Clinician Y/ACRM N outcome than patients reporting an injury etiology of non-blast only.

Patients with suspected PTSD (aOR = 2.01, 95% CI: 1.55–2.60) or depression (aOR = 1.60, 95% CI: 1.25–2.05), compared with patients without these suspected conditions, were more likely to be categorized as Clinician Y/ACRM N, whereby suspected non-PTSD anxiety and drug and alcohol abuse and/or dependence were not associated with this outcome. Higher self-reported somatosensory (aOR = 1.28, 95% CI: 1.01–1.64) and cognitive (aOR = 1.25, 95% CI: 1.06–1.48) symptoms were more likely to have a Clinician Y/ACRM N outcome, whereas those reporting more severe affective symptoms (aOR = 0.67, 95% CI: 0.56–0.81) were less likely to have this outcome. Vestibular symptoms were not associated with a Clinician Y/ACRM N outcome.

Neurobehavioral Symptom Inventory Severity and Mild Traumatic Brain Injury History Evaluation Outcomes

Regarding current neurobehavioral symptom severity, the highest scores were observed in the affective domain, followed by cognitive, somatosensory, and vestibular (Figure 2). Within each domain, there were significant differences among the four outcome groups (all (3, 13,957) > 158.00, all p < 0.001). Post hoc Tamhane significance testing showed that across all domains, in general, the consistent groups reported the highest (Clinician Y/ACRM Y) and lowest (Clinician N/ACRM N) symptom severity, respectively, compared with all other outcome groups (all p < 0.05). The one exception was for the affective symptom domain, in which the mean ± ­standard deviation for the Clinician Y/ACRM N group (2.06 ± 1.00) did not differ from the Clinician N/ACRM N group (1.97 ± 1.03). The inconsistent (Clinician N/ACRM Y, Clinician Y/ACRM N) groups did not differ from one another on any NSI health symptom domain.


Figure 2. Neurobehavioral Symptom Inventory (NSI) health domain average scores by evaluation outcome. Note: For majority of NSI health domains, Clinician Y and American Congress of Rehabilitation Medicine (ACRM) Y > (Clinician N/ACRM Y = Clinician Y/ACRM N) > Clinician N/ACRM N. N = no, Y = yes.

Figure 2.

Neurobehavioral Symptom Inventory (NSI) health domain average scores by evaluation outcome. Note: For majority of NSI health domains, Clinician Y and American Congress of Rehabilitation Medicine (ACRM) Y > (Clinician N/ACRM Y = Clinician Y/ACRM N) > Clinician N/ACRM N. N = no, Y = yes.

Click Image to Enlarge. View as PowerPoint Slide

DISCUSSION

This study aimed to examine concordance of clinician judgment of mTBI history with ACRM-based criteria and explore patient and deployment-related characteristics associated with evaluation outcomes. Although agreement between clinician judgment and ACRM-based criteria (Clinician Y/ACRM Y, Clinician N/ACRM N) for mTBI history was the collective majority (49.0% and 27.0%, respectively), there was disagreement in 24.0 percent of the cases, almost all of which were Clinician N/ACRM Y (21.3%). A major pathway for receiving a CTBIE is to be referred after screening positive for TBI. Thus, prior to the examination, the CTBIE evaluation clinician knows that the Veteran reported events and symptoms that are suggestive of TBI. Therefore, we would have predicted that, if there was going to be a bias, more patients would have been categorized as Clinician Y/ACRM N, but this subgroup comprised only 2.7 percent of the sample.

Numerous factors likely affected the Clinician N/ACRM Y outcome. The natural course of mTBI is symptom reduction or resolution over hours or days, and typically no longer than 1 yr [10–11]. Although our data are incomplete with respect to time between head injury and CTBIE, some injury dates were reported as early as 2001, putting the time between injury and evaluation up to 8 yr. Clinicians may have judged that there was no mTBI history, even if there was a reported TBI incident, because of difficulty relating current symptoms to remote events [14]. This might also explain the association between NSI health symptom severity with outcome groups. Across all NSI health symptom domains, the Clinician Y/ACRM Y group reported the highest severity, suggesting that clinicians may have interpreted current symptom severity as indicative of mTBI history, even though current symptom presentation does not signify mTBI history, nor is it recommended by VA/DOD clinical practice guidelines to identify mTBI based on symptom severity months to years posttrauma [11].

Another possible explanation for the Clinician N/ACRM Y outcome is that clinicians may have in fact considered the ACRM-based criteria but intentionally indicated no mTBI history due to uncertainty about the veracity of reports or ambiguity regarding whether patient-reported TBI sequelae were attributable to a TBI or to other circumstances. For example, delay in recovery from mTBI symptoms is associated with litigation or compensation-seeking [2,10–11,14], and it is possible that some clinicians may have perceived event and symptom reporting as exaggerated or related to secondary gain. Of more clinical relevance, descriptions of TBI-related events may not have appeared credible if patients recalled detailed information, despite reporting loss of or alteration in consciousness or memory [14]. Filling in these gaps is typically due to accounts provided by witnesses or a retrospective piecing together of events by the injured. Conversely, Veterans may have had trouble remembering events because they had disrupted consciousness, and it may have been difficult for clinicians to determine a root cause of fractured recall. For example, Diagnostic and Statistical Manual of Mental Disorders-4th edition [37] criteria for acute stress disorder (which was not assessed during the CTBIE) includes reduced awareness of surroundings (e.g., being in a daze) and dissociative amnesia following a traumatic event, which are markedly similar to the clinical signs of AOC and PTA following a TBI event [38]. Acute stress disorder has also been shown to be positively predictive of subsequent PTSD development [39–40]. Of note, for patients in the Clinician N/ACRM Y group, clinicians indicated that 63.1 percent (1,881 of 2,982) had suspected PTSD (data not shown). The co-occurrence of PTSD with TBI is well-documented in OIF/OEF Veterans, but the extent to which negative health symptoms are attributable to TBI, PTSD, or both remains unclear [2–3,6,27,31–33,35–36,41–43]. Clinicians also indicated possible drug or alcohol abuse and/or dependence in approximately 10 percent of the sample, but neither of these suspected conditions were significant contributors to the clinician and ACRM agreement.

We included NSI health symptoms in analyses because clinicians had access to this information during the CTBIE, even though VA/DOD clinical practice guidelines indicate that such symptoms should not affect decision-making about mTBI history. An mTBI event is defined by presence and duration of LOC, AOC, and PTA at time of injury and not by symptoms reported months to years after a TBI event. Nonetheless, we found significant associations between symptoms and evaluation outcomes. Higher affective and lower somatosensory and cognitive symptoms were more likely to be associated with a Clinician N/ACRM Y outcome. In contrast, lower affective and higher somatosensory and cognitive symptoms were significantly associated with a Clinician Y/ACRM N outcome. It is unclear what accounts for these findings, but clinicians may have interpreted affective symptoms as being more likely related to readjustment issues and psychiatric conditions rather than mTBI. Reports of more severe cognitive and somatosensory symptoms may have been perceived to be more organic and therefore more likely to be indicative of mTBI history.

Of the respondents, 80 percent indicated at least one blast injury, and half of these patients reported that they also experienced at least one additional non-blast injury. We would have expected that relative to a non-blast injury only, reporting both blast and non-blast injuries would be more likely to result in a TBI history diagnosis. Injury etiology findings did not fall into a predictable pattern, further highlighting the complexity of assessing TBI history in OIF/OEF Veterans. The CTBIE data did not specify whether blast and non-blast injuries were linked to multiple injury events at one discrete time (e.g., an IED exploding under a moving vehicle, resulting in both blast and blunt impact injuries) [16] or at separate times. Thus, the categorization of injury etiology may represent a false precision.

In sum, VA clinicians face the difficult challenge of having to make a determination of whether an injury event happened months or years prior to the CTBIE (indicative of TBI history) and whether that event resulted in a TBI that is related to current symptoms (current TBI). In the VA setting, oftentimes, the only source of historical information is the Veteran [44], whose accounting of events may be further complicated if they experienced a TBI. When taking patient history or performing a physical examination to render a determination of TBI history, one of the roles of the evaluating clinician is to assess the patient’s reliability and ability to effectively communicate information. This is based on a synthesis of corroborating and conflicting information, suspected mental health conditions, patients’ responses to other questions, access to medical records, and the clinician’s expertise. If it is a clinician’s judgment that an event (e.g., TBI) or symptom (e.g., memory deficits) did not occur or does not exist currently, then this information needs to be documented in the medical record. Clinicians may still record the patient’s subjective report but will likely provide qualifiers in the medical record. These data were not available in the current study.

LIMITATIONS AND FUTURE DIRECTIONS

Self-report data make it impossible to confirm objectively whether a TBI event occurred. The data reported are from October 2007 to June 2009, the early stages of the VA CTBIE implementation. While VA-wide education to medical specialists was disseminated through national, regional, and individual provider training calls, significant and more standardized educational efforts have since been implemented (e.g., VA-wide training on TBI clinical practice guidelines), as well as changes to the CTBIE template, which clarify the intent of the evaluation and guide the provider through the diagnostic criteria. Therefore, our results are limited to the specified time period. Additionally, we do not have details regarding differences in the evaluation process across locations or across evaluators, such as variation in the extent of clinicians’ additional information-gathering outside of the CTBIE per se, including why clinicians did or did not suspect psychiatric conditions (e.g., opinion, electronic medical record, patient report). Findings concerning suspected psychiatric conditions should be interpreted cautiously here, but there would be benefit to including psychiatrists and psychologists in the algorithm for TBI assessment, because they have the training to consider all the implications of medical and psychological conditions upon the presence or absence of TBI. Because the data set did not contain provider-level data, we were unable to determine whether provider-level characteristics such as clinical specialty and level of TBI expertise were associated with outcomes.

The CTBIE process takes up to an hour, and clinicians indicate whether patient history and clinical course is consistent with a TBI diagnosis toward the end of the CTBIE. Because clinicians may not review responses to the ACRM-based criteria questions prior to making their determination regarding TBI history, inadequate clinician recall may account for some of the observed inconsistencies. To ensure consistency between the VA/DOD clinical practice guidelines for determining mTBI history (i.e., ACRM-based criteria) and indication of deployment-related mTBI, VA has begun automating the CTBIE by linking responses to questions documenting LOC, AOC, and PTA to the questions in which clinicians indicate whether history, clinical course, and current clinical symptom presentation is consistent with TBI.1 This, in addition to the increased education for clinicians and an interdisciplinary team approach to TBI assessment, may help the VA to estimate the extent to which deployment-related TBI symptoms persist or resolve over time. However, this would be only a partial estimation based on a nonrepresentative sample since only Veterans who (1) receive VA healthcare and (2) report current symptoms during the TBI screen would be referred to the CTBIE. Veterans who do not receive VA healthcare or who screen negative because their TBI-related symptoms had resolved would not be accounted for in this estimation because they would not likely receive a CTBIE.

This research focused on concordance between clinician judgment of mTBI history and ACRM-based criteria, but accurate determination of mTBI history is only the start of the treatment process. Future research should examine whether mTBI history and diagnosis affect healthcare utilization and patient outcomes. That is, controlling for symptom reporting, do referral and treatment patterns differ by mTBI diagnostic status? Such analyses should focus on whether mTBI diagnoses confer an advantage (e.g., more appropriate follow-up care), and if so, how utilization is associated with health and reintegration.

CONCLUSIONS

While the majority of VA clinicians were in agreement with ACRM-based guidelines for indicating mTBI history, there was a significant minority (approximately one in five) in which there was disagreement between clinician judgment and ACRM-based criteria, primarily in the direction of Clinician N/ACRM Y. Injury etiology, current self-reported health symptoms, and suspected psychiatric conditions were additional factors that were associated with clinician diagnosis and ACRM-based criteria disagreement. Adherence to established diagnostic guidelines is essential for accurate determination of mTBI history and for understanding the extent to which mTBI symptoms resolve or persist over time in OIF/OEF Veterans.


1

Bidelspach, Douglas (Rehabilitation and Prosthetic Services, Lebanon VA Medical Center, Lebanon, PA). Email with: Terri K. Pogoda (Center for Healthcare Organization and Implementation Research, VA Boston Medical Center, Boston, MA; Department of Health Policy and Management, Boston University School of Public Health, Boston, MA). 2012 Mar 29.

ACKNOWLEDGMENTS
Author Contributions:
Study concept and design: T. K. Pogoda, M. Meterko, E. Baker, A. M. Hendricks.
Acquisition of data: A. M. Hendricks, M. Meterko, K. L. Stolzmann.
Analysis and interpretation of data: K. L. Stolzmann, T. K. Pogoda, M. Meterko, E. Baker, A. M. Hendricks, K. M. Iverson.
Drafting of manuscript: T. K. Pogoda, K. M. Iverson, M. Meterko, A. M. Hendricks, E. Baker, H. L. Lew.
Critical revision of manuscript for important intellectual content: T. K. Pogoda, K. I. Iverson, M. Meterko, E. Baker, A. M. Hendricks, K. L. Stolzmann, M. Krengel, M. P. Charns, J. Amara, R. Kimerling, H. L. Lew.
Statistical analysis: K. L. Stolzmann, T. K. Pogoda, M. Meterko.
Obtained funding: H. L. Lew, A. M. Hendricks.
Administrative, technical, or material support: T. K. Pogoda, K. L. Stolzmann.
Study supervision: T. K. Pogoda, A. M. Hendricks, H. L. Lew.
Financial Disclosures: The authors have declared that no competing interests exist.
Funding/Support: This material was based on work supported by the VA Office of Research and Development, Health Services Research and Development Service (grant SDR 08–405). Dr. Iverson’s contribution was supported by a Health Services Research and Development Service Career Development Award (CDA-2; 10-029).
Additional Contributions: We thank Douglas E. Bidelspach, MPT; David X. Cifu, MD; and Joel D. Scholten, MD, for reviewing this article. Some of the findings were presented at the Third Interagency Conference on TBI; June 14, 2011; Washington, DC.
Institutional Review: All procedures for conduct of this study adhered to the requirements of the institutional review board at the VA Boston Healthcare System.
Participant Follow-Up: The authors do not plan to inform database participants of the publication of this study.
Disclaimer: The opinions expressed in this article are the authors’ and do not reflect those of the VA, Veterans Health Administration, VA Health Services Research and Development Service, Defense and Veterans Brain Injury Center, or DOD.
REFERENCES
1.
Hendricks AM, Amara J, Baker E, Charns MP, Gardner JA, Iverson KM, Kimerling R, Krengel M, Meterko M, Pogoda TK, Stolzmann KL, Lew HL. Screening for mild traumatic brain injury in OEF-OIF deployed US military: An empirical assessment of VHA’s experience. Brain Inj. 2013; 27(2):125–34. [PMID:23384211]
http://dx.doi.org/10.3109/02699052.2012.729284
2.
Hoge CW, McGurk D, Thomas JL, Cox AL, Engel CC, Castro CA. Mild traumatic brain injury in U.S. Soldiers returning from Iraq. N Engl J Med. 2008;358(5):453–63.
[PMID:18234750]
http://dx.doi.org/10.1056/NEJMoa072972
3.
Schneiderman AI, Braver ER, Kang HK. Understanding sequelae of injury mechanisms and mild traumatic brain injury incurred during the conflicts in Iraq and Afghanistan: Persistent postconcussive symptoms and posttraumatic stress disorder. Am J Epidemiol. 2008;167(12): 1446–52. [PMID:18424429]
http://dx.doi.org/10.1093/aje/kwn068
4.
Schwab KA, Ivins B, Cramer G, Johnson W, Sluss-Tiller M, Kiley K, Lux W, Warden D. Screening for traumatic brain injury in troops returning from deployment in Afghanistan and Iraq: Initial investigation of the usefulness of a short screening tool for traumatic brain injury. J Head Trauma Rehabil. 2007;22(6):377–89. [PMID:18025970]
http://dx.doi.org/10.1097/01.HTR.0000300233.98242.87
5.
Tanielian T, Jaycox LH; RAND Corporation. Invisible wounds of war: Psychological and cognitive injuries, their consequences, and services to assist recovery. Santa Monica (CA): RAND Center for Military Health Policy Research; 2008. p. 492.
6.
Terrio H, Brenner LA, Ivins BJ, Cho JM, Helmick K, Schwab K, Scally K, Bretthauer R, Warden D. Traumatic brain injury screening: Preliminary findings in a US Army Brigade Combat Team. J Head Trauma Rehabil. 2009; 24(1):14–23. [PMID:19158592]
http://dx.doi.org/10.1097/HTR.0b013e31819581d8
8.
Owens BD, Kragh JF Jr, Wenke JC, Macaitis J, Wade CE, Holcomb JB. Combat wounds in Operation Iraqi Freedom and Operation Enduring Freedom. J Trauma. 2008;64(2): 295–99. [PMID:18301189]
http://dx.doi.org/10.1097/TA.0b013e318163b875
9.
Gondusky JS, Reiter MP. Protecting military convoys in Iraq: An examination of battle injuries sustained by a mechanized battalion during Operation Iraqi Freedom II. Mil Med. 2005;170(6):546–49. [PMID:16001610]
10.
Carroll LJ, Cassidy JD, Peloso PM, Borg J, von Holst H, Holm L, Paniak C, Pépin M; WHO Collaborating Centre Task Force on Mild Traumatic Brain Injury. Prognosis for mild traumatic brain injury: Results of the WHO Collaborating Centre Task Force on Mild Traumatic Brain Injury. J Rehabil Med. 2004;(43 Suppl):84–105. [PMID:15083873]
http://dx.doi.org/10.1080/16501960410023859
11.
Management of Concussion/mTBI Working Group. VA/DoD Clinical Practice Guideline for Management of Concussion/Mild Traumatic Brain Injury. J Rehabil Res Dev. 2009;46(6):CP1–68. [PMID:20108447]
12.
Ruff RM, Iverson GL, Barth JT, Bush SS, Broshek DK; NAN Policy and Planning Committee. Recommendations for diagnosing a mild traumatic brain injury: A National Academy of Neuropsychology education paper. Arch Clin Neuropsychol. 2009;24(1):3–10. [PMID:19395352]
http://dx.doi.org/10.1093/arclin/acp006
13.
Centers for Disease Control and Prevention [Internet]. Injury prevention & control: Traumatic brain injury. Atlanta (GA): Centers for Disease Control and Prevention; 2010 [cited 2011 May 4]. Available from: http://www.cdc.gov/concussion/signs_symptoms.html
14.
Iverson GL, Langlois JA, McCrea MA, Kelly JP. Challenges associated with post-deployment screening for mild traumatic brain injury in military personnel. Clin Neuropsychol. 2009;23(8):1299–1314. [PMID:19882473]
http://dx.doi.org/10.1080/13854040903153902
15.
U.S. Department of the Army, Office of the Surgeon General, Traumatic Brain Injury Task Force. Report to the surgeon general. Washington (DC): U.S. Army Medical Command; 2008.
16.
Bass CR, Panzer MB, Rafaels KA, Wood G, Shridharani J, Capehart B. Brain injuries from blast. Ann Biomed Eng. 2012;40(1):185–202. [PMID:22012085]
http://dx.doi.org/10.1007/s10439-011-0424-0
17.
Kay T, Harrington DE, Adams R; Mild Traumatic Brain Injury Committee. American Congress of Rehabilitation Medicine, Head Injury Interdisciplinary Special Interest Group’s definition of mild traumatic brain injury. J Head Trauma Rehabil. 1993;8:86–87.
18.
Scott SG, Belanger HG, Vanderploeg RD, Massengale J, Scholten J. Mechanism-of-injury approach to evaluating patients with blast-related polytrauma. J Am Osteopath Assoc. 2006;106(5):265–70. [PMID:16717367]
19.
Sayer NA, Nelson D, Nugent S. Evaluation of the Veterans Health Administration traumatic brain injury screening program in the upper Midwest. J Head Trauma Rehabil. 2011;26(6):454–67. [PMID:22094545]
http://dx.doi.org/10.1097/HTR.0b013e3181ff393c
20.
Donnelly KT, Donnelly JP, Dunnam M, Warner GC, Kittleson CJ, Constance JE, Bradshaw CB, Alt M. Reliability, sensitivity, and specificity of the VA traumatic brain injury screening tool. J Head Trauma Rehabil. 2011;26(6):439–53.
[PMID:21386716]
http://dx.doi.org/10.1097/HTR.0b013e3182005de3
21.
VHA Directive 2010–012. Screening and evaluation of possible traumatic brain injury in Operation Enduring Freedom (OEF) and Operation Iraqi Freedom (OIF) veterans. Washington (DC): Department of Veterans Affairs, Veterans Health Administration; 2010 Mar 8.
22.
U.S. Government Accountability Office. VA health care: Mild traumatic brain injury screening and evaluation implemented for OEF/OIF veterans, but challenges remain. Washington (DC): U.S. Government Accountability Office; 2008. p. 1–55.
23.
McAllister TW, Stein MB. Effects of psychological and biomechanical trauma on brain and behavior. Ann N Y Acad Sci. 2010;1208:46–57. [PMID:20955325]
http://dx.doi.org/10.1111/j.1749-6632.2010.05720.x
24.
Stein MB, McAllister TW. Exploring the convergence of posttraumatic stress disorder and mild traumatic brain injury. Am J Psychiatry. 2009;166(7):768–76.
[PMID:19448186]
http://dx.doi.org/10.1176/appi.ajp.2009.08101604
25.
Cicerone KD, Kalmar K. Persistent postconcussion syndrome: The structure of subjective complaints after mild traumatic brain injury. J Head Trauma Rehabil. 1995; 10(3):1–17.
http://dx.doi.org/10.1097/00001199-199510030-00002
26.
Meterko M, Baker E, Stolzmann KL, Hendricks AM, Cicerone KD, Lew HL. Psychometric assessment of the Neurobehavioral Symptom Inventory-22: The structure of persistent postconcussive symptoms following deployment-related mild traumatic brain injury among veterans. J Head Trauma Rehabil. 2012;27(1):55–62.
[PMID:22190009]
http://dx.doi.org/10.1097/HTR.0b013e318230fb17
27.
Carlson KF, Kehle SM, Meis LA, Greer N, Macdonald R, Rutks I, Sayer NA, Dobscha SK, Wilt TJ. Prevalence, assessment, and treatment of mild traumatic brain injury and posttraumatic stress disorder: A systematic review of the evidence. J Head Trauma Rehabil. 2011;26(2):103–15.
[PMID:20631631]
http://dx.doi.org/10.1097/HTR.0b013e3181e50ef1
28.
Seal KH, Bertenthal D, Miner CR, Sen S, Marmar C. Bringing the war back home: Mental health disorders among 103,788 US veterans returning from Iraq and Afghanistan seen at Department of Veterans Affairs facilities. Arch Intern Med. 2007;167(5):476–82.
[PMID:17353495]
http://dx.doi.org/10.1001/archinte.167.5.476
29.
Hoge CW, Castro CA, Messer SC, McGurk D, Cotting DI, Koffman RL. Combat duty in Iraq and Afghanistan, mental health problems, and barriers to care. N Engl J Med. 2004; 351(1):13–22. [PMID:15229303]
http://dx.doi.org/10.1056/NEJMoa040603
30.
Hoge CW, Terhakopian A, Castro CA, Messer SC, Engel CC. Association of posttraumatic stress disorder with somatic symptoms, health care visits, and absenteeism among Iraq war veterans. Am J Psychiatry. 2007;164(1): 150–53. [PMID:17202557]
http://dx.doi.org/10.1176/appi.ajp.164.1.150
31.
Iverson KM, Hendricks AM, Kimerling R, Krengel M, Meterko M, Stolzmann KL, Baker E, Pogoda TK, Vasterling JJ, Lew HL. Psychiatric diagnoses and neurobehavioral symptom severity among OEF/OIF VA patients with deployment-related traumatic brain injury: A gender comparison. Womens Health Issues. 2011;21(4 Suppl):S210–17.
[PMID:21724143]
http://dx.doi.org/10.1016/j.whi.2011.04.019
32.
Lew HL, Otis JD, Tun C, Kerns RD, Clark ME, Cifu DX. Prevalence of chronic pain, posttraumatic stress disorder, and persistent postconcussive symptoms in OIF/OEF veterans: Polytrauma clinical triad. J Rehabil Res Dev. 2009; 46(6):697–702. [PMID:20104399]
http://dx.doi.org/10.1682/JRRD.2009.01.0006
33.
Lew HL, Vanderploeg RD, Moore DF, Schwab K, Friedman L, Yesavage J, Keane TM, Warden DL, Sigford BJ. Overlap of mild TBI and mental health conditions in returning OIF/OEF service members and veterans. J Rehabil Res Dev. 2008;45(3):xi–xvi. [PMID:18629743]
34.
Polusny MA, Kehle SM, Nelson NW, Erbes CR, Arbisi PA, Thuras P. Longitudinal effects of mild traumatic brain injury and posttraumatic stress disorder comorbidity on postdeployment outcomes in national guard soldiers deployed to Iraq. Arch Gen Psychiatry. 2011;68(1):79–89.
[PMID:21199967]
http://dx.doi.org/10.1001/archgenpsychiatry.2010.172
35.
Sayer NA, Rettmann NA, Carlson KF, Bernardy N, Sigford BJ, Hamblen JL, Friedman MJ. Veterans with history of mild traumatic brain injury and posttraumatic stress disorder: Challenges from provider perspective. J Rehabil Res Dev. 2009;46(6):703–16. [PMID:20104400]
http://dx.doi.org/10.1682/JRRD.2009.01.0008
36.
Wilk JE, Herrell RK, Wynn GH, Riviere LA, Hoge CW. Mild traumatic brain injury (concussion), posttraumatic stress disorder, and depression in U.S. soldiers involved in combat deployments: Association with postdeployment symptoms. Psychosom Med. 2012;74(3):249–57.
[PMID:22366583]
http://dx.doi.org/10.1097/PSY.0b013e318244c604
37.
American Psychiatric Association. Diagnostic and statistical manual of mental disorders: DSM-IV-TR. Washington (DC): American Psychiatric Association; 2000.
38.
Bryant R. Post-traumatic stress disorder vs traumatic brain injury. Dialogues Clin Neurosci. 2011;13(3):251–62.
[PMID:22034252]
39.
Bryant RA. Acute stress disorder as a predictor of posttraumatic stress disorder: A systematic review. J Clin Psychiatry. 2011;72(2):233–39. [PMID:21208593]
http://dx.doi.org/10.4088/JCP.09r05072blu
40.
Bryant RA, Friedman MJ, Spiegel D, Ursano R, Strain J. A review of acute stress disorder in DSM-5. Depress Anxiety. 2011;28(9):802–17. [PMID:21910186]
http://dx.doi.org/10.1002/da.20737
41.
Brenner LA, Ivins BJ, Schwab K, Warden D, Nelson LA, Jaffee M, Terrio H. Traumatic brain injury, posttraumatic stress disorder, and postconcussive symptom reporting among troops returning from Iraq. J Head Trauma Rehabil. 2010;25(5):307–12. [PMID:20042982]
http://dx.doi.org/10.1097/HTR.0b013e3181cada03
42.
Taylor BC, Hagel EM, Carlson KF, Cifu DX, Cutting A, Bidelspach DE, Sayer NA. Prevalence and costs of co-occurring traumatic brain injury with and without psychiatric disturbance and pain among Afghanistan and Iraq War Veteran V.A. users. Med Care. 2012;50(4):342–46.
[PMID:22228249]
http://dx.doi.org/10.1097/MLR.0b013e318245a558
43.
Vanderploeg RD, Belanger HG, Curtiss G. Mild traumatic brain injury and posttraumatic stress disorder and their associations with health symptoms. Arch Phys Med Rehabil. 2009;90(7):1084–93. [PMID:19577020]
http://dx.doi.org/10.1016/j.apmr.2009.01.023
44.
Corrin A, Tuutti C. Congress, GAO take aim at DOD-VA health records program. The Business of Federal Technology. 2013 Mar 7.
This article and any supplementary material should be cited as follows:
Pogoda TK, Iverson KM, Meterko M, Baker E, Hendricks AM, Stolzmann KL, Krengel M, Charns MP, Amara J, Kimerling R, Lew HL. Concordance of clinician judgment of mild traumatic brain injury history with a diagnostic standard. J Rehabil Res Dev. 2014;51(3): 363–76.
http://dx.doi.org/10.1682/JRRD.2013.05.0115
ResearcherID/ORCID: Terri K. Pogoda, PhD: F-6243-2012
iThenticateCrossref

Go to TOP

Last Reviewed or Updated  Wednesday, June 11, 2014 1:52 PM

Valid HTML 4.01 Transitional