Journal of Rehabilitation Research & Development (JRRD)

Quick Links

  • Health Programs
  • Protect your health
  • Learn more: A-Z Health
Veterans Crisis Line Badge
 

Volume 49 Number 9, 2012
   Pages 1377 — 1398

Effectiveness of computerized oculomotor vision screening in a military population: Pilot study

José E. Capó-Aponte, OD, PhD;1* Aaron K. Tarbett, OD;2 Thomas G. Urosevich, OD;1 Leonard A. Temme, PhD;1 Navjit K. Sanghera, OD;3 Melvyn E. Kalich, PhD, OD1

1Visual Sciences Branch, U.S. Army Aeromedical Research Laboratory, Fort Rucker, AL; 2Optometry Services, Walter Reed National Military Medical Center, Bethesda, MD; 3Clinical Services, Illinois College of Optometry, Chicago, IL

Abstract — The prevalence of oculomotor dysfunctions associated with blast-induced mild traumatic brain injury (mTBI) in warfighters has increased as a consequence of recent conflicts. This study evaluated the effectiveness of computerized oculomotor vision screening (COVS) in a military population. Oculomotor functions were assessed with COVS and by conventional methods in 20 U.S. military personnel with and 20 without mTBI. The validity of COVS was determined by Pearson correlation and Bland-Altman method or the kappa coefficient. The repeatability of the COVS was assessed with the coefficient of repeatability or the kappa coefficient. The results showed that COVS had high sensitivity and specificity for screening near oculomotor functions. Overall, the COVS showed excellent validity and repeatability for assessing near lateral and vertical phorias, Worth 4 Dot, and fixation, as well as pursuit and saccadic eye movements. Despite the strong Pearson correlation, the Bland-Altman analysis identified minor to moderate discrepancies for both positive and negative fusional vergence and their associated recovery as well as for the monocular accommodative facility measurements. This study demonstrated that non-eye-care professionals may be able to use the COVS as a tool to efficiently screen oculomotor functions in a military population with or without mTBI.

Key words: accommodation, binocular vision, computerized oculomotor vision screening, mTBI, oculomotor function, TBI, traumatic brain injury, vergence, version, vision.

Abbreviations: AOA = amplitude of accommodation, BVA = Binocular Vision Assessment, CISS = Convergence Insufficiency Symptoms Survey, COR = coefficient of repeatability, COVS = computerized oculomotor vision screening, cpm = cycles per minute, D = diopter, LOA = limit of agreement, MANOVA = multivariate analysis of variance, mTBI = mild traumatic brain injury, NFV = negative fusional vergence, NPC = near point of convergence, PD = prism diopter, PFV = positive fusional vergence, RDS = random dot stereogram, SD = standard deviation, USAARL = U.S. Army Aeromedical Research Laboratory, wpm = words per minute, WRAMC = Walter Reed Army Medical Center.
*Address all correspondence to José E. Capó-Aponte, OD, PhD; Visual Sciences Branch, U.S. Army Aeromedical Research Laboratory, Fort Rucker, AL 36362; 334-255-7933; fax: 334-255-6873.
Email: jose.capoaponte@us.army.mil

As a consequence of the current conflicts in Iraq and Afghanistan, the prevalence of visual dysfunctions attributable to mild traumatic brain injury (mTBI) in U.S. warfighters has dramatically increased [1–8]. mTBI includes concussive head injury and other nonpenetrating head injuries resulting from external force and/or acceleration-deceleration mechanisms (including coup and contre-coup). Such trauma may be the result of a blast, fall, direct impact, or motor vehicle accident and is often associated with an alteration in mental status.

Several studies found oculomotor (i.e., version, vergence, and accommodation) dysfunctions to be the most common visual deficits associated with mTBI [3–6,8–9]. Among these oculomotor dysfunctions are reduced fusional amplitude ranges, abnormal vertical and lateral phorias, convergence insufficiency, diplopia with reduced recovery capability, fixation deficits, nystagmus, inadequate visual pursuit, reduced saccadic velocity, and accommodative deficiencies [3–7,10–11]. Individuals with mTBI who have such oculomotor dysfunctions may experience any of a variety of visual symptoms such as visual confusion (i.e., perception of superimposed images), reduced ability to tolerate a busy visual environment, increased reaction time, decreased attention, and delayed image interpretation, which can produce severe visual performance deficits such as slowed or disrupted reading and impaired visual search [3–7,10–12]. Since many cases of mTBI have a subtle presentation, the associated oculomotor problems may go undiagnosed or misdiagnosed, leading to continued degraded visual abilities.

Oculomotor problems can be crucial for military personnel in combat since their lives and safety, as well as those of others, depend on accurate and rapid situational awareness and environmental perception. Prompt and accurate diagnosis and management of mTBI generally increases an individual’s prognosis for neurological recovery [13–15] and safe return to duty [16–18]. Therefore, the accurate assessment of oculomotor functions is essential for providing an adequate treatment regimen, thereby ensuring that warfighters with mTBI achieve proper recovery of their oculomotor problems before returning to duty [10–11,19–25]. Premature return to duty places warfighters at greater risk of disability if they suffer additional concussive trauma [26]. Fortunately, most oculomotor dysfunctions resulting from mTBI can be ameliorated with noninvasive rehabilitative intervention such as vision therapy (i.e., neurovisual rehabilitation) and/or such passive treatment such as a spectacle prescription with prismatic lenses or other optical devices [11,20–36].

The detection of oculomotor dysfunction remains a challenge, particularly in the military context. Oculomotor dysfunctions, predominantly those affecting near vision, are relatively common in warfighters with mTBI, and many warfighters with these visual deficits go undiagnosed and untreated if a complete oculomotor assessment is not performed [3,6,9,22]. However, only a trained eyecare provider (i.e., optometrist or ophthalmologist) can perform a conventional oculomotor examination, a situation that limits the number of warfighters who can be evaluated for oculomotor dysfunctions before (baseline), during, and after deployment. Given the limited amount of time available to military eye-care providers for a comprehensive eye examination, it would be helpful if some of the oculomotor examination could be delegated to ophthalmic technicians. Furthermore, the absence of standardized methodology among military eye-care clinics for the assessment of oculomotor functions makes the direct comparison of clinical findings over time, particularly pre- and postdeployment, problematic.

Despite the surge in automation of today’s ophthalmic instrumentation, automated comprehensive oculomotor assessment is not available, possibly because there has been little to no economic incentive for ophthalmic equipment manufacturers to develop such technology. A new generation of recently available commercial software designed to screen oculomotor functions has the potential to overcome many of these shortcomings.

The present study evaluated the effectiveness of computerized oculomotor vision screening (COVS) by determining its validity, repeatability, sensitivity, and specificity compared with conventional oculomotor testing methodology. An effective COVS that can be administered by non-eye-care personnel would allow timely identification of oculomotor dysfunctions in warfighters with mTBI and would facilitate possible referrals for a comprehensive oculomotor examination as warranted. Automation of oculomotor function screening can potentially provide a standardized screening tool that will reduce the amount of clinic chairtime per patient while making the oculomotor evaluation of warfighters suspected of having mTBI easier and more effective. In addition to expediting the identification and management of oculomotor deficits, COVS can be useful for monitoring the recovery of warfighters with oculomotor problems and assisting healthcare providers to accurately determine return to duty status.

A total of 40 U.S. military personnel participated in this study; half were recruited from the U.S. Army ­Aeromedical Research Laboratory (USAARL) located at Fort Rucker, Alabama, and the other half were recruited from the Walter Reed Army Medical Center (WRAMC) in Washington, DC. Those recruited and tested at USAARL (14 male and 6 female; mean ± standard deviation [SD] age 32.65 ± 8.94 yr) had no history of mTBI or blast exposure, whereas the 20 subjects recruited and tested at WRAMC (18 male and 2 female; age 29.70 ± 7.68 yr) did have a history of blast-induced mTBI for which they were receiving medical care.

Exclusion criteria included monocularity (e.g., enucleation or no light perception), amblyopia, anisometropia greater than 2.0 diopters (Ds) in any meridian, ocular trauma, or polytrauma that would prevent performing the oculomotor examination. The WRAMC subjects had a documented history of mTBI based on the criteria of the American Congress of Rehabilitation Medicine: (1) a duration of a loss of consciousness of no more than 30 min at the time of the mTBI, (2) a duration of posttraumatic amnesia of no more than 24 h, (3) a Glasgow Coma Scale score of 13 to 15, and (4) alteration of mental state (i.e., evidence of confusion or disorientation) [12].

One of two optometrists (coauthors JEC and TGU) performed a comprehensive eye examination without dilation on all subjects. The optometrists used the identical testing protocol to determine manifest refraction error, distance high contrast best corrected visual acuity measured with a Snellen projector chart, color vision measured with Dvorine pseudoisochromatic plates, confrontation visual fields, intraocular pressure determined using noncontact tonometry, and ocular health status evaluated using biomicroscopy and direct ophthalmoscopy. Since all subjects had normal dilated fundus examinations documented in their electronic medical records as recently as 45 d prior to being enrolled in the study, the undilated examination screened the subjects for recent ocular disease that could confound the interpretation of study results. The manifest refraction was used in the conventional oculomotor and COVS examinations.

The conventional oculomotor examination included tests of oculomotor functions that were assessed by COVS as well as tests that were not. Conventional oculomotor assessments that had no counterpart in COVS were nonetheless important for characterizing the subject’s oculomotor status during the study. These components of the examination were stereo acuity as measured by random dot stereograms (RDSs) using the Randot stereotest, near point of convergence (NPC), ocular alignment at distance and at near determined by cover test, distance phoria (lateral and vertical) and gradient accommodative convergence/accommodation ratio using the von Graefe technique, and the amplitude of accommodation (AOA) measured by phorometry using the minus lens to blur method. The near oculomotor functions assessed with both the conventional methods and COVS were Worth 4 Dot, lateral and vertical phorias, maximum negative fusional vergence (NFV) and positive fusional vergence (PFV) with its associated recovery measured with the von Graefe technique, fixation disparity tested with the Borish near point card (separate nonious lines through polarized lenses), monocular accommodative facility measured with +2.00/–2.00 D lenses flippers, and saccadic and smooth pursuit eye movements assessed by the Northeastern State University College of Optometry Oculomotor Test. The comprehensive eye examination and conventional oculomotor assessment were conducted using standard clinical procedures [37–39].

Following the vision and conventional oculomotor examination, a COVS practice run was performed to ensure that each subject was experienced with the experimental procedures and understood what the methodology entailed. Following this familiarization, all subjects had their oculomotor functions measured three times by COVS, with 15 min rest breaks between test-retest sessions. The purpose and instructions for each COVS test were described to the volunteer before testing. The COVS was administered by one of two medical technicians who were blinded to the results of the conventional oculomotor evaluation.

The COVS system consisted of a metal baseplate that supported a linear translation stage, chinrest, and monitor jacks (Figure 1). The computer monitor was mounted on the monitor jack positioned on top of the linear translation stage. A chinrest system was mounted on the chinrest jack. The linear translation stage ensured the proper testing distance of 40 cm from the monitor pixel plane to the front surface of the eyes. The monitor and chinrest jacks allowed vertical adjustments to ensure the eyes were aligned with the center of the monitor. Stimuli were displayed on a 19 in. monitor (model 1908FP, Dell Inc; Round Rock, Texas) with a native resolution of 1,280 × 1,024 controlled by a laptop computer (Latitude D630, Dell Inc). The average screen luminance was about 35 cd/m2 measured with a spot meter (Minolta CS-100A, Konica Minolta Business Solutions; Ramsey, New Jersey). The subjects wore red-blue anaglyphic lenses for all of the tests, except for those involving saccadic and smooth pursuit eye movements. The anaglyphic lenses provided dichoptic presentation such that each eye viewed a separate and independent field of stimuli. In order to obtain the best cancellation to eliminate crosstalk or ghosting while subjects wore the red-blue anaglyphic lenses, the color setting of the monitor was adjusted to 45 percent red, 45 percent green, and 100 percent blue, with the test administered in subdued room illumination.

During COVS, the subject was comfortably seated with his or her chin positioned on the chinrest and with his or her forehead against the headrest (Figure 1). A trial lens frame held the lenses containing both the manifest refraction and red-blue anaglyphic lenses. Subject responses were made using a joystick game controller (Attack 3, Logitech; Newark, California).


Figure 1. Computerized oculomotor vision screening experimental setup with subject aligned with center of monitor located at 40 cm from subject’s eyes. Trial lens frame with best correction and red-blue anaglyph lenses were worn for data collection. Subject used joystick game controller to respond to individual tests.

Figure 1.

Computerized oculomotor vision screening experimental setup with subject aligned with center of monitor located at 40 cm from subject’s eyes. Trial lens frame with best correction and red-blue anaglyph lenses were worn for data collection. Subject used joystick game controller to respond to individual tests.

Click Image to Enlarge. View as PowerPoint Slide

The COVS system used a modified version of the commercially available Binocular Vision Assessment (BVA) software (Home Therapy Systems; Gold Canyon, Arizona), presently labeled as BVA2 (Figure 2(a)). The BVA2 consisted of the original BVA tests augmented with tests that are part of two other Home Therapy Systems Inc. software systems, VTS3 Computer Orthoptics and ADR iNet Dynamic Reader. The VTS3 Computer Orthoptics software and the BVA software were designed to screen oculomotor functions as part of a vision training package, whereas the ADR iNet Dynamic Reader software was designed to assess reading speed and comprehension (i.e., reading fluency) as part of a reading training package. The software controlled the display and test procedures based on the subject responses. In addition, the size of all the stimuli was automatically controlled by the software based on the monitor size chosen at the initial setup.


Figure 2. Computerized oculomotor vision screening display view for (a) Binocular Vision Assessment 2 test selection screen, (b) Worth 4 Dot, (c) motor field (lateral and vertical phorias), (d) fusional ranges (negative and positive fusional vergence amplitude), (e) fixation disparity, (f) monocular accommodation facility, and (g) saccadic and (h) smooth pursuit eye movements.

Figure 2.

Computerized oculomotor vision screening display view for (a) Binocular Vision Assessment 2 test selection screen, (b) Worth 4 Dot, (c) motor field (lateral and vertical phorias), (d) fusional ranges (negative and positive fusional vergence amplitude), (e) fixation disparity, (f) monocular accommodation facility, and (g) saccadic and (h) smooth pursuit eye movements.

Click Image to Enlarge. View as PowerPoint Slide

Ciuffreda et al. described that the presence of oculomotor dysfunctions resulting from brain injury can produce performance deficits such as slowed reading and impaired visual search [40]. Accordingly, we used the Reading Level Test from the ADR iNet Dynamic Reader software to assess reading fluency. This allowed a functional screening of the integrated oculomotor system (i.e., vergence, version, and accommodative components). Particularly, reading speed and comprehension (i.e., fluency) were assessed because these parameters are acutely sensitive to the integration of oculomotor functions [21,25]. Reading fluency is the ability to read text rapidly and accurately and is the bridge between word recognition and comprehension. The software allows the examiner to adjust the font size and reading grade level tested. Furthermore, the BVA2 includes the 15-question Convergence Insufficiency Symptoms Survey (CISS). This questionnaire has been validated and standardized for the randomized clinical trials (Convergence Insufficiency Treatment Trial Study Group) to subjectively measure the recovery of near vision symptoms in adults [27–28,41]. Each symptom question had five possible answers with an associated value, where 4 = always, 3 = frequently, 2 = sometimes, 1 = rarely, and 0 = never. Thus, the cumulative symptoms score can vary from 0 to 60. An average normal adult should score <21 points. The following describes each of the COVS individual test components.

Binocular fusion requires both motor fusion and sensory fusion. Motor fusion is a prerequisite for sensory fusion; however, complete sensory fusion does not always follow motor fusion. There are three degrees of sensory fusion: (1) simultaneous perception and superimposition, (2) flat fusion, and (3) stereopsis. The Worth 4 Dot test is used to assess flat fusion and determine whether the individual experiences fusion, diplopia, or suppression. Similar to the clinical “flashlight” version of the Worth 4 Dot test, the stimuli consisted of four circles (one white, one blue, and two red), each of which subtended 0.72° (Figure 2(b)). The stimuli were presented on a black background in a diamond configuration covering a 3.58° area at a viewing distance of 40 cm. The subject wore red and blue filters over the right and left eye, respectively, so that the right eye saw one blue and one white dot and the left eye saw two red dots and one white dot. The subject was instructed to report the number of dots displayed on the screen. The examiner pressed the number key corresponding to the subject’s answer. The Worth 4 Dot test was the first test to be administered and ensured that the subject did not have diplopia, which would have prevented a meaningful oculomotor evaluation. The outcome measure was a pass (i.e., normal fusion) or fail (suppression or diplopia) score.

Lateral and vertical phorias, assessed under the COVS motor field testing option (Figure 2(a)), were measured in nine cardinal positions of gaze. The stimuli consisted of a red cross and a blue ring, both of which subtended 1.15° in height and diameter presented on a black background (Figure 2(c)). During the testing, the subject wore red and blue filters over the right and left eye, respectively. The subject was instructed to use the joystick to move a red cross so that it appeared to be exactly inside a blue ring. The subject reported when the alignment was achieved and the examiner advanced the test to the next gaze position on the screen. The test continued in the same fashion until all nine gaze positions were tested. The software allowed the examiner to set the fixating eye (i.e., right or left) and to set the speed of the joystick to allow better control of the responses for those subjects with poor hand dexterity or decreased reaction time. In this study, all subjects were tested using the right eye as the fixating eye and the joystick speed was set to level 2 (on a scale of 1 = slow and 4 = fast). The outcome measures were the near lateral and vertical phorias in prism diopters (PDs) in each of the nine gazes. The measurement in primary gaze was used to determine the validity and repeatability of the COVS. A diagram depicting a deviation at the nine gazes was also generated as a graphical representation of a suspected paretic muscle.

The screenings for PFV and NFV amplitudes were assessed under the COVS fusional ranges testing option (Figure 2(a)). Computer-generated right and left stimuli created vergence disparity RDSs that moved horizontally in opposite directions to create a divergence or convergence demand with an increase in vergence demand of 1 PD on the next schedule presentation [42]. Bifoveal fixation is required to discriminate the location of the small RDS, thus ensuring both appropriate sensory and motor fusion. The RDS inner square subtending an angle of 5.01° was located inside a larger square formed by dot patterns subtending an angle of 16.36° presented on a black background (Figure 2(d)). The subject wore red and blue filters over the right and left eye, respectively. The subject was instructed to locate a smaller square formed by RDSs inside the larger square and to indicate the position of the inner square by moving the joystick in the corresponding direction. As the test progressed, the fusional demand required to maintain fusion of the RDS gradually increased until the subject could no longer discriminate the location of the small square within the larger square. This indicated the maximum NFV amplitude (i.e., break point). When the subject was unable to discriminate the location of the small square, the examiner gradually decreased the fusional amplitude demand until the subject regained fusion, as indicated by the subject’s ability to discern the location of the small square again (i.e., recovery point). Maximum fusional amplitudes and recovery values were automatically recorded. After NFV (i.e., base-in) testing, PFV (i.e., base-out) testing was similarly performed. The outcome measures were the maximum near NFV and PFV amplitudes required to maintain fusion (i.e., break) or to recover fusion (i.e., recovery) measured in PD.

Similar to the fixation disparity stimuli used for the conventional test (i.e., Borish near point card), the stimuli displayed on the screen consisted of vertical and horizontal nonious lines forming a cross, in which each eye only saw half of each stimulus (red or blue), subtending 2.15° in total height and width presented on a black background (Figure 2(e)). The stimuli were centered in a red square border viewed binocularly that subtended 5.72°. The subject wore red and blue filters over the right and left eye, respectively. He or she was instructed to report whether the arms of the cross were perfectly aligned with each other or not. The outcome measure was a pass or fail score based on the subject response. Normal fixation (pass) was indicated when the subject reported that the arms of the cross were perfectly aligned with each other. The examiner entered the subject’s response using the keyboard.

The stimuli for the monocular accommodative facility test consisted of four red or blue Landolt Cs (0.29° diameter) centered in a square border (0.72° width) arranged horizontally and presented on a black background (Figure 2(f)). For this test, the subject wore red and blue filters over the right and left eye, respectively, as well as a +1.50 D lens in front of the right eye and a −1.50 D lens in front of the left eye. This resulted in monocular presentation of the stimuli with the right eye relaxing accommodation and the left eye accommodating for the lens. The subject was instructed that the test was a timed event and to keep all the letters clear (i.e., in focus and single) during the test. Similar to the conventional flipper accommodation facility test, the subject activated (by looking through a minus lens) or relaxed (by looking through a plus lens) accommodation to reduce the blur sufficiently to identify the location of the break in a Landolt C (i.e., up, down, left, or right). The subject was instructed to use the joystick to indicate the location of the break in a Landolt C beginning with the first square located on the left of the screen. As each response was made, the square containing the Landolt C disappeared and the subject responded to the next stimulus to the right. Once the subject had responded to all four stimuli, the computer advanced to the next screen, where the other eye was evaluated. Since this is a monocular test, each eye was evaluated separately by using different color filters (i.e., red for right eye and blue for left eye). The test ran for 60 s, alternating between eyes. The outcome measure was the number of cycles the subject could discriminate the orientation of the Landolt C alternating between +1.50 D and −1.50 D lenses, reported in cycles per minute (cpm).

The test displayed a randomly placed blue arrow (1.58° × 0.72°) pointing either up, down, left, or right; surrounded by a red square border (1.86°); and presented on a black background (Figure 2(g)). The subject was instructed to use the joystick to indicate, as rapidly and accurately as possible, the direction that the arrow was pointing. The subject was also instructed that the test was a timed event and to move only the eyes during the testing, not the head. Each response caused the arrow and the red square to randomly jump to a different location on the screen and to change the direction in which the arrow was pointing. The test ran for 60 s. The outcome measure was a pass or fail score based on the percentage of the correct responses (i.e., correctly identified the direction of the arrow) and the average response time (i.e., time to complete response) in seconds. Saccadic eye movements were considered within normal limits if the percentage of the correct responses was ≥80 percent and if the average response time was ≤1.20 s, as described in the product manual instructions. However, well-developed normative saccadic eye movement data obtained with this method have not yet been published.

The test displayed a continuously moving letter E (1.43° × 1.15°) that was randomly oriented up, down, left, or right on a black background (Figure 2(h)). The subject was instructed to use the joystick to indicate the orientation of the letter E as rapidly and accurately as possible. He or she was also instructed that the test was a timed event and to move only the eyes during the testing, not the head. With each response, the letter E changed orientation randomly as it continuously moved throughout the screen. The test ran for 60 s. Similar to the saccadic eye movement, the outcome measure was a pass or fail score based on the percentage of the correct responses (i.e., correctly identified the orientation of the E) and the average response time (i.e., time to complete response) in seconds. Also similar to the saccadic screening, pursuit eye movements were considered within normal limits if the percentage of the correct responses was ≥80 percent and if the average response time was ≤1.20 s, as described in the product manual instructions. However, well-developed normative smooth pursuit eye movement data obtained with this method have not yet been published.

The subject was instructed to read silently a short story displayed in 12-point font on the computer screen. The subject was also instructed to read the story as rapidly as possible while maintaining comprehension. After reading the story, the subject answered 10 questions about the story as a measure of reading comprehension. The results for the reading speed and comprehension component of the test were automatically recorded. All subjects were tested with materials designed to assess a seventh grade reading level. The outcome measure was a pass or fail score based on the number of words per minute (wpm) read by the subject and the score of the reading comprehension test (0 to 100). The expected reading speed and reading comprehension score at the seventh grade reading level are ≥197 wpm and ≥80 percent, respectively.

After completing all three COVS sessions, the subject was asked to complete the 15-question CISS displayed on the screen by selecting the frequency of the particular visual problem [28]. The results of the survey were automatically scored and saved. The outcome measure was a survey total score. The subject failed the test if he or she scored >20.

A within-subject, repeated-measures correlational design was used to evaluate the effectiveness (i.e., validity, repeatability, sensitivity, and specificity) of the COVS to screen oculomotor functions. The results of the COVS were compared with results of the conventional oculomotor evaluation performed by an optometrist.

Descriptive statistics (mean ± SD) were calculated for most of the outcome measures. Inferential statistics (Student pair t-test or multivariate analysis of variance [MANOVA]) were used to examine the difference between measurements obtained by the two methods. Scatter plots and Pearson correlation coefficients were used to demonstrate the relationship between the oculomotor measurements obtained by both techniques. Based on its correlation coefficient (r) value, the correlation strength was interpreted as very strong (≥0.80), moderately strong (0.79–0.60), fair (0.59–0.30), or poor (<0.30) [43]. In addition, the Bland-Altman analysis of limits of agreement (LOAs) was used to further compare the results of the COVS and conventional testing for oculomotor functions scored on a ratio scale [44]. The Bland-Altman plot is useful to reveal the relation between the differences and the averages, uncover any systematic biases, and identify possible outliers, all of which might be missed with conventional correlations analysis. The validity of the COVS oculomotor assessments producing categorical data (pass or fail) was determined by the kappa coefficient, a measure of agreement between multiple observations [45]. Based on its magnitude, the kappa coefficient was interpreted as almost perfect (0.81–1.00), substantial (0.61–0.80), moderate (0.41–0.60), fair (0.21–0.40), slight (0–0.20), and poor (<0).

The test-retest repeatability of the COVS measures between sessions was assessed using the Bland-Altman method and the coefficient of repeatability (COR). The COR was calculated by multiplying the SD of the differences between sessions by 1.96 (i.e., between sessions 1 and 2, 1 and 3, and 2 and 3). The measurements were compared using a within-subjects repeated MANOVA to determine whether the values were significantly different between sessions [46]. Eta values were calculated to estimate the size of the effect attributable to repeated COVS. The repeatability of the COVS for oculomotor functions producing categorical data (pass or fail) was determined by kappa coefficient and interpreted as described previously.

Analytical measures of sensitivity and specificity, as well as the rates of false positives and false negatives, were calculated to assess the clinical effectiveness of the COVS. The sensitivity was calculated by dividing the number of true positives by the combined number of true positives and false negatives. The specificity was calculated by dividing the number of true negatives by the combined number of true negatives and false positives. All significance levels were p < 0.05. Statistical analyses were performed with SPSS version 11.0 (IBM Corporation; Armonk, New York) and GraphPad Prism (GraphPad Software Inc; La Jolla, California) software.

All subjects were correctable to 20/20 in both eyes. They also had normal color vision, intraocular pressure, and ocular health. Table 1 summarizes oculomotor functions assessed only by conventional oculomotor evaluation. These oculomotor functions were measured to characterize the subjects’ complete oculomotor status, particularly distance-vision oculomotor functions since these were not evaluated by the COVS. There were statistically significant differences between the mTBI and the non-mTBI groups for NPC and monocular AOA.


Table 1. 
mTBI
Group
p-Value
0.66
0.01*
0.50 ± 2.24
0.21
0.60 ± 1.85
0.13
0.65 ± 3.28
0.27
0.59
0.23
0.03*
0.02*
*Statistically significant (p < 0.05).

Table 2 summarizes the mean ± SD of the expected normative values and observed values for those oculomotor functions assessed by the conventional evaluation and COVS with outcome measures scored on a ratio scale. Note that although the COVS was conducted three times in order to assess repeatability (discussed later), the comparison between the COVS and the conventional evaluation of oculomotor function was limited to the first COVS. The results show that there were no statistically significant differences (t-test) between the results obtained by these two methods for lateral phoria, vertical phoria, NFV, and its associated recovery; however, there were statistically significant differences between the results obtained by the two methods for PFV (p < 0.001) and its recovery (p < 0.001) as well as monocular accommodative facility (p < 0.001).


Table 2. 
3 ± 5
1.68 ± 3.68
3 ± 3
17
10
15
20
16
12
12
*Statistically significant (p < 0.001).

To assess the validity of the COVS compared with the conventional evaluation, scatter and Bland-Altman plots with 95 percent LOA were performed for each of the oculomotor function measurements scored on a ratio scale. The scatter plots show the regression lines for the mTBI and the non-mTBI groups separately and for the two groups combined. Each Bland-Altman graph is a plot of the difference against the mean; that is, for each ­subject, the difference in the measurements made by the COVS and the conventional method is plotted as a function of the mean of the measurements made by the two methods. Except when indicated, the differences were calculated such that negative values indicate higher reading (i.e., overestimation) and positive values indicate lower reading (i.e., underestimation) by COVS of the particular oculomotor function measurements. Each plot shows horizontal lines at the mean difference (i.e., bias) as well as at the upper and lower LOAs, which are defined as the mean difference plus and minus 1.96 times the SD. In addition, for completeness, regression lines for the mTBI and the non-mTBI groups are shown separately as well as for the two groups combined. The regression lines indicate the extent to which the difference between the two methods depends on the magnitude of the measurement; if the slope does not differ significantly from 0, the difference between the two methods is independent of the magnitude of the measurement. Note that because of overlapping data points, the total number of plotted points in each graph might seem to be less than the total number of volunteers (n = 40).

Figure 3(a) demonstrates the relationship between the lateral phoria results obtained with COVS and conventional testing for all subjects combined as well as individual groups. The regression line plotted for each comparison demonstrates very strong correlations between the methods (combined: r = 0.936; mTBI: r = 0.944; non-mTBI: r = 0.914). Figure 3(b) shows the Bland-Altman plot for lateral phoria. Consistent with the correlation analysis, there was excellent agreement between the two methods as indicated by the relative low mean difference and the narrow LOA range (combined: −0.63 ± 1.30, range = −3.19 to 1.92; mTBI: −0.82 ± 1.28, range = −3.32 to 1.68; non-mTBI: −0.65 ± 1.35, range = −3.18 to 2.11). The slopes of the regression lines were not significantly different than 0 (combined: m [slope] = 0.11, p = 0.08; mTBI: m = 0.31, p = 0.12; non-mTBI: m = 0.08, p = 0.44), which indicates that the difference remained constant across the range of measurements. COVS had a high sensitivity (83%) and specificity (100%) as a tool to screen lateral phorias (Table 3).


Figure 3. Computerized oculomotor vision screening (COVS) validity for measuring lateral phoria. (a) Lateral phoria measures of mild traumatic brain injury (mTBI) (􀀀) and non-mTBI (×) subjects are presented on scatter plot showing regression line for both groups combined (fine solid line), mTBI group (fine dashed lines), and non-mTBI group (fine dotted line). (b) Bland-Altman plot shows horizontal lines at mean difference (bold dotted line); upper and lower 95 percent limits of agreement (LOAs) (bold dashed lines); and regression line for both groups combined (fine solid line), mTBI group (fine dashed lines), and non-mTBI group (fine dotted line). Minus signs represent exophoria. Area between two vertical dashed lines indicates expected normative value range. PD = prism diopter.

Figure 3.

Computerized oculomotor vision screening (COVS) validity for measuring lateral phoria. (a) Lateral phoria measures of mild traumatic brain injury (mTBI) (􀀀) and non-mTBI (×) subjects are presented on scatter plot showing regression line for both groups combined (fine solid line), mTBI group (fine dashed lines), and non-mTBI group (fine dotted line). (b) Bland-Altman plot shows horizontal lines at mean difference (bold dotted line); upper and lower 95 percent limits of agreement (LOAs) (bold dashed lines); and regression line for both groups combined (fine solid line), mTBI group (fine dashed lines), and non-mTBI group (fine dotted line). Minus signs represent exophoria. Area between two vertical dashed lines indicates expected normative value range. PD = prism diopter.

Click Image to Enlarge. View as PowerPoint Slide


Table 3. 
Sensitivity %
(TP/TP+FN)
Specificity %
(TN/TN+FP)
83 (10/12)
100 (28/28)
86 (12/14)
100 (26/26)
83 (10/12)
100 (28/28)
100 (5/5)
97 (34/35)
100 (8/8)
97 (31/32)
100 (8/8)
94 (30/32)
90 (9/10)
95 (28/30)
100 (2/2)
100 (38/38)
100 (2/2)
100 (38/38)
100 (6/6)
97 (33/34)
100 (12/12)
97 (27/28)

Figure 4(a) shows a strong correlation (r = 0.935) between the vertical phoria results obtained with COVS and conventional testing. Note the two outliers at the extreme right of the graphs. In order to assess the effect of these two suspect data points on the correlation, it was recomputed with the two observations excluded. The ­correlation between the methods was still very strong (r = 0.818, p < 0.001). The Bland-Altman plot (Figure 4(b)) shows that there was an excellent agreement between the two methods as indicated by the narrow LOA range. However, the COVS tended to slightly overestimate left hyperdeviation and underestimate right hyperdeviation as the magnitude of the measurements increased. COVS has a relatively high sensitivity (86%) and specificity (100%) as a tool to assess vertical phorias (Table 3).


Figure 4. Computerized oculomotor vision screening (COVS) validity for measuring vertical phoria. (a) Vertical phoria measures of mild traumatic brain injury (mTBI) (􀀀) and non-mTBI (×) subjects are presented on scatter plot showing regression line for both groups combined (fine solid line), mTBI group (fine dashed lines), and non-mTBI group (fine dotted line). (b) Bland-Altman plot shows horizontal lines at mean difference (bold dotted line); upper and lower 95 percent limits of agreement (LOAs) (bold dashed lines); and regression line for both groups combined (fine solid line), mTBI group (fine dashed lines), and non-mTBI group (fine dotted line). Minus and plus signs represent left and right hyperphoria, respectively. Area between two vertical dashed lines indicates expected normative value range. PD = prism diopter.

Figure 4.

Computerized oculomotor vision screening (COVS) validity for measuring vertical phoria. (a) Vertical phoria measures of mild traumatic brain injury (mTBI) (􀀀) and non-mTBI (×) subjects are presented on scatter plot showing regression line for both groups combined (fine solid line), mTBI group (fine dashed lines), and non-mTBI group (fine dotted line). (b) Bland-Altman plot shows horizontal lines at mean difference (bold dotted line); upper and lower 95 percent limits of agreement (LOAs) (bold dashed lines); and regression line for both groups combined (fine solid line), mTBI group (fine dashed lines), and non-mTBI group (fine dotted line). Minus and plus signs represent left and right hyperphoria, respectively. Area between two vertical dashed lines indicates expected normative value range. PD = prism diopter.

Click Image to Enlarge. View as PowerPoint Slide

Similarly, there was a strong correlation (r = 0.862) between the methods for NFV break (Figure 5(a)). Figure 5(b) shows excellent agreement between the two methods as indicated by the relatively low mean difference and the relatively narrow range of 95 percent LOA. The clinical utility of the COVS to assess NFV can be considered moderate based on its sensitivity (83%) but with excellent specificity (100%) (Table 3). However, note that the expected normative value range for both methods is slightly different as shown in Table 2 (conventional: ≥17 PD; COVS: ≥10 PD). If the expected Morgan normative value range (i.e., ≥17 PD) is used instead to make a pass or fail determination, the sensitivity of the COVS increases to 100 percent at the cost of a lower specificity (88%) and slightly higher false positive rate (12%). Based on the strong Pearson correlation between measurements, these results suggest that adopting the conventional criterion (Morgan) for the COVS NFV measurement to make a pass or fail determination will increase its sensitivity while maintaining the good agreement between methods, as shown by the correlation and Bland-Altman analyses.


Figure 5. Computerized oculomotor vision screening (COVS) validity for measuring negative fusional vergence (NFV). (a) NFV measures of mild traumatic brain injury (mTBI) (􀀀) and non-mTBI (×) subjects are presented on scatter plot showing regression line for both groups combined (fine solid line), mTBI group (fine dashed lines), and non-mTBI group (fine dotted line). (b) Bland-Altman plot shows horizontal lines at mean difference (bold dotted line); upper and lower 95 percent limits of agreement (LOAs) (bold dashed lines); and regression line for both groups combined (fine solid line), mTBI group (fine dashed lines), and non-mTBI group (fine dotted line). Area right of vertical dashed line indicates expected normative value range. PD = prism diopter

Figure 5.

Computerized oculomotor vision screening (COVS) validity for measuring negative fusional vergence (NFV). (a) NFV measures of mild traumatic brain injury (mTBI) (􀀀) and non-mTBI (×) subjects are presented on scatter plot showing regression line for both groups combined (fine solid line), mTBI group (fine dashed lines), and non-mTBI group (fine dotted line). (b) Bland-Altman plot shows horizontal lines at mean difference (bold dotted line); upper and lower 95 percent limits of agreement (LOAs) (bold dashed lines); and regression line for both groups combined (fine solid line), mTBI group (fine dashed lines), and non-mTBI group (fine dotted line). Area right of vertical dashed line indicates expected normative value range. PD = prism diopter.

Click Image to Enlarge. View as PowerPoint Slide

Figure 6(a) shows very a strong correlation (r = 0.916) between the NFV recovery results obtained with COVS and conventional methods. There was also good agreement between the two methods according to the Bland-Altman analysis (Figure 6(b)). When the groups were analyzed individually, the results from the mTBI group showed slightly less variability than those from the non-mTBI group. Nonetheless, the results showed good agreement between the methods with only a slight tendency for the COVS to overestimate the measurements as the magnitude of the measurements increases. However, the COVS demonstrated a sensitivity and specificity of 100 and 97 percent, respectively (Table 3).


Figure 6. Computerized oculomotor vision screening (COVS) validity for measuring negative fusional vergence (NFV) recovery. (a) NFV recovery measures of mild traumatic brain injury (mTBI) (􀀀) and non-mTBI (×) subjects are presented on scatter plot showing regression line for both groups combined (fine solid line), mTBI group (fine dashed lines), and non-mTBI group (fine dotted line). (b) Bland-Altman plot shows horizontal lines at mean difference (bold dotted line); upper and lower 95 percent limits of agreement (LOAs) (bold dashed lines); and regression line for both groups combined (fine solid line), mTBI group (fine dashed lines), and non-mTBI group (fine dotted line). Area right of vertical dashed line indicates expected normative value range. PD = prism diopter.

Figure 6.

Computerized oculomotor vision screening (COVS) validity for measuring negative fusional vergence (NFV) recovery. (a) NFV recovery measures of mild traumatic brain injury (mTBI) (􀀀) and non-mTBI (×) subjects are presented on scatter plot showing regression line for both groups combined (fine solid line), mTBI group (fine dashed lines), and non-mTBI group (fine dotted line). (b) Bland-Altman plot shows horizontal lines at mean difference (bold dotted line); upper and lower 95 percent limits of agreement (LOAs) (bold dashed lines); and regression line for both groups combined (fine solid line), mTBI group (fine dashed lines), and non-mTBI group (fine dotted line). Area right of vertical dashed line indicates expected normative value range. PD = prism diopter.

Click Image to Enlarge. View as PowerPoint Slide

Figure 7(a) demonstrates a very strong correlation between methods (r = 0.975) for the PFV break results. Despite the nearly perfect correlation between the methods, the Bland-Altman plot shows only a moderate agreement between the two methods as indicated by the relative large LOA range (−17.01 to 4.94), with mTBI group results showing slightly less variability than the non-mTBI group. There was a tendency for the COVS to overestimate the measurements as the magnitude increased. Nevertheless, the clinical utility of the COVS to assess PFV is supported by its high sensitivity (100%) and specificity (97%), as summarized in Table 3. Here the sensitivity refers to the ability of the method to ­identify deficient positive fusional amplitude. Therefore, the excellent sensitivity shown here, despite the moderate agreement between the methods shown by the Bland-­Altman analysis, can be attributed to the fact that the COVS appears to overestimate those measurements that are already within the expected normative range (≥20 PD), maintaining its sensitivity to measure lower values outside the normative range.


Figure 7. Computerized oculomotor vision screening (COVS) validity for measuring positive fusional vergence (PFV). (a) PFV measures of mild traumatic brain injury (mTBI) (􀀀) and non-mTBI (×) subjects are presented on scatter plot showing regression line for both groups combined (fine solid line), mTBI group (fine dashed lines), and non-mTBI group (fine dotted line). (b) Bland-Altman plot shows horizontal lines at mean difference (bold dotted line); upper and lower 95 percent limits of agreement (LOAs) (bold dashed lines); and regression line for both groups combined (fine solid line), mTBI group (fine dashed lines), and non-mTBI group (fine dotted line). Area right of vertical dashed line indicates expected normative value range. PD = prism diopter.

Figure 7.

Computerized oculomotor vision screening (COVS) validity for measuring positive fusional vergence (PFV). (a) PFV measures of mild traumatic brain injury (mTBI) (􀀀) and non-mTBI (×) subjects are presented on scatter plot showing regression line for both groups combined (fine solid line), mTBI group (fine dashed lines), and non-mTBI group (fine dotted line). (b) Bland-Altman plot shows horizontal lines at mean difference (bold dotted line); upper and lower 95 percent limits of agreement (LOAs) (bold dashed lines); and regression line for both groups combined (fine solid line), mTBI group (fine dashed lines), and non-mTBI group (fine dotted line). Area right of vertical dashed line indicates expected normative value range. PD = prism diopter.

Click Image to Enlarge. View as PowerPoint Slide

Figure 8(a) demonstrates very strong correlations (r = 0.955) between the PFV recovery results obtained with both methods. However, despite the nearly perfect correlation between the methods, the Bland-Altman plot (Figure 8(b)) shows only a moderate agreement between the two methods. The mTBI group showed slightly less variability than the non-mTBI group, with a tendency for the COVS to overestimate measurements as the magnitude increased. Of all the oculomotor functions evaluated, this was the function with the largest variability on the results. Nevertheless, the clinical utility of the COVS to assess PFV recovery is supported by its high sensitivity (100%) and specificity (94%), summarized in Table 3, which indicate the ability of the COVS to identify defective PFV recovery. The excellent sensitivity shown here, despite the moderate agreement between the methods shown by the Bland-Altman method, can be attributed to the fact that the COVS appears to overestimate those measurements above 30 PD that are already within the expected COVS normative range (≥16 PD) while maintaining its sensitivity to measure lower values that are considered to be outside the normative range.


Figure 8. Computerized oculomotor vision screening (COVS) validity for measuring positive fusional vergence (PFV) recovery. (a) PFV recovery measures of mild traumatic brain injury (mTBI) (􀀀 ) and non-mTBI (×) subjects are presented on scatter plot showing regression line for both groups combined (fine solid line), mTBI group (fine dashed lines), and non-mTBI group (fine dotted line). (b) Bland-Altman plot shows horizontal lines at mean difference (bold dotted line); upper and lower 95 percent limits of agreement (LOAs) (bold dashed lines); and regression line for both groups combined (fine solid line), mTBI group (fine dashed lines), and non-mTBI group (fine dotted line). Area right of vertical dashed line indicates expected normative value range. PD = prism diopter.

Figure 8.

Computerized oculomotor vision screening (COVS) validity for measuring positive fusional vergence (PFV) recovery. (a) PFV recovery measures of mild traumatic brain injury (mTBI) (􀀀 ) and non-mTBI (×) subjects are presented on scatter plot showing regression line for both groups combined (fine solid line), mTBI group (fine dashed lines), and non-mTBI group (fine dotted line). (b) Bland-Altman plot shows horizontal lines at mean difference (bold dotted line); upper and lower 95 percent limits of agreement (LOAs) (bold dashed lines); and regression line for both groups combined (fine solid line), mTBI group (fine dashed lines), and non-mTBI group (fine dotted line). Area right of vertical dashed line indicates expected normative value range. PD = prism diopter.

Click Image to Enlarge. View as PowerPoint Slide

Figure 9(a) shows a moderately strong correlation (r = 0.769) between the monocular accommodative facility results obtained with COVS and conventional testing. Figure 9(b) shows that despite the strong correlation between the methods, there was only a moderate agreement between the two methods as indicated by the relatively large LOA range (−12.75 to 2.67). The mTBI group showed slightly less variability than the non-mTBI group with a tendency for the COVS to overestimate measurements as the magnitude increased. Despite the variability shown here, the clinical utility of COVS was demonstrated by its relatively high sensitivity (90%) and specificity (95%) when assessing monocular accommodative facility (Table 3). Its sensitivity is preserved despite the moderate agreement between methods shown by the Bland-Altman analysis because the overestimation occurs primarily for values already within the expected normative range (≥12 cpm).


Figure 9. Computerized oculomotor vision screening (COVS) validity for measuring monocular accommodative facility. (a) Monocular accommodative facility measures of mild traumatic brain injury (mTBI) (􀀀) and non-mTBI (×) subjects are presented on scatter plot showing regression line for both groups combined (fine solid line), mTBI group (fine dashed lines), and non-mTBI group (fine dotted line). (b) Bland-Altman plot shows horizontal lines at mean difference (bold dotted line); upper and lower 95 peccent limits of agreement (LOAs) (bold dashed lines); and regression line for both groups combined (fine solid line), mTBI group (fine dashed lines), and non-mTBI group (fine dotted line). Area right of vertical dashed line indicates expected normative value range. PD = prism diopter.

Figure 9.

Computerized oculomotor vision screening (COVS) validity for measuring monocular accommodative facility. (a) Monocular accommodative facility measures of mild traumatic brain injury (mTBI) (􀀀) and non-mTBI (×) subjects are presented on scatter plot showing regression line for both groups combined (fine solid line), mTBI group (fine dashed lines), and non-mTBI group (fine dotted line). (b) Bland-Altman plot shows horizontal lines at mean difference (bold dotted line); upper and lower 95 peccent limits of agreement (LOAs) (bold dashed lines); and regression line for both groups combined (fine solid line), mTBI group (fine dashed lines), and non-mTBI group (fine dotted line). Area right of vertical dashed line indicates expected normative value range. PD = prism diopter.

Click Image to Enlarge. View as PowerPoint Slide

The kappa coefficient showed excellent agreement between methods for assessing oculomotor functions with outcome measurements with categorical data (pass or fail) based on large coefficient value for Worth 4 Dot (combined: κ = 1.0; mTBI: κ = 1.0; non-mTBI: κ = 1.0), fixation (combined: κ = 1.0; mTBI: κ = 1.0; non-mTBI: κ = 1.0), and pursuit (combined: κ = 0.875; mTBI: κ = 1.0; non-mTBI: κ = 0.823) and saccadic (combined: κ = 0.828; mTBI: κ = 0.886; non-mTBI: κ = 0.807) eye movements. The sensitivity and specificity were also excellent for the Worth 4 Dot (100% and 100%, respectively), fixation (100% and 100%, respectively), and pursuit (100% and 97%, respectively) and saccadic eye movements (100% and 97%, respectively) (Table 3).

For oculomotor function measurements that produced continuous data, the COR and LOA method of repeated measures assessed the repeatability of the data over sessions 1, 2, and 3 (Table 4 and Figure 10). The CORs were very similar for all session combinations for all the functions. In order to quantitatively evaluate the extent of this similarity, a within-subjects repeated measures MANOVA was performed to compare the data over the three sessions for the seven continuous dependent measures in Table 4. The last two columns of Table 4 contain the p-values (for Pillai trace) and the η2, an estimate of the effect size that could be attributed to the comparison being made. Thus, for example, even though the mean differences among each of the three measurements of lateral phoria were not statistically significant, the differences among the three lateral phoria means account for about 5.7 percent of the total variance in this data set (Figure 10(a)). Accommodative facility is the only measure for which the MANOVA was statistically significant (Pillai trace = 272; F(2,38) = 7.112, p < 0.002), such that about 27.2 percent of the variance can be accounted for by the fact that the first readings were statistically slower than the second, but that the third reading was not different from either the first or the second, as shown in Figure 10(g).


Table 4. 
Oculomotor
Function
Difference
(M ± SD)
95%
LOA
Difference
(M ± SD)
95%
LOA
Difference
(M ± SD)
95%
LOA
p-Value
η-Value
1.47 (1.80)
1.95 (2.11)
0.09 ± 0.62*
1.30 (1.13)
0.01 ± 0.37
0.74 (0.71)
0.72 (0.74)
0.50 (0.56)
5.04 (5.04)
0.50 ± 2.18
4.78 (3.78)
0.50 ± 2.82
6.03 (5.03)
9.09 (9.29)
0.20 ± 3.60
7.26 (6.86)
0.30 ± 4.27
8.68 (8.07)
13.28 (14.63)
12.10 (12.50)
0.48 ± 6.73
13.66 (12.71)
22.18 (20.59)
0.65 ± 10.84
21.89 (20.59)
1.80 ± 5.16
11.92 (8.32)
Monocular Accommodative Facility (cpm)†
7.38 (3.97)
1.11 ± 3.49
7.95 (5.74)
4.37 (5.56)
*Negative (–) denotes exophoria.
Negative (–) denotes hyperphoria.

Figure 10. Computerized oculomotor vision screening measurements for three individual testing sessions: 1 (), 2 (), and 3 (). Average measurement with ±95 percent confident interval error bars are shown for (a) lateral phoria, (b) vertical phoria, (c) negative fusional vergence (NFV) break, (d) NFV recovery, (e) positive fusional vergence (PFV) break, (f) PFV recovery, (g) monocular accommodative (acc) facility, (h) pursuit correct responses (resp), (i) pursuit mean resp time, (j) saccadic correct resp, (k) saccadic mean resp time, (l) reading speed, and (m) reading comprehension. PD = prism diopter.

Figure 10.

Computerized oculomotor vision screening measurements for three individual testing sessions: 1 (●), 2 (█), and 3 (▲). Average measurement with ±95 percent confident interval error bars are shown for (a) lateral phoria, (b) vertical phoria, (c) negative fusional vergence (NFV) break, (d) NFV recovery, (e) positive fusional vergence (PFV) break, (f) PFV recovery, (g) monocular accommodative (acc) facility, (h) pursuit correct responses (resp), (i) pursuit mean resp time, (j) saccadic correct resp, (k) saccadic mean resp time, (l) reading speed, and (m) reading comprehension. PD = prism diopter.

Click Image to Enlarge. View as PowerPoint Slide

The repeatability of the COVS for oculomotor functions producing categorical data was determined by its kappa coefficients. The kappa coefficients show excellent agreement between the three readings obtained by the COVS for the Worth 4 Dot (κ = 0.791), fixation (κ = 0.848), and pursuit (κ = 0.815) and saccadic (κ = 0.880) eye movements. Furthermore, there was excellent repeatability of the individual components (i.e., percent of correct responses and average response time) used by the COVS to make a pass or fail determination for the pursuit and saccadic eye movement (Figure 10(h)–(k)).

Figure 11 shows representative diagrams for one subject with normal (Figure 11(a)) and one with defective (Figure 11(b)) motor fields. A motor field diagram is generated based on the phorias measured at the nine positions of gaze, therefore providing a graphical representation of a suspected paretic muscle. We found that 12 (11 [55%] mTBI and 1 [5%] non-mTBI) of the 40 subjects (30%) had motor field diagrams that were classified as “defective” when compared with the example paretic muscle patterns displayed on the test result screen. The resulting motor field diagram for each of the subjects was highly repeatable in all three test-retest sessions. More importantly, these defective motor field diagrams were present in subjects with marked oculomotor deficit, ­particularly those with large vertical and lateral phorias in primary gaze.


Figure 11. Representative motor field diagrams are shown for (a) one study subject with normal motor fields and (b) another study subject with abnormal motor field with suspected right superior oblique paresis obtained during three computerized oculomotor vision screening sessions.

Figure 11.

Representative motor field diagrams are shown for (a) one study subject with normal motor fields and (b) another study subject with abnormal motor field with suspected right superior oblique paresis obtained during three computerized oculomotor vision screening sessions.

Click Image to Enlarge. View as PowerPoint Slide

Similarly, of the 40 subjects, the 15 (38%) who failed the reading assessment test also had at least two oculomotor function deficits as measured by the COVS or ­conventional evaluation. The repeatability of the reading assessment based on a pass or fail score (categorical data) was determined by the kappa coefficient. The kappa coefficient shows excellent agreement between all three pass or fail determinations obtained by the COVS for the reading assessment test (κ = 0.826). Furthermore, there was excellent repeatability of the individual assessment components (i.e., reading speed and reading comprehension) used by the COVS to make a pass or fail determination for the reading assessment test (Figure 10(l)–(m)). Since the short story passages are different for each test-retest session, there was no practice effect for reading comprehension or speed. On the contrary, there was a small (not statistically significant) decline in reading speed as the number of repeats progressed, possibly caused by fatigue.

The mean total score for the CISS was significantly different between the two groups (mTBI: 20.45 ± 14.88; non-mTBI: 12.15 ± 7.66; p = 0.03). A correlation of the subjects’ reading assessment pass or fail score determined by the COVS with the number of subjects failing the CISS positively identified 13 (86.67%, all mTBI) of the 15 subjects who failed the reading assessment test.

Although the effectiveness of computer-based systems has been previously described as a tool to remediate oculomotor deficits in the form of orthoptic therapy, we know of no validation studies assessing the effectiveness of computerized testing for screening oculomotor functions [31,34–35]. This study demonstrated that non-­eye-care professionals may be able to use the COVS as a tool to efficiently screen version, vergence, and accommodative functions in a military population with or without mTBI. To evaluate the COVS, the present study used 40 Active Duty military subjects, half with a history of mTBI and half without. A comparison of the visual functions measured in these two groups is clearly beyond the scope of the present article, particularly since the number of subjects in each group is so small. However, among all of the oculomotor functions evaluated with both conventional testing and COVS, we found that near lateral phoria, near vertical phoria, near PFV break and recovery, and pursuit and saccadic eye movements were significantly more defective in the mTBI group than the non-mTBI group.

Overall, the COVS showed excellent accuracy and repeatability for assessing near lateral and vertical phorias, flat fusion (by Worth 4 Dot), fixation, and pursuit and saccadic eye movements. Despite the strong Pearson correlation, the Bland-Altman analysis identified minor to moderate discrepancies for both PFV and NFV and their associated recovery, as well as with the monocular accommodative facility measurements. For example, the mean maximum fusional amplitudes and refusion values during divergence and convergence testing measured with the COVS were consistently higher than those measured with phorometry. However, this was an expected finding because the COVS’s RDS stimulus size was larger than that used during the conventional von Graefe phorometry technique. Previous studies have shown that fusional amplitude measurements increase as the stimulus size increases [47–48]. Furthermore, Feldman et al. concluded that vergence fusional amplitude values depend on the type of stimuli. Stereo acuity (i.e., RDS) stimuli produce higher measurements than flat fusion stimuli (i.e., phorometry target) [47]. Feldman et al. found that the largest differences in measurement overestimation were present with PFV measurements. Similarly, the present study found that PFV measurements provide the largest magnitude of overestimation. However, it appears that such overestimation on PFV values has been partially accounted for by the COVS software since its expected normative values (break: ≥20 PD; recovery: ≥16 PD) are high relative to the expected Morgan normative values (break: ≥15 PD; recovery: ≥4 PD). Consequently, even though there was only moderate agreement between methods, the COVS has nearly perfect sensitivity and specificity as a tool to assess positive fusional amplitudes and recoveries due to the adjusted COVS normal range criterion.

Feldman et al. also showed that RDS stimuli produced only a slight overestimation (approx 2–3 PD) of the maximum divergence amplitude and divergence recovery, which is also in agreement with the NFV break and recovery results found here [47]. However, the expected normative values for NFV measurements for the COVS (break: ≥10 PD; recovery: ≥6 PD) are lower than the Morgan normative values (break: ≥17 PD; recovery: ≥8 PD). If the Morgan expected values are used with the COVS to make a pass or fail determination, its sensitivity improves from 83 to 100 percent; however, its specificity decreases from 100 to 88 percent with a slightly higher false positive rate (12%). Despite the great level of repeatability of NFV measurements, it appears that the COVS measurements are overestimated, particularly over what is considered a normal fusional range. Since we are concerned with the identification of individuals with poor fusional vergence (i.e., low values), it would be acceptable to adopt the Morgan normative criterion for COVS to increase the test sensitivity even with its slightly decreased specificity. Further studies incorporating a larger sample size are needed to determine the acceptable cut-off value that will result in the highest sensitivity and specificity.

Similarly, despite the great level of repeatability of accommodative facility, it appears that the COVS overestimated accommodative facility values that are beyond what is considered a normal range (≥12 cpm). It is possible that this overestimation, particularly on those with normal function, was induced by the different magnitude of flipper lens power used for the COVS (+1.50/–1.50 D, as per user manual instructions) and the conventional test (+2.00/–2.00 D). Further studies are needed to determine whether the use of +2.00/–2.00 D flippers with the COVS provides results that are more in line with conventional assessment, decreasing the overestimation on subjects with normal accommodative function without negatively affecting its sensitivity. Since we are concerned with the identification of individuals with poor accommodative amplitude (<12 cpm), overestimation at large values will not substantially affect the sensitivity (i.e., ability to detect subjects with low cycles per minute). Moreover, since the repeatability of the measurement is high, adjustments to the software possibly compensate for this overestimation to meet accuracy tolerances without altering repeatability.

The COVS also incorporated the symptoms CISS, the motor field assessment, and a reading fluency test, which provided a functional assessment of the subjects’ near oculomotor status. These tests (i.e., motor field and reading assessment) also show good repeatability and sensitivity to identify individuals with oculomotor dysfunctions and should be performed during routine as well as predeployment eye examinations to facilitate the diagnosis of oculomotor deficits resulting from deployment.

The findings of this study have important implications regarding the clinical utility of COVS as a screening tool since it has the promise of enabling non-eye-care providers to quickly screen (~10 min) warfighters. The COVS and the methodology used in this study supported the identification of oculomotor deficits commonly present in warfighters with mTBI. The use of computer programs to automate the oculomotor function evaluation in military and Department of Veterans Affairs treatment facilities has the potential to improve the identification of individuals with oculomotor deficits while decreasing the variability of results, decreasing testing time, and allowing direct comparison of pre- and postdeployment results. In addition, an automated system overcomes most of the inherent technical and examiner-induced shortcomings of the conventional oculomotor examination. For example, the COVS may reduce the variability of results among examiners as well as intratesting variability observed with ­conventional oculomotor examinations [32,42,49]. Interexaminer and intratesting variability can be induced by a considerable number of factors, such as variation on the size, brightness, and complexity of the target; speed of target presentation; type of stimulus (i.e., flat fusion vs stereoscopic); methods of presentation (i.e., Risley prism, anaglyph, or vectograph) or of changing the stimulus parameters; and variations in lighting, as well as the detail of instructions and the level of motivational support provided to the patient during the evaluation [30,42,50]. In addition, our finding supports the usefulness of the CISS and the reading assessment as adjuvant tools to identify subjects with near vision oculomotor dysfunctions.

This study is the first step toward a standardized oculomotor screening methodology that can be used to assess and monitor oculomotor deficits associated with mTBI in warfighters. The study suggests that the COVS may be administered by paraoptometric personnel or non-eye-care providers to (1) rapidly and accurately evaluate a significant number of oculomotor functions, (2) provide a functional assessment test, (3) survey the subjective near vision symptoms, and (4) expedite the referral for a comprehensive TBI functional examination by an eye-care provider.

By comparison with a conventional examination of oculomotor functions, this study showed that COVS has promise as an effective screening tool to assess near oculomotor functions in a military population with or without mTBI. Its effectiveness is based on its high validity and repeatability for most of the evaluated functions. In addition, its high sensitivity and specificity, coupled with the relatively low false positive and negative rates, make it an effective tool that can be administered by non-eye-care personnel to screen for near oculomotor dysfunctions as well as associated symptoms and reading problems. The automation of the oculomotor function screening allows for the standardization of the oculomotor examination by presenting the same stimuli under similar conditions (e.g., type of stimuli, speed, distance, and illumination levels). Such standardization has the potential to improve interexaminer and interlocation reliability, thereby permitting the provider to compare pre- and postdeployment oculomotor functions or to monitor recovery of oculomotor deficits over time for those individuals with mTBI. Although most of the oculomotor dysfunctions associated with mTBI affect near oculomotor functions, a complete oculomotor assessment is necessary to make an accurate diagnosis and to prescribe an effective rehabilitative program. In practice, patients who fail the COVS or have associated visual complaints should be referred to the eye-care provider for more thorough near and distance oculomotor evaluation in order to prescribe an appropriate customized neurovisual rehabilitation ­regimen.

Study concept and design: J. E. Capó-Aponte, A. K. Tarbett, T. G. Urosevich, N. K. Sanghera, M. E. Kalich.
Acquisition of data: J. E. Capó-Aponte, T. G. Urosevich.
Analysis and interpretation of data: J. E. Capó-Aponte, T. G. Urosevich, L. A. Temme.
Drafting of manuscript: J. E. Capó-Aponte, T. G. Urosevich, L. A. Temme.
Critical revision of the manuscript for intellectual content: A. K. Tarbett, T. G. Urosevich, L. A. Temme, N. K. Sanghera, M. E. Kalich.
Statistical analysis: J. E. Capó-Aponte, L. A. Temme.
Obtained funding: M. E. Kalich.
Administrative, technical, or material support: A. K. Tarbett, T. G. Urosevich, L. A. Temme, N. K. Sanghera, M. E. Kalich.
Study supervision: J. E. Capó-Aponte, A. K. Tarbett.
Financial Disclosures: The authors have declared that no competing interests exist.
Funding/Support: This material is based on work supported with resources and funding from the USAARL.
Institutional Review: The study protocol was approved by the institutional review boards of the USAARL and the WRAMC Department of Clinical Investigation. Informed written consent was freely obtained from all participants, who received no monetary compensation.
Participant Follow-Up: The authors do not plan to notify study participants of the publication of this article.
Disclaimer: The views, opinions, and/or findings contained in this report are those of the authors and should not be construed as an official Department of the Army position, policy, or decision unless so designated by other official documentation. Citation of trade names in this report does not constitute an official Department of the Army endorsement or approval of the use of such commercial items. None of the authors has financial or other interest in the BVA software or Home Therapy Systems Inc.
1.
Cockerham GC, Goodrich GL, Weichel ED, Orcutt JC, Rizzo JF, Bower KS, Schuchard RA. Eye and visual function in traumatic brain injury. J Rehabil Res Dev. 2009; 46(6):811–18. [PMID:20104404]
http://dx.doi.org/10.1682/JRRD.2008.08.0109
2.
Dougherty AL, MacGregor AJ, Han PP, Heltemes KJ, Galarneau MR. Visual dysfunction following blast-related traumatic brain injury from the battlefield. Brain Inj. 2011; 25(1):8–13. [PMID:21117919]
http://dx.doi.org/10.3109/02699052.2010.536195
3.
Goodrich GL, Kirby J, Cockerham G, Ingalla SP, Lew HL. Visual function in patients of a polytrauma rehabilitation center: A descriptive study. J Rehabil Res Dev. 2007;44(7): 929–36. [PMID:18075950]
http://dx.doi.org/10.1682/JRRD.2007.01.0003
4.
Lew HL, Garvert DW, Pogoda TK, Hsu PT, Devine JM, White DK, Myers PJ, Goodrich GL. Auditory and visual impairments in patients with blast-related traumatic brain injury: Effect of dual sensory impairment on Functional Independence Measure. J Rehabil Res Dev. 2009;46(6): 819–26. [PMID:20104405]
http://dx.doi.org/10.1682/JRRD.2008.09.0129
5.
Lew HL, Weihing J, Myers PJ, Pogoda TK, Goodrich GL. Dual sensory impairment (DSI) in traumatic brain injury (TBI)—An emerging interdisciplinary challenge. NeuroRehabilitation. 2010;26(3):213–22. [PMID:20448311]
7.
Stelmack JA, Frith T, Van Koevering D, Rinne S, Stelmack TR. Visual function in patients followed at a Veterans Affairs polytrauma network site: an electronic medical record review. Optometry. 2009;80(8):419–24.
[PMID:19635432]
http://dx.doi.org/10.1016/j.optm.2009.02.011
8.
Brahm KD, Wilgenburg HM, Kirby J, Ingalla S, Chang CY, Goodrich GL. Visual impairment and dysfunction in combat-injured servicemembers with traumatic brain injury. Optom Vis Sci. 2009;86(7):817–25. [PMID:19521270]
http://dx.doi.org/10.1097/OPX.0b013e3181adff2d
9.
Lew HL, Poole JH, Vanderploeg RD, Goodrich GL, Dekelboum S, Guillory SB, Sigford B, Cifu DX. Program development and defining characteristics of returning military in a VA Polytrauma Network Site. J Rehabil Res Dev. 2007; 44(7):1027–34. [PMID:18075959]
http://dx.doi.org/10.1682/JRRD.2007.05.0073
10.
Ciuffreda KJ, Kapoor N, Rutner D, Suchoff IB, Han ME, Craig S. Occurrence of oculomotor dysfunctions in acquired brain injury: a retrospective analysis. Optometry. 2007;78(4):155–61. [PMID:17400136]
http://dx.doi.org/10.1016/j.optm.2006.11.011
11.
Ciuffreda KJ, Rutner D, Kapoor N, Suchoff IB, Craig S, Han ME. Vision therapy for oculomotor dysfunctions in acquired brain injury: a retrospective analysis. Optometry. 2008;79(1):18–22. [PMID:18156092]
http://dx.doi.org/10.1016/j.optm.2007.10.004
12.
McAllister TW. Mild brain injury and the post-concussive syndrome. In: Silver JM, McAllister TW, Yudofsky SC, editors. Textbook of traumatic brain injury. Washington (DC): American Psychiatric Publishing; 2005. p. 279–308.
13.
Katz DI, Alexander MP. Traumatic brain injury. Predicting course of recovery and outcome for patients admitted to rehabilitation. Arch Neurol. 1994;51(7):661–70.
[PMID:8018038]
http://dx.doi.org/10.1001/archneur.1994.00540190041013
14.
Novack TA, Bush BA, Meythaler JM, Canupp K. Outcome after traumatic brain injury: pathway analysis of contributions from premorbid, injury severity, and recovery variables. Arch Phys Med Rehabil. 2001;82(3):300–305.
[PMID:11245749]
http://dx.doi.org/10.1053/apmr.2001.18222
15.
Povlishock JT, Katz DI. Update of neuropathology and neurological recovery after traumatic brain injury. J Head Trauma Rehabil. 2005;20(1):76–94. [PMID:15668572]
http://dx.doi.org/10.1097/00001199-200501000-00008
16.
Khan F, Baguley IJ, Cameron ID. 4: Rehabilitation after traumatic brain injury. Med J Aust. 2003;178(6):290–95.
[PMID:12633489]
17.
Vandiver VL, Johnson J, Christofero-Snider C. Supporting employment for adults with acquired brain injury: a conceptual model. J Head Trauma Rehabil. 2003;18(5):457–63.
[PMID:12973275]
http://dx.doi.org/10.1097/00001199-200309000-00007
18.
Zasler ND, Devany CD. Traumatic brain injury. In: Belandres PV, Dillingham TR, editors. Rehabilitation of the injured combatant. Washington (DC): Borden Institute; 2000. p. 207–77.
20.
Ciuffreda KJ, Han Y, Kapoor N, Ficarra AP. Oculomotor rehabilitation for reading in acquired brain injury. NeuroRehabilitation. 2006;21(1):9–21. [PMID:16720933]
21.
Ciuffreda KJ, Kapoor N. Oculomotor dysfunctions, their remediation, and reading-related problems in mild traumatic brain injury. J Behav Optom. 2007;18(3):72–77.
22.
23.
Cooper J, Feldman J, Selenow A, Fair R, Buccerio F, MacDonald D, Levy M. Reduction of asthenopia after accommodative facility training. Am J Optom Physiol Opt. 1987; 64(6):430–36. [PMID:3631199]
http://dx.doi.org/10.1097/00006324-198706000-00008
24.
Cooper J, Selenow A, Ciuffreda KJ, Feldman J, Faverty J, Hokoda SC, Silver J. Reduction of asthenopia in patients with convergence insufficiency after fusional vergence training. Am J Optom Physiol Opt. 1983;60(12):982–89.
[PMID:6660282]
http://dx.doi.org/10.1097/00006324-198312000-00007
26.
Slobounov S, Slobounov E, Sebastianelli W, Cao C, Newell K. Differential rate of recovery in athletes after first and second concussion episodes. Neurosurgery. 2007;61(2): 338–44, discussion 344. [PMID:17762746]
http://dx.doi.org/10.1227/01.NEU.0000280001.03578.FF
27.
Scheiman M, Cooper J, Mitchell GL, Land PD, Cotter S, Borsting E, London R, Rouse M. A survey of treatment modalities for convergence insufficiency. Optom Vis Sci. 2002;79(3):151–57. [PMID:11913841]
http://dx.doi.org/10.1097/00006324-200203000-00009
28.
Scheiman M, Mitchell GL, Cotter S, Cooper J, Kulp M, Rouse M, Borsting E, London R, Wensveen J; Convergence Insufficiency Treatment Trial Study Group. A randomized clinical trial of treatments for convergence insufficiency in children. Arch Ophthalmol. 2005;123(1): 14–24. [PMID:15642806]
http://dx.doi.org/10.1001/archopht.123.1.14
29.
Suchoff IB, Gianutsos R. Rehabilitative optometric interventions for the adult with acquired brain injury. In: Grabois M, editor. Physical medicine and rehabilitation: The complete approach. Malden (MA): Blackwell Science; 2000.
30.
Cooper J. Review of computerized orthoptics with specific regard to convergence insufficiency. Am J Optom Physiol Opt. 1988;65(6):455–63. [PMID:3046364]
http://dx.doi.org/10.1097/00006324-198806000-00005
31.
Cooper J. Diagnosis and treatment of accommodative and vergence anomalies using computerized vision therapy. Pract Optom. 1998;(9):6–10.
32.
Cooper J, Feldman J. Random-dot-stereogram performance by strabismic, amblyopic, and ocular-pathology patients in an operant-discrimination task. Am J Optom Physiol Opt. 1978;55(9):599–609. [PMID:742651]
http://dx.doi.org/10.1097/00006324-197809000-00001
33.
Cooper J, Feldman J. Operant conditioning of fusional convergence ranges using random dot stereograms. Am J Optom Physiol Opt. 1980;57(4):205–13. [PMID:7386581]
http://dx.doi.org/10.1097/00006324-198004000-00002
34.
Cooper J, Feldman J. Reduction of symptoms in binocular anomalies using computerized home therapy-HTS. Optometry. 2009;80(9):481–86. [PMID:19716076]
http://dx.doi.org/10.1016/j.optm.2009.05.007
35.
Serna A, Rogers DL, McGregor ML, Golden RP, Bremer DL, Rogers GL. Treatment of symptomatic convergence insufficiency with a home-based computer orthoptic exercise program. J AAPOS. 2011;15(2):140–43.
[PMID:21458340]
http://dx.doi.org/10.1016/j.jaapos.2010.11.023
36.
Scheiman M, Mitchell GL, Cotter S, Kulp MT, Cooper J, Rouse M, Borsting E, London R, Wensveen J. A randomized clinical trial of vision therapy/orthoptics versus pencil pushups for the treatment of convergence insufficiency in young adults. Optom Vis Sci. 2005;82(7):583–95.
[PMID:16044063]
http://dx.doi.org/10.1097/01.opx.0000171331.36871.2f
37.
Carlson NB, Kurtz D. Clinical procedures for ocular examination. 3rd ed. New York (NY): McGraw-Hill, Medical Publishing Division; 2004.
38.
Goss DA. Fixation disparity. In: Eskridge JB, Amos JF, Barlett JD, editors. Clinical procedures in optometry. Philadelphia (PA): Lippincott; 1991. p. 716–26.
39.
Maples WC, Ficklin TW. Interrater and test-retest reliability of pursuits and saccades. J Am Optom Assoc. 1988; 59(7):549–52. [PMID:3403903]
40.
Ciuffreda KJ, Han Y, Kapoor N, Suchoff IB. Oculomotor consequences of acquired brain injury. In: Suchoff IB, Kapoor N, Ciuffreda KJ, editors. Visual & vestibular consequences of acquired brain injury. Santa Ana (CA): Optometric Extension Program; 2001. p. 77–88.
41.
Rouse MW, Borsting EJ, Mitchell GL, Scheiman M, Cotter SA, Cooper J, Kulp MT, London R, Wensveen J; Convergence Insufficiency Treatment Trial Group. Validity and reliability of the revised convergence insufficiency symptom survey in adults. Ophthalmic Physiol Opt. 2004;24(5): 384–90. [PMID:15315652]
http://dx.doi.org/10.1111/j.1475-1313.2004.00202.x
42.
Feldman JM, Cooper J, Carniglia P, Schiff FM, Skeete JN. Comparison of fusional ranges measured by Risley prisms, vectograms, and computer orthopter. Optom Vis Sci. 1989; 66(6):375–82. [PMID:2771323]
http://dx.doi.org/10.1097/00006324-198906000-00007
44.
Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 1986;1(8476):307–10. [PMID:2868172]
http://dx.doi.org/10.1016/S0140-6736(86)90837-8
46.
Armitage P. Statistical methods in medical research. Oxford (UK): Blackwell Scientific; 1971.
47.
Feldman JM, Cooper J, Eichler R. The effect of stimulus parameters (size, complexity, depth and line thickness) on horizontal fusional amplitudes in normal humans. Binocul Vis Strabismus Q. 1993;8(1):23–30.
48.
Boman DK, Kertesz AE. Effect of stimulus parameters on fusional and stereoscopic performance. Am J Optom Physiol Opt. 1985;62(3):222–27. [PMID:3985116]
http://dx.doi.org/10.1097/00006324-198503000-00012
49.
Feldman J, Cooper J. Rapid assessment of stereopsis in pre-verbal children using operant techniques: a preliminary study. J Am Optom Assoc. 1980;51(8):767–71.
[PMID:7440878]
50.
Sheedy JE, Saladin JJ. Validity of diagnostic criteria and case analysis in binocular vision disorders. In: Schor CM, Ciuffreda KJ, editors. Vergence eye movements: Basic and clinical aspects. Boston (MA): Butterworth; 1983.
This article and any supplementary material should be cited as follows:
Capó-Aponte JE, Tarbett AK, Urosevich TG, Temme LA, Sanghera NK, Kalich ME. Effectiveness of computerized oculomotor vision screening in a military population: Pilot study. J Rehabil Res Dev. 2012;49(9):1377–98.
http://dx.doi.org/10.1682/JRRD.2011.07.0128
iThenticateCrossref

Go to TOP

Last Reviewed or Updated  Monday, January 7, 2013 10:24 AM

Valid HTML 4.01 Transitional