Vol. 37 No. 1, January/February 2000
Pages 53 - 64
A practical EMG-based human-computer interface for users with motor disabilities
Armando B. Barreto, PhD; Scott D. Scargle, MSEE; Malek Adjouadi, PhD
Digital Signal Processing Laboratory, Department of Electrical and Computer Engineering, Florida International University, Miami, FL., 33174; Center for Advanced Technology and Education, Department of Electrical and Computer Engineering, Florida International University, Miami, FL 33174
Abstract--In line with the mission of the Assistive Technology Act of 1998 (ATA), this study proposes an integrated assistive real-time system which "affirms that technology is a valuable tool that can be used to improve the lives of people with disabilities." An assistive technology device is defined by the ATA as "any item, piece of equipment, or product system, whether acquired commercially, modified, or customized, that is used to increase, maintain, or improve the functional capabilities of individuals with disabilities." The purpose of this study is to design and develop an alternate input device that can be used even by individuals with severe motor disabilities. This real-time system design utilizes electromyographic (EMG) biosignals from cranial muscles and electroencephalographic (EEG) biosignals from the cerebrum's occipital lobe, which are transformed into controls for two-dimensional (2-D) cursor movement, the left-click (Enter) command, and an ON/OFF switch for the cursor-control functions. This HCI system classifies biosignals into "mouse" functions by applying amplitude thresholds and performing power spectral density (PSD) estimations on discrete windows of data. Spectral power summations are aggregated over several frequency bands between 8 and 500 Hz and then compared to produce the correct classification. The result is an affordable DSP-based system that, when combined with an on-screen keyboard, enables the user to fully operate a computer without using any extremities.
Key words: Assistive Technology Act, assistive technology device, biofeedback, human-computer interface, spinal cord injury.
Address all correspondence and requests for reprints to: Malek Adjouadi, PhD, Acting Chair, Dept. of Electrical and Computer Engineering, and Director, Center for Advanced Technology and Education, Florida International University, Miami, FL 33174; email: email@example.com.
A significant number of individuals have severe motor disabilities, due to a variety of causes, such as Spinal Cord Injury (SCI), Amyotrophic Lateral Sclerosis (ALS), and so forth. For example, it has been estimated that up to 400,000 individuals may be living with the limitations imposed by SCI or Spinal Dysfunction (1). The quality of life of these individuals could be significantly improved by providing them with a practical, reliable means to use standard personal computers (PC)s. The improvement in their quality of life, along with those identified by the Assistive Technology Act of 1998 (2), and as presented in some case studies (3,4), can come about in at least two ways:
- Increased integration to society and productivity by communicating/working through the computer, using standard software
- Increased unassisted control over their environment, using dedicated software/output devices to turn appliances, alarms, etc. ON/OFF.
With today's Graphic User Interface (GUI)-based PC software, most of the human-to-computer interaction is based on selection operations. Even limited data entry can be achieved by selection through approaches such as "on-screen keyboards." Selecting requires two basic types of capabilities:
- Pointing: Positioning the cursor at the desired location of the screen, over the appropriate area or icon.
- Clicking: Executing the Mouse Down/Up function that is interpreted by the computer's operating system as an indication to complete the selection of the item associated with the icon at the location of the screen cursor.
A number of approaches have attempted to make these operations available to individuals with severe motor disabilities. Several of these devices target the motor skills that are still available to some users. The "Tonguepoint" system is based on an IBM Trackpoint IIITM pressure-sensitive isometric joystick fastened to a mouthpiece so that it can be operated by the user's tongue. The joystick provides cursor-control, while two switches (a bite switch and a manual switch located outside of the mouth) allow the user to consider left and right button selections (5).
Another alternative "mouse" based on movement is the HeadmouseTM (Origin Instruments, Grand Prairie, TX), a pointing device that transforms head movement into cursor movement on the screen. This is accomplished by a wireless optical sensor that tracks a small adhesive-backed target on the user's forehead or glasses. When combined with an on-screen keyboard, the left and right mouse button operations are triggered either by dwelling over a particular key for a set period of time or by using a remote adaptive switch that can be mechanically altered.
The approaches outlined above have potential disadvantages for some categories of users. For example, a user with cerebral palsy may not have the fine motor abilities in the tongue to operate the Tonguepoint device. Similarly, a user with spinal vertebrae fusion may not be able to turn his or her head, so the Headmouse would be of little benefit.
Other more complex approaches have attempted to provide computer interface functionality requiring even fewer abilities from the potential users. A prominent example is the eye-gaze tracking interface approach. This method does not presume any mechanical capability on the part of the user other than the control over his/her eye gaze. In the most common types of these systems, an infrared illuminator and video camera are used to obtain continuous images of the subject's eye. Application of digital image processing techniques allows the real-time isolation of two landmark reflections from the subject's eye: the reflection from its pupil and the smaller and brighter reflection from its cornea. Real-time determination of the centers of these reflections and their relative positions in the image captured by the camera is used to define the instantaneous orientation of the eye's line of gaze.
Knowledge of other geometric parameters involved in the system, such as the distance between the eye and the plane of the computer screen, allow the calculation of the "point-of-gaze" of the subject in the plane of the screen, at all times. Software in the computer assigns the screen coordinates of the point-of-gaze to the screen cursor, drawing it at that specified location. The clicking operation in these systems has been attempted by assigning a "dwell latency" and executing a click whenever the cursor remains within a so-called "dwell neighborhood" for at least that amount of time. This clicking procedure, however, may result in false clicks if a user is simply staring attentively at a small area of the screen, a dilemma referred to as the "Midas Touch" problem (6).
Given their complexity and computational requirements, eye-gaze-tracking systems are comparatively expensive and require great attention and effort to achieve proper cursor control (7). However, an effort is made in this technology to make such systems more portable (e.g., head-mounted version). Thus, while they provide the subject with the ability to quickly displace the cursor across the screen, execution of fine, small cursor movements is not easy in these systems, and the stability of the cursor in a single screen position is also limited. Furthermore, the determination of the point-of-gaze from the orientation of the line of gaze depends heavily on the geometric relationships between the subject and the computer, which are captured during a calibration stage in the operation of these systems.
If the user changes position with respect to the plane of the screen during the use of the device, the calibration is lost and cursor position errors ensue. Furthermore, if the subject moves enough to shift his/her eye out of the field of vision of the camera, the operation of the system is disrupted. Initial results of comparative studies that we have carried out clearly indicate that the eye-gaze approach requires more strenuous and stringent control abilities for finer cursor movements than the approach proposed in this study. At present, some eye-gaze systems do attempt to compensate for the movement of the subject by using a pan-tilt camera, and adding a magnetic head tracking device to feed head position information and command compensatory movements to the camera, in real time. Results are improved with this addition, but unfortunately at the expense of added complexity and cost.
Another approach is the Brain-Computer Interface (BCI) that uses electroencephalographic (EEG) waves originating in the brain. Researchers such as Wolpaw, McFarland, Neat, and Pfurtscheller (8,9) have focused on the detection of the mu rhythm, an 8-12 Hz brainwave of sinusoidal nature, occurring at the sensorimotor cortex (8). When a voluntary physical movement takes place, such as right hand movement, the contralateral mu rhythm is suppressed during the preparation of the movement. This is known as an ERD or Event-Related Desynchronization (10). Through the use of biofeedback, some individuals on a limited scale can eventually control these ERDs. The potential of this research as the basis for a computer control device for individuals with physical disabilities resides in the fact that these ERDs actually happen before the physical movement occurs (11). It has been conceptualized that a user of a BCI system would just have to "think" about a movement and the corresponding ERD would be generated in the motor cortex.
One of the major limitations of BCI systems is the high potential for electromyographic (EMG) contamination. EEG signals originate in the neurons of the brain and have to propagate through the skull and the pericranial muscles in order to reach the surface electrodes. Because the EEG signals are small in amplitude (5-300 µV), the EEG biopotential amplifiers are designed to incorporate high amplification (12). Thus, any muscle movement on the head or neck can produce a large noise contamination from the corresponding EMG signal. From an application standpoint, this is a big inconvenience to a user, especially if the user has a condition such as cerebral palsy.
Most BCI researchers have tried their best to eliminate any EMG artifacts, especially eye blinks and neck movements (8,9). However, one particular researcher has embraced certain EMG signals and has actually combined them with EEG signals. Andrew Junker, has developed the CyberlinkTM, a device that analyzes the EMG/EEG signal combination in the 0.5 to 45 Hz range where up to 12 control signals can be independently selected (13). The user's ability to operate the device is improved over time through the use of visual and auditory biofeedback. The training time between individuals varies depending on their ability to command the EMG/EEG control signals. This interface seems applicable to a wide range of users, due to the utilization of the easily produced and controllable EMG signals.
Although Junker's system incorporates the EMG biosignal, it also relies extensively on EEG brain activity and visual biofeedback, which can implicitly require the user to concentrate on producing a certain type of brain activity and to look at a visual bar graph that monitors the control frequencies (13). Even if the cursor-control capabilities are excellent, this intensive concentration and visual distraction will hamper the overall control process. Therefore, a system that is fundamentally dependent on EMG biosignals, which can be easily generated by a vast majority of users, is the aim of our proposed approach.
Rationale for EMG/EEG-Based Cursor Control
Our approach aimed at the design, development, and realization of an EMG/EEG Human-Computer Interface (HCI) that employs biosignals gathered from the head of the subject to be used as control signals for two-dimensional computer cursor movement. The movement commands for the cursor are derived by the interface from EMG signals associated with natural and voluntary movements of the user's face, such as pressing together the upper and lower teeth of each side of the head, and lifting or lowering the eyebrows. These face movements are still accessible and controllable for many individuals whose motor impediments are mainly from the neck down. Other functions include the "Left-Click" function and an ON/OFF switching function. EMG activity is associated with the mouse functions while the EEG activity disengages or engages the EMG electrodes via the ON/OFF mechanism.
For our interface, the data are received through four bio-electrodes placed above pericranial muscles and above the occipital lobe of the cerebrum. After analog-to-digital conversion, these biosignals are transformed into the appropriate mouse functions using digital signal processing (DSP) techniques based primarily on the power spectral density (PSD) estimation. These techniques are implemented in a dedicated DSP board whose output is in reverse TTL protocol for proper RS-232C conversion performed by external circuitry. The RS-232C serial data are routed into the port of a host computer, where they emulate the commands that would normally be generated by normal operation of an ordinary "hand-held computer mouse."
The EMG/EEG HCI for computer-cursor control is a complete system that provides two-dimensional axial movements as well as an ENTER or left-click function for "hands-off" interaction with computer programs. Because the mouse functions are controlled by easily produced EMG signals from muscles on the head, this HCI system can be operated by a large number of users.
Electrode Placement for the HCI System
The HCI System derives the basic communication tokens needed for computer interaction (UP, DOWN, LEFT, RIGHT, left-click) from monitoring the activity of several pericranial muscles. Accordingly, the location of the four electrodes used for the system is as shown in Figure 1.
Illustration of electrode placement on cranium.
Electrode E0 is placed on the forehead of the subject, displaced to the right, approximately three-quarters of an inch from the midline. This location was assigned for the monitoring of the electrical activity of the right frontalis muscle. Electrodes E1 and E2 are placed on the sides of the subject's head, to monitor the activity of the left temporalis muscle and the right temporalis muscle, respectively. Electrode E3 is placed on the back of the subject's head, between positions O1 and O2 of the International 10-20 Electrode Positioning System (14), to monitor the electrical activity of the visual cortex, in the occipital lobe. A reference electrode is placed on the left mastoid of the subject.
The electrodes used in our current prototype are Ag/AgCl cup electrodes applied to the scalp of the subject with double-adhesive disks. Application of these electrodes requires preparing the skin by gently rubbing with an alcohol-soaked cotton swab in order to minimize the impedance between the scalp and the cup electrode that is filled with electrode gel before application.
Basic EMG Cursor Control Approach
Our intent to associate each voluntary facial movement with a cursor movement command called for the detection of the activation of the muscle directly responsible for each motion. The electrodes were positioned with the intent to record the contraction of each one of these muscles primarily in only one of the electrodes. So, for example, the contraction of the left temporalis that results in a "left jaw movement" (i.e., pressing the left lower teeth against the left upper teeth), should be mainly detected as an increase of EMG amplitude at electrode E1. Additionally, a "full jaw clinch," which involves the simultaneous contraction of both temporalis muscles, causes EMG amplitude increases at electrodes E1 and E2. It was decided that the system could use that combined cue to interpret a left-click command. The intended relationship between voluntary facial movements, the main muscles that contract during these movements, the electrodes where the most significant EMG amplitude increase was expected, and the cursor commands associated with the movements is shown in Table 1.
Table 1. Intended relationship between facial movements and cursor commands. Facial
Eyebrows Up Frontalis E0 UP Left Jaw Movement Left Temporalis E1 LEFT Right Jaw Movement Right Temporalis E2 RIGHT Full Jaw Clench Left & Right Temporalis E1 & E2 "Left-Click"
(Left mouse click)
The electrode positions were selected so that they would lie on a circle around the head of the subject that could be covered by a sports headband, or possibly the headband of a baseball cap, as shown in Figure 2. This seemingly minor consideration is important in terms of electrode performance, since the headband will help keep the electrodes pressed against the skin; thus, improving the electrical conditions of the interface. Monitoring the temporalis muscles was preferred over monitoring the masseter muscles, which also contract during jaw movement, because the latter are heavily involved during the act of speaking, which would trigger false LEFT or RIGHT commands.
Application of the monitoring electrodes required for the EMG/EEG HCI System through the use of a headband: a) Mr. Julio Blandon demonstrates the lightweight headband consisting of the four monitoring electrodes; b) DSP board and interface circuitry used in the Human Computer Interface System.
The generation of a DOWN cursor command does not relate directly to the detection of the contraction of any of the muscles being monitored. The indirect mechanisms used to issue the DOWN command are explained in the following sections.
The Need for Frequency-Based EMG Online Classification
In spite of the deliberate placement of the electrodes over the frontalis and temporalis muscles, the volume conduction effect in the head of the subject causes varying levels of EMG signals to be sensed by electrodes other than the one assigned to the muscle contracting during each movement. So, for example, raising the eyebrows may cause a significant increase in the amplitude of electrode E2, originally devoted to the detection of the right temporalis movement. This cross talk between EMG increases due to different muscles prompted us to develop a more stringent classification of the EMG signals picked up by the electrodes.
Fortunately, EMG signals convey more information than just their average amplitude. In particular, it has been observed that EMG signals caused by the contraction of a muscle may have a significantly different frequency composition from EMG signals due to the contraction of another muscle. According to Bazzy and associates, as referenced by LeVeau and Andersson (15), this realization can be partly attributed to the dependence of the frequency content, specifically the mean frequency, on the contraction length of the muscle. Other factors that affect the frequency spectrum of the EMG generated by a muscle are the motor unit recruitment patterns, distinct motor unit properties (fast-twitch, slow-twitch), conduction velocity, and muscle fatigue (15).
For the HCI system, it is critical to differentiate between the EMG associated with frontalis contraction and that resulting from the contraction of a temporalis muscle. We proceeded to assess the frequency composition differences between EMG signals caused by the contractions of these muscles. Figures 3 and 4 show typical frequency representations of these contractions. These plots represent the Power Spectra obtained from analyzing 500-sample records of EMG signals collected from electrodes E0 and E1 during the contraction of the frontalis and the left temporalis, respectively.
500-sample PSD performed on an EMG recording from the frontalis muscle using electrode E0.
500-sample PSD performed on an EMG recording from the left temporalis muscle using electrode E1.
The PSD indicates how the total power of the signal analyzed is distributed along the frequency spectrum. These PSD estimates were obtained by calculating the Discrete Fourier Transform on EMG data digitized at 1 kHz, after application of a Hanning window. It should be noted that these PSD plots also reflect the filter settings in the bio-signal amplifiers used to record the EMG signals: the bandwidth of these amplifiers was limited to the range 10 Hz to 1 kHz. In addition, the 60-Hz notch filter for line interference canceling was turned ON in the amplifiers, which is reflected in both PSD plots.
These typical results show that the temporalis contraction tends to generate EMG signals with significant frequency components above 300 Hz, while the frontalis contraction has very small amounts of power associated with frequencies above 300 Hz. This observation was confirmed in several EMG recordings taken from a number of experimental subjects. In fact, based upon eighteen 500-sample PSD estimates performed on recordings from three subjects, there is approximately 23.45 percent of spectral power in the 300 to 500 Hz range for the temporalis muscle after the EMG data were passed through a bandpass filter, with cut-off frequencies at 8 Hz and 500 Hz. With the same constraints applied to the EMG associated with frontalis contraction, there was only 2.68 percent of spectral power in the 300 to 500 Hz range.
This same observation also serves as the basis for the signal processing approach ultimately implemented to successfully associate voluntary facial movements with cursor control commands. This processing approach, outlined in the EEG/EMG Classification Algorithm section under METHODS, overcomes the negative effects of EMG cross-talk due to volume conduction.
Table 2 shows EMG power content percentages from three subjects along with the final average percentage for each muscle. For each subject, three 500-sample PSD estimates were used from both the temporalis and frontalis EMG biosignals.
Table 2. EMG spectral power percentages from three subjects along with the final average percentage for each muscle. Temporalis Frontalis Subj % Subj % 1 34.32 1 1.85 1 32.89 1 2.09 1 28.52 1 1.13 2 25.93 2 2.85 2 28.88 2 3.71 2 20.57 2 2.85 3 15.60 3 3.40 3 8.31 3 3.28 3 16.07 3 2.98 Avg 23.45 Avg 2.68
Subj=subject; %=percentage of spectral power at 300-500 Hz; Avg=average spectral power percentage.
While the EMG cross-talk due to volume conduction required the implementation of a more complex classification approach, it also enabled the detection of the lowering of the eyebrows without an electrode placed directly over the muscles that contract during this movement. The muscles that effectively contract while an individual lowers his/her eyebrows are the procerus, the corrugators, and the orbicularis. These muscles are located over the bridge of the nose and above the eyelids, where it would be very awkward to place electrodes. Instead, the HCI system is capable of recognizing the contraction of these muscles because of the relative amplitude and frequency composition changes that their contraction causes in the EMG sensed by electrode E0, through volume conduction.
Finally, the electrical activity of the user's visual cortex is monitored with electrode E3, placed over the occipital lobe, in order to detect the presence or absence of the alpha rhythm for prolonged periods of time. The alpha rhythm is a characteristic oscillation at a frequency of 8 to 12 Hz that has been associated with a state of the visual cortex that reflects the lack of concentrated visual attention (16). A subject can voluntarily generate this rhythm while awake, by closing his/her eyes and not evoking any specific mental imagery (17). The HCI system uses the detection of voluntary alpha rhythm through electrode E3 as an indication to "engage" or "disengage" (depending on the previous state) the cursor control effects of the signals sensed by electrodes E0, E1, and E2. The capability of disengaging the EMG-driven cursor control is necessary if the user is about to perform very specific activities, such as eating, which can generate EMG signals similar to the ones purposely generated by the user to move the cursor.
Figure 5 illustrates the overall block diagram of the HCI system. The signals sensed by the four electrodes, E0, E1, E2, and E3, with respect to the reference electrode placed on the left mastoid of the subject, are amplified using four Grass® P5 Series AC biopotential preamplifiers. To accommodate the frequency and amplitude characteristics of the EMG signals being monitored, the bandwidth of these amplifiers was set to the interval 10 Hz to 1 kHz, with a gain of 10,000 V/V. For the signals sensed by electrode E3, monitoring the EEG of the occipital lobe to detect the presence or absence of the alpha rhythm, the bandwidth was set to 3 Hz to 30 Hz, and the gain to 200,000 V/V. In all cases, the 60-Hz notch filters of the biosignal amplifiers were used. The outputs of these preamplifiers have amplitudes within the range of -5 V and +5 V and are sent to 4 independent analog-to-digital converters (ADCs) included in the Innovative Integration® ADC 64 DSP board. In addition to the ADCs, this board includes the TMS320C31 DSP processor, which is the computational engine effectively implementing the algorithm described in the METHODS section entitled "EEG/EMG Classification Algorithm."
Block diagram of the Human Computer Interface System.
This ADC/DSP board is installed inside the PC for which the HCI system is developed. The results of the algorithm will cause the board to generate signals equivalent to those that a hand-held mouse generates to move the cursor UP, DOWN, LEFT, or RIGHT at three different speeds, or no signals at all if the cursor should remain static.
In addition, the ADC/DSP board will generate the appropriate sequence of signals to emulate a click with the left button of a hand-held mouse. Because of its voltage supply characteristics, the ADC/DSP board generates the appropriate sequences using 0 V and 5 V as the voltage levels to represent the binary digits in the codes. An RS-232C driver integrated circuit (MC1488P) was used to match the voltage levels used by the serial port of the computer, according to the RS-232 standard (18). This serial driver is a small component, powered directly from the PC. It can, in fact, be incorporated into the cable that connects the ADC/DSP board to the serial port input of the PC that would normally receive the mouse cable. The serial output of the HCI system has been designed to make it completely compatible with the standard serial mouse driver included with the Windows® operating system. This means that the HCI system can directly and immediately substitute a hand-held serial mouse. The use of an "A/B switch" allows both the HCI system and a mechanical mouse connected to the computer to be available, with the potential to select either one for use.
EEG/EMG Classification Algorithm
The basis for identification of the facial movements used to operate the HCI is the real-time frequency analysis of the electrode signals. The HCI system achieves this analysis by implementing the Periodogram estimation of the PSD for the four electrode signals in the ADC/DSP board. The PSD indicates how the power in a sampled signal is distributed along the frequencies from 0 Hz to half the sampling frequency. In the HCI system, each EMG electrode signal is sampled at a rate fS of 1200 samples/second. Then blocks of 256 sequential samples are collected to apply a Hanning Window and perform a Periodogram PSD estimation approximately every quarter second. These parameters provide a frequency resolution, Δf, of 4.68 Hz for the PSD estimation. Since the detection of the presence or absence of the alpha rhythm requires a finer frequency resolution, the signals from electrode E3 are collected in blocks of 512 samples, providing a PSD estimate approximately every half second, and achieving an improved frequency resolution of 2.34 Hz.
The determination of which facial movement is being executed is made by studying the relative power contents of some predefined frequency intervals within the PSD of the electrode signals. For the signals form electrodes E0, E1, and E2, the power accumulations in frequency ranges of importance are defined as:
Fk: From 0 Hz to 145 Hz
Jk: From 145 Hz to 600 Hz (half the sampling rate)
where k is the electrode number being considered (k=0,1,2).
Several criteria, involving the spectra of several of the electrode signals, must be met before the HCI will issue a cursor or "left-click" command. For example, a contraction of the left temporalis caused by a left jaw movement will be recorded if the PSD of electrode E1 surpasses a pre-defined threshold Th1 (for any frequency) and if the power accumulation in J1 is larger than the power accumulation in F1. This condition verifies that the signal sensed by electrode E1 has a frequency spectrum typical of a temporalis contraction and is different from the typical spectrum due to a frontalis contraction. Before the system acknowledges a left temporalis contraction (LEFT cursor movement), it will also verify that the power accumulation in J1 is larger than its counterpart for electrode E2, in interval J2, as expected in a unilateral jaw movement. The conditions that the system verifies before issuing a LEFT cursor movement command can then be summarized as:
(a) Conditions for LEFT CURSOR MOVEMENT:
If max(PSD1) > Th1
max(PSD0) < Th0, and max(PSD2) < Th2
If J1 > F1, and J1 > J2
Then: LEFT cursor movement.
(b) Conditions for RIGHT CURSOR MOVEMENT:
If max(PSD2) > Th2
max(PSD0) < Th0, and max(PSD1) < Th1
If J2 > F2, and J2 > J1
Then: RIGHT cursor movement.
(c) Conditions for LEFT-CLICK:
If max(PSD1) > Th1, and max(PSD2) > Th2
If J1 > F1, and J2 > F2
[J1/(J1+J2)], and [J2/(J1+J2)] > 0.22
(d) Conditions for UP CURSOR MOVEMENT:
If max(PSD0) > Th0
If F0 > J0
max(PSD1) < Th1, and max(PSD2) < Th2 Then: UP cursor movement.
As mentioned in the previous section, the detection of the lowering of the eyebrows is accomplished in an indirect manner. To discard the possibility of a jaw movement event, the power thresholds for electrodes E1 and E2 should not be surpassed. In fact, to detect activity in the muscles of the forehead that is not a frontalis contraction, the algorithm first requires that the threshold for maximum power in electrode E0 not be surpassed. Instead, the requirement that reveals contraction of other forehead muscles is the preponderance of power contents in the mid-frequency band for the signal of electrode E0. The quantitative index used is the accumulation of power within the following frequency range: M0 from 88 Hz to 250 Hz.
Then, the decision rule for the acknowledgment of a DOWN cursor movement is:
Conditions for DOWN CURSOR MOVEMENT:
If max(PSD0) > Th0
If max(PSD1) < Th1, and max(PSD2) > Th2
If [M0/(F0+J0)] > 0.55
Then: DOWN cursor movement.
In contrast with the spectrum of EMG signals, the presence of alpha rhythm is only expected in a fairly narrow band of frequencies, from 8 Hz to 12 Hz. Therefore, to engage/disengage the mouse control functions, the system requires that the threshold corresponding to the PSD of the occipital EEG (Th3) be surpassed within that alpha band. In addition, it is required that no other PSD peaks outside the alpha band surpass Th3.
To accommodate more realistic patterns of cursor motion, the HCI system incorporates graded speed control, according to the four directions of movement: UP, DOWN, LEFT, and RIGHT. If the system detects five consecutive commands indicating movement in the same direction, and if the following command is also in that direction, the system will issue the serial code corresponding to a bigger step. Initially, the system commands small movements in any direction, but the process of acceleration may repeat itself twice to make it command the largest of three possible step sizes. As soon as the sequence of commands in the same direction is broken, the step size for subsequent movements is reduced to the minimum.
In order to assess the possibility of practical use of the HCI interface, a test program was developed in Visual Basic® to exercise the point-and-clicking capabilities of the HCI system, in a controlled, timed environment. It is noted that no special provisions were taken in the development of this test program for reading the signals from the HCI. It used the standard mouse driver available in the Windows95® operating system.
The test program was used with the HCI system and a 17" color monitor. The test program presents a Start Button to the user in one corner of the screen. The dimensions of the Start Button are always 8.5 × 8.5 mm. The test program also presents a Stop Button; always at the center of the screen. There are four sizes for this target: 8.5 × 8.5 mm, 12.5 × 12.5 mm, 17 × 17 mm, and 22 × 22 mm. Before the beginning of each trial, the cursor is placed for the user at the Start Button. Then the subject is to use the HCI system to a) left-click on the Start Button, to start a timer; move the cursor toward the Stop Button, following any trajectory; and c) left-click on the Stop Button, to stop the timer. At the end of each trial, the test program reports the time, in seconds, taken by the user for the trial.
Each test session consisted of 20 trials with each size Stop Button (which varied from smallest to largest) for a total of 80 trials. Within each group of 20 trials with the same Stop Button size, the Start Button position was rotated through the four corners of the screen, from one trial to the next. Consequently, there were five trials starting at each corner for each Stop Button size.
The HCI system was tested with six subjects, using the Visual Basic program described above. All of the subjects were healthy volunteers aged 21 to 35 years. The ethnic backgrounds of the subjects included: Caucasian, Hispanic, and African-American.
Table 3 shows the average times taken by each subject to complete the trials from each starting corner. Also included is the average of all trial times, irrespective of the starting corner, for each subject. All these measurements are reported for the average of all subjects. The standard deviations associated to each of the computed averages are given.
Table 3. Average times taken by each subject to complete the trials from each corner to the center of the screen, in seconds. Lower Left Upper Left Upper Right Lower Right All Subj AVG SD AVG SD AVG SD AVG SD AVG SD 1 15.567 3.885 16.896 7.294 14.106 2.673 14.149 3.192 15.177 4.684 2 15.990 4.310 12.699 2.355 13.791 2.155 15.936 3.227 14.604 3.383 3 13.269 2.760 13.707 4.740 15.246 5.029 12.978 3.428 13.800 4.113 4 13.194 1.693 14.673 3.890 13.761 3.856 12.600 1.103 13.557 2.965 5 21.768 7.914 21.120 11.142 16.431 6.150 20.049 8.703 19.842 8.749 6 19.479 11.105 23.181 16.396 20.904 9.459 21.114 7.738 21.170 11.489 ALL 16.544 6.832 17.046 9.656 15.706 5.913 16.136 6.181 16.358 7.293
Subj=subject; AVG=average; SD=standard deviation.
The values in Table 3 show the viability of the proposed HCI system. The subjects in the trials required an average close to 16 seconds for displacing the cursor from any of the screen corners to the center and performing a left-click on the Stop Button. While this time is much longer that the one required by unimpaired subjects using a hand-held mouse (1-2 s), it represents a speed of response that is still usable with most standard GUIs.
This occurs especially in applications, such as World Wide Web browsing, that typically require intermittent selection of buttons or "hot links" to continue browsing. Similarly, the capabilities of this type of interface are expanded with the concurrent use of an "on-screen keyboard," allowing simple text strings to be entered. This type of task is often facilitated by the anticipatory feature commonly found in these programs, which will suggest alternatives for complete words or phrases as soon as the first few letters are keyed-in by the user.
Note that, in these particular scenarios, the overall advantage of more complex hands-off HCI systems (e.g., the eye-gaze trackers), with respect to our proposed HCI system, are diminished. Eye-gaze trackers will undoubtedly provide a faster displacement of the cursor across long distances in the screen. However, these systems present limitations in terms of the execution of accurate fine displacements, cursor stability and the required "dwell time" necessary for acknowledging a "left-click" operation. These limitations are likely to be a significant drawback in clicking on the active sections of many typical Web pages, as well as for "typing" letter-by-letter through an "on-screen keyboard."
While eye-gaze tracking computer interfaces have been justly recognized for the degree of independence that they can provide to subjects with severe motor disabilities (19), their own enthusiasts have acknowledged that the cost of these interfaces, estimated between $20,000 and $25,000, may be a limiting factor. In contrast with these observations, an initial cost estimate of the proposed EMG/EEG HCI System indicated that it could be sold for less than $2,000 if the general purpose biopotential amplifiers used for our custom-made amplifiers with fixed gains and frequency characteristics (20). Similarly, the proposed EMG/EEG HCI system is less prone to the loss of calibration (adjustment of the power summation thresholds Th0, Th1, Th2, Th3) during a given session than the eye-gaze trackers.
The EMG/EEG-based Human-Computer Interaction system presented represents a potential alternative for the communication of individuals with severe motor disabilities and their computers. Because the system commands cursor movements on the basis of detection and classification of EMG signals, its operation is relatively simple for the user. Use of the interface only requires the voluntary contraction of a set of cranial muscles, requiring little training on the part of the subject.
The operation of the system is possible due to the difference in the spectral composition of EMG signals generated by contraction of different cranial muscles. Thus, the novel classification approach allows the on-line separation of EMG signals caused by the contraction of different muscles, based on the real-time estimation of their power spectra.
In our evaluation of the interface, six experimental subjects were able to drive the computer cursor from the screen corners to the center and perform a click at each end of that trajectory in an average of 16 seconds. This level of performance achieved by untrained subjects reveals that this interface can be employed for such applications as web browsing and appliance control through appropriate hardware/software setups.
In comparison with other unassisted interfaces for users with disabilities, the EMG/EEG interface proposed has the potential to be more affordable and portable then others, such as eye-gaze tracking devices.
- 1. National Spinal Cord Injury Association, Factsheet #2: spinal cord injury statistics, 1995-98.
- Assistive Technology Act of 1998, Public Law 105-394, Enacted by the 105th Congress, November 13, 1998.
- Sax C, Fisher D, Pumpian I. Outcomes of students with severe disabilities: case studies on the use of assistive technology in inclusive classrooms. Technol Disabil 1996;5:327-34.
- Riviere CN, Thakor NV. Effects of age and disability on tracking tasks with a computer mouse: accuracy and linearity. J Rehabil Res Dev 1996;33(1):6-15.
- Salem C, Zhai S. An isometric tongue pointing device. Proceedings of CHI'97, March 22-27; 1997.
- Jacob RJ. The use of eye movements in human-computer interaction techniques: what you look at is what you get. ACM Trans Inform Syst 1991;9(3):152-62.
- Foulds R, Arthur J, Khan A. Human factors studies in eye movements related to AAC head movement studies. VA Rehabil RD 1996 Progr Rep 1997;34:155-6.
- Wolpaw JR, McFarland D, Neat GW, Forneris CA. Multichannel EEG-based brain - computer communication. Electroencephalogr Clin Neurophysiol 1991;78.
- Pfurtscheller G, Flotzinger D, Pregenzer M, Wolpaw JR, McFarland D. EEG-based brain computer interface (BCI). Med Progr Technol 1996;21.
- Pfurtscheller G, Brain-computer interface--a new communication device for handicapped persons. J Microcomput Appl 1993;16.
- Jasper HH, Penfield W. Electrocorticograms in man: effect of the voluntary movement upon the electrical activity of the precentral gyrus. Arch Psychiat Z Neurol 1949;183.
- Taberner AM, Barreto AB. Real-time signal processing towards an EEG-based human-computer interface. Proceedings of the 1997 Florida Conference on Recent Advances in Robotics, Miami, FL; 1997. p. 56-60. In: Webster JG, editor. Medical instrumentation: application and design, 3rd Ed. Boston: Houghton Mifflin Company; 1998.
- Junker A. Brain-body actuated system. United States Patent, #5,474,082, December 12, 1995.
- Jasper H. The ten-twenty electrode system of the international federation. Electroencephalogr Clin Neurophysiol EEG J 1958;10:Appendix.
- LeVeau B, Andersson G. Output forms: data analysis and applications. Interpretation of the electromyographic signal, selected topics in surface electromyography for use in the occupational setting: expert perspectives. U.S. Department of Health and Human Services, NIOSH Pub. No. 91-100; March 1992.
- Gaarder KR. Eye movements, vision and behavior. Washington, D.C.: Hemisphere Publishing Corporation; 1975.
- Taberner A M, Barreto AB. Real-time signal processing towards an EEG-based human-computer interface. Proceedings of the 1997 Florida Conference on Recent Advances in Robotics; 1997. p. 56-60.
- Electronic Industries Association Standards Publication EIA 232E--Interface between data terminal equipment and data circuit-terminating equipment employing serial binary data interchange.
- Lynch WJ. Hardware update: The Eyegaze Computer System;tm. J Head Trauma Rehabil 1994;9(4):77-80.
- Scargle S. EMG/EEG HCI system for computer-cursor control, (Masters thesis). Florida International University; 1998.
Contents Vol. 37, No. 1
Back to Top