Journal of Rehabilitation Research & Development (JRRD)

Quick Links

  • Health Programs
  • Protect your health
  • Learn more: A-Z Health
Veterans Crisis Line Badge
 

Volume 47 Number 2, 2010
   Pages 87 — 98

Feasibility of controlling prosthetic hand using sonomyography signal in real time: Preliminary study

Jun Shi, PhD;1* Qian Chang, BSc;1 Yong-Ping Zheng, PhD2

1School of Communication and Information Engineering, Shanghai University, Shanghai, China; 2Department of Health Technology and Informatics, Hong Kong Polytechnic University, Hong Kong SAR, China

Abstract — The morphological changes of muscle can be accurately detected by sonography, a process we have termed sonomyography (SMG). This article investigates the feasibility of using muscle thickness deformation SMG as a new signal source to control a prosthetic hand in real time. Thickness deformation SMG of the extensor muscle was measured by a block-matching algorithm during wrist extension-flexion; the amplitude of the deformation was used to control the prosthetic hand. We compared various fast-search algorithms to select the best one for real-time prosthetic control. The two-dimensional logarithmic search (TDL) algorithm, with and without streaming single-instruction multiple-data extensions, showed excellent execution efficiency, with an overall mean correlation coefficient of about 0.99, a mean standard root-mean-square error <0.75, and a mean relative root-mean-square error <8.0% referenced to the cross-correlation algorithm baseline. The mean frame rates were greater than the ultrasound sampling rate (12 Hz), indicating that TDL could be implemented in real-time control. These results demonstrate that only one muscle position is needed to control a prosthetic hand, allowing for proprioception of muscle tension, and that the SMG provides good control of the prosthetic hand, allowing it to proportionally open and close with a fast-search algorithm.

Key words: block-matching algorithm, electromyography, prosthetic control, prosthetic hand, real time, skeletal muscle, sonomyography, streaming SIMD extensions, two-dimensional logarithmic search, ultrasound.


Abbreviations: BMA = block-matching algorithm, CC = cross-correlation, DOF = degree of freedom, EMG = electromyography, MK = myokinemetric, MMG = mechanomyogram, RRMSE = relative root-mean-square error, SAD = sum of absolute difference, SD = standard deviation, SEMG = surface electromyography, SIMD = single-instruction multiple-data, SMG = sonomyography, SRMSE = standard root-mean-square error, SSE = streaming SIMD extensions, TDL = two-dimensional logarithmic search.
*Address all correspondence to Jun Shi, PhD; School of Communication and Information Engineering, Shanghai University, Box 159, 149 Yan Chang Road, Shanghai, PR China; +86-21-56331787; fax: +86-21-56331787.
Email: junshi@staff.shu.edu.cn
DOI:10.1682/JRRD.2009.03.0031
INTRODUCTION

Surface electromyography (SEMG) is the control source for almost all powered upper-limb prosthetic devices. Over the past decade, researchers across the world have expended a great deal of effort to improve the control accuracy of electromyography (EMG)-based prostheses [1-4]. However, some inherent limitations have not yet been overcome [3-4]. For example, it is believed that considerable attention is required to control EMG-based prostheses, often resulting in fatigue [4-5]. It is also difficult for SEMG to detect the activities of deep muscles because the deep muscle EMG may be attenuated and/or mixed with the superficial muscle EMG at the skin surface [4,6-7]. Furthermore, the lack of proprioceptive feedback is one of the major disadvantages for SEMG-based control [2,3-5].

Alternatives to EMG have been investigated for use in prosthetic control, including the mechanical force of muscles measured via a tunnel cineplasty [8]; the morphological changes of residual limb tissues [9]; the myokinemetric (MK) signal measured from the muscle dimensional change that is observable at the skin surface [10]; the mechanomyogram (MMG), which is detected from the sound signal generated by muscle contraction [11]; speech [12]; and the electroencephalogram [13]. Although most of the alternative approaches just described are laboratory prototypes, they represent a diversification of prosthetics research and may accelerate the development of improved methods for prosthetic control.

Muscle structure is one of the primary determinants of muscle function, and structural changes in muscles are closely associated with muscle activities [14]. Therefore, muscle structure could potentially be used to control a prosthetic hand. Ultrasonography can accurately and reliably detect the structural changes of skeletal muscle, such as muscle thickness [7,15-16], muscle fiber pennation angle [15-16], muscle fascicle length [15,17], and cross-sectional area [18] during isometric and dynamic contractions. The relationships between EMG activities and changes in muscle structure extracted from ultrasound images have also been reported [7,15-16,19], resulting in a characterization of the muscle properties from both electrophysiology and structural morphology.

In previous studies, we used the term sonomyography (SMG) to describe the structural and morphological changes of skeletal muscle extracted from ultrasound images [7,20-22] and showed that the SMG signal is different from the sound signal generated by muscle contraction, which is commonly called MMG. We studied the relationship between the thickness deformation SMG of the forearm extensor and wrist angle during wrist extension-flexion. We discovered qualitatively similar profiles between the SMG and the wrist angle, suggesting a high degree of correlation [7,20-22]. Furthermore, we found that SMG profiles from amputees performing hypothetical wrist extension-flexion were similar to those from nondisabled subjects [20]. We then studied a prosthetic hand with 1 degree of freedom (DOF) that was controlled by the SMG signal obtained from nondisabled subjects off-line [23]. However, the time to calculate an SMG value from one image was far slower than the sampling time of one ultrasound image because of the slow speed of the search algorithm that extracted the SMG signal; as a result, we were unable to implement a real-time control experiment [23]. Nevertheless, these studies indicate the potential for SMG to control a prosthetic hand. The key to achieving real-time control is to find an efficient algorithm that is able to extract the SMG signal in real time and replace the currently used full-search block-matching algorithm (BMA) [23].

The process of extracting the SMG signal is similar to motion estimation or object tracking, which commonly make use of optical flow and BMAs [24-25]. BMAs are more popular because of their robustness in complicated environments with a low signal-to-noise ratio [24,26]. They usually use evaluation criteria such as cross-correlation (CC), mean absolute error, mean square error, and the sum of absolute difference (SAD) [24,27]. The most straightforward and optimal BMA is full search, which searches exhaustively inside the search window to find the motion vector but requires high computational complexity [24,26-27]. Many fast BMAs have been proposed to alleviate this problem, such as n-step hierarchical search, new three-step search, cross-search, two-dimensional logarithmic search (TDL), block-based gradient descent search, conjugate directional search, edge-assisted search, diamond search, hexagon search, octagon search, multiresolution BMA, and enhanced predictive zonal search [24,26-27]. The TDL algorithm is a suboptimal, but fast, search algorithm proposed by Jain that could enable a trade-off between search quality and easy program implementation [27-28].

Most modern microprocessors feature special architectural design to boost their computing performance. For instance, a set of single-instruction multiple-data (SIMD) registers have been employed to parallelize data operations within the processor [29]. In 1999, Intel's Pentium III processor family first introduced the streaming SIMD extensions (SSE) instructions (Intel Corp; Santa Clara, California). SSE expand the SIMD execution model by a new set of 128-bit registers to provide the ability to perform SIMD operations on packed and scalar single-precision floating-point values [29-30]. SSE2 was then introduced in 2001 along with the Pentium IV and Intel Xeon processors [30] to enable more computations in parallel. Consequently, parallel algorithms can be implemented and the computational costs can be significantly reduced by fully exploiting the SSE series instruction set. SSE instructions have been extensively used in image- and signal-processing algorithms [31-32].

This article investigates the feasibility of using SMG as a real-time control source for a prosthetic hand with SSE-embedded fast BMA. An SMG signal was extracted by BMA from the extensor carpi radialis muscle to control a 1-DOF prosthetic hand with proprioception of muscle tension. We compared different BMAs with SSE acceleration to select the best one for real-time control. We aim to design a new prosthetic control technique with proprioception.

METHODS
Control Strategy

In previous studies, we have proposed that the thickness deformation SMG of extensor carpi radialis muscle was closely correlated with wrist extension-flexion [7,20,23]. This close relationship is the basis for SMG-based prosthetic control. The percentage SMG ( r) is defined as


Equation 1.

Equation 1. Click Image to Enlarge
In This Window

View Image In New Window

where d0 is the initial muscle thickness and d is the current muscle thickness.

A 1-DOF SEMG-based prosthetic hand was constructed before implementing the SMG-based control strategy. The principle of the SEMG-based prosthetic hand was as follows: two-channel SEMG signals were acquired from the antagonistic flexor and extensor muscles of the forearm, respectively, and a stepping motor was driven by the two SEMG amplitudes to control the opening and closing of the prosthetic fingers. Before using SMG to control this prosthetic hand, we connected two analog signal output pins of a data acquisition card (NI PCI-6024E, National Instruments Corp; Austin, Texas) to the appropriate input pins of the prosthetic hand, as shown in Figure 1. One pin controlled finger opening and another controlled finger closing. A pulse signal with enough amplitude to step-by-step control the fingers opening or closing was then generated through the analog output pins. The step number of the motor from fully closed to fully open was recorded as M and that from fully open to fully closed was N. The new SMG-based control strategy for proportional grasp was implemented on a PC as follows:

1. Sample the ultrasound images from the extensor carpi radialis muscle in the process of wrist extension-flexion and select one image sequence that included more than two continuous extension-flexion cycles as training data.
2. Calculate the SMG values according to Equation (1) from the training ultrasound images by way of BMA.
3. Determine the maximum and minimum SMG values in the training data and then calculate their difference (symbold). Alternatively, symbold could be manually set instead of automatically calculated.
4. Segment the symbold into M and N segments. Each segment is tagged as 1, 2, 3 . . . M or N in sequence, and the numerical ranges are recorded for each segment.
5. Calculate the SMG from the test images and then judge the SMG value at the rising or falling edge by comparing it with the previous three values. The SMG value at the rising edge (corresponding to wrist extension) is used to open the fingers, while the value at the falling edge (corresponding to wrist flexion) is used to close the fingers.
6. Control the fingers to the desired open or closed position by the calculated frame-by-frame SMG values. The step movements of the motor are determined by mapping the SMG value to the corresponding tagged segment range in Step 4. When the SMG values are kept constant, the fingers stay in a certain position or at a certain force.

Figure 1. Click Image to Enlarge
In This Window

View as PowerPoint Slide   ¦    View Image In New Window

Search Algorithms

The core of our control strategy was to accurately and efficiently extract the SMG signal from ultrasound images, which is very computationally expensive for the whole control process. The simple search algorithm CC was successfully used in our previous work to track the SMG [16,21-22]. Figure 2 shows a typical cross-sectional ultrasound image of the extensor carpi radialis muscle with two rectangular blocks on the upper and lower boundaries as two templates. Two blocks were manually added on the first frame, and the centers of the upper and lower blocks were at the subcutaneous fat-muscle interface and the upper boundary of the radius, respectively. Then, block sizes were adjusted to include enough features (the most obviously reflected echo regions of the upper and lower boundaries) for reliable tracking with a correlation coefficient (R2) >0.9 between two conjunctive frames. The vertical distance between the centers of the two blocks was calculated as the muscle thickness for each frame. The equation of normalized two-dimensional CC is given by


Equation 2.

Equation 2. Click Image to Enlarge
In This Window

View Image In New Window

where symbol and symbol are the means of pixel density for the image blocks x(m,n) and y(m+i,n+j), respectively. When matching was finished for one frame, the templates were automatically updated using the most similar image blocks in the current frame. After the initial templates were manually selected, tracking was then automatically performed for each subsequent image.


Figure 2. Click Image to Enlarge
In This Window

View as PowerPoint Slide   ¦    View Image In New Window

Although CC was effective in terms of search quality, its high computational complexity prevented real-time implementation of prosthetic control. Therefore, this study compared the following five fast-tracking algorithms: SAD, TDL, CC with SSE, SAD with SSE2, and TDL with SSE.

SAD

SAD is a popular BMA estimated by summing the absolute difference between each pixel and its corresponding pixel in the reference block [27]. The best-matched reference block is the one with the minimum SAD value. The SAD is given by


Equation 3.

Equation 3. Click Image to Enlarge
In This Window

View Image In New Window

where x(m,n) and y(m+i,n+j) are the pixels in the image blocks.

TDL

TDL is based on a basic coarse-to-fine approach to testing several points at the present step and restricting the search in the next step to the vicinity of the point [27-28]. This algorithm is carried out using three steps.

Step 1

Pick an initial step size as k and set the block center as the search center point. Four points at a distance of k from the center point p(x,y) axes are selected as other search points. Thus, the five positions form a "+" shape.

Step 2

Search the best-matched position by CC. If the best-matched point is at the center, halve the step size. If one of the other four points is the best match, then it becomes the center and step 1 is repeated.

Step 3

When the step size becomes 1, all 9 blocks around the center are chosen for the search and the best among them is picked as the desired block center.

CC with SSE

In CC with SSE acceleration, the 32-bit floating-point SSE data type was used because of the division operation in Equation (2). Two 128-bit packed registers were loaded with four template values (as a vector) and four tracked values, respectively. SIMD operations including parallel addition, parallel subtraction, and parallel multiplication were then conducted on the registers to implement the CC algorithm.

SAD with SSE2

SSE2 can implement up to 16 integers simultaneously, while SAD only needs to perform a subtraction operation of 8-bit integers. Therefore, SAD can be easily and efficiently parallelized with SSE2. After the 8-bit elements were loaded into the 128-bit registers in the same way as in CC with SSE, they were subject to the parallel absolute difference instruction, which directly calculated the SAD of the corresponding data.

TDL with SSE

In the TDL with SSE algorithm, only the CC module was programmed with SSE acceleration, which was previously described.

Experiment

Seven nondisabled subjects participated in the experiment (mean ± standard deviation [SD] age = 26 ± 3 yr; height = 170 ± 3.5 cm; and weight = 62 ± 4.7 kg). None of these subjects had any previous history of neuromuscular disorder.

As shown in Figure 1, the subject sat on a chair with his/her right forearm on the table. Each subject was instructed to repeatedly perform wrist extension starting from the neutral position and then returning to the neutral position. Hereafter, this motion is referred to as wrist extension-flexion. The term "flexion" here indicates the action of returning from an extension state to the neutral position. After several warm-up contractions, each subject performed wrist extension-flexion at an extension rate of 22.5 cycles/min guided by a metronome (MT-40, Wittner GmbH & Co KG; Isny/Allgäu, Germany). Three trials were performed for each subject with a break of 2 minutes between each trial, and more than five wrist extension cycles were performed in each trial. The ultrasound images of a cross-sectional area of the extensor carpi radialis muscle were collected during the whole process using a portable B-mode ultrasound scanner (180 Plus, SonoSite, Inc; Bothell, Washington) with a 7.5 MHz, 38 mm linear probe. The probe was fixed by a custom-made bracket to contact the skin surface without excessive pressure. The video output of the B-mode ultrasound scanner was then digitized by a video capture card (NI PCI-1411, National Instruments Corp) with a sampling rate of 12 Hz. The signal channel SEMG signal and wrist angle were also synchronously sampled for the purpose of other research, and details of this data acquisition can be found in Shi et al. [21].

The data acquisition was controlled by custom-developed software in Visual C++ 6.0 (Microsoft Corp; Redmond, Washington). A total of 200 frames was saved as one sequence for every trial. In the present study, only the ultrasound images were processed off-line for prosthetic control.

A total of 21 image sequences from the 7 subjects were processed. The positions of reflected echoes from the probe-skin interface were almost immovable because the ultrasound probe touched the skin surface. Therefore, the CC algorithm was performed to the block on the upper boundary because of the slight movement of this boundary during the experiment. The five fast-tracking algorithms were only implemented for the block on the lower boundary of the extensor, which moved drastically during wrist extension-flexion. Two kinds of blocks with different block sizes, defined as small and large templates, were selected on the lower boundary for every image sequence. The small templates were able to cover just enough features for tracking, including the most obvious reflected echo regions of radius with the block centered at the bone-muscle interface, while the large templates covered 1.5-2 times larger regions than the small ones. All five tracking algorithms were performed using the same templates for the same sequence data every time on a PC (Intel Pentium IV, 3 GHz CPU, 1 GB RAM). The CC algorithm was also implemented as a baseline for comparison.

Evaluation Indices

Numeric validation was carried out to quantitatively evaluate the tracking accuracy and efficiency of the different algorithms. The R2, standard root-mean-square error (SRMSE), and relative root-mean-square error (RRMSE) were calculated to evaluate the accuracies of the five tracking algorithms in comparison with the CC baseline [22,33]. The RRMSE is given by


Equation 2

Equation 4. Click Image to Enlarge
In This Window

View Image In New Window

where d(i) is the SMG calculated from CC and d¢(i) is the SMG obtained from the tested tracking algorithm. The mean execution time, mean frame rate, and mean time-saving percentage were calculated to evaluate the realized time efficiencies. The execution time is defined as the time to finish searching all 200 frames of one sequence, and the mean execution time is the average of the execution time for all 21 sequences. The time-saving percentage is given by


Equation 5

Equation 5. Click Image to Enlarge
In This Window

View Image In New Window

where T1 is the execution time of CC and T  2 is the execution time of the tested tracking algorithm.

RESULTS

Figure 3 is a typical example that illustrates the qualitatively similar profiles of the original wrist angle and SMG signals, suggesting a high degree of correlation. A more detailed and quantitative analysis of the relationships between SMG and wrist angle during wrist extension-flexion movements can be found in Shi et al. and Xie et al. [21-22].


Figure 3

Figure 3. Click Image to Enlarge
In This Window

View as PowerPoint Slide   ¦    View Image In New Window

Table 1 shows the tracking accuracy indices for different algorithms with the data in the format of mean ± SD. The overall mean R2 is about 0.99, the overall mean SRMSE is less than 0.75, and the overall mean RRMSE is less than 8.0 percent for SAD and TDL with both large and small template blocks, indicating that no significant differences exist between the tracking results of SAD, TDL, and CC. Therefore, the SAD and TDL algorithms could be reliably used for accurate tracking. Table 2 presents the indices for the execution efficiencies of different algorithms. The execution times of different search algorithms are sorted in descending order in Table 2 from CC to TDL with SSE. The SSE-optimized algorithms drastically reduced the execution times compared with the original algorithms. The execution time of TDL with SSE is more than three times shorter than that of CC for both the large and small template blocks. Furthermore, the mean frame rates of SAD with SSE2 and TDL with and without SSE are all greater than the ultrasound sampling rate of 12 Hz for small template blocks, which means that these algorithms could theoretically control a prosthetic hand in real time.


Table 1.
Indices (presented as mean ± standard deviation) to evaluate accuracies of SAD and TLD algorithms.

Parameter
 
SAD
TLD

Large Template
R2
 
0.99 ± 0.01
0.99 ± 0.01
SRMSE
 
0.54 ± 0.19
0.66 ± 0.27
RRMSE (%)
 
6.08 ± 2.86
7.45 ± 2.36
Small Template
R2
 
0.99 ± 0.01
0.98 ± 0.01
SRMSE
 
0.54 ± 0.16
0.71 ± 0.19
RRMSE (%)
 
5.86 ± 2.75
7.31 ± 2.17

RRMSE = relative root-mean-square error, SAD = sum of absolute difference, SRMSE = standard root-mean-square error, TDL = two-dimensional logarithmic search.



Table 2.
Indices (presented as mean ± standard deviation) to evaluate efficiencies of different algorithms.

Algorithm
Execution Time (s)
Frame Rate (Hz)
Time-Saving Percentage (%)
Large Template
Small Template
Large Template
Small Template
Large Template
Small Template

CC
50.32 ± 6.72
38.81 ± 4.87
4.05 ± 0.59
5.15 ± 0.68
-
-
CC SSE
38.20 ± 5.23
29.25 ± 3.18
5.34 ± 0.79
6.84 ± 0.77
24.04 ± 3.40
24.44 ± 2.81
SAD
30.26 ± 3.48
24.07 ± 2.32
6.70 ± 0.84
8.31 ± 0.82
39.69 ± 2.24
37.68 ± 3.38
SAD SSE2
19.57 ± 1.91
16.60 ± 1.01
10.31 ± 1.03
12.05 ± 0.75
60.70 ± 4.26
56.81 ± 3.86
TDL
16.09 ± 1.25
14.66 ± 1.32
12.51 ± 1.01
13.65 ± 1.28
67.62 ± 3.92
61.77 ± 5.25
TDL SSE
13.07 ± 0.57
12.71 ± 0.33
15.32 ± 0.66
15.73 ± 0.39
73.58 ± 3.60
66.75 ± 4.26

CC = cross-correlation, SAD = sum of absolute difference, SSE = streaming single-instruction multiple-data extensions, TDL = two-dimensional logarithmic search.


Using the fast-search algorithms SAD with SSE2 and TDL with and without SSE, we tested real-time control using the sampled data off-line. The prosthetic fingers were opened or closed by the real-time calculated SMG frame by frame. When the SMG values were kept constant, the prosthetic fingers were also held in a certain spatial position. All the data sampled from seven subjects have been tested, and the results demonstrated that SMG could be used to smoothly control a prosthetic hand with the fast-search algorithms SAD with SSE2 and TDL with and without SSE.

A data set previously acquired from an amputee was also used to test the proposed algorithm. More information about these data can be found in Zheng et al. [20]. Because the original frame rate was much greater than 12 Hz, the ultrasound images were resampled to decrease the frame rate to 10 Hz. The prosthetic fingers were able to open or close when driven by SMG calculated in real time, although the opening and closing operations were not very smooth since the SMG signal was not smooth.

DISCUSSION

This article validates the feasibility of using SMG to control a 1-DOF prosthetic hand in real time. The SMG signal extracted from the forearm extensor was able to smoothly control the opening and closing of the prosthetic hand with the fast BMA. SMG-based control has the advantage of proprioception of muscle tension to enable proportional grasping.

As shown in Table 1, the high overall mean R2 and low overall mean SRMSE and RRMSE indicate good consistency between the CC, SAD, and TDL algorithms for both large and small template blocks. However, sometimes it was difficult to set small blocks for SAD. Particularly, tracking errors would occasionally occur when the blocks were set too small. Although a large enough block ensured accurate tracking of SAD, it came at the cost of longer execution time. Moreover, the frame rate of SAD with SSE2 was less than 12 Hz for the large block, which would affect the real-time control. Therefore, SAD is not the best search algorithm in our experiments.

As shown in Table 2, the mean execution time of CC for small blocks is much less than that for large blocks, while the mean execution times of TDL for large and small blocks are very close and the same is true of TDL with SSE. Therefore, the block size will not drastically affect the execution time for TDL with and without SSE, which simplifies the template setting and shortens the training process. Furthermore, the frame rates of TDL with and without SSE are both greater than the ultrasound sampling rate of 12 Hz for both large and small blocks, indicating that the fast-search algorithms TDL with and without SSE provide optimal tracking in this setting. Hence, these results prove the feasibility of using an SMG signal to control a prosthetic hand in real time with TDL with and without SSE. Furthermore, these fast BMA algorithms could be used in other similar experiments of real-time tracking of muscle motion.

SSE can obviously improve the execution efficiency compared with the original algorithms without SSE. The only disadvantage of this method is that the SSE instructions are designed for the Pentium III or higher processor series, which limits the application of SSE to a certain extent. However, the TDL algorithm is fast enough to enable real-time control when using a computer without SSE instructions. Moreover, the TDL algorithm implements about 1.8 × 108 addition/subtraction operations per second and 6 × 107 multiplication operations per second, indicating that it could be migrated to other processors such as the digital signal processor. For example, DSP chip TMS320DM642 (Texas Instruments, Inc; Dallas, Texas) can implement 2.4 × 109 addition/subtraction operations per second and 1.2 × 109 multiplication operations per second.

As just mentioned, although the SEMG-based prosthetic hand has been widely used, it still suffers from some inherent limitations. The MK signal is measured by a Hall sensor as the dimensional change of muscles beneath the skin while they bulge during contraction [10]. The signal processing for electromagnetic effects and sensor attachments are still challenging when collecting the MK signal [7]. The MMG can be affected by many factors such as muscle temperature, skinfold thickness, socket movement, and external mechanical noise [4,10,34]. These factors, together with challenges in low-frequency noise elimination, can reduce the stability and reliability of the MMG signal [4,7]. SMG reflects the structural and morphological features of skeletal muscle, and the morphological changes are easily extracted from ultrasound images with little conscious effort. Ultrasound is more spatially localized than EMG. The changes in muscle thickness are calculated by detecting the shift of the reflected ultrasound echoes from the fat-muscle and muscle-bone interfaces. Therefore, the SMG signal is able to noninvasively detect muscle thickness changes at different depths or locations, effectively avoiding crosstalk from adjacent muscles. SMG is stable and less influenced by external noise such as movement artifacts or power supply noise and also less influenced by disturbance from the fat or surface skin impedance, as shown in our previous studies [20-21]. Therefore, SMG is a robust method for prosthetic control.

In contrast with SEMG and MMG signals, SMG has a proprioceptive feedback aspect for prosthetic control so that the user can be aware of the degree of muscle tension through proprioception. This is a very important advantage of the SMG-based prosthetic hand. Sensory feedbacks, including pressure, vibration, and position, have been widely investigated for use in prostheses [5,35-36]. The extensor thickness deformation appears as the obvious changes in the bulge of the skin surface during the process of wrist extension-flexion due to muscle contraction. Because the ultrasound probe was fixed to contact the skin surface during the whole process, the user can sense the pressure changes from the probe caused by muscle tension through cutaneous receptors. Thus, the user can control where the prosthetic finger is going or what kinds of forces are acting on the finger through tension proprioception. Another proprioceptive feedback provided by SMG-based control is the thickness changes of the extensor originated from the stretch receptor of muscle spindles, although these contribute less than muscle tension. Therefore, the prosthetic hand can be controlled with proprioceptive feedback, and only one muscle position is needed for control. This proprioceptive feedback will improve the control and dexterity of the hand, reduce the conscious effort required for control, and delay muscle fatigue [2-3,5], which will benefit the users.

At present, the control signal was only acquired from the extensor carpi radialis muscle with incomplete wrist extension-flexion. Although this control strategy works well, in future work we will simulate the SEMG-based 1-DOF prosthetic hand to collect the SMG from both the extensor and flexor with a multichannel A-mode ultrasound system for higher control accuracy. Thus, the maximum SMG values of the extensor for full wrist extension will correspond to the full finger-open position, while the minimum SMG values of the flexor for full wrist flexion will correspond to the full finger-closed position. With the increase of the difference (symbol) between maximum and minimum SMG values, the moving motor step is predicted to be driven more accurately.

In our previous work, we studied the relationships among A-mode thickness deformation SMG , SEMG , and wrist angle in wrist extension-flexion, and we also found that the A-mode SMG could be reliably extracted with a single-element ultrasound transducer by way of a one-dimensional CC tracking algorithm [7]. The A-mode SMG-based prosthetic hand is currently being studied. The ultrasound echoes reflected from the fat-muscle and muscle-bone interfaces of the extensor carpi radialis muscle had obvious amplitudes. We could detect these two adjacent echo envelopes automatically by calculating the amplitudes or by comparing them with the preset echo templates. Therefore, the manual template selection could potentially be eliminated in a real-time implementation with A-mode ultrasound. The advantages of A-mode SMG should also be further verified together with other factors that may affect the collection and extraction of the A-mode SMG signals.

Compared with the SEMG, MK, and MMG-based prosthetic hands, the proposed method is still in a preliminary stage of development. Although the traditional B-mode ultrasound probe is too big for real application to prosthetic control, the SMG signal extracted from the B-mode ultrasound image provides visual two-dimensional images. These images enable an investigation of the feasibility of the method and can be used to construct theoretical models describing features of residual muscles for future work using A-mode ultrasound SMG. Moreover, the utilization of a miniature B-mode probe with a custom-developed B-mode ultrasound acquisition system to obtain ultrasound images is currently under study. Our aim is to combine the SMG (both A-mode and B-mode) together with SEMG to control a multi-DOF prosthesis; these two signals could yield more information about the residual muscle activity than either alone, and the SMG signal could provide proprioceptive feedback to improve the qualities of control and dexterity. More multisignal experiments are required in future work.

A data set acquired from an amputee has been used for prosthetic control, preliminarily proving the feasibility of the SMG-based control strategy. Amputees can feel the level of contraction of their residual muscles, which in turn is related to wrist extension-flexion. However, different degrees of injury in the residual limb may affect the amplitude and stability of SMG signals. The problem of fixing the residual muscle should also be considered. The suitability of the SMG-based (A-mode and B-mode) control strategy should be investigated for more amputees in the future.

CONCLUSIONS

This work studied the feasibility of using an SMG signal extracted from the forearm extensor as a real-time control source for a 1-DOF prosthetic hand with the advantage of proprioceptive feedback. The prosthetic hand was successfully driven to proportionally open and close with the TDL algorithm with and without SSE acceleration in this preliminary work. We aim to explore a new technique with proprioception for prosthetic control, and this work can contribute to human rehabilitation engineering for upper-limb amputees.

ACKNOWLEDGMENTS
Author Contributions:
Study concept and design: J. Shi, Y. P. Zheng.
Acquisition of data: J. Shi.
Analysis and interpretation of data: Q. Chang.
Drafting of manuscript: J. Shi.
Financial Disclosures: The authors have declared that no competing interests exist.
Funding/Support: This work was supported by the National Natural Science Foundation of China (grant 60701021), the Leading Academic Discipline Project of Shanghai Educational Committee (grant J50104), and the Hong Kong Research Grant Council (grant PolyU 5331/06E).
Institutional Review: Informed consent was obtained from each subject before the experiment.
Participant Follow-Up: The authors plan to inform participants of the publication of this study.
REFERENCES
1. Zecca M, Micera S, Carrozza MC, Dario P. Control of multifunctional prosthetic hands by processing the electromyographic signal. Crit Rev Biomed Eng. 2002;30(4-6):459-85. [PMID: 12739757]
DOI:10.1615/CritRevBiomedEng.v30.i456.80
2. Light CM, Chappell PH, Hudgins B, Engelhart K. Intelligent multifunction myoelectric control of hand prostheses. J Med Eng Technol. 2002;26(4):139-46. [PMID: 12396328]
DOI:10.1080/03091900210142459
3. Oskoei MA, Hu HS. Myoelectric control systems-A survey. Biomed Signal Process Control. 2007;2(4):275-94.
DOI:10.1016/j.bspc.2007.07.009
4. Ohnishi K, Weir RF, Kuiken TA. Neural machine interfaces for controlling multifunctional powered upper-limb prostheses. Expert Rev Med Devices. 2007;4(1):43-53.
DOI:10.1586/17434440.4.1.43
5. Cipriani C, Zaccone F, Micera S, Carrozza MC. On the shared control of an EMG-controlled prosthetic hand: Analysis of user-prosthesis interaction. IEEE Trans Robot. 2008;24(1):170-84. DOI:10.1109/TRO.2007.910708
6. Lai JC, Schoen MP, Perez Gracia A, Naidu DS, Leung SW. Prosthetic devices: Challenges and implications of robotic implants and biological interfaces. Proc Inst Mech Eng H. 2007;221(2):173-83. [PMID: 17385571]
DOI:10.1243/09544119JEIM210
7. Guo JY, Zheng YP, Huang QH, Chen X. Dynamic monitoring of forearm muscles using one-dimensional sonomyography system. J Rehabil Res Dev. 2008;45(1):187-95.
[PMID: 18566937]
DOI:10.1682/JRRD.2007.02.0026
8. Weir RF, Heckathorne CW, Childress DS. Cineplasty as a control input for externally powered prosthetic components. J Rehabil Res Dev. 2001;38(4):357-63. [PMID: 11563487]
9. Craelius W. The bionic man: Restoring mobility. Science. 2002;295(5557):1018-21. [PMID: 11834819]
DOI:10.1126/science.295.5557.1018
10. Heath GH. Control of proportional grasping using a myo-kinemetric signal. Technol Disabil. 2003;15(2):73-83.
11. Silva J, Heim W, Chau T. A self-contained, mechanomyography-driven externally powered prosthesis. Arch Phys Med Rehabil. 2005;86(10):2066-70. [PMID: 16213256]
DOI:10.1016/j.apmr.2005.03.034
12. Mainardi E, Davalli A. Controlling a prosthetic arm with a throat microphone. Conf Proc IEEE Eng Med Biol Soc. 2007;2007:3035-39. [PMID: 18002634]
13. Müller-Putz GR, Pfurtscheller G. Control of an electrical prosthesis with an SSVEP-based BCI. IEEE Trans Biomed Eng. 2008;55(1):361-64. [PMID: 18232384]
DOI:10.1109/TBME.2007.897815
14. Lieber RL, Fridén J. Functional and clinical significance of skeletal muscle architecture. Muscle Nerve. 2000;23(11): 1647-66. [PMID: 11054744]
DOI:10.1002/1097-4598(200011)23:11<1647::AID-MUS1>3.0.CO;2-M
15. Hodges PW, Pengel LH, Herbert RD, Gandevia SC. Measurement of muscle contraction with ultrasound imaging. Muscle Nerve. 2003;27(6):682-92. [PMID: 12766979]
DOI:10.1002/mus.10375
16. Shi J, Zheng YP, Huang QH, Chen X. Continuous monitoring of sonomyography, electromyography and torque generated by normal upper arm muscles during isometric contraction: Sonomyography assessment for arm muscles. IEEE Trans Biomed Eng. 2008;55(3):1191-98.
[PMID: 18334413]
DOI:10.1109/TBME.2007.909538
17. Mademli L, Arampatzis A. Behaviour of the human gastrocnemius muscle architecture during submaximal isometric fatigue. Eur J Appl Physiol. 2005;94(5-6):611-17. [PMID: 15906075]
DOI:10.1007/s00421-005-1366-8
18. Narici MV, Binzoni T, Hiltbrand E, Fasel J, Terrier F, Cerretelli P. In vivo human gastrocnemius architecture with changing joint angle at rest and during graded isometric contraction. J Physiol. 1996;496(Pt 1):287-97.
[PMID: 8910216]
19. Manal K, Roberts DP, Buchanan TS. Can pennation angles be predicted from EMGs for the primary ankle plantar and dorsiflexors during isometric contractions? J Biomech. 2008;41(11):2492-97.[PMID: 18579147]
DOI:10.1016/j.jbiomech.2008.05.005
20. Zheng YP, Chan MM, Shi J, Chen X, Huang QH. Sonomyography: Monitoring morphological changes of forearm muscles in actions with the feasibility for the control of powered prosthesis. Med Eng Phys. 2006;28(5):405-15.
[PMID: 16115790]
DOI:10.1016/j.medengphy.2005.07.012
21. Shi J, Zheng Y, Yan Z. Prediction of wrist angle from sonomyography signals with artificial neural networks technique. Conf Proc IEEE Eng Med Biol Soc. 2006;1:3549-52.
[PMID: 17946186]
22. Xie HB, Zheng YP, Guo JY, Chen X, Shi J. Estimation of wrist angle from sonomyography using support vector machine and artificial neural network models. Med Eng Phys. 2009;31(3):384-91.[PMID: 18586548]
DOI:10.1016/j.medengphy.2008.05.005
23. Shi J, Zheng Y, Zhou K. A pilot study of the SMG controlled prosthesis. In: Proceedings of the IEEE/ICME International Conference on Complex Medical Engineering 2007. 2007 May 23-27; Beijing, China. Los Alamitos (CA): IEEE Press; 2007. p. 1190-93.
24. Yeo H, Hu YH. A modular high-throughput architecture for logarithmic search block-matching motion estimation. IEEE Trans Circ Syst Video Tech. 1998;8(3):299-315.
DOI:10.1109/76.678625
25. Amiaz T, Lubetzky E, Kiryati N. Coarse to over-fine optical flow estimation. Pattern Recogn. 2007;40(9):2496-2503.
DOI:10.1016/j.patcog.2006.09.011
26. Chan YL, Siu WC. An efficient search strategy for block motion estimation using image features. IEEE Trans Image Process. 2001;10(8):1223-38. [PMID: 18255539]
DOI:10.1109/83.935038
27. Turaga D, Alkanhal M. Search algorithms for block-matching in motion estimation. [midterm project]. [Pittsburgh (PA)]: Carnegie Mellon University: 1998. 881 pages.
28. Jain JR, Jain AK. Displacement measurement and its application in interframe image coding. IEEE Trans Commun. 1981;29(12):1799-1808.
DOI:10.1109/TCOM.1981.1094950
29. Hassaballah M, Omran S, Mahdy YB. A review of SIMD multimedia extensions and their usage in scientific and engineering applications. Comput J. 2008;51(6):630-49.
DOI:10.1093/comjnl/bxm099
30. Intel Corp. Intel 64 and IA-32 architecture software developer's manual, volume 1: Basic architecture. Santa Clara (CA): Intel; 2003.
31. Shahbahrami A, Juurlink B, Vassiliadis S. Implementing the 2-D wavelet transform on SIMD-enhanced general-purpose processors. IEEE Trans Multimedia. 2008;10(1):43-51.
DOI:10.1109/TMM.2007.911195
32. Cunha MT, Telles JC, Ribeiro FL. Streaming SIMD extensions applied to boundary element codes. Adv Eng Soft. 2008; 39(11):888-98. DOI:10.1016/j.advengsoft.2008.01.003
33. Luh JJ, Chang GC, Cheng CK, Lai JS, Kuo TS. Isokinetic elbow joint torques estimation from surface EMG and joint kinematic data: Using an artificial neural network model. J Electromyogr Kinesiol. 1999;9(3):173-83.
[PMID: 10328412]
DOI:10.1016/S1050-6411(98)00030-3
34. Jaskólska A, Brzenczek W, Kisiel-Sajewicz K, Kawczynski A, Marusiak J, Jaskólski A. The effect of skinfold on frequency of human muscle mechanomyogram. J Electromyogr Kinesiol. 2004;14(2):217-25. [PMID: 14962774]
DOI:10.1016/j.jelekin.2003.08.001
35. Meek SG, Jacobsen SC, Goulding PP. Extended physiologic taction: Design and evaluation of a proportional force feedback system. J Rehabil Res Dev. 1989;26(3):53-62.
[PMID: 2666644]
36. Davalli A, Sacchetti R, Fanin S, Avanzolini G , Urbano E. Biofeedback for upper limb myoelectric prostheses. Technol Disabil. 2000;13(3):161-72.
Submitted for publication March 24, 2009. Accepted in revised form December 4, 2009.

Go to TOP

Go to the Table of Contents of Vol. 47 No. 2

Last Reviewed or Updated  Thursday, April 15, 2010 1:19 PM

Valid HTML 4.01 Transitional