Logo for the Journal of Rehab R and D

Volume 45 Number 9, 2008
   Pages 1401 — 1414

Building on residual speech: A portable processing prosthesis for aphasia

Marcia C. Linebarger, PhD;1* John F. Romania, MS;1 Ruth B. Fink, MA, CCC-SLP;2 Megan R. Bartlett, BA;2 Myrna F. Schwartz, PhD2-3

1Psycholinguistic Technologies, Inc, Jenkintown, PA; 2Moss Rehabilitation Research Institute, Albert Einstein Healthcare Network, Philadelphia, PA; 3Thomas Jefferson University, Philadelphia, PA

Abstract — This article examines the challenges of developing electronic communication aids for individuals with mild-to-moderate aphasia and introduces a new portable aid designed for this population. People with some residual speech are often reluctant to use communication aids that replace their natural speech with synthesized speech or the recorded utterances of another individual. SentenceShaper (computer software; Psycholinguistic Technologies, Inc; Jenkintown, Pennsylvania; www.sentenceshaper.com), a computerized "processing prosthesis," allows the user to record spoken sentence fragments and hold them in memory long enough to combine them into larger structures. Previous studies have shown that spoken narratives created with SentenceShaper-composed of concatenated, recorded segments in the user's own voice-may show marked superiority to the individual's spontaneous speech and that sustained use may engender treatment effects. However, these findings do not guarantee the program's efficacy to support functional communication or its acceptance by people with aphasia. Here, we examine strengths and weaknesses of SentenceShaper as the basis for a communication aid for individuals with mild-to-moderate aphasia and review factors guiding the design of SentenceShaper To Go, a portable extension to the program. Data from a "proof-of-concept" pilot study with the portable system suggest the viability of providing computer-based support for users' residual speech in composing and delivering spoken messages.

Key words: AAC, aphasia, assistive technology, augmentative and alternative communication, communication aid, communication disorders, language disorders, nonfluent aphasia, processing prosthesis, rehabilitation.

Abbreviations: AAC = augmentative and alternative communication, ANELT = Amsterdam-Nijmegen Everyday Language Test, CIU = correct information unit, C-VIC = Computer-based VIsual Communication, HH = handheld-aided mode, PDA = personal digital assistant, SGD = speech-generating device, SLP = speech-language pathologist, SSR = SentenceShaper-aided mode, U = unaided mode.
*Address all correspondence to Marcia C. Linebarger, PhD; 93 Old York Road, Suite 1#447, Jenkintown, PA 19046; 215-525-9905; fax: 215-525-9905.

The use of augmentative and alternative communication (AAC) technology has psychosocial costs and benefits. Costs include the effort required to learn and operate a device, the visible evidence of disability if the device is perceptible to others, and the awkwardness of relying on alternatives to natural speech. Benefits include the opportunity to express wants and needs, engage in a wider range of social interactions, and reveal one's preserved cognitive and linguistic abilities to others who may not understand that impaired speech does not transparently reflect mental function.

For individuals with purely motor impairments, the benefits of using this technology often outweigh the costs, as evidenced, for example, by the widespread acceptance of AAC among individuals with amyotrophic lateral sclerosis [1]. Speech-generating devices (SGDs) employing synthesized speech to transmit text or symbolic messages created by the user may effectively compensate for the loss of oral speech. The costs (e.g., the mental and physical effort of composing messages, the use of unnatural-sounding computerized speech) are more than offset by the benefits (e.g., regaining the ability to participate in spoken interactions, demonstrating preserved linguistic and cognitive function).

Aphasia presents somewhat different patterns of impaired and preserved abilities, with correspondingly different trade-offs between costs and benefits. Individuals with aphasia typically experience difficulty with some aspects of language production, such as word-finding and grammatical encoding, and may also show impaired reading, writing, and spelling. Considerable variability exists in the nature and severity of linguistic impairments and in the presence of concomitant cognitive or motor impairments. In developing AAC technology for aphasia, the challenge is to compensate for linguistic impairments while at the same time supporting and even highlighting the user's preserved cognitive function and any residual speech.

Electronic aids for people with aphasia have generally taken the form of SGDs in which words, phrases, or full messages are associated with buttons or other screen controls displaying visual images; when clicked, buttons play their associated items through synthesized speech or the prerecorded speech of another individual. These aids vary in the transparency of their images, which can range from highly stylized symbols to photographs of specific objects or events, and in the number and organization of stored messages. While some aids are designed primarily to replay prefabricated stored messages, others allow the user to create novel utterances by locating and combining multiple icons in a specific area of the screen. This iconic model underlies the first electronic aid developed specifically for aphasia, Computer-based VIsual Communication (C-VIC) [2], commercially released as Lingraphica® (dedicated augmentative communication device; Lingraphicare America, Inc; Princeton, New Jersey; www.aphasia.com), and a variety of other software programs or electronic devices subsequently developed for individuals with aphasia.

A number of studies have reported what we will term aided effects for icon-based communication by individuals with severe aphasia [2-5]; that is, participants were able to express propositional meaning more effectively with visual icons than with their natural speech, at least in laboratory tasks. However, these and other studies suggest that some people with aphasia have quite limited ability to combine icons into complex structures, especially when the message to be expressed is abstract [6-8].

Thus this paradigm has both strengths and weaknesses. In addition to compensating for impairments to oral speech, SGDs incorporating visual icons can provide semantic support to those with impaired word comprehension and bypass the reading, writing, and spelling impairments that are widespread in aphasia. However, abstract or nonimageable words present difficulty, and the ability of individuals with aphasia, even those with preserved comprehension, to combine visual icons into complex messages may be limited. Finally, and perhaps most importantly, most people with mild-to-moderate aphasia prefer to use their own voices rather than synthesized or third-party recorded speech.

This preference may underlie the relatively low acceptance of AAC in people with mild-to-moderate aphasia. While many individuals with aphasia accept nonelectronic AAC aids and strategies, such as gesturing, writing, and so forth, electronic aids such as iconic SGDs have not played the same role for this population as they have for those with purely motor impairments. The literature on C-VIC and Lingraphica, for example, has focused on treatment effects rather than on functional applications of the device. More generally, clinicians typically invoke AAC only as a last resort for the most severely impaired [9-11]. Hux et al. note, "As a group, many [speech-language pathologists (SLPs)] incorporate AAC strategies only when individuals fail to regain the ability to convey even the most basic messages through natural speech" [9]. Given the centrality of language to identity [12], we are not surprised that many individuals with aphasia reject alternatives to oral speech. This preference for natural speech may be shared by family members and SLPs, even when messages communicated with AAC aids are perceived as superior to those expressed in the user's natural speech [13]. In addition, people with aphasia typically wish to continue efforts to remediate their spoken language and perceive SGDs as circumventing such practice [14]. Furthermore, SGDs that play synthesized speech or the digitized speech of another person raise questions in the listeners' minds about whether the person with aphasia actually authored the message [15]. This ambiguity about authorship enters into the cost-benefit considerations just discussed. Employing an SGD to deliver a message that the user could have produced with his or her natural speech, albeit with long pauses and struggles, may actually underrepresent that person's linguistic competence by implying that the user is unable to produce the message her- or himself. This underrepresentation may well offset the benefits of communicative efficacy. Additionally, the expressiveness of the human voice enhances the communication of emotion and personality. Finally, the literature on language production in typical speakers suggests that prosody and word associations are important in sentence formulation; these cues are unlikely to be provided by devices that replace natural speech.

Therefore, although encouraging studies have reported successful SGD use by individuals with severe aphasia for whom natural speech is unlikely to be an option [15-17], a different approach is often suggested for those who have some residual speech. On this approach, AAC is used to supplement or scaffold natural speech rather than replace it [9,18], and a division of labor is maintained between natural and aided speech, with the former used for small talk and the latter used primarily for more complex messages that can be anticipated in advance. Garrett and Beukelman identify as specific-need communicators those individuals who rely on AAC primarily in situations that require "specificity, efficiency, or clarity" [19], e.g., complex or detailed messages to service providers, such as doctors, lawyers, or transportation workers; telephone messages; and storytelling and advice-giving to family members and friends. Similarly, Hux et al. describe an individual who "[used] natural speech to manage small talk interactions and [used] his communication book to communicate detailed information about himself" [9]. Given the important role of storytelling in adult conversation [20], for which opportunities are frequently diminished in aphasia [21], increasing opportunities to engage in this activity could significantly affect quality of life. This approach is also embodied in the script training [22] approach to aphasia therapy, which emphasizes the intense practice of personally relevant, often monologic, scripts appropriate for use in the community. Note that the kinds of functional situations (e.g., service encounters) that evolve in daily life typically require creating novel messages and cannot be negotiated with a set of high-frequency utterances designed for all users. While caregivers may be able to input more personalized messages, this can be labor-intensive and may decrease the user's perception that the device speaks for her or him. Because of caregivers' inability to anticipate the ever-changing set of desired messages [17], supporting the independent composition of novel utterances is a priority in the development of AAC aids for those with mild-to-moderate aphasia.

This article describes a new approach to AAC for this population, one that follows previous work quite closely in two respects: (1) deploying AAC primarily for complex monologic messages that can be prepared beforehand and (2) attempting to scaffold rather than replace natural speech. However, the approach diverges markedly regarding the mechanism used to provide this scaffolding.


SentenceShaper® (computer software; Psycholinguistic Technologies, Inc; Jenkintown, Pennsylvania; www.sentenceshaper.com) is a computer program that facilitates the creation of spoken utterances by providing processing support [23]. It is based on evidence that words are retrieved more slowly and may decay more quickly in persons with aphasia, causing them to have difficulty assembling all the elements of a sentence in memory simultaneously [24-25]. For example, a speaker may produce the subject noun for a sentence ("dog") and then, after struggling to retrieve an appropriate verb ("chase"), may have to reretrieve the subject noun to combine it with the verb, which may itself have decayed from memory during this reretrieval.

SentenceShaper is a "processing prosthesis" that helps the user keep utterances in working memory. The program, which has been detailed elsewhere [26], embeds a user-controlled sound recorder within a visual interface in which every recording the user creates is linked to an arbitrary colored icon. Clicking an icon replays the recorded utterance linked to it; the user creates sentences by dragging these icons (each representing a recorded sound bite) to the Sentence Assembly Area on the screen, where they are combined and ordered from left to right. When the user is satisfied with the sentence, she or he clicks a button to transfer it to the Narrative Assembly Area. There, it is represented by a single icon, an oval "bean," which plays the entire sequence when clicked. The user can build a larger narrative or message by creating additional sentences and adding their associated beans to the ordered slots in the Narrative Assembly Area. Two word-finding tools, which can be disabled or customized for individual users, display printed words along the sides of the main screen (the Side Buttons) or in a separate screen (the WordFinder). Clicking a text word causes it to be spoken by the system; the user must repeat it in his or her own voice to incorporate it into a production.

The program's replay functionality allows the speaker to refresh the memory of words or phrases she or he has previously produced, keeping them activated in working memory long enough to be combined into sentences. Replaying saved fragments, or sequences of fragments, may also support speech production by allowing the user to detect and repair lexical and grammatical errors and by triggering a cloze response [27], helping the user to complete a phrase or sentence on the basis of context and word associations. The description of SentenceShaper as a prosthesis derives from its role in artificially enlarging the user's mental workspace for language. However, to the extent that the program guides language production by fostering strategies such as the replay of partial utterances to trigger a cloze response, it might also be described as an orthosis.

A number of studies have reported aided effects for SentenceShaper [23,28-34]: the sometimes marked superiority of spoken narratives created on SentenceShaper to the same user's spontaneous, unaided narratives on the same topic. These aided effects have been demonstrated with multiple different measures (of structure, rate, and informativeness) and elicited content and with informativeness ratings by unfamiliar listeners as well as quantitative analyses. These studies have varied in the strength and statistical significance of the effects and in the nature of word-finding support, if any, provided by the system. To illustrate qualitatively the differences between aided and unaided speech, Table 1 provides excerpts from unaided and aided descriptions of the same events by the two individuals with nonfluent agrammatic aphasia who showed the strongest aided effects in an early study with the system [23]. Since both word-finding tools were disabled during the creation of the aided productions in this study, the prosthesis provided no word-finding or grammatical assistance, only processing support.

Table 1.
Aided and unaided spoken descriptions of same events by two participants in Linebarger et al.* who showed strongest aided effects. Since WordFinder and Side Buttons were disabled for this study, no word-finding or grammatical support was provided by SentenceShaper; aided effects illustrate impact of "pure" processing support.

Unaided (without SentenceShaper)
Aided (with SentenceShaper)

The, the maid, the maid, the maid, uh, uh, upstair and she, uh,
the maid upstairs and scuse me and um . . . go around but now
uh the . . . . The policeman, she she?, no, the man, two men, and the uh, oh, she, uh, her, she . . . knock them out, knock them out, um hum, knock them out, two men.
The man goes around them. She did not do it. The nurse goes around the baby carriage. The policeman, she fights the, the two men.
Ooh! A fish! Ah, water and . . . uh mmm and attendant, here, and bumped his head. Oh boy, oh my hand, my hand, my hand.
The boy and the fishmonger is taking the fish. The boy hit his hand.

*Linebarger MC, Schwartz MF, Romania JR, Kohn SE, Stephens DL. Grammatical encoding in aphasia: Evidence from a "processing prosthesis." Brain Lang. 2000;75(3):416-27. [PMID: 11112295]

Several small studies have also reported treatment effects following a period of SentenceShaper use [26,28-29,32,34-35]* that is, positive changes in participants' unaided (off-computer) spoken narratives on topics that have never been practiced on the computer. These effects, which have ranged from minimal to quite marked, qualitatively resemble the reported aided effects and are encouraging because treatment interventions rarely generalize to connected speech [36]. Furthermore, they add to the evidence that using communication aids can lead to positive changes in natural speech [5,37]. One motivation for developing SentenceShaper into an assistive device is its potential to engender treatment effects; integrating the program into daily life communication may stimulate ongoing practice with the prosthesis and bring about positive changes in spontaneous speech.

*Schwartz MF, Linebarger MC, Brooks R, Bartlett MR. Combining assistive technology with conversation groups in long-term rehabilitation for aphasia. Unpublished manuscript; Sep 2007. Available at: http://www.ncrrn.org/pdf/SchwartzAssistiveTechnology.pdf/. Cited by permission of Myrna F. Schwartz (http://www.ncrrn.org/people/schwartz).

These studies have primarily targeted individuals with nonfluent aphasia and have suggested that prerequisites for effective use of SentenceShaper include the ability to produce some speech, if only short phrases; relatively intact auditory comprehension of single words; some ability to self-monitor, i.e., to replay one's utterances and subsequently correct or expand them; and good performance on simple grammaticality judgment tasks [23,26]. In addition, severe impairments to executive function have been shown to impede AAC use [38] and would almost certainly limit the effectiveness of the program. Future research may identify other abilities necessary for effective SentenceShaper use and/or additional disorders that it might ameliorate.


Despite the aided effects observed in the laboratory, functional deployment of SentenceShaper faces two major challenges: (1) the "time barrier" and (2) cognitive demands imposed by functional use of such a device.

Regarding the former, the core limitation of SentenceShaper-and perhaps of any aid that attempts to support users' residual language production abilities for the creation of novel messages-is the lengthy process of message composition. A message with a 3-minute playing time may take 45 minutes to create. Therefore, as previously noted, we followed an existing thread in the AAC literature in focusing on monologic messages that can be anticipated in advance, such as information that the user wishes to communicate in service encounters, storytelling, advice-giving, or offline situations in which the listener is either physically distant or noninteractive, such as email [28-29,39], Web postings,* and public speaking [40, p. 150,262].

* Schwartz MF, Linebarger MC, Brooks R, Bartlett MR. Combining assistive technology with conversation groups in long-term rehabilitation for aphasia. Unpublished manuscript; Sep 2007. Available at: http://www.ncrrn.org/pdf/SchwartzAssistiveTechnology.pdf/. Cited by permission of Myrna F. Schwartz (http://www.ncrrn.org/people/schwartz).

The cognitive demands of functional use are also considerable. The user must be able to anticipate novel utterances that she or he will wish to communicate in some future situation or to generate topics for emails or Web postings that will be comprehensible and interesting to the recipient. Previous studies have suggested that some individuals who effectively use SentenceShaper to retell narratives presented to them in the laboratory or on television may nonetheless require support with idea generation when composing their own messages [28-29].* The delivery of previously composed messages can also pose challenges. For example, a person with aphasia preparing comments beforehand for a discussion group may show a strong aided effect using SentenceShaper to create a far more grammatical and complex message than she or he could produce spontaneously but may still need help identifying the appropriate moment to deliver this message to the group.

The issue of support has been addressed carefully in the AAC literature, which posits a core distinction between partner-dependent and independent communicators [41]. Lasker et al. describe partner-dependent communicators as individuals who have difficulty applying AAC strategies without ongoing reminders, or in unpracticed contexts. In contrast, independent communicators are active in thinking of what they want to say and in circumventing or augmenting oral speech when necessary. A subset of this latter group is called generative communicators. They are able to construct messages from multiple or heterogeneous subcomponents, such as combining items stored on the device into novel productions or using oral speech, gesture, and drawings to convey an idea.

If effective SentenceShaper users can be identified within this taxonomy, then we may be able to draw on existing AAC materials and methodologies for assessing and training users of the system. However, because SentenceShaper differs so fundamentally from the iconic SGDs that have typically been employed for individuals with severe aphasia, its users may not fall easily into standard AAC taxonomies. For example, virtually anyone who shows a SentenceShaper-aided effect represents a generative user in this framework, since building a sentence from several recorded words or phrases demonstrates the ability to create messages from multiple subcomponents. Nonetheless, even those who are able to create complex, multicomponent messages on SentenceShaper may require support (and thus lack independence) in idea generation, in getting into "set" to use SentenceShaper and identifying points in a conversation for which a stored message is appropriate. Thus SentenceShaper users may demonstrate both generativity and partner-dependence; the program may allow some individuals to compose messages of a complexity and/or abstractness that outstrips the ability of their author to integrate them into real-life situations without considerable support. A question for future research is whether AAC-readiness assessment procedures such as the Multimodal Communication Screening Test for Persons with Aphasia may identify independent SentenceShaper users by assessing communicative flexibility and other metacognitive skills [42], even though these assessments typically focus on tasks (such as combining visual icons) that are not actually performed on SentenceShaper.

Portability: Design Issues

Functional use of any communication aid obviously requires portability. Because of the time barrier described previously, the most realistic approach to portability may be one in which the person with aphasia anticipates messages she or he would like to convey in an impending social situation; composes the messages on SentenceShaper, typically installed on a desktop computer; and then, using an interface within SentenceShaper, downloads these messages onto a handheld computer that can be carried into the situation. Thus the handheld device is not itself used for message composition in the community but rather to store messages that the user has created in advance of the situation. This architecture is employed in, e.g., Boyd-Graber et al. [43].

Regarding the handheld device, people with poststroke aphasia are often ambulatory but without full control of the right arm and hand (right hemiparesis). Portability in such cases raises a distinct set of challenges [44], because no wheelchair exists to which the device might be affixed. To use the device while standing, therefore, persons need to both carry and operate the device with the nondominant hand. And although affixing the handheld device to the user's body is possible, freeing his or her hand to operate the device, this arrangement is unlikely to be acceptable for individuals who are typically self-conscious about "visible signs of disability" [11,15,41,45]. Therefore, the display on the handheld device must be as simple as possible so that it does not require a stylus or unencumbered hand to operate.

A final design issue is how the spoken messages that have been stored on the device will be conveyed to a listener. SGDs are typically used for direct playback (i.e., a previously recorded message is played directly to the user). However, the artificiality of communicating this way is likely to render it a last resort for individuals with some residual speech, as evidenced by the lukewarm embrace of such technology in this population. Another approach is to use the device for self-cueing, that is, for the individual to replay a stored message privately (through earphones or lowered volume) and then repeat it in real time, live speech, using the handheld device as a prompter. (This approach requires a hardware or software implementation in which the user can switch freely between private and public modes of replay.) Because individuals with mild-to-moderate aphasia have not been the focus of AAC, the technique of private self-cueing is not widely discussed, but see, e.g., Mollica [46] and van de Sandt-Koenderman [11].

Architecture of SentenceShaper To Go

A portable version of SentenceShaper was developed called "SentenceShaper To Go™" in response to the design issues just reviewed. SentenceShaper To Go is a software suite consisting of three interacting programs:

1. The original SentenceShaper program, typically running on a desktop computer, which the user employs to create spoken messages intended for use in a future situation.
2. The Handheld Display, a program running on a handheld computer or personal digital assistant (PDA), which allows the user to replay utterances created on SentenceShaper by clicking buttons on the screen of the PDA.
3. The Handheld Customization Tool, a screen accessed from within SentenceShaper, where the user configures the handheld device (associating individual SentenceShaper utterances with particular buttons on the device) and clicks a button to transfer this configuration onto the device. (The latter two components are described in the following subsections.)
Handheld Display

The Handheld Display program was initially implemented on a Zaurus SL-5500 PDA (Sharp Electronics Corp; Mahwah, New Jersey), selected for its Java runtime environment and reflective color liquid crystal display touch screen, allowing input by either stylus or finger. The display on the handheld device contains up to eight buttons, which play (when clicked) spoken utterances that the user has created on SentenceShaper.

Ergonomic Issues

We conducted preliminary studies to determine whether participants with right hemiparesis would be able to use the nondominant hand to access all eight buttons on the Handheld Display in finger press mode (pressing any one of the eight buttons on the display while holding the device with the same hand) as well as in stylus mode (placing the device on a flat surface to allow use of the stylus). While participants' reaction times were slightly slower to press the button in the finger as opposed to stylus mode (finger press mode mean = 0.70 s/button; stylus mode mean = 0.57 s/button), this slower response was deemed an acceptable trade-off for the portability of the finger press mode.

A second concern related to potential injury or discomfort associated with holding and operating the handheld device in this way. We preliminarily addressed this issue by measuring grip and finger strengths before and after sessions of handheld use. Five participants were evaluated over eight successive sessions. No clinically meaningful changes in strength were found and no participants reported discomfort while or after using the device. However, long-term finger press mode use of the handheld-or any similar PDA-would require careful monitoring for signs of repetitive stress. Encouraging people with hemiparesis to configure the device with fewer than eight buttons may also decrease this risk.

Audio Output

Participants were asked to hold the device toward themselves when using it to self-cue and toward the listener when using it in direct playback mode. The program was subsequently recoded to allow the user to toggle between low and high volume settings for self-cueing and direct playback, respectively.

Handheld Customization Tool

After creating a spoken message on SentenceShaper, the person with aphasia downloads this message to the portable device by means of the Handheld Customization Tool (Figure 1), accessed by clicking a button on the main SentenceShaper screen.

Figure 1. Handheld Customization Tool used by person with aphasia to download a message to portable device.

The eight slots on the left side of Figure 1 (the Handheld Mockup) correspond to the eight available slots on the PDA. Above the Mockup, the SentenceShaper Narrative Assembly Area remains visible; recall that the oval beans in this area play sequences of speech segments that the user has recorded and combined into sentences or other groupings. The user assigns a bean to one of the buttons on the handheld device by dragging that bean to a slot in the Mockup. After the user has dragged the bean to its slot, a checkmark appears on that bean in the Narrative Area at the top to remind the user that the bean has already been loaded. Labels can be added to beans in the Mockup by dragging them from the Label Set area on the right onto a bean. Once in the Mockup, beans can be reordered or dragged down to the Mockup's Trash Hole. The Label Play Buttons (to the right of the Label Set in Figure 1) are intended to help users learn the meanings of the labels; when clicked, they play a sound file pronouncing the word on the label (if it displays text) or explaining the label's meaning (if it is a nonverbal symbol). The Return to Main Screen Button lets the user toggle back and forth between the Handheld Customization Tool and the SentenceShaper Assembly Area to add more material or to rerecord all or part of the message if an error is detected or if a button contains more material than the user can say at one time in the self-cueing mode. Clicking the Synch Button loads the handheld device with the set of sound files and labels arrayed in the Mockup; the PDA shown on the lower left of Figure 1 displays four labeled buttons as in the Handheld Mockup.


A preliminary implementation of SentenceShaper To Go was tested in a single-participant "proof-of-concept" experiment. The pilot study finalized the hardware and software for a larger study currently underway at Moss Rehabilitation Research Institute (Philadelphia, Pennsylvania). A particular focus of the pilot study was to observe the participant's delivery of messages stored on the handheld device, for assessing the viability of self-cueing as opposed to direct playback and ascertaining the participant's preferences in this regard. In this early pilot and subsequently in the larger study, the impact of the new program was tested by eliciting spoken narratives on functional topics in three modes: (1) U (unaided, spontaneous speech), (2) SSR (spoken narratives created on SentenceShaper), (3) and HH (handheld-aided, narratives delivered with spontaneous speech aided and/or supplemented by the handheld device, which had been loaded with the SentenceShaper production of that narrative). Note that the pilot data presented here are purely descriptive, because this preliminary study was conceived as an integral part of the technology development. In contrast, the larger study uses a series of single-subject multiple baseline experiments, with somewhat different procedures from those described here.


The participant, identified as L1, was a 41-year old, right-handed man who had sustained a left-hemisphere stroke 22 months before the experiment. He was diagnosed with mild agrammatic aphasia on the basis of clinical testing; his Aphasia Quotient on the Western Aphasia Battery was 85.6 [47]. He had received 16 years of formal education. After all procedures were fully explained to the participant, he gave his written informed consent to participate.


Two sets of materials were used. Initial practice with the system used scenarios from the Amsterdam-Nijmegen Everyday Language Test (ANELT) [48], which are designed to elicit responses to hypothetical daily life situations, such as purchasing, selling, or returning merchandise. Four test narratives (Wedding, Parking, Cooking, and Ironing) were also developed for this study; these narratives are more complex than the ANELT scenarios and are designed to challenge participants with mild or moderate aphasia. For example, in one scenario, a dinner party host recounts a series of cooking mishaps to the arriving guests to explain why the meal will be late.


L1 had learned to use SentenceShaper during an earlier study. The protocol incorporated an extended training period (16 biweekly sessions) followed by 8 weeks of largely independent home use, during which he created narratives such as retellings of television and movie plots. During the latter part of this training, L1 began to create practice ANELT narratives, incorporating five new Side Buttons on the SentenceShaper screen that played phrases hypothesized to provide useful support in initiating the functional messages and signaling the completion of the monologue ("Hello," "The problem is . . . ," "I will . . . ," "I want you to . . . ," "Goodbye"). These phrases were also added to the Label Set in the Handheld Mockup (Figure 1).

During four sessions, L1 was introduced to the eight-button display on the handheld and shown how to use the Handheld Customization Tool to download the ANELT narratives-previously recorded on SentenceShaper-to the device. He then learned to use the handheld device to communicate these simple functional narratives to both familiar and unfamiliar listeners who asked probe questions targeting missing or incorrect information. He was instructed to use the handheld device when needed, using either the self-cueing or direct playback methods as he found appropriate.

During this phase, L1 also produced retellings of all four test narratives in the U mode. Each test narrative was performed six times (over six sessions) in the U mode. To account for practice effects and thereby to avoid inflating the SSR or HH aided effects, we included only the last three unaided retellings (here denoted as baselines 4, 5, 6) in the U sample to be compared with the subsequent SSR and HH retellings. In all these retellings, the elicitation procedure was to have L1 listen to a recording of the narrative, accompanied by pictures, and then retell the narrative in his own words and without picture support.

After the baseline phase, the SSR and HH retellings of the test narratives were elicited in successive blocks of two sessions each. In the first session, L1 retold a test narrative in SSR mode; that is, he prepared the retelling independently on the desktop computer running SentenceShaper and then downloaded it to the handheld device. In the following session, he used the handheld device to deliver this narrative to an unfamiliar listener (HH mode). Earlier in the session, he rehearsed by retelling the narrative to the examiner. This practice was meant to simulate optimal usage of the device in real life (i.e., for a user to become familiar with the button contents and practice using the handheld to communicate to another person).

Analyses and Results

Two scorers, blind to the experimental conditions, scored the transcripts using a modified version of Nicholas and Brookshire's correct information unit (CIU) methodology [49], which counts as a CIU any word that is "intelligible in context, accurate in relation to the picture(s) or topic, and relevant to and informative about the content of the picture(s) or topic." Table 2 summarizes the results for two derived variables: percentage of words that are CIUs (% CIUs) and CIUs/minute (speech rate). It also displays, as a measure of the aided effects, the percentage gain from baseline for the SSR and HH conditions. Figure 2 displays the percent CIUs across all five retellings of each narrative.

Table 2.
Summary of data from pilot study participant L1 for four narrative stimuli in three modes: U (unaided) (mean of U baselines 4, 5, 6), SSR (retelling produced on SentenceShaper), and HH (handheld) (live speech retelling cued with handheld device).

No. of
% CIUs*
Rate (CIUs/min)
% Gain
% Gain


*Percentage of words that are correct information units (CIUs) (Nicholas LE, Brookshire RH. A system for quantifying the informativeness and efficiency of the connected speech of adults with aphasia. J Speech Hear Res. 1993;36(2): 338-50. [PMID: 8487525]).
% Gain = ((aided value - baseline mean)/baseline mean) × 100.
Mean of three baselines.

Figure 2. Correct information unit (CIU) analysis of participant L1ís retellings of four test narratives

The aided effect for SSR is strong and consistent across measures and narratives. In the Cooking test narrative, for example, percent CIUs increased from 64.0 (baseline mean) to 90.0 percent (SSR), for a percentage gain of 40.6 percent (Table 2).

The aided effect for SentenceShaper To Go is weaker. Still, for three of the four test narratives, the percentage gain in percent CIUs exceeds 15 percent, and in the last two narratives, the informativeness of the HH retelling (86% and 84% of words were CIUs) approaches the high levels of the SSR production (92%, 95%). The SSR and HH productions also show greater fluency than the U retellings, as indicated by the differences in CIUs/minute across the three modes (Table 2).

These observed SSR and HH aided effects cannot be interpreted as reflecting simply increased familiarity with the test scenarios, since L1 had been thoroughly familiarized with the scenarios during the baseline phase. Figure 2 clearly shows that no upward trend was present in baselines 4, 5, and 6. Also relevant is the Linebarger et al. finding that eliciting repeated unaided productions of the same narrative from participants with aphasia had no impact on structural measures or percent CIUs [26].

Regarding direct replay versus self-cueing, L1 strongly preferred live speech, relying exclusively on self-cueing rather than simply playing back the recorded material to the listener. This limited use of direct playback is reflected in the CIU analysis: none of the CIUs in L1's handheld-aided retellings of the final two narratives was contained only in material replayed on the handheld device. In the first two narratives, only 5 and 6 percent, respectively, of the CIUs in the handheld-aided productions were replayed on the handheld device but not expressed in L1's spontaneous speech.


With regard to percent CIUs, the robust SSR aided effect observed here replicates earlier studies, and the impact of SentenceShaper To Go (i.e., the HH aided effect), while less striking, suggests that use of the handheld device to cue spontaneous speech (rather than replay the recorded utterances directly to the listener) is a viable approach for delivering messages created on SentenceShaper. The increased fluency of the SSR and HH productions is also encouraging. Consider, for example, L1's production of the Cooking test narrative. His final unaided baseline of the story took almost 11 minutes (10:47), considerably longer than either his SSR retelling (1:39) or his HH retelling (3:04). As shown in Figure 2, this final unaided baseline (baseline 6) was also the least informative.

Note that narratives produced in HH and U modes differed in one important respect: The final three narratives produced in the U mode were delivered to a familiar listener, while the narratives produced in the HH mode were delivered to an unfamiliar listener. (The SSR retellings were produced independently on the computer, not delivered to a listener.) However, this difference would be expected to underrepresent any HH aided effect, since an unfamiliar listener represents a more difficult condition.

The participant's preference for live speech in this pilot study demonstrates that using a stored message to self-cue is a viable mode of delivery for messages composed on SentenceShaper. Individuals with more severe impairments may need to rely more heavily on direct playback; less impaired users may become more comfortable with direct playback over time, especially because the recorded messages on the handheld component of SentenceShaper To Go are in the user's own voice, providing evidence that the user did author the message [13].

Observation of L1's handheld retellings also clearly showed that he had many different ways to use the recorded utterances stored on the handheld. They include delivering the message entirely by direct playback of the recorded speech; communicating some message elements with direct playback and others with live speech; listening to the stored utterances privately to self-cue and then recapitulating these same elements with live speech, possibly embellishing or correcting them; and communicating a message by direct playback of the recording accompanied by simultaneous live speech that echoes words and phrases in the recording [28-29]. L1 demonstrated that, for some, self-cueing mode may only require private playback of the first few words of a well-practiced production.

During an informal interview at the end of the study (with ML; see "Acknowledgments" section regarding her competing interest), L1 expressed a strong desire to use the handheld device in his own life and described several situations in the upcoming week for which it might be appropriate. His communicative needs in these situations exemplify the kind of message complexity that can disrupt the speech of even moderately impaired individuals: rescheduling of a medical appointment, complicated trip arrangements with a travel agent, athletic coaching advice, a short presentation at a charity event, a speech to his family on Father's Day, and remarks to a stroke group at a local rehabilitation hospital. In the larger study currently underway at Moss Rehabilitation Research Institute, a qualitative methods consultant is conducting one-on-one interviews with participants after the handheld experiment to explore their feelings and attitudes about using AAC aids with these features in everyday life.


Assistive technology for aphasia has typically focused on the most severely impaired and has often been rejected as an undesirable last resort by those who retain the ability to produce some speech. As observed in the AAC literature, the verbal output of this latter group may allow them to negotiate independently many day-to-day activities; they may need communication assistance primarily when the situation requires the clear and timely expression of more complex information. A communication aid for such individuals should support the creation of novel utterances appropriate to new situations as they emerge, rather than rely on a set of prefabricated utterances created by others, and should, if possible, support rather than replace the user's own speech.

Previous studies have demonstrated that SentenceShaper, a processing prosthesis that supports the creation of spoken sentences and narratives, may allow individuals with mild-to-moderate aphasia to create more structured and informative utterances, in their own voices, than they can create unaided and that use of the prosthesis may lead to treatment effects evident in connected speech. When used as a communication aid, SentenceShaper may facilitate not only the transmission of information but also the demonstration of preserved linguistic and cognitive abilities and the expression of the user's identity through more complex utterances.

Our assessment of the strengths and weaknesses of SentenceShaper and more generally of the clinical and technological challenges in developing electronic communication aids for those with mild-to-moderate aphasia has guided the design and pilot deployment of SentenceShaper To Go, an AAC aid that allows people with aphasia to create messages on SentenceShaper and download them onto a portable handheld computer. The results of this pilot study are encouraging: the participant was able to create more informative messages on SentenceShaper than he could produce spontaneously, and his retellings of these messages when self-cued by the handheld component of SentenceShaper To Go were in most instances more informative than his unaided speech. Increases in fluency, quite marked in some cases, were also observed in both aided modes. His effective use of self-cueing and his preference for self-cueing over direct playback suggest that self-cueing may be an effective strategy for message delivery by those with mild-to-moderate aphasia. (If the PDA were used for direct playback, of course, it would allow the user to deliver messages with the informativeness and fluency of the SSR condition.)

The participant's willingness to practice his functional narratives repeatedly before delivering them with the handheld also suggests that the SentenceShaper To Go suite could implement a script training [22] protocol in which the user generates his or her own scripts on SentenceShaper, rehearses them by replaying utterances on the PDA or within the Handheld Mockup, and uses the PDA to support delivery of the script in situ.

Although this article has focused on the development of SentenceShaper for assistive use, most of the strategies and functionalities discussed could be implemented through other means. For example, processing support can be provided without SentenceShaper, through text processing (for individuals with high-level reading skills) or by a clinician, albeit laboriously [50]. Using stored messages to self-cue spontaneous speech is appropriate to virtually any SGD [11,46], and techniques such as playing partial utterances to trigger a cloze response could also be implemented on an iconic, portable SDG , although this strategy may require processing support to be effective. Transporting SentenceShaper productions into functional situations can be done without SentenceShaper To Go, e.g., by means of a portable sound recorder or any SGD to which sound files created on SentenceShaper can be downloaded or simply rerecorded. Also, as previously noted, for situations that cannot be anticipated in advance and/or require access only to single words or high-frequency utterances, other communication aids and strategies may be more appropriate than the processing prosthesis described here.

If these pilot results are replicated in the larger study-and early reports are encouraging in this regard [51]-the next step will be to investigate the use of SentenceShaper To Go in functional encounters outside the laboratory. Real-world deployment poses numerous challenges, such as the need for assessment and training protocols and for ongoing, contextualized coaching as the user attempts to integrate the assistive device into daily life. Nevertheless, the impact of this technology on the spoken narratives of some individuals with aphasia demonstrates that their own residual speech, with appropriate support, may provide a strong foundation for enhanced communication.


We thank Michelene Kalinyak-Fliszar and Jennifer Lowery for their assistance with data analysis.

This material was based on work supported by grant R01 HD043991 (Dr. M. F. Schwartz, principal investigator) from the National Institutes of Health.

A research prototype of SentenceShaper (termed "CS") was used in some of the earlier studies. SentenceShaper uses methods and interfaces covered by U.S. Patent No. 6,068,485 owned by Unisys Corporation and licensed to Psycholinguistic Technologies, Inc, which has released it as a commercial product. A competing interest arises because Dr. Linebarger owns shares in this company; therefore, neither she nor any other representative of Psycholinguistic Technologies has engaged in testing or scoring of raw data in the studies reported and reviewed in this article.

1. Ball LJ, Beukelman DR, Pattee GL. Acceptance of augmentative and alternative communication technology by persons with amyotrophic lateral sclerosis. Augment Altern Commun. 2004;20(2):113-22.
2. Steele RD, Weinrich M, Wertz RT, Kleczewska MK, Carlson GS. Computer-based visual communication in aphasia. Neuropsychologia. 1989;27(4):409-26. [PMID: 2471943]
3. Gardner H, Zurif EB, Berry T, Baker E. Visual communication in aphasia. Neuropsychologia. 1976;14(3):275-92. [PMID: 958599]
4. Koul R, Corwin M, Hayes S. Production of graphic symbol sentences by individuals with aphasia: Efficacy of a computer-based augmentative and alternative communication intervention. Brain Lang. 2005;92(1):58-77. [PMID: 15582036]
5. Weinrich M, McCall D, Weber C, Thomas K. Training on an iconic communication system for severe aphasia can improve natural language production. Aphasiology. 1995; 9(4):343-64.
6. Beck AR, Fritz H. Can people who have aphasia learn iconic codes? Augment Altern Commun. 1998;14(3):184-95.
7. Bertoni B, Stoffel A, Weniger D. Communicating with pictographs: A graphic approach to the improvement of communicative interactions. Aphasiology. 1991;5(4):341-53.
8. Funnell E, Allport A. Symbolically speaking: Communicating with blissymbols in aphasia. Aphasiology. 1989;3(3): 279-300.
9. Hux K, Manasse N, Weiss A, Beukelman D. Augmentative and alternative communication for persons with aphasia. In: Chapey R, editor. Language intervention strategies in aphasia and related neurogenic communication disorders. 4th ed. Philadelphia (PA): Lippincott Williams & Wilkins; 2001. p. 675-89.
10. Kitzing P, Ahlsen E, Jonsson B. Communication aids for people with aphasia. Logoped Phoniatr Vocol. 2005;30(1): 41-46. [PMID: 16040439]
11. Van de Sandt-Koenderman M. High-tech AAC and aphasia: Widening horizons? Aphasiology. 2004;18(3):245-64.
12. Shadden B. Aphasia as identity theft: Theory and practice. Aphasiology. 2005;19(3-5):211-23.
13. Lasker J, Beukelman DR. Peers' perceptions of storytelling by an adult with aphasia. Aphasiology. 1999;13(9/11):857-70.
14. Beukelman DR, Ball LJ. Improving AAC use for persons with acquired neurogenic disorders: Understanding human and engineering factors. Assist Technol. 2002;14(1):33-44. [PMID: 12739848]
15. Lasker JP, Bedrosian JL. Promoting acceptance of augmentative and alternative communication by adults with acquired communication disorders. Augment Altern Commun. 2001; 17(3):141-53.
16. Van de Sandt-Koenderman M, Wiegers J, Hardy P. A computerised communication aid for people with aphasia. Disabil Rehabil. 2005;27(9):529-33. [PMID: 16040556]
17. Waller A, Dennis F, Brodie J, Cairns AY. Evaluating the use of TalksBac, a predictive communication device for nonfluent adults with aphasia. Int J Lang Commun Disord. 1998;33(1):45-70. [PMID: 9673218]
18. Hustad KC, Beukelman DR. Integrating AAC strategies with natural speech in adults. In: Beukelman DR, Yorkston KM, Reichle J, editors. Augmentative and alternative communication for adults with acquired neurologic disorders. Baltimore (MD): Brookes Publishing Co; 2000. p. 83-106.
19. Garrett KL, Beukelman DR. Adults with severe aphasia. In: Beukelman DR, Mirenda P, editors. Augmentative and alternative communication: Management of severe communication disorders in children and adults. 2nd ed. Baltimore (MD): Brookes Publishing Co; 1998. p. 465-99.
20. Hine N, Arnott JL, Smith D. Design issues encountered in the development of a mobile multimedia augmentative communication service. Universal Access Inf Soc. 2003; 2(3):255-64.
21. Davidson B, Worrall L, Hickson L. Identifying the communication activities of older people with aphasia: Evidence from naturalistic observation. Aphasiology. 2003;17(3):243-64.
22. Cherney LR, Halper AS, Holland AL, Cole R. Computerized script training for aphasia: Preliminary results. Am J Speech Lang Pathol. 2008;17(1):19-34. [PMID: 18230811]
23. Linebarger MC, Schwartz MF, Romania JR, Kohn SE, Stephens DL. Grammatical encoding in aphasia: Evidence from a "processing prosthesis." Brain Lang. 2000;75(3): 416-27. [PMID: 11112295]
24. Kolk H. A time-based approach to agrammatic production. Brain Lang. 1995;50(3):282-303. [PMID: 7583191]
25. Kolk HH. How language adapts to the brain: An analysis of agrammatic aphasia. In: Progovac LP, Casielles E, Barton E, editors. The syntax of nonsententials multidisciplinary perspectives. Amsterdam (the Netherlands): John Benjamins Publishing; 2006. p. 229-58.
26. Linebarger M, McCall D, Virata T, Berndt RS. Widening the temporal window: Processing support in the treatment of aphasic language production. Brain Lang. 2007;100(1): 53-68. [PMID: 17069883]
27. Taylor WL. Cloze procedure: A new tool for measuring readability. Journalism Q. 1953;30:415-33.
28. Albright E. Investigating sentence shaper: A processing prosthesis [thesis]. Vancouver (Canada): University of British Columbia; 2006.
29. Albright E, Purves B. Exploring SentenceShaper: Treatment and augmentative possibilities. Aphasiology. 2008; 22(7-8):741-52.
30. Bartlett M, Fink RB, Schwartz MF, Linebarger M. Informativeness ratings of messages created on an AAC processing prosthesis. Aphasiology. 2007;21(5):475-98. [PMID: 18648580]
31. Fink RB, Bartlett MR, Lowery JS, Linebarger MC, Schwartz MF. Aphasic speech with and without SentenceShaper: Two methods for assessing informativeness. Aphasiology. 2008; 22(7-8):679-90. [PMID: 18846247]
32. Linebarger MC, McCall D, Berndt RS. The role of processing support in the remediation of aphasic language production disorders. Cogn Neuropsychol. 2004;21(2/4):267-82.
33. McCall D, Linebarger MC, Berndt RS. Predicting effects of computer-based intervention on structure and content of aphasic patients' spoken language [abstract]. Brain Lang. 2007;103(1-2):207-8.
34. Linebarger MC, Schwartz MF, Kohn SE. Computer-based training of language production: An exploratory study. Neuropsychol Rehabil. 2001;11(1):57-96.
35. McCall D, Virata T, Linebarger MC, Berndt RS. Integrating technology and targeted treatment to improve narrative production in aphasia: A case study. Aphasiology. Forthcoming 2009.
36. Mitchum CC, Berndt RS. The cognitive neuropsychological approach to treatment of language disorders. Neuropsychol Rehabil. 1995;5(1):1-16.
37. Aftonomos LB, Steele RD, Appelbaum JS, Harris VM. Relationships between impairment-level assessments and functional-level assessments in aphasia: Findings from LCC treatment programmes. Aphasiology. 2001;15(10-11):951-64.
38. Nicholas M, Sinotte M, Helm-Estabrooks N. Using a computer to communicate: Effect of executive function impairments in people with severe aphasia. Aphasiology. 2005; 19(10-11):1052-65.
39. Linebarger MC, Schwartz M, Kantner TR, McCall D. Promoting access to the Internet in aphasia [abstract]. Brain Lang. 2002;83:169-72.
40. Fried S. The new rabbi: A congregation searches for its leader. New York (NY): Bantam Books; 2002. p. 150;262.
41. Lasker JP, Garrett KL, Fox LE. Severe aphasia. In: Beukelman DR, Garrett K, Yorkston KM, editors. Augmentative communication strategies for adults with acute or chronic medical conditions. Baltimore (MD): Brookes Publishing Co; 2007. p. 163-206.
42. Lasker J, Garrett K. Using the Multimodal Communication Screening Test for Persons with Aphasia (MCST-A) to guide the selection of alternative communication strategies for people with aphasia. Aphasiology. 2006;20(2-4):217-32.
43. Boyd-Graber JL, Nikolova SS, Moffatt KA, Kin KC, Lee JY, Mackey LW, Tremaine MM, Klawe MM. Participatory design with proxies: Developing a desktop-PDA system to support people with aphasia. In: Proceedings of the 2006 Conference on Human Factors in Computing Systems (ACM CHI); 2006 Apr 22-27; Montréal, Québec, Canada. New York: ACM (Association for Computing Machinery) Press; 2006. p. 151-60.
44. Garrett KL, Kimelman MDZ. AAC and aphasia: Cognitive-linguistic considerations. In: Beukelman DR, Yorkston KM, Reichle J, editors. Augmentative and alternative communication for adults with acquired neurologic disorders. Baltimore (MD): Brookes Publishing Co; 2000. p. 339-74.
45. Stuart S, Lasker J, Beukelman D. AAC message management. In: Beukelman DR, Yorkston KM, Reichle J, editors. Augmentative and alternative communication for adults with acquired neurogenic communication disorders. Baltimore (MD): Brookes Publishing Co; 2000. p. 25-54.
46. Mollica BM. Emerging technologies in augmentative and alternative communication: Restorative and compensatory approaches to acquired disorders of communication. Neuro-Rehabilitation. 1999;12(1):27-37.
47. Kertesz A. Western Aphasia Battery test manual. New York (NY): Grune & Stratton; 1982.
48. Blomert L, Kean ML, Koster C, Schokker J. Amsterdam-Nijmegen Everyday Language Test (ANELT): Construction, reliability & validity. Aphasiology. 1994;8(4):381-407.
49. Nicholas LE, Brookshire RH. A system for quantifying the informativeness and efficiency of the connected speech of adults with aphasia. J Speech Hear Res. 1993;36(2):338-50. [PMID: 8487525]
50. Peach R, Wong P. Integrating the message level into treatment for agrammatism using story retelling. Aphasiology. 2004;18(5-7):429-41.
51. Bartlett MR, Schwartz MF, Fink RB, Lowery JS, Linebarger MC. Spoken-language enhancement with SentenceShaper To Go, a portable AAC system based upon processing support [abstract]. Brain Lang. 2007;103(1-2):217-18.
Submitted for publication October 28, 2007. Accepted in revised form May 28, 2008.

Go to TOP
Go to the Table of Contents of Vol. 45 No. 9

Last Reviewed or Updated  Friday, August 28, 2009 10:41 AM

Valid XHTML 1.0 Transitional