When asked about her stroke, Debi L. Green pulls a worn scrap of paper out of her purse and points to “October 15, 2004.” The paper is a collection of dates, diagnoses and vital facts that helps her communicate when she can’t find the right words.

The 49-year-old former mortgage adjuster suffers from aphasia, a disorder caused by damage to the language center in the brain as a result of stroke or other head trauma. She has spent the last ten years re-learning how to talk, read and write.

Temple researchers are developing a virtual speech therapist that could someday help people like Green by providing a more accessible and cost-effective tool for the repetition and practice exercises that are crucial to recovery.

The two-year study, which is being conducted by Temple’s Nadine Martin, professor of communication sciences; Emily Keshner, professor and chair of physical therapy; and Justin Shi, associate professor of computer science; along with Alex Rudnicky, professor of computer science at Carnegie Mellon University, is supported by a new $275,000 R21 NIH grant.

Currently, economics and the structure of the health care system limit the number of speech therapy sessions people with aphasia can undergo. After the sessions expire, they’re sent home with scripts to practice on their own or with a family member. While therapy doesn’t cure aphasia, it does improve language abilities — generally, those who stay stimulated through practice, therapy and feedback will continue to improve.

It’s the last part — feedback — that’s missing from simply practicing at home on one’s own, says Keshner. Without it, people with aphasia are likely to continue to make the same mistakes over and over again. A speech therapist, and someday a virtual clinician, can help the person catch and correct the mistakes.

Feedback has also not yet been perfected in the virtual world. Current computer-based speech therapy programs are built on rigid scripts. They’re unable to respond to patient mistakes, and there’s no allowance for disordered speech. If a patient uses the wrong word or makes an unfamiliar sound, the computer gets confused.

“That’s the real stumbling block — speech recognition,” said Martin. “We want to develop a virtual clinician that can be more flexible. We are developing scripts, but we’re incorporating possible utterances that the system will have to recognize. How far can we take that? That’s the big challenge for us.”

Another goal of the project is to determine how people with aphasia will respond to a virtual vs. a real therapist. This is where Shi and graduate assistant Greg Teodoro come in. They are creating different versions of the avatar for the virtual therapist — according to gender, ethnicity and even voice texture — that they will test using what is called the “Wizard of Oz” technique. As in the movie, a real person will be stationed behind a curtain and will respond to, in this case, a patient, who is interacting with the avatar of the virtual therapist.

“Ultimately, we’ll be able to program a virtual therapist to creatively respond to almost anything a patient says,” said Keshner. More specifically, she explains, the computer will be able to select content from what the patient says, restructure it and bring it back to the client who can then figure out what to say next.

“That’s the whole process of neural relearning: error recognition, then the generation of a speech plan and a motor plan,” said Keshner.

Rudnicky, of Carnegie Mellon, will focus on the language recognition capability of the virtual clinician, using a natural language processing system that he’s already developed. Shi describes it as the machine or the engine behind the avatar.

“We are putting the face to Rudnicky’s machine,” said Shi. “Our face will reflect what the machine is thinking so it has to be perceptive and expressive.”

Green visits Temple’s Speech and Language Clinic often, both for therapy and support. She also practices on her own, for instance, by talking on the phone with her cousin. But she wants to be able to do more, like make her own doctor’s appointment over the phone. Despite her struggles to find the right words, she stays positive, often tossing her head back and laughing when she has trouble answering a question, like, “What does it mean to be able to speak?”

“A lot! It means a lot of things,” she exclaims. “I will do anything to learn how to speak more,” she said. “I just can’t make a point. Words don’t come out. I want to speak legibly.” 

Martin underlines the vital importance of the ability to communicate.

“If you lose your ability to communicate as you used to, you lose power — in your relationships and in yourself,” said Martin. “You are, or at least think you are, perceived differently.”

The researchers are already preparing to apply for more funding to continue their work. In addition to developing a cost-effective virtual speech therapist, they believe their work could someday lead to broader applications for virtual therapy in many areas of rehabilitation.