A groundbreaking development in brain-computer technology has emerged from the collaboration between researchers at UC San Francisco and UC Berkeley.
They have successfully created a brain-computer interface (BCI) that empowers a woman suffering from severe paralysis due to a brainstem stroke to communicate through a digital avatar. This pioneering advancement represents the first instance where speech and facial expressions have been synthesized directly from brain signals.
The key breakthrough lies in the system’s capability to decode these brain signals into text at an impressive rate of nearly 80 words per minute, a substantial improvement over existing commercial technology.
Spearheading this project is Edward Chang MD, the chair of neurological surgery at UCSF, who has been dedicated to developing this brain-computer interface (BCI) for over a decade. The recent research findings, featured in the journal Nature, are seen as a stepping stone towards an FDA-approved system that could facilitate speech through brain signals in the near future.
The primary objective is to reinstate a comprehensive and natural mode of communication that aligns with our inherent way of connecting with others. Edward Chang envisions this technology as a real solution for patients seeking to restore communication abilities.
In previous work, Chang’s team demonstrated the feasibility of converting brain signals into text in a man who had also experienced a brainstem stroke [1]. However, the current study’s ambition goes further by translating brain signals into the intricacies of speech and the associated facial expressions during conversation.
The technique involved implanting 253 electrodes onto the woman’s brain surface, targeting areas essential for speech generation. These electrodes intercepted brain signals that would have otherwise triggered muscle movements in the tongue, jaw, larynx and facial expressions were it not for the stroke. These signals were then routed through a cable connected to a computer bank via a port attached to her head.
For weeks, the participant collaborated with the research team to train the system’s artificial intelligence algorithms. This process focused on teaching the computer to recognize her unique brain signal patterns for speech.
Instead of teaching the AI to recognize complete words, the researchers opted for a system that decodes words from phonemes – speech sub-units that mirror letters forming written words. This approach streamlined the learning process and significantly enhanced accuracy and speed.
To replicate her voice, the researchers developed an algorithm for speech synthesis, personalized to match her pre-injury voice, using a recording of her speech from her wedding.
An animated avatar was then brought to life with software that simulated muscle movements in the face. This technology, created by Speech Graphics, incorporated customized machine-learning techniques to translate brain signals into facial movements on the avatar.
The result was a synchronized movement of the avatar’s facial features, such as jaw movement, lip protrusion, tongue motion and even emotional expressions like happiness, sadness and surprise.
The team’s future endeavors involve creating a wireless system version, freeing users from physical connections to the BCI. This advancement could empower individuals to independently control their computers and phones, significantly impacting their autonomy and social interactions.
Individuals with paralysis are poised to experience substantial advantages from this technology, as it holds the potential to restore communication abilities and enhance their overall quality of life.
Learn more about “A High-Performance Neuroprosthesis for Speech Decoding and Avatar Control” study in Nature Journal.
[1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9714519/
