Speech neuroprostheses can decode speech based on brain activity

Researchers from HSE University and Moscow State Medical and Dentistry University have developed a machine learning model that can predict the word about to be spoken by a subject based on their activity recorded with a small set of minimally invasive electrodes. The article “Speech decoding from a small set of spatially separated minimally invasive intracranial EEG electrodes with a compact and interpretable neural network” was published in the Neural Engineering Journal . The research was funded by a grant from the Russian government under the national project “Science and Universities”.

Millions of people around the world are affected by speech disorders that limit their ability to communicate. The causes of speech loss can vary and include strokes and certain congenital conditions.

Technology is now available to restore communication function to these patients, including “silent speech” interfaces that recognize speech by tracking the movement of articulatory muscles as the person speaks words without making sound. However, such devices help some patients but not others, such as those with facial muscle paralysis.

Speech neuroprostheses; brain-computer interfaces capable of decoding speech based on brain activity; can provide an accessible and reliable solution to restore communication in these patients.

Unlike personal computers, devices with a brain-computer interface (BCI) are controlled directly by the brain without the need for a keyboard or microphone.

A major barrier to wider use of BCIs in speech prostheses is that the technology requires highly invasive surgery to implant electrodes into brain tissue.

The most accurate speech recognition is achieved by neuroprostheses with electrodes covering a large area of ​​the cortical surface. However, these brain activity reading solutions are not intended for long-term use and pose significant risks to patients.

Researchers from the HSE Center for Bioelectrical Interfaces and Moscow State Medical and Dentistry University investigated the possibility of creating a functional neuroprosthesis capable of decoding speech with acceptable accuracy by reading brain activity from of a small set of electrodes implanted in a limited cortical area. The authors suggest that in the future, this minimally invasive procedure could even be performed under local anesthesia. In the current study, researchers collected data from two epileptic patients who had previously been implanted with intracranial electrodes for presurgical mapping to locate areas of seizure onset.

The first patient was implanted bilaterally with a total of five sEEG rods with six contacts each, and the second patient was implanted with nine electrocorticographic (ECoG) strips with eight contacts each. Unlike ECoG, electrodes for sEEG can be implanted without a full craniotomy via a hole drilled into the skull. In this study, only the six contacts of a single sEEG tree in one patient and the eight contacts of an ECoG strip in the other were used to decode neural activity.

Subjects were asked to read aloud six sentences, each presented 30 to 60 times in random order. The sentences varied in their structure and the majority of the words in the same sentence started with the same letter. The sentences contained a total of 26 different words. As the subjects read, the electrodes recorded their brain activity.

This data was then aligned with the audio signals to form 27 classes, including 26 words and a silence class. The resulting training dataset (containing the signals recorded during the first 40 minutes of the experiment) was fed into a machine learning model with a neural network-based architecture. The learning task for the neural network was to predict the next spoken word (class) based on the neural activity data preceding its utterance.

In designing the architecture of the neural network, the researchers wanted to make it simple, compact and easily interpretable. They proposed a two-step architecture that first extracts internal speech representations from recorded brain activity data, producing log-mel spectral coefficients, and then predicts a specific class, namely a word or a silence.

Thus trained, the neural network achieved 55% accuracy using only six channels of data recorded by a single sEEG electrode in the first patient and 70% accuracy using only eight channels of data recorded by a single ECoG strip in the second patient. Such precision is comparable to that demonstrated in other studies using devices requiring the implantation of electrodes over the entire cortical surface.

The resulting interpretable model makes it possible to explain in neurophysiological terms which neural information contributes the most to predicting a word about to be spoken. The researchers examined signals from different neuronal populations to determine which of them were essential for the downstream task. Their findings were consistent with the speech mapping results, suggesting that the model uses neural signals that are essential and therefore can be used to decode imaginary speech.

Another advantage of this solution is that it does not require manual feature engineering. The model learned to extract speech representations directly from brain activity data. The interpretability of the results also indicates that the network decodes signals from the brain rather than from any concomitant activity, such as electrical signals from articulatory muscles or resulting from a microphone effect.

The researchers point out that the prediction was always based on neural activity data preceding the utterance. According to them, this ensures that the decision rule did not use the response of the auditory cortex to the speech already spoken.

The use of such interfaces involves minimal risks for the patient. If all works, it might be possible to decode imaginary speech from neural activity recorded by a small number of minimally invasive electrodes implanted on an outpatient basis under local anesthesia.”

Alexey Ossadtchi, lead study author, director of the HSE Institute’s Center for Bioelectrical Interfaces for Cognitive Neuroscience