Mindreading technology can turn brain scans into language

By Dennis Thompson Health Day Reporter

MONDAY, May 1, 2023 (HealthDay News) — A mind-reading device sounds like science fiction, but researchers say they’re firmly on the way to building one.

Using functional MRI (fMRI), a new brain-computer interface can read a person’s thoughts and translate them into full sentences, according to a report published on May 1 in Natural neuroscience.

The decoder was developed to read a person’s brain activity and translate what they want to say into continuous natural language, the researchers said.

“Eventually, we hope this technology can help people who have lost the ability to speak due to injuries like strokes or diseases like ALS,” said the study’s lead author. jerry tanggraduate research assistant at the University of Texas at Austin.

But the interface goes even further than that, translating into language the most important thoughts in a person’s mind.

“We also ran our decoder on brain responses while the user imagined telling stories and performed responses while the user watched silent movies,” Tang said. “And we found that the decoder is also able to pick up the gist of what the user was imagining or seeing.”

Because of this, the decoder is able to capture the essence of what a person is thinking, if not always the exact words, the researchers said.

For example, at one point a participant heard the words: “I don’t have my driver’s license yet”. The decoder translated the thought as, “She hasn’t even started learning to drive yet.”

The technology is not at the point where it can be used on anyone, Tang said.

Training the program required at least 16 hours of participation from each of the three people involved in the research, and Tang said one person’s brain readings cannot be used to inform another’s scans.

The actual analysis also involves the person’s cooperation and can be thwarted by simple mental tasks that distract a participant’s attention, he said.

Still, one expert hailed the results.

“This work represents a breakthrough in brain-computer interface research and is potentially very exciting,” said Dr Mitchell Elkinddirector of clinical sciences for the American Heart Association and professor of neurology and epidemiology at Columbia University in New York.

“The major breakthrough here is being able to record and interpret the meaning of brain activity using a non-invasive approach,” Elkind explained. “Previous work required electrodes placed in the brain using open neurosurgery with risks of infection, bleeding and seizures. This non-invasive approach using MRI would have virtually no risk, and MRI scans are performed regularly in patients with brain damage. This approach can also be used frequently in healthy people in the context of research, without exposing them to risks.”

Powerful findings warn ‘mental privacy’ may be at risk

Indeed, the results of this study were so powerful that Tang and his colleagues felt compelled to issue a warning about “mental privacy.”

“All of this could change as technology improves, so we believe it’s important to continue to research the privacy implications of brain-decoding and to adopt policies that protect each person’s mental privacy. “, said Tang.

Previous efforts to translate brain waves into speech have used electrodes or implants to record impulses from speech-related motor areas of the brain, a lead researcher has said. Alexander Huth. He is an assistant professor of neuroscience and computer science at the University of Texas at Austin.

“These are the areas that control the mouth, larynx, tongue, etc., so what they can decode is how the person is trying to move their mouth to say something, which can be very effective” , Huth said.

The new process takes an entirely different approach, using fMRI to noninvasively measure changes in blood flow and blood oxygenation in brain regions and networks associated with language processing.

“So instead of looking at this kind of low-level engine, our system is really working at the level of ideas, semantics, meaning,” Huth said. “That’s where it’s coming from. That’s why what we get is not the exact words someone heard or said. It is essential. It’s the same idea, but expressed in different words.

The researchers trained the decoder by first recording the three participants’ brain activity as they listened to 16 hours of storytelling podcasts like “Moth Radio Hour,” Tang said.

“It’s more than five times larger than existing language datasets,” he said. “And we use this dataset to build a model that takes into account any sequence of words and predicts how the user’s brain would react upon hearing those words.”

The program mapped changes in brain activity to semantic features of podcasts, capturing the meaning of certain phrases and associated brain responses.

Investigators then tested the decoder by having participants listen to new stories.

Make educated guesses based on brain activity

The decoder essentially attempts to make an educated guess about the words associated with a person’s thoughts, based on brain activity.

Using participants’ brain activity, the decoder generated sequences of words that captured the meanings of the new stories. It even generated exact words and phrases from the stories.

An example of a real story versus a decoded story:

Real: “I got up from the air mattress and leaned my face against the glass of the bedroom window expecting to see eyes staring at me, but instead I only found the darkness.”

Decoded: “I just kept walking to the window and opening the glass. I stood on my tiptoes and looked outside, saw nothing and looked up again, I saw nothing.”

The decoder specifically captured what a person was focused on. When a participant actively listened to a story while another simultaneously played, the program identified the meaning of the story that had captured the listener’s attention, the researchers said.

To see if the decoder captured thoughts rather than speech, the researchers also asked participants to watch silent movies and scan their brain waves.

“There is no language. The subjects were not instructed to do anything while watching these videos. But when we put that data into our decoder, what it spits out is kind of a description of what’s happening in the video,” Huth said.

Participants were also asked to imagine a story, and the device was able to predict the meaning of this imagined story.

“Language is the output format here, but whatever we come up with isn’t necessarily the language itself,” Huth said. “It’s definitely about going to something deeper than language and converting it into language, which is sort of on a very high level the role of language, isn’t it?”

The decoder is not yet ready for prime time

Concerns about mental privacy led researchers to further test whether participants could interfere with the device’s readings.

Certain mental exercises, like naming animals or thinking of a different story than the podcast, “really prevented the decoder from picking up anything about the story the user was hearing,” Tang said.

The process still needs more work. The program is “particularly bad” at pronouns and requires additional tweaking and testing to accurately reproduce exact words and phrases, Huth said.

It’s also not very practical because it now requires the use of a large MRI machine to read a person’s thoughts, the study authors explained.

Researchers wonder if cheaper, more portable technology like EEG or functional near-infrared spectrometry could be used to capture brain activity as effectively as fMRI, Tang said.

But they admit they were shocked at how the set-top box ended up working, leading to their concerns about brain privacy.

“I think my cautionary example is the polygraph, which is not an accurate polygraph, but still had a lot of negative consequences,” Tang said. “So I think even though this technology is in its infancy, it is very important to regulate what brain data can and cannot be used for. And then, if one day it becomes possible to get an accurate decode without getting the person’s cooperation, we’ll have a regulatory base in place that we can build on.

SOURCES: Jerry Tang, graduate research assistant, University of Texas at Austin; Alexander Huth, PhD, assistant professor, neuroscience and computer science, University of Texas at Austin; Mitchell Elkind, MD, MS, MPhil, director of clinical sciences, American Heart Association, and professor of neurology and epidemiology, Columbia University, New York; Natural neuroscienceMay 1, 2023

Copyright © 2023 health day. All rights reserved.