22/07/2021
An that interprets the brainwaves of a man who can't speak and turns them into conversations - Researchers at UC San Francisco and UC Berkeley trained a system to interpret electrical impulses from the brain of a man who had lost the ability to speak 15 years ago and displayed them as words on a video screen.
A stack of three LSTMs detected portions of brain activity related to speech. An ensemble of 10 convolutional gated recurrent unit models classified speech signals as one of the 50 words. An n-gram language model predicted the probability that a given word would come next. A custom Viterbi decoder, an algorithm often used in communications that are subject to transmission errors, determined the most likely of the 50 phrases based on the models’ output.
Details/research: https://www.nejm.org/doi/full/10.1056/NEJMoa2027540
On LSTM networks (pdf paper):http://www.bioinf.jku.at/publications/older/2604.pdf
Viterbi decoder: https://en.wikipedia.org/wiki/Viterbi_decoder