Last week, undergraduate sophomore Raymond Xiong presented YuanningLi and colleagues’ new paper on using state-of-the-art deep neural network (DNN) models to investigate the neuro-mechanisms of speech perception. This 🧵 explores our thoughts (🤍 & ❔) www.nature.com/articles/s41...
Using direct intracranial recordings and modern speech AI models, Li and colleagues show representational and computational similarities between deep neural networks for self-supervised speech learnin...
Thank you to all the co-authors for your work, and we look forward to following your progress. cc: Gopala K. Anumanchipalli, Abdelrahman Mohamed, Peili Chen, Laurel H. Carney, Junfeng Lu, Jinsong Wu, & Edward F. Chang
❔3️⃣: How might the cross-linguistic comparison results differ if the participants are bilingual and have linguistic knowledge of both English and Mandarin?
❔2️⃣: In the cross-lingual comparison, when the language of the input speech is different from trained speech, the HuBERT model trained on English systematically achieves higher BPS values than the model trained on Mandarin. What are some possible implications of this?
Similarly, when comparing how speech representations for different layer architectures predict neural responses, the results for HG differ quite a lot from those for other ROIs. What might be some potential reasons behind these findings?
❔1️⃣: The speech representations across different layers of the DNN models do not significantly better predict neural responses in Heschl’s gyrus as compared to the baseline TRF model. Con't:
🤍3️⃣: The cross-linguistic comparison of the performance of Mandarin and English DNNs on predicting neural responses shows that pretraining on large corpora enables the DNN to extract language-specific phonological knowledge, which could help studies in language acquisition.
🤍2️⃣: The comparison of the trained attention-weight matrices in DNN models with templates suggests that the correlation between trained attention-weight matrices and corresponding templates is positively correlated with prediction accuracy in the superior temporal gyrus.
🤍1️⃣: Analyzing the performance of speech representations for different DNN layers in predicting neural encodings suggests a hierarchical structure in the human auditory pathway similar to the one in the DNN models.