Multimodal neural networks for articulatory prediction during ultrasound imaging

Speech acoustics depend on the location and degree of vocal tract constrictions, which are not attainable from ultrasound images of the tongue. This study combines acoustic and articulatory data as input into a deep neural network based on bi-directional gated recurrent units (BiGRU) to predict expanded articulatory data. Specifically, audio and tongue contours attainable from ultrasound imaging were used to predict the vocal tract shape during speech. (Manuscript in progress.)
Presentation:
- “Predicting vocal tract shape information from tongue contours and audio using neural networks,” ASA, 2024. Poster
