Real-time 3D British Sign Language avatar generation from multimodal inputs

Deaf and hard-of-hearing users face persistent barriers in public transport, healthcare and customer-facing services, where interactions are time-sensitive and access to qualified interpreters is limited. SilenceSpeaks is developing AI sign-language avatars that translate text and voice into sign-language video. This PhD will develop the next-generation pipeline: real-time 3D British Sign Language (BSL) avatar generation from multimodal inputs (speech/audio, English text and BSL gloss), enabling scalable deployment in high-impact settings.

The research challenge is to produce faithful, temporally coherent signing with realistic 3D hands, face and body motion under strict latency constraints. Building on an existing 2D pipeline, the PhD will investigate a modular end-to-end approach: (1) multimodal conditioning to align speech/text to BSL gloss with controllable timing; (2) 3D motion synthesis for full-body and fine-grained hand articulation using diffusion-based generative models and neural 3D representations (e.g., NeRF and/or 3D Gaussian splatting, and/or mesh+rig pipelines), with constraints for sign correctness and temporal coherence; (3) real-time 3D avatar rendering with occlusion-aware post-processing, texture/colour consistency and artifact removal; and (4) systems optimisation for interactive deployment, including distillation/quantisation and memory-aware PyTorch implementations.

You will work jointly with the University of Bath (Computer Science) and Silence Speaks, with access to engineering support, domain experts, and Deaf-community annotation and evaluation to establish ground truth for BSL signing. The project is expected to produce publishable research suitable for leading machine learning, computer vision and computer graphics venues, alongside translational outputs supporting real-world accessibility deployments.