QuakeFace: Large-Scale Automatic Facial Feature Annotation

This project presents initial findings regarding cross-cultural differences in eyebrow movement during speech. Using the newly developed QuakeFace software, an automatic facial feature annotator designed to be used for... [ view full abstract ]

This project presents initial findings regarding cross-cultural differences in eyebrow movement during speech. Using the newly developed QuakeFace software, an automatic facial feature annotator designed to be used for large-scale annotation of audiovisual corpora, eyebrow movement between ethnically self-identified Māori and Pākehā speakers of New Zealand English is compared. Building on Gruber et al. (2016), who found greater rates of eyebrow flashes for speakers of Māori ethnicity, this study uses a larger dataset and objective coding to examine the rate of eyebrow flashes between ethnicities. In the present study, eyebrow flashes are examined for a total of 16 speakers (8 Māori, 8 Pākehā) from the QuakeBox Corpus (Walsh et al. 2013) housed at the New Zealand Institute of Language, Brain and Behaviour.

In addition to findings regarding eyebrow movement, this study introduces QuakeFace, a tool that opens the door to countless avenues of research regarding the links between facial movement and speech. For each video file, QuakeFace identifies 71 different facial landmarks, including mouth, nose, eyebrows, eyes, and pupils, and then records the coordinates on a frame-by-frame basis. Work is underway to annotate the entire QuakeBox Corpus, which consists of 700 high quality videos of speakers telling their earthquake stories.

Previous work using facial feature annotation has required time-intensive manual annotation and faced difficulties with low rates of annotator agreement (Gruber et al. 2016). Automatic facial feature annotation allows for objective, large-scale annotation, which in turn allows for large-scale research on the link between speech and facial motion.

Gruber, J., King, J., Hay, J., & Johnston, L. (2016). The hands, head, and brow: A sociolinguistic study of Māori gesture. Gesture, 15(1), 1-36. 

Walsh, L., Hay, J., Bent, D., Grant, L., King, J., Millar, P., Papp, V., & Watson, K. (2013). The UC QuakeBox Project: Creation of a community-focused research archive. New Zealand English Journal, 27, 20-32.