Tag: sign-language

Automatic sign language identification

My final project for Stanford CS 231N was on automatically identifying sign languages from publicly licensed YouTube clips. For this project I learned from scratch about working with neural networks, computer vision, and video data.

Automatic processing of sign languages can only recently potentially advance beyond the toy problem of fingerspelling recognition. In just the last few years, we have leaped forward in our understanding of sign language theory, effective computer vision practices, and large-scale availability of data. This project achieves better-than-human performance on sign language identification, and it releases a dataset and benchmark for future work on the topic. It is intended as a precursor to sign language machine translation.

Identifying sign languages from video: SLANG-3k

As I haven’t yet created a permanent place to hold the dataset I collected for my most recent class project, I’m hanging it here for now.  SLANG-3k is an uncurated corpus of 3000 clips of 15 seconds each of people signing in American Sign Language, British Sign Language, and German Sign Language, intended as a public benchmark dataset for sign language identification in the wild.  Using 5 frames, I was able to achieve accuracies bounded around 0.66/0.67.  More details can be found in the paper and poster created for CS 231N, Convolutional Neural Networks for Visual Recognition.

Many thanks to everyone who helped with this project — and most especially to the anonymous survey respondents who received only warm fuzzies as compensation for taking the time to help with this early-stage research.

Introduction to ASL Theoretical Linguistics

One of my favorite parts of studying linguistics was being presented with data and being asked to find the system within it. Language data, with linguistic theory’s insistence that everything must make sense, make the most excellent data and logic puzzles.

Screenshot of ASL Linguistics Problems

As part of preparing for a spatial grammar-heavy meeting of the Montgomery Blair High School Linguistics Club, I developed three American Sign Language morphology problems.  These problems illustrate interesting properties of American Sign Language that spoken languages do not have (non-manual markers, spatial agreement, and a rich temporal inflection system based in manual phonology).

Try your hand at doing the problems if you’re interested in any of the following:

  • What it means to do theoretical linguistics (or the sort of logic skills that linguists develop)
  • Unique properties of spatial languages
  • Basic American Sign Language linguistics
  • Similarities between American Sign Language and other world languages

Unlike most materials on ASL linguistics, the problems don’t assume that readers are fluent in American Sign Language or in linguistic theory — I developed these problems because I couldn’t find any resources aimed at an intelligent lay non-Deaf audience.  The problems deliberately walk users through the steps to answer a question, whereas most theoretical linguistics problem sets jump straight to the questions at hand and assume existing familiarity with linguistic features not observed in English.

Once the club and I meet, I’ll post the answer sheet as well.

Introduction to ASL Linguistics

During the discussion we focused on introducing different non-voiced communication forms and on linguistic anthropology/linguistic creativity.  We postponed theoretical linguistics until another time (in which we did some experiential learning on morphology). This page consists of a set of links, prepared videos, and notes designed to support real-time interaction with students at the linguistics club at Montgomery Blair High School.

The big take-away is that American Sign Language is not “English on the hands”.  ASL is independent from English both in grammar and linguistic culture.


  • Caveats for posterity: I’m hearing, I don’t possess native-like fluency in ASL, and I don’t have an advanced degree in this; I do have general and ASL linguistic training, I read widely, and I’m more or less aware of what I don’t know
  • What are some ways deaf people communicate? [YouTube]
  • Compare ASL structure [.avi | .ogv | .gif] with PSE structure [.avi | .ogv | .gif] with English structure [.txt]
  • Charts might help [fingerspelling: ASL | BSL | LSF] [cued speech]

Anthropological Linguistics

  • Big idea: Linguistic creativity
  • ABC stories [YouTube]
  • Sign jokes [King Kong, “please but”, environments, CODAs]
  • Music & poetry [YouTube]
    • Rhyme (handshape, movement path, location, non-manual markers)
    • Rhythm (movement, handedness)
    • Meter (heavy & light syllables)
  • Also, Black ASL [WaPo | HuffPost | YouTube (uncaptioned but 5:37 has a chart)]

Theoretical Linguistics [postponed]

  • Big idea: Spatial grammar
  • Basic structure
    • English consonants have place and manner of articulation, plus voicing [IPA chart]
      • Place of articulation (cat/tat/pat)
      • Manner of articulation (pat/bat/mat)
    • ASL signs have five “parameters”
      • Handshape (think/know) *
      • Location (summer/dry) *
      • Palm orientation (sock/star)
      • Movement (sit/chair) *
      • Nonmanual markers (late/not yet)
  • The movement piece is more complicated (Christian/Congress, one-handed children/die) –> movement-hold theory
    • M (always)
    • H (color, study)
    • M H (think, know, my, sit)
    • H M H (week, guess)
    • M H M H (Congress, flower)
    • M M M H (chair, school, paper)
    • Other structures are possible, but not any other structure (e.g., exclude H M)
  • Nonmanual markers are extremely important grammatical markers; they are frequently unrecognized by hearing people
    • Questions (yes-no/wh)
    • Rhetorical questions
    • Adjectives and adverbs (mm, th, cha, cs — more in a .doc)
    • Topicalization
  • Grammatical use of space of ASL (verb classes, classifiers, aspect, etc.)

Further Resources

  • Deaf people with linguistics training
  • ASL [language | grammar]
  • Gallaudet University [map]
    • 10th-12th grade summer ASL immersion [link]
    • Linguistics department [dept. | event blog]
    • Center for Continuing Studies teaches ASL courses for $230/credit (most classes are 3 credits) [dept.]
    • Theatre performances are captioned or voice-interpreted [link]
  • Books
    • Linguistics of American Sign Language by Valli et al. (ASL linguistics textbook)
    • Signing Naturally (ASL language textbook series)
    • The American Sign Language Handshape Dictionary by Tennant and Brown (dictionary)
    • For Hearing People Only by Moore and Levitan (deaf culture/language in context)
  • Apps
    • ASL Dictionary — 5000 signs [Android | iPhone]
    • ASL spelling game — beginner’s fingerspelling app [Android]
    • Marlee signs — phrases and words [iPhone]
  • Media
    • “Switched at Birth” (ABC Family)
    • YouTube has a variety of performances, lectures and vlogs