The Datta lab studies how information from the outside world is detected, encoded in the brain, and transformed into meaningful behavioral outputs. Here we describe a new model- based approach we have recently developed, which combines 3D machine vision with unsupervised machine learning, to characterize the underlying structure of mouse behavior. We refer to this approach as Motion Sequencing (MoSeq). Using MoSeq we have discovered that mouse behavior can be segmented into a fundamental set of components that we call “behavioral syllables.” Each behavioral syllable is a brief and well-defined motif of 3D behavior that the brain places in into specific sequences via definable transition statistics (or behavioral “grammar”) to flexibly create complex patterns of action. By characterizing mouse behavior in terms of its component parts, we can use our behavioral characterization technique to identify subtle differences in the pattern of motor output under different experimental conditions with an unprecedented level of sensitivity. By combining MoSeq with in vivo imaging of neural circuits in behaving animals, we have also identified context-dependent neural correlates for the sub- second structure, suggesting that MoSeq provides direct insights into the relationship between neural circuit activity and patterns of action. MoSeq will therefore afford insight into mechanisms that allow animals to flexibly navigate the outside world, enable better characterization of mouse models of disease, and serve as a quantitative prism through which the function of genes and neural circuits can be understood.