Human beings have a sophisticated understanding of emotions. Lots of studies have looked at people’ ability to infer emotions from emotional cues such as facial expressions and vocalizations. In a partially observed world, however, our studies show that even young children can recover much richer information than previous research has claimed. In three studies, we found that before 2 years old, toddlers can make nuanced distinction between within-valence emotional expressions and match these expressions to their external causes in the world; by 5 years old, children can jointly recover an agent’s beliefs and desires from (and only from) a valence change in facial expressions. At last, we developed a generative model that captures the high-level structure of the causal relationships between beliefs, desires, actions, outcomes and emotional reactions. This model can quantitatively predict adults’ joint inferences of beliefs and desires from emotional expressions in diverse contexts. Based on these results, we propose that even young children have a rich representation of emotions. This representation is intertwined with the external world and the internal mental states in a way that allows rich and rational inferences from sparse observations of emotional cues.