Color is a fundamental aspect of visual experience that confers a myriad of behavioral advantages: finding objects in cluttered scenes, recognizing familiar objects, and gleaning information about the state of objects (e.g. the edibility of fruit) and agents in the world (e.g. health or emotional status). As famously pointed out by Marr (1980), a full understanding of perception requires an analysis of the computations performed, the algorithms that carry out those computations, and the implementation of those algorithms in the physical hardware of the brain. This thesis employs psychophysical methods and functional imaging to tackle questions about human color vision at all three levels: what it is used for, how we solve the classic problem of color constancy, and how our color processing machinery is functionally organized in the brain.  Chapter 1 provides a brief survey of the background to these questions. Chapter 2 describes functional MRI studies in humans that find both segregation and convergence of the processing of color and shape in the brain, as well as evidence for the homology of the color system between humans and macaques. Chapter 3 uses psychophysics to investigate the cues used by the human visual system to constrain the classic ill-posed problem of inferring the intrinsic reflectance of an object by discounting the spectral properties of the illuminant. Specifically, these studies find evidence that color constancy is mediated by perceptual and cognitive factors (i.e., low-level features, inferences about 3D scene geometry, prior knowledge, and attention), and provide the first evidence that human skin is a sufficient cue to recover the illuminant to bring about color constant percepts.  Chapter 4 uses psychophysics to evaluate the impact of memory on color appearance of objects and faces. The study finds a novel perceptual illusion that reveals the role of memory for face color in perceptual experience and social communication, and sheds light on the selective pressures for the evolution of trichromatic vision in primates. Taken together, these studies provide clues about the perceptual and neural mechanisms underlying our rich experience of a colorful world.