Every person has a unique set of individual experiences that make up their memories, yet surprisingly, recent work has shown that people tend to remember and forget the same images. This is because these images differ in their memorability – a predictive value of whether an image is likely to be later remembered or not. Here, I describe the first characterization of the neural and psychophysical bases of memorability. I show that memorability is highly consistent in the domain of face images, despite their similar perceptual features and same basic-level category, and also consistent across images of the same identity. I then compare memorability to several phenomena known to influence memory – bottom-up attention, top-down attention, and priming – and find that memorability effects remain independent of these phenomena. Lastly, through human functional magnetic resonance imaging experiments, I find sensitivity to memorability in the medial temporal lobe, with a memorability-centric representational geometry in the neural patterns in these regions. Importantly, this sensitivity is dissociable from classical individual subsequent memory effects that I show to be localized in the prefrontal cortex. These results indicate that until now, memory work has largely confounded the effects of the participant (individual memory) and the stimulus (memorability), and I propose a re-examination of past memory findings through the lens of memorability. In whole, this work presents memorability as a novel phenomenon, easily quantified for images and entities, with its own dedicated neural signatures at the intersection of perception and memory.