In conversation, the discourse history, including past referents and how they were described, shapes future language use. While it is widely known that interlocutors form representations of the discourse history, the veracity and similarity of these representations among interlocutors has not been widely explored. Through the study of referential form in conversation, combined with explicit measures of memory for past referents, I show that interlocutors are likely to walk away from a conversation with distinct memories for the contents, and in some cases the context of conversation. In general, speakers tend to remember what was said better than listeners do. Studies of conversational language use in persons with anterograde amnesia offer insights into the biological memory systems involved. The findings have implications for how common ground is formed in conversation, and suggest that there are limits on the degree to which interlocutors can achieve coordinated representations of the discourse history. More generally, this work demonstrates that the successful exchange of meaning in conversation involves imperfect, asymmetric representations of the jointly experienced past.