The locus coeruleus (LC) is a small brainstem nucleus and the primary source of the neuromodulator norepinephrine (NE) in the brain. Receiving diverse, widespread inputs, while also projecting throughout the brain, LC-NE neurons are thought to signal global brain states such as arousal and attention. LC-NE is also implicated in more precise roles such as mediating learning and signaling unexpected uncertainty, though it’s exact function in learned behaviors is unknown. Here, we explore the role of LC-NE in reinforcement learning by studying the spatiotemporal dynamics of transient LC-NE release during learned behaviors, the impact of this release on behavior, and the effects of LC-NE on task representation and processing in cortical target regions. We show that LC-NE is important for two aspects of reinforcement learning: promoting task execution under uncertain conditions and facilitating improved performance after a surprising outcome. We find that these two components of phasic activity are heterogeneously represented in cortical outputs, such that LC neurons encoding uncertainty project to the motor cortex to facilitate task execution, while surprise/reward prediction error signals are widely distributed to improve response sensitivity on the next trial. We show that LC-NE activity enhances encoding of task variables in motor and prefrontal cortical neurons, and that LC-NE activity following a surprising outcome alters PFC population dynamics to enhance discriminability of the stimulus on the next trial. Finally, we find that cortical astrocytes are responsive to high phasic LC-NE activity in our task, and that this astrocyte calcium signaling mediates performance optimization after a surprising outcome. Taken together, these data indicate that LC-NE neurons signal critical components of reinforcement learning, including reward uncertainty and reward prediction error or surprise, and selectively project these signals throughout the brain to facilitate reinforcement learning by altering astrocyte calcium and neuronal population activity in the cortex.