Abstract: One puzzle of moral cognition is that while our moral theories are often described in terms of absolute rules (e.g., the greatest amount of good for the greatest number, or the doctrine of double effect), our moral judgments are graded. We hypothesize that since moral judgments are particularly sensitive to the agent's mental states, uncertainty in these inferred mental states might partially underlie these graded responses. While previous computational models have shown that mental states such as beliefs and desires can be inferred from behavior, they have critically lacked a third mental state, intentions, which play a central role in moral judgment. In this work, we develop a novel computational representation for reasoning about other people's intentions based on counterfactual contrasts over influence diagrams. This model captures the future-oriented aspect of intentional plans and distinguishes between intended outcomes and unintended side effects. Finally, we give a probabilistic account of moral permissibility which produces graded judgments by integrating uncertainty about inferred intentions with utilitarian maximization. By grounding moral permissibility in an intuitive theory of planning, we quantitatively predict the fine-grained structure of both intention and moral permissibility judgments in classic and novel moral dilemmas.