Neural network language models have achieved impressive performance on a broad variety of tasks, completing our emails and even generating believable paragraphs of news text. But to what extent are these algorithms models of human linguistic cognition? We address this question using two complementary approaches: First, we import methods from psycholinguistics to assess how well models are learning static, grammatical generalizations that match those of English syntax. Using center embedding as a case study, we provide evidence that neural models are able to approximate the stack-like processing required to process hierarchically structured language. Scaling up this approach, we present an online platform that facilitates targeted evaluation of neural language models across a broad range of grammatical phenomena. Second, we assess the power of neural network models to predict human online processing data, such as eye-gaze duration in naturalistic reading. For the range of architectures and training datasets tested, we find that when it comes to syntactic generalizations, inductive bias is responsible for making the model more humanlike; however, when it comes to predicting real-time processing behavior, minimizing perplexity is the key factor. Our results indicate that more careful, targeted testing regimes can provide a clearer basis for assessing the extent to which contemporary neural networks are models of human linguistic cognition.