The majority of us easily picked up our mother tongue as we started to make sense of this world. However, it is not always the case that language learning happens effortlessly. Some children struggle to master their first language for effective communication, and almost all adults find it a challenging task to learning a new language. What happens in the brain of the struggling learners? In other words, why do others not seem to struggle that much? In the first half of my talk, I will present a multi-modal neuroimaging project investigating the neural substrates of phonological impairment in school-age children with ASD. Both structural and functional connectivity data reveal reduced coupling between left frontal and left posterior temporal regions in children with ASD. An MEG experiment further examines children's neural sensitivity to speech distribution information. Children with ASD showed a left-hemisphere hyposensitivity to the probabilistic information embedded in speech content, but not that embedded in speech acoustics. In the second half of my talk, I will present two training studies aiming to understand the interface between individuals' language learning aptitudes and specific functional and structural brain characteristics. Our results reveal learners' initial phonological skills and how they tune their hemispheric sensitivity to foreign speech sounds are critical for successful language learning. Moreover, individual differences in processing native-language semantics and syntax also play specific roles in supporting vocabulary and syntactic acquisition in a new language.