Non-convex optimization and learning play an important role in data science and machine learning. Despite their wide use, our understanding is limited in many aspects. In this thesis, I study two important aspects of non-convex optimization and learning: Riemannian optimization and deep neural networks.

In the first part, I develop iteration complexity analysis for Riemannian optimization, i.e. optimization problems defined on Riemannian manifolds. Through bounding the distortion introduced by the metric curvature, iteration complexity of Riemannian (stochastic) gradient descent methods is derived. I also show that some fast first-order methods in Euclidean space, such as Nesterov's accelerated gradient descent (AGD) and stochastic variance reduced gradient (SVRG), have Riemannian counterparts that are also fast under certain conditions.

In the second part, I challenge two common practices in deep learning, namely empirical risk minimization (ERM) and normalization. Specifically, I show (1) training on convex combinations of samples improves model robustness and generalization, and (2) a good initialization is sufficient for training deep residual networks without normalization. The method in (1), called mixup, is motivated by a data-dependent Lipschitzness regularization of the network. The method in (2), called ZeroInit, makes the network update scale invariant to its depth at initialization.

This thesis can be read here: https://www.dropbox.com/s/xih6vzou0nj6ekg/hongyi_thesis_draft.pdf?dl=0