With Generative AI rapidly gaining attention in the machine learning community, it is important to understand its foundational theories and inherent properties. Despite its burgeoning popularity, comprehensive research elucidating its underlying principles remains scant. This dissertation delves into the theoretical properties of generative models and unnormalized sampling, framing challenges as optimization quandaries within a probabilistic space. We establish the connections between prevailing algorithms and first-order methods in the probabilistic landscape.

Our first part focuses on the first-order algorithm to perform unnormalized sampling. Our algorithm uses a different discretization strategy than MCMC, which is proven to be more efficient from both theoretical and empirical pers...[ Read more ]

With Generative AI rapidly gaining attention in the machine learning community, it is important to understand its foundational theories and inherent properties. Despite its burgeoning popularity, comprehensive research elucidating its underlying principles remains scant. This dissertation delves into the theoretical properties of generative models and unnormalized sampling, framing challenges as optimization quandaries within a probabilistic space. We establish the connections between prevailing algorithms and first-order methods in the probabilistic landscape.

Our first part focuses on the first-order algorithm to perform unnormalized sampling. Our algorithm uses a different discretization strategy than MCMC, which is proven to be more efficient from both theoretical and empirical perspectives. We also incorporate new designs of adaptive distance metrics, facilitating the derivation of sampling algorithms attuned to specific geometrical landscapes. The adaptation of loss landscapes makes it more suitable for ill-conditioned target distributions.

A pivotal observation we make pertains to the evident challenges presented by multimode distributions in both generative modeling and unnormalized sampling. In response, we introduce two novel methodologies: (1) A paradigm shift leveraging variational inference, effectively transforming standard transport challenges into conditional transport scenarios. (2) An innovative departure from traditional WGF-based methodologies, advocating a reverse SDE process transitioning from multi-mode distribution to Gaussian distribution models. Our empirical and theoretical evaluations accentuate the efficacy and improvements inherent in these methodologies.

Furthermore, the versatility of sampling techniques is illuminated through their application in analyzing sophisticated systems, notably two-layer neural networks and non-convex non-concave min-max problems. By randomization, we convert these non-convex systems to convex systems in probability space, whose existence of solutions and convergence properties can be guaranteed.

These explorations spotlight the indispensable role of generative model theories in both practical and theoretical domains.