{'fraction_non_alphanumeric': 0.019687026754164564, 'fraction_numerical': 0.0005047955577990914, 'mean_word_length': 5.53960396039604, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Modelling, inference and optimization in probabilistic machine learning

Bayesian machine learning has gained tremendous attention in the machine
learning community over the past few years. Bayesian methods offer a
coherent reasoning for quantifying uncertainties in the decision making
procedure, based on the Bayes rule. One of the core advantages of Bayesian
methods is the separation of modelling and inference. In other words, the
likelihood models are completely independent of the computation of the
posterior distribution of the parameters. There are many Bayesian models that are widely used in the machine
learning community. For example, non-parametric models such as Gaussian
Processes and Dirichlet Processes are flexible models which are able to
capture and learn the structure of the data. Bayesian deep learning
models, which are based on neural networks, are another example of
flexible Bayesian models that are rich enough to represent non-linear
structures in the data. The process of inferring the posterior lies at the center of Bayesian inference.
When computing the posterior distribution exactly is not feasible, due to
intractability of the posterior and the computational or memory constraints,
approximate Bayesian inference comes to play. In this PhD thesis, I develop and investigate various Bayesian modelling
and inference techniques and apply them to multiple interesting domains
and tasks. We begin with Tucker Gaussian Processes(TGP), a class of
flexible non-parametric models based on Gaussian Processes (GP). We
apply the method to 1) regression problems on structured input data, and
2) collaborative filtering problems where TGP offers an elegant way of
incorporating side information. We demonstrate superior results compared
with benchmarks on a number of examples across different domains. A closely related line of research based on GPs is Bayesian Optimization
(BO). It is a black-box optimizer where one optimizes an objective function
through subsequent queries about next input locations to be evaluated at. However, this method does not work well when the input space is
non-Euclidean or combinatorial. We alleviate the problem by learning a
low dimensional Euclidean representation of the combinatorial input space
with variational inference, using Variational Auto-encoder (VAE). The
optimization can then be conducted on the low dimensional embedding
instead. We apply our method to Automatic Statistician and natural scene
understanding, which give promising results. For approximate Bayesian inference, we first propose an algorithm called
Relativistic Hamiltonian Monte Carlo (RHMC) which is a variant of
MCMC. In particular, we replace Newton’s kinetic energy in the Hamiltonian
with Einstein’s relativistic kinetic energy, which makes the algorithm
more robust. There are several extensions to RHMC, including a stochastic
gradient version for scalability, a thermostat version based on the temperature
of the physical system and a resulting optimization algorithm which
gives comparable performance compared with the state-of-the-art. Finally, we propose another sampling based inference method called the
Adaptive Importance Sampling with Exploration and Exploitation (Daisee),
where we look into the problem of exploration-exploitation in adaptive
importance sampling through establishing a natural connection between
importance sampling and multi-armed bandit problem. In particular,
through a finite-time regret analysis we show that the regret of the proposed
algorithm grows sublinearly with time. Further, we propose a hierarchical
extension of Daisee to encourage exploration in the region with high
uncertainty. The new models proposed in this thesis help to allow for more flexible
Bayesian modelling and the inference techniques introduced can open new
research directions for efficient and accurate posterior inference. These
contribute to Bayesian inference and probabilistic machine learning.