Learning with structured covariance matrices in linear Gaussian models

We study structured covariance matrices in a Gaussian setting for a
variety of data analysis scenarios. Despite its simplistic nature, we
argue for the broad applicability of the Gaussian family through its
second order statistics. We focus on three types of common structures
in the machine learning literature: covariance functions, low-rank and
sparse inverse covariances. Our contributions boil down to combin-
ing these structures and designing algorithms for maximum-likelihood
or MAP fitting: for instance, we use covariance functions in Gaus-
sian processes to encode the temporal structure in a gene-expression
time-series, with any residual structure generating iid noise. More
generally, for a low-rank residual structure (correlated residuals) we
introduce the residual component analysis framework: based on a
generalised eigenvalue problem, it decomposes the residual low-rank
term given a partial explanation of the covariance. In this example
the explained covariance would be an RBF kernel, but it can be any
positive-definite matrix. Another example is the low-rank plus sparse-
inverse composition for structure learning of GMRFs in the presence
of confounding latent variables. We also study RCA as a novel link
between classical low-rank methods and modern probabilistic counter-
parts: the geometry of oblique projections shows how PCA, CCA and
linear discriminant analysis reduce to RCA. Also inter-battery factor
analysis, a precursor of multi-view learning, is reduced to an itera-
tive application of RCA. Finally, we touch on structured precisions of
matrix-normal models based on the Cartesian factorisation of graphs,
with appealing properties for regression problems and interpretabil-
ity. In all cases, experimental results and simulations demonstrate the
performance of the different methods proposed.