White Rose University Consortium logo
University of Leeds logo University of Sheffield logo York University logo

Learning with structured covariance matrices in linear Gaussian models

Kalaitzis, Alfredo (2013) Learning with structured covariance matrices in linear Gaussian models. PhD thesis, University of Sheffield.

Text (Thesis of Alfredo Kalaitzis, library version, deposited June 7 2013)
Available under License Creative Commons Attribution-Noncommercial-No Derivative Works 2.0 UK: England & Wales.

Download (7Mb)


We study structured covariance matrices in a Gaussian setting for a variety of data analysis scenarios. Despite its simplistic nature, we argue for the broad applicability of the Gaussian family through its second order statistics. We focus on three types of common structures in the machine learning literature: covariance functions, low-rank and sparse inverse covariances. Our contributions boil down to combin- ing these structures and designing algorithms for maximum-likelihood or MAP fitting: for instance, we use covariance functions in Gaus- sian processes to encode the temporal structure in a gene-expression time-series, with any residual structure generating iid noise. More generally, for a low-rank residual structure (correlated residuals) we introduce the residual component analysis framework: based on a generalised eigenvalue problem, it decomposes the residual low-rank term given a partial explanation of the covariance. In this example the explained covariance would be an RBF kernel, but it can be any positive-definite matrix. Another example is the low-rank plus sparse- inverse composition for structure learning of GMRFs in the presence of confounding latent variables. We also study RCA as a novel link between classical low-rank methods and modern probabilistic counter- parts: the geometry of oblique projections shows how PCA, CCA and linear discriminant analysis reduce to RCA. Also inter-battery factor analysis, a precursor of multi-view learning, is reduced to an itera- tive application of RCA. Finally, we touch on structured precisions of matrix-normal models based on the Cartesian factorisation of graphs, with appealing properties for regression problems and interpretabil- ity. In all cases, experimental results and simulations demonstrate the performance of the different methods proposed.

Item Type: Thesis (PhD)
Keywords: covariance matrix, Gaussian distribution, low-rank, sparsity, lasso, L1 regulatisation, inverse covariance estimation, residual component analysis, principal component analysis, canonical correlation analysis, linear discriminant analysis
Academic Units: The University of Sheffield > Faculty of Engineering (Sheffield) > Computer Science (Sheffield)
The University of Sheffield > Faculty of Science (Sheffield) > Computer Science (Sheffield)
Identification Number/EthosID: uk.bl.ethos.574078
Depositing User: Alfredo Kalaitzis
Date Deposited: 18 Jun 2013 10:36
Last Modified: 03 Oct 2016 10:39
URI: http://etheses.whiterose.ac.uk/id/eprint/4038

You do not need to contact us to get a copy of this thesis. Please use the 'Download' link(s) above to get a copy.
You can contact us about this thesis. If you need to make a general enquiry, please see the Contact us page.

Actions (repository staff only: login required)