GPBoost is a software library for combining tree-boosting with Gaussian process and grouped random effects models (aka mixed effects models or latent Gaussian models). It also allows for independently applying tree-boosting as well as Gaussian process and (generalized) linear mixed effects models (LMMs and GLMMs). The GPBoost library is predominantly written in C++, it has a C interface, and there exist both a Python package and an R package.
For more information, you may want to have a look at:
The GPBoost library allows for combining tree-boosting with Gaussian process (GP) and grouped random effects models (aka mixed effects models or latent Gaussian models) in order to leverage advantages and remedy drawbacks of these two approaches (see below for a list with advantages and disadvantages of these modeling techniques). In particular, the GPBoost / LaGaBoost algorithms are generalizations of classical boosting algorithms which assume (conditional) independence across samples. Advantages include that this can allow for more efficient learning of predictor functions which, among other things, can translate into increased prediction accuracy, it can be used as a solution for high-cardinality categorical variables in tree-boosting, and it can be used for modeling spatial or spatio-temporal data when, e.g., spatial predictions should vary continuously , or smoothly, over space. Further, the GPBoost / LaGaBoost algorithms can be seen as non-linear extensions of classical mixed effects or latent Gaussian models, where the linear predictor function is replaced by a non-linear function which is learned using tree-boosting (=arguably, often the "best" approach for tabular data).
The GPBoost library implements two algorithms for combining tree-boosting with Gaussian process and grouped random effects models:
For Gaussian likelihoods (GPBoost algorithm), it is assumed that the response variable (aka label) y is the sum of a potentially non-linear mean function F(X) and random effects Zb:
y = F(X) + Zb + xi
where xi is an independent error term and X are predictor variables (aka covariates or features).
For non-Gaussian likelihoods (LaGaBoost algorithm), it is assumed that the response variable y follows some distribution p(y|m) and that a (potentially multivariate) parameter m of this distribution is related to a non-linear function F(X) and random effects Zb:
y ~ p(y|m) m = G(F(X) + Zb)
where G() is a so-called link function.
In the GPBoost library, the random effects can consist of
Learning the above-mentioned models means learning both the covariance parameters (aka hyperparameters) of the random effects and the predictor function F(X). Both the GPBoost and the LaGaBoost algorithms iteratively learn the covariance parameters and add a tree to the ensemble of trees F(X) using a gradient and/or a Newton boosting step. In the GPBoost library, covariance parameters can (currently) be learned using (Nesterov accelerated) gradient descent, Fisher scoring (aka natural gradient descent), and Nelder-Mead. Further, trees are learned using the LightGBM library.
Tree-boosting has the following advantages and disadvantages:
|Advantages of tree-boosting||Disadvantages of tree-boosting|
|- State-of-the-art prediction accuracy||- Assumes conditional independence of samples|
|- Automatic modeling of non-linearities, discontinuities, and complex high-order interactions||- Produces discontinuous predictions for, e.g., spatial data|
|- Robust to outliers in and multicollinearity among predictor variables||- Can have difficulty with high-cardinality categorical variables|
|- Scale-invariant to monotone transformations of predictor variables|
|- Automatic handling of missing values in predictor variables|
Gaussian process (GPs) and grouped random effects models (aka mixed effects models or latent Gaussian models) have the following advantages and disadvantages:
|Advantages of GPs / random effects models||Disadvantages of GPs / random effects models|
|- Probabilistic predictions which allows for uncertainty quantification||- Zero or a linear prior mean (predictor, fixed effects) function|
|- Incorporation of reasonable prior knowledge. E.g. for spatial data: "close samples are more similar to each other than distant samples" and a function should vary continuously / smoothly over space|
|- Modeling of dependency which, among other things, can allow for more efficient learning of the fixed effects (predictor) function|
|- Grouped random effects can be used for modeling high-cardinality categorical variables|
This project is licensed under the terms of the Apache License 2.0. See LICENSE for more information.