gpb

gpboost

Combining tree-boosting with Gaussian process and mixed effects models

Showing:

Popularity

Downloads/wk

0

GitHub Stars

263

Maintenance

Last Commit

18d ago

Contributors

3

Package

Dependencies

4

License

Apache License, Version 2.0, + see LICENSE file

Categories

Readme

GPBoost icon

GPBoost: Combining Tree-Boosting with Gaussian Process and Mixed Effects Models

Table of Contents

  1. Get Started
  2. Modeling background
  3. News
  4. Open issues - contribute
  5. References
  6. License

Get started

GPBoost is a software library for combining tree-boosting with Gaussian process and grouped random effects models (aka mixed effects models or latent Gaussian models). It also allows for independently applying tree-boosting as well as Gaussian process and (generalized) linear mixed effects models (LMMs and GLMMs). The GPBoost library is predominantly written in C++, it has a C interface, and there exist both a Python package and an R package.

For more information, you may want to have a look at:

Modeling background

The GPBoost library allows for combining tree-boosting with Gaussian process (GP) and grouped random effects models (aka mixed effects models or latent Gaussian models) in order to leverage advantages and remedy drawbacks of these two approaches (see below for a list with advantages and disadvantages of these modeling techniques). In particular, the GPBoost / LaGaBoost algorithms are generalizations of classical boosting algorithms which assume (conditional) independence across samples. Advantages include that this can allow for more efficient learning of predictor functions which, among other things, can translate into increased prediction accuracy, it can be used as a solution for high-cardinality categorical variables in tree-boosting, and it can be used for modeling spatial or spatio-temporal data when, e.g., spatial predictions should vary continuously , or smoothly, over space. Further, the GPBoost / LaGaBoost algorithms can be seen as non-linear extensions of classical mixed effects or latent Gaussian models, where the linear predictor function is replaced by a non-linear function which is learned using tree-boosting (=arguably, often the "best" approach for tabular data).

GPBoost and LaGaBoost algorithms

The GPBoost library implements two algorithms for combining tree-boosting with Gaussian process and grouped random effects models:

  • The GPBoost algorithm (Sigrist, 2020) for data with a Gaussian likelihood (conditional distribution of data)
  • The LaGaBoost algorithm (Sigrist, 2021) for data with non-Gaussian likelihoods

For Gaussian likelihoods (GPBoost algorithm), it is assumed that the response variable (aka label) y is the sum of a potentially non-linear mean function F(X) and random effects Zb:

y = F(X) + Zb + xi

where xi is an independent error term and X are predictor variables (aka covariates or features).

For non-Gaussian likelihoods (LaGaBoost algorithm), it is assumed that the response variable y follows some distribution p(y|m) and that a (potentially multivariate) parameter m of this distribution is related to a non-linear function F(X) and random effects Zb:

y ~ p(y|m)
m = G(F(X) + Zb)

where G() is a so-called link function.

In the GPBoost library, the random effects can consist of

  • Gaussian processes (including random coefficient processes)
  • Grouped random effects (including nested, crossed, and random coefficient effects)
  • Combinations of the above

Learning the above-mentioned models means learning both the covariance parameters (aka hyperparameters) of the random effects and the predictor function F(X). Both the GPBoost and the LaGaBoost algorithms iteratively learn the covariance parameters and add a tree to the ensemble of trees F(X) using a gradient and/or a Newton boosting step. In the GPBoost library, covariance parameters can (currently) be learned using (Nesterov accelerated) gradient descent, Fisher scoring (aka natural gradient descent), and Nelder-Mead. Further, trees are learned using the LightGBM library.

See Sigrist (2020) and Sigrist (2021) for more details.

Background on Gaussian process and grouped random effects models

Tree-boosting has the following advantages and disadvantages:

Advantages of tree-boostingDisadvantages of tree-boosting
- State-of-the-art prediction accuracy- Assumes conditional independence of samples
- Automatic modeling of non-linearities, discontinuities, and complex high-order interactions- Produces discontinuous predictions for, e.g., spatial data
- Robust to outliers in and multicollinearity among predictor variables- Can have difficulty with high-cardinality categorical variables
- Scale-invariant to monotone transformations of predictor variables
- Automatic handling of missing values in predictor variables

Gaussian process (GPs) and grouped random effects models (aka mixed effects models or latent Gaussian models) have the following advantages and disadvantages:

Advantages of GPs / random effects modelsDisadvantages of GPs / random effects models
- Probabilistic predictions which allows for uncertainty quantification- Zero or a linear prior mean (predictor, fixed effects) function
- Incorporation of reasonable prior knowledge. E.g. for spatial data: "close samples are more similar to each other than distant samples" and a function should vary continuously / smoothly over space
- Modeling of dependency which, among other things, can allow for more efficient learning of the fixed effects (predictor) function
- Grouped random effects can be used for modeling high-cardinality categorical variables

News

Open issues - contribute

Software issues

Computational issues

  • Add GPU support for Gaussian processes
  • Add CHOLMOD support

Methodological issues

  • Add a spatio-temporal Gaussian process model (e.g. a separable one)
  • Add possibility to predict latent Gaussian processes and random effects (e.g. random coefficients)
  • Implement more approaches such that computations scale well (memory and time) for Gaussian process models and mixed effects models with more than one grouping variable for non-Gaussian data
  • Support sample weights

References

License

This project is licensed under the terms of the Apache License 2.0. See LICENSE for more information.

Rate & Review

Great Documentation0
Easy to Use0
Performant0
Highly Customizable0
Bleeding Edge0
Responsive Maintainers0
Poor Documentation0
Hard to Use0
Slow0
Buggy0
Abandoned0
Unwelcoming Community0
100
No reviews found
Be the first to rate

Alternatives

No alternatives found

Tutorials

No tutorials found
Add a tutorial