feat: add soft gradient boosting (#72)

* update code * update code * update code * update unit tests * update code * Update README.rst
TorchEnsemble-Community · May 17, 2021 · 14171a5 · 14171a5
1 parent 3cd7b77
commit 14171a5
Show file tree

Hide file tree

Showing 7 changed files with 580 additions and 17 deletions.
diff --git a/CHANGELOG.rst b/CHANGELOG.rst
@@ -18,6 +18,7 @@ Changelog
 Ver 0.1.*
 ---------
 
+* |Feature| |API| Add :class:`SoftGradientBoostingClassifier` and :class:`SoftGradientBoostingRegressor` | `@xuyxu <https://github.com/xuyxu>`__
 * |Fix| Fix missing functionality of ``use_reduction_sum`` for :meth:`fit` of Gradient Boosting | `@xuyxu <https://github.com/xuyxu>`__
 * |Enhancement| Relax :mod:`tensorboard` as a soft dependency | `@xuyxu <https://github.com/xuyxu>`__
 * |Enhancement| |API| Simplify the training workflow of :class:`FastGeometricClassifier` and :class:`FastGeometricRegressor` | `@xuyxu <https://github.com/xuyxu>`__

diff --git a/README.rst b/README.rst
@@ -92,23 +92,25 @@ Example
 Supported Ensemble
 ------------------
 
-+-------------------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------+
-|    **Ensemble Name**    |  **Type**  |                                                                                                                             **Paper**                                                                                                                            |      **Repository**     |
-+-------------------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------+
-|          Fusion         |    Mixed   |                                                                                                              Not listed for its Perceptual Intuition                                                                                                             |        fusion.py        |
-+-------------------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------+
-|          Voting         |  Parallel  |                                                                                                              Not listed for its Perceptual Intuition                                                                                                             |        voting.py        |
-+-------------------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------+
-|         Bagging         |  Parallel  |                                                                                       `Bagging Predictors <https://link.springer.com/content/pdf/10.1007/BF00058655.pdf>`__                                                                                      |        bagging.py       |
-+-------------------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------+
-|    Gradient Boosting    | Sequential | `Greedy Function Approximation: A Gradient Boosting Machine <https://www.jstor.org/stable/pdf/2699986.pdf?casa_token=3fkT9safZHUAAAAA:HT_MeRk_xNsUZkOpbixOtXc950xnRSXNAyl7WjGZgjLtwBTAzZaQe2urnVyp5sK1dIXRL-9hVrdvjT-Ex_PEvov5tTyFg6wMaSbhCzkJRfUj4uBJ6l_PHA>`__ |   gradient_boosting.py  |
-+-------------------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------+
-|    Snapshot Ensemble    | Sequential |                                                                                 `[ICLR'17] Snapshot Ensembles: Train 1, Get m For Free <https://arxiv.org/pdf/1704.00109.pdf>`__                                                                                 |   snapshot_ensemble.py  |
-+-------------------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------+
-|   Adversarial Training  |  Parallel  |                                                                  `[NIPS'17] Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles <https://arxiv.org/pdf/1612.01474.pdf>`__                                                                 | adversarial_training.py |
-+-------------------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------+
-| Fast Geometric Ensemble | Sequential |                                                                     `[NeurIPS'18] Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs <https://arxiv.org/pdf/1802.10026;Loss>`__                                                                       |    fast_geometric.py    |
-+-------------------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------+
++-------------------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------+
+|    **Ensemble Name**    |  **Type**  |                                                                                                                             **Paper**                                                                                                                            |       **Repository**      |
++-------------------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------+
+|          Fusion         |    Mixed   |                                                                                                              Not listed for its Perceptual Intuition                                                                                                             |         fusion.py         |
++-------------------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------+
+|          Voting         |  Parallel  |                                                                                                              Not listed for its Perceptual Intuition                                                                                                             |         voting.py         |
++-------------------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------+
+|         Bagging         |  Parallel  |                                                                                       `Bagging Predictors <https://link.springer.com/content/pdf/10.1007/BF00058655.pdf>`__                                                                                      |         bagging.py        |
++-------------------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------+
+|    Gradient Boosting    | Sequential | `Greedy Function Approximation: A Gradient Boosting Machine <https://www.jstor.org/stable/pdf/2699986.pdf?casa_token=3fkT9safZHUAAAAA:HT_MeRk_xNsUZkOpbixOtXc950xnRSXNAyl7WjGZgjLtwBTAzZaQe2urnVyp5sK1dIXRL-9hVrdvjT-Ex_PEvov5tTyFg6wMaSbhCzkJRfUj4uBJ6l_PHA>`__ |    gradient_boosting.py   |
++-------------------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------+
+|  Soft Gradient Boosting |  Parallel  |                                                                                         `[arXiv'20] Soft Gradient Boosting Machine <https://arxiv.org/abs/2006.04059>`__                                                                                         | soft_gradient_boosting.py |
++-------------------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------+
+|    Snapshot Ensemble    | Sequential |                                                                                 `[ICLR'17] Snapshot Ensembles: Train 1, Get m For Free <https://arxiv.org/pdf/1704.00109.pdf>`__                                                                                 |    snapshot_ensemble.py   |
++-------------------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------+
+|   Adversarial Training  |  Parallel  |                                                                  `[NIPS'17] Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles <https://arxiv.org/pdf/1612.01474.pdf>`__                                                                 |  adversarial_training.py  |
++-------------------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------+
+| Fast Geometric Ensemble | Sequential |                                                                        `[NIPS'18] Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs <https://arxiv.org/pdf/1802.10026;Loss>`__                                                                       |     fast_geometric.py     |
++-------------------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------+
 
 Package Dependency
 ------------------

diff --git a/docs/parameters.rst b/docs/parameters.rst
@@ -85,6 +85,25 @@ GradientBoostingRegressor
 .. autoclass:: torchensemble.gradient_boosting.GradientBoostingRegressor
     :members:
 
+Soft Gradient Boosting
+----------------------
+
+In soft gradient boosting, all base estimators could be simultaneously
+fitted, while achieving the similar boosting improvements as in gradient
+boosting.
+
+SoftGradientBoostingClassifier
+******************************
+
+.. autoclass:: torchensemble.soft_gradient_boosting.SoftGradientBoostingClassifier
+    :members:
+
+SoftGradientBoostingRegressor
+*****************************
+
+.. autoclass:: torchensemble.soft_gradient_boosting.SoftGradientBoostingRegressor
+    :members:
+
 Snapshot Ensemble
 -----------------
 

diff --git a/torchensemble/__init__.py b/torchensemble/__init__.py
@@ -12,6 +12,8 @@
 from .adversarial_training import AdversarialTrainingRegressor
 from .fast_geometric import FastGeometricClassifier
 from .fast_geometric import FastGeometricRegressor
+from .soft_gradient_boosting import SoftGradientBoostingClassifier
+from .soft_gradient_boosting import SoftGradientBoostingRegressor
 
 
 __all__ = [
@@ -29,4 +31,6 @@
     "AdversarialTrainingRegressor",
     "FastGeometricClassifier",
     "FastGeometricRegressor",
+    "SoftGradientBoostingClassifier",
+    "SoftGradientBoostingRegressor",
 ]