|
Abstract : |
We study the decomposition of statistical estimators into linear and higher order parts, or equivalently, the decomposition of the objective function that they optimize into quadratic and higher order terms. We show that bagging reduces the variability of the nonlinear component by replacing it with an estimate of its expected value, while leaving the linear part unaffected. It is therefore most successful when used with highly nonlinear estimators such as decision trees and neural networks. We investigate different resampling schemes and show that half--sampling without replacement is virtually equivalent to traditional bootstrap sampling. It is shown that sampling fractions other than 1/2 often work better with bagging, and that there can be a bias--variance trade--off in choosing an optimal value. In addition to reducing variance, bagging is seen to also reduce bias with certain types of estimators that include decision trees. 1, |
