Ecological Archives E088-015-A1

Glenn De'ath. 2007. Boosted trees for ecological modeling and prediction. Ecology 88:243–251.

Appendix A. Simulations comparing aggregated boosted trees and boosted trees.

Simulations

A series of simulations were used to compare the performance of aggregated boosted trees (ABT) and boosted trees (BT) for regression and classification. Since for simulated data we know the true values of the response variables, we can estimate the accuracy, bias and precision for predictions. This can assist our understanding of how these methods work and to what types of problem they are best applied.

For regression, the responses were iid normal, and for classification, binary responses were iid binomial. The simulations used from 4–8 explanatory variables with effect sizes tapered to a random exponential distribution. Tapered polynomial effects for each predictor were randomly varied in degree from linear to cubic, and interactions between them were also included. The inputs were also randomly correlated (-1 < r < 1).

For regression, the error levels were set at 10, 20, and 50% of the total variation of the outputs. The numbers of cases in the training data sets were 100, 200, 400, and 1000, and the test data comprised 2000 cases both with and without error. Shrinkage rates were set at 0.01 and 0.001. For regression and classification, and each size of data and error level, from 10–50 sets of model parameters were generated; more sets were used for data with fewer cases. For each set, 3 replicate training and data sets were generated.

For classification, the error levels were set to give typical misclassification rates of ~10% and 25%. The sizes of the training data sets were either 100 or 1000 cases. For each of the 4 combinations, 20 sets of model parameters with 3 replicate training data sets were generated. Accuracy, bias, and precision were calculated from the model fits from the estimated probabilities (not log-odds).

The corresponding test data set was used to assess their accuracy, bias, and precision of predicting true mean values. PE and misclassification rates were also calculated for regression and classification data respectively. The precision was estimated from the within replicate predictions of the training data; the bias from the mean absolute difference between means of the replicate predictions and the true values of the test data.

For regression (Table A1 and Fig. A1), ABT was more accurate than BT for 94% of data simulations, with a mean improvement of 9.4% that was consistent across the size and noise level of the training data. The improvement in accuracy of ABT over BT was due to a substantial improvement in precision (12.0%) and a small improvement in bias (3.5%). For classification (Table A2), ABT and BT were indistinguishable in accuracy, bias and precision of predicting true mean values (probabilities). The misclassification error of ABT was marginally, but consistently better than BT.

 

 
   FIG. A1. Simulations show the relative predictive performance of aggregated boosted trees (ABT) and boosted trees (BT) for regression. Performance is expressed as the measure for BT divided by the measure for ABT, and thus scores >1 indicate better performance of ABT. The four relative measures of performance are accuracy, bias, and precision of true mean values and accuracy of predictions of individual observations (PE).

The patterns of relative performance (Fig. A1) show that ABTs consistently outperform BTs, particularly at higher levels of shrinkage (less trees), and the difference increases with the size of the data. That improved performance is largely due to improved precision.

 

 
   FIG. A2. Prediction error (PE) for aggregated boosted trees (ABTs: black curves) and single boosted trees (BTs: gray curves) grown on a typical simulated data set of 200 cases. The shrinkage rates were varied from 0.03, 0.01, 0.003, and 0.001 across the panels (a) – (d). The ABT achieved its minimum PE (0.312) with shrinkage 0.003 using 1021 trees, whereas the SBT required a smaller shrinkage rate 0.001 and 5410 trees to achieve its minimum PE (0.341); a value matched by the ABT using shrinkage of 0.01 and 252 trees.

 

The estimated PE (CVI) of ABTs and BTs obtained from the cross-validations was compared to PE estimated from the test data (Table A1). CVI was a conservative estimate of PE for ABTs being 3.4% higher on average than the test data estimate and exceeding it for 92% of the estimates. For BTs however CVI was a slightly optimistic estimate, being on average 1.6% lower than test data estimates and exceeding them 25% of the time.

Further insights into the better performance of ABTs over BTs can be seen in relative rates of learning as ABTs and BTs are grown (Fig. A2). For an ABT with components of a given number of trees, the PE of a BT of the same number of components typically has higher PE. The ABTs are also less prone to over-learning.


TABLE A1. Results of simulations assessing the predictive performance of aggregated boosted trees (ABT) and boosted trees (BT) for regression.

Error

N

Shrinkage

Prediction error (mean)

Bias

Variance

Prediction
error (individual)

     

ABT

BT

ABT

BT

ABT

BT

ABT

BT

CVI

0.5

100

0.01

0.268

0.295

0.164

0.165

0.104

0.130

1.275

1.301

1.281

0.5

100

0.001

0.257

0.269

0.157

0.154

0.100

0.116

1.260

1.272

1.249

0.5

200

0.01

0.167

0.187

0.097

0.101

0.070

0.086

1.173

1.192

1.200

0.5

200

0.001

0.158

0.165

0.094

0.093

0.064

0.072

1.158

1.164

1.197

0.5

400

0.01

0.114

0.129

0.065

0.069

0.049

0.060

1.108

1.122

1.128

0.5

400

0.001

0.114

0.119

0.069

0.069

0.046

0.049

1.110

1.115

1.115

0.5

1000

0.01

0.063

0.072

0.035

0.038

0.028

0.034

1.071

1.081

1.072

0.5

1000

0.001

0.061

0.064

0.035

0.036

0.026

0.028

1.074

1.078

1.067

0.2

100

0.01

0.529

0.569

0.330

0.315

0.199

0.254

1.536

1.576

1.624

0.2

100

0.001

0.510

0.523

0.317

0.296

0.193

0.228

1.511

1.525

1.572

0.2

200

0.01

0.317

0.351

0.183

0.184

0.134

0.167

1.324

1.357

1.389

0.2

200

0.001

0.300

0.313

0.175

0.169

0.125

0.144

1.301

1.314

1.364

0.2

400

0.01

0.206

0.235

0.115

0.121

0.090

0.114

1.200

1.229

1.242

0.2

400

0.001

0.208

0.218

0.121

0.120

0.088

0.098

1.204

1.214

1.227

0.2

1000

0.01

0.115

0.134

0.064

0.069

0.051

0.065

1.123

1.142

1.137

0.2

1000

0.001

0.109

0.116

0.060

0.062

0.049

0.054

1.122

1.130

1.126

0.1

100

0.01

0.839

0.888

0.546

0.506

0.293

0.381

1.843

1.892

1.996

0.1

100

0.001

0.896

0.884

0.599

0.530

0.297

0.354

1.902

1.889

2.000

0.1

200

0.01

0.478

0.536

0.280

0.279

0.198

0.257

1.483

1.541

1.618

0.1

200

0.001

0.468

0.485

0.277

0.262

0.190

0.224

1.470

1.487

1.555

0.1

400

0.01

0.319

0.360

0.184

0.189

0.135

0.171

1.314

1.356

1.385

0.1

400

0.001

0.303

0.314

0.179

0.173

0.124

0.141

1.305

1.317

1.354

0.1

1000

0.01

0.169

0.203

0.094

0.105

0.075

0.098

1.183

1.217

1.219

0.1

1000

0.001

0.172

0.184

0.099

0.101

0.073

0.083

1.172

1.184

1.205


TABLE A2. Results of simulations assessing the predictive performance of aggregated boosted trees (ABT) and gradient boosting (BT) for classification. ABT and BT were indistinguishable in accuracy and RF was less accurate than both types of boosting. The misclassification error of ABT was marginally but consistently better than BT.

N
Error
Accuracy
Bias
Precision
Misclassification error
   

ABT

BT

ABT

BT

ABT

BT

ABT

BT

100

High

0.056

0.057

0.042

0.042

0.014

0.015

0.272

0.288

 

Low

0.026

0.026

0.018

0.018

0.007

0.008

0.110

0.115

1000

High

0.011

0.011

0.007

0.007

0.004

0.004

0.182

0.184

 

Low

0.005

0.006

0.003

0.003

0.002

0.002

0.043

0.044



[Back to E088-015]