Tree-Based Methods


It might be a overfit if we have a huge tree since we have fitted the model so well to the training data set. We need a termination condition, or we simple build a large tree then prune it to a smaller one.

Cost complexity pruning: add a punishment to the cost function

\sum_{t} \sum_i (y_i - \tilde y_i)^2 + \alpha t

where \alpha can be determined from cross validation.


  1. Gini index: purity
  2. Cross entropy

Tree vs Linear Regression

  1. Trees are box based


Bootstrap aggregation

For categorical

  1. Majority voting: most frequent categorical predictions
  2. Average

References and Notes

© 2018, Lei Ma| GitHub| Statistical Mechanics Notebook | Index | Page Source| changelog| Created with Sphinx