By John Mount
We have been recently working on and presenting on nested modeling issues. These are situations where the output of one trained machine learning model is part of the input of a later model or procedure. I am now of the opinion that correct treatment of nested models is one of the biggest opportunities for improvement in data science practice. Nested models can be more powerful than non-nested, but are easy to get wrong.
At first glance nested models seem like they should be anathema. Using data to build a model and then applying the model or transform to that same data breaks the exchangeability that statistical machine learning depends on for correct behavior. It leads to overfit. The overfit can be big (where you have a chance to notice it) or small (where you miss it, but have unknowingly have somewhat inferior models). However when one looks further we see such nested procedures are already common statistical practice:
- Using training data to build a principal components projection.
- Stacking or super-learning (for a good intro see the talks and writings of Dr. Erin Ledell).
- Variable selection.
- Dimension reduction.
- Variable transform/centering (such as
carret::preProcess()). - Our own y-aware data preparation.
- Deep models (such as multi-layer neural nets).
- Estimation of Bayesian hyper-parameters.
Our point is: the above procedures are useful, but they are strictly correct only when a disjoint set of calibration data is used for the preparation design (and then never re-used in training, test, or later application). The strictness is taught and remembered for the marquee steps (such as model fitting and evaluation), and sometimes forgotten for the “safe steps” (such as principal components determination).
In the age of “big data” the statistical inefficiency of losing some data is far less than the statistical inefficiency of breaking your exchangeability. The recommended experimental design is similar to the Cal/Train/Test split taught in “The Elements …read more
Source:: r-bloggers.com