UPENN | ML | Cross validation

Question

Suppose you have picked the parameter \( \theta \) for a model using 10-fold cross-validation. The best way to pick a final model to use and estimate its error is to

(a) pick any of the 10 models you built for your model; use its error estimate on the held-out data

(b) pick any of the 10 models you built for your model; use the average CV error for the 10 models as its error estimate

(c) average all of the 10 models you got; use the average CV error as its error estimate

(d) average all of the 10 models you got; use the error the combined model gives on the full training set

(e) train a new model on the full data set, using the \( \theta \) you found; use the average CV error as its error estimate

Answer 1

The best way to pick a final model and estimate its error after performing 10-fold cross-validation is:

(c) average all of the 10 models you got; use the average CV error as its error estimate

The reason for this is that 10-fold cross-validation is performed to get a more reliable estimate of the model's performance by training and evaluating the model on different subsets of the data. Taking the average of the 10 models helps to reduce the variability in the performance estimates and provides a more stable and representative measure of the model's generalization performance.

Options (a) and (b) suggest picking any single model from the 10-fold cross-validation, which may be influenced by the random splitting of the data and might not provide a robust estimate. Options (d) and (e) involve training on the full dataset, which might lead to overfitting to the training data and may not reflect the model's ability to generalize to new, unseen data. Therefore, (c) is generally the recommended approach for model selection and error estimation after cross-validation.

Answer 2

 train a new model on the full data set, using the � you found; use the average CV error as its error estimate