Evaluating model proficieny #
Devloping a machine learning model is one half of the coin, the other half however is measure the performance of the model and improving its efficiency. There are multidude ways of measuring the performance of model, most of them are generic in nature and can be used across mutliple models, and some are specifically designed towards specific learnging algorithms.
Cross Validation #
- Statistical method to evalaute the model
- Lower bias
k-fold Cross Validation #
- Evaluating models with limited set of data
Pseudocode,
- Split the data set into k parts
- Iterate through the parts and take one part as hold/test data set and the rest as train data set
- Evaluate the model for each split and record the values
- Repeat until all parts are taken as test/hold
Optimal k value #
- Typically 5 or 10
- Representative of the broader population
- k=n, each data sample is held and the rest is used for training (LOOCV)
Variations, #
Test/Train Split Split the data into 2 parts. One is for testing and the other is for training Leave one out Cross Validation (LOOCV) each sample is held and everthing else is used to train Stratified Splitting based on criteria; each fold has same proportion of observation Repeated Usual method of splitting into n folds