Permutation Testing for Machine Learning Model Validation using Sklearn

Have you ever had a model perform slightly better than chance and wondered if this was significant? Here I'll show you one way of answering this question.

Permutation testing is a great tool to use in machine learning in order to express if a model performance is greater than chance. Even though the theoretical boundary for chance is 50% for a two balanced class problem, when you are dealing with a small dataset it might not be the case. If you get 50.5%, 51, or even up to 60% accuracy you might still have a classifier that is performing worst than chance for your given dataset.

"An increasingly common statistical tool for constructing sampling distributions is the permutation test (or sometimes called a randomization test). Like bootstrapping, a permutation test builds - rather than assumes - sampling distribution (called the “permutation distribution”) by resampling the observed data. Specifically, we can “shuffle” or permute the observed data (e.g., by assigning different outcome values to each observation from among the set of actually observed outcomes). Unlike bootstrapping, we do this without replacement.

Permutation tests are particularly relevant in experimental studies, where we are often interested in the sharp null hypothesis of no difference between treatment groups. In these situations, the permutation test perfectly represents our process of inference because our null hypothesis is that the two treatment groups do not differ on the outcome (i.e., that the outcome is observed independently of treatment assignment). When we permute the outcome values during the test, we therefore see all of the possible alternative treatment assignments we could have had and where the mean-difference in our observed data falls relative to all of the differences we could have seen if the outcome was independent of treatment assignment. While a permutation test requires that we see all possible permutations of the data (which can become quite large), we can easily conduct “approximate permutation tests” by simply conducting a vary large number of resamples. That process should, in expectation, approximate the permutation distribution."

An excellent paper on the subject in my field is "Exceeding Chance Level by Chance: The Caveat of Theoretical Chance Levels in Brain Signal Classification and Statistical Assessment of Decoding Accuracy " from Dr. Etienne Combrisson and Dr. Karim Jerbi.

----

----
Follow Me Online Here:

___

Have a great week! 👋

[Mohamm-ed]

Awesome video thanks for sharing. Could you please make the pain detection video available for us it says not public. Thanks