How to calculate statistical significance for accuracy from two model?
I have two models (model1,model2) which are run on a test dataset of5,800instances,So I show each data point to each model, and they generate a response.If the response from the model matches with the expected outcome, then I count it as it as “1”, else it would be “0”. So itsbinaryoutcome.Accuracy frommodel1is 68%.Accuracy frommodel2is 75%.But I was asked that the improvement with model2 compared to model1 is not statistically significant.I was further asked to report a confidence interval for a two-sample difference in proportion tests.Also what is the statistical significance of my result?I don’t understand this query.I ran inference with all the data points, and it is not like I only tested for a proportion of the test dataset.Please note I have read about confidence interval, Paired Samples T-Test from different blogs, Khan academy, and watched many youtube videos.But I cant figure this out. All the videos apply these techniques for numeric values, and not for a binary metric like the above.As an example, in this tutorialPaired Samples T-Test (How to calculate and interpret) - YouTubethey applied this for weight loss. As a result they were able to getmean,standard deviation. But for my case its a binary outcome i.e.0or1.So how to calculate statistical significance from the accuracy from two models?Can anyone please help me?
2022-11-22T13:11:42Z
You can use t-testlook at thisrasbt.github.iocombined_ftest_5x2cv: 5x2cv combined *F* test for classifier comparisons -...A library consisting of useful tools and extensions for the day-to-day data science tasks.