T-Test, F-Test and P-value

Two very important tests in statistical analysis are the t-test and the f-test. However, some confusion may arise for a new user as to the difference between the two tests. In this post I will try and present the difference between the two tests and when each should be used.

But before we understand the test, let’s understand what a p-value is. The p-value is the probability of getting results as extreme as the observed values under null hypothesis. For example, after performing a t-test you find out that the p-value is 0.06. Does this mean that the null hypothesis can be rejected? Suppose you decide that due to random errors, even if the null hypothesis is true, 5 out of 100 experiments would inevitable fail the null hypothesis and you can live with that. However, in our experiment the p-value is 0.06, which says that ‘by looking at your data, I think that 6 out of 100 times you would tend to reject the null hypothesis even if it is true’. You are happy with that statement, you were happy if it fails five times, but the data says that six failures are expected to occur due to chance.
For practical purposes, reject a null hypothesis if the p-value is less than alpha (generally 5% or 0.05)

T-test: The t-test is used to find out if the means between two populations is significantly different.
Characteristics of the test are;
1) The test statistic follows a t distribution under null hypothesis.
2) The test can be used to find if the mean of a population is different from a known mean.
3) The test can be used to find out if the means of two samples are significantly different. Note that the two populations need to follow the normal distribution. Also the variances of the two populations need to be equal if sample size is less than 30.
4) The test can be used to find out if the difference between values of a single variable measured at different times is zero.
5) The test can be used to find out if the regression line has a slope different from zero.
6) Paired vs un-paired: A test of type 3 is a paired test. The samples are independent. A test of type 4 is an unpaired test. In many cases of unpaired data, it is the same variable undergoing repeated observations. For example, measurements taken before and after an experiment.
7) The questions that need to be answered before using the t-test are: is it a single population or multiple populations, are the sample sizes equal, are the variances equal, and is it a paired or un-paired test.

F-test: F-test is used to find out if the variances between the two populations are significantly different.
Characteristics of an F-test are:
1) The test statistic has an F distribution under null hypothesis. I.e. the ratio of variances follows an F distribution.
2) F-test can be used to find out if the means of multiple populations having same standard deviation differ significantly from each other. (ANOVA)
3) F-test can be used to find out if the data fits into a regression model obtained using least square analysis. Here we compare is the mean square due to error is significantly different from the mean square due to regression.
4) The test can be a two tailed test or a one tailed test.
5) F-test for ANOVA for two variables is equivalent to performing the t-test. Also the relation is given by F=t squared.
6) For ANOVA the F test is the measure of ratio of variance between groups and variance with the sample groups.

1 thought on “T-Test, F-Test and P-value”

Leave a Comment