This lecture note is still being edited: the text may contain typos, inaccuracies, and locally unpolished passages. If you find something, please let the author know (contact information on the course page).
Let \(\overrightarrow {X} = \left(X_{1}, \ldots , X_{n}\right)^T\) and \(\overrightarrow {Y}=\left(Y_{1}, \ldots , Y_{n}\right)^T\) be two paired samples from univariate distributions. That is, we have a sample from a 2-dimensional distribution \(\operatorname {Law}\left(X_1, Y_1\right)\):
\[
\begin{pmatrix} \overrightarrow {X} & \overrightarrow {Y} \end{pmatrix} = \begin{pmatrix} X_1 & Y_1 \\ X_2 & Y_2 \\ \vdots & \vdots \\ X_n & Y_n \end{pmatrix}
\]
We will be interested in whether \(\overrightarrow {X}\), \(\overrightarrow {Y}\) are samples from independent distributions: \(\operatorname {Law}\left(X_1, Y_1\right) = \operatorname {Law}\left(X_1\right)\cdot \operatorname {Law}\left(Y_1\right)\). In other words, we put forward the hypothesis
\[
H_{0}: F_{X, Y}(s, t)=F_{X}(s) F_{Y}(t) \text{ for all } s, t,
\]
where \(F_{X}\) is the distribution function of \(X_{1}, F_{Y}\) is the distribution function of \(Y_{1}\), and \(F_{X, Y}\) is the distribution function of the vector \(\left(X_{1}, Y_{1}\right)\). The alternative to the hypothesis \(H_{0}\) is the hypothesis \(H_{1}\) that the samples \(\overrightarrow {X}\) and \(\overrightarrow {Y}\) are dependent, i.e.
\[
H_{1}: F_{X, Y}(s, t) \neq F_{X}(s) F_{Y}(t) \text{ for some } s, t .
\]
Checking the independence property in general is problematic, since it must hold for all pairs \(s, t\); it is much easier to confirm that the samples are dependent via a correlation coefficient. Recall that if \(X_1\) and \(Y_1\) have finite second moments, then
\[
\operatorname {Corr}\left[ X_1, Y_1 \right] = \frac{\operatorname {Cov}\left[ X_1, Y_1 \right]}{\sqrt{\operatorname {Var}\left[X_1\right]} \sqrt{\operatorname {Var}\left[Y_1\right]}}.
\]
By the Cauchy-Bunyakovsky-Schwarz inequality, \(\left|\operatorname {Corr}\left[ X_1, Y_1 \right]\right| \leq 1\), and if \(X_1\), \(Y_1\) are independent, then \(\operatorname {Corr}\left[ X_1, Y_1 \right] = 0\).
For samples from unknown distributions, the true correlation coefficient \(\operatorname {Corr}\left[ X_1, Y_1 \right]\) cannot be computed. Instead, sample correlation coefficients are used. The general decision rule in our problem is as follows: if the correlation coefficient lies sufficiently far from zero, we reject the hypothesis of independence.
Correlation analysis (namely, the analysis of dependencies between observations) plays an important role in regression problems, as well as in time series theory, in particular in the analysis of stationary sequences and autoregressive models.
Definition 1 For a paired sample \(\left(\overrightarrow {X} \; \overrightarrow {Y}\right)\), a statistic \(\widehat{r}\) is called a sample correlation coefficient if
\(\left|r\right| \leq 1\), and the values \(1, -1\) can be attained by the statistic \(\widehat{r}\);
Under the hypothesis \(H_0\) of independence of the sample distributions, we have
\[
\mathbb {E}\left[\widehat{r}\right] = 0, \qquad \operatorname {Var}\left[\widehat{r}\right] \xrightarrow [n \to \infty ]{} 0.
\]
Note that, under the remaining conditions, \(\operatorname {Var}\left[\widehat{r}\right] \xrightarrow [n \to \infty ]{} 0\) is equivalent to \(\widehat{r} \xrightarrow [n \to \infty ]{\mathbb {P}, d} 0\).
In this chapter we consider the three most commonly used sample correlation coefficients: the Pearson, Spearman, and Kendall correlation coefficients.
Pearson Correlation Coefficient
Definition and properties
Let \(X_{1}, \ldots , X_{n}\) and \(Y_{1}, \ldots , Y_{n}\) be two samples with finite second moments: \(\mathbb {E}\left[X_1^2\right], \mathbb {E}\left[Y_1^2\right] < \infty\). To estimate the correlation, we need to estimate the variances \(\sigma_X^2\), \(\sigma_Y^2\) of each sample, as well as their covariance. We already constructed estimators for the variances back in the first chapter:
\[
\widehat{\sigma _X^2} = s^2_X = \overline{\overrightarrow {X}^2} - \overline{\overrightarrow {X}}^2 = \frac{1}{n}\sum _{i=1}^n\left(X_i - \overline{\overrightarrow {X}}\right)^2, \qquad \widehat{\sigma _Y^2} = s^2_Y = \overline{\overrightarrow {Y}^2} - \overline{\overrightarrow {Y}}^2 = \frac{1}{n}\sum _{i=1}^n\left(Y_i - \overline{\overrightarrow {Y}}\right)^2
\]
Let us estimate the covariance. Recall that
\[
\operatorname {Cov}\left[ X_1, Y_1 \right] = \mathbb {E}\left[\left(X_1 - \mathbb {E}\left[X_1\right]\right)\left(Y_1 - \mathbb {E}\left[Y_1\right]\right)\right].
\]
By analogy with the variance, the covariance can be estimated as follows:
\[
\widehat{\operatorname {Cov}\left[ X_1, Y_1 \right]} = \frac{1}{n}\sum _{i=1}^n\left(X_i - \overline{\overrightarrow {X}}\right)\left(Y_i - \overline{\overrightarrow {Y}}\right)
\]
Prove that \[
\widehat{\operatorname {Cov}\left[ X_1, Y_1 \right]} = \frac{1}{n}\sum _{i=1}^nX_iY_i - \left(\frac{1}{n}\sum _{i=1}^n X_i\right) \cdot \left(\frac{1}{n}\sum _{i=1}^n Y_i\right) = \overline{\overrightarrow {X}\overrightarrow {Y}} - \overline{\overrightarrow {X}} \; \overline{\overrightarrow {Y}}
\]
Definition 2 The Pearson correlation coefficient, or the ordinary (sample) correlation coefficient, is the following statistic: \[
\widehat{\rho }(\overrightarrow {X}, \overrightarrow {Y}) := \frac{\widehat{\operatorname {Cov}\left[ X_1, Y_1 \right]}}{\sqrt{\widehat{\sigma _X^2}}\sqrt{\widehat{\sigma _Y^2}}} = \frac{\sum _{i=1}^{n}\left(X_{i}-\overline{\overrightarrow {X}}\right)\left(Y_{i}-\overline{\overrightarrow {Y}}\right)}{\sqrt{\sum _{i=1}^{n}\left(X_{i}-\overline{\overrightarrow {X}}\right)^{2} \sum _{i=1}^{n}\left(Y_{i}-\overline{\overrightarrow {Y}}\right)^{2}}}.
\] Two more equivalent expressions are useful. The geometric one β the cosine of the angle between the centered samples: \[
\widehat{\rho }(\overrightarrow {X}, \overrightarrow {Y}) = \frac{ \left\langle \overrightarrow {X} - \overline{\overrightarrow {X}}\cdot \overrightarrow {1}, \overrightarrow {Y} - \overline{\overrightarrow {Y}}\cdot \overrightarrow {1} \right\rangle }{\left\| \overrightarrow {X} - \overline{\overrightarrow {X}}\cdot \overrightarrow {1}\right\| \left\| \overrightarrow {Y} - \overline{\overrightarrow {Y}}\cdot \overrightarrow {1}\right\| },
\] and the expression via sample means: \[
\widehat{\rho }(\overrightarrow {X}, \overrightarrow {Y}) = \frac{\overline{\overrightarrow {X}\overrightarrow {Y}} - \overline{\overrightarrow {X}} \; \overline{\overrightarrow {Y}}}{\sqrt{\overline{\overrightarrow {X}^2} - \overline{\overrightarrow {X}}^2}\sqrt{\overline{\overrightarrow {Y}^2} - \overline{\overrightarrow {Y}}^2}}.
\]
Scatter plots for 7 samples with different Pearson correlation coefficients \(r\): from an exact positive (\(r=1\)) to an exact negative (\(r=-1\)) linear relationship, passing through complete absence of a relationship (\(r=0\)).
Pearson coefficient for samples of linearly dependent quantities of the form \((X_1, Y_1)\): \(X_1 \overset{\text{a.s.}}{=} \theta_0 + \theta_1 Y_1\). The coefficient equals \(1\) if \(\theta_1 > 0\), equals \(-1\) if \(\theta_1 < 0\), and is undefined if \(\theta_1 = 0\) (in which case \(\operatorname{Var}[X_1]=0\)).
Before turning to the properties of \(\widehat{\rho }\), it is useful to train your eye: to be able to estimate the sign and the approximate magnitude of the correlation coefficient from the point cloud of a scatter plot alone.
A point cloud and the sample Pearson correlation coefficient \(\hat\rho\): the intuition behind the formula β how βstretchedβ the cloud is along a line, and in which direction. Move the target correlation slider \(\rho\) to see how the shape of the cloud changes; the button below regenerates the points with the same \(\rho\).
Theorem 1 The sample Pearson correlation converges to the true correlation: \[
\widehat{\rho } \xrightarrow [n \to \infty ]{\mathbb {P}} \operatorname {Corr}\left[ X_1, Y_1 \right]
\]
Prove the property \(\widehat{\rho } \xrightarrow [n \to \infty ]{\mathbb {P}} \operatorname {Corr}\left[ X_1, Y_1 \right]\) of the Pearson correlation coefficient.
The proof consists of successively applying the strong law of large numbers (for independent identically distributed random variables with finite expectation) and the theorem on inheritance of convergence.
By the strong law of large numbers,
\[
\frac{1}{n}\sum _{i=1}^n \left(X_i - \mathbb {E}\left[X_i\right]\right)\left(Y_i - \mathbb {E}\left[Y_i\right]\right) \xrightarrow [n\to \infty ]{\mathbb {P}} \mathbb {E}\left[\left(X_1-\mathbb {E}\left[X_1\right]\right)\left(Y_1 - \mathbb {E}\left[Y_1\right]\right)\right]
\]
and also \(\overline{\overrightarrow {X}} \stackrel{\mathrm{P}}{\rightarrow } \mathrm{E} X_{1}\) and \(\overline{\overrightarrow {Y}} \stackrel{\mathrm{P}}{\rightarrow } \mathrm{E} Y_{1}\) (of course, even almost sure convergence holds). Consequently, convergence of the vectors \((\overline{\overrightarrow {X}}, \overline{\overrightarrow {Y}}) \stackrel{\mathrm{P}}{\rightarrow } \left(\mathbb {E}\left[X_1\right], \mathbb {E}\left[Y_1\right]\right)\) also holds. Hence, by the theorem on inheritance of convergence,
\[
\overline{\overrightarrow {X}}\, \mathbb {E}\left[Y_1\right] - \mathbb {E}\left[X_1\right]\mathbb {E}\left[Y_1\right] + \overline{\overrightarrow {Y}}\, \mathbb {E}\left[X_1\right]-\overline{\overrightarrow {X}}\cdot \overline{\overrightarrow {Y}} \xrightarrow [n\to \infty ]{\mathbb {P}} 0,
\]
and direct expansion of the brackets shows that this expression is exactly equal to the difference
\[
\frac{1}{n}\sum _{i=1}^{n}\left(X_{i}-\overline{\overrightarrow {X}}\right)\left(Y_{i}-\overline{\overrightarrow {Y}}\right) - \frac{1}{n}\sum _{i=1}^{n}\left(X_{i}-\mathrm{E} X_{i}\right)\left(Y_{i}-\mathrm{E} Y_{i}\right).
\]
Hence,
\[
\frac{1}{n} \sum _{i=1}^{n}\left(X_{i}-\overline{\overrightarrow {X}}\right)\left(Y_{i}-\overline{\overrightarrow {Y}}\right) \stackrel{\mathrm{P}}{\rightarrow } \mathrm{E}\left(X_{1}-\mathrm{E} X_{1}\right)\left(Y_{1}-\mathrm{E} Y_{1}\right)=\operatorname {cov}\left(X_{1}, Y_{1}\right) .
\]
As we proved in previous chapters, the sample variance \(s_{X}^{2}=\frac{1}{n} \sum_{i=1}^{n}\left(X_{i}-\overline{\overrightarrow {X}}\right)^{2}\) is a consistent estimator of \(\mathrm{D} X_{1}\), and an analogous statement holds for the sample \(Y_{1}, \ldots , Y_{n}\). Then, by the theorem on inheritance of convergence,
\[
\widehat{\rho }=\frac{\sum _{i=1}^{n}\left(X_{i}-\overline{\overrightarrow {X}}\right)\left(Y_{i}-\overline{\overrightarrow {Y}}\right)}{\sqrt{\sum _{i=1}^{n}\left(X_{i}-\overline{\overrightarrow {X}}\right)^{2} \sum _{i=1}^{n}\left(Y_{i}-\overline{\overrightarrow {Y}}\right)^{2}}} \stackrel{\mathrm{P}}{\longrightarrow } \frac{\operatorname {cov}\left(X_{1}, Y_{1}\right)}{\sqrt{\mathrm{D} X_{1} \mathrm{D} Y_{1}}}
\]
Theorem 2 The Pearson coefficient is indeed a sample correlation coefficient in the sense of Definition Definition 1.
Prove the first property from Definition Definition 1 for the Pearson coefficient.
Theorem 3 (Relation to linear regression) If we have a linear regression model with an intercept
\[
\overrightarrow {X} = \mathbf{Z}\overrightarrow {\theta } + \overrightarrow {\varepsilon }, \qquad \overrightarrow {Z}_{\cdot 0} = \overrightarrow {1}, \quad \dim (\mathbf{Z}) = n \times (k+1), \quad \dim (\overrightarrow {\theta }) = k + 1,
\]
then its coefficient of determination equals the square of the sample correlation between the dependent variable \(\overrightarrow {X}\) and the fitted model \(\widehat{\overrightarrow {X}}\):
\[
R^2 = \widehat{\rho }^2\left(\overrightarrow {X}, \widehat{\overrightarrow {X}}\right)
\]
Moreover, if \(k=1\), i.e. we have a linear regression model with an intercept and a single regressor \(\overrightarrow {Y}:= \overrightarrow {Z}_{\cdot 1}\):
\[
\overrightarrow {X} = \theta _0 \cdot \overrightarrow {1} + \theta _1 \overrightarrow {Y} + \overrightarrow {\varepsilon },
\]
then the coefficient of determination equals the square of the sample correlation between the dependent variable and the regressor:
\[
R^2 = \widehat{\rho }^2\left(\overrightarrow {X}, \overrightarrow {Y}\right). \qquad \text{Moreover, } \widehat{\theta }_1 = \widehat{\rho }\sqrt{\frac{\widehat{\sigma _X^2}}{\widehat{\sigma _Y^2}}}.
\]
Prove the theorem.
Recall that if the regression has an intercept, then the coefficient of determination is defined as
\[
R^2 = \frac{\operatorname {ESS}}{\operatorname {TSS}} = \frac{\left\| \widehat{\overrightarrow {X}} - \overline{\overrightarrow {X}}\cdot \overrightarrow {1}\right\| ^2}{\left\| \overrightarrow {X} - \overline{\overrightarrow {X}}\cdot \overrightarrow {1}\right\| ^2} = \cos ^2(\varphi )
\]
where \(\varphi\) is the angle between the vectors \(\overrightarrow {X} - \overline{\overrightarrow {X}} \cdot \overrightarrow {1}\) and \(\widehat{\overrightarrow {X}} - \overline{\overrightarrow {X}} \cdot \overrightarrow {1}\). It is easy to show that \(\overline{\widehat{\overrightarrow {X}}} = \overline{\overrightarrow {X}}\). We have
\[
\cos (\varphi ) = \frac{ \left\langle \overrightarrow {X} - \overline{\overrightarrow {X}} \cdot \overrightarrow {1}, \widehat{\overrightarrow {X}} - \overline{\overrightarrow {X}} \cdot \overrightarrow {1} \right\rangle }{\left\| \overrightarrow {X} - \overline{\overrightarrow {X}} \cdot \overrightarrow {1}\right\| \left\| \widehat{\overrightarrow {X}} - \overline{\overrightarrow {X}} \cdot \overrightarrow {1}\right\| },
\]
and, replacing \(\overline{\overrightarrow {X}}\) in the second argument by the equal quantity \(\overline{\widehat{\overrightarrow {X}}}\), we recognize the definition of the Pearson correlation coefficient:
\[
\cos (\varphi ) = \frac{ \left\langle \overrightarrow {X} - \overline{\overrightarrow {X}} \cdot \overrightarrow {1}, \widehat{\overrightarrow {X}} - \overline{\widehat{\overrightarrow {X}}} \cdot \overrightarrow {1} \right\rangle }{\left\| \overrightarrow {X} - \overline{\overrightarrow {X}} \cdot \overrightarrow {1}\right\| \left\| \widehat{\overrightarrow {X}} - \overline{\widehat{\overrightarrow {X}}} \cdot \overrightarrow {1}\right\| } = \widehat{\rho }(\overrightarrow {X}, \widehat{\overrightarrow {X}}).
\]
Next, let \(k=1\). Let us prove that in this case the vectors \(\widehat{\overrightarrow {X}} - \overline{\overrightarrow {X}}\cdot \overrightarrow {1}\) and \(\overrightarrow {Y} - \overline{\overrightarrow {Y}}\cdot \overrightarrow {1}\) are collinear β this will give the required result, since both sides of the equality being proved do not depend on the sign of the cosine. We have \(\dim (L) = 2\), where \(L = \operatorname {span}(\overrightarrow {1}, \overrightarrow {Y})\), and both vectors under consideration lie in \(L\) and are perpendicular to \(\overrightarrow {1}\). Hence they are collinear (pointing the same way when \(\widehat{\theta }_1 > 0\) and opposite ways when \(\widehat{\theta }_1 < 0\)).
Next, \(\mathbf{Z} = \left(\overrightarrow {1} \; \overrightarrow {Y}\right)\),
\[
\begin{align} \mathbf{Z}^T & = \begin{pmatrix} 1 & 1 & \ldots & 1 \\ Y_1 & Y_2 & \ldots & Y_n \end{pmatrix}, \quad \left(\mathbf{Z}^T \mathbf{Z}\right) = \begin{pmatrix} n & n\overline{\overrightarrow {Y}} \\ n \overline{\overrightarrow {Y}} & n\overline{\overrightarrow {Y}^2} \end{pmatrix}, \quad \left(\mathbf{Z}^T \mathbf{Z}\right)^{-1} = \frac{1}{n^2\overline{\overrightarrow {Y}^2} - n^2\overline{\overrightarrow {Y}}^2}\begin{pmatrix} n\overline{\overrightarrow {Y}^2} & -n\overline{\overrightarrow {Y}} \\ -n \overline{\overrightarrow {Y}} & n \end{pmatrix} \\ \widehat{\overrightarrow {\theta }} & = \left(\mathbf{Z}^T \mathbf{Z}\right)^{-1} \mathbf{Z}^T \overrightarrow {X} = \frac{1}{n^2\overline{\overrightarrow {Y}^2} - n^2\overline{\overrightarrow {Y}}^2}\begin{pmatrix} n\overline{\overrightarrow {Y}^2} & -n\overline{\overrightarrow {Y}} \\ -n \overline{\overrightarrow {Y}} & n \end{pmatrix} \begin{pmatrix} n\overline{\overrightarrow {X}} \\ n\overline{\overrightarrow {X}\overrightarrow {Y}} \end{pmatrix} = \frac{1}{n^2\overline{\overrightarrow {Y}^2} - n^2\overline{\overrightarrow {Y}}^2}\begin{pmatrix} n^2\overline{\overrightarrow {X}}\; \overline{\overrightarrow {Y}^2} - n^2\overline{\overrightarrow {X}\overrightarrow {Y}}\; \overline{\overrightarrow {Y}} \\ -n^2\overline{\overrightarrow {X}}\; \overline{\overrightarrow {Y}} + n^2\overline{\overrightarrow {X}\overrightarrow {Y}} \end{pmatrix} = \\ & = \frac{1}{\overline{\overrightarrow {Y}^2} - \overline{\overrightarrow {Y}}^2}\begin{pmatrix} \overline{\overrightarrow {X}}\; \overline{\overrightarrow {Y}^2} - \overline{\overrightarrow {X}\overrightarrow {Y}}\; \overline{\overrightarrow {Y}} \\ \overline{\overrightarrow {X}\overrightarrow {Y}} - \overline{\overrightarrow {X}} \cdot \overline{\overrightarrow {Y}} \end{pmatrix} \\ \widehat{\theta }_1 & = \frac{\overline{\overrightarrow {X}\overrightarrow {Y}} - \overline{\overrightarrow {X}} \cdot \overline{\overrightarrow {Y}} }{\overline{\overrightarrow {Y}^2} - \overline{\overrightarrow {Y}}^2} = \frac{\overline{\overrightarrow {X}\overrightarrow {Y}} - \overline{\overrightarrow {X}} \cdot \overline{\overrightarrow {Y}}}{\sqrt{\overline{\overrightarrow {Y}^2} - \overline{\overrightarrow {Y}}^2}\sqrt{\overline{\overrightarrow {X}^2} - \overline{\overrightarrow {X}}^2}} \cdot \sqrt{\frac{\overline{\overrightarrow {X}^2} - \overline{\overrightarrow {X}}^2}{\overline{\overrightarrow {Y}^2} - \overline{\overrightarrow {Y}}^2}} \end{align}
\]
Geometric interpretation of OLS with a single regressor: the vector \(X\) is projected onto the plane \(\operatorname{span}(\vec 1, Y)\); the projection \(\widehat X = X_L = \widehat\theta_0\vec 1 + \widehat\theta_1 Y\) is the point of this plane closest to \(X\), and the residual \(\widehat{\vect\varepsilon} = X - X_L\) is orthogonal to it. Numerical example: \(n=3\), \(Y=(1,2,4)\), \(X=(2.2,3.6,6.4)\). Interactive 3D model on geogebra.org: link.
Gaussian samples
Let \((\overrightarrow {X}, \overrightarrow {Y})\) be a sample from a 2-dimensional normal distribution:
\[
\begin{pmatrix} X_1 & Y_1 \end{pmatrix} \sim \mathscr {N}\left(\begin{pmatrix} \mu _X\\ \mu _Y \end{pmatrix}, \begin{pmatrix} \sigma _X^2 & \rho \cdot \sigma _X \sigma _Y \\ \rho \cdot \sigma _X \sigma _Y & \sigma _Y^2 \\ \end{pmatrix}\right),
\]
where \(\rho := \operatorname {Corr}\left[ X_1, Y_1 \right]\). Since we are working with a sample from a 2-dimensional Gaussian distribution, independence of the components is equivalent to their being uncorrelated, i.e. \(H_0 \iff \rho = 0\).
In this case there exists an exact test for testing the hypothesis of independence, i.e. a test in which the distribution of the statistic is known.
Theorem 4 If the hypothesis \(H_0\) of independence of \(X_1, Y_1\) holds, then \[
T=\frac{\hat{\rho } \sqrt{n-2}}{\sqrt{1-\hat{\rho }^{2}}} \sim T_{n-2}
\]
The level-\(\alpha\) test for checking the hypothesis \(H_{0}\) for normal samples is as follows: if \(T \notin \left(Q_{t_{n-2}}(\frac{\alpha }{2}), Q_{t_{n-2}}(1 - \frac{\alpha }{2})\right)\), where \(Q_{t_{n-2}}(\cdot )\) is the quantile function of Studentβs distribution with \(n-2\) degrees of freedom, then we reject the hypothesis \(H_{0}\) of independence.
Pearson correlation coefficient for 7 samples from 2-dimensional Gaussian distributions with different values of \(\rho\). The Pearson coefficient is especially effective at detecting dependence precisely for samples from a 2-dimensional normal distribution.
Prove Theorem Theorem 4.
We have already encountered Studentβs distribution in the chapter on Gaussian linear regression. Let us build from \((\overrightarrow {X}, \overrightarrow {Y})\) the linear regression model
\[
X_i = \theta _0 + \theta _1 Y_i + \varepsilon _i, \quad i = \overline{1,n}
\]
Let us arrange for \(\varepsilon_i\) to be independent of \(Y_1\). Since we are dealing with components of a Gaussian vector, it suffices to arrange for them to be uncorrelated. We have
\[
0 = \operatorname {Cov}\left[ Y_1, \varepsilon _1 \right] = \operatorname {Cov}\left[ Y_1, X_1 - \theta _0 - \theta _1 Y_1 \right] = \rho \cdot \sigma _X \sigma _Y - \theta _1 \sigma ^2_Y \; \Rightarrow \; \theta _1 = \rho \frac{\sigma _X}{\sigma _Y}
\]
Next, we require \(\mathbb {E}\left[\varepsilon_i\right] = 0\). We have
\[
0 = \mathbb {E}\left[\varepsilon _i\right] = \mathbb {E}\left[X_i - \theta _0 - \theta _1 Y_i\right] = \mu _X - \theta _0 - \theta _1 \mu _Y \; \Rightarrow \; \theta _0 = \mu _X - \theta _1\mu _Y = \mu _X - \rho \frac{\sigma _X}{\sigma _Y}\mu _Y
\]
Let us find the variance \(\sigma^2\) of the resulting error. Substituting \(\theta_1 = \rho \sigma_X/\sigma_Y\):
\[
\begin{align} \sigma ^2 & = \operatorname {Var}\left[\varepsilon _i\right] = \operatorname {Var}\left[X_1 - \theta _0 - \theta _1Y_1\right] = \sigma _X^2 - 2\theta _1 \rho \sigma _X \cdot \sigma _Y + \theta _1^2\sigma _Y^2 = \\ & = \sigma _X^2 - 2\rho \frac{\sigma _X}{\sigma _Y} \cdot \rho \sigma _X \sigma _Y + \rho ^2\frac{\sigma _X^2}{\sigma _Y^2}\sigma _Y^2 = \sigma _X^2 - 2 \rho ^2 \sigma ^2_X + \rho ^2 \sigma _X^2 = \sigma _X^2 \left(1 - \rho ^2\right) \end{align}
\]
For the given model, let us first find the auxiliary quantities entering the statistic \(T_{n-2}\). The element \(a_{1,1}\) of the matrix \(\left(\mathbf{Z}^T\mathbf{Z}\right)^{-1}\) corresponding to \(\theta_1\) (see the computation above):
\[
a_{1,1} = \frac{n}{n^2\overline{\overrightarrow {Y}^2} - n^2\overline{\overrightarrow {Y}}^2} = \frac{1}{n \widehat{\sigma _Y^2}}.
\]
Next, since the regression model contains \(k=1\) regressor and an intercept:
\[
\begin{align} \operatorname {TSS} & = \sum _{i=1}^n\left(X_i - \overline{\overrightarrow {X}}\right)^2 = n\widehat{\sigma ^2_X}, \\ (n-2)\widehat{\sigma ^2} & = \operatorname {RSS} = \operatorname {TSS} - \operatorname {ESS} = \operatorname {TSS}\left(1 - R^2\right) = \operatorname {TSS}\left(1 - \widehat{\rho }^2\right). \end{align}
\]
Substituting what we found into the statistic:
\[
\begin{align} T_{n - 2} & \sim \frac{\widehat{\theta }_1}{\sqrt{a_{1,1} \widehat{\sigma ^2}}} = \widehat{\rho } \sqrt{\frac{\widehat{\sigma _X^2}}{\widehat{\sigma _Y^2}}}\frac{1}{\sqrt{a_{1,1} \widehat{\sigma ^2}}} = \widehat{\rho } \sqrt{\frac{\widehat{\sigma _X^2}}{\widehat{\sigma _Y^2}}}\frac{\sqrt{n\widehat{\sigma _Y^2}} \sqrt{n-2}}{\sqrt{\operatorname {TSS}(1 - \widehat{\rho }^2)}} = \\ & = \widehat{\rho } \sqrt{\frac{\widehat{\sigma _X^2}}{\widehat{\sigma _Y^2}}}\frac{\sqrt{n\widehat{\sigma _Y^2}} \sqrt{n-2}}{\sqrt{n\widehat{\sigma _X^2}(1 - \widehat{\rho }^2)}} = \frac{\widehat{\rho } \sqrt{n-2}}{\sqrt{1 - \widehat{\rho }^2}} \end{align}
\]
Drawbacks
Since, for normal samples, being uncorrelated is equivalent to independence, the Pearson correlation coefficient is best suited to working with normal samples. Unfortunately, as with many parametric methods, using this correlation coefficient can lead to inadequate values of the statistic (meaning that the resulting statistical decisions are incorrect), since the correlation coefficient is not robust to outliers (when the sample contains far-lying data points, for example due to measurement errors). A well-known example (the so-called βAnscombeβs quartetβ) demonstrates how badly such methods of processing statistical data can βlie,β even when there is only a single outlier among 10 βordinaryβ results.
Anscombeβs quartet (F. Anscombe, 1973): in all four datasets the Pearson correlation coefficient equals \(0.816\), and the regression line is the same β yet the scatter plots are completely different. Dataset I is the ordinary linear case, where the Pearson coefficient works adequately. Dataset II shows that the Pearson coefficient does a poor job of detecting nonlinear dependencies. Datasets III and IV show that it is overly sensitive to outliers.
The Pearson correlation coefficient estimates the degree of linear dependence between variables. It can be close to zero even when a dependence between \(X_1\) and \(Y_1\) exists β it is simply nonlinear (and, typically, symmetric).
Rank Correlation Coefficients
Ranks of a sample
Suppose we have a sample \(X_{1}, \ldots , X_{n}\) from some continuous distribution, i.e. the sample elements do not coincide a.s. Let us order the sample elements in increasing order, i.e. construct the order statistics of the sample.
Definition 3 The numbers that the sample elements receive under such an ordering are called their ranks. Notation: \(\overrightarrow {R}^X\).
Assign ranks to a realization of a sample of 10 elements in the experiment \(\omega_0\): \[
\overrightarrow {X}(\omega _0) = \overrightarrow {x}_0 = \left(9.356, \frac{29}{5}, 2\pi , 3e, \frac{11}{2}, 5\sqrt{2}, 7\sin (1), 8.78, 4.51, \frac{e^\pi }{4}\right)^T
\]
We have \[
\begin{align} \overrightarrow {X}(\omega _0) & = \left(9.356, \frac{29}{5}, 2\pi , 3e, \frac{11}{2}, 5\sqrt{2}, 7\sin (1), 8.78, 4.51, \frac{e^\pi }{4}\right) \approx \\ & \approx \left(9.356, 5.800 , 6.283 , 8.155, 5.500 , 7.071 , 5.890 , 8.780, 4.510, 5.785\right), \\ \overrightarrow {R}^X(\omega _0) & = \left(\; \; 10, \hspace*{6mm} 4, \hspace*{6mm} 6, \hspace*{6mm} 8, \hspace*{6mm} 2, \hspace*{6mm} 7, \hspace*{6mm} 5, \hspace*{6mm} 9, \hspace*{6mm} 1, \hspace*{6mm} 3 \quad \right) \end{align}
\]
The basic property of ranks:
\[
\mathbb {P}\left(R^X_1 = r_1, \ldots , R^X_n = r_n\right) = \frac{1}{n !},
\]
where \(\left(r_{1}, \ldots , r_{n}\right)\) is an arbitrary permutation of the numbers \((1, \ldots , n)\), since for any \(i<j\) we have \(\mathbb {P}\left(X_i > X_j\right) = \mathbb {P}\left(X_j > X_i\right)\), because the sample components are independent and identically distributed. The distribution of the ranks does not depend on the distribution of the sample.
Show that
\(\overline{R^X} = \frac{n+1}{2}, \quad \overline{(R^X)^2} = \frac{(n+1)(2n+1)}{6}\);
\(\mathbb {E}\left[R^X_i\right] = \frac{n+1}{2}, \; \forall i = \overline{1,n}\);
\(\mathbb {E}\left[(R^X_i)^2\right] = \frac{(n+1)(2n+1)}{6}, \; \forall i = \overline{1,n}\);
\(\operatorname {Var}\left[R^X_i\right] = \frac{n^2 - 1}{12}, \; \forall i = \overline{1,n}\);
\(\mathbb {E}\left[R^X_i \cdot R^X_j\right] = \frac{(n+1)(3n+2)}{12} , \; \forall i \neq j\);
\(\operatorname {Cov}\left[ R^X_i, R^X_j \right] = -\frac{n+1}{12}, \; \forall i \neq j\).
Notice that none of the characteristics of the ranks listed in this problem depend on the distribution \(\operatorname {Law}\left(X_1\right)\) from which the sample was drawn.
Spearman Correlation Coefficient
Definition
As before, consider two samples \(X_{1}, \ldots , X_{n}\) with distribution function \(F_{X}\) and \(Y_{1}, \ldots , Y_{n}\) with distribution function \(F_{Y}\).
Using the results of Problem Problem 3, we have
\[
\begin{align} \overline{(R^X)^2} - \overline{R^X}^2 & = \frac{(n+1)(2n+1)}{6} - \frac{(n+1)^2}{4} = \frac{n^2-1}{12} \\ \rho _S & = \frac{12}{(n-1)(n+1)} \left(\overline{R^X R^Y} - \frac{(n+1)^2}{4}\right) = \\ & = \frac{12}{n(n-1)(n+1)} \left( \left\langle \overrightarrow {R}^X, \overrightarrow {R}^Y \right\rangle - \frac{n(n+1)^2}{4}\right) \end{align}
\]
The Pearson coefficient estimates the degree of linear dependence, while the Spearman coefficient estimates the degree of monotonicity of the dependence. Left: a strictly monotone but nonlinear relationship β the Spearman coefficient equals 1, while the Pearson coefficient is noticeably lower. Center: a weak, nearly absent relationship β both coefficients are close to each other and to zero. Right: an overall linear relationship spoiled by outliers β the Spearman coefficient is more robust to them than the Pearson coefficient.
Properties
Theorem 5 The following representation holds: \[
\rho _S = 1 - \frac{6}{n^3 - n}\sum _{i=1}^n \left(R^X_i - R^Y_i\right)^2
\tag{2}\] In practice, the Spearman coefficient is computed from this formula. It is much simpler than formula Equation 1.
Prove formula Equation 2.
Note that
\[
\sum _{i=1}^{n}\left(R^X_{i}-R^Y_{i}\right)^{2} = \left\langle \overrightarrow {R}^X-\overrightarrow {R}^Y, \overrightarrow {R}^X-\overrightarrow {R}^Y \right\rangle = 2 \frac{n(n+1)(2n+1)}{6} - 2 \left\langle \overrightarrow {R}^X, \overrightarrow {R}^Y \right\rangle ,
\]
whence
\[
\left\langle \overrightarrow {R}^X, \overrightarrow {R}^Y \right\rangle = \frac{n(n+1)(2n+1)}{6} - \frac{1}{2}\sum _{i=1}^n\left(R^X_i - R^Y_i\right)^2.
\]
Substitute this expression into the formula for \(\rho_S\) above, and separately simplify the term that is free of the sum:
\[
\frac{12}{n(n-1)(n+1)}\left(\frac{n(n+1)(2n+1)}{6} - \frac{n(n+1)^2}{4}\right) = \frac{12}{n(n-1)(n+1)} \cdot \frac{n(n+1)(n-1)}{12} = 1.
\]
Hence,
\[
\begin{align} \rho _{S} & = \frac{12}{n(n-1)(n+1)} \left( \left\langle \overrightarrow {R}^X, \overrightarrow {R}^Y \right\rangle - \frac{n(n+1)^2}{4}\right) = 1 - \frac{6}{n(n-1)(n+1)}\sum _{i=1}^n\left(R^X_i - R^Y_i\right)^2 \end{align}
\]
Theorem 6 The Spearman coefficient is indeed a sample correlation coefficient in the sense of Definition Definition 1. In particular, if the samples are drawn from independent distributions, then \[
\mathbb {E}\left[\rho _S\right] = 0, \qquad \operatorname {Var}\left[\rho _S\right] = \frac{1}{n-1}
\]
Prove that, if the hypothesis of independence of the samples holds, \(\mathbb {E}\left[\rho_S\right] = 0, \; \operatorname {Var}\left[\rho_{S}\right]=\frac{1}{n-1}\).
Testing the hypothesis of independence
To test the hypothesis \(H_0\) of independence of the distributions using the Spearman coefficient \(\rho_S\), we need to know the distribution of \(\rho_S\) under \(H_0\). Note that
\[
-1 \leq \rho _S \leq 1 \iff 0 \leq \frac{6}{n^3 - n} \sum _{i=1}^n \left(R_i^X - R_i^Y\right)^2 \leq 2 \iff 0 \leq \sum _{i=1}^n \left(R_i^X - R_i^Y\right)^2 \leq \frac{n^3 - n}{3} =: \psi _n \in \mathbb {N}.
\]
Moreover, obviously \(\sum_{i=1}^n \left(R_i^X - R_i^Y\right)^2 \in \mathbb {Z}_+\). Hence, for any \(n\), the coefficient \(\rho_S\) is a discrete random variable taking values in the set
\[
A_n = \left\{ 1 - 2\frac{k}{\psi _n}: \; k \in \left\{ 0, 1, \ldots , \psi _n\right\} \right\} = \left\{ -1, -1 + 2\frac{1}{\psi _n}, -1 + 2\frac{2}{\psi _n}, \ldots , -1 + 2\frac{\psi _n - 2}{\psi _n}, -1 + 2\frac{\psi _n - 1}{\psi _n}, 1 \right\}
\]
The distribution of \(\rho_S\) can be computed by finding the probability mass function on the set \(A_n\).
Since the distribution of \(\rho_S\) is discrete, computing the probability mass function of \(\rho_S\) is a combinatorial problem. It can be solved for any \(n\) simply by enumerating all possible values of the ranks \(\overrightarrow {R}^X, \overrightarrow {R}^Y\) by hand or by computer. However, as \(n\) grows, the number of cases to enumerate grows like \(n!\), i.e. very fast. In practice, statistical packages have exact values of the distribution of \(\rho_S\) only for \(n \leq N\), for some bound \(N\). For example, in the pspearman package of the R language the bound equals \(22\): \(N = 22\).
For \(n > N\), statistical packages use an approximation to the distribution of \(\rho_S\) when testing the hypothesis of independence. For example, from the CLT, for large \(n\) we have
\[
\operatorname {Law}\left(\frac{\rho _S}{\sqrt{\operatorname {Var}\left[\rho _S\right]}}\right) = \operatorname {Law}\left(\sqrt{n-1}\rho _S\right) \approx \mathscr {N}\left(0, 1\right).
\]
Or, from the properties of the Pearson coefficient for normal samples, we have
\[
\operatorname {Law}\left(\frac{\rho _S\sqrt{n-2}}{\sqrt{1 - \rho _S^2}}\right) \approx T_{n-2}
\]
A corrected statistic can also be used, equal to the half-sum of the two previous ones:
\[
\tilde{\rho }_S = \frac{\rho _S}{2}\left(\frac{\sqrt{n-2}}{\sqrt{1 - \rho _S^2}} + \sqrt{n-1}\right).
\]
The distribution function and quantile function of \(\tilde{\rho }_S\) are approximately equal to the half-sum of the corresponding functions for \(\mathscr {N}\left(0, 1\right)\) and for \(T_{n-2}\).
Other ways of computing an approximation to the distribution of \(\rho_S\) are also used. For example, the cor.test function of the R language uses algorithm AS 89 for \(n < 1290\). See (Best and Roberts 1975) for details.
In summary, a level-\(\alpha\) test for checking the hypothesis \(H_0\) of independence can look as follows:
if \(n \leq 22\), reject the hypothesis \(H_0\) if \(\rho_S \not\in \left(Q_n(\frac{\alpha }{2}), Q_n(1 - \frac{\alpha }{2})\right)\), where \(Q_n(\cdot )\) is the exact quantile function of the distribution of \(\rho_S\);
if \(n > 22\), and we use, for example, the corrected statistic \(\tilde{\rho }_S\), reject the hypothesis \(H_0\) if
\[
\tilde{\rho }_S \not\in \left(\frac{Q_{\mathscr {N}\left(0, 1\right)}(\alpha /2) + Q_{T_{n-2}}(\alpha /2)}{2}, \frac{Q_{\mathscr {N}\left(0, 1\right)}(1 - \alpha /2) + Q_{T_{n-2}}(1 - \alpha /2)}{2}\right)
\]
Find the distribution of \(\rho_S\) for \(n=3\), given that \(H_0\) holds. Find the quantile function \(Q_3(\cdot )\) of this distribution.
Define the statistic \(T_{i}\) as follows: if \(R^X_{k}=\) \(i\), then \(T_{i}=R^Y_{k}\). That is, \(T_i = R^Y_{(R^X)^{-1}(i)}\):
\[
\begin{pmatrix} R^X_1 & R^Y_1 \\ R^X_2 & R^Y_2 \\ \vdots & \vdots \\ R^X_n & R^Y_n \\ \end{pmatrix} = \begin{pmatrix} R^X_1 & R^Y_1 \\ \vdots & \vdots \\ 1 & T_1 \\ \vdots & \vdots \\ 2 & T_2 \\ \vdots & \vdots \\ R^X_n & R^Y_n \\ \end{pmatrix}
\]
The Spearman coefficient between \(\overrightarrow {R}^X, \overrightarrow {R}^Y\) equals the Spearman coefficient between \(\overrightarrow {T}, (1, \ldots , n)^T\). The collection of values \(\left\{ T_{i}\right\}_{i=1}^{n}\) is a permutation of the set \(\{ 1, \ldots , n\}\), and its distribution under the hypothesis \(H_0\) is, obviously, uniform. We have
\[
\begin{align} \rho _S = 1 - \frac{6}{n^{3}-n}\sum _{i=1}^n \left(i - T_i\right)^2 = 1 -\frac{1}{4}\sum _{i=1}^3 \left(i - T_i\right)^2 \end{align}
\]
Distribution of \(\sum_{i=1}^3 (i - T_i)^2\):
| Probability |
\(\sum_{i=1}^3 d_i^2\) |
| \(1/6\) |
0 |
| \(2/6\) |
2 |
| \(2/6\) |
6 |
| \(1/6\) |
8 |
Distribution of \(\rho_S\):
| Probability |
\(r_S\) |
| \(1/6\) |
\(-1\) |
| \(1/3\) |
\(-0.75\) |
| \(1/3\) |
\(0.75\) |
| \(1/6\) |
\(1\) |
Distribution function:
\[
F_3(r) = \begin{cases} 0, & r < -1 \\ 1/6, & r \in [-1, -0.5) \\ 1/2, & r \in [-0.5, 0.5) \\ 5/6, & r \in [0.5, 1) \\ 1, & r \geq 1 \end{cases}
\]
Quantile function:
\[
Q_3(p) = \inf \left\{ r \in \mathbb {R}: p \leq F_3(r)\right\} = \begin{cases} -1, & p \in (0, 1/6] \\ -0.5, & p \in (1/6, 1/2] \\ 0.5, & p \in (1/2, 5/6] \\ 1, & p \in (5/6, 1] \\ \end{cases}
\]
Find the distribution of \(\rho_S\) for \(n=4\), given that \(H_0\) holds. Find the quantile function \(Q_4(\cdot )\) of this distribution.
Using the statistics \(\left\{ T_{i}\right\}\) introduced in Example Example 6, prove that \[
\rho _{S} = 1-\frac{12}{n^{3}-n} \sum _{i<j}(j-i) \; \mathbb {1}_{T_{i}>T_{j}}.
\]
This problem is worth 3 points.
Kendall Correlation Coefficient
Another example of a correlation coefficient robust to outliers, constructed using rank-based methods, is the Kendall correlation coefficient.
Definitions
Definition 5 A pair of 2-dimensional observations \(\left(X_{i}, Y_{i}\right), \left(X_{j}, Y_{j}\right)\) is called concordant if \[
\left[ \begin{gathered} X_i < X_j, \; Y_i < Y_j \\ X_i > X_j, \; Y_i > Y_j \\ \end{gathered} \right.
\] Equivalently: \[
\operatorname {sign}\left(X_{i}-X_{j}\right) \operatorname {sign}\left(Y_{i}-Y_{j}\right)=1
\]
For the observation \((X_1, Y_1)\), the shaded zone contains observations concordant with it (the product of the signs of the deviations \(\operatorname{sign}(X_i-X_1)\operatorname{sign}(Y_i-Y_1)=1\)), while the unshaded zone contains discordant observations.
Let \(C\) be the number of concordant pairs of observations, and \(D\) the number of discordant pairs. The total number of pairs is
\[
C + D = \frac{n(n-1)}{2}
\]
As before, we assume there are no identical elements within the samples. Note that the difference \(C - D\) can range from \(-\frac{n(n-1)}{2}\) to \(\frac{n(n-1)}{2}\), and if the hypothesis \(H_0\) of independence holds, then \(C - D \approx 0\).
Definition 6 The Kendall correlation coefficient is the statistic \[
\tau =\frac{2}{n(n-1)} \left(C - D\right)= 1-\frac{4}{n(n-1)} D.
\]
Properties
Corollary 1 Like the Spearman coefficient, the Kendall coefficient is a rank coefficient, i.e. its value depends only on the ranks \(\overrightarrow {R}^X, \overrightarrow {R}^Y\) of the samples \(\overrightarrow {X}, \overrightarrow {Y}\).
Theorem 8 The Kendall coefficient is indeed a sample correlation coefficient in the sense of Definition Definition 1. In particular, if the samples are drawn from independent distributions, then \[
\mathbb {E}\left[\tau \right] = 0, \qquad \operatorname {Var}\left[\tau \right] = \frac{2(2n+5)}{9n(n-1)}
\]
Theorem 9 (Relation between the Spearman and Kendall coefficients) Under the hypothesis \(H_{0}\), the Spearman and Kendall correlation coefficients are strongly correlated: \(\operatorname {Corr}\left[ \rho_{S}, \tau \right] > 0.98\) for \(n>5\). However, the Spearman correlation coefficient is more sensitive to the number of discordant pairs.
Testing the hypothesis of independence
Like the Spearman coefficient, the Kendall coefficient is a discrete random variable. For small \(n \leq N\), the distribution function and quantile function of \(\tau\) can be computed fairly easily. For example, in the cor.test function of the R language, \(N = 50\). For \(n > N\), statistical packages use the normal approximation from the CLT: under \(H_0\), we have
\[
\operatorname {Law}\left(\frac{\tau }{\sqrt{\operatorname {Var}\left[\tau \right]}}\right) = \operatorname {Law}\left(\sqrt{\frac{9n(n-1)}{2(2n+5)}} \tau \right) \approx \mathscr {N}\left(0, 1\right)
\]
A level-\(\alpha\) test for checking the hypothesis \(H_0\) of independence can look as follows:
if \(n \leq 50\), reject the hypothesis \(H_0\) if \(\tau \not\in \left(Q_n(\frac{\alpha }{2}), Q_n(1 - \frac{\alpha }{2})\right)\), where \(Q_n(\cdot )\) is the exact quantile function of the distribution of \(\tau\);
if \(n > 50\), reject the hypothesis \(H_0\) if
\[
\sqrt{\frac{9n(n-1)}{2(2n+5)}}\tau \not\in \left(Q_{\mathscr {N}\left(0, 1\right)}(\alpha /2), Q_{\mathscr {N}\left(0, 1\right)}(1 - \alpha /2)\right)
\]
Find the distribution of \(\tau\) for \(n=4\), given that \(H_0\) holds. Find the quantile function \(Q_4(\cdot )\) of this distribution.
(*) Other Correlation Coefficients
Generalized correlation coefficient
Suppose we are given samples \(\overrightarrow {X}=\left(X_{1}, \ldots , X_{n}\right)^T, \overrightarrow {Y}=\left(Y_{1}, \ldots , Y_{n}\right)^T\). Consider an arbitrary antisymmetric function \(f: \mathbb {R}^{2} \to \mathbb {R}\):
\[
f(x_1, x_2) = -f(x_2, x_1).
\]
For any \(1 \leq i < j \leq n\) and for \(\overrightarrow {x} \in \mathbb {R}^{n}\), set \(c_{i j}(\overrightarrow {x}) := f\left(x_{i}, x_{j}\right)\).
Definition 7 Consider some \(n\)-dimensional characteristic \(\tilde{\overrightarrow {X}} = \tilde{\overrightarrow {X}}(\overrightarrow {X})\) of the sample \(\overrightarrow {X}\). For example, \(\tilde{\overrightarrow {X}}\) may be the sample \(\overrightarrow {X}\) itself or the vector of ranks \(\overrightarrow {R}^X\). Similarly for \(\overrightarrow {Y}\). The generalized correlation coefficient is the following statistic: \[
\widehat{r}_f(\overrightarrow {X}, \overrightarrow {Y}) = \frac{\sum _{i<j} c_{i j}(\tilde{\overrightarrow {X}}) c_{i j}(\tilde{\overrightarrow {Y}})}{\sqrt{\sum _{i<j} c^2_{i j}(\tilde{\overrightarrow {X}}) \sum _{i<j} c^2_{i j}(\tilde{\overrightarrow {Y}})}}.
\]
Prove the first property from Definition Definition 1 for the generalized correlation coefficient.
Express the
Pearson,
Spearman,
Kendall correlation coefficients as generalized correlation coefficients.
Fechner Sign Correlation Coefficient
Definition 8 The Fechner sign correlation coefficient is defined by the formula \[
I = \frac{C-H}{C + H},
\] where \(C\) is the number of pairs for which the signs of the deviations of the values from their means agree, and \(H\) is the number of pairs for which the signs of the deviations of the values from their means disagree.
Find the distribution of \(I\) under the hypothesis of independence of the samples, given that the samples are drawn from the standard normal distribution.
Practical Problems
A study was conducted to detect a relationship between the physical weight and the IQ of first-graders in the city of Ust-Ilimsk.
Test the hypothesis of independence of weight and IQ at significance level \(0.1\).
Let us compute the Spearman correlation coefficient for our data. Note that some of the studentsβ weight values coincide, so it is natural to assign them equal ranks. This is achieved using so-called mid-ranks: the ranks corresponding to equal values in the sample are replaced by the arithmetic mean of the ranks for those values. Thus, the ranks of the values of our samples are as follows (the first row corresponds to weight, the second to \(\mathrm{IQ})\):
Note: in the presence of ties (equal values), formula CoeffCorr:SpearmanSimpleRepresentation ceases to be exact β in deriving it, we used the fact that the vector of ranks is a permutation of the numbers \(1, \ldots , n\). Therefore we compute \(\rho_S\) from the definition, as the Pearson coefficient between the vectors of mid-ranks. We obtain \(\rho_{S} \approx -0.355\) (formula CoeffCorr:SpearmanSimpleRepresentation would give a close but inexact value of \(-0.348\)).
Problem data: weight and IQ of first-graders (\(n = 12\))
Real data from the problem statement (not a simulation): the weight (kg) and IQ of 12 first-graders in Ust-Ilimsk. Two observations β IQ \(145\) and IQ \(215\) β stand out sharply from the rest: the sample is clearly non-Gaussian, and the Pearson coefficient is inapplicable here, whereas the rank-based Spearman coefficient is robust to such outliers. Click the highlighted (red) outlier point to exclude/include it from the correlation calculation, or switch the data view below.
Observations used:
\(\hat\rho\) (Pearson) =
\(\rho_S\) (Spearman) =
For the full sample (\(n=12\)), the Spearman coefficient computed from mid-ranks is \(\rho_S \approx -0.355\); the Pearson coefficient is distorted by the outliers.
The value of the correlation coefficient is rather far from 0, which might suggest that the samples are dependent. Nevertheless, let us use the corrected statistic \(\tilde{\rho }_{S}\) to correctly test the hypothesis \(H_{0}\). We have (\(n=12\)):
\[
\begin{align} \tilde{\rho }_S & = \frac{\rho _S}{2}\left(\frac{\sqrt{n-2}}{\sqrt{1 - \rho _S^2}} + \sqrt{n-1}\right) = \frac{-0.355}{2}\left(\frac{\sqrt{10}}{\sqrt{1-0.355^2}} + \sqrt{11}\right) = \\ & = \frac{-0.355}{2}\left(\frac{3.162}{0.935} + 3.317\right) \approx \frac{-0.355}{2}\cdot 6.699 \approx -1.19, \end{align}
\]
whereas the critical value (the half-sum of the \(0.05\)-level quantiles of the distributions \(\mathscr {N}\left(0, 1\right)\) and \(T_{10}\)) equals
\[
\frac{Q_{\mathscr {N}\left(0, 1\right)}(0.05) + Q_{T_{10}}(0.05)}{2} = \frac{-1.645 - 1.812}{2} \approx -1.73.
\]
Since \(-1.19 > -1.73\), despite the fairly large absolute value of the correlation coefficient, the hypothesis of independence of the samples cannot be rejected.
GDP growth rates of Russia and Ukraine for 2006-2012, in percent (total and per capita, relative to 2005):
GDP growth rates of Russia and Ukraine for 2006-2012, in % relative to 2005: total and per capita.
Test the hypothesis of independence of these samples at significance level \(0.05\) (choose the most convenient and applicable test in this case).
A study was conducted at an institute to detect a relationship between a studentβs academic performance and the average number of donuts they eat per day.
Average number of donuts eaten per day, and final grade β for 14 students.
Test the hypothesis of independence of academic performance and the number of donuts eaten at significance level \(0.05\) (choose the most convenient and applicable test in this case).
Can we, at significance level \(0.05\), consider the sequence of numbers \[
1.05,1.12,1.37,1.50,1.51,1.73,1.85,1.98,2.03,2.17
\] to be a realization of a random vector all 10 of whose components are independent identically distributed random variables? Draw a conclusion based on correlation coefficients (choose the most convenient and applicable test in this case).
References
Best, D. J., and D. E. Roberts. 1975.
βAlgorithm AS 89: The Upper Tail Probabilities of Spearmanβs Rho.β Journal of the Royal Statistical Society. Series C (Applied Statistics) 24 (3): 377β79.
http://www.jstor.org/stable/2347111.