Similarities and differences with PCA

• Similarly, LDA achieves dimensionality reduction by finding a linear combination of synthetic variables which best explain the data
• Classes (or groups) for points are known a priori
• LDA is a supervised classification algorithm
• Goal is to determine a linear discriminant that separates our groups of interest

• Assumptions
  • Sensitive to outliers
  • Size of the smallest group must be larger than the number of predictor variables
  • Multivariate normality: Independent variables are normal for each level of the grouping variable.
  • Homoscedasticity: Variances among group variables are the same across levels of predictors
  • Multicollinearity: Predictive power can decrease with an increased correlation between predictor variables
  • Independence: Samples are assumed to be randomly sampled, and a sample's score on one variable is assumed to be independent of scores on that variable for all other samples

Formulations of LDA

• A generic classification problem involves a random variable $X$ from one of $K$ classes

• A discriminant rule tries to divide the data space into $K$ disjoint regions that represent all classes

• The Maximum Likelihood (ML) discriminant rule allocates $x$ to the population with the largest likelihood at $x$, thus

  $$ d(x)=\arg\max \delta_k(x) $$

• In a simple univariate case where $K=2$ and both $K_1\sim\mathcal{N}(\mu_1,\sigma_1^2)$ and $K_2\sim\mathcal{N}(\mu_2,\sigma_2^2)$, the ML rule will assign $x$ to $K_1$ if and only if $f(x_1)>f(x_2)$ , which is solvable by the PDF of Gaussian distribution.

$$ x^2(\frac{1}{\sigma^2_2}-\frac{1}{\sigma^2_1})+x(\frac{2\mu_1}{\sigma_1^2}-\frac{2\mu_2}{\sigma_2^2})+\frac{\mu_2^2}{\sigma_2^2}-\frac{\mu_1^2}{\sigma_1^2}+2\log\frac{\sigma_2}{\sigma_1}>0 $$

• If we have standardised our data such that now $\sigma_1=\sigma_2$, then we are left with a linear discriminant rule because the $x^2$ term will cancel

$$ 2x(\mu_1-\mu_2)+\mu_2^2-\mu_1^2>0 $$