Title: Datenanalyse

Histogram Formula

Q: Choice of interval length of a histogram
A: 1. Rule of Sturges:
Assumes underlying normal distribution and default for hist() in R
2. Rule of Scott:
More general, assumes underlying density function to be continuous and f' and f'' to be bounded and continuous. Since f is unknown in general, we assume normal distribution and it simplifies to:
where s is the empirical ("inferred from the data") standard deviation
3. Rule of Freedman and Diaconis:
In case of outliers, rule of Scott may be non-optimal because the standard deviation (s) can be inflated by outliers. Idea to use more robust scale estimator -> IQR (Interquartile Range)

Q: Kernel Density Estimation (Univariate)
A: Kernel density estimation (kde) is a non-parametric method to estimate the probability density function of a continuous random variable. Key points:
1. For each data point \(x_i\), we center a kernel (weight) function W(t) at that location
2. The kernel function gives more weight to points near \(x_i\) and less weight to distant points
3. The bandwidth h controls the smoothness - larger h means more smoothing
4. The density estimate at any point x is: \(\hat{f}(x) := \frac{1}{n}\sum_{i=1}^n \frac{1}{h}W\left(\frac{x-x_i}{h}\right)\) where n is the sample size
The result is a smooth curve showing the shape of the data distribution, like a continuous version of a histogram. Common kernel functions include rectangular (boxcar) and cosine functions.
Key properties:
- Integrates to 1 (proper density)
- Non-parametric (no assumed distribution)
- Smoothness controlled by bandwidth h
- More robust to bin width choice than histograms

Q: Empirical Distribution Function
A: The Empirical Distribution Function is trying to estimate the underlying cumulative distribution function (cdf). For \(n \rightarrow \infty\) it converges against the theoretical distribution function.

Q: Normal probability plot
A: Empirical distribution function is very simple to compute without the need to estimate any parameter but it is usually hard to see if the underlying distribution is normally distributed (or any other distribution we would like to compare with).
Idea of normal probability plot is to rescale \(F_n(x)\) such that the S-shape turns into a line. Deviations from the line are easier to grasp.
This tests for a normal distribution. We take the inverse of the normal distribution and plug-in our empirical distribution function. The distributions "cancel" each other and a line is created:
On this line you can see an estimate for the mean at 50% probability and for the standard deviation + mean at 84%.

Q: Quantile-quantile plots
A: For the normal probability plot we used a probability scale, but we could as well just show
a scale in terms of quantiles of the standard normal distribution.
Similar as for the normal probability plot, the QQ-plot compares an empirical distribution
function, computed from the data sample, with a hypothetical one, e.g. with normal dis-
tribution

Boxplot

Q: Classical univariate estimators
A: - Arithmetic mean:
- Empirical Standard Deviation:
If data quality is good those estimators are fine but if the data quality is not great (outliers) we should look towards robust estimators

Q: Robust Univariate Estimators
A: Robust estimators of location (middle of the data):
- Trimmed mean
- Median
Robust estimators of scale (spread of the data):
- Interquartile Range
- Trimmed Standard Deviation
- Median Absolute Deviation (MAD)
- \(Q_n\) Estimator

Q: Outlier Indentification Rules
A: - Boxplot rule: outliers are smaller than \(\hat q_{0.25} - 1.5*IQR\) or bigger than \(\hat q_{0.75} + 1.5*IQR\) (this rule is also robust against outliers!)
- Sigma Rule: We know that that \(\mu \pm 2*\hat\sigma\) contains 95% of the data in a normal distribution, values outside could be potential outliers -> we need to estimate location \(\mu\) and scale \(\sigma\): \(x_{median} \pm 2*s_{MAD}\) would be the most appropriate since both estimators are robust against outliers

Q: Bivariate density estimation
A: Similar to the univariate case we can also estimate the density function in a 2-dimensional case.
Basic principle the same: Select local region (this time in two dimensions) -> evaluate a given weight function W (kernel)
Based on window width \(h\) (which could be different for x and y) we optain the formula:

Q: Least-squares (LS) estimator/regression
A: LS estimator is not robust! And a few outliers can drastically change the regression line!

Q: Robust regression line after Tukey
A: We estimate the parameters of the regression line iteratively:
1. Sort the data pairs \((x_i,y_i)\) according to their x-values
2. Group the data pairs into 3 groups of about equal size (left, middle, right)
3. Compute medians of the x and y values in the singe groups:
Thus take all the x values of a group and get the median and then get all the y values of a group and get the median
\(x_L = \underset{(x_i,y_i) \in L}{median}\) \(x_i\)
\(y_L = \underset{(x_i,y_i) \in L}{median}\) \(y_i\), etc. for all other groups
This gives you \((x_L,y_L), (x_M,y_M),(x_R,y_R)\)
4. \(\hat \beta_0 = \frac{y_R-y_L}{x_r-x_L}\) first estimate of \(\beta\) (slope)
\(\hat\alpha_0 = \frac{1}{3}((y_L+y_M+y_R)-\hat\beta_0(x_L+x_M+x_R))\) first estimate of \(\hat \alpha_0\) (intercept)
5. Calculate Residuals:
\(r_i^0 = y_i - (\hat\alpha_0 + \hat\beta_0(x_i-x_M))\) for \(i=1,...,n\)
6. Repeat steps 3-5 with data pairs \((x_i,r_i^0)\) for \(i=1,...,n\) yields line
\(r_i^0=\hat\alpha_1+\hat\beta_1(x_i-x_M)\) and has the residuals \(r_i^1=r_i^0-(\hat\alpha_1 + \hat\beta_1(x_i-x_M))\)
Which means we fit a line to our previous residuals and calculate the residuals of that until parameters are under a certain threshold and dont move that much anymore (we try to capture any remaining linear trend)
7. \(\hat\beta=\hat\beta_0+\hat\beta_1+\hat\beta_2+...\)
\(\hat\alpha=\hat\alpha_0+\hat\alpha_1+\hat\alpha_2+...\)
Summation of our estimates gives us the final estimates of our parameters for the regression line
After the 3rd iteration the line does not change anymore.

Q: Robust regression line after Theil
A: Calculate the slope by considering the median of the slopes given by all pairs of data points (x values need to be distinct otherwise we would divide by 0)
\(\hat\beta_{ij}=\frac{y_j-y_i}{x_j-x_i}\) \(1\le i < j \le n\)
this gives us \(\frac{n(n-1)}{2}\) slopes
The Theil estimator is then given by:
\(\hat\beta_T=\underset{1\le i<j\le n}{median} \ \hat\beta_{ij}\) (we take the median of all previously calculated slopes)
\(\hat\alpha_T=\underset{1\le i<j\le n}{median} \ (y_i-\hat\beta_Tx_i)\) (we take the median of all derived intercepts with the Theil estimator slope)

Q: Repeated median regression
A: Repeated median regression estimator of Siegel is very similar to Theils robust regression line. We calculate the median of the medians for each point:
\(\hat\beta_{RM} = \underset{1\le i \le n}{median} \ (\underset{\underset{j \ne i}{1 \le j \le n}}{median(\hat\beta_{ij})})\)
\(\hat\alpha_{RM} = \underset{1 \le i \le n}{median} \ (y_i - \hat\beta_{RM}*x_i)\)
1. Step 1: Calculate Slopes Between Pairs of Points
For each point \((x_i,y_i)\), we calculate the slope with every other point \((x_j,y_j)\) except itself:
2. Step 2: Find the Median Slope for Each Point
Next, we calculate the median slope for each point based on the list of slopes we computed.
For example:
  - For Point 1, the median of \((2,0.5,1,24.5)\) is \(1.5\)
We would repeat this for each of the remaining points to get a median slope for each point.
3. Step 3: Calculate the Overall Median Slope
Now, we take the median of all the median slopes obtained for each point.
4. Step 4: Calculate the Intercept
Now we calculate the median of all the intercepts with the previously calculated slope

Q: Least Median of Squares (LMS) regression
A: The LMS regression is very similar to Least Squares but is more robust since we replace the sum in LS regression with the median.
\((\hat\alpha_{LMS},\hat\beta_{LMS}) = argmin_{\alpha,\beta} \ median_i \ r^2_i\)

Q: Least Trimmed Squares (LTS) regression
A: The LTS estimator minimizes the sum of the smallest squared residuals i.e. a trimmed sum. Thus the squared (!) residuals need to be sorted in ascending order!
\((\hat\alpha_{LTS},\hat\beta_{LTS})=\underset{\alpha,\beta}{argmin}\sum_{i=1}^{h}r^2_i\) with \(\frac{n}{2} < h < n\)
The parameter \(h\) determines the breakdown point of the LTS estimator which is between 0% and 50%

Q: Statistical efficiency
A: Statistical efficiency measures how well an estimator uses data to achieve accurate estimates, with higher efficiency meaning lower variance and better accuracy for a given sample size. An efficient estimator requires less data to achieve the same precision as less efficient estimators.

Q: Local polynomial regression fitting (LOWESS / LOESS)
A: Is a procedure for smoothing scatter plots but can also be applied to time series data. It does not assume equidistant points along the x-axis.
Key idea is to use a locally weighted regression where the regression model is either linear (a line) or quadratic (a parabola)
We fix a parameter \(s\) (span, default \(0.75\)) which defines the proportion of data points to be used for the local smoothing.
Thus \(q=\lfloor ns+0.5 \rfloor\) data points.
We also need two weight functions:
Tri-cube function \(T(t)= \begin{cases} (1-|t|^3)^3, & |t|<1 \\ \ 0, & \text{otherwise} \end{cases}\)
Biweight function \(B(t)= \begin{cases} (1-|t|^2)^2, & |t|<1 \\ \ 0, & \text{otherwise} \end{cases}\)
We want to smooth at index \(i\) so to obtain the smoothed value:
1. Compute all distances along x-coordinate:
\(d_{ik} = | x_i-x_k| \ \text{for} \ k=1,...,n\)
2. Compute weights according to distance in x-coordiante
\(t_i(x_k)=\begin{cases} T(\frac{d_{ik}}{d_{iq}}), & d_{iq} \ne 0 \\ \ 1, & d_{iq}=0 \end{cases} \ \text{for} \ k=1,...,n\)
3. Then we do a weighted LS regression
\((\hat\alpha^i, \hat\beta^i)=\underset{\alpha,\beta}{argmin} \sum_{k=1}^{n} t_i(x_k)(y_k-\alpha-\beta x_k)^2\)
This gives us the smoothed value \(\hat y_i = \hat\alpha^i+\hat\beta^ix_i\)
4. Repeat Steps 1-3 for all \(i=1,...,n\)
5. Compute residuals \(r_i=y_i-\hat y_i \text{, for }i=1,...,n\)
Estimate residual scale \(\hat\sigma\)
6. Compute weights for the size of scaled residuals:
\(w(x_k)=B(\frac{r_k}{2\hat\sigma}) \ \text{for } k=1,...,n\)
and use them in Step 3 for weighted regression -> the weights are \(w(x_k)t_i(x_k)\)
7. Repeat all these steps serveral times

Q: Time Series Components (explained)
A: A time series of the form
\(x_t=\tau_t+\delta_t+e_t \text{ for t=1,...,T}\)
\(\tau_t\): Trend means that the time series increases or decreases for a longer period of time
\(\delta_t\): Seasonality refers to regular variation such as regular changes for the same months over the years
\(e_t\): remainder or error component

Q: Exponential smoothing (Simple Version)
A: The idea behind exponential smoothing is that previous values are considered and weighted according to their history (More recent values receive a heigher weight, while values that are further away in the past receive small weight)
The smoothing factor \(0 < \alpha < 1\) determines how much weight the most recent value receives and how much past values are weight \((1-\alpha)\)
The smaller \(\alpha\) is chosen, the smaller the contribution of the most recent values which implies that the value \(\tilde x_t\) gets smoother.
\(\tilde x_t =\alpha x_t + (1-\alpha)\tilde x_{t-1}\)
To determine \(\alpha\) we use a sum of least squares (but more robust methods can be used)
\(\sum_{t=m}^T (x_t-\tilde x_{t-1})^2 \rightarrow min\)
it is immediate to see that this results in a recursive computation (thus the name, the weights into the past decrease exponentially)
\(\tilde x_t=\alpha x_t + \alpha(1-\alpha)x_{t-1}+\alpha(1-\alpha)^2x_{t-2}+\alpha(1-\alpha)^3x_{t-3}+...\)
For predictions into the future \(h\) time periods away
\(\hat x_{t+h|t} = \tilde x_t\) this implies that it doesnt matter how we select \(h\) the last value is always the "right" prediction since for simple exponential smoothing we dont consider neither trend nor saisonality

Q: Exponential Smoothing (Holt-Winters)
A: The so-called Holt-Winters method includes a trend variable \(b_t\):
\(\tilde x_t = \alpha x_t + (1-\alpha)(\tilde x_{t-1} + b_{t-1})\)
\(b_t=\beta(\tilde x_t-\tilde x_{t-1})+(1-\beta)b_{t-1}\)
At every time point \(t\) the parameter \(b_t\) estimates the local slope of the time series. The parameters \(\alpha, \beta\) are again in the interval (0,1) and they determine the degree of smoothing.
The forecast with Holt-Winters for a horizon \(h\) is given by:
\(\hat x_{t+h|t} = \tilde x_t+hb_t\)
Thus the forecast is done by a linear equation with the most recent slope \(b_t\) starting from \(\tilde x_t\).

Q: Autocovariance
A: Some time series models make use of dependencies which are given by a shift of the series of \(k\) steps (lag)
Dependencies are analyzed with the autocovariance \(Cov(x_t,x_{t-k})\), it can be estimated by: (order \(k)\)
\(c_k=\frac{1}{T}\sum_{t=k+1}^T(x_t-\overline x)(x_{t-k}-\overline x)\)
with the arithmetic mean \(\overline x=\frac{1}{T}\sum_{t=1}^T x_t\)
The autocovariance for lag 0 thus \(c_0\) is the variance of \(x_t\)

Q: Autocorrelation
A: The autocorrelation of order \(k\) is defined as:
\(\rho_k=Corr(x_t,x_{t-k})=Cov(x_t,x_{t-k})/Var(x_t)\)
and can be estimated by:
\(r_k=c_k/c_0\)

Q: Moving Average (MA) model
A: A stationary time series follows a Moving Average process of order 1, MA(1) if:
\(x_t=a+u_t-\theta u_{t-1}\)
with the unkown parameters \(a, \theta\)
A Moving Average process of order \(q\) MA(q) is defined as:
\(x_t=a+u_t-\theta_1u_{t-1}-\theta_2u_{t-2}-...-\theta_qu_{t-q}\)

Q: Autoregressive Model (AR)
A: A stationary time series follows an autoregressive process of order 1 denoted as AR(1) if
\(x_t=a+\phi x_{t-1}+u_t\)
with the unknown parameters \(a,\phi\)
Similarly an autoregressive process of order \(p\) is defined as:
\(x_t=a+\phi_1 x_{t-1}+\phi_2x_{t-2}+...+\phi_px_{t-p}+u_t\)

Q: ARMA (Autoregressive-Moving-Average)
A: If neither ACF plot nor PACF plot show an "implosion" to zero from a certain lag, the process could be a mix of AR and MA denoted as ARMA(p,q) and defined as:
\(x_t=a+\phi_1x_{t-1}+\phi_2x_{t-2}+...+\phi_px_{t-p}+u_t-\theta_1u_{t-1}-\theta_2u_{t-2}-...-\theta_qu_{t-q}\)

Q: ARIMA model
A: ARMA models require a stationary time series. Time series with a trend are named as non stationary or integrated. Trends can be eliminated using differences defined as:
\(\Delta x_t=x_t-x_{t-1}\)
Linear trends can be eliminated by applying \(\Delta\) once, while quadratic trends can be eliminated by applying it twice and so on...
A time series \(x_t\) follows an ARIMA(p,d,q) model if \(\Delta^dx_t\) follows an ARMA(p,q) model
Example: an ARIMA(1,1,0) model is of the form:
\(\Delta x_t=a+\phi\Delta x_{t-1}+u_t\) and thus
\(x_t=x_{t-1}+a+\phi(x_{t-1}-x_{t-2})+u_t\)

Q: Covariance
A: Theoretically the covariance between a pair of random variables \(x_j\) and \(x_k\) is defined as:
\(\sigma_{jk}=E[(x_j-E(x_j))(x_k-E(x_k))]\)
The classical estimator for the covariance is the sample covariance:
\(s_{jk}=\frac{1}{n-1} \sum_{i=1}^n (x_{ij}-\overline x_j)(x_{ik}-\overline x_k)\)
for \(j=k\) we obtain the sample variance

Q: Correlation
A: The correlation is a measure for the linear relationship between two variablews which is always in the interval [-1,1]. Therefore correlation is easier to interpret than covariance because it does not depend on the variances  of the variables.
The population correlation coefficent is defined as:
\(\rho_{jk}=\frac{\sigma_{jk}}{\sqrt{ \sigma_{jj} \sigma_{kk}}}\)
The sample correlation coefficent also called Pearson correlation is defnied as:
\(r_{jk}=\frac{s_{jk}}{\sqrt{s_{jj}s_{kk}}}\)

Q: Covariance / Correlation Matrix
A: If we determine the covariance or correlation for all pairs of variables we obtain a matrix of dimension \(p \times p\)
The theoretical covariance matrix is denoted by \(\Sigma\) and the sample covariance matrix is denoted by \(S\)
The theoretical correlation matrix is denoted by \(\rho\) and its sample counterpart by \(R\)

Q: Spearman rank correlation
A: More robust estimation of correlation.
We compute Pearson correlations from the ranks of the observations which are values bewteen 1 and \(n\).
The Spearmans rank correlation isa measure of monotone relationship and captures relationship outside of linearity

Q: Minimum Coveriance Determinant (MCD)
A: MCD aims to find a subset of the data (usually half of the total points or a proportion close to it) that has the smallest possible covariance determinant.
The covariance determinant reflecctrs the volume of the data distribution, so minimizing it finds a subset that most tightly clustered together
Denote by \(C\) the MCD estimator for the covariance matrix and its elements by \(c_{jk}\) for \(j,k=1,...,p\)
The robust variances are given by \(c_{jj}\) thus the robust correlation matrix is given by:
\(C=\frac{c_{jk}}{\sqrt{c_{jj}}\sqrt{c_{kk}}}\)
Steps to Compute MCD:
1. Choose Subset Size:
Select a subset of the data, often with a size h, where h is slightly larger than half the data points (usually h≈n/2+some small constant).
2. Compute Mean and Covariance for Random Subset:
Calculate the mean vector and covariance matrix for several random combinations of h points from the dataset.
3. Determinant Calculation:
For each subset, compute the covariance matrix's determinant (a measure of how spread out the data is).
4. Minimization:
Out of all possible subsets, select the one with the smallest covariance determinant. This subset is the one that minimizes the influence of outliers and captures the core data structure.
5. Final Estimation:
Calculate the mean and covariance matrix of the selected "best" subset to obtain robust estimates.

Q: Euclidean Distance
A: The Euclidean Distance between \(x_A\) and \(x_B\) is defined as:
\(d_E(x_A,x_B)=(\sum^p_{j=1} (x_{Bj}-x_{Aj})^2)^\frac{1}{2} = ||x_B-x_A||\)

Q: Manhattan Distance
A: Also called City Block Distance is based on the absolute coordinatewise distances:
\(d_M(x_A,x_B)=\sum^p_{j=1}|x_{Bj}-x_{Aj}|\)

Q: Minkowski Distance
A: The Euclidean Distance and Manhattan Distance can be generalized to the Minkowski Distance that is defnied as:
\(d_{Mink}(x_A,x_B)=(\sum^p_{j=1}(x_{Bj}-x_{Aj})^m)^{1/m}\)

Q: Mahalanobis Distance
A: This distance measure accounts for the covariance structure of the data and thus does not depend on the scale of the variables.
Usually this distance is most interesting for an observation \(x_i\) to the center \(\mu\) of the distribution and is defnied as:
\(d_{Mahal}(x_i,\mu)=[(x_i-\mu)^T\Sigma^{-1}(x_i-\mu)]^{1/2}\) for \(i=1,...,n\)
If we would replace the covariance matrix \(\Sigma\) by the identity matrix \(I\) we would obtain the Euclidian distance (of an observation to the center)
As one can see the Mahalanobis Distance is better suited if variables are correlated!

Q: Linear Combinations of Variables
A: Linear combinations are used for dimensional reduction
If we have a \(n\times p\) data matrix \(X\) a linear combination would look like this:
\(u=x_1b_1+x_2b_2+...+x_pb_p=Xb\)
It is common to call the coefficents \(b\) loadings and the values \(u\) scores
The coefficents are usually normed to length 1 thus \(\sum^p_{j=1}b_j^2=1\)
if we want all linear combinations of all "dimensions" we would arrive at:
\(U=XB\) where \(U\) is the matrix of the score vectors, \(X\) is our data matrix and \(B\) is the coefficent vector matrix

Q: Principal Component Analysis (PCA)
A: Assume we have given a centered and possibly scaled data matrix \(X\) of dimension \(n \times p\). Then the PCs are defnied via linear combinations like:
\(U=XB\)
\(U\) as \(u_1,...,u_p\) are the principal components
Now the problem at hand is how do we choose \(B\)? -> Optimization Problem under constraints:
- The first loadings vector \(b_1\) is chosen such that the variance of \(u_1=Xb_1\) is maximized, under the constraint \(b_1^Tb_1=1\) (normalized to length 1).
- The second loadings vectors \(b_2\) is chosen such that the variance of \(u_2=Xb_2\) is maximized under the constraints \(b_2^Tb_2=1\) and \(b_2^Tb_1=0\) (orthogonality to previous direction)
- The j-th loadings vector \(b_j\) is chosen such that the variance of \(u_j=Xb_j\) is maximized under the constraints \(b_j^Tb_j=1\) and \(b_j^Tb_l=0\) (orthogonality to all previous directions)

Q: Number of relevant PCs
A: Since the goal of PCA is to reduce dimensionality and at the same time keep loss of information small we try to express as much information with as little PCs as possible.
We can compute the variance expressed by the first \(k\) PCs as \(\lambda_1+...+\lambda_k\) and the total variance is \(\lambda_1+...+\lambda_p\). Therefore the proportion of explained variance by the first \(k\) PCs is:
\(\frac{\lambda_1+...+\lambda_k}{\lambda_1+...+\lambda_p}\)
This can be visualized by the so-called scree-plot. The optimal number of PCs could then be selected at that point before the curve "flattens out"

PMF CDF PDF

Q: K-means Clustering
A: The k means clustering algorithm takes the \(k\) (the number of clusters) and uses centroids or cluste centers which can be taken as the arithmetic means of the observations per cluster:
\(\overline x_j=\frac{1}{n_j}\sum_{i \in I_j} x_i\) for \(j=1,...,k\)
And the objective function is definied as:
\(\sum^k_{j=1}n_j\sum_{i \in I_j} ||x_i- \overline x_j ||^2 \rightarrow min\)
There is also a very similiar algorithm PAM (Partitioning Around Mediods) which uses medians to calculate the cluster centroids

Q: Cluster Validity Measures
A: A cluster validtity measure should help with decision about an appropriate number of clusters \(k\).
The variance of observations of a cluster is a good measure of homogenity:
\(W_k=\sum_{j=1}^k\sum_{i \in I_j}||x_i-\overline x_j||^2\), the value of \(W_k\) should be small
For heteorgenity we can consider the between-cluster sum-of-squares:
\(B_k=\sum_{j=1}^k || \overline x_j - \overline x ||^2\), the value of \(B_k\) should be big
Calinkski-Harabasz Index is a normalized ratio of theese quantities:
\(CH_k=\frac{B_k/(k-1)}{W_k/(n-k)}\)
Another proposal is the Hartigan-Index:
\(H_k=log\frac{B_k}{W_k}\)
Then we plot these indexes against \(k\) and take the \(k\) for which the decline starts to flatten out.

Q: Discriminant Analysis
A: Discriminant Analysis is used to find a boundary or rule that separates different classes (or categories) of data based on some features or characteristics. Once this boundary is established, it can be used to classify new data points into one of the predefined classes.
Since Discriminant Analysis requires the classes to be known in advance, it is a supervised learning method. The model is trained on labeled data where the correct class for each observation is already known.

Q: Linear Discriminant Analysis (LDA)
A: Linear Discriminant Analysis (LDA) is one of the most commonly used supervised classification methods. It is particularly useful when you have data points that you want to classify into pre-known categories (classes) based on predictor variables (features).
LDA works under some fundamental assumptions:
- The data from each class is normally distributed.
- The covariance matrices of the features (i.e., how the features vary together) are the same across all classes. This assumption simplifies the problem, resulting in linear boundaries.
- Classes are well-separated: though this assumption can be relaxed, LDA works best when there’s a good separation between the classes.
The linear discriminant rule is defined as:
\(x\) is assigned to the j-th (and not the l-th group) if
\(\delta_j(x)>\delta_l(x)\) where
\(\delta_j(x)=x^T\Sigma^{-1}\mu_j-\frac{1}{2}\mu^T_j\Sigma^{-1}\mu_j+log \ p_j\)
is called linear discriminant function of the j-th group
In order to be able to apply the rule we need to estimate the unknown parameters from the training data.
The prior probabilities \(p_j\) can be estimated by the group proportions \(n_j/n\).
The sample mean:
\(\hat\mu_j=\overline x_j=\frac{1}{n_j}\sum_{i \in I_j}x_i \ \text{for } j=1,...,k\)
The sample covariance matrix
\(\hat\Sigma=S_{pooled}=\frac{1}{n-k}\sum_{j=1}^k\sum_{i \in I_j}(x_i-\overline x_j)(x_i-\overline x_j)^T\)

Q: Quadradtic Discriminant Analysis
A: Quadratic Discriminant Analysis (QDA) is a supervised classification method similar to Linear Discriminant Analysis (LDA) but with a key difference: QDA allows each class to have its own covariance matrix. This results in quadratic decision boundaries instead of linear ones.
Steps in QDA:
1. Estimate the means \(\mu_j\) and covariance matrices \(\Sigma_j\) for each class using the training data.
2. For a new observation \(x\), calculate the discriminant score \(\delta_j(x)\) for each class \(j\).
3. Classify the observation into the class Cj that has the highest discriminant score.
When to Use QDA:
- When you suspect or know that the covariance (spread and relationship between features) differs across classes.
- When the decision boundary between the classes is non-linear or complex.
The quadratic discriminant functions are defined as:
\(\delta^q_j(x)=-\frac{1}{2}log(det(\Sigma_j))-\frac{1}{2}(x-\mu_j)^T\Sigma^{-1}(x-\mu_j)+log \ p_j\)
To estimate the unknown paramters we do it similarly as in LDA but we now need to estimate the individual covariances by taking the sample covariance matrices:
\(\hat\Sigma_j=S_j=\frac{1}{n_j-1}\sum_{i \in I_j}(x_i-\overline x_j)(x_i-\overline x_j)^T\)
Since we need to estimate way more parameters in QDA than LDA this could lead to overfitting.