Linear Regression: Introduction

- Master Kan: "Truth is hard to understand."

- Kwai Chang Caine: "It is a fact, it is not the truth. Truth is often hidden, like a shadow in darkness."

Just want to quickly understand how linear regression works? The best way to understand the regression analysis is to calculate it yourself step by step. To do this, jump directly to the example below.

For all other curious people: Welcome to the world of regression, a branch of statistics that allows us to understand relationships within data and make predictions. In linear regression analysis, we're looking for truth in the form of a linear relationship between variables - but we must rely on evidence, or data, to identify these relationships.
It's important to recognize that while the relationships found through linear regression may reveal facts, the truth may be more complex. Linear regression gives us a tool to shed light on data and outline relationships, but it doesn't always give us the whole picture. It is a simplified representation of reality, a projection that is useful but limited.
In this introduction, we'll explore how linear regression works, how it's applied, and what insights it can provide. At the same time, we'll keep in mind that our models are approximations of reality, built on the basis of available evidence, and that there is always room for interpretation and further investigation.
Experts know that linear regression is the original data science. The technical concepts behind the linear regression model are so fundamental that they lay the foundation for modern methods in machine learning and AI.

Linear Regression for Beginners: The Linear Regression Line

At its core, linear regression is a simple statistical technique. It determines the best fitting straight line through a cloud of points. The goal is to identify the linear relationship between two variables represented by that data.


The line that best fits through the cloud of points is called the regression line. It can be written using a standard linear equation such as $\hat{y}={\hat{\beta }}_0+x\times {\hat{\beta }}_1$. The parameters are interpreted as follows:

• $\hat{y}$ :   The estimated dependent variable (the "^" indicates that this is an estimate).
• ${\hat{\beta }}_0$ :  The estimated y-intercept.
• $x$ :   The independent variable.
• ${\hat{\beta }}_1$ :  The estimated slope of the line (regression coefficient).

${\hat{\beta }}_0$ is the value of $\hat{y}$ when $\stackrel{=}{x}$ ( $\hat{y}={\hat{\beta }}_0+0\times {\hat{\beta }}_1$), which is precisely the point where our "best" line (the regression line) intersects the Y-axis. ${\hat{\beta }}_1$ tells us how much Y increases or decreases (in the case of a negative slope) for each additional unit of X.

Linear Regression Advanced: Least Squares

- Young Caine: "I am puzzled."

- Master Po: "That is the beginning of wisdom."

The key question now is: How do we find this "best" line? And what exactly are our parameters ${\hat{\beta }}_0$ and ${\hat{\beta }}_1$ that uniquely define the regression line? And why is it this line and not another one slightly higher - what does "best" line actually mean? In this case, the best line is the one for which the overall differences (or "distances") between the actual data points and the line are as small as possible. We try to position the line through the cloud of points so that the distances between the true values and the estimated values (i.e., the regression line) are minimal.

Mathematically, this optimization problem can also be formulated as follows:

$$\min \sum _{i=1}^n{\left(e_i\right)}^2=\min \sum _{i=1}^n{\left(y_i-\widehat{y_i}\right)}^2$$
We use the summation symbol ∑ because we consider not just one but all distances for our minimization. And to be precise, we're minimizing not just all distances, but all squared distances. This is because data points are sometimes above and sometimes below the line, and we don't want the distances to cancel each other out. The distances are positive for points above the line and negative for points below the line. We solve this problem by squaring the distances. Minimizing these squared distances also minimizes the "normal" distances - this method is called the method of least squares (or OLS, Ordinary Least Squares).
Continue: We apply the expectation value to both sides of the equation to detach our optimization problem from the specific data set with points i = 1, ..., n. In other words, we want to abstract from the sample and make inferences about the "true" parameters ${\beta }_0$ and ${\beta }_1$ of the population. We achieve this by minimizing the expected error in the population:

Calculating Linear Regression Parameters

The sum of squared distances $\sum _{i=1}^n{\left(e_i\right)}^2$, is sometimes simplified as SSR (Sum of Squared Residuals) or SSE (Sum of Squared Errors). By minimizing SSR, we also minimize the (squared) expected value of $e$. To do this, we need to take the derivatives of SSR with respect to ${\hat{\beta }}_0$ abd ${\hat{\beta }}_1$ set them equal to zero:

(1.) Partial derivative with respect to ${\hat{\beta }}_0$:

$\frac{\partial \mathrm{SSR}}{{\hat{\beta }}_0}=-2E\left[\right(Y-{\hat{\beta }}_0-{\hat{\beta }}_{1}X\left)\right]\stackrel{!}{=}0$

From this, it follows that:

$E\left[Y\right]={\hat{\beta }}_0+{\hat{\beta }}_{1}E\left[X\right]\hspace{0.17em}\iff \hspace{0.17em}{\hat{\beta }}_0=E\left[Y\right]-{\hat{\beta }}_{1}E\left[X\right]$

(2.) Partial derivative with respect to ${\hat{\beta }}_1$:

$\frac{\partial \mathrm{SSR}}{{\hat{\beta }}_1}=E\left[2\right(Y-{\hat{\beta }}_0-{\hat{\beta }}_{1}X\left)\right(-X\left)\right]=-2E\left[X\right(Y-{\hat{\beta }}_0-{\hat{\beta }}_{1}X\left)\right]\stackrel{!}{=}0$

This results in:

$E[XY-{\hat{\beta }}_{0}X-{\hat{\beta }}_1{X}^2]=0\hspace{0.17em}\iff \hspace{0.17em}E\left[XY\right]-{\hat{\beta }}_{0}E\left[X\right]={\hat{\beta }}_{1}E\left[X^2\right]$

When we use ${\hat{\beta }}_0$ from (1.):

$\begin{array}{ccc}E\left[XY\right]-\left(E\right[Y]-{\hat{\beta }}_{1}E[X\left]\right)E\left[X\right]={\hat{\beta }}_{1}E\left[X^2\right]& \hspace{0.17em}\iff \hspace{0.17em}& E\left[XY\right]-E\left[Y\right]E\left[X\right]={\hat{\beta }}_1\left(E\right[X^2]-{E\left[x\right]}^2)\\ {\hat{\beta }}_1=\frac{E\left[XY\right]-E\left[Y\right]E\left[X\right]}{\left(E\right[X^2]-{E\left[X\right]}^2)}& \hspace{0.17em}\iff \hspace{0.17em}& {\hat{\beta }}_1=\frac{\mathrm{Cov(X,Y)}}{\mathrm{Var(X)}}\end{array}$

As a reminder: $E\left[XY\right]-E\left[Y\right]E\left[X\right]=\mathrm{Cov}\left(\mathrm{X,Y}\right)$ and $E\left[X^2\right]-{E\left[X\right]}^2=\mathrm{Var}\left(X\right)$.

Why did we do all these calculations? The takeaway: We can now uniquely determine our regression line by the parameters ${\hat{\beta }}_0$ and ${\hat{\beta }}_1$. We have shown how to solve the least squares minimization problem and now know that our regression line is defined by these two estimators: and
We also assume that as we collect more observations, our estimated sample parameters ${\hat{\beta }}_0$ and ${\hat{\beta }}_1$ will converge to the true population parameters ${\beta }_0$ and ${\beta }_1$ (called "unbiasedness"). This means that even though we are working with a random sample of data, we can still estimate a "true" regression line.

Linear Regression for Experts: Regression Analysis with Matrices

- Kwai Chang Caine: "A worker is known by his tools. A shovel for a man who digs. An ax for a woodsman."

And we use regression.
So far, we have not used any terms or notation that go beyond basic algebra and calculus (and probability). This has forced us to limit ourselves to the simple linear regression model with one predictor variable. However, this becomes unbearable when we have multiple predictor variables, i.e. we want to examine the influence of multiple X's on Y. Fortunately, with a little (for some, a little more) linear algebra, we can abstract away from many of the 'bookkeeping' details, so multivariate linear regression hardly seems more complicated than the simple version.
This section will not remind the reader how matrix algebra works. Rather, some prior knowledge is assumed, because now the part for experts begins. In the following, the bold letters such as a denote matrices throughout, as opposed to the scalar a.

Our data consist of $n$ observations of the predictor variable $X$ and the response variable $Y$, i.e. $(x_1,y_1),\dots ,(x_n,y_n)$. We would like to fit the following model: where $\mathbb{E}\left[\epsilon \right|X=x]=0,\mathrm{Var}[\epsilon |X=x]={\sigma }^2$ and ε is uncorrelated between samples. First, we summarize all our observations of the target variables in an (n×1) matrix: We also summarize our coefficients in a (2×1) matrix: We also want to summarize the observations of the predictor variable $X$, but we need something that looks a little unusual at first glance: $$x=\left[\begin{array}{cc}1& x_1\\ 1& x_2\\ \text{⋮}& \text{⋮}\\ 1& x_{\mathrm{n}}\end{array}\right]$$

This is an (n×2) matrix, where the first column is always 1 and the second column contains the actual observations of $X$. We have this seemingly redundant first column because it helps us when we multiply x by β:

$$x𝛽=\left[\begin{array}{c}{𝛽}_0+{𝛽}_{1}x{}_1\\ {𝛽}_0+{𝛽}_{1}x{}_2\\ \text{⋮}\\ {𝛽}_0+{𝛽}_{1}x{}_{\mathrm{n}}\end{array}\right]$$ $x$β is therefore an (n×1) matrix that contains our predictions. The matrix $x$ is sometimes also referred to as a design matrix.

The Mean Squared Error MSE

For each data point, the use of the linear regression model, i.e. the application of the coefficients β, leads to a certain error in the prediction - so we have $n$ prediction errors. These form a vector: $$\mathbf{e}\left(\beta \right)=\mathbf{y}-\mathbf{x}\beta$$ Here, an (n×1) matrix is subtracted from an (n×1) matrix. In the section above, we said that we minimize the expected error by minimizing the sum of squared deviations / errors. Here we write this as follows: $$\mathrm{MSE}\left(\beta \right)=\frac{1}{n}\sum _i^n{e}_i^2\left(\beta \right)$$ But how do we formulate this expression using matrices? Like this: $$\mathrm{MSE}\left(\mathbf{\beta }\right)=\frac{1}{n}{\mathbf{e}}^T\mathbf{e}$$

To see that this works, let's take a look at what matrix multiplication really looks like: $$\left[\begin{array}{cccc}e{}_1& e{}_2& \dots & e{}_{\mathrm{n}}\end{array}\right]\left[\begin{array}{c}e{}_1\\ e{}_2\\ ⋮\\ e{}_{\mathrm{n}}\end{array}\right]$$

This clearly corresponds to $\sum _{i=1}^n{\left(e_i\right)}^2$, the form from the section above. With a little more math we get this:

$$\begin{array}{lll}\mathrm{MSE}\left(\beta \right)& =& \frac{1}{n}{\mathbf{e}}^T\mathbf{e;}\\ & =& \frac{1}{n}{(\mathbf{y}-\mathbf{x}\beta )}^T(\mathbf{y}-\mathbf{x}\beta )\\ & =& \frac{1}{n}\left({\mathbf{y}}^T-{\beta }^T{\mathbf{x}}^T\right)\left(\mathbf{y}-\mathbf{x}\beta \right)\\ & =& \frac{1}{n}\left({\mathbf{y}}^T\mathbf{y}-{\mathbf{y}}^T\mathbf{x}\beta -{\beta }^T{\mathbf{x}}^T\mathbf{y}+{\beta }^T{\mathbf{x}}^T\mathbf{x}\beta \right)\end{array}$$

Note that $({{\mathbf{y}}^T\mathbf{x}\beta )}^T={\beta }^T{\mathbf{x}}^T\mathbf{y}$ and that this is an (1×1)-Matrix such that ${{\mathbf{y}}^T\mathbf{x}\beta }^{}={\beta }^T{\mathbf{x}}^T\mathbf{y}$. Thus we get:

$$\mathrm{MSE}\left(\beta \right)=\frac{1}{n}({\mathbf{y}}^T\mathbf{y}-2{\beta }^T{\mathbf{x}}^T\mathbf{y}+{\beta }^T{\mathbf{x}}^T\mathbf{x}\beta )$$

Minimizing MSE

First, we find the gradient of MSE as a function of β:

$$\begin{array}{lll}\nabla \mathrm{MSE}\left(\beta \right)& =& \frac{1}{n}(\nabla {\mathbf{y}}^T\mathbf{y}-2\nabla {\beta }^T{\mathbf{x}}^T\mathbf{y}+\nabla {\beta }^T{\mathbf{x}}^T\mathbf{x}\beta )\\ & =& \frac{1}{n}(0-2{\mathbf{x}}^T\mathbf{y}+2{\mathbf{x}}^T\mathbf{x}\beta )\\ & =& \frac{2}{n}({\mathbf{x}}^T\mathbf{x}\beta -{\mathbf{x}}^T\mathbf{y})\end{array}$$

In the optimum, we set this equation to zero, so there is a$\hat{\beta }$ for which the following holds:

$${\mathbf{x}}^T\mathbf{x}\hat{\beta }-{\mathbf{x}}^T\mathbf{y}\stackrel{!}{=}0$$

This equation for the two-dimensional vector $\hat{\beta }$ is also called the normal equation. If we solve for $\hat{\beta }$, we get:

$$\hat{\beta }={\left({\mathbf{x}}^T\mathbf{x}\right)}^{-1}{\mathbf{x}}^T\mathbf{y}$$

We have now found a matrix equation with which we can calculate both coefficients of our regression line. We can now use our solution for the estimators ${\hat{\beta }}_0$ and ${\hat{\beta }}_1$ from the previous section for this equation:

$${\hat{\beta }}_1=\frac{c_{XY}}{{s^2}_X}=\frac{\overline{xy}-\overline{x}\overline{y}}{\overline{x^2}-{\overline{x}}^2}$$

und

$${\hat{\beta }}_0=\overline{y}-{\hat{\beta }}_1\overline{x}$$

So now we know how the linear regression is derived and structured. We have definitions for ${\hat{\beta }}_0$, the Y-intercept and ${\hat{\beta }}_1$, the linear slope(s) of our explanatory X variable(s) However, the mere functioning of the model is not enough. After applying the data analysis, the results of the model still need to be interpreted. You can see how this is done here.

Example: Calculating a Linear Regression

To understand the linear regression even better, it is worth calculating the regression analysis manually. This allows us to see how all the formulas are used and how the data are related. Here is an example of how to proceed step-by-step if you want to calculate a linear regression “manually” with specific numbers.

1. Write down the x and y values of a sample
2. From the values in step 1, calculate the corresponding $x^2\mathrm{,}{y}^2\mathrm{and}xy$
3. Compute the sum over all observations (i = 1, ..., N) from $x\mathrm{,}y\mathrm{,}{x}^2\mathrm{,}{y}^2\mathrm{and}xy$
4. Calculate mean values from step 3
5. Caluclate regression coefficients ${\hat{\beta }}_0$ and ${\hat{\beta }}_1$

i	X	Y	$X^2$	$Y^2$	$\mathbf{X}\mathbf{Y}$
1	40	80	1600	6400	3200
2	45	80	2025	6400	3600
3	60	90	3600	8100	5400
4	80	140	6400	19600	11200
5	75	100	5625	10000	7500
Sum	300	490	19250	50500	30900
Mean	60	98	3850	10100	6180

Calculate the covariance or variance in an intermediate step:

$$\mathrm{Cov}(X,Y)\hspace{0.17em}=\hspace{0.17em}E\left[XY\right]-E\left[X\right]E\left[Y\right]\hspace{0.17em}=\hspace{0.17em}6180-60\times 98\hspace{0.17em}=\hspace{0.17em}300$$
$$\mathrm{Var}\left(X\right)\hspace{0.17em}=\hspace{0.17em}E\left[X^2\right]-{E\left[X\right]}^2\hspace{0.17em}=\hspace{0.17em}3850-{\left(60\right)}^2\hspace{0.17em}=\hspace{0.17em}250$$
Now calculate the regression coefficients ${\hat{\beta }}_0$ and ${\hat{\beta }}_1$ by substituting into the formulas:

$${\hat{\beta }}_1\hspace{0.17em}=\hspace{0.17em}\frac{\mathrm{Cov(X,Y)}}{\mathrm{Var(X)}}\hspace{0.17em}=\hspace{0.17em}\frac{300}{250}\hspace{0.17em}=\hspace{0.17em}\mathrm{1,2}$$
$${\hat{\beta }}_0\hspace{0.17em}=\hspace{0.17em}E\left[Y\right]-{\hat{\beta }}_{1}E\left[X\right]\hspace{0.17em}=\hspace{0.17em}98-\mathrm{1,2}\times 60\hspace{0.17em}=\hspace{0.17em}26$$

The regression line is therefore: $\hspace{0.17em}\hat{Y}=26+\mathrm{1,2}\times \hat{X}$

Thus, the Y-intercept of the regression line = 26 and Y increases by +1.2 for an increase of X by +1 (one unit). The exemplary calculation worked well for the 5 data points from the sample. For more data, the calculation is of course much too complex, so we have built the online calculator for linear regression!