Least-Squares by Left Inverse

• • Called design matrix in statistics.
  • Tall matrix (Typically has more number of data than the number of features)
  • Full column rank meaning columns form the linearly independent set. If is , the rank is .
    • If it's not full column rank meaning is a reduced rank matrix, also meaning the columns in a matrix forms linearly dependent set, it's called multicollineariy in statistics.
  • itself is not invertible.
  • But tall matrix can have one-sided, left inverse if it has linearly independet set of columns.
  • We are given this data.
• • A vector of coefficient parameters.
  • We are not given this data. We don't know.
• • Dependent variable
  • A vector
  • We are given this data.

The goal is to remove from the left side of the equation, and to compute the numbers for by using and .

If is a square, full-rank matrix, we can remove by left-multiplying inverse of , , and making it identity matrix to disappear from the left side. But is not square, so we can't do it, so the below is wrong.

Instead, we first try to make it square by left-multiply , because suppose has , is times , so is square.

We wanna remove so left-multiply the inverse of , . We can do that because it's square.

Because , and as if

This equation has a weird looking, but we know why we have this, because we wanted to first square it, and take the inverse, and get identity matrix to leave the left side only .

In linear algebra, is called left-inverse. It appears when we have a tall matrix and we wanna get the inverse of it.

We got numbers of , but typically does not produce the number exactly same as . When produces exact , it called exact solution. In linear algebra, it's the situation where belongs to the columns space of . But in real-world problem that statistics solves, is unlikely to be in the column space, because statistics tries to develop a simplified equation to formulate the complex real-world problem. The exact solution is below, but it's unlikely to happen

Instead, we find which is , and use it as

Normal Equation

Left-multiply the both sides of the equation by

That is normal equation. It's used in the following least-squares by reduced row echelon form (RREF) and least-squares by QR decomposition.

Least-Squares by Reduced Row Echelon From (RREF)

First, convert it to normal equation.

Apply reduced row echelon form (RREF) to compute matrix inverse to the given data. It's about making augmented matrix by concatenating matrix at the left and identity matrix at the right, apply RREF, and get augmented matrix of identity matrix at the left concatenated by matrix inverse at the right, as below

When we apply that to the augmented matrix concatenated by left and right side of normal equation,

Least-Squares by QR Decomposition

The key part is, before applying QR decomposition to , convert the equation to normal equation by left-multiplying by

Replace with as QR decomposition. is orthogonal matrix. is residual matrix.

Apply LIVE EVIL rule to apply transpose to inner matrices.

gives us identity matrix , because they are orthogonal matrices.

Left-multiply both sides of the equation by

Because, letting ,