Minimization (maximization) of a quadratic form

Based on Chaudhury 2024, ch. 4 (linear algebraic tools for ML)

Given a quadratic form , we wish to find the unit vector that minimizes (or maximizes) .

To begin, we observe that the existence of the quadratic form implies that is symmetric. As such, by the spectral theorem, we know that can be diagonalized by an orthogonal matrix S, i.e., that we can express q as

By the definition of an orthogonal matrix, we know that , and so

Let us now define . Because the transpose of a matrix product is the reversed product of the transposes, we also have that . Substituting the definitions of and , we have

Now, again we recall that must be orthogonal because is symmetric. Orthogonal matrices are length preserving. Hence we know that must have the same magnitude as the unit vector , i.e.

Therefore, we can find the that minimizes without loss of generality. Expanding the right hand side of this relation, we see that this is equivalent to saying that

We observe that

Multiplying this by , we see that

i.e., that is a weighted sum of the eigenvalues of ​, with weights . Being squared, these weights are non-negative; being terms of a unit vector , they sum to 1. Clearly, to minimize the quadratic form, we must put all of the weight on the smallest eigenvalue of ; and to maximize, we must put all of the wight on the largest eigenvalue.