Random projection

In general, you can use an orthogonal projection to reduce the dimensionality of a vector. However, doing so promises nothing about the preservation of information from that vector. Surprisingly, if you fill a matrix, , with random numbers with and , it ends up doing a nice job preserving distance when used for projection. It doesn’t even need to be Gaussian.

As a result, you can literally just generate a random matrix to project high-dimensional data to lower dimensional data. The resulting embeddings should do a respectable job making the dimensionality of a dataset tractable for a neural network.

Of course, these days the embedding matrix is often trained, with the model learning its own optimal projection.