A feature space is a vector space that corresponds to the input or output features of a machine learning model, typically (but not always) a neural network. Often times, the model’s input feature space is an Embeddings of a higher-dimensional set of original features.
In many neural networks, some (but not all) of the model’s hidden states can be considered as learned feature spaces. These “hidden feature spaces” represent learned correspondences between the input and output features, but are not directly observable.
In convolutional neural networks, the output of convolutional layers can be considered as learned feature spaces, as they discover features that associate the input features to output features. Similarly, in Transformer models, the output of the feed-forward sublayers can be considered as learned feature spaces, as they discover correspondences between the input and output features.
However, not all hidden states in neural networks necessarily correspond to feature spaces. For example, the output of the self-attention sublayers in transformers does not directly associate input features to output features, and therefore may not be considered as feature spaces in the same sense.