In commercial applications of machine learning, offline metrics refer to what we usually think of as “performance metrics” for statistical models: precision, accuracy, F-beta score (“f-score”), etc.

Online metrics refer to business-specific measures that are typically of most concern in practice: click-through rate, session duration, and so forth.

Models are typically trained using offline metrics and evaluated using online metrics, but certain kinds of models, such as reinforcement learning models, can be trained directly on online metrics.