Updated 2026-07-25
Out of sample describes data that took no part in building, fitting or tuning a model. In-sample performance tells you how well a model describes the data it was given, which is nearly always flattering. Out-of-sample performance is the only measurement that speaks to whether it generalises.
The subtlety is that out of sample is easy to claim and easy to contaminate. Choosing features or thresholds by looking at the test set, re-running a study until a period cooperates, or reusing the same holdout across dozens of variants all leak information back in, and the result quietly becomes in-sample again. That is why the discipline matters as much as the label: pre-register the rule, test forward, and count every attempt. See walk-forward validation and overfitting.