# Overfitting

Overfitting is when a model learns quirks of its training data rather than patterns that generalize, making its apparent performance unreliable on new data.

## What it means

A trading model can look convincing on historical data and still perform poorly on observations it has never seen. That is the central concern with overfitting. The model may have learned noise, a temporary market condition, or details peculiar to the sample rather than a relationship that persists.

Overfitting can affect a price model, an execution-cost estimate, a risk estimate, or an AI-generated assessment. It can also arise without complex AI: repeatedly adjusting a simple trading rule until its historical results look attractive creates the same problem. A good historical fit alone does not establish that a model will work on future data.

## How it is calculated

Overfitting has no single universal score. One useful diagnostic is the **validation gap**: the difference between a model's result on data used to fit it and its result on separate, unseen data. If larger values of a chosen metric mean better performance, calculate:

`Validation gap = training metric − out-of-sample metric`

- **Training metric** is the selected performance measure calculated on data used to fit the model.
- **Out-of-sample metric** is the *same* measure, in the *same units*, calculated on data withheld from fitting.
- **Validation gap** is expressed in those same units. A large positive gap can be a warning sign, not proof of overfitting.

For a metric where smaller is better, such as average prediction error, reverse the subtraction to express deterioration as a positive number: `Validation gap = out-of-sample error − training error`. The samples must be comparable, and the withheld data must not have influenced model selection. In time-ordered market data, testing on later observations is often more informative than randomly mixing past and future observations.

## Worked example

Suppose, hypothetically, a model estimates whether a stock order will fill within a specified time. The examples below are invented, not evidence about any live model.

1. A developer fits the model on 100 historical orders. It correctly classifies 90: `90 ÷ 100 = 90%` training accuracy.
2. The developer then evaluates the fixed model on 100 later orders that were not used to fit or select it. It correctly classifies 60: `60 ÷ 100 = 60%` out-of-sample accuracy.
3. The validation gap is `90% − 60% = 30 percentage points`.

That gap raises a question about generalization. It does not establish the cause: the later orders might differ in liquidity, order type, market conditions, or data quality. Accuracy also hides whether the model's probability estimates are well calibrated and whether errors on particular orders matter more than others.

## Why it matters before you trade

A pre-trade decision can depend on estimated costs, risks, fill conditions, and model-generated probabilities. If an estimate is overfitted, apparent precision may disguise uncertainty. A probability is not a promise, and a strong historical result is not a substitute for testing on genuinely separate data.

Useful questions are whether evaluation used later, untouched observations; whether the same metric and assumptions were applied throughout; and whether results hold across relevant stocks and market conditions. Repeatedly trying models against a supposedly untouched test set can itself make that set part of the fitting process.

## How PreTrAIde uses it

PreTrAIde provides pre-trade analysis for a proposed stock basket before the user places an order. Its deterministic, reproducible quantitative layer estimates measures including spread, market impact, daily volatility, and one-day 95% value at risk. Its AI agents reason from quantitative scores, news, and analyst ratings; guardrails correct outputs that contradict specified data conditions.

The product facts do not state that PreTrAIde calculates an overfitting score or describe out-of-sample validation results. Its success probability is a model estimate bounded between 5% and 95%, not a guarantee or a return forecast. The analysis verdicts are decision support, not recommendations.

## Common mistakes

- **Treating historical fit as future reliability.** A backtest can reflect choices tailored to the same data.
- **Leaking future information into testing.** Data unavailable at the decision time can make results look artificially strong.
- **Tuning to the test set.** Repeated adjustments based on test results undermine its independence.
- **Calling every gap overfitting.** Changed market conditions, small samples, and measurement errors can also explain differences.
- **Confusing accuracy with calibration.** A model can classify many cases correctly while assigning misleading probabilities.

---
Source: https://pretraide.com/glossary/overfitting