What it means
A probability estimate is well calibrated when outcomes occur about as often as the estimate suggests. If a model assigns 70% probability to many comparable events, roughly 70% of those events should occur over a sufficiently large, representative sample. An individual 70% prediction cannot be verified by a single outcome.
Calibration requires a specific event, a time horizon and a rule for recording the outcome. In stock trading, “success” could mean a condition defined for a proposed execution or a later market outcome; these are different events. Without knowing which event was predicted, a percentage is hard to evaluate. Calibration does not say whether a trade is profitable, whether its costs are acceptable or whether its risk fits an investor’s circumstances.
How it is calculated
One practical check groups past predictions into probability bands and compares each band’s average prediction with its observed outcome rate:
calibration gap = |(sum of predicted probabilities / n) − (number of observed successes / n)|
- Predicted probability: The model’s estimate for the defined event, expressed as a decimal; 70% is 0.70.
- Observed success: An outcome recorded as 1 if the defined event occurred and 0 otherwise.
- n: The number of predictions in the band.
- Calibration gap: The absolute difference between the average prediction and the observed success rate, in probability units. Multiply by 100 to express it in percentage points.
A smaller gap is better for that band, but it does not prove that the model is calibrated overall. Analysts also examine multiple bands, sample sizes, changing market conditions and whether the cases tested resemble future cases. A model can have a small aggregate gap while giving misleading probabilities to particular groups.
Worked example
Suppose, hypothetically, that a model assigns 70% to each of 100 independently reviewed, comparable pre-trade cases. The event and review period are specified before outcomes are checked. Later, 60 cases meet that event definition.
- Add the predictions: 100 × 0.70 = 70; divide by 100 for an average predicted probability of 0.70.
- Calculate the observed rate: 60 ÷ 100 = 0.60, or 60%.
- Calculate the gap: |0.70 − 0.60| = 0.10, or 10 percentage points.
In this hypothetical group, the predicted frequency exceeds the observed frequency. That finding alone does not establish why: the sample might be unrepresentative, the event might be recorded inconsistently, or conditions might have changed. A different group could produce a different gap.
Why it matters before you trade
A probability can look precise even when its relationship to actual outcomes is uncertain. Knowing what it measures and how comparable predictions have behaved helps a private investor interpret it as an estimate rather than a promise. Calibration is distinct from discrimination: a model may rank higher- and lower-risk cases effectively while assigning probabilities that are systematically too high or too low.
Calibration checks should also avoid selecting only cases that were acted on without accounting for cases that were skipped. Those groups can differ, and missing outcomes can distort a comparison. Past calibration does not guarantee that probabilities will remain calibrated as liquidity, volatility or available data change.
How PreTrAIde uses it
PreTrAIde provides a model-estimated success probability between 5% and 95% with each basket and line verdict. It is neither a guarantee nor a return forecast. Its pre-trade analysis also provides a risk assessment, a suggested execution strategy and plain-language reasoning. Quantitative scores anchor its AI analysis, and guardrails downgrade a GO verdict when its success probability is below 50%.
The product facts do not specify a calibration method, a measured calibration gap or a claim that these probabilities are calibrated. PreTrAIde Memory can use logged trading outcomes to personalize later analyses at the AI layer; quantitative scores remain identical. Personalization should not be mistaken for evidence of probability calibration.
Common mistakes
- Treating 70% as a guaranteed result: It describes uncertainty about a defined event, not the result of one case.
- Confusing calibration with profitability: Even a well-calibrated event probability says nothing by itself about gains, losses or transaction costs.
- Comparing unlike events: Predictions and observed outcomes must use the same event definition and time horizon.
- Judging by one bucket or a tiny sample: A single average can conceal large errors elsewhere, and small samples produce noisy observed rates.
- Assuming safeguards prove calibration: Limiting a probability range or correcting a contradictory verdict is not the same as testing predicted frequencies against outcomes.