Trifecta Signals

Back to Learn

Sample Size and Statistical Significance for Investors

Why sample size, outliers, and overlapping signals matter when reading investing statistics, with simple worked examples of how much a percentage can really be trusted.

Last reviewed: November 2026.

A percentage on its own can be misleading. "70% of signals gained 10%" sounds firm, but its reliability depends on how many signals sit behind it, how they were spread over time, and whether a few extreme cases are doing the work. This guide is a companion to How to Read a Track Record, going one level deeper into the statistics.

How uncertain is a percentage?

Suppose a method's past hit rate is measured at 70%. If the true underlying rate were the same, a different set of cases would still give slightly different results just by chance. The standard error of a proportion gives a rough sense of the spread:

standard error = √( p × (1 − p) ÷ n )

where p is the observed rate and n is the number of independent cases. A rough 95% range is about two standard errors either side. Using 70%:

The same "70%" is a loose estimate at n=20 and a fairly tight one at n=1,000. (Simplified illustration; it assumes the cases are independent.)

Averages and outliers

Return averages are even more fragile than hit rates, because a single extreme result can dominate. Consider ten hypothetical signals: nine gain 10% each and one gains 1,000%.

The mean says the typical signal more than doubled. The median says the typical signal gained 10%. Neither is wrong, but they describe different things, and the gap between them signals that outliers are driving the average. When a record shows very large individual gains, look at the median, at the distribution, and at what the average would be without the top few results.

Independence: the hidden assumption

The standard-error math assumes each case is an independent event. Investing data often violates this:

When cases cluster, the effective sample size is smaller than the raw count, and the real uncertainty is larger than the simple formula suggests. This is why a record covering many years and several market environments is more informative than one with a large count packed into a single stretch.

Statistical significance, plainly

A result is called "statistically significant" when it would be unlikely to arise by chance alone if there were no real effect. That is a narrow claim. It does not say the effect is large, that it will persist, or that it can be captured after costs. And when many patterns are tested, some will look significant purely by chance. This is the multiple-testing problem: try enough rules on past data and a few will appear to work.

Backtest versus live results

A backtest applies a rule to historical data after the fact. It can be honest and useful, but it carries risks that live, forward results do not: the rule may have been shaped while looking at the past, and data limits such as survivorship can flatter the outcome. A record that separates backtested history from results published in real time is easier to evaluate.

A checklist

Key points

Related: How to Read a Track Record · Drawdowns and Position Sizing · Glossary

Educational material only — not investment advice.