Confidence interval

An interval describes uncertainty in an estimate

A sample mean is usually not exactly the population mean. A confidence interval uses the observed sample and a specified method to express uncertainty about the parameter being estimated.

For a conventional 95% procedure, repeated samples generated under the assumed model would produce intervals containing the fixed true parameter about 95% of the time. That is different from saying 95% of individual observations fall inside the interval.

An engineering example

Suppose an illustrative experiment compares build times before and after a cache change. The estimated reduction is useful, but its interval may be wide because the sample contains only a few builds with very different workloads.

More observations can improve precision. However, repeatedly measuring the same unusual workload does not make the result representative of every repository. The design of the comparison still matters.

Precision is not the same as validity

A narrow interval can surround a biased estimate. If the new configuration was tested only on smaller projects, uncertainty calculations do not correct the unequal comparison.

Check whether observations are independent, how the sample was selected, and whether the statistical method fits the data. Builds from the same project or measurements from the same developer may be related. Treating every record as independent can exaggerate precision.

For a decision, inspect the effect size and its practical importance as well as the interval. A precisely measured improvement can still be too small to justify an expensive rollout.