Outliers Explained: How to Find Them, When to Keep Them, and a Worked Example

An outlier is a value that sits far away from the rest of a dataset. It might be a typing error, a faulty sensor, a genuinely exceptional event, or the first sign of something new. The distinction matters enormously: deleting a real discovery because it looked strange is as bad as keeping a data-entry error because nobody checked. Outlier detection is therefore not a mechanical rule for deleting awkward numbers. It is a screening step that flags values worth investigating, after which human judgement decides what to do with them. The two most widely used screening methods are the interquartile range method and the z-score method, and understanding both will cover most practical situations.

The interquartile range method, usually shortened to IQR, works from the quartiles of the data. The first quartile, Q1, is the value below which roughly a quarter of the data falls. The third quartile, Q3, is the value below which roughly three quarters fall. The IQR is simply Q3 minus Q1, and it measures the spread of the middle half of the data. The standard fences are drawn at Q1 minus 1.5 times the IQR and at Q3 plus 1.5 times the IQR. Any value outside those fences is flagged as an outlier. The factor 1.5 is a convention introduced by the statistician John Tukey; it is not a law of nature, but it has proven to be a sensible default across many kinds of data, flagging genuinely unusual values without crying wolf on ordinary variation.

Consider a worked example that you can verify by hand. A small dataset of ten delivery times in minutes is: 12, 14, 15, 15, 16, 17, 18, 19, 21, 45. The data is already sorted, which is the necessary first step. To find Q1, take the lower half, which is 12, 14, 15, 15, 16, and its median, the middle value, is 15. To find Q3, take the upper half, which is 17, 18, 19, 21, 45, and its median is 19. The IQR is therefore 19 − 15, which is 4. The lower fence is 15 − 1.5 × 4, which is 15 − 6, which is 9. The upper fence is 19 + 1.5 × 4, which is 19 + 6, which is 25. The value 45 lies above the upper fence of 25, so it is flagged as an outlier, while every other value sits comfortably inside the fences. The method has done its job: it has pointed at the one value that needs a story.

The z-score method approaches the same question from a different angle. A z-score measures how many standard deviations a value sits from the mean, computed as the value minus the mean, divided by the standard deviation. Values with a z-score beyond about 3, or beyond −3, are commonly flagged. Apply it to the same dataset. The sum of the ten values is 12 + 14 + 15 + 15 + 16 + 17 + 18 + 19 + 21 + 45, which is 192, so the mean is 192 / 10, which is 19.2. The standard deviation of this set is about 9.4. The z-score of 45 is therefore (45 − 19.2) / 9.4, which is 25.8 / 9.4, about 2.74. That is below the usual threshold of 3, so the z-score method with a threshold of 3 would not flag 45 in this small dataset, while the IQR method did. This disagreement is instructive rather than embarrassing: with only ten points, one extreme value inflates both the mean and the standard deviation, which pulls the z-score back towards respectability. The IQR method is more robust here precisely because quartiles resist the pull of a single extreme value.

That sensitivity difference guides which method to use. The IQR method needs no assumption about the shape of the distribution, works well on skewed data, and stays reliable in small datasets, which makes it the better default for most everyday analysis. The z-score method is natural when data is roughly bell-shaped and the dataset is large enough that one value cannot dominate the mean and standard deviation. Some analysts use a stricter z-score threshold of 2.5 for small samples, or a modified z-score based on the median, which combines the robustness of the IQR approach with the familiar standard-deviation scale. Whatever method you choose, report it alongside the result: "flagged by the IQR method" is a claim someone can reproduce, while "looked like an outlier" is not.

Flagging is only the beginning. The next step is investigation, and the possible verdicts are more varied than most people expect. The value may be a plain error: a decimal point slipped, a unit was misrecorded, or two fields were swapped. In that case correct it if the true value can be established, or remove it with a note if it cannot. It may be a measurement failure: a sensor glitched or a stopwatch was misread, in which case the value carries no information and should be excluded from analysis but retained in the raw record with an explanation. It may be a genuine but unrepresentative event: a delivery delayed by a road closure is real, but including it in an average delivery time misleads anyone planning normal operations. Or it may be the most important observation in the dataset: the first case of a new failure mode, a record-breaking performance, or an early signal of a change. Deleting that last kind is how organisations miss their most valuable warnings.

To see the stakes, compare the mean with and without the flagged value in the example. With all ten values the mean is 19.2 minutes. Without the 45, the sum of the remaining nine values is 192 − 45, which is 147, and the mean is 147 / 9, which is 16.33 minutes. The single value moves the average by almost three minutes, about eighteen percent. The median, by contrast, is barely touched: with all ten values it is the average of the fifth and sixth values, 16 and 17, which is 16.5, and without the outlier it is 17. That stability is why the median is often reported alongside, or instead of, the mean when outliers are suspected. Reporting both, and stating clearly whether flagged values were included or excluded, is the mark of honest analysis.

A few practical habits will keep outlier handling defensible. Never delete values silently: keep the raw data intact, record which values were flagged, by which method and threshold, and what decision was taken for each. Never let an automated rule delete data without review in any analysis that affects people, money, or safety. Be extra careful with small datasets, where every value carries more weight and where the difference between "outlier" and "small sample" is genuinely hard to draw. And remember that a value can be unusual in one variable but perfectly normal in context: a high delivery time on a day of storms, or a high spend by a customer who always spends highly, may be explained by another variable rather than flagged as an error. Outlier detection on one column alone is a starting point, not a verdict.

Try it next: use the Outlier Calculator on your own dataset, then explore the spread with the Standard Deviation Calculator and the Z-Score Calculator. For background reading, see Profit, Margin and Break-Even Explained.