Statistical significance

A result is significant when the gap between two measurements would rarely turn up by chance alone, given the sample size. Sample size does most of the work here. Two thumbnails split across four hundred impressions can differ by a couple of percentage points for no reason at all, while the same gap across four hundred thousand impressions is real.

Creators trip on the word in two ways. Significance is a statement about noise, and it says nothing about whether the difference is big enough to care about. A result that clears the bar once also needs to hold up when the test runs again.