Biased vs. Unbiased Standard Deviation and Variance

[minus27]

Thanks for the jq recipes! jq is very powerful, but horribly underdocumented.

While looking at your standard deviation recipe, I threw the numbers into Google Sheets (and Excel) and your the results did not match - which bothered me. Long story short, your recipe is for the biased standard deviation (and variation) while the Google Sheets and Excel functions calculate the unbiased standard deviation (and variation). While I do not fully appreciate correcting for biases, I am happy to know why the results did not match.

In your existing recipe, I did the following:

• changed mean to simply def mean: add / length;
• removed pow2 altogether and replaced its call with . * .
• simplified your data with just an array of the same numbers

Here is where I ended up:

def mean: add / length;
def variance_biased: . | mean as $mean | map_values(. - $mean | . * .) | mean;
def stdev_biased: . | variance_biased | sqrt;
def variance_unbiased: . | mean as $mean | map_values(. - $mean | . * .) | add / (length-1);
def stdev_unbiased: . | variance_unbiased | sqrt;

[remy]

I'll update my post with biased - thank you 👍