StDev error

[johnrowse]

Hi

I have a weird error, when using the StDev method on certain number of values.

In the spreadsheet itself, using the formula StDev() on:

1.428,1.428,1.428,1.428,1.428,1.428 [(x6)] = 0 CORRECT
1.428,1.428,1.428,1.428,1.428,1.428, 1.428 [(x7)] = 0 CORRECT

In VBA

1.428,1.428,1.428,1.428,1.428,1.428 [(x6)] = 0 CORRECT
1.428,1.428,1.428,1.428,1.428,1.428, 1.428 [(x7)] = 2.39835584666591-16 WRONG
1.428,1.428,1.428,1.428,1.428,1.428,1.428,1.428 [(x8)] = 0 CORRECT

To be clear, the above values come from a range of cells, and as I add another value, I am keeping the original value. So the values in the incorrect calculation of SD, are also in the correct ones, so it is not like there is an erroneous value being picked up.

I have also tried all the different versions of StDev (P/S...) and they all do exactly the same.

Does anyone know ehere I am going wrong?

Thanks

John

 

[johnrowse]

Sorry

I guess writting the problem out to myself, was enough for me to solve it. 2.39835584666591-16 is actually an incrediably small number (need to shift number by 16 decimal places to the right). This very minor inaccuracy, must be due to floating point precision rounding. I can round this level of inaccuracy out, without impact.

Unless i am still missing something and i am making a major assumption?

Thanks though

John

 

[PHV]

You're right.
Floating point are only approximations.

Hope This Helps, PH.
FAQ219-2884
FAQ181-2886