Problem with Double Data Types

[NSNewey]

Hi,

I've been scathing my head because a function that returns a Double has been giving me a strange result.

I have realised it's VBA and not my function which is at fault.

Can someone tell me why when you type...
?CDbl(110.4) - CDbl(110)
into the intermediate window the result is 0.400000000000006

I have now converted the function's return value using CDec() which returns 0.4 as I would expect.

Is there another way?

 

[SkipVought]

You need to read up on Floating Point Precision.

Skip,
[sub]
Just traded in my old subtlety...
for a NUANCE![/sub]

 

[TheAceMan1]

How are ya NSNewey . . .

Try:

[blue]? CCur(110.4) - CCur(110)[/blue]

Ahhhhh Hahhhhh!

[blue]SkipVought[/blue] is right! You need to read up on Floating Point Precision.

See Ya! . . . . . .

Be sure to see faq219-2884 [blue]Worthy Reading![/blue]
Also faq181-2886 [blue]Worthy Reading![/blue]

 

[strongm]

Or, if you know how the parser works:

? cdec(110.4) - 110

or

? 110.4 - cdec(110)