Strange result when subtracting

[emozley]

Hi,

For some reason if I perform the calculation:

12.425 - 11.75

in vbscript I get 0.675000000000001

Do you know why this is happening? I am wary of using formatnumber(n, 3) as this won't fix the underlying problem.

Thanks very much

Ed

 

[Remou]

You need to use the right type for maths, for example:

ccur(12.425) - cdbl(11.75)

http://lessthandot.com

 

[strongm]

Even if you use the 'right type' you will still get oddities like this on occassion, because there are certain floating point numbers that cannot be accurately represented in binary

 

[emozley]

Yep you're right - i've just tried cdbl and it still happens. If I can format the answer to 3 decimal places but is there a way of making it so that an answer that had for example 1 decimal place wouldn't have trailing zeros?

Thanks!

Ed

 

[Remou]

Some futher notes.

'Various data types:

Answer = "1. " & 1000.92 - 500.41 - 500.51 & vbCrLf
'Answer: -5.6843418860808E-14

'Currency  -922,337,203,685,477.5808 to 922,337,203,685,477.5807.
Answer = Answer & "2. " & CCur(1000.92) - 500.41 - 500.51  & vbCrLf
'Answer: 0

'Double  -1.79769313486231E308 to -4.94065645841247E-324 for negative values;
'4.94065645841247E-324 to 1.79769313486232E308 for positive values.
Answer = Answer & "3. " & CDbl(1000.92) - 500.41 - 500.51  & vbCrLf
'Answer: -5.6843418860808E-14

'Decimal +/-79,228,162,514,264,337,593,543,950,335 for zero-scaled numbers,
'that is, numbers with no decimal places. For numbers with 28 decimal places,
'the range is +/-7.9228162514264337593543950335. The smallest possible
'non-zero number is 0.0000000000000000000000000001.
'Decimal is not available in VBScript

'Integer -32,768 to 32,767; fractions are rounded.
Answer = Answer & "4. " & CInt(1000.92) - 500.41 - 500.51  & vbCrLf
'Answer: 7.99999999999841E-02

'Long -2,147,483,648 to 2,147,483,647; fractions are rounded.
Answer = Answer & "5. " & CLng(1000.92) - 500.41 - 500.51  & vbCrLf
'Answer: 7.99999999999841E-02

'Single -3.402823E38 to -1.401298E-45 for negative values;
'1.401298E-45 to 3.402823E38 for positive values.
Answer = Answer & "6. " & CSng(1000.92) - 500.41 - 500.51  & vbCrLf
'Answer: -1.70898437659162E-05

Msgbox Answer

http://lessthandot.com

 

[emozley]

Thanks for this that has cleared things up a great deal. In the end I used the following code which seems a little clumsy but has done the job:

TimeLeft = UserArray(3, i)-UserArray(4, i)
Remainder=TimeLeft-Fix(TimeLeft)
Select Case Len(Remainder)
Case 0, 1, 2, 3, 4
Response.Write(TimeLeft)
Case Else
Response.Write(FormatNumber(TimeLeft, 3))
End Select

 

[PHV]

Have you tried this ?

TimeLeft = UserArray(3, i)-UserArray(4, i)
Response.Write CCur(TimeLeft)

Hope This Helps, PH.
FAQ219-2884
FAQ181-2886