Mathematical Accuracy

[Mighty]

Is there an accuracy issue with ASP/VBScript with numbers with several decimal places. For example, I am testing a value to see if it is within specification. I have the following code:

if (CDbl(trim(value)) < CDbl(limit - tolerance)) or (CDbl(trim(value)) > CDbl(limit + tolerance)) then
checkLimits = &quot; STYLE=&quot;&quot;color: red;&quot;&quot;&quot;
pass = false
else
checkLimits = &quot;&quot;
end if

In this case, the value is 0.0027 and the limit is 0.0024 and the tolerance is 0.0003. So basically I am saying:

if 0.0027 < 0.0021 or 0.0027 > 0.0027 then
pass = false
end if

I have checked all the values and printed them out but it always fails saying that 0.0027 is greater than 0.0027. However, if I modify the data so that it is equal to the lower limit (0.0021) it passes - it only seems to fail it when it is equal to the upper limit.

In addition, it only fails for tests with small values. I have another test where the upper limit is 6.1. If the value is equal to this limit, it passes.

This is really frustrating me - I would appreciate it if anyone has any ideas. Mighty

 

[mwolf00]

Shouldn't it fail?

0.0027 < 0.0021 False
0.0027 > 0.0027 False
Get the Best Answers! faq333-2924
&quot;A witty saying proves nothing.&quot; - Voltaire
mikewolf@tst-us.com

 

[Mighty]

No it shouldn't fail!
0.0027 is not less than 0.0021
0.0027 is not greater than 0.0027 Mighty

 

[mwolf00]

I'm having a rough time after a three-day weekend so please help me out... I think I'm still confused...

Does this help?

if (CDbl(trim(value)) - CDbl(limit - tolerance)< 0 ) or (CDbl(trim(value)) - CDbl(limit + tolerance) > 0) then
checkLimits = &quot; STYLE=&quot;&quot;color: red;&quot;&quot;&quot;
pass = false
else
checkLimits = &quot;&quot;
end if
Get the Best Answers! faq333-2924
&quot;A witty saying proves nothing.&quot; - Voltaire
mikewolf@tst-us.com

 

[Mighty]

mwolf000,

Thanks for the idea - I tried that out and it was still failing. So I printed out the result of the calculation.

The result was as follows:

0.0027 - 0.0027 = -4.33680868994202E-19

As this is less than zero, it was failing. Have you any idea why this just doesn't return zero??? Mighty

 

[mwolf00]

how are you establishing the values of value, limit and tolerance? Maybe there is something in that... Then other solution would be to allow yet another tolerance to your calculation (argh!)


if (CDbl(trim(value)) - CDbl(limit - tolerance)< 0 ) or (CDbl(trim(value)) - CDbl(limit + tolerance) > 0.00000001) then
checkLimits = &quot; STYLE=&quot;&quot;color: red;&quot;&quot;&quot;
pass = false
else
checkLimits = &quot;&quot;
end if
Get the Best Answers! faq333-2924
Is this an asp FAQ? faq333-3048

mikewolf@tst-us.com

 

[Mighty]

The value of &quot;value&quot; is inputted by the user. Limit and tolerance are specified and stored in a database.

I have got a way around it but I'm not too happy with it. I have used the following:

if (formatNumber((CDbl(value) - CDbl(limit - tolerance)),5) < 0) or (formatNumber((CDbl(value) - CDbl(limit + tolerance)), 5) > 0) then
pass = false
end if

This works but I don't really like it as a solution. Mighty

 

[mwolf00]

What type of database? How are you retrieving them?

SELECT convert(money,limit,2) from mySQLTable..... - will give you limit to a precision of 4 decimals.... Get the Best Answers! faq333-2924
Is this an asp FAQ? faq333-3048

mikewolf@tst-us.com

 

[Mighty]

It is an MS Access database. The values in the database for the specification are only in 4 decimal places. The problem might be coming from the value - this is stored as a string in the database and I am using CDbl to convert this to a number for the calculations. Mighty

 

[mwolf00]

Try this (so we can see what's happening):

response.write cDbl(1 - value & &quot;<br>&quot
response.write cDbl(1 - limit & &quot;<br>&quot
response.write cDbl(1 - tolerance & &quot;<br>&quot Get the Best Answers! faq333-2924
Is this an asp FAQ? faq333-3048

mikewolf@tst-us.com

 

[Mighty]

mwolf00,

Tried that out - inconclusive.
I got the proper answers when I outputted them to the screen. Mighty

 

[mwolf00]

Have you tried:

dim test
test = 0.0027000001
response.write cDbl(test) & &quot;<br>&quot;
test = (fix(test*10000))/10000
response.write cDbl(test) & &quot;<br>&quot;
Get the Best Answers! faq333-2924
Is this an asp FAQ? faq333-3048

mikewolf@tst-us.com