Tek-Tips is the largest IT community on the Internet today!

Members share and learn making Tek-Tips Forums the best source of peer-reviewed technical information on the Internet!

  • Congratulations Mike Lewis on being selected by the Tek-Tips community for having the most helpful posts in the forums last week. Way to Go!

Comparing values

Status
Not open for further replies.

AOLBoy

IS-IT--Management
May 7, 2002
68
GB
I have a strange situation and I do not what to do about it.

I have two fields

Dim A As Double
Dim B As Double

In my code I am accumulating values from Excel cells and then I am doing a comparison of A and B.

With the 'Add Watch' facility in the debugger I have
A = -3263.18
B = -3263.18

In my code the following check returns true:

If A <> B

I added a new field x = A - B
and this contains the value

4.54747350886464E-13

This makes no sense to me

Can anybody please help me out here? The code works OK in a VB6 application.



 
if you format "a" and "b" (look at the format function) then compare this issue should disappear , the number shown is very small

 
attually the round function is more applicable , (sorry it is early in the morning , not woken up yet)
 
I think it depends on how the numbers are generated and why you're comparing them. Do they really have to be rigorously equal, or is there a tolerance on your comparison. That is, you can compare using: if A-B<tolerance ....

If the numbers are generated from calculations, and gathered in different paths, they can, actually, be different due to truncating and "machine noise".

_________________
Bob Rashkin
 



The PRECISION of Double is not precise as you have experienced.

Best to use integer arithmetic OR establish a tolerance in your expression...
Code:
Const TOLER = .0001

if Abs(a - b) < TOLER then


Skip,
[sub]
[glasses] [red][/red]
[tongue][/sub]
 
Thanks very much for all your comments and setting me straight on this issue.
 
Status
Not open for further replies.

Part and Inventory Search

Sponsor

Back
Top