Double precision issue 2

[TCARPENTER]

Why does this line of code:

Dim x as Double
Dim y as Double
Dim Rise As Double
Dim Span As Double

For x = -1 * Span To Span Step 0.05
  y = x * x
  Debug.Print x; y
Next x

Produce:

-3  9 
-2.95  8.7025 
-2.9  8.41 
-2.85  8.1225 
-2.8  7.84 
-2.75  7.56250000000001 
-2.7  7.29000000000001 
-2.65  7.02250000000001 
-2.6  6.76000000000001 
-2.55  6.50250000000001 
-2.5  6.25000000000001 
-2.45  6.00250000000001 
-2.4  5.76000000000001 
-2.35  5.52250000000001 
-2.3  5.29000000000001 
-2.25  5.06250000000001 
-2.2  4.84000000000001 
-2.15  4.62250000000001 
-2.1  4.41000000000001 
-2.05  4.20250000000001 
-2  4.00000000000001 
-1.95  3.80250000000001 
-1.9  3.61000000000001 
-1.85  3.42250000000001 
-1.8  3.24000000000001 
-1.75  3.06250000000001 
-1.7  2.89000000000001 
-1.65  2.72250000000001 
-1.6  2.56000000000001 
-1.55  2.40250000000001 
-1.5  2.25000000000001 
-1.45  2.10250000000001 
-1.4  1.96000000000001 
-1.35  1.82250000000001 
-1.3  1.69000000000001 
-1.25  1.56250000000001 
-1.2  1.44000000000001 
-1.15  1.32250000000001 
-1.1  1.21000000000001 
-1.05  1.10250000000001 
-1  1.00000000000001 
-0.950000000000003  0.902500000000005 
-0.900000000000003  0.810000000000005 
-0.850000000000003  0.722500000000004 
-0.800000000000002  0.640000000000004 
-0.750000000000002  0.562500000000004 
-0.700000000000002  0.490000000000003 
-0.650000000000002  0.422500000000003 
-0.600000000000002  0.360000000000003 
-0.550000000000002  0.302500000000002 
-0.500000000000002  0.250000000000002 
-0.450000000000002  0.202500000000002 
-0.400000000000002  0.160000000000002 
-0.350000000000002  0.122500000000002 
-0.300000000000002  9.00000000000014E-02 
-0.250000000000002  6.25000000000011E-02 
-0.200000000000002  4.00000000000009E-02 
-0.150000000000002  2.25000000000007E-02 
-0.100000000000002  1.00000000000005E-02 
-5.00000000000023E-02  2.50000000000023E-03 
-2.28983498828939E-15  5.24334427359425E-30 
 4.99999999999977E-02  2.49999999999977E-03 
 9.99999999999977E-02  9.99999999999954E-03 
 0.149999999999998  2.24999999999993E-02 
 0.199999999999998  3.99999999999991E-02 
 0.249999999999998  6.24999999999989E-02 
 0.299999999999998  8.99999999999986E-02 
 0.349999999999998  0.122499999999998 
 0.399999999999998  0.159999999999998 
 0.449999999999998  0.202499999999998 
 0.499999999999998  0.249999999999998 
 0.549999999999998  0.302499999999997 
 0.599999999999998  0.359999999999997 
 0.649999999999998  0.422499999999997 
 0.699999999999998  0.489999999999997 
 0.749999999999998  0.562499999999997 
 0.799999999999998  0.639999999999997 
 0.849999999999998  0.722499999999997 
 0.899999999999998  0.809999999999996 
 0.949999999999998  0.902499999999996 
 0.999999999999998  0.999999999999996 
 1.05  1.1025 
 1.1  1.21 
 1.15  1.3225 
 1.2  1.44 
 1.25  1.5625 
 1.3  1.69 
 1.35  1.8225 
 1.4  1.96 
 1.45  2.1025 
 1.5  2.25 
 1.55  2.4025 
 1.6  2.56 
 1.65  2.7225 
 1.7  2.89 
 1.75  3.0625 
 1.8  3.24 
 1.85  3.4225 
 1.9  3.61 
 1.95  3.8025 
 2  4 
 2.05  4.2025 
 2.1  4.40999999999999 
 2.15  4.62249999999999 
 2.2  4.83999999999999 
 2.25  5.06249999999999 
 2.3  5.28999999999999 
 2.35  5.52249999999999 
 2.4  5.75999999999999 
 2.45  6.00249999999999 
 2.5  6.24999999999999 
 2.55  6.50249999999999 
 2.6  6.75999999999998 
 2.65  7.02249999999998 
 2.7  7.28999999999998 
 2.75  7.56249999999998 
 2.8  7.83999999999998 
 2.85  8.12249999999998 
 2.9  8.40999999999998 
 2.95  8.70249999999998 
 3  8.99999999999997

I get the precision is and can be off, but what I don't get is why it works for -3, but +3 comes out 8.99999999999997. How can I ensure I'm getting consistent return values - or can I? I've searched the forum and didn't see anything related to my issue. Most replies were either just truncate it, round it, or format it. for the inbetween number, I need the precision.

Any advice is greatly appreciated.

Thanks
Todd

 

[combo]

You are doing a series of additions that cumulate errors (For x = -1 * Span To Span Step 0.05 - adds 0,05). Try:

Dim Rise As Double
Dim Span As Double
Dim R As Integer

Span = 3
Rise = 0.05
R = Span / Rise

For i = -R To R Step 1
  x = i * Rise
  y = x * x
  Debug.Print x; y
Next i
End Sub

combo

 

[PHV]

Floating point is only approximation.
What about this ?
Debug.Print CDec(x); CDec

Hope This Helps, PH.
FAQ219-2884
FAQ181-2886

 

[combo]

Note At this time the Decimal data type can only be used within a Variant, that is, you cannot declare a variable to be of type Decimal. You can, however, create a Variant whose subtype is Decimal using the CDec function.

So the variables should be declared as Variant.
Anyway, my point was that redefinition of the loop allows to avoid cumulation of extra errors (60 times x=x+0.05).

combo

 

[TCARPENTER]

Thanks guys!

 

[PHV]

So the variables should be declared as Variant
Which variables ????

 

[TCARPENTER]

X and Y right, since those are the ones to be converted?

 

[combo]

There is a description how to use decimals in MSDN.

combo

 

[PHV]

The destination of the conversion should be defined as Variant, not the source !

 

[TCARPENTER]

Okay - that does make better sense. Thanks again guys!

 

[combo]

The destination of the conversion should be defined as Variant, not the source !

What's the use of converting less precise calculations (in Doubles) to higher resolution output only?

combo

 

[PHV]

I personally never use floating point math as I don't play with scientific data but business and thus can't deal with approximations.