Single/Double conversion

[gkrogers]

I've come across this anomaly and wondered if anybody could shed some light on it for me. I've fixed the problem by using a Single instead of a Double, but am curious as to why the problem arises in the first place.

* I'm using a SQL Server database containing a table with a field that stores a decimal number to a precision of 4 decimal places.
* A stored procedure returns that number as a REAL.
* The stored procedure is called from VB and the return value (temporarily stored in a variant, in an array) is assigned to a Double variable.
* The value stored in the table is (for example) 1.5384. The value that ends up in the Double is (for example) 1.5384004020451. How/why does that happen? I'd have thought that the extra precision would be padded out with zeroes if necessary, not random digits.
* If I assign the value to a variable of type Single, then it's fine.

TIA,
gkrogers

 

[gkrogers]

Incidentally, this problem can be demonstrated in a much simpler way by typing "Debug.Print CDbl(CSng(1.5384))" in the Immediate window - you get 1.53840005397797.

Regards,
gkrogers

 

[CajunCenturion]

The reason for this behavior is the in-ability to store the fractional parts of decimal numbers exactly in binary notation. Good Luck
--------------
As a circle of light increases so does the circumference of darkness around it. - Albert Einstein

 

[jtm111]

I don't use SQL server much, but you said it STORED the number to 4 decimal precision. I don't think that's true. It might DISPLAY at 4 decimal precision but I doubt it would store a real number that way.

When you saw the double, you saw the actual stored value of the single.

I could be wrong.

 

[strongm]

It is as cajuncenturion explains.

Another trivial example that illustrates the point:
[tt]
Dim lp As Single

For lp = 0.1 To 1 Step 0.1
Debug.Print lp
Next

 

[gkrogers]

Thanks guys.

gkrogers