numbers slightly different immediately after equal assignment 3

[voodoojon]

Can anyone explain this?

I've been stepping thru my code (vb6)with f8 to find the source of a wrong value. here's the line where it's assigned

variables before:
temp: 100509.222 (type: variant)
thecontrol: undefined (type: control)

thecontrol = temp

variables after:
temp: 100509.222 (type: variant)
thecontrol: 100509.22199999999(type: control)



it's bizarre. the equals sign doesn't work!


Jon

One day I will find a signature worthy of this space. That day has not yet come.

 

[ADoozer]

probably something to do with converting from single to double. (it has been discussed somewhere on this forum quite a bit.. just do a search for "decimal precision" or something similar)

try using format statement when assigning to thecontrol

If somethings hard to do, its not worth doing - Homer Simpson

 

[Golom]

Variant" is not a basic data type the way "Double", "Integer", "Long", etc. are. A variant is a generic type capable of containing data of any type and you identify the specific format of the data in a variant by means of the sub-type.

When you assign a value such as 100509.222 to a control(such as command button, text box, etc.), you are really assigning it to some default property of the control such as .Text, .Caption or .Value which will in turn have its own basic data type.

VB is probably converting the variant sub type to the required property data type and, in the process, changing it's precision. To check this, try some simple code like

Debug.Print TypeName(Temp)
Debug.Print TypeName(TheControl)

to determine what the basic source and destination data types really are.

 

[DrJavaJoe]

I'm with Adoozer the problem is how the computer stores floating point decimals. Try writing .222 in binary.

Here's a couple sites to look at.

http://en.wikipedia.org/wiki/Binary_numeral_system

http://babbage.cs.qc.edu/courses/cs341/IEEE-754.html

Two strings walk into a bar. The first string says to the bartender: 'Bartender, I'll have a beer. u.5n$x5t?*&4ru!2[sACC~ErJ'. The second string says: 'Pardon my friend, he isn't NULL terminated'.

 

[chiph]

Yup. You're seeing round-off happen when the computer tries to store a repeating fractional value in a Double datatype. It's just something you have to code for.

Chip H.


____________________________________________________________________
If you want to get the best response to a question, please read FAQ222-2244 first

 

[chiph]

Typically, you decide to how many decimal places are significant to your app, and discard anything beyond that for your comparison.

Chip H.


____________________________________________________________________
If you want to get the best response to a question, please read FAQ222-2244 first

 

[voodoojon]

cheers for the help, now I've got a few things to try.

unfortunately this part of the code deals with showing the onscreen data for pretty much every datacontrol in the app(~100 front end forms), in any type or format so any changes to it could cause any amount in disruption somewhere else.

Jon

One day I will find a signature worthy of this space. That day has not yet come.

 

[BobRodes]

I tried this

Dim x
Dim ctl As Control
x = 100509.222
Set ctl = Text1
ctl = x

and don't get the behavior you describe. Can you give more detail?

Bob