Displaying calulated values without losing the precision 4

[CP60]

Debug.? 12345678.12345678# + 123456.67#

So, how do I display the right sum without losing the precision?

Thank you!

 

[Andrzejek]

When I do:[tt]
Debug.Print 12345678.1234568 + 123456.67[/tt]

I get:
[tt]12469134.79345689[/tt]

What do you get that is wrong...?

Have fun.

---- Andy

A bus station is where a bus stops. A train station is where a train stops. On my desk, I have a work station.

 

[CP60]

Sorry, I should have mentioned that I was testing it in the Debug window (if you'll notice, the values you posted are different from what I posted).
(even though, I still do not get what you have - your last digit is a 9)

Try:

Debug.? CDbl("12345678.12345678") + CDbl("123456.67")

I get:    12469134,7934568 
Expected: 12469134,793456[b][COLOR=#EF2929]7[/color][/b]8

 

[gmmastros]

try the decimal data type.

Debug.? CDec("12345678.12345678") + CDec("123456.67")

-George
Microsoft SQL Server MVP
My Blogs
SQLCop
twitter
"The great things about standards is that there are so many to choose from." - Fortune Cookie Wisdom

 

[CP60]

@gmmastros

I guess I was tring to make it too simple, and mislead things in my second post, using the values as strings. I should have used doubles, as in the OP.

That may be fine, using CDec(), if there was a Decimal type.

But, I would need to always store the input values in strings and for each calculation, use CDec() in order for that to work
(there many more calculations to be done).

Please allow me to clear up my mistake:

    Dim x As Double
    Dim y As Double
    Dim z As Double
    
    x = "123456789.123456789" 'Store Input from user
    y = "123456.67"  'Store Input from user
    z = x + y
    
    Debug.Print CDec(z)

 

[SkipVought]

Observe:

debug.Print 1234567.12345678#
debug.Print 1234567[highlight #FCE94F]8[/highlight].12345678#
debug.Print 1234567[highlight #FCE94F]89[/highlight].12345678#

15 places?

 

[CP60]

I guess so.

X = CDbl(12345678.12345678)
Debug.? x + x + x + x + x + x + x + x

The values will be calculated right.


Just the displayed value will be cut off and rounded and cannot be shown to a higher precision.

Thanks

 

[CP60]

I guess I could just create the function in a Visual Basic 2005 dll, using the decimal type (29 significant digits) and call that from VB6, passing the results back as a string.

 

[strongm]

>Please allow me to clear up my mistake:


Please allow me to give you an alternative:

[blue]    Dim x as Variant
    Dim y As Variant
    Dim z As Variant
    
    x = CDec("123456789.123456789")  'Store Input from user
    y = CDec("123456.67")  'Store Input from user
    z = x + y
    
    Debug.Print z[/blue]

This has exactly the same accuracy as your idea of using VB 2005 decimal type

 

[CP60]

Yes, prior to my last post yesterday, I saw that on another site, posted by dilettante:

http://www.vbforums.com/showthread....30433b22e6af&p=4033201&viewfull=1#post4033201

and though the Decimal type is also slow, I decided to go with that, rather than using the variant type.

I guess I was, to begin with, just naive concerning this and have since read up some on the IEEE 754 specification.

My thanks to everyone!

 

[dilettante]

The code there was using Variants as containers for Decimal too. There is no separate Decimal type in VB6, only the Decimal subtyped Variant.

When you omit the type you get a Variant. These do the same thing:

Dim A
Dim B As Variant

 

[CP60]

Yes, I realize that.

What I meant with
"and though the Decimal type is also slow, I decided to go with that,"
is to do what I said in my post prior to that: To use the stronger Decimal type variable in a VS 2005 dll and run it through that.

Thank you.

 

[strongm]

>To use the stronger Decimal type variable in a VS 2005 dll

Stronger? Please explain why you think this.

BTW, VB6/VBA documentation on decimal subtype:

96-bit (12-byte) integer numbers scaled by a variable power of 10. The scaling factor specifies the number of digits to the right of the decimal point; it ranges from 0 through 28. With a scale of 0 (no decimal places), the largest possible value is +/-79,228,162,514,264,337,593,543,950,335 (+/-7.9228162514264337593543950335E+28). With 28 decimal places, the largest value is +/-7.9228162514264337593543950335, and the smallest nonzero value is +/-0.0000000000000000000000000001 (+/-1E-28)

And here's what VB.NET says:

96-bit (12-byte) integer numbers scaled by a variable power of 10. The scaling factor specifies the number of digits to the right of the decimal point; it ranges from 0 through 28. With a scale of 0 (no decimal places), the largest possible value is +/-79,228,162,514,264,337,593,543,950,335 (+/-7.9228162514264337593543950335E+28). With 28 decimal places, the largest value is +/-7.9228162514264337593543950335, and the smallest nonzero value is +/-0.0000000000000000000000000001 (+/-1E-28)

Seriously. These are cut'n'pasted from each set of documentation. Can you spot a difference? Of any sort?

 

[CP60]

I did write "stronger Decimal type variable".

Ok, then: "variable declared explicitly as a Decimal type".

But if taken in context to my previous post to that one, I assumed it would be understood that I was referring to my preference in using a variable declared as a Decimal type over that a Variant type.

So, though I wouldn't have imagined that there would have been a problem with what I wrote, I am sorry that there was.

BTW: Is there an explicit definiton of "Strong typing" or "Weak typing" defined somewhere, as I understand you are referring to? (I didn't find any on the MS site, so I looked in the not-always-reliable Wikipedia).

 

[CP60]

I had searched for "Strong Type" instead of "Strong typing" on the MS Site. So now I have found this: Link

 

[strongm]

>my preference in using a variable declared as a Decimal type over that a Variant type

But you are not. Not in VB6. The fact that it is typed in VB.NET is neither here nor there.