rounding problem. round to 10th place? 1

[simon551]

Hi.
I have some equations that require very long decimals. I'm not having luck getting it to go past 4 places. Is there a master setting for this? I have my properties set to 10 decimals but it's rounding at the 4th.

Thanks in advance.

 

[earthandfire]

Does it round in the actual calculation or just in the display. I {i]believe[/i] Microsoft's floating point calculations go up to 15 decimal places, though I may be wrong there.

For display purposes you could try:

Format(variable,"0.0000000000")


Hope this helps.

 

[simon551]

Thanks for the reply. It's in the calculation. Even though the display shows 10 places, everything past the 4th is 0.

????

 

[earthandfire]

Try this in the Immediate window:

?format(123.4567891256789,"0.0000000000")

You should get this:

123.4567891257

Note that .....567 has rounded to .......57

By the way sorry about the bad italics around believe above - got it right this time!!

 

[earthandfire]

I've just tried it as part of a calculation, with and without formatting - very advanced calculation, I divided by 2.

?format(123.4567891256789/2,"0.0000000000")
61.7283945628

?123.4567891256789/2
61.7283945628394

 

[earthandfire]

By the way, what data type are you using?

Currency only uses 4 decimal places

Double & Single are the true Floating Point data types. I think they are IEEEE (however many Es) compliant.

 

[simon551]

thanks dude.
not sure what you are saying there. pretty new to this stuff. I changed the type to number and bumped up precision. That worked. I was using currency.

Take care.