Tek-Tips is the largest IT community on the Internet today!

Members share and learn making Tek-Tips Forums the best source of peer-reviewed technical information on the Internet!

  • Congratulations Mike Lewis on being selected by the Tek-Tips community for having the most helpful posts in the forums last week. Way to Go!

How to deal with Decimal Number

Status
Not open for further replies.

vbdewd

Programmer
Oct 7, 2006
11
US
Greetings Folks,
This is an old issue but I still wonder if there's a quick solution.

When I multiply a number such as (0.0000001 * 1000000), The answer comes up to 0.099999999999999999 instead of 0.1

Aside from using parseFloat and/or .toFixed, can you tell me of a way to get the correct result of 0.1?

Many Thanks
 
In general there isn't! Google for 'floating point' to understand why.

If you want the best response to a question, please check out FAQ222-2244 first.
'If we're supposed to work in Hex, why have we only got A fingers?'
Drive a Steam Roller
 
thanks johnwm.

I do understand why but I was hoping somebody has found a soultion or some type of conversion.
 
Basically it is a built-in problem due to the fact that computers cannot natively understand decimal numbers, only binary numbers. No one yet has invented a five-way hardware computational switch.

Since your problem cannot be solved with hardware, you must do your computations to a large number of significant digits, then round to a lower number of significant digits to filter out the built in errors.

mmerlinn


"We've found by experience that people who are careless and sloppy writers are usually also careless and sloppy at thinking and coding. Answering questions for careless and sloppy thinkers is not rewarding." - Eric Steven Raymond
 
Status
Not open for further replies.

Part and Inventory Search

Sponsor

Back
Top