Tek-Tips is the largest IT community on the Internet today!

Members share and learn making Tek-Tips Forums the best source of peer-reviewed technical information on the Internet!

  • Congratulations SkipVought on being selected by the Tek-Tips community for having the most helpful posts in the forums last week. Way to Go!

Floating/Double Precision Problem

Status
Not open for further replies.
Jan 20, 2005
180
0
0
US
I am reading from a file a floating point number thats precise to 5 decimals. However when I store it, it sometimes gets changed to .xxxx(x-1)999999916 or .xxxxx0000000011 or similar.
I need this to be .xxxxx0000000000 for later presicion work. How do I go about fixing this?
 
Floating point numbers are inexact approximations of real numbers. There is only so much you can do with a finite number of bits.

Sooner or later, you will always end up with the situation you describe because it will have to choose between the next lowest or next highest approximate value which is closest to your actual value, and that choice will be represented in the manner you describe.

If you need it exact, then you need a different data type, floats and doubles will not meet your need.

--
 
Addition: don't forget that floating point numbers represented as binaries (radix 16) - so you may see a very long decimal fraction when you print a binary fraction as a decimal...
 
After all the work, the easiest solution seems to present itself for me... I had totally forgotten about floating numbers were as such. For me I just ended up outputing my data to 1 extra place, so when it was inputted elsewhere it rounded it correctly.
 
Status
Not open for further replies.

Part and Inventory Search

Sponsor

Back
Top