Tek-Tips is the largest IT community on the Internet today!

Members share and learn making Tek-Tips Forums the best source of peer-reviewed technical information on the Internet!

  • Congratulations gkittelson on being selected by the Tek-Tips community for having the most helpful posts in the forums last week. Way to Go!

Calculation that returns incomprehensible values

Status
Not open for further replies.

caifa

Programmer
Jul 21, 2009
3
IT
I have a field, let's call it Field D, that is the result of a calculation.
field D = field A + Field B - Field C

sometimes the calculation works, but in some cases it returns incomprehensible values, i.e. -8,8817842E-16 (whereas I expected to find 0)
 
It's all due to the wonderful ways which computers (and hence databases) store decimal numbers using binary methods. Just use ROUND or formatting where necessary to remove or mask this problem.

soi la, soi carré
 
Status
Not open for further replies.

Part and Inventory Search

Sponsor

Back
Top