Tek-Tips is the largest IT community on the Internet today!
Members share and learn making Tek-Tips Forums the best source of peer-reviewed technical information on the Internet!
-
Congratulations SkipVought on being selected by the Tek-Tips community for having the most helpful posts in the forums last week. Way to Go!
Sin(pi) not zero
- Thread starter fawkes
- Start date
- Thread starter
- #2
What type of variable are you assigning the result to?
If you assign it to an Integer it comes out as zero e.g.
and if you assing it to a double it comes out as 0.00000000000000012246063538223773 e.g.
____________________________________________________________
Need help finding an answer?
Try the Search Facility or read FAQ222-2244 on how to get better results.
If you assign it to an Integer it comes out as zero e.g.
Code:
Dim i As Integer = System.Math.Sin(System.Math.PI)
Code:
Dim i As Integer = System.Math.Sin(System.Math.PI)____________________________________________________________
Need help finding an answer?
Try the Search Facility or read FAQ222-2244 on how to get better results.
- Thread starter
- #4
Oops...just noticed my last statement had a Type of Integer. It should have read:
and if you assing it to a double it comes out as 0.00000000000000012246063538223773 e.g.
____________________________________________________________
Need help finding an answer?
Try the Search Facility or read FAQ222-2244 on how to get better results.
and if you assing it to a double it comes out as 0.00000000000000012246063538223773 e.g.
Code:
Dim i As Double = System.Math.Sin(System.Math.PI)____________________________________________________________
Need help finding an answer?
Try the Search Facility or read FAQ222-2244 on how to get better results.
And I must stop copying that bit of text that says "and if you assing it"!
____________________________________________________________
Need help finding an answer?
Try the Search Facility or read FAQ222-2244 on how to get better results.
____________________________________________________________
Need help finding an answer?
Try the Search Facility or read FAQ222-2244 on how to get better results.
From MSDN:
System.Double and System.Single have inexact precision of decimal digits because internally they are a floating point binary numbers. As such, beyond a certain number of significant digits, their decimal representation becomes approximate.
Two strings walk into a bar. The first string says to the bartender: 'Bartender, I'll have a beer. u.5n$x5t?*&4ru!2[sACC~ErJ'. The second string says: 'Pardon my friend, he isn't NULL terminated'.
System.Double and System.Single have inexact precision of decimal digits because internally they are a floating point binary numbers. As such, beyond a certain number of significant digits, their decimal representation becomes approximate.
Two strings walk into a bar. The first string says to the bartender: 'Bartender, I'll have a beer. u.5n$x5t?*&4ru!2[sACC~ErJ'. The second string says: 'Pardon my friend, he isn't NULL terminated'.
Yup. It's a binary representation rounding error.
Chip H.
____________________________________________________________________
Donate to Katrina relief:
If you want to get the best response to a question, please read FAQ222-2244 first
Chip H.
____________________________________________________________________
Donate to Katrina relief:
If you want to get the best response to a question, please read FAQ222-2244 first
- Status
