Decimal number getting rounded up 3

[mais123]

Hi, I have a function that does some calc and returns a decimal which i try to put into a datarow. so:
decimal myvar= SomeCalc(..);
//lets assume myvar=0.9 after calculation
//update the output row in place with value
newOutputRow["column"]= myvar;

But then when I do this in command window:

? newOutputRow[["column"] I get 1 back, not 0.9. Some the rounding happends when I assign the value to table cell as it is 0.9 after the procedure calll

I seen the same with other numbers, like 0.012 gets rounded to 0.
I tried using decimal and varchar column types for the output table:
Output.Columns.Add(new DataColumn("column", SqlDbType.VarChar.GetType())); or
Output.Columns.Add(new DataColumn("column", SqlDbType.Decimal.GetType()));

Why is it getting rounded up???

Thanks!

 

[JurkMonkey]

SomeCalc(..);

Make sure you are not doing any mixing of DataTypes here.

If you multiply or divide a float or double by an integer you can start to lose precision. Check your value coming out of SomeCalc and see if it is what you expected. If this is the case then look further into how you are storing these values.

http://www.ubuntu.com/

 

[jmeckley]

if you do any rounding you may want to incorporate a Money Object to encapsulate the logic.

Jason Meckley
Programmer
Specialty Bakers, Inc.

 

[chiph]

Like JurkMonkey says -- if you start with a decimal datatype, you need to make sure any intermediate math operations on it involve ONLY other decimals. Otherwise the runtime will do an implicit datatype conversion, and you stand a chance of losing precision.

Chip H.


____________________________________________________________________
www.chipholland.com

 

[mais123]

thanks guys, it turned out to be i was setting the column type incorrectly