float question

[Jeremy21b]

Hello.

I have a calculation using ASP which generates a small number like 0.0001. My problem is that ASP converts it into something like 1.0000000000E-02
Is there any easy way to avoid this or convert it to a simple float?

 

[onpnt]

cDbl()

exactely how is ASP (vbscript) converting the numer? is it actually a database converting it?

_____________________________________________________________________
[sub]You can accomplish anything in life, provided that you do not mind who gets credit.
Harry S. Truman[/sub]

http://www.onpntwebdesigns.com

 

[dmhirsch]

1.0000000000E-02 and 0.0001 are the same value, formatted differently. VBScript's FormatNumber(Expression [,NumDigitsAfterDecimal [,IncludeLeadingDigit [,UseParensForNegativeNumbers [,GroupDigits]]]]) function will allow you to format it however you want for display.

 

[dmhirsch]

Actually, 1.0000000000E-02 and 0.0001 are not the same number. 1.0000000000E-04 and 0.0001 are the same number. I'm hoping that the latter is what's happening (you said "something like&quot...

 

[onpnt]

I knew there was a reason I should ahve gone to math class....

really though, yes of course you're right. and my first sugestion after thinking about it with cDbl() was incorrect as it would format to 0.01.

_____________________________________________________________________
[sub]You can accomplish anything in life, provided that you do not mind who gets credit.
Harry S. Truman[/sub]

http://www.onpntwebdesigns.com

 

[Jeremy21b]

Yes the numbers are equivalent, just the formatting is not what I want. I am unfamiliar with that VBScript FormatNumber function. How exactly could I use it to format my number to 0.0001 instead of 1.0000E-04.

There isn't a database converting it. When I use cDbl() it still leaves it in the expanded form instead of 0.0001

Basically, its trying to calculate the percent difference between 2 numbers. Whenever the percentage is very small ASP insists on automatically formatting it with the 000E-02 stuff.

strDiff = (1 - strDiff) * 100

 

[onpnt]

1.0000000000E-04 and 0.0001 are the same number

told you...math classes

_____________________________________________________________________
[sub]You can accomplish anything in life, provided that you do not mind who gets credit.
Harry S. Truman[/sub]

http://www.onpntwebdesigns.com

 

[Jeremy21b]

Ok i got the format number function working. Thanks for the immediate help.

 

[dmhirsch]

Response.Write(FormatNumber(myNumber, 5))

would do the trick, leaving 5 places after the decimal.

 

[onpnt]

the more i think about it wouldn't
strDiff = (1 - cDbl(strDiff) * 100)
result correctly if the numeric value in strDiff is 0.0001



_____________________________________________________________________
[sub]You can accomplish anything in life, provided that you do not mind who gets credit.
Harry S. Truman[/sub]

http://www.onpntwebdesigns.com

 

[dmhirsch]

onpnt, it sounds like it was just a formatting issue, not a data conversion/storage issue.

 

[onpnt]

I'll stop trying to think... thanks

_____________________________________________________________________
[sub]You can accomplish anything in life, provided that you do not mind who gets credit.
Harry S. Truman[/sub]

http://www.onpntwebdesigns.com