forcing precision 1

[tomdagliusa]

Hi,

I have an embedded version of tcl inside of an app, and I think the version of tcl is older than God. The problem is that to the following command:
set x [expr $membernum * 601.3439942]

I get a result of 3006.72
instead of 3006.719971

I tried forcing a type by adding a double cast to the
expr command, but it still gets truncated.

Suggestions?
Tom

 

[ddrillich]

Please try:

set z "121.343434"
set z2 [format "%10.2f" $z]

It returns in tcl 8.2: 121.34

Regards,
Dan

 

[ulis]

Try to set ::tcl_precision to 17.

From the tclvars page of the Tcl Manual:

tcl_precision
This variable controls the number of digits to generate when converting floating-point values to strings. It defaults to 12. 17 digits is ``perfect'' for IEEE floating-point in that it allows double-precision values to be converted to strings and back to binary with no loss of information. However, using 17 digits prevents any rounding, which produces longer, less intuitive results. For example, expr 1.4 returns 1.3999999999999999 with tcl_precision set to 17, vs. 1.4 if tcl_precision is 12.


HTH

ulis

 

[tomdagliusa]

First of all, thank you to all for replying.

I tried the format %4.6f which still truncated, but did it
more precisely, i.e. 3006.120000 instead of 3006.117791.

I'm afraid set the global precision var as it may break other scripts.

I'm probably going to multiply out the decimal portion and
save that value, along with any carry, then multiply the integer part, add the carry, if any, to it, and parse the two numbers together with a period in the middle.

It's a hack, but it'll work.

Tom