formatting error: 9.94581 rounded = 9.8???

[caconklin]

I have a datawindow that is making use of a MSS6.5 stored procedure. All columns are of type real and the stored procedure produces a table with columns of type real. Without formatting the values I can see the data is correct. When I add formatting to "0.0" all values are correctly rounded except 9.9xxx. Somehow in PowerBuilder's infinite wisdom it takes 9.9xxx and rounds it to 9.8. The unformatted value I'm dealing with is 9.94581. With PB's special formatting technique, that rounds to 9.8! After looking into it a bit more I've found the following:

9.95 < Value; Value rounds to 10.0

9.75 <= Value <= 9.95; Value rounds to 9.8

Has anyone else noticed this? Is there a logical cause or solution?

Thanks,
Cathy

 

[zibi]

Hi,
Have you tried using your own function for roundnig values? Then you can place a compute field with f.e. &quot;myround(value,1)&quot;
Or, mayby if this error occurs only on exe-file try to complile to byte-code, not machine-code.
Best regards, Zibi
--
Zbyszek Ogledzki, email:zbyszek@pangea.com.pl

 

[caconklin]

That's a possible work-around and thank you...but I was wondering why it was happening. It makes no sense that all other numbers (as far as I can tell) round correctly except for 9.85 to 9.95.

 

[zibi]

Yes, why...
I have another problems with datawindows and parameters - decimal.
If you make a table with column, lets say id and insert rows with values:
...,1000000000088700, 1000000000088701, 1000000000088702, ... try this:
Build DW with argument - compared to column id. Try to retrieve:
1000000000088700, next: 1000000000088701, next: 1000000000088702.
Its not possible to get the middle row! (yes, there is a workaround, but why...)
regards, Zibi
--
Zbyszek Ogledzki, email:zbyszek@pangea.com.pl