Help with integer conversion

[sagobogger]

I have a datafile that includes dollar amounts. I want to only select the records where that $ amount is a multiple of another $ amount. I thought it would be simple to divide the first amount by the second and determine if the result is an integer;

DEFINE FILE MYFILE
NEWNUM/D8.2 = AMOUNT1/39.56;
INEWNUM/I8 = INT(NEWNUM);
ITSANINT/A1 = IF (NEWNUM - INEWNUM) EQ 0 THEN 'Y' ELSE 'N';
END

Doesn't work. Some values of NEWNUM e.g. the exact values 13.00 and 15.00 do not convert correctly to integers, 13 = 12 and 15.00 = 14.

If NEWNUM is defined as a P8.2 instead of a D8.2 it works, at least with the range of values I have seen so far, but maybe it will fail next month when there are new numbers?

It's not the case that the result of AMOUNT1/39.56 is not exact, if the amount is defined as a multiple e.g. 514.28 the result is the same.

Is there a better way to do this?

Is there an explanation for what's happening?

What are the format D vs P trade off's? I tend to use D's having had (or thought I had) some rounding issues with summing a lot of P's a long time ago.

WebFocus 4.3.6, source edit mode in MRE.

 

[focwizard]

The problem is HOW double precision numbers are stored. What you see is usually NOT what is stored. That's because double precision stores a binary fraction, and an exponent (a power of 2 on PCs). Because most numbers can't be represented EXACTLY in this manner (unless they're a power of two themselves), you get a number which is 'slightly less' than what you think. What is displayed is the rounded off value, to the precision specified. Thus, your '13.00', as a D8.2, may ACTUALLY be stored as 12.9999999998 (to whatever precision the native mode format can hold). The INT function just drops the decimal portion.

With PACKED numbers, the digits themselves are stored, so you GET the number that's displayed. Again, 13.00 may be stored as ...01300c (the ... are leading zeros and the 'c' is the sign 'nibble' for positive numbers).

 

[sagobogger]

Ah, thanks, that helps a lot.

Is there a better/simpler way to find the whole numbers?

 

[sagobogger]

oops hit the button too soon...

Is there a MOD function?

 

[jimster06]

Yes there is a MOD function.

DMOD returns the answer a a decimal number,
IMOD returns the answer as an integer, and
FMOD returns the answer as a floating-point number.

Generically, the function looks like
function(dividend,divisor,outfield).

TABLE FILE EMPLOYEE
PRINT ACCTNUMBER AND COMPUTE
LAST3_ACCT/I3L = IMOD(ACCTNUMBER, 1000, LAST3_ACCT);
BY LAST_NAME
BY FIRST_NAME
WHERE (ACCTNUMBER NE 000000000) AND (DEPT EQ 'MIS')
END

LAST_NAME FIRSTNAME ACCTNUMBER LAST3_ACCT
BLACKWOOD ROSEMARIE 122850108 108
CROSS BARBARA 163800144 144

HTH

 

[sagobogger]

Great thanks, I'll try those. DMOD with a divisor of 1 should return the part after the DP.