Incorrect results using exponentiation with the INT() function

[longda]

We have just encountered a problem with using exponentiation inside the INT() function with regards to the SET DECIMALS setting. Below is sample code for you to try in your command window.

SET DECIMALS TO 2
l_nNewValue = 4.9811
? INT(l_nNewValue*(10^5)) / (10^5) = 4.9811
? INT(l_nNewValue*10*10*10*10*10) / (10*10*10*10*10) = 4.9811

SET DECIMALS TO 4
l_nNewValue = 4.9811
? INT(l_nNewValue*(10^5)) / (10^5) = 4.9811
? INT(l_nNewValue*10*10*10*10*10) / (10*10*10*10*10) = 4.9811

This code should return the following on your screen:
.T.
.T.
.T.
.T.
but instead returns:
.T.
.T.
.F.
.T.

With SET DECIMALS TO 4 it is returning 498109 when using the (10^5) inside the INT() function but should be returning 498110 just like all the other equations. Why is this happening? Why does it work when set decimals is 2 and not when it is 4? Is their a setting I am missing? I know I can avoid the exponentiation equation and type out the 10*10 five times but my trust in VFP to return a correct mathimatical result will be diminished.

I did do a search and the only thing I saw that mentioned anything close this is Thread184-407116 and that had to due with double rounding.

Any help is much appreciated.

Thanks,
Dave L.

 

[tvign]

Dave,

You are right, the INT function is incorrect. INT(l_nNewValue*(10^5)) is returning the FLOOR since the result must be a smidgen below 498110 (in binary). It should be returning the CEILING.

I tried the same program in TRU BASIC which is a mathematical language (compiler written by mathematicians from Princeton) and it correctly returns 498110.

That's Microsoft for you!! Probably not a mathematician in the bunch.

Tony

 

[Stella740pl]

Maybe I don't understand your problem to the fullest.

Why do you say INT() is incorrect if it's description in Help is the following:"Evaluates a numeric expression and returns the integer portion of the expression." The examples in Help are following:

CLEAR
? INT(12.5) && Displays 12
? INT(6.25 * 2) && Displays 12
? INT(-12.5) && Displays -12
STORE -12.5 TO gnNumber
? INT(gnNumber) && Displays -12

If you need rounding (which is returning either FLOOR() or CEILING() according to the rules), why don't you use ROUND(MyNumber,0)?

 

[tvign]

INT must respect the mathematical meaning not the internal problem of representing integers with binary numbers. When a binary result translates to 44.99999999999999...etc, the compiler should realize that it is 45 not the floor, 44.

Tony

 

[longda]

It is still returning the wrong value regardless. Here is a more simplified example:

SET DECIMALS TO 2
? INT(4.9811 * (10^5))
? 4.9811 * (10^5)

The above code will work and both integer portions of the answers will be 498110

SET DECIMALS TO 4
? INT(4.9811 * (10^5))
? 4.9811 * (10^5)

The above code fails with the first answer having an integer portion of 498109 and the second answer having an integer portion of 498110. Why does the SET DECIMAL TO 4 affect how the INT() function returns the integer portion of a number?

Thanks,
Dave L.

 

[vazagothic]

Stella740pl,
the problem may be easier to understand here:

SET DECIMALS TO 18
l_nNewValue = 4.9811
? l_nNewValue   && 4.981100000000000000  / CORRECT
? l_nNewValue * 10   &&  49.810999999999990000  / INCORRECT

The same with:

SET DECIMAL TO 18
? 4.9811 *  10*10    && 498.109999999999900000
? 4.9811 * (10*10)   && 498.110000000000000000

In theory they should both return the same value .. but it seems Microsoft's programmers attended "special" math classes ;-)

 

[Stella740pl]

Well, in all four longda's examples I got one and the same 498110.

In vazagothic's examples 2nd two are exactly as shown, while 1st two return 4.9811 and 49.8110 respectively.

But I understand your grief better now.

In any case, did you try ROUND()?

 

[longda]

Tony thanks for the explanation. I looked up the FLOOR() and CEILING() functions in VFP and tried them instead of the INT function. The CEILING() gave me the correct answer and the FLOOR() gave me the same as the broken INT(). I remember doing this kind of thing in math class having to draw a X axis on paper to better understand it.

I did another test and it only seemed to happen with variables. Try the below code:

SET DECIMALS TO 4
l_nNewValue = 4.9811
? INT(l_nNewValue * (10^5)) && returns 498109
? INT(4.9811 * (10^5)) && returns 498110

Don't why Microsoft would do this. We are using VFP 8 so I am not sure if this affects all versions or not.

Thanks,
Dave L.

 

[Stella740pl]

This example works as you say in my VFP.

It's no wonder INT() works just as the FLOOR() does - it's basically the same function.

I personally use ROUND(...,0) and not INT() most of the time, but I must note I don't have do math calculations often. Worth a try, anyway. It worked correctly for me in your last examples.

 

[vazagothic]

I've tried it with versions 6,8 and 9 - same results. And I wouldn't be so sure about the code working without variables.

It all depends how long the variable is (and how many decimal places you want to show with SET DECIMALS)

While testing different numbers and different values for SET DECIMALS, I've found something "funny".

According to VFP (6.0, 7.0, 9.0)

SET DECIMALS TO 7
? 4981.1 * 100000   && 498110000.0000001

or

SET DECIMALS TO 2
? 4981.1 * 100000000000 && 498110000000000.10

Maybe someone made a mistake in a math library used by VFP, or something, since the last digits shouldn't be there at all.

I wonder who's to blame if a software written in VFP calculates things wrongly (and lets say a bank uses it ..)

 

[tvign]

This gets even worse.
Try this and see totally inconsistant results proving that INT is a Microsoft special math function - Hungarian Math, maybe??

Avoid the INT function!

SET DECIMALS TO 4
FOR l_nNewValue = 4.9810 to 4.9815 step .0001
? l_nNewValue
? l_nNewValue*(10^5)
? INT(l_nNewValue*(10^5))
? INT(round(l_nNewValue,4)*(10^5))
ENDFOR

Tony

 

[Mike Gagnon]

Tony

Can I ask that if the INT() function of VFP returns the integer portion of the expression., why would even bother with SET DECIMAL?


Mike Gagnon

If you want to get the best response to a question, please check out FAQ184-2483 first.

 

[wgcs]

It's no wonder INT() works just as the FLOOR() does - it's basically the same function.

Well, the same until you get to negative numbers:

INT( -5.5 ) = -5
FLOOR( -5.5 ) = -6

- Bill

Get the best answers to your questions -- See FAQ481-4875.

 

[tvign]

Mike,

I was just keeping the original test environment the same as defined by Dave L.

Why does it work when set decimals is 2 and not when it is 4?

Tony

 

[DSummZZZ]

The DECIMALS setting has a major impact on the math functions of VFP. For a demo, go to this FAQ and play with it a little.
While the form is running, go to the command window and try different SET DECIMALS TO settings and note the results. Interesting:

Conversion Functions: Dec,Hex,Binary,Oct Back and Forth
faq184-4461

It was a little enlightening for me anyway.

-Dave Summers-

Even more Fox stuff at:

http://www.davesummers.net/foxprolinks.htm

 

[tvign]

Dave,

Good show! If you put in 4.9811 with DECIMALS 4 you get back 4.98109961 and int(4.98109961*10^5) = 498109.

Voila!

It looks like assigning 4.9811 to a variable causes it to be stored internally as Single Precision Hex:409f652b which is 4.98109961 when DECIMALS is 4 and 4.9811 when DECIMALS is 2

You will also note that the double precision Hex: 4013eca57a786c22 always yields the correct result of 4.9811 for any DECIMAL setting (that I tried).

I don't believe we have control of precision in VFP.

Definitely, Hungarian Math invented by Microsoft!

Tony

 

[longda]

Thanks for all your help with this issue. After reading your last reply Tony, I did a search for "numeric precision" in the VFP help. The first result was the "numeric data type" for VFP. If you read this it even states that floating point calculations may produce wrong results. Then they pass the buck on to Intel and the IEEE. They even have a link to a knowledgebase article describing the problem in detail. Here is the link:

http://support.microsoft.com/default.aspx?scid=kb;en-us;Q78113

This link only states that Excel is affect and does not mention VFP. I wonder how many people actually read this help to see if they will get correct mathematicall results (I know I didn't). I guess Microsoft would never be held responsible for any issues because they said it may happen (even though they kind of hid it).

Thanks again for all of your help.

Dave Long