Is there some Limitation in Division 2

[vj]

Hi guys,

I have a cell in excelsheet which has formula = A1/A2 ,,, where the value in A1 is 150 and value in A2 is 1584242000 and the calculated cell gives me the answer as : 0.0000000947 ..... this is all fine so far . Now in the FoxPro command window ... if I type in ?150/1584242000 ... I get 0 ... then I .... SET DECIMALS TO 10 and type in ?150/1584242000 again and I get 0.0000000000 . any special condition I need to put in to get my answer - 0.0000000947

thnakx
Vijay

 

[Mike Lewis]

Vijay, are you sure about SET DECIMALS? The result you are seeing is consistent with leaving SET DECIMALS at its default setting.

If you SET DECIMALS 10, you should see either 0.000000094683 or 0.0000000947, depending on the setting of SET FIXED.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

 

[Olaf Doschke]

I can confirm what you see.

SET DECIMALS TO 10
? 150/1584242000

As I said already in thread184-1738764:

VFP is not only converting a decimal number into the binary float representation and approximation, it also stores how many decimals the original decimal representation had. And then it does a side calculation about how many decimal places the result can have and displays that number of decimal places. If you multiply a number with N decimal places with another number of M decimal places, the result can have up to N+M decimal places. You simply have to think about an extreme case. Eg for N=1 and M=2 multiply 0.1 with 0.01 you get 0.001 having B+M=1+2=3 decimal places.

VFP isn't doing that very well, though, eg try ? 0.001^3, this will not be the same as 0.001*0.001*0.001, with the ^ exponent operator VP strays with the number of decimal places or precision of the base number. In division VFP sums the decimal places, too, though the rule doesn't fit for division.

Putting in the number of decimal places you want helps:
? 150.0000000000/1584242000 is correct
>0.0000000947

Also:
a = 150
b = 1584242000
? a/b
> 0.000000094683
Because now a and b are variables and VFP trades them as ordinary double floats without making a side calculation of the accuracy.

Bye, Olaf.

 

[vj]

ok .. thankx guys ..... it's ok .. now ..

 

[vgulielmus]

Dividing one integer constant into another integer constant, in VFP :
- if the result is >5x10^-7, is considered significant
- if the result is <=5x10^-7, is considered unsignificant then and rounded to 0

Because VFP6 behaves exactly the same, I suspect this being the default behaviour of Foxpro.

SET DECIMALS TO 18
?1/1000000 && 0.000000010
?1/10000000 && 0.000000000
?1/1234567 && 0.0000000081000
?1/1999999 && 0.0000000050000
?1/2000000 && 0.0000000000000

?
?5/1000000 && 0.000000050
?5/10000000 && 0.000000000
?6/1000000 && 0.000000060
?6/10000000 && 0.0000000060

?
?50/10000000 && 0.000000050
?50/100000000 && 0.000000000
?51/100000000 && 0.0000000051

?
?500/100000000 && 0.000000050
?500/1000000000 && 0.000000000
?501/1000000000 && 0.0000000051

Respectfully,
Vilhelm-Ion Praisach
Resita, Romania