FLOOR error ? 3

[florindaniel]

I have this values: (SET DECIMALS TO 6 environnement)
value = 265.96 and qty = 20.
obviously, price = value / qty = 13.298

BUT FLOOR(1000000*vt/qq)/1000000 = 13.297999 !!!

Actually, what I am trying to do is to calculate the truncated rounded value to the sixth digit following the decimal point.

i.e. 14.736256732 goes to 14.736256, not 14.736257

the method I found is that: Floor(1000000*value/quantity)/1000000 but it gives that peculiar (and wrong) result.

Any ideas?
Thank you, Daniel

 

[Olaf Doschke]

You know floating point and why it's called floating?

You always have this kind of rounding errors with the one or other values, this is just normal and also no hardware bug (remember the pentium 1 fdiv bug?)

Your specific issue can be solved by using ROUND instead: Round(265.96/20,6)

Bye, Olaf.

 

[Mike Lewis]

Hi Daniel,

This is not an error or an incorrect result. You are working with numbers whose precision is too big for an integer, so VFP is using floating point. And floating point simply can't display every possible value in a given range. It's not a bug. It's just the way it works.

If your aim is to round down rather than round to the nearest, you could try something like this:

[tt]x = 14.736256732
? ROUND(x - 0.0000005, 6)[/tt]

That will give you 14.736256, which is what you want (if I've understood it correctly).

Mike


__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

 

[Olaf Doschke]

Your example of 14.736256732 points out you dont want the truncated rounded value, but you want the truncated NON rounded value.
So ROUND() won't help you, as Round(14.736256732,6) is 14.736257

STR() used without the decimals parameter will truncate, so you could use

lcNum =  Alltrim(Str(1000000*vt/qq,20))
lcResult = Left(lcNum,Len(lcNum)-6)+"."+Right(lcNum,6)

Bye, Olaf.

 

[florindaniel]

Thank you, Mike
I'll do some tests but at a glance I think it will work. That - 0.0000005 it's great
I've managed somehow to fix that using stings and macrosubstitution but it's not elegant at all.

Daniel

 

[Olaf Doschke]

I think you could find a case where mikes solution doesn't work, but testing this with values on the edge of two values by randomly generating a number x I couldn't yet find a wrong case.

So this might work out for a larger majority than the solution using FLOOR or STR.

Bye, Olaf.

 

[Mike Lewis]

Actually, my solution won't work for x < 0 - assuming your aim is to get the lower value, not the value closest to zero.

You could fix that like so:

[tt]? ROUND(x + (SIGN(x) * -0.0000005), 6)[/tt]

That would also work for x = 0.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

 

[NasibKalsi]

The following may also work for your situation

x = 14.736256732
y = 1000000.000000
? int(x*y)/y

 

[florindaniel]

Thank you all... the 14.736 etc was (perhaps an unfortunate) example.

The problem is when calculating:
[highlight #CC0000]value = 265.96 and qty = 20.
obviously, price = value / qty = 13.298

BUT FLOOR(1000000*vt/qq)/1000000 = 13.297999 !!![/highlight]

and is solved with Mike's solution (thank you again Mike)

Daniel

 

[vgulielmus]

This is the VFP reason for FLOOR(value /qty*1000000)=13297999 :

SET DECIMALS TO 6
value = 265.96 
qty = 20
?value /qty*1000000,value /qty*1000000==13298000 && .F.

A few more insight.

?value /qty*1000000>13297999.99999999720603229 && .T.
?value /qty*1000000>13297999.999999997206032290 && .F.
?value /qty*1000000<13297999.999999997206032290 && .F.
?value /qty*1000000==13297999.999999997206032290 &&.T.
?13297999.99999999720603229<13297999.999999997206032290 && .T.

So value /qty*1000000 is equal to :
13297999.999999997206032290
and is bigger than
13297999.99999999720603229

But if you use strings
?VAL(TRANSFORM(value /qty*1000000))==13298000 && .T.

I've made some experiments here

http://www.foxite.com/archives/interesting-behavior-of-inputmask-0000406050.htm

Respectfully,
Vilhelm-Ion Praisach
Resita, Romania

 

[Olaf Doschke]

Daniel, it's good the solution works, that doesn't mean it's still valid to exchange other ideas.

I'll start a new thread for that matter, Nasib, and we may discuss this further there. If we find a better solution or a case to alarm Daniel about Mikes solution also not working in some cases, we can add that here, but shouldn't annoy him with further notices of possible alerts or improvements, as long as there aren't any.

Therefore I started thread184-1738764

Bye, Olaf.