Calculation not showing correct value 2

[kiwieur]

Hi I wonder if someone can help me please with this issue

I have the following code I am working on

Dim MinWidth, MaxWidth, MinSWidth3, MaxSWidth3, MinSWidth1, MaxSWidth1, MinSWidth2, MaxSWidth2, MIW, MAW, MIW1, MAW1 As Variant

MinWidth = 2165
MaxWidth = 2465


MinSWidth1 = RndUp((MinWidth / Slit1), 2)
MaxSWidth1 = RndUp((MaxWidth / Slit1), 2)
MinSWidth2 = RndDwn(MinWidth / Slit1)
MaxSWidth2 = RndDwn(MaxWidth / Slit1)
MIW = MinWidth1 - MinWidth2
MAW = MaxWidth1 - MaxWidth2

Slit1 = 485 therfore

MinSWidth1 equals 4.47
MaxSWidth1 equals 5.09
MinSWidth2 equals 4
MaxSWidth2 equals 5
MIW = 0.47 which is correct
MAW = 8.99999999999999E-02

And I cannot figure out why this is so

Any help would be appreciated

Regards

Paul

 

[MajP]

It is an issue of point arithmetic. It has to do with how computers represent non integer values.
There are several ways to fix it.
1. Round the answer. Since your other values are precise to 2 digits then round your answer to 2.
2. Use a decimal datatype

You should not declare your variables as variants.

 

[SkipVought]

Hi,

Do this...

MinSWidth1 = Round((MinWidth / Slit1), 2)
MaxSWidth1 = Round((MaxWidth / Slit1), 2)
MinSWidth2 = Int(MinWidth / Slit1)
MaxSWidth2 = Int(MaxWidth / Slit1)

My results...
[tt]
4.46
5.08
4
0.46
8.00000000000001E-02
[/tt]
BTW, that happens with floating point arithmetic. If you want 5.08 - 5, resulting in EXACTLY .08, use integer arithmetic.

Dim MinWidth As Integer, MaxWidth As Integer, MinSWidth3 As Integer, MaxSWidth3 As Integer, MinSWidth1 As Integer, MaxSWidth1 As Integer, MinSWidth2 As Integer, MaxSWidth2 As Integer, MIW As Integer, MAW As Integer, MIW1 As Integer, MAW1  As Integer
Dim Slit1  As Integer
MinWidth = 2165
MaxWidth = 2465
Slit1 = 485

MinSWidth1 = Int((MinWidth / Slit1) * 10 ^ 2)
MaxSWidth1 = Int((MaxWidth / Slit1) * 10 ^ 2)
MinSWidth2 = Int(MinWidth / Slit1) * 10 ^ 2
MaxSWidth2 = Int(MaxWidth / Slit1) * 10 ^ 2
MIW = MinSWidth1 - MinSWidth2
MAW = MaxSWidth1 - MaxSWidth2
Debug.Print MinSWidth1
Debug.Print MaxSWidth1
Debug.Print MinSWidth2
Debug.Print MIW
Debug.Print MAW

'the payoff
Debug.Print MIW / 10 ^ 2
Debug.Print MAW / 10 ^ 2

Skip,

_{Just traded in my old subtlety...
for a NUANCE!}

 

[kiwieur]

Hi MajP & SkipVought,

Thank you for your very prompt responses to my question, I have decided to go with Skip's suggestion however Skip I do have one more question to ask you.

using my values
2165/485 = 4.463
2465/485 = 5.082

And using the code you supplied my values are

4.46 & 5.08

Is there anyway to amend the code so that it would round up to the following

2165/485 = 4.47
2465/485 = 5.09

so that I eventually end up with
.47 & .09

I ask this because I was asked to make the values round up but if this is not possible then fine

Regards

Paul

 

[SkipVought]

MinSWidth1 = Int(((MinWidth / Slit1) + 10 ^ -2) * 10 ^ 2)
MaxSWidth1 = Int(((MaxWidth / Slit1) + 10 ^ -2) * 10 ^ 2)

Skip,

_{Just traded in my old subtlety...
for a NUANCE!}

 

[kiwieur]

Skip,

I can't thank you enough for your help in solving this issue, one star does not seem adequate

[

Regards

Paul

 

[SkipVought]

Pleased to help.

Skip,

_{Just traded in my old subtlety...
for a NUANCE!}