Significant Figures and Resolution

[blboyd]

Upon exploring a problem with our software we ran into an anomly that none of us understand.

Using this code - all I am doing is adding .001 to .001 an so on. Looks that simple right?? If you run the code it will start generating numbers like: 6.000001E-03

Can anyone explain this or have a fix for this??

Thanks,
Brandon

Sub Main()
Dim TestArray(1000) As Single
Dim i As Integer

TestArray(0) = 0.001

    For i = 1 To 999
        TestArray(i) = TestArray(i - 1) + 0.001
    Next i
    
End Sub

 

[vb5prgrmr]

try currency...
[tt]
Dim TestArray(1000) As Currency
Dim i As Integer

TestArray(0) = 0.001

For i = 1 To 999
TestArray(i) = TestArray(i - 1) + 0.001
Next i

[/tt]

Good Luck

 

[hookbob]

Hi there

The code is fine. I've tested it also and it works properly. How are you testing the resulting values ? Are you formatting the value in a certain way to view it ?

If you add the following line in the For Loop you will see the results are correct.

Debug.Print TestArray(i)

 

[chandraIRL]

change the datatype from Single to Double.
TRY IT.

 

[MichaelRed]

read hookbob's post. It is a bit pithy -but absoloutly accurate.

or (with SLIGHTLY more context)

? 6 * 0.001
 0.006 
? 6e-3
 0.006

e.g. 0.006 = 6e-3


Now -if you want / need / desire a SPECIFIC format for this trivia, you should avail yourself of the ever present {F1} and snoop through the ubiquitious "Format" function.




MichaelRed
m.red@att.net

Searching for employment in all the wrong places