Redim 5 dimensional array => Out of Memory

[9550GT]

I have an array that needs to be dimensioned to the following specifications:

Dim m_dMTMDeals() as Double
.
.
.
ReDim m_dMTMDeals(1 To 2, 1 To 5100, 2003 To 2022, 1 To 12, 1 to 108)

The numbers above are represented as variables in the actual code. When VB tries to redim the variable, I get an out of memory error. The computer that is running the application has Windows 2000, 4 Intel XEO processors @ 2.20GHz, and 2 GB of RAM. I can't figure out why I don't have enough memory.

I need to get this resolved because the fifth dimension is only going to increase from 108...

Any help would be greatly appreciated!

 

[9550GT]

I forgot to mention, if you count Virtual Memory also, the computer has over 10 GB available for its use...

 

[TheVampire]

Well, the dimensions that you have there, @ 8 bytes per double, gives us 1.9 gig just for the array. Plus some overhead that the array needs for itself.

Are all of the values you are storing in this array doubles? Do you really need doubles or will a smaller type do?

Robert

 

[9550GT]

Certainly not every value in the array will be large enough to warrent the double type, but the possibility of such a large number does exist. If I dim the array as single and try to store a 'double sized' number during the procedure, I'll get an overflow error.... but I guess its better to get the overflow error at that time than it is to get the error when I'm trying to dimension the array.

I'll give it a shot and check back.

Does anyone know of any limitations in VB itself relating to memory usage? Shouldn't the virtual memory help in managing the size of this array?

 

[johnwm]

In theory VB can use all of virtual memory, but I seem to recall reading somewhere that there is a addressing limit related to using a Long as a pointer. Your data structure appears to be about 2 GB, which by the time you've got an operating system and a program in memory will require lots of disc paging and will be craaaaaaawlingly slow.

Is there any way you can revisit your data structure (database, chunk processing or whatever???)


________________________________________________________________
If you want to get the best response to a question, please check out FAQ222-2244 first

'People who live in windowed environments shouldn't cast pointers.'

 

[TheVampire]

You can look in MSDN for the information on VB limitations on data sizes and so forth.

It does have information about the stack space and if the variables are declared in a local procedure or at the module level.

Are all elements in the array going to have a value, or only some of them going to have values at any one time? If it's the latter, then you might want to look into sparse matrices, and see if that may help you with your memory problem.

You might also try an "Array of arrays" like this:

Dim m_dMTMDeals(5) As Variant
Dim E1() As Double
Dim E2() As Double
Dim E3() As Double
Dim E4() As Double
Dim E5() As Double

ReDim E1(1 To 2)
ReDim E2(1 To 5100)
ReDim E3(2003 To 2022)
ReDim E4(1 To 12)
ReDim E5(1 To 108)

m_dMTMDeals(1) = E1
m_dMTMDeals(2) = E2
m_dMTMDeals(3) = E3
m_dMTMDeals(4) = E4
m_dMTMDeals(5) = E5

This works fine on my system and gives no errors.

But, depending on how you have your code structured and what you want to do with it, this may not work for you.

Hope this helps or stimulates some ideas!

Robert

 

[9550GT]

johnwm:

The data structure has worked flawlessly up to today. The last dimension (108) is the number of deals in a certain portfolio that is being evaluated. Thats the most deals I've ever seen in this particular portfolio, but there could certainly be many, many more depending on trading activity.

We bought the computer that this process runs on because of its speed and memory. We figured we'd be safe with 4 processors and 10 GB total memory.... Unfortunately it looks like we were wrong.

Do you remember where you read that about VB using a Long pointer? If its on the web somewhere, I'd definitely like to check it out. Whether that is correct or not, the suggestion seems to make sense and is in line with what I've experienced so far.

I'm going to see if I get the error when I use an array declared as single that crosses the Long threshold (2.147 whatever million).

 

[CajunCenturion]

One of the problems that you may be running into with the single multi-dimensional array, is that the memory allocated for the array may need to be continguous. That means that not only do need 2.1GB for this array, but you need a contiguous block of 2.1GB for the array. TheVampire's suggestion significantly eases this situation in that your overall memory requirements are the same (or just slightly larger since you more than one array), but you break up the requirements into smaller memory blocks.

Good Luck
--------------
As a circle of light increases so does the circumference of darkness around it. - Albert Einstein

 

[9550GT]

TheVampire:

I'll check MSDN out when I'm finished here. Not all the elements will have a value. I guess I should explain what the structure represents:

1st Dimension : Calc Type (End of Quarter or End of Year)
2nd Dimension : Scenario number (5100 simulations)
3rd Dimension : Year
4th Dimension : Month
5th Dimension : Deal number

Each deal gets simulated 5100 times from its startdate to its enddate, both for the end of the next quarter and the end of the year. Each element in the array represents the payoff amount for a deal in a given month, for a given scenario, for each calculation type. If the deal doesn't start until Jan-2005, all elements where the 3rd dimension is 2003 or 2004 will be zero. I hope that makes sense.

I have a question about the array of arrays that you described. Say I wanted to reference this element in my array: m_dMTMDeals(1,1234,2010,7,59); how would I do this in your example?

 

[strongm]

The long data type issue is merely restricted to an individual index, not the array as a whole.

In theory, a VB array (multidimensioned or not) is limited in size only by available memory. However, under Win32, each individual process is limited to a maximum of 4GB of memory.

 

[TheVampire]

I tried this also:

Option Explicit
Dim BigE() As Double
Dim LittleE(1 To 108) As Variant

Private Sub Form_Load()
Dim I As Integer

ReDim BigE(1 To 2, 1 To 5100, 2003 To 2022, 1 To 12)

For I = 1 To 108
LittleE(I) = BigE
Next I

End Sub


And it took a very long time creating the temp file on my HD, then eventually bombed when the variable 'I' got to 62.

Checking my available disk space, I see that it's just slightly lower than 1 Gig ( I know, get a bigger drive, sheesh ) so I could't make a temp file large enough to hold all of the elements.

With more drive space this would work, but as johnwm says it would be VERY slow ( it took about 1 minute on my 500 Mhz machine before it finally came back with the error ).

Your list of types is as follows:

1st Dimension : Calc Type (End of Quarter or End of Year)
2nd Dimension : Scenario number (5100 simulations)
3rd Dimension : Year
4th Dimension : Month
5th Dimension : Deal number

So, you want to keep all of the results from each type of calcualtion ( EOQ or EOY ), for each month of each year from 2003 to 2022, and also for each deal number and 5100 different scenarios?

Why do you need to keep all of the data in memory at one time? Can you not just extract the information from a database when you need it?

Robert

 

[strongm]

Oh, and a further bit of research on that 4 GB suggests that "The virtual address space is divided such that 2 GB is available to the application and the other 2 GB is available only to the system", which would explain your problem...

The following link may be of interest:

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/memory/base/4gt_ram_tuning.asp

 

[9550GT]

CajunCenturion:

I think I'll give that a try in the morning.


strongm:

Thanks for the link and your suggestions.


TheVampire:

I keep the data in memory because its used in a later procedure to calculate expected deal values and percentiles by conterparty. I could save each element to our SQL Server database, but that would require ~264 million individual inserts. When this application was still in development, I tried writing the data to a text file and using a bulk insert to get it into SQL Server. The bulk insert took a little over an hour to upload all of the data and build the table indices.

Using the database would be ideal, but unfortunately its just too much I/O.

I'm going to use your array of arrays example to build a new structure that will hopefully eliminate the continuous memory issue. Thanks very much for your help. It did stimulate some ideas...

 

[TheVampire]

Oh, seeing what you were wanting to do, I don't think my "Array of Arrays" idea will work for you.

The sparse matrices idea may though. As you say, if you have a deal that does not cover all years and months, then those elements of the array will not have any values in them. This could make quite a diffence in the memory used.

Robert

 

[MichaelRed]

Hmmmmmmmmmm!

Why are you even attempting to use (Memory) arrays for such large 'datasets'? Isn't this a basic instance of the rationale for DATABASES? Unless you are implementing ARRAY functions like Array Multiplication, Transposition, Division, Addition Subtraction, you are probably just doing some logical and math functions on the elements in certain combiniations.

For starters, I would take at least the largest dimensions of the array and place them in recordsets and simply references these through an indexed field. It shouldn't be much different than the array dimension/index and allows the app to fit into memory. Further, I would 'go the whole route' and just make ALL of the dimensions of the array into recordsets (tables) with appropiate indicies. In general, this would eliminate the memory limitations even for much larger problems / data sets. There WILL a penalty on the time of ececution going through each element of every array, on the other hand (depending on the actula calculations), simple queries may generate enough of the answer to overcome the slowdown from the use of database recordkeeping.

Again, I do not pretend to know anything aboout YOUR app / problem and the soloution imnplementation - but database operations ARE matrix soloutions, so in many instances a database IS the more appropiate soloution.




MichaelRed
m.red@att.net

Searching for employment in all the wrong places

 

[TheVampire]

Exactly what I said in an earlier post, I just didn't use as many words!

Robert

 

[MichaelRed]

Well, I HAVE been known to ramble on a bit, and I admit I did not note that part of the post. I think my eyes were beginning to glaze a bit with all the number crunching along the threads' path.

Reviewing at least some of the posts here seems to indicate that the array only holds the calculated values of the matrix - but that the values are themselves indic=vidually calciulated, and no "Matrix Math" is involved.

In this case, the matrix formulation is entirely arbitrary and (one presumes) set up merely to reflect the 'logical' orientation of the problem. This being the case, the entire 'matrix' couls easily be 'linearized' into a single dimension.

That, of course, does nothing to alleviate the memory requirement, It DOES however, open the possability of (conceptually) restructuring the database to have different groupings (sets; RECORD structure) to reduce the 264M insert operations to ~~1M, which shoud be quirte a bit more manageable.

While the arrangement scheme would need some careful thought, it should be acceptable to arrange the results into groups of 200 to 250 or so and have each group be inserted as a record. If the group to matrix mapping becomes to awkward, the actual results fields could be reduced (by ~ 1/2) and a Matrix index field could be included to facillitate the retrieval in the "matrix" organization.

Again, this presumes that no matrix math is involved, otherwise, it STILL looks like a database is the more reasonable soloution.




MichaelRed
m.red@att.net

Searching for employment in all the wrong places