Tek-Tips is the largest IT community on the Internet today!

Members share and learn making Tek-Tips Forums the best source of peer-reviewed technical information on the Internet!

  • Congratulations strongm on being selected by the Tek-Tips community for having the most helpful posts in the forums last week. Way to Go!

Create 250^4 or 250^5 elements-arrays

Status
Not open for further replies.

armeros

Programmer
Aug 26, 2014
23
AU
Hi,

How can I declare the arrays of such size

REAL, DIMENSION(250,250,250,250) :: A ??[ponder]


I use intel visual Fortran and it generates the error message: The size of the array dimension is too large, and overflow occurred when computing the array size.

I cannot get away with sparse matrix as well.

Please give some suggestion.

Thanks.
 

Well, let's think for a moment...you are requesting an array that will have 250*250*250*250 = 3906250000 items in it; if each item was 1 byte long, you would need 3.9 GB of memory...but you want this array to be REAL. If the length of the default REAL is 8 bytes, you are requesting 31.25 GB of RAM...does your computer have this kind of memory?

You really need to figure out what the problem at hand is and how else you can solve it.

If sparse techniques cannot be used, maybe you need to stage your handling by working only on a portion of your matrix at a time and keep the rest on disk....read part of it, process, write it down, load another portion, process, write it down and so on.
 
How do you need to access these values? Are you just reading in the values once and then using them all over the place or will you be writing to them all the time?
 
Thanks for the answer.
For xwb, the values are calculated from some other part of the code and need to be stored in this huge arrays. Then, I need to pick the largest element from these 250^4 values simultaneously. Do you think I can reduce the dimension, find the MAXLOC from each dimension, and then MAXLOC again from the 1st MAXLOC ?

Thanks.


 
If all you are going to do is figure out which is the largest value out of all those, there is no need to store them at all...all you need is one variable.
 
To salgerman,
Nope. It is the algorithm that must be done. It works for 2*(250^3) and I tried to extend that to 250^4.

Thanks for your answer.
 
The algorithm is called the gridsearch method.
 
Uh Oh

I have no idea on how to work only a portion. It seems that I cannot avoid declaring things like

REAL, DIMENSION(250,250,250,30) :: A1
REAL, DIMENSION(250,250,250,30) :: A2
...
REAL, DIMENSION(250,250,250,30) :: A8
REAL, DIMENSION(250,250,250,10) :: A9

What is worse is that the arrays I really need to work on is of dimension (260,260,2,260,260,260), which can be declared for (260,260,2,260,3,3) (3 is the highest number allowed)



 
If speed isn't really a problem, there is no difference in syntax between accessing an array and a function when you are reading values
Code:
    x = A1(i,j,k,l)
    y = FUNC(i,j,k,l)
You could get your algorithm working first with an array, then switch to a function, where the function reads in blocks depending on whether the area you are looking at is in scope or not.
 
To xwb,

Suppose the working code look like this.

Code:
REAL, DIMENSION :: A(250,250,250,250)
A=0.0
        DO i=1,250           
            DO j=1,250       
                DO jj=1,250  
                   DO ii = 1,250
                    x=f(i,ii,j,jj)  !f is some function
                    WHERE (y>0.0)                         
                        y=g(x)      !g is some function
                    ELSEWHERE 
                        y=1e-12
                    END WHERE
                    A(i,ii,j,jj)=y                      
                END DO
            END DO
        END DO
     END DO     
z = MAXOC(A)

How can I do as you suggested ?

Thank you very much.

 

My previous comment about not needing a huge matrix and simply needing a single variable is this:
Code:
real Vmax
Vmax = -999999.99
        DO i=1,250           
            DO j=1,250       
                DO jj=1,250  
                   DO ii = 1,250
                    x=f(i,ii,j,jj)  !f is some function
                    WHERE (y>0.0)                         
                        y=g(x)      !g is some function
                    ELSEWHERE 
                        y=1e-12
                    END WHERE
                    if (y > Vmax) Vmax = y
                END DO
            END DO
        END DO
     END DO     
write(*,*) 'Maximum value  = ', Vmax
and that is it.
 
To salgerman, xwb

Actually, the actual code is not this simple just to choose the largest element from A. I need part of the A matrix to do some matrix operation, i.e.

Code:
REAL, DIMENSION (250,250) :: B

A(i,:,j,jj) + B(j,:)

(Sorry I did not show you my actual code since it would be too long here.)

So far I still don't have any idea other than declare thing bit by bit.

Someone is telling me that I need to access a super computer in order to do this.
 
armeros said:
I need part of the A matrix to do some matrix operation

If you do matrix operation you are only using 2 indices.
So IMO you really need only 2D-portion of matrix A and therefore should be able to reduce your 4D-task to smaller 2D tasks.
Then you can compute the result X of every 2D-tier in dependence of the previous tier for example something like this:
Code:
! matrix of ij=th tier
real, dimension (:,:), allocatable :: A
! result of the ij-th tier to be used in the next tier
real, DIMENSION (250,250) :: X
...
...
X = 0
do i=1, 256
  do j=1, 256
    ! for every i,j-th tier allocate the matrix A 
    allocate(A(256,256))
    call initialize_matrix(A)
    call compute_result(A, X)    
    deallocate(A)
  end do
end do
...
subroutine initialize_matrix(A)
  ! set initial_values of the matrix A
  ...
end subroutine
 
subroutine compute_result(A, X)
  ! compute result X in dependence of previous X and A
  ...
  X = some_function (X, A)
end subroutine
...

The result from one 2D-tier to the next tier will be transfered using the matrix X, but maybe you only need a vector or a skalar - it depends of your problem.
 
mikrom said:
do i=1, 256
do j=1, 256
! for every i,j-th tier allocate the matrix A
allocate(A(256,256))
call initialize_matrix(A)
call compute_result(A, X)
deallocate(A)
end do
end do

This seems to be a good way to try. I will try to do this. Thank you very much.

 
Looking back on the code I posted, I think that you can omit the usage allocatable.
Maybe, even if you initialize the matrix on the current tier, it would be good to know the matrix of the previous tier.
So then the code simplify to:
Code:
! A= matrix of ij-th tier
! X = result of the ij-th tier to be used in the next tier
real, DIMENSION (250,250) :: A, X
...
...
X = 0
do i=1, 256
  do j=1, 256
    ! for every i,j-th tier use the matrix A 
    call initialize_matrix(A)
    call compute_result(A, X)    
  end do
end do
...
subroutine initialize_matrix(A)
  ! set initial_values of the matrix A
  ...
end subroutine
 
subroutine compute_result(A, X)
  ! compute result X in dependence of previous X and A
  ...
  X = some_function (X, A)
end subroutine
...
 
mikrom, I really appreciate your help. All I need to do is just to fix the indices.
 
Status
Not open for further replies.

Part and Inventory Search

Sponsor

Back
Top