I have a program that has many calloc allocations for 2D arrays and finally a calloc call a = calloc(rc, sizeof( char)); The program runs fine for smaller file sizes but when file sizes get larger this calloc returns a pointer to NULL; I have freed all previous calls to calloc before this last call. I watch the memory manager in the task bar and see deallocation of memory when memory is freed. There should be plenty of memory available for this last call to calloc but as I said It returns pointer to NULL. Anyone got any ideas? Is heap corruption possible?
Tek-Tips is the largest IT community on the Internet today!
Members share and learn making Tek-Tips Forums the best source of peer-reviewed technical information on the Internet!
-
Congratulations SkipVought on being selected by the Tek-Tips community for having the most helpful posts in the forums last week. Way to Go!
Calloc memory allocation
- Thread starter gismo
- Start date
It could be any number of things.
When you allocate the rows of your 2D array, do you do this?
When you allocate the columns, do you allow for appending a \0 to the end?
--
If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
When you allocate the rows of your 2D array, do you do this?
Code:
char **array = malloc( nRows * [red]sizeof(char*)[/red] );When you allocate the columns, do you allow for appending a \0 to the end?
--
If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
A null return indicates a failure, often times meaning out of memory. I noticed that you are using Calloc. I can't recall the details, but I think I heard / read that using malloc is better. More standards compliant or something like that.
I would try switching to malloc from calloc and run a test where you allocate some memory, write some garbage data to it (loop through the range assigning incrementing values or something), free the memory, then re-allocate it and see if you get the same address pointer back (you should).
I would try switching to malloc from calloc and run a test where you allocate some memory, write some garbage data to it (loop through the range assigning incrementing values or something), free the memory, then re-allocate it and see if you get the same address pointer back (you should).
- Thread starter
- #4
All 2d array calls of calloc prior to a = calloc(rc, sizeof( char)); are sizeof(int) or sizeof(double) and all are freed. I have tried malloc and get the same results. The syntax is correct since smaller files do run correctly. Watching the usage on the taskbar I have about 1.5 GB of free memory at the time of the space allocation problem. I'm only trying to allocate less than 100 MB. So why can't I allocate 100 MB when I have 1.5 GB of ram free? I realize the pointer to NULL means that calloc was unable to allocate the 100 MB of memory. Does all the allocations and freeing of memory cause fragmentation or corruption of the heap to the point that I can't get my 100 MBs?
The problem with calloc (x,y) is that it allocates an array of x items, each at least y bytes long. It may not do what you think. If y is less than the word size, it may allocate a word for each item. On a 64 bit machine, calloc (10, 1) may allocate 80 bytes instead of 10.
In your case, if y is 100M, then it may be allocating 400M or 800M depending on your word size. If however, you used malloc as suggested by Salem, then it would allocate 100M
In your case, if y is 100M, then it may be allocating 400M or 800M depending on your word size. If however, you used malloc as suggested by Salem, then it would allocate 100M
> The syntax is correct since smaller files do run correctly.
Bad deduction on your part.
Early success followed by failure when scaling up just meant you were "lucky" early on (and NOT correct).
Please post your code for allocating a 2D array.
--
If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
Bad deduction on your part.
Early success followed by failure when scaling up just meant you were "lucky" early on (and NOT correct).
Please post your code for allocating a 2D array.
--
If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
Sorry, some addition.
The calloc function does not take into account an element (the 2nd argument) alignment. It allocates 1starg*2ndarg bytes. Of course, the allocated memory chunk size is not equal to 1st*2nd exactly (add internal heap control block) but no 1starg or 2ndarg factors in this overheads.
Incidentally, both calloc parameters have unsigned (size_t) type. If we pass negative int as the 1st arg, the calloc can't allocate this huge unsigned number of elements and returns NULL in all cases!
Moral: calloc is equal to malloc with additional zero-bit initialization so OP problem is out of the calloc per se scope...
The calloc function does not take into account an element (the 2nd argument) alignment. It allocates 1starg*2ndarg bytes. Of course, the allocated memory chunk size is not equal to 1st*2nd exactly (add internal heap control block) but no 1starg or 2ndarg factors in this overheads.
Incidentally, both calloc parameters have unsigned (size_t) type. If we pass negative int as the 1st arg, the calloc can't allocate this huge unsigned number of elements and returns NULL in all cases!
Moral: calloc is equal to malloc with additional zero-bit initialization so OP problem is out of the calloc per se scope...
- Thread starter
- #10
My fault for not making my initial post more clear. The program creates many 2d arrays of ints and doubles prior to the final call to calloc which is for a 1D array of pointer to char a = calloc(rc, sizeof( char));. To check to see if I was just getting "lucky" with smaller files, I isolated the second part of the program and ran it with the large file and the program allocated and processed correctly. I'm running windows xp on a 32 bit machine and visual studio 6.0. so my sizeof(char) is 1 byte. rc=48750000 or 48.75 MB.
ArkM, I think you're on to something with the chunk size. You say to "add internal heap control block". I don't know how to do that. Could you explain. ALSO XWB you said that if y is less than 1 machine word, which it is in this case, that the system might still allocate a full machine word. So in my case my 48.75 MB if allocated as a full machine word would be 195 MB. How can I test your possibility? Watching the memory manager on the task bar I do not see a huge jump in memory usage when the allocation takes place. Thanks everyone for your comments.
ArkM, I think you're on to something with the chunk size. You say to "add internal heap control block". I don't know how to do that. Could you explain. ALSO XWB you said that if y is less than 1 machine word, which it is in this case, that the system might still allocate a full machine word. So in my case my 48.75 MB if allocated as a full machine word would be 195 MB. How can I test your possibility? Watching the memory manager on the task bar I do not see a huge jump in memory usage when the allocation takes place. Thanks everyone for your comments.
gismo,
no need "to add internal ctrl block" explicitly, C run time library functions do this work implicitly. The real memory allocation size is requested_memory_size + memory_control_block_size_and_padding. The last item as usually is 8-16 bytes per calloc/malloc call or near that.
It seems 48.7 Mb allocation is not a crucial moment for the old good VC 6.0...
By the way, if you want to allocate "array of pointer to char", why did you make a request of n*sizeof(char)? Array of N pointer(s?) to char size is N*sizeof(char*)...
no need "to add internal ctrl block" explicitly, C run time library functions do this work implicitly. The real memory allocation size is requested_memory_size + memory_control_block_size_and_padding. The last item as usually is 8-16 bytes per calloc/malloc call or near that.
It seems 48.7 Mb allocation is not a crucial moment for the old good VC 6.0...
By the way, if you want to allocate "array of pointer to char", why did you make a request of n*sizeof(char)? Array of N pointer(s?) to char size is N*sizeof(char*)...
Yet another possible source of troubles is a great number of tiny allocations in 2D arrays construction.
About memory fragmentation case: let's try to pre-allocate large enough (~100 Mb, for example) dummy array on the program start then free it just before that ill-starred calloc
About memory fragmentation case: let's try to pre-allocate large enough (~100 Mb, for example) dummy array on the program start then free it just before that ill-starred calloc
- Thread starter
- #14
ArkM, I'll try your suggestion Monday I'm not in the office today. Previously I have tried to allocate the 48.75 MB at the beginning of the program and carry it through to the end of the program only to receive a pointer error, cannot write to .... when I use files that allocate more than 10 MB. I'll try and allocate 100 MB at the beginning of the program and free it just prior to allocating the 48.75 MB on Monday. You were correct I shouldn't have said an "array of pointers to char" I should have said a pointer to an array of chars. Thanks for the suggestion.
- Status
Similar threads
