Memory management

[kodr]

I've got a few applications that run daily, that I use to track equipment utilization (port counts, used vs. available), stuff like that.

The number of nodes (equipment) is steadily growing, as the network gets larger.

I'm starting to have to allocate more memory for these programs (-Xms...m -Xmx...m) and this is working fine. Right now there's an abundance of memory, so I'm not too worried about it.

I'm in the process of changing from relying pretty heavily on array's to different data structures (Linked lists and so on) to make the programs more dynamic (that's what happens when you start to develop a program in a language BEFORE you take a data structures course...)

So, my question is this, I'd like to be able to review the amount of memory a particular program is actually using when it runs, and generate some type of file that I could review. Can anyone point me in the right direction?

What I'd like to accomplish is to be able to determine ahead of time when the data a program uses begins to approach an upper threshold, instead of waiting for the day that the program just crashes due to lack of resources.

Thanks.

 

[preachie]

Hi,

I have 2 quick ideas for you.

You can either run your application with a verbose garbage collector (-verbose:gc detailed information here

http://java.sun.com/developer/technicalArticles/Programming/GCPortal/)

or monitor the java process and check the virtual memory size regularly.

 

[TonHu]

It would be wise to 'clean up after yourself', by setting objects to null, if you don't need them eny longer, instead of just relying on getting things out of scope to be cleaned up automatically. Setting stuff to null get them bigger attention from GC.
Then forcing the GC to process stuff, f.e. just before a large (memory consuming) operation is to be performed should make it easier on yourself in the memory management area.
This is the code I use to force GC:

public void runGc(int loops) {
	for (int i = 0; i < loops; i++) {
		System.gc();
	}
}

and I call it at crucial points like this:

runGc(3);

If I really want to force all garbage to be collected, I call it with 10 loops, more doesn't really improve anything, and calling it once, is only helpful in very memory-tight situations.

HTH
TonHu

 

[stefanwagner]

Sorry, TonHu, but that's completly nonsens.

You don't need to set things to null before they get out of scope. Bigger attention from GC - how should that work?

Calling gc manually isn't guaranteed to do anything. You're discouraged to do that. Thinking, you would know better than the gc when to run is - well, optimistic.

Before a program crashes because of out-of-memory, GC is guaranteed to run.

don't visit my homepage:

http://home.arcor.de/hirnstrom/minis/index.html

 

[tom62]

I fully agree with Stefan. Calling GC manually only tells Java that is might be a good time now to get rid of the garbage. Running GC 10 times in a loop is therefore rather silly. As long as you've got object references hanging around in your program these objects will not be removed from memory. Even GC won't help you there.

 

[kodr]

I'm not too worried about cleaning up old data. I think by using more dynamic memory structures like linked lists and tree's instead of array's, I'll head that issue off before it becomes a problem.

It's just that there is so much raw data now to deal with that I need to keep in memory for a while, and the amount of data is growing daily. I'm looking for a way to monitor how much memory I'm actually using so that I can bump up the amount of memory available to the vm, or when I may be in need of a memory upgrade on the pc.

For example, one of the older programs I'm still maintaining started out with 1,000,000 lines of data that I had to parse through and act upon, keeping a large portion in memory to compare against other files. It's now well over 5x that large, in just over a year. So now I'm setting the vm to 256m to 512m, were originally, I just used the default.

 

[TonHu]

Well, sorry guys, for the 'old-scool' clean-up act I tend to perform. It actually solved a lot of memory and performance issues in our constellation.

It may be related to the way I get the JVM handed to me, running on top of our own (C + Cobol-based) Middleware layer, with a very low memory-footprint. I also have systems running with per-session heapsizes of 4m (that's set by -Xmx4m, actually) so I don't always have memory to burn. You have to do something when 1200+ middleware sessions are running on a hp-ux server...

The GC stuff actually does some work, when you tell it to, if it doesn't (or hardly) get a chance to run on it's own, like in our single-threaded Middleware, but it does have about 10 JVM subthreads running, so one of them should be a gc-process.
On busy systems, every GC call seems to shift the actual garbage a generation further (older), and in tight systems, that, combined with assigning null to no longer used objects, does have a measurable effect on responsiveness and reliability.

HTH
TonHu

 

[stefanwagner]

Sorry - I never heard of any 'old-school' clean-up. Maybe your JVM behaves in a different way, than the usual Sun-Products?

An object is eligible for GC when the last reference to it is out of scope, and you needn't set it to null. If you set it to null, while there are still references to it, you're introducing security holes from type nullpointer exception.

If your scope ist usually greater than the real lifetime of your references, you're program structure is maybe bad - you might be using some kind of global variable.

For the normal JVM there are rarely cases to set objects to null. If your JVM is different, you shouldnd't give common advices which depend on a special type of JVM.

don't visit my homepage:

http://home.arcor.de/hirnstrom/minis/index.html

 

[TonHu]

My jvm is Sun 1.5.0_14 (on Windows, hp-ux latest official sun-based HP-release), it is just started in a way to not load deemed unneeded modules at startup, they may be loaded later.

Setting object references to null should ofcourse only be done if nothing will be accessing them, that was a natural assumption, sorry. As explained, in our applications under tight memory situations it does make a huge difference, when combined with repeated enforced GC calls. It is possible, that by our way of loading up the jvm, the normal gc process may be inhibited, and fall back to some emergency scenario, I have been unable to confirm these statements atm, the Architect/Developer involved is temporarily not available to answer questions of this kind.

HTH
TonHu