CE 10 WCS Consumes Memory Until System Fails

[DanENW]

We are running Crystal Enterprise 10, with Service Pack 3, in a 3-machine cluster.

The CE cluster is built on 3 machines named SERVER85, SERVER86, and SERVER87. SERVER85 and SERVER87 are older machines (1.39 GHz P3 with 2 GB RAM) running Windows 2003 Server, Standard version. SERVER86 is a new machine (3.19 GHz Xeon with 2 GB RAM) running Windows 2003 Server, Standard version with SP1. Each machine has CE 10, SP3 loaded. Each machine is running the WCS with identical switches (-service -name MachineName.WCS -ns ClusterName -restart -defaultSessionTimeout 30 -trace).

We have experienced a problem with the WCS on the new machine. The memory used by the new WCS ramps up continuously during the daily usage cycles, and stops or slows during off-normal hours. The memory used continues to increase without limit until the memory usage approaches 1GB, at which time the memory usage drops, shifting to the virtual memory which now starts growing continuously until the virtual memory used approaches the physical memory size. At this point the CE system starts to produce error messages and eventually fails.

The memory used by the WCS on each of the old machines is in the neighborhood of 45 to 50 MB.

We have been monitoring the WCS memory usage daily and restarting the WCS during off-peak usage times when the memory used exceeds 700MB. This normally takes 2-3 days.

Any Ideas?

 

[eyeswideclosed]

Are your reports on-demand? How are they being called?

 

[DanENW]

We have over 400 reports in our library, and fewer than 10 are authorized for on-demand usage. At present I would estimate that we have fewer than 20 on-demand viewed instances per day.

Most of our reports are scheduled on demand (run now) and then either printed directly when scheduled or opened in the Crystal Enterprise viewer (we set the ActiveX viewer for the users' viewer; new users in the system get the HTML viewer by default until we set it for them).

We do have a few reports that are on-demand (only run when the user opens the report in the viewer).

One on-demand report runs against a business view and presents employee specific information. This report is always on-demand. This report was run quite heavily when it was released; current usage is sporadic. It is expected to see heavy usage briefly in the future for short intervals (1-2 weeks) and then taper off quickly to sporadic usage.

We have 4-6 other reports that may be run either as scheduled reports or (by limited audiences) as on-demand reports.

The reports that run on-demand all execute quickly and return limited data sets to the viewer.

 

[eyeswideclosed]

I'm thinking the process that calls the reports is not being released properly, but I need more specific information, when you open task manager/processes, what is the name of the process that is consuming all the memory?

 

[DanENW]

The image name shown in the task manager is WebCompServer.exe.

 

[Turkbear]

Hi,
Just a couple of things to check:

Are all the servers on the same sub-net?

Have you tried deleting and recreating the WCS on the problem server?





To Paraphrase:"The Help you get is proportional to the Help you give.."

 

[eyeswideclosed]

What are you using to call the reports? ASP? If you are, make sure you call Dispose() if available on the report objects after the call is made, and then destroy them (obj = Nothing), after every call.

 

[DanENW]

The servers are all on the same sub-net.

I will try deleting and recreating the WCS. It will take a couple of days, maybe a week, for me to process the paperwork to permit the change.

The reports are all being initiated and viewed thru the Crystal Enterprise web desktop; or using an html link. We do not use a locally developed interface.

THanks for the tips. Ill try recreating the WCS as soon as I can and report the results back.

If anyone has other ideas I'd like to hear them.

 

[eyeswideclosed]

Link to what? To the .rpt directly? That could be the problem, that is unmanaged reporting which is no longer supported in v10>, we had resouce problems with that too.

 

[DanENW]

No, not a link to the report. We don't employ unmanaged reporting. This is what one of the links looks like:

http://crystalenterprise10/crystal/...sword=&apsauthtype=secWinAD&promptOnRefresh=0

 

[eyeswideclosed]

Well I'm out of ideas, I would try Turk's solution, the WCS on that server may be corrupted or not set up properly.

 

[DanENW]

Turkbear,

If you are still looking at this, I tried deleting and recreating the WCS on the problem server (I had som personal problems that took me out for a few weeks) after I was able to do the paperwork. I deleted the WCS on Tuesday and then recreated it. Today (Thursday) the memory usage had climbed to 575 MB for the WCS. I can only guess that the issue is somewhere else.

Any other Ideas.

 

[Turkbear]

Hi,
Not at present but I will see if I can find anything
( I assume you have checked the server for malware, viruses, etc..with multiple scanners)





To Paraphrase:"The Help you get is proportional to the Help you give.."

 

[DanENW]

Yes. This system is under continuous examination for malware and is also behind a firewall.