CMS hangs in 'stopping' mode 1

[jovunik]

Hi folks,

We are currently running CE10 SP1 on a Windows 2003 server.

We have encountered on several occasions a critical production issue where the Crystal Management Server (CMS) loses connectivity to the CE10 database on the Database Servers. This loss is due to various reasons such as network outages or database server maintenance, etc.

Unfortunately this causes the CMS process (CrystalMS.exe) to enter a ‘stopping’ state, but it has not completely stopped. Basically, the process is running in a ‘hung’ state, and the Crystal Enterprise application fails to function correctly – affecting reports in production. The process cannot be ‘killed’ and the only fix at this time is to reboot the server.

We have attempted to kill the process using Task Manager but get an access denied exception, even after stopping all CE related servers/services. We have even tried to use utilities like "PrcView" and "Process Explorer (procexp.exe)" to kill the process without success. The only solution so far is to reboot the server.

The following is the error we see in the event viewer when connectivity is lost:

Error - Crystal_CrystalMS - General - 33007
CMS is unstable and will shut down immediately. Reason: Database access error.
Reason [Microsoft] [ODBC SQL Server Driver] Communication link failure.

We have contacted Business Objects and others companies have reported the same issue to them. However, we were told a fix is not to be expected until the next CE product release.

Some suggestions they made were:
- To move the CMS database to the server locally. However, under our overall enterprise security model and zoning of servers and applications, this is not possible and is unacceptable.
- We have already looked at KB article c2015557, and although we have this implemented, it does not help us in our case.

So, my question to all of you is, short of rebooting the server each time this happens, what other things could we try? Have any of you experienced this and found any solutions?

Thanks in advance for any information or suggestions,

joVuNik

 

[Turkbear]

Hi,
We have also found that the only way out of this problem is a reboot of the server running the CMS service..

Since our CE server is only used for CE, and since without a valid CMS connection no one can do anything anyway, we just reboot and move on..

 

[MJRBIM]

Ditto to Turkbear's comment, once it's "STOPPING" reboot is the only way to go...

To avoid this we have started a scheduled process that STOPS all of our CE services (including CMS) at 11 PM and restarts them at 6 AM. This avoids most of our DB and network outages that take place during the overnight hours.

Don't know if that is a workable solution for your environment.

 

[vinay2167]

Hi,
We had the same issue, I went ahead to the task manager and select the CMS task and end task. It then comes as Stopped. Let me also know if this is a harm full way of doing it !

 

[Turkbear]

Hi,
Probably not harmfull, but it could depend on the database you use for the CMS..Sometimes killing a client process ( which is what the CMS service acts like)when connected to a database can leave a connection open to the database and that can be problematic.

 

[LMCRYER]

Dunno if theres a difference in terms of "damage" but I was fed up with the processes not letting me STOP them, so I just go in aod do a cmd kill proc id. Works every time.

LMC

LMC

 

[jovunik]

Hi All,

I thank you all for your responses. We have put in an auto-reboot of the server for every weekend to re-enable any services that might have hung.

I'm hoping BO can fix this in the future. Happy Holidays!

joVuNik

 

[KingfisherINC]

An early Christmas present:

Go to Windows Services ->
Crystal Management Server ->
Right click Properties ->
Recovery Tab ->
First Failure = Restart the Service
Second Failure = Restart the Service
Third Failure = Restart the Service
Reset fail count = 0 Days
Restart service after = 2 minutes <or whatever you want>

Kingfisher [CECP]

 

[Turkbear]

Hi,
Have you checked this , Kingfisher? If the service is in the 'stopping' state, is that a Failure that would trigger the recover actions?

 

[KingfisherINC]

I've used this to solve "Stopping" issues before when the database has gone down, and issues where the CMS does not recover after a database reboot. I don't know exactly when the fault is trapped to trigger these actions though, there may well be scenarios where this does not solve the problem, and a hard reboot is the only option.

Kingfisher [CECP]

 

[Turkbear]

HI,
Thanks..It can't hurt to set it that way for those cases where it will help resolve it. So, a star...