Request could not be submitted for background processing

[mdwyer]

We have been running CE 9 for several years and suddenly began having total processing failure with the error message, "The request could not be submitted for background processing. File C:\temp\procSched\devserv2005.report\~tmp...rpt."

This applies to ALL reports run under CE, and the failure is immediate; the event dependency is not even checked.

When this first occurred, BO support suggested changing the Job Server to run under a domain account rather than the default LocalSystem. With a later failure, we changed to a different domain account to get things working. The last time, the job server was reinstalled.

We're out of ideas, and so is BO. They, of course, do not want to support v9 any longer, but we have some substantial ePortfolio customizations that would be lost on an upgrade.

Are there any ideas with you folks?

Thanks,
Mike

 

[KingfisherINC]

It sounds like a security / disk access issue to me.

Some quick thoughts / questions:
[ul]
[li]Crystal service pack / hot break fix installed?[/li]
[li]Windows SP installed?[/li]
[li]Any info from a trace log on the job server?[/li]
[li]Are the temp directories full?[/li]
[li]Are the temp directories accessible?[/li]
[li]Do all reports fail, even simple ones?[/li]
[li]How about running a process monitor to track exactly where/when it fails.[/li]
[/ul]


Kingfisher

 

[mdwyer]

Thanks, Kingfisher.

The service packs are up to date for CR (sp6), CE (sp5), and W2k3 (sp1).
C:\Temp has one 1.2MB file. C: has 20GB free.
We've given everyone full control over C:\Temp and subfolders.
All in-house reports fail. Interestingly: The CE sample reports do not fail. Our reports access an Oracle database; CE 's do not.

Can you explain what you mean by a trace log and a process monitor? I've only checked the Event log (nothing there).

- Mike

 

[laghes]

Hey Mike,

Its not like BO don't want to support CE9 anylonger. May be u had not got the right guy to help. You can reopen your case & request for reassignment of the case to L2 engineer.

Regarding the problem, could you tell me when exaclty it started. Was there any update in the software/OS/computer/network etc ? Was there any update on the database side..e.g. patches or service pack.
Which Database you are using

Kind Regards
JS

 

[KingfisherINC]

Can you explain what you mean by a trace log and a process monitor? I've only checked the Event log (nothing there).

By trace log I mean add a -trace switch to the report job server, this will create a log file in your logging directory. IIRC v9 may have a verbosity setting too? you'll have to check the documentation.

By process monitor I mean tools that monitor exctly what is happening on the server. There are several, I don't know which is best, but for example:
Filemon and ProcessExplorer



Kingfisher

 

[mdwyer]

I'm not knocking BO Support; the tech that assisted me went way overboard in trying to solve the problem, including several hours spent in Webex sessions. But CE9 is technically off support as of April 1.

Naturally, "nothing has changed," but it's a large environment where *something* may have changed and it would not be obvious. To the best of my ability, I cannot determine if there was a change. However, the frequency of occurrence is about once a month, just before or just after the month changes. There could be something there. Also, we have two systems with nearly the same configuration (one development/test and one production). Oddly, the failures are showing a pattern of alternating between these systems.

The reports that fail are "managed" reports that are run on a recurring schedule prior to start of business each day. Very few reports are run manually during the day. I will discover that on a particular day all the scheduled reports have failed with the "could not be submitted" error.

All the failng reports are hitting an Oracle 10g database running on a Solaris 9 server. CE is running on Windows 2003 Server. They were using the CR Oracle ODBC Driver 4.10. I have changed that to Oracle in Ora10g with no change.

 

[mdwyer]

The -trace option is not documented in the online help, but it did show the following:

[small][Wed May 03 16:36:53 2006] 5148 3068 assert failure: (Y:\backend\jobserver\dlls\procreport\src\crpemgr.cpp:1815). (0 : procReport.dll: Unknown exception caught: caused by: CrpeMgrPEOpenPrintJobWithAPSInfo).
[Wed May 03 16:36:53 2006] 5148 3068 trace message: procReport.dll: CrpeMgrPEGetErrorCodeAndSubstituteStrings(0) in

[Wed May 03 16:36:53 2006] 5148 3068 trace message: procReport.dll: CrpeMgrPEGetErrorCodeAndSubstituteStrings(0) out returns 1 and errorInfo->lastErrorCode is 685

[Wed May 03 16:36:53 2006] 5148 3068 trace message: procReport.dll: CrpeMgrPEGetExtendedErrorMessage() in

[Wed May 03 16:36:53 2006] 5148 3068 trace message: procReport.dll: CrpeMgrPEGetExtendedErrorMessage() out returns 1[/small]

Filemon is a strange beast, but it seems to show that data is accessed through the Oracle ODBC driver and that formating is taking place. (There are many entries with SolidDocuments/SolidPDF/SolidGraphics strings in them.)

So, a lot more detail than I ever wanted to see, but nothing that helps me resolve or better understand the error. Does it mean anything to you?

Thanks!

 

[KingfisherINC]

I'm not knocking BO tech either, but if they did not ask for a trace log I'd be a little concerned.

Do the reports View on Demand without error?
Do the reports View on Demand using Advanced DHTML viewer without error?
Does even a very simple Oracle report ("select sysdate from dual") fail?
How many Job servers do you have, and on how many servers in the cluster?


Kingfisher

 

[mdwyer]

View on Demand: Yep. It works. (DHTML and ActiveX viewers)

VOD/Interactive DHTML: Yep.

My simplest report also fails when scheduled. It queries a 1-row table as (Show SQL Query):

SELECT "FIN_PARAMETERS"."DB_NAME", "FIN_PARAMETERS"."SOURCE_DB", "FIN_PARAMETERS"."LAST_UPDATE_DATE"
 FROM   "FIN"."FIN_PARAMETERS" "FIN_PARAMETERS"

Jobservers: One. No clustering. All CE services run on the same server. They are also installed on a second server as a development environment.

 

[KingfisherINC]

OK so the problem is only on the Job server (that's good).

Here is what I would do next:
* Delete the Job server from CCM
* Reboot
* Add new Job server from CCM
* Set Job server login account details
* Start Job server
* Enable Job server
* Cross fingers
* Schedule report
* Think possitive thoughts...and?

Kingfisher

 

[mdwyer]

I had pretty much done that twice before, but without the reboot. I also used a slightly different configuration of the fingers. I even copied the jobserver files from the working server to the non-working one.

With the reboot after deleting the jobserver, and the crossing of the fingers: Sorry, same results. But thanks for the assist.

 

[KingfisherINC]

Can you compare the version numbers of procReport.dll on prod vs dev?
Any anti-virus software running?
Are prod and dev on different network segments?
I'm running out of ideas I assume there is no good backup of the server?

Kingfisher

 

[mdwyer]

procReport was copied from prod (working) to dev (broken) already so both have the same version: 9.0.1.773, 3/10/05, in C:\Program Files\Crystal Decisions\Enterprise 9\win32_x86. (There are other copies and versions in the Patches folders.)

Symantec Anti-virus is running. What are you thinking in this regard?

I'm not sure about "network segments," but I suspect they are different. How might that affect things?

There may be a backup available, but I hadn't thought to use it because this is a recurring error, and it's currently on the development system. If we can't fix it without rolling back, that limits our solutions if the production system gets hit again. Of course, that might reveal what (if anything) really did change.

 

[mdwyer]

Same network segment. I knew the first three IP octets were the same, but wasn't sure that translated into "segments."

 

[darran6]

What database are you using for the Central Management Server (if CE9 works in this way), and is there any chance some limit could have been reached on it? (either filesize of DB, or number of objects?)

Cheers,
Darran

 

[mdwyer]

CE 9 calls it an APS (Automated Process Scheduler, I think). The CMS was introduced in version 10. We're using an Oracle 10g database for it. The server has been having some performance problems, but should have the necessary resources. We will be migrating to a higher-powered server in the near future, but it might be worth migrating back to the default MSDE database and see what that affects.