In serious need of performance improvement - suggestions?

[eo]

Hi

I have a BOExi r2 environment with approx 2500 report objects, approx 250 preset scheduled staggered as much as possible, output to either default, SMTP or anumanaged disk.

Report schedules complete at around 9:30 in the morning when users start viewing scheduled reports. We have approx 500 registered users of which approx 100 are active.

We are currently experiencing a massive drop in the system's performance and I have no idea why. CPU usage runs at 100%, and the "culprits" (for example currently) are: Page server x 5 and Job Server Child x 2

I have tried all the prescribed tricks, but now need some special advise to try and improve performance:

Server specs:
Intel(R) Xeon(TM) CPU 3.6 GHz, 3.5 GB of RAM

Capacity:

C: Total - 29.2 GB; Free - 3.08 GB
D: Total - 242 GB; Free - 205 GB

Herewith my server setup:
Server * Location of files * Comments
-------------------------------------

1) Cache * D: * Max Cache size 102400 Kb, min before idle connection closed 60, oldest on demand data given to a client 60

2) CMS * NA

3) Connection * NA

4) Desktop Intelli Cache * C: * we do not use Deskt Intelli so file size is 0bytes

5) Desktop Intelli Job * C: * we do not use Deskt Intelli so file size is 0bytes

6) Desktop Intelli Rpt * C: * we do not use Deskt Intelli so file size is 0bytes

7) Destination Job * * C: * file size is 0bytes

8) Event * NA

9) List of Values * C: * file size is 0bytes

10) Page * D: * Min before idle connection closed 85, Min before idle job closed 85, oldest on demand data given to a client 60, Database Records to Read when Previewing or refreshing a report - unlimited

11) Program Job * C: * file size is 0bytes

12) RAS * NA

13) Rpt Job * D: * Max jobs allowed 8

14) Webi Job * C: * we do not use WebI so file size is 0bytes

15) Webi Rpt * C: * we do not use WebI so file size is 0bytes

16) Input * D: * Max idle time 60

17) Output * D: * Max idle time 60

As you will see, whereever there are overhead intensive services, they were moved to D: which has so much more capacity then C:

Any ideas what else can be tried?


EO
Hertfordshire, England

 

[eo]

I forgot to add that many report instances (successful schedule outputs) which are viewed by users are very big, but I would have hoped that the liberal space for the Cache server on D: would be able to deal with this.

EO
Hertfordshire, England

 

[MJRBIM]

When do you experience the performance drop? Overnight when the Scheduled RPT files are running - or during the day when people are viewing instances?

From your server info, it looks like you are running on a single-server with just one processor. You might be maxing-out concurrent sessions if users are all trying to view RPTs at the same time.

Do you have AUDITING running? Can you track how many concurrent USER sessions you are running when you hit these peaks?

Are you using any CR-specific features (ie. On-Demand Sub-Reports, Report Linking) that would be lost if you dropped all the RPT Instances to PDF format (with Bookmarks on Groups) rather than RPT format? That would take the load of your page/cache servers when people are viewing them.

 

[eo]

Hi,

95% of our reports are pretty interactive so PDF not an option.
We have a single processor with a processor licence so are not limited to a number of concurrent users, although the peaks are most likely when many users view the reports, thereby hitting the Cache and Page servers hard. These two are on a drive with loads of pace. But it still does not seem as if reports are Cached as they still take approx 15 seconds to render (if you are lucky during peaks 15 seconds are good). I am not sure if th actual size of the reports come to play here??
Nio magic fix, except investing in another server with another licence? Would this resolve the peak time issues?

EO
Hertfordshire, England

 

[MJRBIM]

Sorry, should have said "simultaneous user requests", not concurrent sessions...the problem isn't with licenses, it's with the volume of activity that you can support on one CPU that is doing everything in the BOE-XI (R2) infrastructure.

Please confirm which RPT-viewer your users are using for InfoView - as that effects the services that generate the pages (RAS vs Page/Cache).

When you say your reports are "large and interactive" - can you give me a ballpark on numbers of small, medium and large reports that you host (and how often they are viewed).

SMALL RPT = Returns 100 to 200 records and takes less than 1 minute to process.
AVERAGE RPT = Returns 1,500 to 2,000 records.
LARGE RPT = Return more than 50,000 records and takes more than 1 minute to process.

Also, see what your memory allocations are for the BOE-XI services when your CPU is maxed-out. Which services are eating up the Memory/CPU capacity?

The numbers below are from the BOE-XI sizing-guide when services are running ALONE on a single-processor, so you can expect a major drop from these numbers since you are running all services on a single-processor.

CMS = 1 CPU can handle 150 simultaneous user requests
CACHE = 1 CPU can handle 200 simultaneous viewing sessions
PAGE = 1 CPU can handle 25 simultaneous viewing sessions
RAS = 1 CPU can handle 25 simultaneous processing threads
CR JOB = 1 CPU can handle 5 concurrent report jobs

My "back of the napkin" estimate that you might need to cluster your environment with at 1 or 2 more processors running just the CACHE and PAGE services.

Your best bet is to contact your BOBJ Pre-Sales consultant who should be able to scope you needs.

I have a UK passport - if you need on-site services...:->

 

[hilfy]

Also, the Cache only contains the pages that have already been viewed. If a report hasn't been viewed yet or a requested page has not yet been viewed, it's not in the cache and so, there will be a lag-time while it is rendered.

-Dell

A computer only does what you actually told it to do - not what you thought you told it to do.

 

[bleuys]

It would be also good to know wich DB you are using for both system DB and the reports.
Also it would be good to know at wich rate is the cpu usage on the DB Server. Maybe you're running to many concurrent jobs while scheduling and that's while the scheduling task takes so long...