ISSUE: We are experiencing a situation where reports are running in a single thread thus affecting performance. They should be running in multiple threads. Also see attached technical description.
Problem in Detail:
I work on the GCS project and I was wondering if you can help us out with an issue that's baffled us for days. We're basically running CE 9 deployed as 1 cluster across 5 NT machines (see bottom of email for a list of processes by server). The middle tier is implemented as 3 ActiveX dll components (VB 6) and CSP pages.
We've been running some load test on the GCS application this year when we discovered a strange behavior by the WCS and Page Servers. It seems that all all requests are executed sequentially even though we've configured say the Page Server to run with Maximum Simultaneous Processing Threads = 75. For instance, if we have only one page server running and we submit multiple/simultaneous requests to execute 2 reports from different machines; what we see in the log file is that one request will block until the the other request comes back before it starts executing (please see sample log below)
Page Server Log
12-07-05 23:02:13 - GetRSData() - HRCDR004 - SQL: {call PKG_CD_RPT_BMX.sp_get_ .... etc.
12-07-05 23:02:21 - GetRSData() - HRCDR004 - Time Elapsed (secs): 8
12-07-05 23:02:21 - GetRSData() - HRCDR004 - Records Returned: 168
12-07-05 23:02:21 - GetRSData() - HRCDR004 - SQL: {call PKG_CD_RPT_EXT.sp_comp_extra.... etc.
12-07-05 23:02:47 - GetRSData() - HRCDR004 - Time Elapsed (secs): 26
12-07-05 23:02:47 - GetRSData() - HRCDR004 - Records Returned: 121
We had first thought that it's an ADO connection problem, but we ran some tests to verify that requests are blocking on any DLL Function/Sub and not necessarily on database requests. The test we ran was as follows
- we wrote a simple Sub that writes to the log file, closes it and then sleeps for 10 seconds.
- we wrote a csp page that calls that 'sleepy' method
- we hit the csp page from 2 different browsers simultaneously
What we saw in the log file is that no two requests are executing the Sub at the same time. one always blocks until the first request returns.
Our next goal was to eliminate the DLL itself as the reason for the single thread execution (the way it's written, compiled, etc.), so we wrote a _VBScript/ASP page so we're not using the WCS to execute it. When hitting the ASP page simultaneously in the same fashion, we could see that request 1 and request 2 were executing the same Sub at the same time.
As further proof that there are no issues with the DLL, we noticed that unlike the WCS and the Page Server, the Job Server is able to handle multiple requests at the same time. Using the example above, if we had one Job Server running and we scheduled 2 reports at the same time, what we see in the log are 2 interlaced messages like so:
Job Server Log
12-07-05 23:02:13 - GetRSData() - HRCDR004 - SQL: {call PKG_CD_RPT_BMX.sp_get_ .... etc.
12-07-05 23:02:21 - GetRSData() - HRCDR004 - SQL: {call PKG_CD_RPT_EXT.sp_comp_extra.... etc.
12-07-05 23:02:21 - GetRSData() - HRCDR004 - Time Elapsed (secs): 8
12-07-05 23:02:21 - GetRSData() - HRCDR004 - Records Returned: 168
12-07-05 23:02:47 - GetRSData() - HRCDR004 - Time Elapsed (secs): 26
12-07-05 23:02:47 - GetRSData() - HRCDR004 - Records Returned: 121
Given the information above, could you please help us understand if:
- This is a known problem or expected behavior
- This is a limitation of this particular implemenation (VB/COM)
- There's anything we can do to mitigate this issue without introducing too much risk
++++++++++++++++++++++++++++ 5 Server Deployment+++++++++++++++++++++++++++++++++
Machine Name Type Server Group Selected
HRCDR001.aps1 HRCDR001 Crystal APS Member of...
--------------------------------------------------------------------------------
HRCDR001.eventserver hrcdr001 Crystal Event Server Member of...
--------------------------------------------------------------------------------
HRCDR001.RAS hrcdr001 Crystal Report Application Server Member of...
--------------------------------------------------------------------------------
HRCDR001.report hrcdr001 Crystal Job Server Member of...
--------------------------------------------------------------------------------
hrcdr002.cache_2 hrcdr002 Crystal Cache Server Member of...
--------------------------------------------------------------------------------
HRCDR002.cacheserver hrcdr002 Crystal Cache Server Member of...
--------------------------------------------------------------------------------
HRCDR002.WCS_1 hrcdr002 Crystal Web Component Server Member of...
--------------------------------------------------------------------------------
hrcdr002.WCS_2 hrcdr002 Crystal Web Component Server Member of...
--------------------------------------------------------------------------------
hrcdr002.WCS_3 hrcdr002 Crystal Web Component Server Member of...
--------------------------------------------------------------------------------
hrcdr002.WCS_4 hrcdr002 Crystal Web Component Server Member of...
--------------------------------------------------------------------------------
HRCDR003.pageserver_1 hrcdr003 Crystal Page Server Member of...
--------------------------------------------------------------------------------
hrcdr003.pageserver_2 hrcdr003 Crystal Page Server Member of...
--------------------------------------------------------------------------------
HRCDR003.report_2 hrcdr003 Crystal Job Server Member of...
--------------------------------------------------------------------------------
Input hrcdr003 Crystal File Repository Server Member of...
--------------------------------------------------------------------------------
Output hrcdr003 Crystal File Repository Server Member of...
--------------------------------------------------------------------------------
HRCDR004.cache_1 hrcdr004 Crystal Cache Server Member of...
--------------------------------------------------------------------------------
hrcdr004.cache_2 hrcdr004 Crystal Cache Server Member of...
--------------------------------------------------------------------------------
hrcdr004.wcs_1 hrcdr004 Crystal Web Component Server Member of...
--------------------------------------------------------------------------------
hrcdr004.wcs_2 hrcdr004 Crystal Web Component Server Member of...
--------------------------------------------------------------------------------
hrcdr004.wcs_3 hrcdr004 Crystal Web Component Server Member of...
--------------------------------------------------------------------------------
hrcdr004.wcs_4 hrcdr004 Crystal Web Component Server Member of...
--------------------------------------------------------------------------------
hrcdr005.page_2 hrcdr005 Crystal Page Server Member of...
--------------------------------------------------------------------------------
HRCDR005.pageserver hrcdr005 Crystal Page Server Member of...
--------------------------------------------------------------------------------
HRCDR005.report hrcdr005 Crystal Job Server Member of...
Problem in Detail:
I work on the GCS project and I was wondering if you can help us out with an issue that's baffled us for days. We're basically running CE 9 deployed as 1 cluster across 5 NT machines (see bottom of email for a list of processes by server). The middle tier is implemented as 3 ActiveX dll components (VB 6) and CSP pages.
We've been running some load test on the GCS application this year when we discovered a strange behavior by the WCS and Page Servers. It seems that all all requests are executed sequentially even though we've configured say the Page Server to run with Maximum Simultaneous Processing Threads = 75. For instance, if we have only one page server running and we submit multiple/simultaneous requests to execute 2 reports from different machines; what we see in the log file is that one request will block until the the other request comes back before it starts executing (please see sample log below)
Page Server Log
12-07-05 23:02:13 - GetRSData() - HRCDR004 - SQL: {call PKG_CD_RPT_BMX.sp_get_ .... etc.
12-07-05 23:02:21 - GetRSData() - HRCDR004 - Time Elapsed (secs): 8
12-07-05 23:02:21 - GetRSData() - HRCDR004 - Records Returned: 168
12-07-05 23:02:21 - GetRSData() - HRCDR004 - SQL: {call PKG_CD_RPT_EXT.sp_comp_extra.... etc.
12-07-05 23:02:47 - GetRSData() - HRCDR004 - Time Elapsed (secs): 26
12-07-05 23:02:47 - GetRSData() - HRCDR004 - Records Returned: 121
We had first thought that it's an ADO connection problem, but we ran some tests to verify that requests are blocking on any DLL Function/Sub and not necessarily on database requests. The test we ran was as follows
- we wrote a simple Sub that writes to the log file, closes it and then sleeps for 10 seconds.
- we wrote a csp page that calls that 'sleepy' method
- we hit the csp page from 2 different browsers simultaneously
What we saw in the log file is that no two requests are executing the Sub at the same time. one always blocks until the first request returns.
Our next goal was to eliminate the DLL itself as the reason for the single thread execution (the way it's written, compiled, etc.), so we wrote a _VBScript/ASP page so we're not using the WCS to execute it. When hitting the ASP page simultaneously in the same fashion, we could see that request 1 and request 2 were executing the same Sub at the same time.
As further proof that there are no issues with the DLL, we noticed that unlike the WCS and the Page Server, the Job Server is able to handle multiple requests at the same time. Using the example above, if we had one Job Server running and we scheduled 2 reports at the same time, what we see in the log are 2 interlaced messages like so:
Job Server Log
12-07-05 23:02:13 - GetRSData() - HRCDR004 - SQL: {call PKG_CD_RPT_BMX.sp_get_ .... etc.
12-07-05 23:02:21 - GetRSData() - HRCDR004 - SQL: {call PKG_CD_RPT_EXT.sp_comp_extra.... etc.
12-07-05 23:02:21 - GetRSData() - HRCDR004 - Time Elapsed (secs): 8
12-07-05 23:02:21 - GetRSData() - HRCDR004 - Records Returned: 168
12-07-05 23:02:47 - GetRSData() - HRCDR004 - Time Elapsed (secs): 26
12-07-05 23:02:47 - GetRSData() - HRCDR004 - Records Returned: 121
Given the information above, could you please help us understand if:
- This is a known problem or expected behavior
- This is a limitation of this particular implemenation (VB/COM)
- There's anything we can do to mitigate this issue without introducing too much risk
++++++++++++++++++++++++++++ 5 Server Deployment+++++++++++++++++++++++++++++++++
Machine Name Type Server Group Selected
HRCDR001.aps1 HRCDR001 Crystal APS Member of...
--------------------------------------------------------------------------------
HRCDR001.eventserver hrcdr001 Crystal Event Server Member of...
--------------------------------------------------------------------------------
HRCDR001.RAS hrcdr001 Crystal Report Application Server Member of...
--------------------------------------------------------------------------------
HRCDR001.report hrcdr001 Crystal Job Server Member of...
--------------------------------------------------------------------------------
hrcdr002.cache_2 hrcdr002 Crystal Cache Server Member of...
--------------------------------------------------------------------------------
HRCDR002.cacheserver hrcdr002 Crystal Cache Server Member of...
--------------------------------------------------------------------------------
HRCDR002.WCS_1 hrcdr002 Crystal Web Component Server Member of...
--------------------------------------------------------------------------------
hrcdr002.WCS_2 hrcdr002 Crystal Web Component Server Member of...
--------------------------------------------------------------------------------
hrcdr002.WCS_3 hrcdr002 Crystal Web Component Server Member of...
--------------------------------------------------------------------------------
hrcdr002.WCS_4 hrcdr002 Crystal Web Component Server Member of...
--------------------------------------------------------------------------------
HRCDR003.pageserver_1 hrcdr003 Crystal Page Server Member of...
--------------------------------------------------------------------------------
hrcdr003.pageserver_2 hrcdr003 Crystal Page Server Member of...
--------------------------------------------------------------------------------
HRCDR003.report_2 hrcdr003 Crystal Job Server Member of...
--------------------------------------------------------------------------------
Input hrcdr003 Crystal File Repository Server Member of...
--------------------------------------------------------------------------------
Output hrcdr003 Crystal File Repository Server Member of...
--------------------------------------------------------------------------------
HRCDR004.cache_1 hrcdr004 Crystal Cache Server Member of...
--------------------------------------------------------------------------------
hrcdr004.cache_2 hrcdr004 Crystal Cache Server Member of...
--------------------------------------------------------------------------------
hrcdr004.wcs_1 hrcdr004 Crystal Web Component Server Member of...
--------------------------------------------------------------------------------
hrcdr004.wcs_2 hrcdr004 Crystal Web Component Server Member of...
--------------------------------------------------------------------------------
hrcdr004.wcs_3 hrcdr004 Crystal Web Component Server Member of...
--------------------------------------------------------------------------------
hrcdr004.wcs_4 hrcdr004 Crystal Web Component Server Member of...
--------------------------------------------------------------------------------
hrcdr005.page_2 hrcdr005 Crystal Page Server Member of...
--------------------------------------------------------------------------------
HRCDR005.pageserver hrcdr005 Crystal Page Server Member of...
--------------------------------------------------------------------------------
HRCDR005.report hrcdr005 Crystal Job Server Member of...