Connection Pooling .NET 3.5 vs .NET 1.1

[andegre]

I have a very large ASP.NET web application that was recently upgraded from 1.1 to 3.5 framework. This has NOT hit production yet because I am still having performance problems.

One thing I am noticing is the number of "connections" on the application servers. (Test environment has 4 servers running Microsoft's NLB, Network Load Balance, while the production environment is running 8 application servers).

In production (which is still .NET 1.1) each server's # of connections doesn't get higher than 50, usually averages around 25 to 35.

In test (.NET 3.5), I had a "load test" last Friday and also Tuesday of this week and both times I have seen the # of connections on the [4] application servers get as high as 450, and is commonly between 100 and 300.

Also, in production, I average 1000 active users, and for the load test I had 500.

This is a HUGE difference between the environments/frameworks.

For the upgrade process, the code was only changed in order to get it to compile, NO CHANGES except for 1 class. That one class that had to be reworked was the XslTransform (1.1) changed to XslCompiledTransform (3.5). That whole class changed pretty drastically in their upgrade.

Do any of you architects/ASP pros/IIS pros know of any big changes that could be causing my issue? Is there some setting that I could change in a web.config or IIS website property that is new and needed for .NET 3.5?

Any help would be greatly appreciated! Thanks!

 

[jmeckley]

when you say connections are you talking about IIS connections or database connections?

in either case there have been advancements with .net, IIS and SqlServer to handle more throughput. I would think this means you can handle more connections simultaneously. which may explain your statistics.

while the numbers increased have you noticed a negative impact on performance? Is the response time for a user slower/faster? are the servers using more/less CPU?

1.1 to 2.0 is a big enough change. 1.1 to 3.5 is huge. properly disposing of objects is important. usually if an object implements IDisposable it's good practice to dispose of those objects. I would assume if this was the problem it would have been a problem in 1.1.

you should be disposing of ado.net objects (database connections) and streams, where your code is using them. there are a number of other objects implementing IDisposable, but these are usually the main culprits.

Jason Meckley
Programmer
Specialty Bakers, Inc.

faq855-7190
faq732-7259

 

[andegre]

These are IIS connections.

Users are complaining that this is slower than the production environment. Contrary to that, they are able to do approximately the same number of transactions in the load test as they do in the same time period in the production environment, but also complian that the system is "slower/much slower" than production. (That may be because they just continually do work in test, where in production they will help a customer, then wait for the next customer to do another transaction)

Also, the servers CPU usage is also AT LEAST twice as high. For example, the production servers will run on average about 20-30% CPU usage including spikes up to 80% (but it's just a spike). Whereas, the test servers will CONSISTENTLY run 50-80% with spikes at 100%.

I will go through to check for disposing, but like you said, it wasn't a problem in .NET 1.1 so their supposedly shouldn't be an issue.

 

[andegre]

Jason, I've been searching through the code to make sure everything is getting disposed correctly (mostly all connections are in "using" statements), but I came across this method:

        public DataSet ExecuteDataset(IDbConnection connection, IDbTransaction transaction, CommandType commandType, string commandText, params IDbDataParameter[] commandParameters)
        {
            try
            {
                //create a command and prepare it for execution
                IDbCommand cmd = connection.CreateCommand();
                this.PrepareCommand(cmd, connection, transaction, commandType, commandText, commandParameters);

                //create the DataAdapter & DataSet
                DataSet ds = new DataSet();
                IDbDataAdapter da = null;

                //Determine type of connection and create appropriate adapter
                switch (connection.GetType().FullName)
                {
                    case "System.Data.SqlClient.SqlConnection":
                        da = new SqlDataAdapter((SqlCommand)cmd);
                        //fill the DataSet using default values for DataTable names, etc.
                        break;

                    case "System.Data.OleDb.OleDbConnection":
                        da = new OleDbDataAdapter((OleDbCommand)cmd);
                        //fill the DataSet using default values for DataTable names, etc.
                        break;
                }
                //Fill the dataset
                da.Fill(ds);

                //return the dataset
                return ds;

            }
            catch (SqlException e)
            {
                LogError(String.Format("{0}::{1}", this.GetType().FullName, System.Reflection.MethodBase.GetCurrentMethod().Name), String.Format("Message: \n{0}\n\nStack:\n{1}", e.Message, e.StackTrace));
                DBApplicationException dbEx = new DBApplicationException(connection.ConnectionString, commandText, String.Format("A SqlClient.SqlException happened in {0}! {1}", System.Reflection.MethodBase.GetCurrentMethod().Name, e.Message), e);
                dbEx.MethodName = String.Format("{0}::{1}", this.GetType().FullName, System.Reflection.MethodBase.GetCurrentMethod().Name);
                throw dbEx;
            }
            catch (OleDbException e)
            {
                LogError(String.Format("{0}::{1}", this.GetType().FullName, System.Reflection.MethodBase.GetCurrentMethod().Name), String.Format("Message: \n{0}\n\nStack:\n{1}", e.Message, e.StackTrace));
                DBApplicationException dbEx = new DBApplicationException(connection.ConnectionString, commandText, String.Format("A OleDbException happened in {0}! {1}", System.Reflection.MethodBase.GetCurrentMethod().Name, e.Message), e);
                dbEx.MethodName = String.Format("{0}::{1}", this.GetType().FullName, System.Reflection.MethodBase.GetCurrentMethod().Name);
                throw dbEx;
            }
            catch (Exception e)
            {
                LogError(String.Format("{0}::{1}", this.GetType().FullName, System.Reflection.MethodBase.GetCurrentMethod().Name), String.Format("Message: \n{0}\n\nStack:\n{1}", e.Message, e.StackTrace));

                DBApplicationException dbEx = new DBApplicationException(connection.ConnectionString, commandText, String.Format("Error in {0}! {1}", System.Reflection.MethodBase.GetCurrentMethod().Name, e.Message));
                dbEx.MethodName = String.Format("{0}::{1}", this.GetType().FullName, System.Reflection.MethodBase.GetCurrentMethod().Name);

                throw dbEx;
            }

        }

Does the IDbCommand need to have a close and dispose inside this method?

 

[jmeckley]

ok, so you have both a preceived problem, "it's slower", and a measureable problem, 20-30% to 50-80%.

do you have a profiling tool? both jetbrains and ants have a profilers. i use jetbrain's dotTrace. if you can get the metrics for the 1.1 and 3.5 code base you can find your bottlenecks and solve them one at a time. sometimes resolving 1 or 2 bottlenecks can have a drastic effect on the user experience.

are you using any 3rd party libraries in you're project? if so have you updated these binaries as well to take advantage of the 3.5 features?

if possible can you target the 2.0 framework, rather than 3.5 and measure the statistics from 1.1 to 2.0. once this is acceptable you could upgrade from 2.0 to 3.5.

Jason Meckley
Programmer
Specialty Bakers, Inc.

faq855-7190
faq732-7259

 

[jmeckley]

it should, but this wouldn't be the cause. at least I highly doubt it.

Jason Meckley
Programmer
Specialty Bakers, Inc.

faq855-7190
faq732-7259

 

[andegre]

Yes, I've run ANTS Profiler on one of the servers before and it pointed to the XslTransform to XslCompiledTransform change that I implemented. I have not run it since that was fixed though. Per Microsoft's request, it was removed while they were trying to help with performance problems.

This program was developed by a 3rd party vendor and they were the ones that did the inital upgrade to 3.5, but we ran out of contract hours so they have not been available since giving us the upgraded code.

Basically, I have access to all of the 3rd party code and can change it how I see fit.

About the 3.5 features, no features beyond 1.1 have been implemented, all that was done by the vendor was get the code to compile in .NET 3.5 (2.0 for web) and then they handed it off.

 

[jmeckley]

so, if only change was to compile the code on 3.5, but no updates were made in user functionality, no bug fixes, or even technical refactorings to gain the value of .net 3.5 BCL, why update at all?

Jason Meckley
Programmer
Specialty Bakers, Inc.

faq855-7190
faq732-7259

 

[ca8msm]

It may be useful to start implementing some performance counters throughout areas that you see as a potential bottleneck. Some carefully placed Trace.Warn statements may help identify which sections are causing the application to appear slower. This should then hopefully give you a starting point to track down any issues.

Mark,

Darlington Web Design
Experts, Information, Ideas & Knowledge
ASP.NET Tips & Tricks

 

[andegre]

Jason, we needed to be in at least .NET 2.0 to receive their (same Vendor) "new" Cash Drawer system. Also, our servers are getting updated to 64-bit in the very near future.

 

[andegre]

Doing another load test as I type this (for a 1/2 hour) after I implemented some more caching. This was our last try to try and speed up performance because we "want" to go live this weekend. If it's not better, then obviously will look back into everything.

I wasn't able to get the dispose change noted above implemented, i'd like to know more about the trace.warn thing...I'll google it in the meantime.

Thanks for all suggestions fo far!

 

[jmeckley]

how much caching is implemented currently? this may be the problem. caching puts a higher load on the server because the data is sorted in memory (usually).
and if the cached data is specific to the user the amount of memory is that much more.

Jason Meckley
Programmer
Specialty Bakers, Inc.

faq855-7190
faq732-7259

 

[andegre]

None of it is per user (server-side). There IS caching on client-side that would obviously be user specific.

I guess I don't know how to give a good number of "how much is cached", There are 1000's of web methods, where probably 100 implement caching (just a guess).

Is that alot (insert movie reference here)?

 

[jmeckley]

caching on the client shouldn't be a problem. i was concerned you were caching on the server per user.

Jason Meckley
Programmer
Specialty Bakers, Inc.

faq855-7190
faq732-7259

 

[andegre]

I meant to say "none of it on the server-side is per-user", but there are approximately 100 methods that are cached on the server side. Then some more implemented on client-side.