web app effected by database server change - please help! 2

[woodrg]

Please forgive the lengthy post, but this really is the short version – honest...

A couple of weeks ago we moved our oracle databases from a Windows 2003 server to a Windows 2008 server and upgraded them from 32 bit processing to 64 bit processing. The transition has been perfect with all our applications except in one case.

We have a web app, developed with .net 2003 several years ago, that has had a recurring error we have suspected to be caused by corrupted cache, session variables or application variables. Prior to the database move (and subsequent server upgrade) the error only occurred once every 2 to 3 weeks. The accepted "fix" was to just restart the application pool and all would be well for another 2 to 3 weeks (yes, we know – not cool, but those folks aren't here anymore). However, once the application began being used after the database move, it started happening 3 or 4 times a day and it almost seems to be getting worse.

How it starts is that one or two of the end users will get an error message indicating "index out of range" or "object not set to an instance of an object" – they are in the same process/page of the application (because this is the time of day that this process is accomplished across the contract), but they all bomb out on different lines. So they close the app/browser and go back in, they may be able to work for a bit longer but then will get a similar error again. Eventually all of the users (about 6 at a time) will begin receiving the same errors and no one can do anything – then we restart the app pool and everything's okay for a while. Again, previously this was fine and gave us a few more weeks before we saw it again, but now we're lucky to get 20 more minutes out of it.

I know this is an issue that needs to be addressed in the code itself, but it's painfully obvious that something to do with the moving the databases to a 2008 server or to 64 bit processing has had an effect on this application and escalated this error that was once tolerable to bringing the application to its knees.

Anyone have any ideas? I'm hoping that someone else has experienced something similar after moving to or upgrading a server and may be able to point us in the right direction so that we can at least get the application back in a useable state on a more consistent basis for the end users.

(previoulsy posted in VB.NET 2002-2008 Forum, thanks to RiverGuy for directing me to this forum)

Becky,
Ft. Rucker

 

[Turkbear]

Hi,
What Oracle client software is on the web server?

Have you tried connecting using that client's SqlPlus?





To Paraphrase:"The Help you get is proportional to the Help you give.."

 

[ca8msm]

I'd guess that the real errors are actually getting masked and the ones that you see are just untrapped side effects of what is really going on. What exception logging is done in the application, how are errors handled and where (if at all) do the errors get stored and sent to?

Mark,

[URL unfurl="true"]http://lessthandot.com[/url] - Experts, Information, Ideas & Knowledge
[URL unfurl="true"]http://mdssolutions.co.uk[/url] - Website Design
[URL unfurl="true"]http://aspnetlibrary.com[/url] - An online resource for professional ASP.NET developers

 

[woodrg]

Hi Turkbear - we have oracle 10g running across the board. connectivity has been verified on the app server as well as all other workstations requiring access to the data. Regardless, the the application returns data fine, at least until it decides to take a dirt-nap.

ca8msm is right i'm sure; the error's we're seeing are just the side effects. Errors are e-mailed to us as well as to a mail-box for easy reference. however, they're not trapping the event that is causing the error, only the errors that are caused by it. below are a few examples of what we are getting back. Of course the users only see "an error has occured - the helpdesk has been notified - yada yada yada"


Error occurred: mscorlib USER = FRMAINT\OlsonE
Source: mscorlib USER = FRMAINT\OlsonE
Error Message: Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index
Stack trace: at System.Collections.ArrayList.RemoveAt(Int32 index)
at System.Data.RecordManager.NewRecordBase()
at System.Data.DataTable.NewRecord(Int32 sourceRecord)
at System.Data.DataRow.BeginEdit()
at System.Data.DataRow.set_Item(DataColumn column, Object value)
at System.Data.DataRow.set_Item(Int32 columnIndex, Object value)
at Aircraft.Schedule.DataGrid_From_Variable(TypeData DisplayWhat) in C:\webprojects\Aircraft_soln\Aircraft_proj\Schedule.aspx.vb:line 382 Error String: System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index
at System.Collections.ArrayList.RemoveAt(Int32 index)
at System.Data.RecordManager.NewRecordBase()
at System.Data.DataTable.NewRecord(Int32 sourceRecord)
at System.Data.DataRow.BeginEdit()
at System.Data.DataRow.set_Item(DataColumn column, Object value)
at System.Data.DataRow.set_Item(Int32 columnIndex, Object value)
at Aircraft.Schedule.DataGrid_From_Variable(TypeData DisplayWhat) in C:\webprojects\Aircraft_soln\Aircraft_proj\Schedule.aspx.vb:line 382


Error occurred: Aircraft USER = FRMAINT\nancer
Source: Aircraft USER = FRMAINT\nancer
Error Message: Object reference not set to an instance of an object.
Stack trace: at Aircraft.Schedule.DataGrid_From_Variable(TypeData DisplayWhat) in C:\webprojects\Aircraft_soln\Aircraft_proj\Schedule.aspx.vb:line 500
at Aircraft.Schedule.btnSchedule_Click(Object sender, EventArgs e) in C:\webprojects\Aircraft_soln\Aircraft_proj\Schedule.aspx.vb:line 1237 Error String: System.NullReferenceException: Object reference not set to an instance of an object.
at Aircraft.Schedule.DataGrid_From_Variable(TypeData DisplayWhat) in C:\webprojects\Aircraft_soln\Aircraft_proj\Schedule.aspx.vb:line 500
at Aircraft.Schedule.btnSchedule_Click(Object sender, EventArgs e) in C:\webprojects\Aircraft_soln\Aircraft_proj\Schedule.aspx.vb:line 1237


Becky,
Ft. Rucker

 

[jmeckley]

you may not be disposing of disposable object properly. when logging an exception the tostring() member will output everything related to the exception.
1. type
2. message
3. stack trace
4. inner exception(s)

I would also guess that somewhere in your code you are doing this
try
{
...
}
catch(exception)
{
return a default value,
but no logging is done.
}

I would ditch the entire try/catch block and let the exception bubble up. at the most I would catch a specific exception type, but that's only if I knew that situation could occur.

I would also assume you have code like this

var connection = new XConnection(connection string);
var command = new xCommand(connection);
...configure command
var results = new DataTable();

connection.Open();
results.Load(command.ExecuteReader());
connection.Close();

return results;

this will cause problems because the connection and command are not properly disposed of.

var results = new DataTable();
using(var connection = new xConnection(...))
using(var command = connection.CreateCommand())
{
   ...configure command
   connection.Open();
   results.Load(command.ExecuteReader());
}
return results;

is a better alturnative as this will properly close and dispose of commands and connections for you.

Jason Meckley
Programmer
Specialty Bakers, Inc.

 

[woodrg]

thanks, that will help with our error handling, but doesn't sound like it will help with the session variables being dropped or whatever is causing the errors to be triggered.

Any idea of how the differences between a Windows 2003 server and a Windows 2008 server that hosting an oracle database might have an effect on a web app's sessions or cache?

i know the answer is to correct what is dropping the session variables, but if we can just slow it back down to it's original impact it would buy us some time to do so.

Becky,
Ft. Rucker

 

[jmeckley]

Prior to the database move (and subsequent server upgrade) the error only occurred once every 2 to 3 weeks. The accepted "fix" was to just restart the application pool and all would be well for another 2 to 3 weeks (yes, we know – not cool, but those folks aren't here anymore).

you are now paying for the sins of the past. Sooner or later you need to fix the root cause. Sooner is now

However, once the application began being used after the database move, it started happening 3 or 4 times a day and it almost seems to be getting worse.

Does this mean the application wasn't (heavily) used prior to the OS/DB upgrade? If so, then the OS/DB is not the issue. The fact that users are now using the system exposes a coding bug.

I would also upgrade your .net 1.x project to .net 2.0 at a minimum. the jump from 1.x to 2.0 is night and day difference (hence the breaking changes upgrade). Once you're at 2.0 moving to 3.x is easy as the changes are only additive, not breaking.

If this is an OS/DB issue, then you will want to find a forum (tech support) dedicated to those platforms. as asp.net has nothing to do with either of those technologies.

Jason Meckley
Programmer
Specialty Bakers, Inc.

 

[woodrg]

Thanks so much for your interest in this.

I agree 100%, there's definetly a coding bug. unfortunately, doing what it takes to fix the app is not going to get it up and running anytime soon. this application is such a mess that we've grown leary of making any changes to it as it's probably a good text book example of what not to do. In fact the last changes we had to make we handled thru a database trigger to keep from having to alter the applciation (sad, i know, but current workload per current staffing is highly out of balance). we do have another developer muddling thru the application now, but it's been quite a chore just following it thru trying to figure out what it's doing, let alone what edits are safe to make without messing up someting else.

my thought at this point is to at least slow it back down to the tolerable level by undoing whatever effect the OS/DB upgrade had on it. that would safely put us back in a position of meeting contractual requirements and buy us time so that we can correct the application as it should be using our current development standards with visual studio 2008.

the app has been used the same since it was first written 4 or 5 years ago: very heavily during a 1.5 to 2 hour period twice a day. so no change there.

i'll post else where as well as you suggest, i was just hoping that someone else working from the application point of view might have experienced a similar effect with a similar upgrade and might recognize the issue.

thanks again for your help and any other ideas you may have!


Becky,
Ft. Rucker

 

[jmeckley]

In fact the last changes we had to make we handled thru a database trigger to keep from having to alter the applciation (sad, i know, but current workload per current staffing is highly out of balance)

I completely understand. I have a few apps like that myself. written by me no less

When you have some down time (either at work or home) check out the concepts of uniting testing and unit testing frameworks. .net has a bunch: nunit, mbunit, xunit, mstest, etc.

creating unit/integration test allow for automated regression testing and help deter the fear of making breaking changes. because if changes are made that do break code, you will know right away.

combine unit testing with the concepts of SOLID and application development becomes easier to manage.

good luck hunting down the problem.

Jason Meckley
Programmer
Specialty Bakers, Inc.

 

[ca8msm]

I'd guess that the OS isn't really responsible, and there's probably another explanation of why the crash is happening more often. Have you tried remote debugging the live application to see where it is crashing out when the errors are starting?

Mark,

[URL unfurl="true"]http://lessthandot.com[/url] - Experts, Information, Ideas & Knowledge
[URL unfurl="true"]http://mdssolutions.co.uk[/url] - Website Design
[URL unfurl="true"]http://aspnetlibrary.com[/url] - An online resource for professional ASP.NET developers

 

[woodrg]

actually, i've been called off the hunt and we've been directed to rewrite the framework that contains the data provider and handles the connectivity using the oracle oracle provider instead of the microsoft oracle provider. not sure this is going to help, but off we go!

regardless, my intention was to get some relief to the end-users without having to wait for us to complete the development process.

thanks again to each of you for the advice, we'll definetly be checking into it.

Becky,
Ft. Rucker

 

[jmeckley]

actually, i've been called off the hunt and we've been directed to rewrite the framework that contains the data provider and handles the connectivity using the oracle oracle provider instead of the microsoft oracle provider. not sure this is going to help, but off we go!

I would highly recommend not waisting your time re-inventing the wheel. data access is a solved problem.
checkout the following frameworks:
nhibernate
active record
wilson ORMapper
LLBL Gen Pro 2

they are all ORM/DAL frameworks which solve the data access issue. I have worked will all but wilson ORMapper. LLBL is good if you like the MS RAD development technique. I personally love NHibernate. AR is great too, but with the FluentNH project AR isn't really necessary.

the time spent learning anyone of these frameworks will take less time than trying to write your own.

Jason Meckley
Programmer
Specialty Bakers, Inc.