Dataset and System Out of Memory Exception

[Meleagant]

All,

I'm trying to figure out why on occasion my production web site throws a System.OutofMemory Exception. It seems to happen when my users are returning a dataset with over 40,000 rows by 14 columns.

Just a little background. Users can search for records and those records are returned in an Asp:GridView control. There is an Asp:ObjectDataSource on the page which will go out and get the records. The EnableViewState of both the data source and grid are set to False. When users don't limit their search parameters and return 40,000+ records the page will usually fail and I will see an Out of Memory exception.

Anyone know what is the approx limit of records a dataset can handle (or even that a grid view can handle)?

I've brought up the solution of paging, but the users and the boss don't like it for reasons only they understand.

Any ideas?

Thanks,
-- Joe --


* Sine scientia ars nihil est
* Respondeat superior

 

[jmeckley]

they want 40,000+ records on screen at a time?
I would start by requiring at least 1 search criteria. however that may only reduce the results to 10,000 in which case you back where you started.

10 - 20 records per page is considered average. Without paging any solution is a hack or the core problem; volumn of data. Relate paging this data to paging on Yahoo or Google. Get there buy in, or better yet, let them think of it "first".

Once they do buy into paging, make sure you are paging on the database; not in memory.



Jason Meckley
Programmer
Specialty Bakers, Inc.

 

[Meleagant]

I'll never understand the mind of some of my users. Why the hec would they want to wade through a datagrid on 40,000+ records. When they could just as easily limit their search criteria to one Insurance Plan, or one Diagnosis (we deal with medical records), work those records and then run another search when they are done.

I even gave then a "Search Options" area where they could turn Paging On/Off and even input the number of records per page they would like. They don't seem to understand the the more records per page, the less responsive the page is going to be because there us just so darned much data. The source of the page is huge. Each row has a click event and sometimes there is a little lag between the click and the desired action.

A QUESTION ON PAGING: I've got paging set on the Asp:GridView. Its' DataSourceID is set to an AspataSource on the page. I've never thought of this but am I paging on the disconnected dataset (memory) or on the database? Their complaints has been that flipping between pages takes too long.

My rant aside, I was reading about serialization of a dataset. I don't really know what this means. I think it means I should write the dataset to a local file and have the web page's grid read the file? Can anyone give me the Dummy's version of dataset serialization and an opinion on if it would help?

Thanks,
-- Joe --


* Sine scientia ars nihil est
* Respondeat superior

 

[jbenson001]

What Jason is referring to is custom paging. With built in paging, you are retrieving all 40K+ rows each time you swtich the page, even if you are only showing x# of records. With custom paging, you only go get the records you want to show.

 

[jmeckley]

this might shed some light on why they don't want paging.
if loading a page takes 2-3 seconds then they want everything at once on screen so they don't have to wait for it to load again.

you don't need to serialize anything, so don't worry about this.
you may also want to disable viewstate. if you look at the source of the page, you will see viewstate is probally 40% of the total text. disabling viewstate for this page may allow for a preformance gain.
Note: this could cause other side effects so test throughly.

if you can disable viewstate and implement paging on the database the page will become much more responsive. So much so the users might actually use searching/paging

as said before though, without database paging this is a lost cause.

Jason Meckley
Programmer
Specialty Bakers, Inc.