Is using HTLM to present large amounts of data in one page unreasonabl 1

http://www.codecouch.com/dan/

tbtcust · Feb 4, 2006

Wish it were that simple CliveC. Powers that be don't want to spend resources doing something that may not make sense. Hence, my solicitation to the group for comments (ammunition) to fight back.

BillyRayPreachersSon · Feb 4, 2006

Powers that be don't want to spend resources doing something that may not make sense.

That's very odd. What business would not want to carry out their own feasibility study before embarking on a project? It would not be hard, lengthy, or expensive for an HTML/JavaScript developer to mock up a page with 300,000 lines to find out the answer, so Im surprised your business do not want to do this.

This aside, I believe your developer is right. 300,000 rows of data help in an HTML page at one time will slow the browser down, IMHO. Receiving each page at a time from the DB may well give more hits on the DB, but it will be a hell of a lot quicker for the browser than trying to work through 300,000+ DOM nodes (and most likely more, because you're not just going to have 1 cell per row, I'd imageine... so for 2 cells it would be 600,000, etc).

Dan

[tt]Dan's Page [blue]@[/blue] Code Couch

[/tt]

CliveC · Feb 4, 2006

It would not be hard, lengthy, or expensive for an HTML/JavaScript developer to mock up a page with 300,000 lines to find out the answer, so Im surprised your business do not want to do this.

TBT103
I just tried it for 300000 lines of text data.

This is a row of data<br /> repeated 300000 times.

Loaded with no problem. How much data do you expect in each row?

Clive

tbtcust · Feb 4, 2006

CliveC, the average number per row 64 text data. And than you so much.

CliveC · Feb 4, 2006

For an "acid" test, assuming your database query is already coded and you are going to display the results with a table, then the whole "real test" could be done in less than half an hour.

Glad to be of help.

Clive

Foamcow · Feb 4, 2006

(for once) I agree with Clive, I just ran a test with 73+ characters per line and it displayed in the browser in 40 seconds.

The database query will have no effect on browser rendering times. Though a query returning 300,000 rows may make your database server hiccup (depending on the type of database server of course - I'm not really a large scale database guy).

Foamcow Heavy Industries - Web design and ranting
Buy Languedoc wines in the UK

CliveC · Feb 4, 2006

Foamcow,

My test worked in less than a second. Did you get your computer free in a box of cornflakes?

I upgraded to 4.0 in the Blackberry. Would you like to see some screenprints of your site (eyes and all)?

Clive

Foamcow · Feb 4, 2006

I did expect it to run work faster actually. It was just a simple for loop in PHP running in Apache 2 on my local machine. PHP is running as CGI though and I requested the page with Firefox 1.5.
The machine I tested it on is an Athlon64 3200+ Venice Core with 1Gb of RAM.

Knock yourself out with the screenshots.

Foamcow Heavy Industries - Web design and ranting
Buy Languedoc wines in the UK

CliveC · Feb 4, 2006

means I am just joking. Actually I just did it with an editor, repeating the line 300000 times.

If your are interested I will do the screenshots and send them to you privately. Don't want to waste time if you don't really want to see them.

It might, however, result in you agreeing with me more.

Kind regards,

Clive

Foamcow · Feb 4, 2006

Well I figured that running a loop server side was going to give a fair idea of what would happen. And it was easier for me to do that then keep pressing "ctrl+v".

I just tried it with Javascript and barring the script timeouts it took much the same amount of time.

Incidentally, IE crashed

Yeah, go ahead and send the screenshots.

Foamcow Heavy Industries - Web design and ranting
Buy Languedoc wines in the UK

CliveC · Feb 4, 2006

The editor I use allows me to say "r 300000" and the job is done. Your test is probably more realistic of the time it would take to display text created dynamically.

My point is, as in other recent debates, that you do not need to rely on what other people "say" but that you can just try things out for yourself.

Will send some screenshots soon, maybe by monday.

Clive

http://www.codecouch.com/dan/

Ghodmode · Feb 5, 2006

Raw HTML won't work. The browser will slow down too much. Based on my own experience, 5000 rows is already pushing it on IE or FF.

Just convert it to a different format before presenting the data. Choose a format that can embed itself into the browser window. A spreadsheet/CSV file is out of the question. I think that Excel is limited to 64k rows.

I suggest PDF. Then it's up to the speed of the actual application rendering the data. With 300,000 rows, there will still be a considerable delay, but I'll bet there are applications that are better suited for presenting the data than IE.

How to do it is beyond the scope of this forum, but you could do it in either PHP or Perl and you could post the question in either of those forums.

Another alternative is something that only presents a subset of the results at a time, then relies on something dynamic (like AJAX) rather than next/prev buttons to retrieve more data when your user is looking for it. I don't know how to do that, but I've seen it done.

--
-- Ghodmode

ChrisHunt · Feb 5, 2006

HTML won't work. The browser will slow down too much [...] I suggest PDF.

You are joking aren't you? That PDF would be enormous - much slower to download than any HTML equivalent.

Clive & Foamcaow have demonstrated that browsers can handle HTML files of this size, even if it takes time for them to do so. I don't think technical feasibility is the problem.

The problem is, what do you expect anybody to do with 300,000 rows of data on a web page? Assuming they are prepared to wait for it to download and that their browser is indeed able to handle it.

If they're gonna search it or summarize it or process it in some other way - do that process server-side and just serve up the results. If they're gonna drag it off into another application, Excel for example, you'd be better off doing it in, say, CSV format and putting it in a zip file.

A bit more information on what your data is and what you're planning to do with it may elicit some firmer advice, but I'm betting the answer won't be a 300,000 row table.

-- Chris Hunt
Webmaster & Tragedian
Extra Connections Ltd

chrissie1 · Feb 5, 2006

The problem is, what do you expect anybody to do with 300,000 rows of data on a web page? Assuming they are prepared to wait for it to download and that their browser is indeed able to handle it.

That's the best argument for not showing so much information to a user, even I get bored after 30 seconds.

Christiaan Baes
Belgium

"Time for a new sig." - Me

Foamcow · Feb 5, 2006

Well it shows the first few thousand results more or less immediately. It just takes time to work it's way to the full 300,000.

As Chris said though, 300,000 lines of data is a heck of alot to deal with. I'm not quite sure why you'd need 300,000 lines of data on a single page.

Foamcow Heavy Industries - Web design and ranting
Buy Languedoc wines in the UK

BillyRayPreachersSon · Feb 5, 2006

When I said that I thought this amount of data would slow the browser down, I should have clarified that I was speaking in terms of acceptabilty to most users.

At my last contract, we had around 100,000 rows of name/company data being returned in one hit, which took long enough to render in the browser (around 20-30 seconds - which most of the "power" business users reporting this delay as being unacceptable)... and paging through this also took a long time.

I stand by what I said earlier - take the hit from the database each time the user pages, and return just the records needed for that page. Your users will thank you for it in the long run. You [!]will[/!] get users complaining if you opt for the slow version - save yourself the hassle.

Dan

[tt]Dan's Page [blue]@[/blue] Code Couch

[/tt]

BabyJeffy · Feb 5, 2006

Ghodmode said:
Another alternative is something that only presents a subset of the results at a time, then relies on something dynamic (like AJAX) rather than next/prev buttons to retrieve more data when your user is looking for it.

This is another possibility - but does require round trip access to the server for each new page request. If it's ok to have this kind of traffic (but you just don't want to have to refresh the page) then AJAX could be just what you're after. It would easily degrade so that without javascript it worked (albeit you'd have to request the page and perform a full refresh).

Anyway... it sounds like there are plenty of options available to you... go with the most cost-effective one and be done with it [smile]

Cheers,
Jeff

[tt]Jeff's Page @ Code Couch

http://www.codecouch.com/jeff/

http://www.codecouch.com/jeff/blog/

[/tt]

What is Javascript? FAQ216-6094

CliveC · Feb 5, 2006

The originator did not say that he needed to display 300,000 rows, I just took it as a theoretical maximum to see how it worked.

Clive