AJAX and viewing very large datasets (50K+ documents)

Problem: How to facilitate a user request to get rid of the "paging" options at the bottom of a AJAX enabled web view and allow them to use only the scroll bar to navigate very large datasets (50k + documents).

I have been thinking about this all weekend and have drawn numerous wee drawings trying to get my head around what is required. The bull in a china shop approach would be download the 50k+ view data and then allow the JS or XSLT to render 50,000 documents .... Tried it .. OOOOO it was slow and required lots of answers to "this script is taking a long time it may be looping do you want to end it?" messages.

So the next option and the currently active one is to have 20 records per page and have the standard sort of page that has Page x of 2500 and allows the user to set x the background process goes off gets that page from the server and displays it. This has the benefit that the browser only has to get 20 documents from the server and render them which is quick and easy. BUT the user group doesn't really like this and reports back that they prefer a UI that has a scroll bar. Their main complaint is when you are looking at a Categorized view and the category is large when expanded then the data may be take several pages which you have to shuttle back and forth between.

And finally the last option.. what the user wants... a scroll bar option like in the client...This is the idea I am playing with at the minute,all be it as a mind game. My thoughts are these.

1. To ensure the Scrollbar accurately reflects the size of the dataset the holding element must be as near as possible the correct size to contain the dataset, but it does not necessarily need to be full of data. The size of this containing object will equal
   height of a single row * total number of rows in the dataset
   
   
2. The viewport need only contain the grid of Visible Rows say for example a table of 20
   rows. The size of the viewport if defined in rows is
   height of a single row * total number of rows in the viewport
   
   
3. When a scroll bar is clicked on either the down or up arrows the viewport is moved up or down by one row.
   
   
4. When the scroll bar is clicked either above or below the scroll tab the viewport is moved up or down by one page.
   
   
5. The current pixel position of the scroll bar in the holding element is returned by the scrollTop value. This can be used to work out the row that is at the top of the view port
   current position as returned by scrollTop / height of a row given as an integer value
   

that need to have a height of 50,000 *14 = 700,000px
So...lets say we have a view that is 50,000 rows long, we want to display this in a grid that has 20 visible rows and each row has a height of 14px.

Step 1 - Create a viewport which we know needs to have a height of 20*14 = 280px
Step 2 - Inside the viewport create a holder [div] which has a height of 700000px (50,000*14)

Step 3 - Place a grid inside in the Holder [div] which contains 20 rows 14px high

Step 4 - Register an event handler on the "scroll" event of the Holder [div]

Step 5 - When this event is triggered return the scrollTop and calculate the new starting row.
Step 6 - Ask the server to return 20 view entries from this starting row
Step 7 - Populate the table with these entries.
Step 8 - Position the table at scrollTop,0 within the holder [div]

Basically this will move the table which holds only the data visible in the UI to be centered in the viewport [div], the holder [div] has a height of 700,000px but only contains the table of visible data thus preserving the scale of the scroll bar without the requirement for having it full of data.

I have had a play and this seems to be "do-able" although there are some browser specific niceties that I have to get my head around as well as how to do interesting things like sortable columns, searches and how to return categorized views.

Has anyone else done something similar I wonder or would anyone be interested in the proof of concept I am working on? If so let me know ....

Ohh I wonder ...how does Ext.ND cope with large datasets?

UPDATE - In the comments below Joerg Michael didn't think this was a good idea...and Nathan Freeman followed up with the the point that ... perhaps the Notes Client view should be redesigned too.. :) Bum.. I was distracted by a user saying something that sounded like a good idea.. LOL...
Oh well ...being distracted are sunday afternoons in Feb are for :) However a non-notes solution of a similar nature has been looked at and put forward as a user extension for Ext as pointed out by Rich Waters as something to look at and as a point to think on... :)