3.5.08

Rails endless page plugin

Few days ago I have looked for implementation of the 'endless page' in the Rails. When I'm talking about endless pages I mean alternative to the standard paging used all over Internet. Alternative variant works like initial list of items that sends background Ajax requests and adds received data to the end of list. You can see examples in the DZone and Google Reader if you want. I have googled a lot and talked with authors of various articles related to this functionality. It seems like, there was no such Rails plugins and I have decided to create it.

Theory.
Common view on the functionality. When user opens web page in the background JavaScript timer starts and periodically checks position of the scroller. If user scrolled page and scroller is in the bottom part of scrollbar then Ajax request is sended and additional data recieved and added to the bottom of initial page. If user have recieved all existing data then timer is stopped.

Software need to have.
Plugin doesn't require any gems or plugins installed. Ajax request is executed using Prototype library.

Example of usage.
In the beginning we should have basic Rails application that is easy to create so I'll not tell about this part of project. It should be simple index page with list of items limited by, lets say, 500 first items. Each item look and feel is set by Rails partial.

These are steps needed to implement:
0) Download plugin by entering 'ruby script/plugin install git://github.com/shine/endless_page' in the project root folder. In the vendor/plugins/endless_page folder enter 'ruby install.rb'. During installation endless_page folder will be created in the app/views. Several files will be copied to the new folder and to the public/javascripts.

1)Controller. In the index action session variable setting should be added. Variable will contain id of last item shown to the user:
def index
@articles = Article.find(:all, :order => 'id', :limit => '500')

session[:last_article] = @articles.last.id.to_s
end


2)Additional method should process Ajax request:
def append_items
@additional_objects = Article.find(:all, :conditions => [params[:sort_by]+' > '+session[:last_article]], :limit => '100')

if @additional_objects.blank?
render :template => '/endless_page/have_nothing_to_add'
else
session[:last_article] = @additional_objects.last.id.to_s
render :template => '/endless_page/append'
end
end

Here is one important detail: name of the array passed to the RJS should be @additional_objects. Otherwise RJS will have nothing to process. If..else..end block should be equal for the all implementations of endless pages in the project with exception of session variable name, of course.

3) View. In the head of the page Prototype library and additional JS file with plugin related functions should be included:
<%= javascript_include_tag :defaults %>
<%= javascript_include_tag 'endless_page' %>


4) HTML component that will contain elements added by Ajax should have "id" attribute. The most simple variant of this attribute value will be 'container'. If you need to change it then you will see a way to do so below.

5) After HTML container there should be script that will configure and run JavaScript timer:
<script type="text/javascript">
var ep_config = new EPConfig();

ep_config.auth_token = encodeURIComponent('<%= form_authenticity_token%>');
ep_config.url = '<%= url_for(:controller => 'articles', :action => 'append_items')%>';
ep_config.partial = '/articles/item';

start_endless_page_listener(ep_config);
</script>

In the beginning we are creating configuration object with several default values. After that we are setting few attributes that are unique for each endless page: Rails auth token, url of the controller/action that should proces Ajax request and partial that should be used to show new added items. Finally, we are starting JavaScript scroller checker.

If you need to change container id then you can do it before start of listener by adding next line:
ep_config.container = 'my_favourite_name';


Additionally, you can change other settings like period of scroller checking, point of Ajax activation and so on. Full list you can see in the bottom of the endless_page.js in the public/javascripts folder.

It seems like enough for plugin usage.

Any thoughts and comments are welcome.

Update: some JavaScript changes were made

18 комментариев:

Pete said...

Awesome!

I did a little write-up on our Unspace Rethink blog:

http://rethink.unspace.ca/2008/5/3/check-out-the-rails-endless-page-plugin

Maxim Kulkin said...

Nice plugin, although it seems like a bit awkward to use (judging by example given, I haven't chance to play with the code).

First, I really don't like storing next page number in session. It would be better if javascript component tracked that itself and passed desired number as query param.

Second, well, not a bad thing actually, but I would like to see some component, that would 1) show real scrollbar (e.g. scrollbar that wouldn't expand every time next page is loaded, but rather show final item count); and if so 2) provide ability to access pages randomly (e.g. you open browser page and see 1st chunk of results, then you click on scrollbar near the end and see, say, 50th chunk of data, then you scroll up and see 49th chunk and so on) and component would track what chunks are already loaded and what are not; and if so (to get this list complete) 3) allow items of variable height (width?) and provide ability to calculate the current chunk when you jump to some place using scrollbar.

I might devote some time developing it some time soon.

bewhite said...

Maxim, thanks for your comments.

JavaScript instead of session. Good idea. I'll work with it.

Scrollbar component. Interesting idea but I don't sure how are you planning to get whole height of scrollbar without loading all data to the page. If you will do it then data with 1592 items can kill poor users dialup connection. :)

I have another idea of floating div with search. It should let user to find(and load from server if needed) items without scrolling through all the list and waiting for all data loading.

Maxim Kulkin said...

bewhite, no, I don't suggest loading all data. I suggest that when page is loaded (and pageless loader component is instantiated) you pass the total number of items in the list to pageless component (and optionally the height of each item so that scroller could position properly). Then, pageless scroller component keeps track of what chunks of data were loaded and loads required chunks when necessary). It can also unload chunks if needed (e.g. to keep memory consumption adequate).

The problem I'm facing which stops me from writing it is that I can't decide how to layout chunks. There are two variants:
1. Absolute positioning - each chunk is positioned relatively from it's container. Pretty easy, can be implemented on top of any element combination (eg. <ul> + <li>, or <div>s). Downside: doesn't allow resizing elements when they were loaded (e.g. I have a design that allows unfolding items and changing them into inplace edit forms).
2. Use 'spacers' - some element (e.g. <div>) that will be placed in place of 'not yet loaded' data. When data is loaded, spacer can be shrunk, split into two or removed. This allows resizing elements, but is harder to implement on some particular element combination (e.g. it is not correct on <ul> + <li>s, using <li> as a spacer element).

The second variant I like more so I probably will stick with it.

bewhite said...

I think that second variant is better. I have checked several websites with big lists (Google, Yandex, Rambler, DZone, ...). Only Rambler uses ul+li. The rest websites are using divs. So I think we have to make it workable with divs and tables. It will be enough for most of component users.

As I see you are planning to create functionality. If so then I think it will be handy to use common SVN repository. If you are agree then send me email to the victor.brylew at the gmail_com with handy login and password. BTW, plugin is stored as the part of blank Rails project so you can use this project during component implementation.

bewhite said...

Update.

Last used value is stored in the JavaScript only and passed to the Controller in the params[:last_value] variable.

This is why in the View JavaScript plugin users have to add another one string that defines initial last value:

ep_config.last_value = <%= @articles.last.id.to_s %>;

Maxim Kulkin said...

Hey, what that "last_value" option for ? I though you just show first page and user javascript to load the next one and so on. In fact, the controller could just treat the data as pages. In this case if controller was designed to do pagination in traditional way, nothing needs to be changed (except that it should now return rendered items if called via Ajax request).

BTW, I just finished implementing the pager that I was talking about. I'm testing it right now. Probably will make a post on it on my blog soon.

bewhite said...

last_value is part of EPConfig structure. It contains id value of the last item recieved by View. This value is sended to the Controller by POST so Controller can determine what last information was sent to view and what next information View expects to get..

ep_config.last_value should be set during initial page rendering and changed by RJS during each page update.

What is the URL of your blog?

Maxim Kulkin said...

When you render page as a whole, the list position should be RESET. When you open (or reload) that page, you start viewing the list FROM THE BEGINNING. That's why (if used as described above) this setting will always be zero: you open the page, see *first* chunk of items, scroll down, the page loads *second* chunk of items, you scroll down more (to the end of second chunk), the page loads *third* chunk, and so on... Then you reload the page and see *the first* chunk of items again.

Second, why are you using ID as a mean to determine, which items to select next ? The easier solution is to use offset in ordered collection (assuming that you ordered collection or use natural order). I can think of just one application of "ID as a stop point" thing - when you order by ID (or use natural database order where items also ordered by ID): select * from table where id > 123. This most likely won't work if you order by some other field.

PS You can click on my name and see my profile. From that page you can see all blogs that I have (I have just one blog which I update from time to time).

bewhite said...

We are talking about different things. Last_value is the ID of the last item in the previous chunk.

ID usage is just a convention. You can use different fields by setting alternative value in the 'sort_by' field of the JavaScript configuration object.
There will be problems with offset usage if someone will delete records already recieved by user. In this case user can get same list item several times.

Maxim Kulkin said...

Ok, your thoughts on ID being used to workaround problems if item is removed make sense. However, I don't think that storing ID of last record will solve the problem because if user deleted record with that ID, there will also be problems. Also, with your approach (loading data that _follows_ already received ones) will most likely cause only the problem, when some items ARE NOT SHOWN.

Also, I think almost all traditional implementations (pagination with links on particular page) do no workarounds on it. Solutions like ours could cause more confusion to users, because it will make impression that items follow one another (but in fact there would be some items that are skipped).

The workaround I see (and it is possible in my solution) is to detect if list is changed and reload it.

Moreover, I think it is not worth dealing with, at least in my case.

bewhite said...

maxim, can you describe more detaily cases when items are not shown?

Errors in traditional implementations are not reason to repeat it in the new plugins. Meanwhile, I'll write about this detail to the will_paginate author. Maybe he have some thoughts about this problem.

Maxim Kulkin said...

Those cases are when you open page with first 10 items and while you are watching them someone else deletes 9th item (which is although shown on your page). Then you go to page 2 (with items 11 to 20), but because of 9th item was deleted you can't see 11th item (because it has position 10 in the result now).
The same error will happen in my implementation (because I also rely on offset when displaying results). I'm not sure if it will/won't work in your implementation, but I don't understand what will you do if someone deletes item with the ID you have remembered to be last.

Hey, btw, I have finished my scroller and ready to publish it/show to the public. The only thing is that I don't know how to do it properly: there sure should be some usage examples and _demos_. Demos require some server that could provide data in proper format (which is parts of html code). Unfortunately I don't have any server to host demos on.

bewhite said...

The way to fix problem you have mentioned is forgetting about offsets and using additional WHERE statement instead of it. For example:

SELECT * FROM articles LIMIT 20, 10

should be changed to

SELECT * FROM articles WHERE id > 20 LIMIT 10

In the second case you will make offset not by row number in DB table but by ID number that were not changed after 9th item was deleted.

It will works correctly even if deleted item was last in the previous received chunk. It doesn't matter for SQL.

Maxim Kulkin said...

WTF? If you have item A with id 1, item C with id 2 and item B with id 3, how are you going to display them ordered by title (A, B then C) 2 items per page ? With your scenario you will have items A (with id=1) and B (id=3) on the first page. And what then ? You will do "select id > 3" to get items for the second page ?

bewhite said...

You can use other fields, not just id:

SELECT * FROM `titles` WHERE title > 'B' ORDER BY title LIMIT 2

ID is the set by default but you can change it if you really need it.

karmoo said...

Endless great, this endless_page. I sought for such thing for making a book more readable, i.e. without button-clicking from page to page.
First try: without success. Prototype was looking for an item-file (for new added items) in /app/view/members and didn't find it. Neither on Linux nor on Vista did install.rb produce this file. So I wrote _item.html.erb and placed in it

/div\/% for article in @additional_objects -%\
/%= article.text %\;
/% end %\//div\

For testing I limited item articles and additional objects to 10. Now there was nice scrolling possible from chunk of 10 items to the next ... But unfortunately after the first 10 articles (book pages) the next 10 items are always the same in never ending repetition.

Since my Rails-expertise isn't endless great, I would appreciate some helpful hints.

Thanks in advance.

bewhite said...

It is general propose plugin. So it can be used to show various things in thread: articles, news, table rows and so on. This is why you have to create partial for list item yourself according to your needs. Item partial should contain view of single item list. For example, in your case it should be something like:

<%= article.text %>

Another one thing you have to remember is id="container" for the div with set of articles.