Livelets

What is it?

The idea of this extension is to allow live articles to be transcluded which update automatically on change in a non-polling, fully event-driven way. It will use the parser-functions syntax to make it very similar in usage to normal templates. It's called a livelet since it's very similar in concept to the re-usable areas of options and information in a web page usually referred to as a portlet, except that these are able to accept spontaneous incoming requests, not just to responses to their own requests, which means that livelets are able to stay up to date or be communicated with dynamically without any regular polling being necessary.

It's being partially paid for by client needs, but will of course be free and LGPL - I say partially paid for because it's being done in a far more powerful and generic way than the client requires - in fact they don't even need the live aspect, it's only the division of the page into separate requests to make local caching more efficient that they really need!

But since we already have all the code written to handle sockets between PERL and SWF, which won't ever be used for anything now since our peer interface is now in C and SDL, it seemed like a good opportunity to get it working in the field.

Development Plan

The highest priority aspect is the new transclusion method allowing the contents to be a separate request so that it can be independently cached by the browser. So with that in mind, the order of development tasks is as follows:

• livelets.php - parser function and tag hook returning java XMLHTTP requestor, livelet action
• livelets.js - the function that does the request and update of content
• livelets.as - set up XMLSocket object in JS using SWF (just use to replace XMLHTTP requests first)
• livelets.pl - spawn a thread for each SWF, and propagate changes

Livelet Components

livelets.php

• Ensure livelets.pl daemon is running
• Add the #livelet parser function and internal hook
• Add the livelet action to the UnknownAction hook
• Add the AfterSaveComplete hook
• Insert the SWF (compiled from livelets.as) into the page somewhere invisible but active
• Add the livelets.js headscripts

Parser Function

Converts the syntax to tag-hook along with the parameters

Internal Tag Hook

There's no HTML content because the parser-function doesn't actually do any transclusion, the information that arrives here is the article name and parameters. This hook returns the JavaScript container which makes a separate request for the content when it runs in the browser. The request uses the live action and passes the parameters used in the original parser-function transclude statement.

• see Rob's people-search.js for example of this kind of GET and insert method.

UnknownAction Hook

If an article is requested with action=livelet, then it will be returned without any HTML head, scripts, body or any skin components suitable for being inserted directly into its container by the client JS.

The JS must pass any parameters the transclusion statement originally exhibited, so rather than just returning the article, it would probably change the content AfterDatabaseFetch to a normal transclude with the parameters.

AfterSaveComplete Hook

The job of this hook is to send notification to all the SWF's which have live-containers that are out of date. To do this we need a persistent list of all the current SWF clients and their live article titles with Last Modified times. Each time a save occurs, we update the list.

Only the livelets.pl daemon has up-to-date information about which SWF's need which articles, so this save hook should first ask the daemon for its version of this list, and then return an updated one. The daemon can then notify each SWF with out of date content by sending a list of the container id's needing to request new content.

livelets.js

The JS can send HTTP requests itself with the XMLHTTPRequest object, but must rely on the SWF to call some kind of onData event handler function. Currently the onData function doesn't need any data or parameters, it's simply a notification that something on the wiki has changed and that the live containers should check if their content is up to date.

Note: It may be a good idea to get the SWF to do the send as well (or make it optional) because the XMLHTTP object seems quite browser dependent compared to Flash6.

After that's working, we'd want to get the livelets.pl to deliver a list of the containers which are known to have out of date content, so the Last Modified headers aren't even used.

livelets.as

Even the SWF XMLSocket is crippled in three main ways:

• It can only recieve incoming data from servers in the same domain the SWF was served from
• It can only use port number >1023
• Most importantly - it is not a true listener, it can only listen on streams it has already established with the server. A server cannot spontaneously request a new connection from the SWF socket.

This third item is very important, because it means that a permenantly running server must be present, so we may need to include a simple server daemon using the code from server.pl.

Note1: If we use livelets.pl, then the SWF must be served by that, not by the wiki, because it can only recieve data from the same domain (which includes port number) as that which it was served from.

Note2: Although most PHP installations have socket support as standard now, we're still better using PERL because it still needs to be a separate port >1023 and run as a persistent daemon and we already have the code for this in PERL.

Note3: The third limitation listed above is irrelevent to this application because the SWF has to establish the stream to the server rather than vica versa since there can be many SWF's runningon the local client. They would all have to have separate ports which the server would have to be notified of anyway.

livelets.pl

Although PHP has socket capability it seems that many users would not have the functionality easily available as it often requires PHP to be recompiled with the --enable-sockets switch. Also, the PHP server script would need to be set up to run as a daemon, and many PHP installs do not support command-line PHP by default. So it seems to me that a PERL server would be simplest to implement, especially since we already have working socket code in PERL, it's available by default on virtually all Linux's, and is easy to install on Win32 as well.

Rather than installing the daemon into init.d or as a service, the PHP script could check whether an instance is running and execute it if not so that the installation would then be no different than for most extensions, of putting the files into the extensions directory and ensuring they have the right permissions.

On linux-like OS's it will run as a daemon, but doesn't install itself into init.d for automatic startup, instead livelets.php launches it if it isn'r already running (it can send a simple ping command to the port to test if an instance is running).

The code should work on windows, but is being developed and tested for Linux first. The socket code itself has been tested in windows, and we have code to make it run as a windows service, but no doubt there will be trouble and dedicated windows development time will be required which is unlikely to occur for some time since we have little interest in supporting corporate closed-source solutions. Windows users need to have PERL installed with the IO::Socket module, we recommend the ActivePerl package.

Examples & Usage Ideas

• Ultra changes:
• Forms: Allowing forms to be submitted which can be posted to the server without reloading the page. Any items which may change are made into live templates.
• Caching: Normal templates which are used across many articles like those which include category links would be far more efficient when wrapped inside a live template.
• Collaboration: The edit form could be modified to post the form without reloading the page, and to have a live preview of the content which would make article editing much more chat-like.
• Channels: having live content opens up the whole channel aspect...