Distributed TiddlyWiki

A distributed wiki that might work

Frustrated with the progress of *Diaspora, an idea kept nagging at me: why can't a TiddlyWiki be distributed via Bit Torrent? I've been wanting to start a blog that is serverless and I thought there must be a way using existing technologies.

The foundation of my blog would be some implementation of TiddlyWiki, probably mGSD (previously known as MonkeyGTD or D3 ("D-Cubed"), which I tried before and found to be quite impressive.

TiddlyWiki is essentially an index html page (it uses the Single Page Application architecture) that is loaded with ingenious javaScripts that load flat datafiles, called tiddlers. As I understand it, every time a new page is created, it is stored in a new tiddler, and every time a page is edited, a new tiddler file is created that, rather than replacing it, supersedes the previous version. (The application only needs create access, not modify or delete). Thus, each page has a version history.

It is the index, or base file and its collection of tiddlers that need to be distributed.

Distribution

GitHub?

My first idea was to distribute this collection of files using Github, which nicely deals with files added to the collection. But one must have a Github account and set up crypto keys to use it - not really easy for the marginally-technical person. Worse, after I upload or sync my files to GitHub, in order for anyone to read my wonderful blog/wiki, they too must set up a GitHub account, which is really expecting too much for the average reader. Not only that, but this wikiblog will only be found by search engines as a project directory on GitHub, with no easy way for someone to discover what it's all about.

According to Git is the next Unix:

Git was originally not a version control system; it was designed to be the infrastructure so that someone else could build one on top. And they did; nowadays there are more than 100 git-* commands installed along with git. It's scary and confusing and weird, but what that means is git is a platform. It's a new set of nouns and verbs that we never had before. Having new nouns and verbs means we can invent entirely new things that we previously couldn't do. Git is a new kind of filesystem, and it's faster than any filesystem I've ever seen: git checkout is faster than cp -a. It even has fsck. Git stores revision history, and it stores it in less space than any system I've ever seen or heard of. Often, in less space than the original objects themselves! Git uses rsync-style hash authentication on everything, as well as a new "tree of hashes" arrangement I haven't seen before, to enforce security and reliability in amazing ways that make the idea of "guaranteed identical every time" not something to strive for, but something that's always irrevocably built in. Git names everything using globally unique identifiers that nobody else will ever accidentally use, so that being distributed is suddenly trivial.

— Appenwarr

Github offers gists, issues, and wiki, though it's not clear that these are part of the DRCS (they don't seem to ever be copied to the local file system with the source), maybe they are stored only on the website. In any case, Git can be used as a distributed datastore for a wiki. Some people have already thought of the idea:

Git-wiki

is a wiki that relies on git to keep pages' history and Sinatra (a DSL Ruby for quickly creating web applications in Ruby with minimal effort) to serve them.

I wrote git-wiki as a quick and dirty hack, mostly to play with Sinatra. It turned out that Sinatra is an awesome little web framework and that this hack isn't as useless as I first though since I now use it daily. However, it is definitely not feature rich and will probably never be because I mostly use it as a web frontend for git, ls and vim. If you want history, search, etc. you should look at other people's forks.

— sr

• most active fork appears to be olelo

Cappuccino Github Issues

On Cappuccino demos is a project called GitHub Issues :

Cappuccino front-end for GitHub Issues. Available as a website and as NativeHost desktop app available in app as a download.

• If I understand correctly, the app synchronizes with Github issues, which I believe is distributed version control filesystem. The essential features of editing , commenting, and tagging documents are all there. hmmm...

• Github gists are a simple way to share snippets and pastes with others. All gists are git repositories, so they are automatically versioned, forkable and usable as a git repository.

• The Future of Open Data Looks Like…Github? - lots of people are thinking about a git-ish data repository.

BitTorrent

Far more people know how to use a bittorent client - at least for downloading - than know how to use GitHub. A very small subset of bittorrent users know how to create a torrent file and upload it to a tracker on a torrent index site. Not only that, but trackers and torrent files are quickly being replaced by trackerless sites using magnet links, and many bittorrent clients now can distribute and search for magnet links on a DHT (and for the Vuze client, a special messaging network called PEX.

The advantage of magnet links, is that there is no .torrent file to download; it is simply a URL specifying the hash of the file and its title. Indexing sites that use magnet links don't have to store .torrent files and use less bandwidth. Even better, magnet links can be stored in the Mainline DHT, and all major bittorent clients can search the distributed database directly, without the need for an indexing site like TPB. Their only downside is that the links are long and ugly, and not easy to read or share.

The average web surfer or blogger does not understand all this, and the amount of maintenance to run a site this way is significant. And merely making a torrent file available does not a site make. Unless one knows where to look, readers will never find it, never download the torrent, and never read the site.

Also, since a torrent is a static file that describes static content, it is not possible to add new tiddlers to an existing torrent - this changes the hash of the file collection, and necessitates creating a whole new torrent, and somehow revoking the old version. This problem also applies to magnet links, since they are a hash of the entire distribution, not its individual files.

This means that it would be necessary to create a new .torrent file, or magnet link for every single tiddler, and somehow associate it with the rest of the dynamic distribution (which is not a challenge for GitHub) -- perhaps using a common namespace in the magnet links sent to the DHT. But using a bittorrent client won't do, the reader would continually have to find a way to download the newest tiddlers to keep his mirror of the site up-to-date.

There has to be an easier way. To begin with, what is needed is browser integration with Bittorrent.

Vuze

Vuze (formerly known as Azureus) is essentially a bitttorent client with a built-in browser. I don't know the specifics of its data structure, but it is essentially a multi-page site whose content indexes videos and their metadata, and then efficiently loads videos (and apparently pre-fetches previews on the current content page). Vuze by default uses an incompatible DHT but a plugin is available that integrates it with the Mainline DHT. It also can use magnet links. However, Vuze is proprietary, and although it is possible to upload videos, it is not clear (to me) how one goes about creating html content for the Vuze network, if that is even possible.

But the concept shows that it is possible to create a distributed site, or at least, distributed video content that is indexed in the DHT, with existing technology.

Opera

Opera is another proprietary browser known for its speed - almost as fast as google chrome. It's innovative in a quirky sort of way, for example, it has desktop widgets that are familiar to Mac and Windows 7 users. It does not support all the nifty plugins and extensions people have come to expect on Firefox, but the more important ones, like NotScript are available, and there are many cool extensions, for example Facebook integration and real-time site translation. Opera also includes a mail client. Lack of integrated email support is a common complaint for TiddlyWiki users using it for GTD.

Another nicety: online bookmark syncronising, accessible from anywhere.

Version 9 introduced an bittorent client. At first, most people laughed at the idea, unable to imagine any possible use for a built-in bittorrent client. But downloading a torrent really is transparent and hassle-free for those who are still bittorrent newbies. It is even possible to integrate Opera with Transmission. Opera also can use magnet links, although some websurfing is often required to get it to work. It also claims to be faster and use less memory for pages that are heavy on javaScript - a plus for our javaScript tiddlyWiki.

Lastly, Opera Unite is a built-in personal webserver, that offers the user the capability to host content, share files, composite and share photo albums, live messenger chat and webcam, and even stream your media library. Unite includes a community proxy that forwards your subdomain to your Unite server or use a dynamic DNS plugin. Pretty attractive and powerful, if you don't mind the proprietary technology.

At the end of the day, Opera has almost everything needed to create a personal, self-hosted site, and seamlessly download bittorent files. It seems easy enough to host a tiddlyWiki using Unite, but this is not distributed; it depends on Opera's proxy service.

What is missing is a way to manage new tiddlers, calculate their hashes, create magnet links, and upload them into the Mainline DHT. Opera apparently has no mechanism to create new torrents at all.

• A utility script called Magnet Catcher is available for Opera. (More below) This could be extremely useful in creating a distributed TiddlyWiki, because although it does rely on .torrent files, it makes friendly magnet links that eliminate the need to download .torent files, making it possible to replace (manually) torrent links with magnet links in local content.

• Unite-tracker aims to be a Bittorrent tracker service for Opera Unite.
  • Support for serving/tracking existing .torrent files
  • Support for creating .torrent files from folders/individual files
  • Can track any torrent that decides to use this service's announce URL, or can be restricted to only track .torrents that it serves up.
  • Provides a list of all peers connected to the tracker, and some information about them.
  • Tracker supports a scrape URL for gathering information about the tracked torrents.
  • Update check, with direct link to latest version (supports both stable/unstable release types)
  • Wiki documentation was last updated in 2009.

If Unite-tracker works with the current version of Opera, this is almost all the nuts-and-bolts needed to close the upload-download cycle of distributed web content, as long as an accessible page (primarily, the base index of the tiddlywiki) provides a link that references the index generated by the tracker.

However, Opera, at the end of the day, is proprietary. What is needed are open source solutions that can be maintained and improved...

Firefox Plugins

Magnet Catcher identifies torrent links on a page and automatically creates and adds a magnet link next to the torrent link. Simply click the magnet icon to get the magnet link.

Magnet Catcher also does away with the need to click on a torrent description in a search result page in order to download the torrent. (Apparently it fetches the metadata from the torrent file in the background). The magnet links are displayed directly on the browse pages next to the torrent titles.

Magnet Catcher works on almost any web site.

Resources and Addons To Make BitTorrent Magnet Life Easier

• Mainline DHT Plugin for Vuze

• Magnet Tracker - a handy script that scans a web page looking for torrent hashes. The script then displays a window in the bottom right of a compatible browser window which allows the user to download a Magnet link. Magnet Tracker supports many of the main torrent sites and even offers functionality on Wikileaks in response to the site’s adoption of the technology last year.

• Magnet Catcher - Continuing on the theme, Magnet Catcher strips the concept of adding Magnet links to a page right back. As can be seen from the before and after screenshots below, this script adds Magnet links straight to the main searchpage of a site containing torrent hashes, with no need to click-through to the torrent details page.

• Magnet Link Generator - If you already know the hash value of the material you want to download from BitTorrent, this basic webpage will convert it into a Magnet link.

• Mgnet.me – The Magnet URI shortening service. One of the downsides to Magnet links is that they can be very long and therefore difficult to share.
  • Crucially they often have too many characters to be shared via Twitter, they are not clickable in IM apps like GTalk and MSN, and can be unfriendly on the eye. The Mgnet.me service changes all that.
  • Introduced earlier this year, Mgnet.me is a shortening service, much like Bit.ly or Tinyurl, designed specifically to convert Magnet URIs into shorter and more manageable links. As can be seen from the screenshot below, it also provides HTML code and a feature to post a newly shortened Magnet link directly to Twitter.
  • Depends on centralised service.

All of the above plugins, available for Firefox, suggest that it is currently possible to create a distributed wiki with very little modification and integration of code.

Bittorrent Streaming - the future

Sneak peak: BitTorrent expands live streaming tests - BitTorrent Live is a whole new P2P protocol to distribute live streamed data across the internet without the need for infrastructure, and with a minimum of latency. The inventor's efforts included writing a complete new P2P protocol from scratch. The BitTorrent protocol itself, he said, simply introduced too much latency to be a viable live streaming solution. The tests have so far (as of Oct 2011) been restricted to “simple pre-recorded content loops to test latency and audio/visual sync.”

I tried it, and either it didn't work on Ubuntu, or I did not know how to use it properly. That's because there is no documentation.

This is likely to be commercial closed-source. Still, it's interesting

Mechanism for updating tiddler index via RSS subscription

A Vuze plugin called Distributed Database Trusted Feed can act as a publisher or a subscriber to content.

As a publisher it takes a source of content (either a local file or a webpage) and makes a copy of this available as a torrent. It also makes a descriptor to this available for you to share with other people so they can subscribe to the content and get their own copy of it. The descriptor includes a public key so that when subscribers download the content they are safe in assuming that it came from the publisher. The plugin periodically scans the shared resource and will make a newer copy available to subscribers if it changes.

As a subscriber it takes a publication descriptor (for example a magnet link to the publish descriptor created by the publisher) and periodically downloads the latest content associated with it. It makes the content available via a local http port so the subscriber can easily consume it.

• It remains to be seen whether this technology exists for other bittorrent clients.

Tribler distributed channel subscriptions (3/01/2010)

Tribler released a beta of its Bittorrent client with a new feature that the researchers behind the project have dubbed P2P moderation. The idea in a nutshell is that users can aggregate channels and content and distribute them through DHT. From the official announcement:

"In Tribler V5.2 every user can start their own "Channel" to publish torrents. When people like your torrents you become popular and essentially become the owner of an Internet TV channel. You can moderate this RSS-like stream of torrents. This feature is designed to stop the flow of spam in P2P bittorrent, without the requirement of any server."

Channels can be pre-populated with an existing RSS feed, or personally aggregated by manually adding torrent files. The client lists a number of popular channels and also offers the option to search for channels.

However, the search seems to be restricted to the actual channel name, which makes it impossible to find a channel by searching for the content you're looking for. Users also can't add any description, tags or artwork to their channels. Add to this the fact that I didn't even find an easy way to rename your channel, and you'll see why this is still a pretty experimental feature.

The idea itself of course isn't really new: The original eDonkey client already included the ability to publish collections of files, and Vuze users have been able to publish distributed feeds through the Distributed Database Trusted Feed plug-in and the RSS Feed Generator plug-in since 2008.

Nodewiki

Nodewiki is wiki software created using Node.js. It uses redis for data storage, with the redis node client to talk to redis. Page markup is written using Showdown, a Markdown implementation written in javascript.

• Why You Should Pay Attention to Node.js
• no updates since Jan 2012 :(

fanout.js

fanout.js - a simple and robust fanout pubsub messaging server for node.js

See also

• TiddlyWiki5
• Common vision/Information Technology requirements
• Single Page Application
• TiddlyGuv - A tool for open-source governance. Reporting, tracking, discussing
  • last update: 30 September 2008
• TiddlyWeb - Tiddy's official backend persistence mechanism - this would be the aspect to extend into p2p

Vuze

• Vuze on Wikipedia
• Vuze plugins
• Vuze wiki