Bitgroup

These days the issue of privacy online is becoming more and more important as we hear about the shocking extent to which organisations like the NSA are violating people's privacy and the giant corporations like Facebook and Google are misusing our personal information. Social networks have become an important tool for people from diverse locations and cultures to get together and organise, but when this organisation is used for purposes that oppose them they're shut down like this and this. It's become very apparent to many people that a new form of social networking tool is required that meets a number of important criteria.

Free and open source: For it to gain widespread use and be as accessible as possible to all people like Facebook and Google applications are it must be completely free to use. But it's very important that the source code be available for peer review so that the users can be satisfied that it's secure and there's no nasty tricks involved such as misuse of personal information or the inclusion of back doors through the security systems.

Peer-to-peer: One of the big problems we're starting to see nowadays is that the centralised model for setting up sites on a web-server for the users to access is not very resilient. We're seeing many sites that start off with great ideals when it comes to upholding peoples rights to privacy, but when they become too popular they come under attack by "the global agenda" by being accused of aiding child pornography or "supporting terrorism". They're then forced by law to turn over their user's data or face being prosecuted and shut down, some such as Silent Circle and Lavabit just shut down voluntarily rather than violate their user's privacy or face prosecution. However there are some online applications and services which have come under attack but have not been successfully shut down such as The Pirate Bay and other file sharing applications and the Bitcoin crypto-currency. Although these applications are considered a major problem from the perspective of the global agenda, it's so far proven impossible to do anything about them. The reason is that they have no central point of attack, shutting down accounts or servers or prosecuting people has no effect on these systems. It's for this reason that it's absolutely necessary for us, the people, to have access to a new social networking tool that is completely peer-to-peer.

Supports privacy and anonymity: Another thing that has been growing in importance a lot over the last few years is the ability for users to maintain their anonymity when working online. It's rapidly becoming a violation of law simply to voice opinion that opposes the status quo, which is in itself ample reason, or even a moral obligation, to oppose it! It's important for people who oppose this draconian system to be able to do so anonymously, especially in exceedingly oppressive countries that don't allow basic freedom of speech rights such as China and the USA.

Bitmessage

Anyone whose thought seriously about developing a any kind of peer-to-peer application that has more than basic file sharing functionality will know that there are some extremely difficult challenges to overcome, particularly when it comes to the "sign up/sign in" and the ability for many users to "subscribe" to information. There are many complex issues involved in security and content distribution which are orders of magnitude more difficult to accomplish in a system where there are no fixed servers, and every peer must be considered as a potentially hostile to the network.

The first application which has really made headway in moving the world of peer-to-peer beyond the simple file-sharing application is the Bitcoin crypto-currency. The fact that it's financially oriented creates much more motivation for attackers to try and exploit any weakness in it, but it has stood up to everything the hackers and the governments with their billions of dollars have thrown at it. Now a few years down the track, the Bitcoin code is starting to be used in other more diverse applications such as in the Namecoin alternative domain name system and the Bitmessage secure anonymous messaging system.

Bitgroup

Bitmessage is of particular interest because messaging is a very general application that can be used as the foundation for many other applications - social networking being one of them. And that's what the Bitgroup project is about, building the foundations of a secure peer-to-peer anonymous social network using Bitmessage as it's fundamental communications over which users and groups are formed and connected, hence the name "Bitgroup".

Bitmessage is programmed in Python, and since I've now had some experience working with the Bitessage API through Python when I made the Bitmessage gateway so the main networking aspect of Bitgroup will also be written in Python. The interface will be in JavaScript running within an embedded browser component within the Python application. This is so that it will be very easy to allow it to be run as a web-application server and will also make the interface programming and theming much simpler.

The code is being developed in our tools repo which you can view on our Github mirror here. It'll get moved into it's own repo after it's our of the pre-alpha stage.

How groups work in Bitgroup

The basic functionality required for building a social network application is the channel which is a collection of changing content maintained by one or more users that can be "subscribed" to, or "followed" by many others. This functionality allows for all the different behaviours that make up a social network system such as creating pages, posting items, following and "liking" things, as well as other potential functionality such as blogs, wikis or scheduling applications.

The first step in this "channels" idea is to extend the basic messaging and subscription functionality of Bitmessage to allow for the formation of groups that have not only subscribers that can view the public content generated by the group, but also "members" who produce and publish the group's content together and also have access to the private members-only content.

There are three key features of Bitmessage that allow for the formation of private groups.

• Any number of running instances can have the same From address, so sharing the private keys associated with an address (or in the case of a deterministically generated address, sharing the key used to create it from) with other people allows them to become a member of the group by being able to publish information from that group to its subscribers or other members.
• Users are able to subscribe to their own addresses which means that you will receive broadcast messages from others sending from that same address. This means that a many-to-many messaging system can be set up by a group all using the same address to broadcast from and subscribing to it. They can then use the private key used to create the address as a shared encryption key so that messages broadcast to the group's private address are only readable by members of the group.
• Broadcast messages do not reveal the identity of the sender, so the group can be private and the members anonymous even though its possible for a non-member to subscribe to the group's private address if they were to find it somehow (for example by observing the block chain).

When a user creates a new group, the request is processed by the newGroup method in the main Python App class (defined in app.py) which simply instantiates a new Group instance (the Group class is defined in the group.py module) with the desired name for the new group as the only parameter. When a Group instance is created with a name instead of an existing Bitmessage address as its construction parameter, it goes through the process of creating a new private key and address pair (public and private) for the group. The Group constructor then adds an entry for the new group to the "groups" section of the Bitgroup configuration file using the group's private key as the entry key and the group's private Bitmessage address as the entry value (see the config.sample file for the exact format). And then finally it creates a new data structure for the group based on a template structure in the group.py module and adds the new group's name and public Bitmessage address to the structure. The data structure is automatically saved onto disk by the Node class which is a base-classes of Group. The group's stored data is encrypted using the group's private key.

Nodes

The group functionality created by the Group class is a messaging-only system with no persistent content, so the next level of functionality is done by a class called Node (defined in the node.py module) which this content aspect in the form of a structure of properties and attachments. A group can have any number of nodes which can be viewed and modified in different ways depending on the extensions the group has installed, for example as a folder of files, a list of historical posts, as a collaborative document etc. Later as more extensions are created, this channel functionality can form the basis for many other applications such as group schedules and decision-making tools.

Bitgroup messages

Bitgroup messages are Bitmessage messages that are designed to be interpreted by the Bitgroup application rather than by the recipient user directly. Bitgroup messages are distinguished from normal messages by a simple means of having a subject line that consists of the Bitgroup version that generated the message, the class that it should be instantiated as, and a human readable comment indicating that the message is to be interpreted by Bitgroup. The message body is in the form of a Base64 encoded JSON properties structure. It's up to the specific message class to ensure that the data of the message is valid and from a legitimate source.

Bitgroup messages exist in the Bitmessage network and in user's Bitmessage in-boxes and out-boxes, but at runtime they exist as instances of the Message class, or more specifically one of the sub-classes of Message which are all defined in the message.py module. The Message class has methods for sending or broadcasting and other basic Bitmessage operations and also has methods for creating and updating the message's properties, and the sub-classes the refine this common functionality for the classes specific purpose such as data synchronisation or inviting new members.

The Message class has a static method called getClass which takes a Bitmessage message data structure as its single parameter and returns the Bitgroup class that the message should be instantiated as. It does this by checking if the subject line indicates that the message is for Bitgroup to interpret and extracts the class that's specified. It also checks that the class not only exists, but is indeed a sub-class of the Message class before returning it. If it's not a Bitgroup message, or the class was not a Message sub-class, then the Message class is returned which is the Bitgroup wrapper for a plain Bitmessage message.

Since the getClass method takes a Bitmessage message data structure as its parameter always returns a valid message class, and all Message classes are also instantiated with a Bitmessage message parameter, a Bitmessage message can be converted to its appropriate Bitgroup class with the following slightly odd looking syntax - in fact I think it's odd enough that my Voodoo icon could be in order ;-)

<python>msg = Message.getClass(msg)(msg)</python>

If the returned instance is a sub-class of Message, it's still possible that it could be a forgery with a subject line intended to make the method instantiate a sub-class, for example to try and forge a group invitation acceptance. After the sub-class is instantiated, it will be able to determine the authenticity of the message from the data contained in it, and if there's a problem it can set the invalid property of the message which will cause the App class to dump the newly created instance and re-instantiate it as a basic Message class.

Bitgroup messages are also present in the interface and so the interface also has a Message class which is defined in the message.js script. The message data is stored in the user's node data so that it's automatically synchronised with the Python side and the back-end storage but is not part of any distributed group data.

Note that only low-level functionality used by the Bitgroup system itself uses these message types. High level workflow for group organisation applications will be implemented later and will be done within the context of group node data which is on a higher level of abstraction than dealing with individual Bitmessages.

Invitations

The first type of Bitmessage implemented is the Invitation class which is used to invite other Bitgroup users to become a member of a group. If the recipient doesn't yet run Bitgroup (or even Bitmessage), then they'll need to be contacted directly first with a message requesting that they install Bitgroup so that they can then accept an invitation message. Later this process of first inviting users to install Bitgroup will be incorporated into the interface, but initially only the Bitgroup invitation messages are being implemented.

An Invitation has a property called status which is set to created by default when the instance is first created. Once a recipient has been set and the message sent to their address, the message which resides in the local Bitmessage out-box will have a status set to "sent". When the recipient's Bitgroup application receives the message an Invitation instance will be instantiated for it and its status will be set to received and it will at this point have a potential "accept" action associated with it which the Bitgroup interface will present. If the user chooses to accept the invitation, then the accept method of the Invitation will be executed and the message's status will be set to a final accepted value and the process of creating the group's public and private Bitmessage addresses locally will begin.

Presence & real-time connections

The Bitgroup system connects groups members directly if they're online so that changes to group data can synchronise in near real-time. This also allows for functionality such as group chat to be added to the system at some point. For offline users the data updates are compiled into DataSync messages which are broadcast to the group private address periodically.

For connections to be established between online users, the user's connectivity status needs to be broadcast when they come online. This includes their IP address and approximate bandwidth information. This is done via a Bitgroup message of the Presence class which is broadcast to the group's private Bitmessage address. It's done via Bitmessage rather than direct connection so that no connectivity information is required about any of the other members in order for a user to announce their presence.

The real-time connections are organised in a client-server topology by designating one of the members as the "server" for the group in a deterministic way that doesn't require any communication or organisation. The online status of the members is stored along with the rest of the group's data so that it's available to all members. Each member's Bitgroup instance then takes this list and extracts the members with the fastest bandwidth and then selects the first one in order of Bitmessage address as the server for the group which is essentially random, but will be the same result for all members.

Data synchronisation

Each group consists of a number of users each running one or more instances of the software (only one instance per host though), and each of these instances may have many interface pages open in their browser. The data for the group needs to be synchronised across all of these instances and interface pages.

The general idea is that each context propagates it's changes over real-time bidirectional channel where they can, and where real-time connections are not possible or practical, the changes to data are queued up synchronised on a regular interval. The regular synchronisations are done using Bitmessage and are broadcast to the whole group, the real-time connections are done via a designated server. In the case of communicating changes between members with real-time connections, the server one of the members and is chosen in a deterministic way so that no communications is required to decide who will be the server.

As changes to data occur they're queued up to be sent when the next time a synchronisation request occurs. All the data is in the form of a hierarchical key:value structure with every value being a two-element array contain the value and an accompanying time-stamp denoting when the current value was set. The time-stamp originates at the source of the change and accompanies the value in the queue and then in the sent synchronisation data.

When any instance receives incoming data to be synchronised it checks each items time-stamp and compares it with the time-stamo if the local version of the same value (if one exists) and only stores it if it's more recent. This way all the instances' data will always eventually end up in a consistent state with each other as long as all the messages eventually arrive, see eventual consistency.

Interface synchronisation

Each instance has a unique ID so that the service can keep track of the last time each one connected last and merge it's queued client changes with it's local changes that have occurred during that time. I'm using an HTTP header called X-Bitgroup-ID for assigning an ID to each client because the Python socket implementation doesn't seem to allow one to obtain the unique handle of each connected stream.

This bidirectional channels connecting the Bitgroup service with the interface browser pages in its basic form is a standard Ajax-based mechanism. An Ajax request to the server is made by each page on a regular one-second interval at which time all the queued changes to the group data that have occurred within the context of that page are sent and the server responds with the changes it has accumulated during that time that have come from other group member's hosts or from other local interface pages.

Each interface instance also includes a 1x1 pixel SWF at the bottom of the page which attempts to connect with the JavaScript application running in the same it's embedded in, and also connects to the local Python service with an XMLSocket which remains open and automatically reconnects if it disconnects. If the XMLSocket successfully connects woth both the Python service and the local JavaScript, then it forwards all data sent by the Python service to the JavaScript application. In this way changes can be sent directly by the Python service as they happen rather than periodically sending queued data. The changes going from the page to the Python service are done via Ajax requests regardless of whether the SWF is running, but will be sent immediately if it is running, and queue them for periodic sending if it's not running.

The SWF binary is compiled by the open source MTASC compiler to the version 8 Flash standard and consists of just the socket.as script. The make-swf.sh script is included with the source to automatically download the MTASC compiler if it's not present and then build the interface/socket.swf binary from the socket.as source.

Encryption & Security

Most of the privacy aspects of Bitgroup rely on Bitmessage's security because the underlying messaging infrastructure of the groups is built on Bitmessage messages. But there are also real-time connections between on-line members in groups and group data is stored on disk and can be optionally distributed to third party storage services that the group has access to, so encryption is needed for these two mechanisms. I'm no encryption expert at all, so I've essentially delegated this aspect of the work to Bitmessage by utilising its encryption code for Bitgroup's encryption requirements.

This is done by importing the required modules directly from the Bitmessage source directory and calling the same high-level encrypt method as the Bitmessage code does to encrypt its message's. These modules are OpenSSL, pyelliptic and highlevelcrypto which offers a simple high-level interface to encrypting content using ECDHE-AES256-CBC (Diffie Hellman Elliptic Curve, 256 bit AES in Cipher-Block-Chaining mode).

Another aspect requiring attention is the connection between the local Python background service and the interface running in the browser. This uses the standard HTTP digest access authentication method which is necessary even though the connection is purely local, because it's possible for an attacker to set up a malicious Ajax request on another site that the same browser may visit which can then make a local request to the background service. Also it's useful to have authentication in place so that the application can be served over the web, although this is not a very good idea currently as there's no SSL capability on its HTTP server, but at least the user's login and password are not exposed by using the digest method for authentication, only hashes are sent to the server.

Interface

Browser-based interfaces are extremely convenient because they allow very easy linking to other sites, and are very easy to develop since there's so many useful tools and technologies available such as jQuery and CSS, and allows groups to open up some of their content to the web for easy access. Some groups could even run a complete web gateway if privacy and security were less of a concern to them so that their members and subscribers could log in over the web like a normal social networking site. Since the browser will be connecting only to our local socket, and our local socket only accepts requests from our own application the browser is safer than the usual situation. However later we'd like to offer the option of an in-application version of the interface using PyWebkit or PyGtkMozembed.

The interface uses the Single Page Application pattern for the structure of the interface code. This keeps the roles of the Python and the JavaScript very clear with no need for a messy mish-mash of interface functionality spanning both the sides. The entire interface application will be written in JavaScript and only reloads the page/application when the user changes the context to a different group (since each group decides for themselves which extensions will be active and therefore which JavaScript will be loaded in the page).

The local part of the URL consists only of the group name or Bitmessage address and everything after that is the hash-fragment, so that only a change in group will result in a page-reload. The URL format after the hash is a slash-separated path, the first element being the currently selected view and subsequent elements being decoded by that view's URL-routing method and sent to the relevant method for processing.

To allow for more rapid interface development and make it easier for other developers to be involved and extend the system we're including jQuery and using jQueryUI for all the interface elements such as tooltips, progress bars, tab-sets etc.

The Python side of the application is purely involved with Group communications across the network and with keeping the group's property structures up to date. Any new messages or changes to properties on either side are kept in sync with the other side via a per-second polling request from the client side. Both sides queue any changes to data and on the next request, the server merges the queues, updates its data and then responds with a final list of items changed for the client to update its data with.

Application start-up sequence

First the Python-based Bitgroup service needs to be run before the interface can be loaded since the interface code and resources are served to the browser from this service. The starting point is main.py which reads in the .config file, ensures that the application isn't already running, and then instantiates an instance of the main App class which is defined in the app.py module. When th App class initiates it establishes itself as a "superglobal" called app which is available from all scopes and modules of the application. This may not be a very Pythonic way of doing things, but I'm too noob and found the easiest way to get things rolling so that's how it is for now. I used the following line in app's __init__ method to create the app super-global:

{{{1}}}

After the interface is running in the browser it can be used even if the Python service is stopped as long as the group is not changed. Any data changes made while the background service isn't running will be queued up and sent after it can be connected to again. A change in group requires a page reload (and therefore needs the service to be present) since only one group's data and extensions are loaded at once in case one group you're a member of chooses to run extensions that another group you're a member of doesn't trust.

When the browser first requests a page from Bitmessage's HTTP server (defined in http.py), it returns a page containing just an empty body element and a head element containing the main application script, the default views (currently "overview" and "newgroup") and some common resources such jQuery and jQueryUI. The page also contains a small number of variables specifying the current group, user information and the group's extension names. Everything then starts with the execution of the main application script, main.js.

The main.js script defines a class called App which encapsulates all the main functionality of the entire application, and then instantiates a singleton instance of this class called window.app. Execution is then handed over to the constructor of the App class.

The constructor first copies all the data that was sent in the request HTML head element into the application instance, and then does an Ajax request to retrieve i18n.json which contains all the internationalisation messages used by the application (this file is also loaded into the Python side so that both sides can share the same system messages). After the i18n messages have finished loading, the group's extensions are loaded and a handler is registered to respond to changes in the "hash fragment" portion of the URL (see Single Page Application for more details about what this means). Execution is then passed to the application's main start-up method, run().

The run() method first calls the locationChange() method to simulate the changing of the hash-fragment in the URL so that the current view can be rendered into the content area of the page. If there is no group selected, then the pageRender method is called since it will not be automatically rendered in response to data events in this case. The final step in the start-up process is to set up an event that runs every second which keeps the data synchronised between the backend and client side.

From then on, everything that happens in the ongoing running of the application is in response to either user interface events or data-change events coming from the Python service.

Page rendering sequence

In a pure Single Page Application page rendering is a rare occurrence because there are no page reloads. In Bitgroup a page render occurs when the user changes the group as described above since it causes a page reload. A full page render is also done if data synchronisation hasn't occurred for some time or the number of changes received is over a certain amount.

The first response from the server side for a dataSync request returns the entire data structure (this happens if the last sync was more than a certain time ago or the number of changes is more than a certain amount). The dataSync method detects if the response is a list of changes or the whole structure, and if it's the whole structure it calls renderPage which builds the entire page from scratch.

The renderPage method first loads the CSS for the group's selected skin, or the default if no skin is selected. It then renders the personal links in the top bar and the views menu, and then renders empty div elements for the main page items such as notification. page title and content areas which will be filled later. The skin's JavaScript (if it has any) is then executed which may completely alter the layout of the page as long as it keeps the elements ID attributes intact.

After the skin's script as finished executing the renderPage method will then finish off the page by populating the title area, connecting the live menu elements to their data sources and executing the selected page view method, or overview if none is selected. The view method is responsible for populating the main content area of the page and is called not only in a full page render, but also whenever a new view is selected from the view menu or when the currently selected node changes.

Forms & data

In a Single Page Application, the input elements that allow the user to update data and content in the application do not use the classic form/post methodology because everything is handled by the currently loaded page. In fact the entire form element is quite redundant and so is the name attribute of input elements. Instead the form components simply connect into the application functionality directly via event handlers.

In the Bitgroup application we have some methods of the main App class that make the handling of inputs much simpler. First we have componentType which returns a general type string when passed an input element. This type string is used to determine how the value of the input is set or retrieved and how changes in the value are to be connected in to the application. The componentRender method is a high-level helper function for constructing an input element given some data such as a list, and componentSetValue / componentGetValue are used for setting or retrieving the value of a form or other interface component.

Components can have their own getValue and/or setValue functions by adding them directly to the element and then these will be used by componentSetValue and componentGetValue if they're present. This way it's easy to make complicated custom components that are connected to live data changes.

App.componentConnect: The main function is componentConnect which creates a "live" connection between the specified path into the group's data and the interface component. The connection is bidirectional (if it's an input component) and asynchronous so that if the user changes the value it will automatically propagate back to the server-side to be stored and will automatically be reflected in any other open instances of the interface. Conversely if the data undergoes change from another remote instance via another user in the group or by some other back-end service, all interface components connected to the changed data will automatically have their values updated.

Here's a simple example of how it can be used:

<js> $('#foo').html( app.componentRender( 'checklist', ['foo', 'bar', 'baz'] ) ); app.componentConnect( 'settings.extensions', $('#foo .checklist') ); </js>

In this example, a checklist (like a select list but with checkboxes instead) is rendered with items "foo", "bar" and "baz". The second line then connects this new input element to the data source settings.extensions in the group's shared data structure. As soon as this is done, any of the checklist items that are in the settings.extensions value will become checked, and if any of the items checked state are changed by the user, the group's data and any other pages containing the input will update accordingly.

Here's a slightly more complicated example which creates a live table connected to a data source that's an array. Whenever the data in the array changes, the element's custom setter method is called which renders the data as a table into the element.

{{{1}}}

One complication (bug#2) is that the event handler prevents the DOM element that it's connected to from dying when it's no longer present in the DOM, so the event handler first checks this by using a jQuery statement to check if the component is a descendant of the body element, and removes itself from the event if not.

{{{1}}}

The handler function is assigned to a variable in the local scope so that it can be referred to when it's called to unhook itself from the event using the jQuery .off() method.

Skins

The main content is generated unconditionally by the main application and is a plain set of div and ul elements with CSS class and ID attributes allowing CSS to define the style and page layout. Skins are a directory of the skins name that go in the skins directory and contain at least a style.css stylesheet that will be loaded if the group has selected that skin. There can then be any number of support files such as images etc in the skin's directory that the stylesheet can refer to.

Extensions

Extensions play an important role in the Bitgroup system. The idea is that each group's members can choose what extensions are active for them based on their needs and their security concerns. The extensions are only implemented in the JavaScript side currently as that's where the majority of the application's functionality resides - the Python side is only the foundation network functionality which is unlikely to require extending and raises security issues at least until the application has reached a mature state of development. Extensions are in the form of directories that reside in the extensions directory. The directory should be the name of the extension, and it should contain a JavaScript file of the same name within it that has the .js file extension. This file will be loaded prior to application startup if the current group has selected to use it. If the user changes to a different group, the page reloads so that a new selection of extension can load depending on the group's preferences.

Adding new views

To add a new view, an extension defines a class, adds a render method which populates the content div, and adds the class to the available views in the app.views list. It may also add the view name to the various node types that should have the view.

{{{1}}}

Application events

We need to add custom events throughout the code that extensions can subscribe to so they can adjust or extend the behaviour of the application. Extensions will initially only be available on the JavaScript side and will use the jQuery trigger method. Events are triggered with parameters passed to the handler that are specific to the event are packaged inside an args object so that they can be adjusted since JavaScript doesn't support passing-by-reference on non-object variables.

This is an example of an event the could be triggered by an extension:

{{{1}}}

Here the argument foo is being output after the new bgNewEvent event is triggered to see if any handlers have changed it's value. We precede the event name with the "bg" prefix to ensure there's no conflict with existing events in the environment.

A handler could subscribe to this event and modify the foo value as follows:

{{{1}}}

Current events

bgHashChange

bgPoller

bgDataChange-KEY

See also

• Bitgroup source code: WebSVN | GitHub mirror
• Bitgroup/Bugs - a record of the bugs and fixes
• Bitgroup/Tasks - a record of the tasks to do and done
• Bitmessage
• Bitcoin
• Python
• Single Page Application
• Closure