|
|
| Line 1: |
Line 1: |
| − | These days the issue of privacy online is becoming more and more important as we hear about the shocking extent to which organisations like the NSA are violating people's privacy and the giant corporations like Facebook and Google are misusing our personal information. Social networks have become an important tool for people from diverse locations and cultures to get together and organise, but when this organisation is used for purposes that oppose them they're shut down like [http://worldtruth.tv/government-orders-youtube-to-censor-protest-videos/ this], [http://www.globalresearch.ca/search-engine-manipulation-google-and-youtube-suppress-911-truth/5352982 this] and [http://arstechnica.com/business/2013/09/facebook-suddenly-deletes-social-fixers-facebook-pages/ this]. It's become very apparent to many people that a new form of social networking tool is required that meets a number of important criteria.
| + | The Bitgroup documentation has now moved to its own wiki at [http://www.bitgroup.org bitgroup.org]. |
| − | | |
| − | '''Free and open source:''' For it to gain widespread use and be as accessible as possible to all people like Facebook and Google applications are it must be completely free to use. But it's very important that the source code be available for peer review so that the users can be satisfied that it's secure and there's no nasty tricks involved such as misuse of personal information or the inclusion of [http://www.newscientist.com/article/dn24165-how-nsa-weakens-encryption-to-access-internet-traffic.html back doors] through the security systems.
| |
| − | | |
| − | '''Peer-to-peer:''' One of the big problems we're starting to see nowadays is that the centralised model for setting up sites on a web-server for the users to access is not very resilient. We're seeing many sites that start off with great ideals when it comes to upholding peoples rights to privacy, but when they become too popular they come under attack by "the global agenda" by being accused of aiding child pornography or "supporting terrorism". They're then forced by law to turn over their user's data or face being prosecuted and shut down, some such as [http://www.washingtonpost.com/blogs/the-switch/wp/2013/08/09/another-e-mail-service-shuts-down-over-government-spying-concerns/ Silent Circle] and [http://www.washingtonpost.com/blogs/the-switch/wp/2013/08/08/snowdens-e-mail-provider-is-closing-cannot-legally-say-why/ Lavabit] just shut down voluntarily rather than violate their user's privacy or face prosecution. However there are some online applications and services which have come under attack but have not been successfully shut down such as The Pirate Bay and other file sharing applications and the [[Bitcoin]] crypto-currency. Although these applications are considered a major problem from the perspective of the global agenda, it's so far proven impossible to do anything about them. The reason is that they have no central point of attack, shutting down accounts or servers or prosecuting people has no effect on these systems. It's for this reason that it's absolutely necessary for us, the people, to have access to a new social networking tool that is completely [[peer-to-peer]].
| |
| − | | |
| − | '''Supports privacy and anonymity:''' Another thing that has been growing in importance a lot over the last few years is the ability for users to maintain their anonymity when working online. It's rapidly becoming a violation of law simply to voice opinion that opposes the status quo, which is in itself ample reason, or even a moral obligation, to oppose it! It's important for people who oppose this draconian system to be able to do so anonymously, especially in exceedingly oppressive countries that don't allow basic freedom of speech rights such as China and the USA.
| |
| − | | |
| − | == Bitmessage ==
| |
| − | Anyone whose thought seriously about developing a any kind of peer-to-peer application that has more than basic file sharing functionality will know that there are some extremely difficult challenges to overcome, particularly when it comes to the "sign up/sign in" and the ability for many users to "subscribe" to information. There are many complex issues involved in security and content distribution which are orders of magnitude more difficult to accomplish in a system where there are no fixed servers, and every peer must be considered as a potentially hostile to the network.
| |
| − | | |
| − | The first application which has really made headway in moving the world of peer-to-peer beyond the simple file-sharing application is the [[Bitcoin]] crypto-currency. The fact that it's financially oriented creates much more motivation for attackers to try and exploit any weakness in it, but it has stood up to everything the hackers and the governments with their billions of dollars have thrown at it. Now a few years down the track, the Bitcoin code is starting to be used in other more diverse applications such as in the [[Namecoin]] alternative domain name system and the [[Bitmessage]] secure anonymous messaging system.
| |
| − | | |
| − | == Bitgroup ==
| |
| − | Bitmessage is of particular interest because messaging is a very general application that can be used as the foundation for many other applications - social networking being one of them. And that's what the Bitgroup project is about, building the foundations of a secure peer-to-peer anonymous social network using Bitmessage as it's fundamental communications over which users and groups are formed and connected, hence the name "Bitgroup".
| |
| − | | |
| − | Bitmessage is programmed in Python, and since I've now had some experience working with the Bitessage API through Python when I made the [[Bitmessage#Bitmessage gateway|Bitmessage gateway]] so the main networking aspect of Bitgroup will also be written in Python. The interface will be in JavaScript running within an embedded browser component within the Python application. This is so that it will be very easy to allow it to be run as a web-application server and will also make the interface programming and theming much simpler.
| |
| − | | |
| − | The code is being developed in our tools repo which you can view on our Github mirror [https://github.com/OrganicDesign/tools/tree/git-svn/Bitgroup here]. It'll get moved into it's own repo after it's our of the pre-alpha stage.
| |
| − | | |
| − | == How groups work in Bitgroup ==
| |
| − | The basic functionality required for building a social network application is the [[channel]] which is a collection of changing content maintained by one or more users that can be "subscribed" to, or "followed" by many others. This functionality allows for all the different behaviours that make up a social network system such as creating pages, posting items, following and "liking" things, as well as other potential functionality such as blogs, wikis or scheduling applications.
| |
| − | | |
| − | The first step in this "channels" idea is to extend the basic messaging and subscription functionality of Bitmessage to allow for the formation of groups that have not only subscribers that can view the public content generated by the group, but also "members" who produce and publish the group's content together and also have access to the private members-only content.
| |
| − | | |
| − | There are three key features of Bitmessage that allow for the formation of private groups.
| |
| − | *Any number of running instances can have the same ''From'' address, so sharing the private keys associated with an address (or in the case of a deterministically generated address, sharing the key used to create it from) with other people allows them to become a member of the group by being able to publish information from that group to its subscribers or other members.
| |
| − | *Users are able to subscribe to their own addresses which means that you will receive broadcast messages from others sending from that same address. This means that a many-to-many messaging system can be set up by a group all using the same address to broadcast from and subscribing to it. They can then use the private key used to create the address as a shared encryption key so that messages broadcast to the group's private address are only readable by members of the group.
| |
| − | *Broadcast messages do not reveal the identity of the sender, so the group can be private and the members anonymous even though its possible for a non-member to subscribe to the group's private address if they were to find it somehow (for example by observing the block chain).
| |
| − | | |
| − | When a user creates a new group, the request is processed by the ''newGroup'' method in the main Python ''App'' class (defined in [http://svn.organicdesign.co.nz/filedetails.php?repname=tools&path=%2FBitgroup%2Fapp.py app.py]) which simply instantiates a new ''Group'' instance (the ''Group'' class is defined in the [http://svn.organicdesign.co.nz/filedetails.php?repname=tools&path=%2FBitgroup%2Fgroup.py group.py] module) with the desired name for the new group as the only parameter. When a ''Group'' instance is created with a name instead of an existing Bitmessage address as its construction parameter, it goes through the process of creating a new private key and address pair (public and private) for the group. The ''Group'' constructor then adds an entry for the new group to the "groups" section of the Bitgroup configuration file using the group's private key as the entry key and the group's private Bitmessage address as the entry value (see the [http://svn.organicdesign.co.nz/filedetails.php?repname=tools&path=%2FBitgroup%2Fconfig.sample config.sample] file for the exact format). And then finally it creates a new data structure for the group based on a template structure in the ''group.py'' module and adds the new group's name and public Bitmessage address to the structure. The data structure is automatically saved onto disk by the ''Node'' class which is a base-classes of ''Group''. The group's stored data is encrypted using the group's private key.
| |
| − | | |
| − | === Nodes - structured group data and semantic relationships ===
| |
| − | The group functionality created by the ''Group'' class is a messaging-only system with no persistent content, so the next level of functionality is done by a class called ''Node'' (defined in the [http://svn.organicdesign.co.nz/filedetails.php?repname=tools&path=%2FBitgroup%2Fnode.py node.py] module) which defines this content aspect in the form of a structure of nodes and file attachments. Nodes are a hierarchical structure of properties (key:value pairs) which are are stored in JSON form, and at run-time are in the form of a JavaScript object (called ''app.data'') on the client side, and a Python dictionary on the server-side (called ''[GROUP].data'').
| |
| − | | |
| − | A group can have any number of nodes which can be viewed and modified in different ways depending on the extensions the group has installed, for example as a folder of files, a list of historical posts, as a collaborative document etc. Later as more extensions are created, this channel functionality can form the basis for many other applications such as group schedules and decision-making tools.
| |
| − | | |
| − | The job of the ''Node'' class is to use both the Bitmessage network and real-time connections between members to keep the group's data structure synchronised across all members of the group. It also keeps the data synchronised across all interface pages open in the local member's browser and keeps the member's local storage up to date. Extensions also allow the group to define additional external backup storage repositories and sites on which to publish selected aspects of the group's data.
| |
| − | | |
| − | The hierarchical aspect of the data is generally only one level deep allowing a group to contain settings, members and nodes. Further structure is created via semantic relationships between the nodes. These relationships are still just in the form of key:value pairs like all the other data, but the difference is that both the ''key'' and ''value'' parts are references to other nodes. This creates a relationship between one node and another whereby the type of relationship and its meaning or effect on the node having the connection is determined by the node referred to by the ''key'' part of the property.
| |
| − | | |
| − | The function of a relationships may be as simple as a tag whereby it does nothing at all except that it has particular meaning to the users, and allows them to organise with it for example by searching for all nodes that exhibit that tag. But on the other hand the type of relationship could be very active having an effect of the nodes content or appearance, or restricting the nodes that can exhibit the relationship to only those that satisfy particular criteria. Extensions can even allow some kinds of relationships to automatically be applied to nodes that satisfy particular criteria.
| |
| − | | |
| − | One very useful relationship that will be added very early on will be a kind that requires a given number of members agreement before it gets applied to a node. This will allow for other useful sub-relations to be developed such as an "Administrator" relation that gives a member higher privileges in the group (for example being able to invite other members or terminate memberships) but for any user to gain this privilege would require the agreement of a number (or all) of the other members first.
| |
| − | | |
| − | === Bitgroup messages ===
| |
| − | Bitgroup messages are Bitmessage messages that are designed to be interpreted by the Bitgroup application rather than by the recipient user directly. Bitgroup messages are distinguished from normal messages by a simple means of having a subject line that consists of the Bitgroup version that generated the message, the class that it should be instantiated as, and a human readable comment indicating that the message is to be interpreted by Bitgroup. The message body is in the form of a Base64 encoded JSON properties structure. It's up to the specific message class to ensure that the data of the message is valid and from a legitimate source.
| |
| − | | |
| − | Bitgroup messages exist in the Bitmessage network and in user's Bitmessage in-boxes and out-boxes, but at runtime they exist as instances of the ''Message'' class, or more specifically one of the sub-classes of ''Message'' which are all defined in the [http://svn.organicdesign.co.nz/filedetails.php?repname=tools&path=%2FBitgroup%2Fmessage.py message.py] module. The ''Message'' class has methods for sending or broadcasting and other basic Bitmessage operations and also has methods for creating and updating the message's properties, and the sub-classes the refine this common functionality for the classes specific purpose such as data synchronisation or inviting new members.
| |
| − | | |
| − | The ''Message'' class has a static method called ''getClass'' which takes a Bitmessage message data structure as its single parameter and returns the Bitgroup class that the message should be instantiated as. It does this by checking if the subject line indicates that the message is for Bitgroup to interpret and extracts the class that's specified. It also checks that the class not only exists, but is indeed a sub-class of the ''Message'' class before returning it. If it's not a Bitgroup message, or the class was not a ''Message'' sub-class, then the ''Message'' class is returned which is the Bitgroup wrapper for a plain Bitmessage message.
| |
| − | | |
| − | The ''Message'' constructor (''__new__()'') calls ''getClass'' to determine the sub-class (if any) that the ''Message'' should be instantiated as. It instantiates the returned class and before returning form the constructor it first checks the new instances ''invalid'' property. If the returned instance is a sub-class of ''Message'', it's still possible that it could be a forgery with a subject line intended to make the method instantiate a sub-class, for example to try and forge a group invitation acceptance. After the sub-class is instantiated, it will be able to determine the authenticity of the message from the data contained in it, and if there's a problem it can set the ''invalid'' property so that the ''Message'' constructor can return the original ''Message'' instance.
| |
| − | | |
| − | All of the ''Message'' sub-classes are based on an intermediate "abstract" class called ''BitgroupMessage'' which takes care of all the functionality in common with all the Bitgroup Message sub-classes which are as follows:
| |
| − | *Determines if the message is incoming or outgoing based on whether the first parameter is a ''Group'' class or a Bitmessage message data structure.
| |
| − | *Adds the ''Group'' instance that the message was or will be broadcast to.
| |
| − | *Base64-encodes and encrypts the data for outgoing messages using the group's password.
| |
| − | *Base64-ddecodes and decrypts the data for incoming messages
| |
| − | | |
| − | Bitgroup messages are also present in the interface and so the interface also has a ''Message'' class which is defined in the [http://svn.organicdesign.co.nz/filedetails.php?repname=tools&path=%2FBitgroup%2Finterface%2Fmessage.js message.js] script. The message data is stored in the user's node data so that it's automatically synchronised with the Python side and the back-end storage but is not part of any distributed group data.
| |
| − | | |
| − | Note that only low-level functionality used by the Bitgroup system itself uses these message types. High level workflow for group organisation applications will be implemented later and will be done within the context of group node data which is on a higher level of abstraction than dealing with individual Bitmessages.
| |
| − | | |
| − | === Invitations ===
| |
| − | The first type of Bitmessage implemented is the ''Invitation'' class which is used to invite other Bitgroup users to become a member of a group. If the recipient doesn't yet run Bitgroup (or even Bitmessage), then they'll need to be contacted directly first with a message requesting that they install Bitgroup so that they can then accept an invitation message. Later this process of first inviting users to install Bitgroup will be incorporated into the interface, but initially only the Bitgroup invitation messages are being implemented.
| |
| − | | |
| − | An ''Invitation'' has a property called ''status'' which is set to ''created'' by default when the instance is first created. Once a recipient has been set and the message sent to their address, the message which resides in the local Bitmessage out-box will have a status set to "sent". When the recipient's Bitgroup application receives the message an ''Invitation'' instance will be instantiated for it and its ''status'' will be set to ''received'' and it will at this point have a potential "accept" action associated with it which the Bitgroup interface will present. If the user chooses to accept the invitation, then the ''accept'' method of the ''Invitation'' will be executed and the message's status will be set to a final ''accepted'' value and the process of creating the group's public and private Bitmessage addresses locally will begin.
| |
| − | | |
| − | === Presence & real-time connections ===
| |
| − | The Bitgroup system connects groups members directly if they're online so that changes to group data can synchronise in near real-time. This also allows for functionality such as group chat to be added to the system at some point. For offline users the data updates are compiled into ''Changes'' messages (messages of the ''Changes'' class) which are broadcast to the group private address periodically.
| |
| − | | |
| − | For connections to be established between peers (group members who are currently online), the members's connectivity status needs to be broadcast when they come online. This includes their IP address, Bitmessage address and approximate bandwidth information. This is done via a Bitgroup message of the ''Presence'' class which is broadcast to the group's private Bitmessage address. It's done via Bitmessage rather than direct connection so that no connectivity information is required about any of the other members in order for a user to announce their presence.
| |
| − | | |
| − | The real-time connections are organised in a client-server topology by designating one of the members as the "server" for the group in a deterministic way that doesn't require any communication or organisation. When member comes online all other peers will receive their ''Presence'' message and can store their online information in a list, but when a peer disconnects, only the server knows, and so all other peers must then be notified so they can remove the peer from their list.
| |
| − | | |
| − | Each peer maintains a list of currently online members for each group in an array called ''group::peers'' which has the peer's ID as its ''key'' and the peers current status information as its ''value''. This differs from the ''server::clients'' array which is a list of all current connections including HTTP and SWF connections not just connections to peers. Since the peers for each group form into a client-server topology, only the server-peer will be connected with all the peers in the ''group::peers'' array (except itself), all the others will only be connected with the server-peer, so the array contains all the group's members that are online but are not necessarily active connections.
| |
| − | | |
| − | To determine which peer is currently the server for the group, each peer's Bitgroup instance then takes their list and extracts the members with the fastest bandwidth and then selects the first one in order of Bitmessage address as the server for the group which is essentially random, but will be the same result for all members.
| |
| − | | |
| − | When a ''Presence'' message is received by the group server, it responds by establishing a real-time connection to the IP address and port that the peer specified in its ''Presence'' message, and then sends a ''Welcome'' message over the new connection which contains any group data that has changed since the peer was last online, and the list of connectivity information for all the other peers. ''Presence'' messages are broadcast to the whole group, so all peers can update their online member data when they receive the message, which means that there's no need for the server to notify the peers about the new peer after it receives the message.
| |
| − | | |
| − | If the peer who sends a ''Presence'' message is behind a firewall so that they cannot receive incoming connections, they can set the ''firewall'' configuration option to "1" to ensure that their firewalled status is sent with their request. When the server responds to a ''Presence'' message from a firewalled peer, it returns the ''Welcome'' message over the Bitmessage network instead of establishing a connection with the peer and sending it through that. This allows the peer to then connect to the server instead of the other way around.
| |
| − | | |
| − | When a peer goes offline, it's connection with the server will be terminated. The server then removes the peer from its client data and sends a notification to all other peers to do the same. The message it sends to the peers is of type ''Status'' which is also sent to the server by peers when other aspects of their online status such as their availability changes.
| |
| − | | |
| − | When the members online data changes this may lead to a change in server-peer (both ''group::addPeer'' and ''group::delPeer'' call ''group::determineServer''). If the server-peer changes, then the peer that was the server will close the connections to all peers except the new server-peer, and all other peers will establish a connection with the new server-peer.
| |
| − | | |
| − | Whenever the data of the group changes, the server sends a message of type ''Changes'' to all the peers. If the change originated from one of the peers then that peer is excluded from the list of peers notified about the changes. Actually any peer that creates changes will push the changes to all the peers connected to it, but in the case of all the peers except the server-peer, they will only have one other peer connection to push the changes to; i.e. the server-peer. This process is all initiated in ''node::setData'' which calls the ''server::pushChanges''.
| |
| − | | |
| − | === Data synchronisation ===
| |
| − | Each group consists of a number of users each running one or more instances of the software (only one instance per host though), and each of these instances may have many interface pages open in their browser. The data for the group needs to be synchronised across all of these instances and interface pages.
| |
| − | | |
| − | The general idea is that each context propagates it's changes over real-time bidirectional channel where they can, and where real-time connections are not possible or practical, the changes to data are queued up synchronised on a regular interval. The regular synchronisations are done using Bitmessage and are broadcast to the whole group, the real-time connections are done via a designated server. In the case of communicating changes between members with real-time connections, the server one of the members and is chosen in a deterministic way so that no communications is required to decide who will be the server.
| |
| − | | |
| − | As changes to data occur they're queued up to be sent when the next time a synchronisation request occurs. All the data is in the form of a hierarchical key:value structure with every value being a two-element array contain the value and an accompanying time-stamp denoting when the current value was set. The time-stamp originates at the source of the change and accompanies the value in the queue and then in the sent synchronisation data.
| |
| − | | |
| − | When any instance receives incoming data to be synchronised it checks each items time-stamp and compares it with the time-stamo if the local version of the same value (if one exists) and only stores it if it's more recent. This way all the instances' data will always eventually end up in a consistent state with each other as long as all the messages eventually arrive, see [[w:Eventual consistency|eventual consistency]].
| |
| − | | |
| − | ==== Interface synchronisation ====
| |
| − | Each instance has a unique ID so that the service can keep track of the last time each one connected last and merge it's queued client changes with it's local changes that have occurred during that time. An HTTP header called '''X-Bitgroup-ID''' is included in data sync requests for assigning an ID to each client since each Ajax request opens a different socket even if it's originating from the same page. The Python service's HTTP server is defined in the [http://svn.organicdesign.co.nz/filedetails.php?pass=115b171623f9895a36633&repname=tools&path=%2FBitgroup%2Fhttp.py http.py] module.
| |
| − | | |
| − | This bidirectional channels connecting the Bitgroup service with the interface browser pages in its basic form is a standard Ajax-based mechanism. An Ajax request to the server is made by each page on a regular one-second interval at which time all the queued changes to the group data that have occurred within the context of that page are sent and the server responds with the changes it has accumulated during that time that have come from other group member's hosts or from other local interface pages.
| |
| − | | |
| − | Each interface instance also includes a 1x1 pixel SWF at the bottom of the page which attempts to connect with the JavaScript application running in that same page, and also connects to the local Python service with an [http://help.adobe.com/en_US/FlashPlatform/reference/actionscript/2/help.html?content=00001761.html XMLSocket] which remains open and automatically reconnects if it's disconnected. If the ''XMLSocket'' successfully connects with both the Python service and the local JavaScript, then it forwards all data sent by the Python service to the JavaScript application. In this way changes can be sent directly by the Python service as they happen rather than periodically sending queued data. The changes going from the page to the Python service are done via Ajax requests regardless of whether the SWF is running or not, but if it is, then they'll be sent immediately, otherwise they'll be queued for periodic sending.
| |
| − | | |
| − | The SWF binary is compiled by the open source [http://www.mtasc.org MTASC] compiler to the version 8 Flash standard and consists of just the [http://svn.organicdesign.co.nz/filedetails.php?repname=tools&path=%2FBitgroup%2Fsocket.as socket.as] script. The [http://svn.organicdesign.co.nz/filedetails.php?repname=tools&path=%2FBitgroup%2Fmake-swf.sh make-swf.sh] script is included with the source to automatically download the ''MTASC'' compiler if it's not present and then build the ''interface/socket.swf'' binary from the ''socket.as'' source.
| |
| − | | |
| − | == Other back-end aspects ==
| |
| − | The description of the groups and message structure covers most of the necessary information about the Bitgroup application, but there are a few other aspects worthy of mention which are covered here.
| |
| − | | |
| − | === Encryption & Security ===
| |
| − | Most of the privacy aspects of Bitgroup rely on Bitmessage's security because the underlying messaging infrastructure of the groups is built on Bitmessage messages. But there are also real-time connections between on-line members in groups and group data is stored on disk and can be optionally distributed to third party storage services that the group has access to, so encryption is needed for these two mechanisms. I'm no encryption expert at all, so I've essentially delegated this aspect of the work to Bitmessage by utilising its encryption code for Bitgroup's encryption requirements.
| |
| − | | |
| − | This is done by importing the required modules directly from the Bitmessage source directory and calling the same high-level ''encrypt'' method as the Bitmessage code does to encrypt its message's. These modules are ''OpenSSL'', ''pyelliptic'' and ''highlevelcrypto'' which offers a simple high-level interface to encrypting content using ECDHE-AES256-CBC ([[w:Integrated Encryption Scheme|Diffie Hellman Elliptic Curve]], 256 bit [[w:Advanced Encryption Standard|AES]] in [[w:Block_cipher_mode_of_operation#Cipher-block_chaining_.28CBC.29|Cipher-Block-Chaining]] mode).
| |
| − | | |
| − | Another aspect requiring attention is the connection between the local Python background service and the interface running in the browser. This uses the standard HTTP [[w:Digest access authentication|digest access authentication]] method which is necessary even though the connection is purely local, because it's possible for an attacker to set up a malicious Ajax request on another site that the same browser may visit which can then make a local request to the background service. Also it's useful to have authentication in place so that the application can be served over the web, although this is not a very good idea currently as there's no SSL capability on its HTTP server, but at least the user's login and password are not exposed by using the digest method for authentication, only hashes are sent to the server.
| |
| − | | |
| − | === External IP address ===
| |
| − | The external IP address of the running instance needs to be known so that it can include it in its broadcast ''Presence'' information. Later when the network is more established this information will be able to be gained from the network without requiring a connection to a "legacy" server. Initially however connections to other peers cannot be relied upon, so the ''getExternalIP'' method obtains its information from [http://checkip.dyndns.org checkip.dyndns.org].
| |
| − | | |
| − | === Constants ===
| |
| − | I'm used to programming with constants and find them pretty useful, but there's no constants in Python or JavaScript (at least not in a browser-independent way), so I've made the [http://svn.organicdesign.co.nz/filedetails.php?repname=tools&path=%2FBitgroup%2Fconstants.py constants.py] module which declares a function called ''const'' that allows for a fairly standard way of defining constants and makes them "super-global" (available in every module and every scope). They're not really constants because they can be reassigned, but by sticking to all upper-case names makes it quite clear what their role is so they can be constants by convention. The ''const'' function also stores them all in an dictionary so that the ''http'' module can send them to the client side so that all the same constants are also available there.
| |
| − | | |
| − | == Interface ==
| |
| − | Browser-based interfaces are extremely convenient because they allow very easy linking to other sites, and are very easy to develop since there's so many useful tools and technologies available such as [[jQuery]] and CSS, and allows groups to open up some of their content to the web for easy access. Some groups could even run a complete web gateway if privacy and security were less of a concern to them so that their members and subscribers could log in over the web like a normal social networking site. Since the browser will be connecting only to our local socket, and our local socket only accepts requests from our own application the browser is safer than the usual situation. However later we'd like to offer the option of an in-application version of the interface using [http://code.google.com/p/pywebkitgtk/ PyWebkit] or [http://www.pygtk.org/pygtkmozembed/ PyGtkMozembed].
| |
| − | | |
| − | The interface uses the [[Single Page Application]] pattern for the structure of the interface code. This keeps the roles of the Python and the JavaScript very clear with no need for a messy mish-mash of interface functionality spanning both the sides. The entire interface application will be written in JavaScript and only reloads the page/application when the user changes the context to a different group (since each group decides for themselves which extensions will be active and therefore which JavaScript will be loaded in the page).
| |
| − | | |
| − | The local part of the URL consists only of the group name or Bitmessage address and everything after that is the hash-fragment, so that only a change in group will result in a page-reload. The URL format after the hash is a slash-separated path, the first element being the currently selected ''view'' and subsequent elements being decoded by that view's URL-routing method and sent to the relevant method for processing.
| |
| − | | |
| − | To allow for more rapid interface development and make it easier for other developers to be involved and extend the system we're including [http://jquery.com/ jQuery] and using [http://jqueryui.com/ jQueryUI] for all the interface elements such as tooltips, progress bars, tab-sets etc.
| |
| − | | |
| − | === Internationalisation (i18n) ===
| |
| − | All the interface messages are stored in the [http://svn.organicdesign.co.nz/filedetails.php?repname=tools&path=%2FBitgroup%2Finterface%2Fi18n.json i18n.json] file (this will probably be changed to separate files for each language later such as ''i18n.es.json'' and ''i18n.de.json'' etc). This file is shared by both the Python and JavaScript sides.
| |
| − | | |
| − | === Application start-up sequence ===
| |
| − | First the Python-based Bitgroup service needs to be run before the interface can be loaded since the interface code and resources are served to the browser from this service. The starting point is [http://svn.organicdesign.co.nz/filedetails.php?repname=tools&path=%2FBitgroup%2Fmain.py main.py] which reads in the ''.config'' file, ensures that the application isn't already running, and then instantiates an instance of the main ''App'' class which is defined in the [http://svn.organicdesign.co.nz/filedetails.php?repname=tools&path=%2FBitgroup%2Fapp.py app.py] module. When the ''App'' class initiates it establishes itself as a "superglobal" called ''app'' which is available from all scopes and modules of the application. This may not be a very [http://blog.startifact.com/posts/older/what-is-pythonic.html Pythonic] way of doing things, but I'm too noob and found the easiest way to get things rolling so that's how it is for now. I used the following line in ''app's'' ''__init__'' method to create the ''app'' super-global:
| |
| − | {{code|<python>__builtin__.app = self</python>}}
| |
| − | | |
| − | | |
| − | After the interface is running in the browser it can be used even if the Python service is stopped as long as the group is not changed. Any data changes made while the background service isn't running will be queued up and sent after it can be connected to again. A change in group requires a page reload (and therefore needs the service to be present) since only one group's data and extensions are loaded at once in case one group you're a member of chooses to run extensions that another group you're a member of doesn't trust.
| |
| − | | |
| − | When the browser first requests a page from Bitmessage's HTTP server (defined in [http://svn.organicdesign.co.nz/filedetails.php?repname=tools&path=%2FBitgroup%2Fhttp.py http.py]), it returns a page containing just an empty body element and a head element containing the main application script, the default views (currently "overview" and "newgroup") and some common resources such jQuery and jQueryUI. The page also contains a small number of variables specifying the current group, user information and the group's extension names. Everything then starts with the execution of the main application script, [http://svn.organicdesign.co.nz/filedetails.php?repname=tools&path=%2FBitgroup%2Finterface%2Fmain.js main.js].
| |
| − | | |
| − | The ''main.js'' script defines a class called ''App'' which encapsulates all the main functionality of the entire application, and then instantiates a ''singleton'' instance of this class called ''window.app''. Execution is then handed over to the constructor of the ''App'' class.
| |
| − | | |
| − | The constructor first copies all the data that was sent in the request HTML head element into the application instance, and then does an Ajax request to retrieve ''i18n.json'' which contains all the internationalisation messages used by the application. After the ''i18n'' messages have finished loading, the group's extensions are loaded and a handler is registered to respond to changes in the "hash fragment" portion of the URL (see [[Single Page Application]] for more details about what this means). Execution is then passed to the application's main start-up method, ''run()''.
| |
| − | | |
| − | The ''run()'' method first calls the ''locationChange()'' method to simulate the changing of the hash-fragment in the URL so that the current view can be rendered into the content area of the page. If there is no group selected, then the ''pageRender'' method is called since it will not be automatically rendered in response to data events in this case. The final step in the start-up process is to set up an event that runs every second which keeps the data synchronised between the backend and client side.
| |
| − | | |
| − | From then on, everything that happens in the ongoing running of the application is in response to either user interface events or data-change events coming from the Python service.
| |
| − | | |
| − | === Page rendering sequence ===
| |
| − | In a pure Single Page Application page rendering is a rare occurrence because there are no page reloads. In Bitgroup a page render occurs when the user changes the group as described above since it causes a page reload. A full page render is also done if data synchronisation hasn't occurred for some time or the number of changes received is over a certain amount.
| |
| − | | |
| − | The first response from the server side for a ''dataSync'' request returns the entire data structure (this happens if the last sync was more than a certain time ago or the number of changes is more than a certain amount). The ''dataSync'' method detects if the response is a list of changes or the whole structure, and if it's the whole structure it calls ''renderPage'' which builds the entire page from scratch.
| |
| − | | |
| − | The ''renderPage'' method first loads the CSS for the group's selected skin, or the default if no skin is selected. It then renders the personal links in the top bar and the views menu, and then renders empty ''div'' elements for the main page items such as notification. page title and content areas which will be filled later. The skin's JavaScript (if it has any) is then executed which may completely alter the layout of the page as long as it keeps the elements ID attributes intact.
| |
| − | | |
| − | After the skin's script as finished executing the ''renderPage'' method will then finish off the page by populating the title area, connecting the live menu elements to their data sources and executing the selected page view method, or ''overview'' if none is selected. The view method is responsible for populating the main content area of the page and is called not only in a full page render, but also whenever a new view is selected from the view menu or when the currently selected node changes.
| |
| − | | |
| − | === Forms & data ===
| |
| − | In a Single Page Application, the input elements that allow the user to update data and content in the application do not use the classic form/post methodology because everything is handled by the currently loaded page. In fact the entire ''form'' element is quite redundant and so is the ''name'' attribute of input elements. Instead the form components simply connect into the application functionality directly via event handlers.
| |
| − | | |
| − | In the Bitgroup application we have some methods of the main ''App'' class that make the handling of inputs much simpler. First we have ''componentType'' which returns a general type string when passed an input element. This type string is used to determine how the value of the input is set or retrieved and how changes in the value are to be connected in to the application. The ''componentRender'' method is a high-level helper function for constructing an input element given some data such as a list, and ''componentSetValue / componentGetValue'' are used for setting or retrieving the value of a form or other interface component.
| |
| − | | |
| − | Components can have their own ''getValue'' and/or ''setValue'' functions by adding them directly to the element and then these will be used by ''componentSetValue'' and ''componentGetValue'' if they're present. This way it's easy to make complicated custom components that are connected to live data changes.
| |
| − | | |
| − | '''App.componentConnect:''' The main function is ''componentConnect'' which creates a "live" connection between the specified path into the group's data and the interface component. The connection is bidirectional (if it's an input component) and asynchronous so that if the user changes the value it will automatically propagate back to the server-side to be stored and will automatically be reflected in any other open instances of the interface. Conversely if the data undergoes change from another remote instance via another user in the group or by some other back-end service, all interface components connected to the changed data will automatically have their values updated.
| |
| − | | |
| − | Here's a simple example of how it can be used:
| |
| − | {{code|<js>
| |
| − | $('#foo').html( app.componentRender( 'checklist', ['foo', 'bar', 'baz'] ) );
| |
| − | app.componentConnect( 'settings.extensions', $('#foo .checklist') );
| |
| − | </js>}}
| |
| − | In this example, a checklist (like a select list but with checkboxes instead) is rendered with items "foo", "bar" and "baz". The second line then connects this new input element to the data source ''settings.extensions'' in the group's shared data structure. As soon as this is done, any of the checklist items that are in the ''settings.extensions'' value will become checked, and if any of the items checked state are changed by the user, the group's data and any other pages containing the input will update accordingly.
| |
| − | | |
| − | Here's a slightly more complicated example which creates a live table connected to a data source that's an array. Whenever the data in the array changes, the element's custom setter method is called which renders the data as a table into the element.
| |
| − | {{code|<js>
| |
| − | var inbox = document.getElementById( 'inbox' );
| |
| − | inbox.setValue = function( val ) {
| |
| − | var rows = '<tr><th>From</th><th>Subject</th></tr>';
| |
| − | for( var i in val ) rows += '<tr><td>' + val[i].from + '</td><td>' + val[i].subject + '</td></tr>';
| |
| − | $(this).html( '<table>' + rows + '</table>' );
| |
| − | };
| |
| − | app.componentConnect( 'data.messages', inbox );
| |
| − | </js>}}
| |
| − | | |
| − | | |
| − | One complication (bug#2) is that the event handler prevents the DOM element that it's connected to from dying when it's no longer present in the DOM, so the event handler first checks this by using a ''jQuery'' statement to check if the component is a descendant of the ''body'' element, and removes itself from the event if not.
| |
| − | {{code|<js>
| |
| − | var handler = function( event ) {
| |
| − | if( $(element).parents().filter('body').length > 0 ) event.args.app.componentSetValue( element, event.args.val );
| |
| − | else $(document).off( event.type, null, handler );
| |
| − | };
| |
| − | $(document).on( "bgDataChange-" + key, handler );
| |
| − | </js>}}
| |
| − | The handler function is assigned to a variable in the local scope so that it can be referred to when it's called to unhook itself from the event using the [http://api.jquery.com/off/ jQuery .off() method].
| |
| − | | |
| − | === Skins ===
| |
| − | The main content is generated unconditionally by the main application and is a plain set of ''div'' and ''ul'' elements with CSS class and ID attributes allowing CSS to define the style and page layout. Skins are a directory of the skins name that go in the ''skins'' directory and contain at least a ''style.css'' stylesheet that will be loaded if the group has selected that skin. There can then be any number of support files such as images etc in the skin's directory that the stylesheet can refer to.
| |
| − | | |
| − | === Extensions ===
| |
| − | Extensions play an important role in the Bitgroup system. The idea is that each group's members can choose what extensions are active for them based on their needs and their security concerns. The extensions are only implemented in the JavaScript side currently as that's where the majority of the application's functionality resides - the Python side is only the foundation network functionality which is unlikely to require extending and raises security issues at least until the application has reached a mature state of development. Extensions are in the form of directories that reside in the ''extensions'' directory. The directory should be the name of the extension, and it should contain a JavaScript file of the same name within it that has the ''.js'' file extension. This file will be loaded prior to application startup if the current group has selected to use it. If the user changes to a different group, the page reloads so that a new selection of extension can load depending on the group's preferences.
| |
| − | | |
| − | ==== Adding new views ====
| |
| − | To add a new view, an extension defines a class, adds a render method which populates the ''content'' div, and adds the class to the available views in the ''app.views'' list. It may also add the view name to the various node types that should have the view.
| |
| − | {{code|<js>
| |
| − | function MyView() {
| |
| − | // do any initialisation of the view here such as loading dependencies or subscribing to events etc
| |
| − | }
| |
| − | | |
| − | // Populate the content div element
| |
| − | MyView.prototype.render = function(app) {
| |
| − | $('#content').html( 'Hello World!' );
| |
| − | };
| |
| − | | |
| − | // Create a singleton instance of our new view in the app's available views list
| |
| − | window.app.views.push( new MyView() );
| |
| − | </js>}}
| |
| − | | |
| − | ==== Application events ====
| |
| − | We need to add custom events throughout the code that extensions can subscribe to so they can adjust or extend the behaviour of the application. Extensions will initially only be available on the JavaScript side and will use the jQuery [http://api.jquery.com/trigger/ trigger] method. Events are triggered with parameters passed to the handler that are specific to the event are packaged inside an ''args'' object so that they can be adjusted since JavaScript doesn't support passing-by-reference on non-object variables.
| |
| − | | |
| − | This is an example of an event the could be triggered by an extension:
| |
| − | {{code|<js>
| |
| − | var args = { foo: 'bar' };
| |
| − | $.event.trigger({ type: "bgNewEvent", args: args });
| |
| − | alert( args.foo );
| |
| − | </js>}}
| |
| − | Here the argument ''foo'' is being output after the new ''bgNewEvent'' event is triggered to see if any handlers have changed it's value. We precede the event name with the "bg" prefix to ensure there's no conflict with existing events in the environment.
| |
| − | | |
| − | A handler could subscribe to this event and modify the ''foo'' value as follows:
| |
| − | {{code|<js>
| |
| − | $(document).on( "bgNewEvent", function( event ) {
| |
| − | event.args.foo = 'baz';
| |
| − | });
| |
| − | </js>}}
| |
| − | | |
| − | ==== Current events ====
| |
| − | '''bgHashChange'''
| |
| − | | |
| − | '''bgPoller'''
| |
| − | | |
| − | '''bgDataChange-KEY'''
| |
| − | | |
| − | == Development and testing ==
| |
| − | It can be very time consuming and difficult to set up an environment to test a group running in the Bitmessage network because it's difficult to set up and maintain multiple instances of Bitmessage. Running them on multiple machines presents its own problems and running them all on a single host is very processor and bandwidth intensive. Even after getting multiple instances set up and running it's still a maintenance headache and is a very laborious process having to wait minutes for messages to route across the network especially when many debugging processes often require many iterations which all involve long delays waiting for message propagations. These problems are multiplied when working in a low bandwidth environment like [[our rural net connection]].
| |
| − | | |
| − | So for these reasons I've created the [http://svn.organicdesign.co.nz/filedetails.php?repname=tools&path=%2FBitgroup%2Fdev.py dev.py] module which does two main jobs to help ease the development and testing process which are activated by adding the ''dev'' parameter to the command line arguments to launch Bitgroup in development mode.
| |
| − | | |
| − | The first things the ''dev.py'' module provides is a fake Bitmessage API which provides replicas of all the functions that Bitgroup requires from Bitmessage. Instead of sending messages peer-to-peer across the network, this fake version simply stores the data locally on disk allowing any number of local Bitgroup programs to communicate together as if they were communicating via Bitmessage. This is done in ''app.py'' by checking if the program is running in ''dev'' mode and replacing the ''app.api'' value with the ''fakeBitmessage'' class instead of the proper ''xmlrpclib'' connection.
| |
| − | {{code|1=<python>if app.dev: self.api = fakeBitmessage()
| |
| − | else: self.api = xmlrpclib.ServerProxy("http://"+username+":"+password+"@"+interface+":"+str(port)+"/")</python>}}
| |
| − | | |
| − | | |
| − | The second thing that development does is to make it easy to start and stop a number of Bitgroup instances with different user names and Bitmessage addresses. By adding a second command line argument to specify the number of instances to run, the first instance will then spawn a number of additional instances each with their own determinate fake Bitmessage addresses based on the hash of their nickname.
| |
| − | | |
| − | The nickname names are based on the nickname of the initial user which stays the same, the subsequent ones are appended with 2,3,4 etc, for example, if the main user's nickname is set to "Foo" and the script is run with the ''dev 3'' parameters, then three instances will start up with the nicknames ''Foo'', ''Foo2'' and ''Foo3''.
| |
| − | | |
| − | The port numbers that each instance's interface runs on start at the initial configured port plus one and then each successive instance is on the next port and so on. So for example if the port is set to ''8080'' in the configuration and three instances are running, their interfaces will be on port 8081, 8082 and 8083. This was done rather than starting at 8080 so that their right-most digit matches their nickname's digit.
| |
| − | | |
| − | All the settings (except the user) such as port number (initial port number in the case of multiple instances), groups and other settings are copied from the main configuration file. All the data for the development instances is stored in their own directories using their fake Bitmessage addresses for a name and which all reside in a directory called ''dev'' which resides in the same location as the Bitgroup Python scripts.
| |
| − | | |
| − | Each instance's subscriptions are in a file called ''subscriptions.json'' in their respective data directories. Unlike the subscriptions, the mailboxes are all stored in a single file which has a name based on the initial user's nickname and is directly under ''.dev'', for example ''.dev/messages.Foo.json''. This is done because each instance needs to access other instance's mailboxes to store broadcast messages and the file needs to be locked when it's being updated to avoid conflicts, so having only one file to deal with makes this a lot simpler. This is not necessary for subscriptions because instances only ever modify their own subscriptions.
| |
| | | | |
| | == See also == | | == See also == |
