Mastodon

Mastodon is a free, open-source social network server based on ActivityPub. Follow friends and discover new ones. Publish anything you want: links, pictures, text, video. All servers of Mastodon are interoperable as a federated network, i.e. users on one server can seamlessly communicate with users from another one. This includes non-Mastodon software that also implements ActivityPub such as GNUsocial, Friendica, Hubzilla and PeerTube! The easiest way to get started on Mastodon is to join one of the existing instances, but here at OD we're running our own to get familiar with it all which we're documenting here.

Mastodon is federated which means that you can interact with users who reside on other servers that also use the ActivityPub protocol. Users can export their data including connections and toots, and can easily move everything over to a new instance at any time. It's becoming increasingly clear that corporate controlled "walled gardens" like Twitter and Facebook are not worth spending time and effort building up a following in since they can delete accounts at any time for whatever reasons they like, including simply not falling in line with the mainstream narrative.

Personally I never had much of a following on Facebook or Twitter - it felt like a futile things to do since all the data I built up was in somebody else's hands, but in the "fediverse" (the universe of connections using the AcitviyPub protocol) you control all the information that composes your posts and connections yourself.

Mastodon is similar to Twitter but has some differences. It uses "toots" instead of "tweets", and favourite stars instead of Twitter's hearts or Facebook's likes. Instead of retweets, Mastodon uses a concept called boosting which works the same way except that you can't add you own text to the boosted message. This has been done deliberately so that only the original message's intent gets spread and trolling in the network is reduced. To mention a remote user in a toot you need to also include their domain, such as @fred@example.com. Another thing they've done differently is that the favourite and reply count is not shown, they didn't want to create the competitive rating behaviour on toots which I guess I can understand. Here's @Gargron talking about the reasoning behind this.

To follow a remote user, you go to the user's profile page on their server and follow them there, that brings up an option to enter your @name@server ID so the remote server can request the follow action from your local server that you're logged in to. To unfollow a remote user, you can do it from the follow notification if they've followed you as well, otherwise you need to block and unblocking them.

For a more detailed look into how to use Mastodon and why it's been made the way it has, see this excellent introductory guide by @kev@fosstodon.org. The rest of this article will focus on the installation and administration of a Mastodon instance.

General architecture

The main application is written in Rails and runs in the "web" container. There are also two other containers for the application, "streaming" which handles the long-term web-socket connections, and "sidekiq" for background processing such as mailing and push notifications. The main database is postgresql, but there's also a redis database which is heavily used throughout the application and is best backed up as well even though the loss of its data can be survived. The web-server is not in the Docker containers, instead it's expected that your main web-server will reverse proxy into the ports exposed by the application containers.

Installation

Mastodon has a lot of dependencies that we don't have installed on our server such as PostgreSQL and Ruby, so for us the Docker image is definitely the preferred route, but it's till quite complicated and needs to be done via Docker Compose. This is our procedure which is based on the official installation.

Clone the Docker repo

First, create a mastodon group with number 991 which is used by the project, then create a directory for the persistent data that will be used by the containers (we're putting our repo and data in /var/www/domains along with other web applications - this is not under our document root!), clone the Mastodon Docker repo and checkout the latest stable version.

groupadd -g 991 mastodon
useradd -u 991 -g 991 -c "Mastodon User" -s /usr/bin/nologin -d /var/www/domains/mastodon-data mastodon
mkdir /var/www/domains/mastodon-data
cd /var/www/domains
git clone https://github.com/tootsuite/mastodon.git mastodon-docker
chown -R mastodon:mastodon /var/www/domains/mastodon*
cd mastodon-docker
git checkout $(git tag -l | grep -v 'rc[0-9]*$' | sort -V | tail -n 1)

Dockerfile

The Dockerfile has a chown -R command in it that takes up to an hour to run whenever docker-compose build is run. This issue is known about, but they've chosen to keep it like this for now so that it doesn't break on older versions of Docker. But this is a real show stopper, so to get around it you can comment out the slow separate chown command and add it as an option to the previous copy command instead as follows:

COPY --chown=mastodon:mastodon . /mastodon
# RUN chown -R mastodon:mastodon /mastodon

docker-compose.yml

docker-compose.yml is the file that determines what services will be included in the instances and their versions and data locations. This file should be backed up in case a full rebuild of your Mastodon instance is required at some point. There is nothing private in this file so you can store it in your configuration repo or wiki etc.

Before running any docker-compose commands we need to edit the docker-compose.yml file. Change all the images to use the version of the repo you chose above, e.g. "image: tootsuite/mastodon:v2.5.2". Note also that there are three services that use the mastodon container, web, streaming and sidekiq - all need to have the version added. You may want to enable the elastic search section too. I like to change the restart options from "always" to "unless-stopped" as well.

Uncomment all the volume path lines for data persistence. By default the host part (the path before the colon) of each is just a relative path which means that the data will end up residing in directories within the docker repo directory. We've decided to use a separate mastodon-data directory instead to keep the data separate from the main codebase, so the relative paths need to changed to the absolute paths we set above (/var/www/domains/mastodon-data).

But, since the data is in another location, a symlink needs to be created in the repo public directory pointing to the public public/system directory in the data directory, since the web-server's document_root will be pointing at the repo's public directory which contains all the web files.

.env.production

.env.production is the file that represents your Mastodon instance's basic configuration such as domain, secret keys, database and SMTP connections and other services. This file is best backed up (at least the non-reproducible parts such as the secrets and keys) in a secure location as it is needed if you ever need to do a complete rebuild of your mastodon instance.

Copy the .env.production.sample to .env.production and run the setup wizard. Note that most of the questions can be just set as default by entering nothing. Answer "yes" to save the configuration, create the schema and admin user etc.

docker-compose run --rm web bundle exec rake mastodon:setup

For the email configuration, it's best to run through the live tests, because the settings can be very temperamental. I've found that localhost doesn't work and an actual external domain that resolves to the SMTP server is needed. No login details are needed though because the mail server still sees that the request is local and allows it to be sent without credentials or TLS.

The configuration that will be used is output to the screen as well, and I've found that it's best to copy this so that if there's any problems you can manually put these into the .env.production and then run docker-compose build to make the changes take effect. You can also re-run the setup script by deleting all the persistent data from mastodon-data to create a fresh install, and then building again.

Note: There's a step that runs the chown -R command over the whole mastodon directory structure which for some reason takes a very long time to run, just let it be and it eventually ends.

Start the instance

Then if all has gone well, you can now run the main Mastodon instance with docker-compose up -d which should give something like the following. To stop the instance use docker-compose down. Note that the docker-compose commands must be run from within the mastodon-docker directory.

# docker-compose up -d
Creating network "mastodon-docker_internal_network" with the default driver
Creating network "mastodon-docker_external_network" with the default driver
Creating mastodon-docker_es_1    ... done
Creating mastodon-docker_redis_1 ... done
Creating mastodon-docker_db_1    ... done
Creating mastodon-docker_sidekiq_1   ... done
Creating mastodon-docker_web_1       ... done
Creating mastodon-docker_streaming_1 ... done

You can now see all the containers running with docker ps, the container into which you can enter or log for debugging is the mastodon_docker_web_1 container which is shown in the last column of the docker ps table.

CONTAINER ID  IMAGE                       COMMAND                  PORTS                                NAMES
01d48ec588c0  tootsuite/mastodon:v2.5.2   "/sbin/tini -- bundl…"   3000/tcp, 4000/tcp                   mastodon-docker_sidekiq_1
1aeb232bfd5d  tootsuite/mastodon:v2.5.2   "/sbin/tini -- yarn …"   3000/tcp, 127.0.0.1:4000->4000/tcp   mastodon-docker_streaming_1
6973bc5bb637  tootsuite/mastodon:v2.5.2   "/sbin/tini -- bash …"   127.0.0.1:3000->3000/tcp, 4000/tcp   mastodon-docker_web_1
6335b6ff1f8e  docker.elastic.co/elas...   "/usr/local/bin/dock…"                                        mastodon-docker_es_1
ede8d06a420f  postgres:9.6-alpine         "docker-entrypoint.s…"                                        mastodon-docker_db_1
0ca51e27180c  redis:4.0-alpine            "docker-entrypoint.s…"                                        mastodon-docker_redis_1

Connecting with Nginx

Now that we have a running Mastodon instance in a container, we need to connect it to our web-server outside the container. This simply involves creating an appropriate server block to connect request to our Mastodon domain to the ports exposed by the containers. I'm basing my server block on this page of the official documentation, see also this excellent guide by Dave Lane (but note that CSP and XSS headers are added by Mastodon automatically now).

Theming

The basics of adjusting themes is shown here, but these instructions only cover how to make adjustments to the default theme. We're running the mastodon-light theme, so in our case we created an app/javascript/styles/organicdesign.scss containing our new CSS rules and change variables etc. The variable settings from your existing theme go at the top before your custom variable changes, and the other CSS files your parent theme originally included go at the bottom after your variable changes, check the Github repo to see what your original theme includes. You can also see all the variables used by your theme and the common application variables in the repo. You may also want to add your own custom CSS rules which can go before or after the original CSS includes as required. Here's a short example, our full scss file is here.

/* Original theme variable definitions */
@import 'mastodon-light/variables';

/* My new variable definitions */
$classic-secondary-color: #f2f2f9;
$ui-base-color: #f2f2f9;
$ui-highlight-color: #4e507f;

/* Additional CSS imported by the theme */
@import 'application';
@import 'mastodon-light/diff';

For these changes to take effect, you also need to edit the config/themes.yml file and change the mastodon-light entry to point to your newly created custom scss file instead of the original one. After you've made your changes, you'll need to stop the system, rebuild it and bring it up again.

docker-compose build
docker-compose down
docker-compose up -d

Images: One slightly annoying thing about the way the Mastodon skin is done is that it uses a lot of img elements instead of using div elements with background images. But you can actually make an image invisible and change it to show only a css background image as shown here by Marcel Shields.

Using the API

Mastodon comes with a web API so that bots and other apps can interact with instances and their content and users. The main documentation is here. Some calls are publicly available, but others such as posting a status require an access key. Although there is a workflow for creating via the API, the easiest way is to just log in to the account (or create one) manually, then go to the "development" item in the accounts settings and add an application. Clicking on the newly created application in the list shows a Client Key, a Client Secret and the API Access Token.

Administration

You can check the logs of the Mastodon instance by running the docker logs command on the main container ID, there are --follow, --tail, --timestamp options. See the Docker page for other Docker-specific commands.

docker logs mastodon-docker_web_1

To list the available rake tasks (admin functions):

docker-compose run --rm web rake -T

But note that now most of the application-specific tasks have been migrated into the tootctl utility, do the following to get a list of available tasks:

docker exec -it mastodon-docker_web_1 bin/tootctl help

Upgrade

See the upgrading section of the Docker guide. It basically just involves updating the repo and checking out the latest tag while preserving your docker-compose.yml settings (but changing the image versions in it). Then run the db:migrate task too in case any changes have been made to the database schema.

Note: I like to manually back up the changed files (check with git status what's changed, but mainly docker-compose.yml and Dockerfile are the important ones to get right), then do a git reset --hard then fetch and checkout the new tag. You can then compare the new config files with the old ones and migrate the necessary changes across. This is better than simple stashing and popping because, the configs can undergo a lot of change from version to version.

docker-compose build
docker-compose run --rm web rake db:migrate
docker-compose up -d

Backup

The mastodon-data directory holds everything needed for a backup, but it's important to back up the postgresql database with a proper dump as well since the files can become corrupted and unusable. The redis database is also very important, but a dump is automatically maintained in the redis/dump.rdb file in the data directory and is safe to use for live backup, see this post about backing up and restoring redis databases. Both databases should be backed up at least daily, but the rest of the files can be backed up less regularly.

docker exec mastodon-docker_db_1 pg_dump -Fc -U postgres postgres > dump.pgsql
7za a mastodon-backup.7z dump.pgsql /path/to/data/redis/dump.rdb

Tips & Tricks

• To see a list of you muted/blocked users click the triple-dot more menu above the compose toot input. This is not really a tip, but it can be really hard to find!
• To do a remote unfollow, either use the icon on the notification if they've followed you too, or block/unblock them.
• Blocking: Blocking someone also stops them from following you, but you have to first follow them to be able to then block them. If you then unblock them again, they're still not following you but you can still see things from them in you feed via other people, and they can still follow you again later or message you etc.
• You can follow an account from your own instance (which is useful for sites like friendica which don't provide a public follow link) by using the https://organicdesign.pub/authorize_interaction?acct=user@instance URL format.
• Currently there is no way unfollow remote users, you have to block them and then unblock again (you can see your blocked and muted contacts in the lists option)

Related Articles

• ActivityPub - our local article
• Resources for Noobs
• Motherboard.Vice's deep-dive into Mastodon
• Guide to Mastodon including MastodonToTwitter.py

Fediverse bots

• Resources for making fediverse bots
• fedievrse bots at BotWiki
• @lou.lt@mastodonusercount - regular reports on number of users in the fediverse
• @federationbot@mastodon.host - regular report on most active instances
• @trendybender@organicdesign.pub - Organic Design's own trending tags bot :-)
• @TrendingBot@mastodon.social - regular report on trending hashtags in the network (stopped running in October)
• @fediverse@pleroma.fr - reports on changes in the state of instances

Mastodon tools & resources

• Masto.Host - fully managed mastodon hosting starting at €5/month
• Awesome mastodon - great list of groups, tools and bots
• Search.Mastodon.Tools - list of various mastodon apps and tools
• Mastodon instances list - opt-in list with searcher
• Mastopeek - crawler-based lister/searcher
• Mastodon instances and users with creation times
• Mastodon network overview
• HellThread visualiser
• CommunityWiki/Trunk - lists of people in the fediverse to follow based on topic

Mastodon plugins

• Autopost-to-Mastodon - Wordpress posts automatically replicated to a Mastodon account
• Bitbee Mastodon - Bitbee IRC server with connectors to various protocols including Mastodon

See also

• Installation on a dedicated server without Docker
• Administration guide
• Backup guide
• List of admin functions
• NextCloud joins the fediverse