Mastodon

https://post.lurk.org is a mastodon service. Mastodon is a federated microblogging software that speaks both ActivityPub and OStatus and can thus communicate with other microblogging softwares like GnuSocial, Pleroma, Pump.io etc.

admin resources

Useful pages from the mastodon documentation

• Installing Mastodon the guide is meant for Ubuntu 16.04 but it worked flawlessly on Debian Stretch
• Tuning mastodon performance TODO
• Mastodon admin commands from ruby terminal
• What and how to back up in Mastodon
• Updating to newer versions

Admin community / help

• Mastodon forum some discussions happen here
• Mastodon git issues some happen there

Installation

post.lurk.org followed the mastodon install almost literally since it was one-to-one applicable on debian stretch. Quite boring really.

This means that mastodon runs as the user mastodon. All the mastodon files live in:

/home/mastodon/live/

Differences are:

• When running the interactive set up during install, the smtp address is set as localhost and the postfix relay takes care of the rest.
• Mastodon-web runs on port 3001 instead of 3000, the changes to this are reflected in the systemd service files and in the nginx virtualhost config

Maintenance

Mastodon can be (re)started by:

systemctl stop mastodon-*.service
systemctl start mastodon-web.service
systemctl start mastodon-sidekiq.service
systemctl start mastodon-streaming.service

Sometimes, specially when disk is full, postgresql may have crashed, and mastodon-streaming service will start *but* internally enter a loop of complaining that it cannot connect to the db (super annoying because to see that you need to examine the status of the service), anyway:

service postgresql restart

Removing federated media attachments

RAILS_ENV=production ./bin/tootctl media remove

reduce disk space usage by cleaning out old versions of ruby, yarn etc after upgrades

rm the cache of yarn (nodejs package manager):

yarn cache clean

rm old versions of ruby you no longer need:

rbenv uninstall 2.5.3 

source: https://toot.cafe/@nolan/101450836285521185

script for pruning remote / old media

#!/bin/bash
export PATH="$HOME/.rbenv/bin:$PATH"
eval "$(rbenv init -)"
export RAILS_ENV=production

/home/mastodon/live/bin/tootctl statuses remove
/home/mastodon/live/bin/tootctl media remove --days=3

deleting remote inactive account and their associated avatars and headers

Deleting media attachments is not enough! Since Mastodon builds a local copy of every account it knows in the fediverse you will see the folders /home/mastodon/live/public/system balloon over time. Many instance admins are dealing with this issue and the prevailing attitude to solving it is to get more storage, which is bs.

So here is an interim solution based on this:

First query the mastodon_production database to find out which accounts haven't been active for 6 months or more:

sudo -i -u postgres /bin/bash -l -c "psql -A -d mastodon_production -c \"SELECT username||'@'||domain FROM public.accounts WHERE last_webfingered_at < (CURRENT_TIMESTAMP - interval '6 months') AND id NOT IN (SELECT target_account_id FROM public.follows)\"" | tail -n +2 | head -n -1 > stale.txt

Once you have that list use a scripting language to parse it. The below example is in python. Its suuuuuper slow so probably not the best way to do it. Calling rails like this is not so smart I guess. But hey it works. It went from 11.5GB worth of headers and avatars to 7.5GB.

mastodon@server:~$ cat del_stale_users.py 

#get the output from: 
#sudo -i -u postgres /bin/bash -l -c "psql -A -d mastodon_production -c \"SELECT username||'@'||domain FROM public.accounts WHERE last_webfingered_at < (CURRENT_TIMESTAMP - interval '12 months') AND id NOT IN (SELECT target_account_id FROM public.follows)\"" | tail -n +2 | head -n -1 > stale.txt
#and delete them

import os

stale_users=open('stale.txt').read().split('\n')

os.chdir('/home/mastodon/live')

command = """RAILS_ENV=production bundle exec rails r '
begin
    a = Account.find_by(username: "{}", domain: "{}")
    a.destroy
rescue => err
end'
"""

for user in stale_users:
    if user:
        username, domain = user.split('@')
        os.system(command.format(username,domain))
        print('deleted', user)

linkdump of bespoke maintenance scripts

https://mastodon.zombocloud.com/@staticsafe/103121989384729357

Make all the accounts force follow one specific account

Assuming you want everyone to follow @lurk@post.lurk.org then you would do the following:

su mastodon
cd ~/live
RAILS_ENV=production bin/tootctl accounts follow lurk --verbose

Notes:

• It can be ran live while the instance is up and running
• It takes a while for the command to get going, and it may spit a few warnings, then after that it's very quick and the --verbose flag helps to keep track of progress
• At time of writing, there's no way to specify which account(s) will be affected, everyone will be processed

Performance tweaks

Increasing character limit on posts

Search and replace '500' by whatever you want in these two files:

modified:   app/javascript/mastodon/features/compose/components/compose_form.js
modified:   app/validators/status_length_validator.rb

Next, we probably want to let the world know, too. Currently there’s no official way to include your character length in the API, but unofficially, you’ll need to set the max_toot_chars attribute in your instance’s API response.

in app/serializers/rest/instance_serializer.rb

Change line 8 from :languages, :registrations, :approval_required to :languages, :registrations, :approval_required, :max_toot_chars (don’t forget the comma).

Change line 65, after the approval_required block, and add a definition for max_toot_chars

 def max_toot_chars
     1500
 end

from https://indented.space/2019/07/28/change-max-character-limit-for-mastodon-instance/

Make sure you recompile the web assets afterwards:

RAILS_ENV=production bundle exec rails assets:precompile

Getting high scores on ssl comparison sites

instances.social automatically rates each fediverse instance using two different SSL testing sites:

• https://tls.imirhil.fr/https/post.lurk.org
• https://observatory.mozilla.org/analyze.html?host=post.lurk.org.

At the time of writing we got A and B (untweaked mastodon config). We are good boys and want to get A+ grades.

weak DH primes

The first is the weak Diffie-Hellman key primes described here and here.

Generate like so (this take a looong time):

cd /etc/ssl/certs
openssl dhparam -out dhparam.pem 4096

in the post.lurk.org nginx config we point to this new prime by adding this line:

ssl_dhparam /etc/ssl/certs/dhparam.pem;

content security policy, xss etc

In order to get A+ one hast to set explicit policies the sources and origins of where post.lurk.org gets loaded. The mozilla observatory has a lot of documentation on these topics. Because it is unclear how mastodon loads all of its resources it was a bit of fiddling to find out how strict we could be without breaking the site. This is done by adding headers in the nginx config:

 add_header Strict-Transport-Security "max-age=31536000; includeSubdomains; preload";
 add_header X-Frame-Options "DENY";
 add_header Referrer-Policy "strict-origin-when-cross-origin";
 add_header Content-Security-Policy "default-src 'none'; script-src 'self'; object-src 'self'; style-src 'self'; img-src 'self' data: https: blob:; media-src 'self'; frame-src 'none'; font-src 'self' data: https://post.lurk.org; upgrade-insecure-requests; frame-ancestors 'self'; form-action 'self'; base-uri 'self'; connect-src 'self' blob: wss://post.lurk.org *.lurk.org";

Backups

the Mastodon project advises to back up the following things:

• Postgres database
• Assets (avatars, uploaded files etc)
• Application secrets

We do so using the following shell script:

 today=(`date +"%F"`)
 expiry=(`date +'%F' -d "-3 days"`)
 
 /bin/mkdir  /var/backups/mastodon/${today}/
 /usr/bin/pg_dump mastodon_production > /var/backups/mastodon/${today}/mastodon_production_${today}.sql
 /bin/cp /home/mastodon/live/.env.production /var/backups/mastodon/${today}/
 
 /bin/tar -cvzf /var/backups/mastodon/${today}/system${today}.tar.gz /home/mastodon/live/public/system
 /bin/rm -rf /var/backups/mastodon/${expiry}/

Which is called in cron like so:

30 02 * * * /bin/bash /home/mastodon/backup_mastodon.sh > /home/mastodon/backups/backup.log 2>&1

Two weeks worth of backups are stored remotely using a shell script:

today=(`date +"%F"`)
expiry=(`date +'%F' -d "-14 days"`)
expiry_path=(/media/lurk_backup/mastodon/${expiry})

rsync -auv /var/backups/mastodon/${today} x@x.x.x.x:/media/lurk_backup/mastodon/
ssh x@x.x.x.x rm -rf $expiry_path

This is called in cron like so: 30 03 * * * /bin/bash /home/mastodon/backup_backup.sh > /home/mastodon/backups/backup_copy.log 2>&1

Statistics

Via the public API one can see the amount activity per week:

https://post.lurk.org/api/v1/instance/activity

and the amount of instances in the federation a server is connected to:

https://post.lurk.org/api/v1/instance/peers