Aller au contenu

How to develop on Ma Dada

This document is written in English in the hope that it might help other Alaveteli site owners.

Cette page est écrite en Anglais puisqu'elle peut aussi être utile aux autres site alaveteli.

Setup your development environment

We have a vagrant file that will build a virtual machine with the same setup as the production and staging servers. After following the steps below, you should be able to point your browser at http://192.168.56.2/ and see a local version of Ma Dada, that you can edit.

  • Install vagrant on your machine: https://www.vagrantup.com/docs/installation
  • install ansible: https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html
  • Clone this repository: git clone https://gitlab.com/madada-team/dada-core.git or git clone git@gitlab.com:madada-team/dada-core.git
  • cd dada-core
  • make setup-ansible-deps to install the ansible roles needed to build the machine
  • Copy ansible/group_vars/localdev/local.yml.example to ansible/group_vars/localdev/local.yml and set the transifex api token (you can get yours from https://www.transifex.com/user/settings/api/ after logging in to transifex)
  • vagrant up --provision to start the VM. This will take a while the first time, probably close to 15-30 minutes depending on your internet connection.
  • Once this is done, you should be able to connect to http://192.168.56.2/ from your machine and see a clone of madada.fr that is locally editable.

Make changes to the files locally, then:

  • vagrant ssh to connect to the development server, then
  • cd /var/www/alaveteli
  • sudo service nginx restart (or reload) to see your changes.

Alternative approach

Is is possible to autoreload the site on changes, using sudo -u alaveteli bundle exec rails server on the vm, and loading http://192.168.56.2:3000/. This does not load CSS properly at the moment.

Workflow overview

Overall, the development workflow is expected to function like this:

  • Make changes locally on the vagrant box you installed in the previous step
  • Once you're happy with your changes, commit them to a clean branch:

    • off staging if you are making changes to anything except the docs you are reading now.
    • off doc-public for changes to the documentation.
  • Open a merge request (MR) on https://gitlab.com/madada-team/dada-core/ using the same base branch as above.

  • Once reviewed, your MR is merged onto the base branch, which triggers an automatic deployment:

Note that some parts of the setup related to email antispam features in particular are not replicated on the staging setup, because they involve a lot of DNS setup that is hard to automate fully via ansible (though this ansible plugin could help with this!).

Comments

Make sure to leave plenty of comments when you change code, in one of three places:

  • Directly in the code. This is probably the easiest if changes are limited to a specific file or section of it. If a bit of code relates to some other non-obvious place, it can help to mention the connection in the code.
  • On the related ticket on gitlab. Link to it from the code or commit to help connect the various bits.
  • Commit messages are underused, and are a good place to connect changes that happened in multiple files (and they can be found easily via git blame).

Assume you are writing for yourself in 6 months. There is no need to explain that apt-get install installs a package, but it is probably helpful to mention why a package is needed if it's not obvious. Don't hesitate to link to documentation that helped you decide what to do, ideally summarising in your comment if possible, in case the original document disappears.

Sometimes you have to do things in a less-than-ideal way. It's a good idea to mention that in a comment, also explaining why you did so (urgent bug fix, waiting for upstream fix...).

Deployment

Commits to specific branches trigger deployments automatically via the CI mechanism:

The CI process should require no specific credentials once you've been allowed to commit to the branches listed above.

Where to make your changes

There are several bits of software working together, and finding where to write your code can be a bit confusing. This section tries to explain how things work together. It's a very good idea to read the infrastructure page before continuing.

Alaveteli code

Alaveteli is a ruby on rails application, and is developped in ruby. The source code is available on github.

We try to use the stock alaveteli code as much as possible. This is in big part to make upgrades less painful, as they happen 2-4 times per year.

Where possible, ie. for features that can be useful to other sites than just Ma Dada, we try to feedback our changes to the upstream alaveteli repository via pull requests. This requires agreeing with mySociety's team on how to do things so can take longer, but makes your work available to 25+ other sites.

For changes that are only useful to our site (for instance, if they relate to France's specific regulation only), then we can either:

  • use a patch system in the deployment process, which we have used for minor changes (a few lines of code),
  • switch to using our own fork of alaveteli. This is easier for bigger changes, but also makes upgrades potentially harder. We have used this for temporary changes, while waiting for our patches to be merged upstream.
  • use the patching mechanism they provide, described in the section below. This is the cleanest mechanism, and should be prefered when possible.

Model and controller patches

Alaveteli has a patch system for models and controllers that allows modifying its internal classes.

  • dada-france-theme/lib/model_patches.rb can be used to add custom actions on various models,
  • dada-france-theme/lib/controller_patches.rb to do the same on controllers.

Together and added to the custom theme, they allow adding specific behaviour to alaveteli to better suit our needs. Follow the examples in these files to get started. There is a custom action to allow downloading a full PDF summary of the entire conversation to appeal the initial decision ("saisine CADA").

Other software components (database...)

Virtually everything else should be modified in the dada-core repository. This includes:

  • upgrading software that Alaveteli relies on (database, ruby, operating system...)
  • modifying translations, text on the main site, documentation, statistics... (see details below for each of these)

Backups

The backup system is split in two parts, both defined in the ansible playbook.

It relies on wal-g to backup the database contents, and restic for other files (incoming raw emails).

Restoring a backup

On the server which will receive the restored database: - sudo service postgresql stop - edit /etc/postgresql/12/main/postgresql.conf and update the restore_command to read:

restore_command = '/usr/bin/envdir /etc/wal-g.d/env /usr/local/bin/wal-g wal-fetch \"%f\" \"%p\" >> /tmp/wal.log 2>&1'
- sudo -su postgres - envdir /etc/wal-g.d/env /usr/local/bin/wal-g backup-fetch /var/lib/postgresql/13/main LATEST to fetch the latest base backup (which should be from last night) - Optionally, to do a point in time recovery (PITR), for instance to reset the database to right before a major disaster: - echo "recovery_target_time = '2020-07-27 01:23:00.000000+00'" >> /etc/postgresql/13/main/postgresql.conf - echo "recovery_target_action = 'promote'" >> /etc/postgresql/13/main/postgresql.conf - touch /var/lib/postgresql/13/main/recovery.signal This file will be automatically deleted by postgresql once the recovery is complete. - exit to return to the main user - sudo service postgresql start which will start the database server, and restore all the WAL delta backup files. This might take a few minutes, during which the db server will return errors saying that it in starting up. - sudo tail -f /var/log/postgresql/13/postgresql-13-main.log will show progress. When the restore process is complete, it should show a line like archive recovery complete followed by database system is ready to accept connections

You might need to update passwords for the main db user.

Deployment process

The ansible playbook should hold enough info to rebuild an almost working Ma Dada site from a clean server (meaning a fresh machine with ubuntu installed, and an ssh access).

This means that the configuration details of all our services can be found in the dada-core repository.

Our ansible playbook lives under the ansible folder.

Variables and Secrets

Under group_vars you will find most config values, which are then reused across the playbook. They are split per deployment (staging/production, and shared all). In each folder, you will find a main.yml file for publicly visible values, and a vault.yml that is encrypted using ansible-vault. Each encrypted value is named vault_some_value and is then renamed some_value = {{ vault_some_value }} in the main.yml file. This means that all configurable values should appear in one of the main.yml files, to make it easier to get a complete view of available options. There is no vault.yml file under all as secrets should not be shared between servers.

The vault password is stored as a CI environment variable in gitlab so that the deployment process has access to it.

Tasks

The rest of the playbook lives under roles/alaveteli, which is split in 4 sections:

  • tasks holds the actual actions that are performed on the server (copy a file, install a package...). There are several files in that folder. main.yml holds pretty much everything except for the metabase deployment tasks, which are in their own file. Ideally, each service should be managed in its own file for clarity.
  • meta has configuration for the few dependencies which use shared ansible roles (postgresql and a swap file at the moment)
  • vars holds some extra variables. Ideally these should be moved to the group_vars section, so that everything lives in one place.
  • templates has ansible templates for all config files that are used on the server (and require more than 1-2 lines of changes compared to their default versions). ansible uses jinja2 templates. We don't have a strong naming scheme for these, as long as the name you choose makes the intent clear, it should be fine. The variable placeholders in them are filled with config values described in the previous section.

Most changes will involve working in the tasks and maybe the templates folders.

Pin your dependencies

Please pin versions wherever possible. As we rely on automated deployments, we aim to make them reproducible, and to know that once the process works, it should keep working for a reasonable period of time.

Mailing system

Emails are central to Alaveteli and Ma Dada, and rely on fairly custom behaviours, so we run our own Mail Transfer Agent (MTA). We use postfix.

We pay particular attention to not getting marked as a spam site, as this would seriously jeopardise the usefulness of the site.

There are several paths for emails leaving madada.fr and coming back to it:

Outgoing emails

Emails sent by the platform (mostly CADA requests) are sent from a "magic address" in the format dada+request-<request_id>-<some_hash>@madada.fr. These are kept secret so that only the intended recipient can see them and reply to the request.

These and all other emails sent from the site are sent by our MTA (on the same machine as the website itself).

We also use gandi's MTA for some (low volume) emailing directly as contact@madada.fr.

Incoming magic emails

Incoming emails are all routed to our MTA, and follow different paths:

  • magic emails for individual requests are piped by postfix to a script in alaveteli that handles them.
  • magic emails for batch requests and requests sent by pro users are sent to our POP server (dovecot)

This split logic is defined by alaveteli, we just follow it.

Other incoming emails

Other email addresses are aliases, and incoming messages to them are forwarded to their respectives recipients (for instance, contact@... is such an alias, which is forwarded to our core team. The exact list of recipients is configured in an encrypted variable in the ansible playbook).

DNS and anti-spam measures

We receive a little bit of spam, but our main concern is not being blacklisted as a spam server. For this, we follow several industry standards:

SPF

This DNS record lists which email servers are allowed to send email from any @madada.fr address.

TLSA / DANE

This protocol helps verify that TLS certificates used in the TLS connection setup are indeed issued by who they claim to be. We use letsencrypt everywhere (web and email share the same certificate). It relies on the domain DNS using DNSSEC (which is automatically provided by gandi).

DKIM

This signs outgoing messages using a private key so that receiving servers can check that signature against our DNS record, which provides the public key to use. This helps ensure that an email that appears to come from a @madada.fr address was indeed authorised by us. This is a combination of DNS records and server-side setup (for the private key), which is defined in the ansible playbook.

DMARC

This specifies which measures we have put in place to prevent spam (SPF/DKIM) and tells receiving servers what to do with email that does not pass those checks (discard, mark as spam, do nothing).

There are more details in this gitlab issue, including the record of changes made over time.

There are many tools available online to check the setup of the email server. This one has helpful results, as they are broken down per topic and reasonably explained.

Templates and translations

Alaveteli uses the ruby on rails templating mechanism to allow us to customise the looks and behaviour of our site.

The Ma Dada theme is here. To update our theme, you need to modify files in this folder, not in the upstream alaveteli repository.

Finding the right template to modify is a bit hit and miss. Using a mix of:

  • searching for the text that appears on the site that you want to change,
  • exploring the generated HTML for CSS ids or class names,
  • looking for templates with a relevant name,

you should be able to find what you want.

If you cannot find the file in our them, you need to override a standard template:

  • Find which is the template in the alaveteli code (they live under app/views)
  • Copy the file to our template folder (under dada-france-theme/lib/views). Make sure to reproduce the same folder structure and keep the filename.
  • Modify the copied file as needed. Keep the text in its original language, as the translation happens via a different system.

Split templates

Note that templates for "Pro/Madada++" and regular users are sometimes different. The former are usually found under the dada-france-theme/lib/views/alaveteli_pro folder.

This page (in French) explains how to translate content on the site.

Documentation

Our documentation is written in MarkDown and built using Material for MkDocs. The content is deployed using CI to gitlab pages. The result is visible at doc.madada.fr.

The aim here was to have documentation that is easy to maintain and update. The original docs in HTML included in Alaveteli worked well, but were very heavy to update, as they had to be written in plain HTML.

The source code for documentation lives in dada-core/docs.

To develop on the docs, from the / root folder of the repo:

  • virtualenv -p /usr/bin/python3.10 venvdoc to create a virtual environment named venvdoc
  • . venvdoc/bin/activate to activate it (don't forget the initial dot .)
  • pip install -r docs/requirements.txt to install dependencies
  • mkdocs serve to start a local build of the docs, which will auto-update every time you save your work.

If you add a page to the documentation site, make sure to list it under the nav section in mkdocs.yml or it won't appear.

For further documentation on how to create content in Markdown with MkDocs or how to create pages and navigation see the official user guide for MkDocs.

For some extra Markdown features (like extended lists, extra icons, etc.) see the Material for MkDocs reference guide.

Use the correct branch

Make sure you are working against the doc-public branch when updating docs, as this is the one that will deploy changes to the main site.

Statistics

We use metabase to display the data visualisation dashboard.

It is deployed in a docker container, which is defined in the playbook in ansible/roles/alaveteli/templates/docker-compose_yml.j2 and ansible/roles/alaveteli/tasks/metabase.yml.

Metabase does not have access to the live database, instead it works off a cleaned copy of it, which only contains anonymised metadata.

This dump is produced every night by a cron job.

External data sources

We are working towards connecting Ma Dada with various external services to increase the value of the information we produce.

CADA maintained list of FOI officers (PRADA)

In France, the CADA is the organisation in charge of access to information. They maintain a list of PRADAs (Personne Responsable de l'Accès aux Documents Administratifs). We track such changes with a simple CI job that runs every night. It downloads the latest version and compares it to the previous one. The resulting diff can be used to trigger various actions, such as a change request on a public body.

The code lives in a separate repository, which has specific explanations on how to work with it.

Upgrading alaveteli

The process for upgrading alaveteli is usually straightforward. Dependencies can be more difficult, particularly if we wait several months before installing the latest release.

The team at mySociety post notifications about releases on their dev mailing list.

Before upgrading, start by reading the changelog and particularly Upgrade Notes for the version you want to upgrade to. Make sure to upgrade one release at a time.

Upgrading dependencies

It is a good idea to regularly upgrade dependencies and software versions. Alaveteli upgrades will force some of these (particularly around postgresql and ubuntu versions), but other parts like the backup software won't be impacted. Versions should be pinned wherever possible, and are normally defined directly in the alaveteli/tasks/main.yml file.

Programs to keep up-to-date include:

  • restic for file backups
  • wal-g for database backups

Reusing our deployment mechanism for your site

We have tried to setup our deployment system to be country-agnostic. Although our priority is obviously to make Ma Dada function, where possible, we've made things configurable. All our code is freely reusable, just like Alaveteli itself. What we have added is the tooling to deploy the various bits that alaveteli requires to function (database, mail servers, etc...). If you would like to reuse our code, it's probably best to get in touch with us so we can check that it will help you and not hinder you in what you are trying to do :)


Dernière mise à jour: 2023-06-27