Taskotron Development Guide

So, you’re interested in helping out with libtaskotron? Great, this will help get you started on that.

General information

Libtaskotron is written mostly in Python.

Source code, bugs, tasks

The source code for libtaskotron is available at: https://pagure.io/taskotron/libtaskotron

This project is also used to track bugs and tasks. If you submit patches, please use the process of Submitting code.

Installing a Development Environment

Consider whether you really need a libtaskotron development environment. Maybe you simply want to develop tests that can be executed by libtaskotron runner, or you want to use the libtaskotron Python library in them? In that case, please read Running Taskotron Tasks and Writing Taskotron Tasks. If you really want to develop libtaskotron itself, please continue.

For the moment, libtaskotron can’t be fully installed by either pip or rpm and needs a bit of both for now.

On your Fedora system, install the necessary packages:

sudo dnf install       \
createrepo_c           \
gcc                    \
git                    \
python3-doit           \
python2-hawkey         \
python3-hawkey         \
python2-koji           \
python3-koji           \
python2-libvirt        \
python3-libvirt        \
python2-pip            \
python3-pip            \
python2-rpm            \
python3-rpm            \
python2-rpmfluff       \
python3-rpmfluff       \
python2-testcloud      \
python3-testcloud      \
python3-tox            \
python2-virtualenv     \
python3-virtualenv     \
rpm-build              \
rsync                  \
testcloud

If you have not yet cloned the repository, do it now:

git clone https://pagure.io/taskotron/libtaskotron.git
cd libtaskotron

Then, set up the virtualenv:

virtualenv --system-site-packages env_taskotron
source env_taskotron/bin/activate
pip install -r requirements.txt

If you encounter any installation issues, it’s possible that you don’t have gcc and necessary C development headers installed to compile C extensions from PyPI. Either install those based on the error messages, or install the necessary packages directly to your system. See requirements.txt to learn how.

Finally, you should install libtaskotron in editable mode. This way you don’t need to reinstall the project every time you make some changes to it, the code changes are reflected immediately:

pip install -e .

Before running any task, you also need to manually create a few required directories. First, create a taskotron group if you don’t have it already, and add your user to it (you’ll need to re-login afterwards):

getent group taskotron || sudo groupadd taskotron
sudo usermod -aG taskotron <user>

Now create the directories with proper permissions:

sudo install -d -m 775 -g taskotron /var/tmp/taskotron /var/log/taskotron \
  /var/cache/taskotron /var/lib/taskotron /var/lib/taskotron/artifacts \
  /var/lib/taskotron/images

Configuration

Libtaskotron searches /etc/taskotron and a conf/ dir in your local checkout for configuration files. There’s currently one config file that is mandatory to be present, so if you don’t have libtaskotron RPMs installed, make sure it is present (and keep it up-to-date), like this:

cp conf/yumrepoinfo.conf.example conf/yumrepoinfo.conf

If you need to override the default values of other config files, you can copy over the *.example files in the conf/ dir in a similar fashion and edit them. The configuration files in conf/ take precedence over anything in /etc.

In the development environment, it’s also useful to have taskotron-generated files automatically cleaned up, so that they don’t occupy disk space in vain. There is a tmpfiles.d template prepared for you, look into conf/tmpfiles.d.

Running the Test Suite

We place a high value on having decent test coverage for the libtaskotron code. In general, tests are written using pytest and are broken up into two types:

  • Unit Tests test the core logic of code. They do not touch the filesystem or interact with any network resources. The unit tests should run very quickly
  • Functional Tests are a set of more comprehensive tests which are allowed to touch the filesystem and/or interact with networked resources. Functional tests are often much slower than unit tests but they offer coverage which is not present in the unit tests.

To run the unit tests, execute this in the checkout directory:

tox

The testsuite is seamlessly executed in both Python 2.7 and 3.6 environments, as defined in tox.ini.

A nice HTML-based representation is available if you add --cov-report=html command line parameter.

If you write new code, be sure to run this to see whether the code is sufficiently covered by the tests.

Submitting code

Libtaskotron follows the gitflow branching model and with the exception of hotfixes, all changes made should be against the develop branch.

If you want to use the gitflow plugin for git to make this process more user-friendly, simply install the gitflow package.

Start a new feature

To start work on a new feature, use:

git checkout -b feature/XXX-short-description develop

or if you want to use gitflow, use:

git flow feature start XXX-short-description

where XXX is the issue number in Pagure and short-description is a short, human understandable description of the change contained in this branch.

In general, short reviews are better than long reviews. If you can, please break up features into chunks that are smaller and easier to manage.

Submitting code for review

Make sure to run all unit and functional tests before submitting code for review. Any code that causes test failure will receive requests to fix the offending code or will be rejected. See Running the Test Suite for information on running unit and functional tests.

Updating code reviews

There will often be requests for changes to the code submitted for review. Once the requested changes have been made in your feature branch, commit them and make sure that your branch is still up to date with respect to origin/develop.

Pushing code

Note

Changes to git are very difficult to undo or cleanup once they have been pushed to a central repository. If you are not comfortable going through the process listed here, please ask for help.

If you run into any problems before pushing code to origin, please ask for help before pushing.

Once the review has been accepted, the code needs to be merged into develop and pushed to origin.

Make sure that your local develop branch is up-to-date with origin/develop before starting the merge process, else messy commits and merges may ensue. Once develop is up-to-date, the basic workflow to use is:

git checkout feature/XXX-some-feature
git rebase develop

To merge the code into develop, use one of two commands. If the feature can be reasonably expressed in one commit (most features), use:

git flow feature finish --squash XXX-some-feature

Else, if the feature is larger and should cover multiple commits (less common), use:

git flow feature finish XXX-some-feature

After merging the code, please inspect git commit description and make it prettier if needed. Groups of commits should at least have a short description of their content and a link to the issue in Pagure. Once the feature is ready, push to origin:

git push origin develop

Documentation

The documentation for libtaskotron is made up of several sources which are combined together into a single document using Sphinx.

Most of the libtaskotron docs are written in reStructuredText (reST) and unless otherwise mentioned, follow the Python documentation style guide.

Building documentation

The documentation is easy to build once deps are installed. Have the virtualenv active and run:

doit builddocs

Docstrings

Docstrings should be formatted using a reST-like format which Sphinx’s autodoc extension can easily render into HTML.

Sphinx has several built-in info fields which should be used to document function/method arguments and return data. Read the Sphinx documentation on info fields.

reST documentation

The majority of libtaskotron’s documentation is in the form of reST files which live in the docs/source directory in the git repository.

For more information see:

ReST headings

We use the suggested reST headers as outlined in the Python documentation style guide on section headers.

Build Automation

There are several development related tasks which are at least somewhat automated using doit.

You can see a list of available tasks and a short description of those tasks by running doit list. Some of the available tasks are:

  • buildsrpm takes a snapshot of current git repo and uses the in-repo spec file to build a matching srpm in the builds/<version> directory. Note that if a snapshot already exists for a given version, a new snapshot will not be generated until the existing one is deleted.
  • chainbuild uses mockchain and the existing COPR repo to build a noarch binary rpm from the latest srpm.
  • builddocs builds documentation from current git sources

By default, the tool is pretty quiet but if you would like to see more verbose output, add --verbosity 2 to the doit command and all stdout/stderr output will be shown.