API Docs for libtaskotron

If you’re writing a new Taskotron task, you’ll probably mostly interact with Directives instead of libtaskotron API. There’s one big exception, though, the check module for easily managing and exporting results in the Task Result Format. Of course you’re very welcome to any other module in your task as well.

Apart from that, this API is useful when creating new directives or developing Taskotron.

Core modules

arch_utils

class libtaskotron.arch_utils.Arches

Helper class for working with supported architectures inside taskotron

base = [‘x86_64’, ‘armhfp’, ‘ppc64le’, ‘aarch64’, ‘ppc64’, ‘s390x’, ‘i386’]

base architectures

binary = {‘x86_64’: [‘x86_64’], ‘armhfp’: [‘armhfp’, ‘armv7hl’], ‘ppc64le’: [‘ppc64le’], ‘aarch64’: [‘aarch64’], ‘ppc64’: [‘ppc64’], ‘s390x’: [‘s390x’], ‘i386’: [‘i386’, ‘i486’, ‘i586’, ‘i686’]}

a mapping from base arches to concrete binary arches

known = [‘aarch64’, ‘armhfp’, ‘armv7hl’, ‘i386’, ‘i486’, ‘i586’, ‘i686’, ‘noarch’, ‘ppc64’, ‘ppc64le’, ‘s390x’, ‘src’, ‘x86_64’]

all known architectures

meta = [‘noarch’, ‘src’]

meta arches

libtaskotron.arch_utils.basearch(arch=None)

This returns the ‘base’ architecture identifier for a specified architecture (e.g. i386 for i[3-6]86), to be used by YUM etc.

Parameters:arch (str) – an architecture to be converted to a basearch. If None, then the arch of the current machine is used.
Returns:basearch, or arch if no basearch was found for it
Return type:str

buildbot_utils

Utility functions for dealing with Buildbot

libtaskotron.buildbot_utils.get_urls(jobid, masterurl, task_stepname)

Form url to the job and its logs based on given jobid.

Parameters:
  • jobid (str) – jobid from runner
  • masterurl (str) – taskotron master url
  • task_stepname (str) – taskotron master step name
Returns:

(job_url, log_url)

libtaskotron.buildbot_utils.parse_jobid(jobid)

Parse the incoming jobid which should either be ‘-1’ to indicate that dummy values should be used or in the form of <builder>/<buildid>.

If the passed in format is invalid, no exceptions will be raised but a warning message will be emitted to logs

Parameters:jobid (str) – jobid from runner
Returns:(builder, buildid)

check

Helper tools for managing check status, outcome, and output.

class libtaskotron.check.CheckDetail(item, report_type=None, outcome=None, note=”, output=None, keyvals=None, checkname=None, artifact=None)

Class encompassing everything related to the outcome of a check run. It contains convenience methods for tracking, evaluating and reporting check results.

For some checks, it’s trivial to parse its output at the end of its execution and evaluate the results. For some, however, it’s much easier to do this continuously during check execution. This is especially true for multi-item checks (a single check providing results for many items - builds, updates, etc). That’s when this class comes very handy (it can be used even for the simple use cases, of course).

Variables:
  • outcome_priority (tuple) – a tuple of outcome keywords sorted by priority from the least important to the most important
  • item (str) – a description the item being tested; for example a build NVR (vpnc-0.5-1.fc20), update ID (FEDORA-2014-3309) or a repository name (f20-updates)
  • report_type (str) – a definition of the type of the object being checked; for example a Koji build or a Bodhi update, The allowed values are attributes in ReportType. You don’t have to fill this in (or you can provide a custom string value), but the reporting directives react only to the known types (you can always find it in ResultsDB, though).
  • outcome (str) –

    a keyword specifying the final outcome of the check. Available outcome keywords:

    • PASSED - everything went well
    • INFO - everything went well, but there is some important information that needs to be pointed out
    • FAILED - the item in question fails the check
    • NEEDS_INSPECTION - the outcome can’t be determined and a manual inspection is needed
    • ABORTED - the check aborted itself because of some unexpected problems, i.e. a necessary network server is not reachable. Running this check with the same arguments later can help to mitigate this problem.
    • CRASHED - the check crashed and did not provide usable results

    If no outcome has been set, this attribute returns NEEDS_INSPECTION.

    Raises TaskotronValueError if you try to assign an unknown keyword.

  • note (str) – a few words or one-sentence note about the result of the check run (e.g. 5 WARNINGS, 1 ERROR for an rpmlint result). Should not unnecessarily duplicate item or outcome, if possible.
  • output (list) – output from the check run (a list of strings). You can easily populate this by using store(), or you can modify it directly (for example filter out some messages you don’t want to see in the check output).
  • keyvals (dict) – all key-value pairs in this dictionary are stored in ResultsDB as ‘extra data’. This can be used to store e.g. a compose id or a kernel version, if that information is important for querying the results. Keep it as short as possible.
  • checkname (str) – name of the check, which the CheckDetail belongs to. This is usually not needed, as the check name is devised from the task metadata, but if a single task produces results for multiple checks, this is the way to override the default behavior.
  • artifact (str) – path to a file or directory placed in the artifacts directory, either absolute $artifactsdir/report.html or relative report.html. It will represent task output specific for this particular item.
broken()

Tells whether the check outcome is set to one of the broken states (i.e. ABORTED or CRASHED).

classmethod cmp_outcome(outcome1, outcome2)

Compare two outcomes according to outcome_priority and return -1/0/1 if outcome1 has lower/equal/higher priority than outcome2.

Parameters:
  • outcome1 (str) – an outcome keyword to compare
  • outcome2 (str) – an outcome keyword to compare
Raises:

TaskotronValueError – if any of the outcome keywords are not specified in outcome_priority

classmethod create_multi_item_summary(outcomes)

Create a string containing a sum of all outcomes, like this: 3 PASSED, 1 INFO, 2 FAILED

Parameters:outcomes – either one CheckDetail instance or an iterable of CheckDetails or an iterable of outcome strings
store(message, printout=True)

Add a string to the output, and print it optionally as well.

This is just a convenience method for most common use case, you can of course always access and modify output directly, and print or use logging facilities directly as well. This combines both into a single call.

Parameters:
  • message (str) – a string to store in output
  • printout (bool) – whether to print to standard output or not
update_outcome(outcome)

Update outcome with the provided value only and only if it has a higher priority than its current value (according to outcome_priority). Otherwise this call does nothing.

This is useful if your check performs a number of ‘sub-tasks’, each passing or failing, and you want to final outcome to be the worst/most serious one of those. You can just keep calling update_outcome() and the highest priority outcome type will be stored at outcome at the end.

Parameters:outcome (str) – the outcome value to assign to outcome if it has a higher priority than its current value. Handles None values gracefully (no action).
Raises:TaskotronValueError – if any of the outcome keywords are not specified in outcome_priority
libtaskotron.check.RESERVED_KEYS = (‘item’, ‘type’, ‘outcome’, ‘note’, ‘results’, ‘checkname’, ‘artifact’, ‘_internal’)

a list of reserved keywords for ResultYAML output, which can’t be overridden in keyvals

class libtaskotron.check.ReportType

Enum for different types of CheckDetail.report_type

BODHI_UPDATE = ‘bodhi_update’
COMPOSE = ‘compose’
DIST_GIT_COMMIT = ‘dist_git_commit’
DOCKER_IMAGE = ‘docker_image’
GIT_COMMIT = ‘git_commit’
KOJI_BUILD = ‘koji_build’
KOJI_TAG = ‘koji_tag’
MODULE_BUILD = ‘module_build’
libtaskotron.check.export_YAML(check_details)

Generate YAML output used for reporting to ResultsDB.

Note: You need to provide all your CheckDetails in a single pass in order to generate a valid YAML output. You can’t call this method several times and then simply join the outputs simply as strings.

Parameters:check_details – iterable of CheckDetail instances or single instance of CheckDetail
Returns:YAML output with results for every CheckDetail instance provided
Return type:str
Raises:TaskotronValueError – if CheckDetail.item or CheckDetail.outcome is empty for any parameter provided

Example output:

results:
  - item: xchat-0.5-1.fc20
    type: koji_build
    outcome: PASSED
    note: 5 ERRORS, 10 WARNINGS
    artifact: xchat-0.5-1.fc20.x86_64.log
libtaskotron.check.import_YAML(source)

Parses YAML and returns a list of CheckDetails.

Parameters:source (str) – YAML-formatted text
Raises:TaskotronValueError – if YAML syntax is incorrect

config

Global configuration for Taskotron and relevant helper methods.

libtaskotron.config.CONF_DIRS = [‘/home/src/taskbot_hub/libtaskotron/conf’, ‘/etc/taskotron’]

A list of directories where config files are stored. The config files are loaded from these locations in the specified order, and only the first config file found is used (further locations are ignored). The first location is dynamic, relative to the package location and it reflects the usual config location in a git checkout.

libtaskotron.config.CONF_FILE = ‘taskotron.yaml’

the name of our configuration file

libtaskotron.config.NS_CONF_FILE = ‘namespaces.yaml’

the name of result namespaces configuration file

libtaskotron.config.PROFILE_VAR = ‘TASKOTRON_PROFILE’

environment variable name for setting config profile

libtaskotron.config.get_config()

Get the Config instance. This method is implemented using the singleton pattern - you will always receive the same instance, which will get auto-initialized on the first method call.

Returns:either Config or its subclass, depending on taskotron profile used.
Raises:TaskotronConfigError – if config file parsing and handling failed
libtaskotron.config.load_namespaces_config()

Load and parse namespaces config file into a dictionary.

Returns:dictionary constructed from the yaml document
libtaskotron.config.parse_yaml_from_file(filename)

Parse given file in YAML format into a dictionary.

Parameters:filename (str) – a filename that yaml data are loaded from
Returns:dictionary constructed from the yaml document
Raises:TaskotronConfigError – when YAML parsing fails

config_defaults

This includes the default values for Taskotron configuration. This is automatically loaded by config.py and then overridden by values from config files available in system-wide location.

class libtaskotron.config_defaults.Config

Global configuration for Taskotron (development profile).

The documentation for individual options is available in the config files (unless they’re not present in the config files, then they’re documented here).

Implementation notes:

  • If you want to add a new option, put it here and optionally into the config file as well.
  • If you modify a default value for some option, don’t forget to modify it in both places - here and in the config file (if present).
  • Don’t assign None as a default value. We need to know a value type in order to check for correct type of user-provided values.
artifacts_baseurl = ‘http://localhost/artifacts’
artifactsdir = ‘/var/lib/taskotron/artifacts’
bodhi_staging = False
buildbot_task_step = ‘runtask’
cachedir = ‘/var/cache/taskotron’
client_taskdir = ‘/var/tmp/taskotron/taskdir’
config_filename = ”
default_disposable_arch = ‘x86_64’
default_disposable_distro = ‘fedora’
default_disposable_flavor = ‘taskotron_cloud’
default_disposable_release = ‘25’
download_cache_enabled = True
execdb_server = ‘http://localhost:5003’
force_imageurl = True
imagesdir = ‘/var/lib/taskotron/images’
imageurl = ‘http://download.fedoraproject.org/pub/fedora/linux/releases/25/CloudImages/x86_64/images/Fedora-Cloud-Base-25-1.3.x86_64.qcow2’
koji_url = ‘https://koji.fedoraproject.org/kojihub’
log_file_enabled = False
log_level_file = ‘DEBUG’
log_level_stream = ‘INFO’
logdir = ‘/var/log/taskotron’
minion_repos = [‘https://copr.fedorainfracloud.org/coprs/tflink/taskotron/repo/fedora-23/tflink-taskotron-fedora-23.repo’]
pkg_url = ‘https://kojipkgs.fedoraproject.org/packages’
profile = ‘development’
report_to_resultsdb = False
resultsdb_frontend = ‘http://localhost:5002’
resultsdb_server = ‘http://localhost:5001/api/v2.0’
runtask_mode = ‘local’
ssh_privkey = ”
supported_arches = [‘x86_64’, ‘i386’, ‘armhfp’]
taskotron_master = ‘http://localhost/taskmaster’
tmpdir = ‘/var/tmp/taskotron’
class libtaskotron.config_defaults.ProductionConfig

Configuration for production profile. Inherits values from Config and overrides some. Read Config documentation.

download_cache_enabled = False
log_file_enabled = True
log_level_file = ‘DEBUG’
log_level_stream = ‘INFO’
profile = ‘production’
report_to_resultsdb = True
runtask_mode = ‘libvirt’
class libtaskotron.config_defaults.ProfileName

Enum of available profile names. These can be specified in the config file or as the environment variable.

DEVELOPMENT = ‘development’
PRODUCTION = ‘production’
TESTING = ‘testing’
class libtaskotron.config_defaults.RuntaskModeName

Enum of available runtask mode names. These can be specified in the config file.

LIBVIRT = ‘libvirt’
LOCAL = ‘local’
class libtaskotron.config_defaults.TestingConfig

Configuration for testing suite profile. Inherits values from Config and overrides some. Read Config documentation.

artifactsdir = ‘/var/tmp/taskotron-test/artifacts’
cachedir = ‘/var/tmp/taskotron-test/cache’
log_level_file = ‘DEBUG’
log_level_stream = ‘DEBUG’
logdir = ‘/var/tmp/taskotron-test/log’
profile = ‘testing’
tmpdir = ‘/var/tmp/taskotron-test/tmp’

exceptions

This module contains custom Taskotron exceptions

exception libtaskotron.exceptions.TaskotronConfigError

All errors related to Taskotron config files

exception libtaskotron.exceptions.TaskotronDirectiveError

All errors related to Taskotron directives

exception libtaskotron.exceptions.TaskotronError

Common ancestor for Taskotron related exceptions

exception libtaskotron.exceptions.TaskotronImageError

All generic image related errors

exception libtaskotron.exceptions.TaskotronImageNotFoundError

Requested image not found error

exception libtaskotron.exceptions.TaskotronImportError(e)

All issues with Extensions

exception libtaskotron.exceptions.TaskotronNotImplementedError

NotImplementedError for Taskotron classes, methods and functions

exception libtaskotron.exceptions.TaskotronPermissionError

Insufficient permissions or privileges

exception libtaskotron.exceptions.TaskotronRemoteError

All network and remote-server related errors

exception libtaskotron.exceptions.TaskotronRemoteProcessError

A process/command executed remotely returned a non-zero exit code, crashed or failed in a different way

exception libtaskotron.exceptions.TaskotronRemoteTimeoutError

A remote-server did not send any output in a given timeout

exception libtaskotron.exceptions.TaskotronValueError

Taskotron-specific ValueError

exception libtaskotron.exceptions.TaskotronYamlError

Error in YAML config file of the executed check

executor

class libtaskotron.executor.Executor(formula, arg_data, workdir=None)

This class serves the purpose of actual task execution. It is instantiated by Overlord, PersistentMinion or DisposableMinion.

Variables:
  • default_namespace (str) – namespace to be used when the formula doesn’t contain a namespace definition
  • formula (dict) – parsed task formula
  • arg_data (dict) – processed cli arguments with some extra runtime variables
  • workdir (str) – path to working directory; a temporary one is created during execution if None
  • working_data (dict) – variables exported from formula directives, to be injected when referenced using ${variable}
  • exitcode (int) – exit code set by exitcode directive; it is None if exitcode directive was not used
execute()

Execute the task (consists of preparing the task and running it).

file_utils

class libtaskotron.file_utils.Tee(*files)

Helper class for writing data to different streams.

libtaskotron.file_utils.download(url, dirname, filename=None, cachedir=None)

Download a file.

Parameters:
  • url (str) – file URL to download
  • dirname (str) – directory path; if the directory does not exist, it gets created (and all its parent directories).
  • filename (str) – name of downloaded file; if not provided, the basename is extracted from URL
  • cachedir (str) – If set, the file will be downloaded to a cache directory specified by this parameter. If the file is already present and of the same length, download is skipped. The requested destination file (dirname/filename) will be a symlink to the cached file. This directory is automatically created if not present.
Returns:

the path to the downloaded file

Return type:

str

Raises:

TaskotronRemoteError – if download fails

libtaskotron.file_utils.makedirs(fullpath)

This is the same as os.makedirs(), but does not raise an exception when the destination directory already exists.

Raises:OSError – if directory doesn’t exist and can’t be created

logger

Configure logging in libtaskotron.

There are two modes how to operate - as an external library, or as the main script runner:

  • In the external library mode, we try not to change any global defaults, not touch the root logger, and not attach any handlers. The main script author should be in control of all these things.
  • In the main script runner mode, we control everything - we configure the root logger, attach handlers to it, and set verbosity of this and any other library as we see fit.
libtaskotron.logger.add_filehandler(level_file=None, filelog_path=None, remove_mem_handler=False)

Add a file handler.

Parameters:
  • level_file (int) – level of file logging as defined in logging. If None, a default level from config file is used.
  • filelog_path (str) – path to the log file. If None, the value is loaded from config file.
  • remove_mem_handler (bool) – after adding last file handler, the memory handler is no longer needed and can be removed so its buffer is not filling with messages
Returns:

created file handler

Raises:

IOError – if log file can’t be opened for writing

libtaskotron.logger.init(level_stream=None, stream=True, syslog=False)

Initialize Taskotron logging.

Note: Since this touches the root logger, it should be called only when Taskotron is run as the main program (through its runner), not when it is used as a library.

Parameters:
  • level_stream (int) – level of stream logging as defined in logging. If None, a default level from config file is used.
  • stream (bool) – enable logging to process stream (stderr)
  • syslog (bool) – enable logging to syslog
libtaskotron.logger.init_prior_config(level_stream=None)

Initialize Taskotron logging with default values which do not rely on a config file. Only stream logging is enabled here. This is used before the config file is loaded. After that a proper initialization should take place through the init() method.

Note: Since this touches the root logger, it should be called only when Taskotron is run as the main program (through its runner), not when it is used as a library.

Parameters:level_stream (int) – message level of the stream logger. The level definitions are in logging. If None, the default level is used (i.e. logging.NOTSET).
libtaskotron.logger.log = <logging.Logger object>

the main logger for libtaskotron library, easily accessible from all our modules

libtaskotron.logger.mem_handler = None

our current memory handler sending logged messages to memory log prior to creating file log (after its creation, content of the memory log is flushed into the file log)

libtaskotron.logger.stream_handler = None

our current stream handler sending logged messages to stderr

libtaskotron.logger.syslog_handler = None

our current syslog handler sending logged messages to syslog

main

Main class used by the runtask runner

Below is a list of examples of item input values for different item types:

  • bodhi_update: Bodhi update ID,
    e.g. FEDORA-2017-04459ef8cf
  • compose: Full URL to the compose or image,
    e.g. http://server/Fedora-Atomic-25-20170512.0.x86_64.qcow2
  • dist_git_commit: Triplet of namespace/repo#commit from distgit,
    e.g. rpms/gcc#fe7fce2ad1cbf374be1a1b99af27eff733f639a0 or modules/dhcp#5c6331592b6c92bf7c52a2ac735f92cb8fc11304
  • docker_image: Docker registry URL,
    e.g. candidate-registry.fedoraproject.org/f26/mongodb:0-1.f26docker
  • git_commit: Quadruplet of server/repo#branch#commit,
    e.g. https://pagure.io/rpmdeplint#refs/heads/master#080d9599f01978ad554658c6a5da1642a383b969
  • koji_build: Build NVR from Koji (without epoch, even if it exists),
    e.g. cups-2.2.0-9.fc25
  • koji_tag: Koji tag name,
    e.g. f25-updates-testing-pending
  • module_build: Triplet of modulename-stream-version,
    e.g. nodejs-f26-20170511113257
libtaskotron.main.check_args(parser, args)

Check if passed args doesn’t have conflicts and have proper format. In case of error, this function prints error message and exits the program.

Parameters:
  • parser (argparse.ArgumentParser) – parser object used to show error message and exit the program
  • args (dict) – arguments previously returned by argument parser converted to dict
libtaskotron.main.get_argparser()

Get the cmdline parser for the main runner.

Return type:argparse.ArgumentParser
libtaskotron.main.main()

Main entry point executed by runtask script

libtaskotron.main.process_args(raw_args)

Processes raw input args and converts them into specific data types that can be used by tasks. This includes e.g. creating new args based on (item, item_type) pairs, or adjusting selected architecture.

Parameters:raw_args (dict) – dictionary of raw arguments. Will not be modified.
Returns:dict of args with appended/modified data

minion

class libtaskotron.minion.BaseMinion(formula, arg_data)

Base Minion class that shouldn’t be used on its own, it solely initiates inner attributes and environment. It also provides method _run() that delegates the task execution to Executor over SSH.

Variables:
  • arg_data_exclude (tuple) – a tuple of cmdline options/arguments which should not be forwarded from the host to the minion when execution the task. Use the same names as the stored variables in arg_data use (i.e. dashes converted to underscores, etc).
  • formula (dict) – parsed task formula
  • arg_data (dict) – processed cli arguments with some extra runtime variables
  • artifactsdir (str) – path to artifactsdir for storing logs
  • exitcode (int) – exit code of libtaskotron runner executed on the minion
  • ssh – an instance of ParamikoWrapper which is used to execute all remote commands on a minion. None by default, you need to set this before you start execute().
execute()

This method has to be implemented by classes that inherits from BaseMinion. It should contain environment initialization (namely set ssh to properly instantiated ParamikoWrapper) and cleanup (if necessary). Moreover, the order of private methods has to be: _prepare_task(), _run() and _get_output().

class libtaskotron.minion.DisposableMinion(formula, arg_data)

Minion class that creates a disposable client, connects to it over SSH and runs the task there.

execute()

Init environment and connect to disposable client for task execution.

class libtaskotron.minion.PersistentMinion(formula, arg_data)

Minion class that connects to alredy running machine using SSH and runs the task there.

execute()

Init environment and connect to remote machine for task execution.

os_utils

Utility methods related to an operating system

libtaskotron.os_utils.has_sudo()

Determine whether we currently have a password-less access to sudo.

Note: It’s not possible to say whether the access will stay password-less in the future (the credentials might be set to expire in time), just for this exact moment.

Return type:bool
libtaskotron.os_utils.is_root()

Determine whether we’re currently running under the root account.

Return type:bool
libtaskotron.os_utils.popen_rt(cmd, stderr=-2, bufsize=1, **kwargs)

This is similar to subprocess.check_output(), but with real-time printing to console as well. It is useful for longer-running tasks for which you’d like to both capture the output and also see it printed in terminal continuously.

Please note that by default both stdout and stderr are merged together. You can use stderr=subprocess.PIPE, and it will be returned to you separated, but it won’t be printed out to console (only stdout will).

The parameters are the same as for subprocess.Popen. You can’t use stdout parameter, that one is hardcoded to subprocess.PIPE.

Returns:

tuple of (stdoutdata, stderrdata). stderrdata will be None by default (because stderr is redirected to stdout).

Raises:
  • TaskotronValueError – if you provide stdout parameter
  • subprocess.CalledProcessError – if the command exits with non-zero exit code (helpful attributes are provided, study its documentation)

overlord

class libtaskotron.overlord.Overlord(arg_data)

Overlord class encapsulates decision whether the task is run locally or remotely (in persistent or disposable client) and orchestrates the execution.

Variables:
  • arg_data (dict) – processed cli arguments with some extra runtime variables
  • exitcode (int) – exit code of the task; if None, it means start() was not called yet
  • formula (dict) – parsed task formula
start()

Start the overlord, get runner and execute the task (either locally or remotely).

python_utils

A collection of convenience methods related to Python base libraries.

libtaskotron.python_utils.iterable(obj, item_type=None)

Decide whether obj is an Iterable (you can traverse through its elements - e.g. list, tuple, set, even dict), but not basestring (which satisfies most collections’ requirements, but we don’t often want to consider as a collection). You can also verify the types of items in the collection.

Parameters:
  • obj (any) – any object you want to check
  • item_type (type) – all items in obj must be instances of this provided type. If None, no check is performed.
Returns:

whether obj is iterable but not a string, and whether obj contains only item_type items

Return type:

bool

Raises:

TaskotronValueError – if you provide invalid parameter value

libtaskotron.python_utils.reverse_argparse(args, ignore=())

Take cmdline arguments parsed by argparse and revert it back to a command line.

Example input: {item: 'foo', debug: True, arch: ['x86_64', 'i386'], job_id: 1234, t: 'koji'}
Example output: ['--item', 'foo', '--debug', '--arch', 'x86_64', '--arch', 'i386', '--job-id', '1234', '-t', 'koji']

This is a very naive implementation and has several limitations:

  • You should put positional arg names into the ignore list. Handling those is not implemented because we don’t need them.
  • All options need to use action='store', action='store_true' or action='append'. Nothing else is supported. If you use some other actions, you need to put those options into the ignore list.
  • All multi-word options names will be assumed to be using dashes as a separator. Be sure you do not use underscores, or convert it accordingly yourself.
Parameters:
  • args (dict) – arguments parsed from argparse converted to a dict. This is what you get by running vars(parser.parse_args()).
  • ignore (list of str) – list of option names which should be ignored. If you use any positional arguments, you need to include their variables names here as well. The ignore list is checked before the underscore-to-dash conversion takes place.
Returns:

reversed command line as a list of strings. Please note the returned strings are not shell-escaped. Be sure to do that if you run it in a shell mode.

Return type:

list of str

libtaskotron.python_utils.sequence(obj, item_type=None, mutable=False)

This has the same functionality and basic arguments as iterable() (read its documentation), but decides whether obj is a Sequence (ordered and indexable collection - e.g. list or tuple, but not set or dict).

Parameters:mutable (bool) – if True, the obj must be a mutable sequence (e.g. list, but not tuple)

remote_exec

Tools for remote execution primary for disposable clients

class libtaskotron.remote_exec.ParamikoWrapper(hostname, port, username, key_filename, stdio_filename=None)

Wrapper for SSH communication using paramiko library

TIMEOUT = 900

timeout for network operations in seconds (900 seconds = 15 minutes)

close()

Close open connections and files.

cmd(cmd, debug=True)

Execute a command.

Parameters:
  • cmd (str) – A command to be executed. Make sure you escape it properly to prevent shell expansion, in case it is not desired.
  • debug (bool) – Whether print out debugging log messages about what’s happening.
Returns:

returncode of the command

Raises:
connect()

Connect to a machine over ssh. Open sftp channel and, if applicable, file that stdout/err from the machine will be saved to.

Raises:TaskotronRemoteError – when the connection does not succeed
get_dir(remote_path, local_path, debug=True)

Get a directory from a remote path. File permissions are not preserved.

Parameters:
  • remote_path (str) – A path to the remote directory
  • local_path (str) – A path to the local directory. This directory will be created, if its parent exists. Otherwise you need to create the full tree structure manually beforehand.
  • debug (bool) – Whether print out debugging log messages about what’s happening.
Raises:

TaskotronRemoteError – If the directory could not be downloaded

get_file(remote_path, local_path, debug=True)

Get a file from a remote path. File permissions are not preserved.

Parameters:
  • remote_path (str) – A path to the remote file
  • local_path (str) – A path to the local file
  • debug (bool) – Whether print out debugging log messages about what’s happening.
Raises:

TaskotronRemoteError – If the file could not be downloaded

install_pkgs(pkgs)

Install packages via dnf.

First tries to install packages using dnf --best, which ensures updating to the latest version. If this doesn’t work, we drop --best to allow for broken deps of latest packages, in case there are older packages with working deps.

Parameters:pkgs (list) – A list of packages to be installed (supports any argument that dnf install accepts)
Raises:TaskotronRemoteError – If the command has non-zero return code or times out (see cmd()).
put_dir(local_path, remote_path, overwrite=True, debug=True)

Copy a directory to a remote path. This method creates a tarball from contents of local_path, copies the tarball to remote machine and extracts it to remote_path. File permissions and symlinks are preserved.

Parameters:
  • remote_path (str) – A path to the remote directory. This directory will be created, if its parent exists. Otherwise you need to create the full tree structure manually beforehand.
  • local_path (str) – A path to the local directory
  • overwrite (bool) – Whether to overwrite remote path (merge local dir with the remote dir). If you choose to not overwrite and remote_path exists, this method with just immediately return.
  • debug (bool) – Whether print out debugging log messages about what’s happening.
Raises:

TaskotronRemoteError – If the directory could not be copied

put_file(local_path, remote_path, overwrite=True, debug=True)

Copy a file to a remote path. File permissions are not preserved.

Parameters:
  • local_path (str) – A path to the local file
  • remote_path (str) – A path to the remote file
  • overwrite (bool) – Whether to overwrite remote path. Default is True.
  • debug (bool) – Whether print out debugging log messages about what’s happening.
Returns:

paramiko.SFTPAttributes object containing attributes about the given file, if successful. None otherwise.

Raises:

TaskotronRemoteError – If the file could not be copied

write_file(remote_path, data, overwrite=True, debug=True)

Write data to a remote file.

Parameters:
  • remote_path (str) – A path to the remote file
  • data (str) – Data to be written
  • overwrite (bool) – Whether to overwrite remote path. Default is True.
  • debug (bool) – Whether print out debugging log messages about what’s happening.
Raises:

TaskotronRemoteError – If data could not be written

taskformula

Methods for operating with a task formula

class libtaskotron.taskformula.DotTemplate(template)

Redefines the string.Template’s idpattern property to allow using . in the variable names. This is then later used for key/property access in variable replacement.

libtaskotron.taskformula.devise_environment(formula, arg_data)

Takes a parsed formula, and returns a required run-environment (i.e. distro, arch, fedora release, and base-image flavor), based on item and type.

Parameters:
  • formula (dict) – parsed formula file (or dict with equivalent structure)
  • arg_data (dict) – parsed command-line arguments. item, type and arch are used in this method
Returns:

dict containing release, flavor, arch. Each either set, or None

Raises:

TaskotronValueError – when environment can’t be parsed from the formula

libtaskotron.taskformula.replace_vars_in_action(action, variables)

Replace variables (${var} or ${var.key}) with their values in an action (YAML dictionary).

Only basestring values are searched for variables syntax, which means variables can either be inside leaves (in a tree sense, meaning they can’t be traversed further and are of a primitive type), or inside key names in a dictionary.

Parameters:
  • action (dict) – An action specification parsed from the task formula. See Runner.do_actions() to see what an action looks like.
  • variables (dict) – names (keys) and values of variables to replace
Raises:

TaskotronYamlError – if text contains a variable that is not present in variables; or if the variable syntax is incorrect; of if you try to replace a dictionary key with something else than a string

Disposable minion modules

vm

Interface to locally spawned virtual machines that are used as disposable clients for executing Taskotron tasks.

class libtaskotron.ext.disposable.vm.ImageFinder

Retrieve images from either local or remote sources

classmethod get_all_filenames(imagesdir=None)

Get list of images present on the system.

Parameters:imagesdir – absolute path to directory containing the images, path from config is used if None
classmethod get_latest(distro, release, flavor, arch=’x86_64’, imagesdir=None)

Search for the most recent image available on the system.

Parameters:
  • distro – distro of the image (e.g. fedora)
  • release – release of the image (e.g. 23)
  • flavor – flavor of the image (e.g. minimal)
  • imagesdir – absolute path to directory containing the images, path from config is used if None
  • arch – architecture of the image
Returns:

file:// url of the latest image available

Raises:

TaskotronImageError – if no such image for given release and flavor was found

classmethod get_latest_metadata(distro, release, flavor, arch=’x86_64’, imagesdir=None)

Search for the most recent image available on the system.

Parameters:
  • distro – distro of the image (e.g. fedora)
  • release – release of the image (e.g. 23)
  • flavor – flavor of the image (e.g. minimal)
  • imagesdir – absolute path to directory containing the images, path from config is used if None
  • arch – arch of the image (e.g. ‘x86_64’)
Returns:

metadata of the most recent image

Return type:

dict {‘date’: str, ‘version’: str, ‘release’: str, ‘arch’: str, ‘filename’: str}

class libtaskotron.ext.disposable.vm.TestCloudMachine(uuid)

Launch virtual machines with TestCloud and prepare them for executing tasks (install packages etc.)

Parameters:uuid – unicode string uuid for the task being executed
hostname = None

hostname to use for spawned instance - based on username of current user

instancename = None

name of the testcloud instance spawned for this task

ipaddr = None

ip address of the remote machine (only set after prepared and spawned successfully)

password = None

password for user on the remote machine

prepare(distro=None, release=None, flavor=None, arch=None)

Prepare a virtual machine for running tasks. :param str distro: Distro to use in image discovery :param str release: Distro’s release to use in image discovery :param str flavor: base-image flavor to use in image discovery :param str arch: arch to use in image discovery

Raises:TaskotronRemoteError – if there are any errors while preparing the virtual machine.
teardown()

Tear down the virtual machine by stopping it and removing it from the host machine.

Raises:TaskotronRemoteError – if there is a failure while stopping or removing the virtual machine instance
username = None

username to use when connecting to the virtual machine

uuid = None

uuid of the task which spawned this instance

wait_for_port(port, timeout=60)

Wait until port is open. Repeatedly tries to socket.connect on given port.

Parameters:
  • port – port to check
  • timeout – timeout in seconds
Raises:

TaskotronInstanceError – when timeouted

Fedora modules

bodhi_utils

Utility functions for dealing with Bodhi

class libtaskotron.ext.fedora.bodhi_utils.BodhiUtils(client=None)

Helper Bodhi methods.

Variables:client (fedora.client.Bodhi2Client) – Bodhi2 client instance

Create a new BodhiUtils instance.

Parameters:client – custom Bodhi2Client instance. If None, a default Bodhi2Client instance is used.
build2update(builds, strict=False)

Find matching Bodhi updates for provided builds.

Parameters:
  • builds (iterable of str) – builds to search for in N(E)VR format (foo-1.2-3.fc20 or foo-4:1.2-3.fc20)
  • strict (bool) – if False, incomplete Bodhi updates are allowed. If True, every Bodhi update will be compared with the set of provided builds. If there is an Bodhi update which contains builds not provided in builds, that update is marked as incomplete and removed from the result - i.e. all builds from builds that were part of this incomplete update are placed in the second dictionary of the result tuple.
Returns:

a tuple of two dictionaries:

  • The first dict provides mapping between builds and their updates where no error occured.

    {build (string): Bodhi update (Munch)}

  • The second dict provides mapping between builds and their updates where some error occured. The value is None if the matching Bodhi update could not be found (the only possible cause of failure if strict=False). Or the value is a Bodhi update that was incomplete (happens only if strict=True).

    {build (string): Bodhi update (Munch) or None}

  • The set of keys in both these dictionaries correspond exactly to builds. For every build provided in builds you’ll get an answer in either the first or the second dictionary, and there will be no extra builds that you haven’t specified.

Raises:

TaskotronValueError – if builds type is incorrect

get_update(updateid)

Get the last Bodhi update for the specified update ID.

Parameters:updateid (str) – update ID, e.g. ‘FEDORA-2015-13787’
Returns:Bodhi update object with that ID, or None when no such update is found
Return type:munch.Munch

koji_utils

Utility methods related to Koji

class libtaskotron.ext.fedora.koji_utils.KojiClient(koji_session=None)

Helper Koji methods.

Variables:session (koji.ClientSession) – Koji client session

Create a new KojiClient

Parameters:koji_session (koji.ClientSession) – an existing Koji session instance or None if you want a new default session to be created
get_build_log(nvr, dest, arches=[‘all’], arch_exclude=[])

Download a build.log file for NVR for each specified architecture and save it in dest as build.log.<arch>.

Note: Koji cleans up old log files regularly, i.e. they might not be available for a certain NVR. This method will not raise an error if that happens (the log file can’t be downloaded). Instead, it will be simply missing in that directory and also marked in the returned object.

For parameters undocumented here, see nvr_to_urls().

Parameters:
  • nvr (str) – build NVR
  • dest (str) – location where to store the log file
Returns:

a dictionary of this structure:

{‘ok’: [‘/path/build.log.x86_64’, ‘/path/build.log.i686’],
 ‘error’: [‘armv7hl’]
}

which describes which build logs were correctly downloaded and for which architectures we failed to download the logs.

Return type:

dict

Raises:

TaskotronRemoteError – when the requested build doesn’t exist or its info can’t be retrieved

get_nvr_rpms(nvr, dest, arches=[‘all’], arch_exclude=[], debuginfo=False, src=False)

Retrieve the RPMs associated with a build NVR into the specified directory.

For parameters undocumented here, see nvr_to_urls().

Parameters:
  • nvr (str) – build NVR
  • dest (str) – location where to store the RPMs
Returns:

list of local filenames of the grabbed RPMs (might be empty, according to your option choices and the particular NVR)

Return type:

list of str

Raises:

TaskotronRemoteError – if the files can’t be downloaded

get_tagged_rpms(tag, dest, arches=[‘all’], arch_exclude=[], debuginfo=False, src=False)

Downloads all RPMs of all NVRs tagged by a specific Koji tag.

Note: This works basically the same as get_nvr_rpms(), it just downloads a lot of builds instead of a single one. For description of all shared parameters and return values, please see that method.

Parameters:tag (str) – Koji tag to be queried for available builds, e.g. f20-updates-pending
latest_by_tag(tags, pkgname)

Get the latest Koji build for the given package name in the given tag(s).

Parameters:
  • tags (list) – list of tags to be searched in
  • pkgname (str) – name of the package
Returns:

str containing NVR of the latest build, None if no build was found

nvr_to_urls(nvr, arches=[‘all’], arch_exclude=[], debuginfo=False, src=True)

Get list of URLs for RPMs corresponding to a build.

Parameters:
  • nvr (str) – build NVR
  • arches (list of str) –

    restrict the arches of builds to provide URLs for. By default, all Taskotron-supported architectures are considered. If you want to consider just some selected arches, provide their names in a list.

    Note

    If you specify base arches (like i386), all concrete binary arches for that base arch will be automatically added (e.g. i[3-6]86), because Koji query requires concrete binary arches.

  • arch_exclude (list of str) – exclude some specific arches, even if they are specified in arches or they are implicit in the default 'all' value
  • debuginfo (bool) – whether to provide URLs for debuginfo RPM files or ignore them
  • src (bool) – whether to include a URL for the source RPM
Return type:

list of str

Raises:

TaskotronRemoteError – when the requested build doesn’t exist

rpms_to_build(rpms)

Get list of koji build objects for the rpms. Order of koji objects in this list is the same as order of the respective rpm objects.

Parameters:rpms (list of str) – list of filenames as either /my/path/nvr.a.rpm or nvr.a.rpm
Returns:list of Koji buildinfo dictionaries (as returned e.g. from koji.getBuild()) in the same respective order as in“rpms“
Return type:list
Raises:TaskotronRemoteError – if rpm or it’s related build is not found
libtaskotron.ext.fedora.koji_utils.getNEVR(build)

Extract RPM version identifier in NEVR format from Koji build object

Parameters:build (dict) – Koji buildinfo dictionary (as returned e.g. from koji.getBuild())
Returns:NEVR string; epoch is included when non-zero, otherwise omitted
Raises:TaskotronValueError – if build is of incorrect type

rpm_utils

Utility methods related to RPM

libtaskotron.ext.fedora.rpm_utils.cmpNEVR(nevr1, nevr2)

Compare two RPM version identifiers in NEVR format.

Parameters:
  • nevr1 (str) – RPM identifier in N(E)VR format
  • nevr2 (str) – RPM identifier in N(E)VR format
Returns:

-1/0/1 if nevr1 < nevr2 / nevr1 == nevr2 / nevr1 > nevr2

Return type:

int

Raises:

TaskotronValueError – if name in nevr1 doesn’t match name in nevr2

libtaskotron.ext.fedora.rpm_utils.get_dist_tag(rpmstr)

Parse disttag from an RPM package version string.

Parameters:rpmstr (str) – string to be manipulated in a format of N(E)VR (foo-1.2-3.fc20 or bar-4:1.2-3.fc20) or N(E)VRA (foo-1.2-3.fc20.x86_64 or bar-4:1.2-3.fc20.i686)
Returns:string containing dist tag (fc20)
Raises:TaskotronValueError – if rpmstr does not contain dist tag
libtaskotron.ext.fedora.rpm_utils.install(pkgs)

Install packages from system repositories using DNF. Either root or sudo access required.

Parameters:

pkgs (list of str) – packages to be installed, e.g. ['pidgin'], or any other argument supported by dnf install command

Raises:
libtaskotron.ext.fedora.rpm_utils.rpmformat(rpmstr, fmt=’nvr’, end_arch=False)

Parse and convert an RPM package version string into a different format. String identifiers: N - name, E - epoch, V - version, R - release, A - architecture.

Parameters:
  • rpmstr (str) – string to be manipulated in a format of N(E)VR (foo-1.2-3.fc20 or bar-4:1.2-3.fc20) or N(E)VRA (foo-1.2-3.fc20.x86_64 or bar-4:1.2-3.fc20.i686)
  • fmt (str) – desired format of the string to be returned. Allowed options are: nvr, nevr, nvra, nevra, n, e, v, r, a. If arch is not present in rpmstr but requested in fmt, noarch is used. Epoch is provided only when specifically requested (e.g. fmt='nevr') and being non-zero; otherwise it’s supressed (the only exception is fmt='e', where you receive 0 for zero epoch).
  • end_arch (bool) – set this to True if rpmstr ends with an architecture identifier (foo-1.2-3.fc20.x86_64). It’s not possible to reliably distinguish that case automatically.
Returns:

string based on the specified format, or integer if fmt='e'

Raises:

TaskotronValueError – if fmt value is not supported

yumrepoinfo

A wrapper object for yumrepoinfo.conf to access its information easily

class libtaskotron.ext.fedora.yumrepoinfo.YumRepoInfo(arch=None, filelist=None)

This class is a wrapper for easily accessing repoinfo.conf file.

Parameters:
  • arch (str) – architecture for which to adjust repo URLs. By default it refers to the architecture of the current machine. It’s always converted to basearch.
  • filelist (iterable of str) – list of config files to read information from. The first available config file is used. If None, then the default list of locations is used.
Raises:

TaskotronConfigError – if no YUM repositories data is found (empty or non-existent config file). It’s not raised if you specifically request no data to load (filelist=[]).

arches(reponame)

Get a list of all supported (primary and alternate) architectures for a repo

Parameters:reponame (str) – repository name
branched()

Get branched Fedora release (or None if it doesn’t exist).

Return type:str or None
get(reponame, key)

Get a specific key value from a repo

Parameters:
  • reponame (str) – repository name
  • key (str) – name of the key you want to retrieve from a repository (section)
Raises:

TaskotronConfigError – if the key can’t be retrieved (e.g. wrong key or repo name)

release_status(reponame)

Return release status of specified repo. For non-top-parent repos, return release_status of top parent repo.

Parameters:reponame (str) – repository name
Returns:release status of specified repo, lowercased. One of: rawhide, branched, stable, obsolete.
Return type:str
releases()

Get the list of stable (supported) Fedora releases.

Return type:list of str
repo(reponame)

Given a repo name, return the yumrepoinfo dict with keys: arches, parents, tag, url, path and name

Parameters:reponame (str) – repository name
Return type:dict
repo_by_tag(tag)

Given a Koji tag, return the corresponding repo dict.

Parameters:tag (str) – a koji tag, e.g. f20-updates. Note: rawhide is not used as a Koji tag, use number identifier instead or use repo('rawhide')['tag'] to discover it first.
Returns:repo dict as from repo(), or None if no such Koji tag is found
repos()

Get the list of all known repository names.

Return type:list of str
top_parent(reponame)

Go through the repo hiearchy and find the top parent for a repo

Parameters:reponame (str) – repository name
Returns:the top parent reponame. If reponame doesn’t have any parent, its name is returned (it’s its own top parent)
Return type:str
Raises:TaskotronConfigError – if infinite parent loop detected
libtaskotron.ext.fedora.yumrepoinfo.get_yumrepoinfo(arch=None, filelist=None)

Get YumRepoInfo instance. This method is implemented using the singleton pattern - you will always receive the same instance, which will get auto-initialized on the first method call.

Parameters:
  • arch (str) – architecture to return the YumRepoInfo for. It’s always converted to basearch. If None, then local machine arch is used.
  • filelist – list of config files to read information from. The first available config file is used. If None, then the default list of locations is used.
Returns:

shared YumRepoInfo instance

Raises:

TaskotronConfigError – if file config parsing and handling failed