Task Result Format

The required output format for tasks in Taskotron is YAML with particular values, which we call ResultYAML.

We recommend using libtaskotron Python library or the provided taskotron_result script to create the YAML output. Read Saving task results in ResultYAML format to see how. If you can not use it, then please adhere to the specification described below.

Minimal version

If you don’t need anything fancy, then this is the minimalistic version of the output which, at the same time, is valid for results reporting in Taskotron:

results:
  - item: xchat-0.5-1.fc20
    type: koji_build
    checkname: mytest
    outcome: PASSED

You just need to provide the item, type, checkname and outcome values in the YAML block.

The outcome field defines whether the check result was OK or not OK. Values allowed at the moment, are PASSED, INFO, FAILED, ERROR, WAIVED, and NEEDS_INSPECTION. Using other values will cause the ResultsDB reporting to fail.

type describes “what” was tested on a high level - whether it was a build from Koji, DVD image, update in Bodhi, or something completely different. You can see the supported list of items and their types in libtaskotron.main.

You can, of course, use any other value for the type parameter, but if you have a check for repositories, Koji builds or Bodhi updates, please stick to the already defined ones, so the tools next in the chain will recognize your results properly.

item is a string representation of the specific piece, that was tested - NVR for the koji_build type, update-name or update-id for the bodhi_update, etc. Once again - you can set whatever value you think is right, but sticking to the reasonable/obvious choices will ensure seamless processing.

checkname represents the test case identification. For development purposes, it doesn’t really matter what you use. Once you want to have your task run in production, the admins will suggest an appropriate checkname for you. You can provide test case subresults by using dot as a nesting separator.

Note

item, type, outcome and checkname are just some of the “reserved” key-names in the YAML block. See Note on reserved field names.

Full version

Although the minimal version of the output is sufficient, you can use additional fields to provide more information:

results:
  - item: tzdata-2014f-1.fc20
    type: bodhi_update
    outcome: ABORTED
    artifact: upgradepath_details.html
    checkname: dist.upgradepath
    arch: x86_64
    package: tzdata
  - item: bluez-5.34-1.fc23
    type: bodhi_update
    outcome: PASSED
    artifact: upgradepath_details.html
    checkname: dist.upgradepath
    arch: x86_64
    package: bluez
  - item: tzdata-2014f-1.fc20
    type: bodhi_update
    outcome: PASSED
    artifact: depcheck_tzdata-2014f-1.fc20.txt
    checkname: dist.depcheck
    arch: x86_64

Going top-down, the new fields are artifact, checkname, note, arch and package.

artifact contains the name of a file in the $artifactsdir, that contains detailed information about the check run. In this example the upgradepath check produced an user-friendly report in HTML format, and the author decided to link it as an artifact.

note adds a short valuable information that should be easily visible in the result overview.

arch and package are custom fields that do not have any special meaning for the Taskotron tooling, but provide additional information about the check results as such.

Note on reserved field names

The full list of fields with a pre-defined meaning is available at libtaskotron.check.RESERVED_KEYS.

Apart from those, you can add any number of custom key-value pairs (like the package in the above example), which will get stored in ResultsDB, and can be then used for searching. Those fields will be specific for your task and you can give them any meaning and use them for any purpose that you need.

Special cases

Empty results

There might be a case when you need to specify that your task finished fine, but there are no results that should be posted. A ResultYAML file that contains no results is a dictionary with results key and either no value:

results:

or an empty list value:

results: []