ReleaseEngineering/Applications/Tooltool

Tooltool basics

Some of the jobs in the releng infrastructure make use of generic binary artifacts, which are stored in dedicated artifacts repositories.

Tooltool is the client program, written in Python, which is used to retrieve these artifacts from the servers, and to perform integrity checks on them based on hashcode verification. Hence, the servers storing these artifacts are named "tooltool servers".

A set of files to be fetched by tooltool is specified via a tooltool manifest, usually a file in JSON format with tt extension.

The following is an example of a valid tooltool manifest:

 [
 {
 "size": 139308,
 "digest": "b2a463249bb3a9e7f2a3604697b000d2393db4f37b623fc099beb8456fbfdb332567013a3131ad138d8633cb19c50a8b77df3990d67500af896cada8b6f698b4",
 "algorithm": "sha512",
 "filename": "file2.pdf"
 },
 {
 "size": 3017536,
 "digest": "630d01a329c70aedb66ae7118d12ff7dc6fe06223d1c27b793e1bacc0ca84dd469ec1a6050184f8d9c35a0636546b0e2e5be08d9b51285e53eb1c9f959fef59d",
 "algorithm": "sha512",
 "filename": "file1.pdf"
 },
 {
 "size": 3420686,
 "digest": "931eb84f798dc9add1a10c7bbd4cc85fe08efda26cac473411638d1f856865524a517209d4c7184d838ee542c8ebc9909dc64ef60f8653a681270ce23524e8e4",
 "algorithm": "sha512",
 "filename": "file3.pdf"
 }
 ]

The simplest usecase for tooltool is to provide a manifest and the url of a tooltool server (this url is not longer used, the last modified file is from 13-Oct-2015) , and run a "fetch" command to download the files mentioned in the manifest:

 python  tooltool.py fetch -m my-manifest.tt

If a manifest name is not provided, tooltool will default to manifest.tt.

Tooltool is build on https://mozilla-releng.net/tooltool/ and to find the uploaded file you can search after : Filename,SHA512 hash,Upload author or Upload message (including bug number).

example: search after "mitmproxy"

The tooltool download servers are simply apache server folders with a flat structure (all files are stored at root level with no subfolders).

There are global options and command arguments. All terminal interactions after the option parser finishes is done through the Python logging API. The default is to print logging.INFO and higher messages. Currently, the following global options exist:

   -q/--quiet tells Tooltool to print only logging.ERROR and higher messages
   -v/--verbose specifies to print logging.INFO and higher
   -m/--manifest <file> instructs Tooltool to reference a manifest file located at the specified path
   -d/--algorithm <algorithm> instructs Tooltool to use the specified algorithm
   -o/--overwrite tells Tooltool to overwrite a local file if the filename matches the manifest but the hash value is different to the manifest
   --url specifies the base url to be used for remote operations

Where's tooltool code?

Tooltool's main dev repo is https://github.com/mozilla/build-tooltool.

The version of tooltool which is actually deployed in our infrastructure is in https://github.com/mozilla/build-puppet/blob/master/modules/packages/templates/tooltool.py. The tooltool.py file in the puppet repo should be in sync with the main dev repo (they should be identical, apart from the she-bang line).

Listing and Validating

The two most basic commands list a manifest and validate the local files against the manifest. The list command lists out all of the files in the manifest as well as whether they are present and/or valid. The return code from listing is zero unless there was an error in listing the files. Absent or invalid files will still result in an exit code of zero if there was no error in the listing process. The validate command is used to check if all the files in the manifest are present and valid. The exit code for validating is zero if all files in the manifest are present and their hash matches the manifest. It is non-zero if any file is missing locally or the file does not have the same hash as the manifest.

How To Upload To Tooltool

Note: Make sure you're using the latest tooltool.py from https://github.com/mozilla/build-tooltool

First, plan out your upload. Are all of the files you need to upload public? If not, tooltool also offers storage of "internal" files, which are not made publicly available but can be downloaded by those with proper permissions. Internal files might be under a non-redistribution license (e.g., Google SDKs, Microsoft DLLs), but must not include secrets such as passwords or private keys.

Use the `tooltool.py` client to build a manifest containing the files you would like to upload, annotating each file with its visibility level:

   python tooltool.py add --visibility public gcc.tar.gz

Next, you will need credentials for the upload. Tooltool is hosted on RelengAPI, so the credentials take the form of a RelengAPI token. To get such a token, go to https://mozilla-releng.net/tokens, log in, click on "Tokens", and issue a user token with the permissions tooltool.upload.internal and/or tooltool.upload.public, depending on the nature of your upload. If you don't have internal permission, then the option won't be available to you. When you click "Issue", you'll get a long string which you should copy/paste into a (protected) file on your system, and pass to tooltool.py with --authentication-file.

Once all of that is in place, you're ready to perform the upload. Just like for a Git or Mercurial commit, you'll need to provide a message to indicate why you're making this upload. Including a bug reference is always helpful.Also make sure the "file" is in the same repository from where you run tooltool.

   python tooltool.py upload --authentication-file=~/.tooltool-token --message "Bug 1234567: add new frobnicator binaries"

The tool will begin uploading and, when it's finished, may wait a minute or two longer until the files are verified by RelengAPI. If your upload fails for some reason, just restart it. Any files already completely uploaded will not need to be uploaded again.

You can view the available files and uploads at https://mozilla-releng.net/tooltool.

Other tooltool features

The tooltool cache

This has been implemented in bug https://bugzilla.mozilla.org/show_bug.cgi?id=858635 All the bellow examples are used for the older packages that are on "http://tooltool.pub.build.mozilla.org/temp-sm-stuff" url. When connecting to a tooltool server to fetch a file, it is now possible to use a local cache specifying the local cache folder with the -c option, e.g.:

 python  tooltool.py fetch --url http://tooltool.pub.build.mozilla.org/temp-sm-stuff -c ~/cache

In the previous example, folder ~/cache will be inspected for artifacts before connecting to the specified tooltool server. If the specified cache folder does not exist, tooltool will create it (if the user running tooltool has permissions to do so!). Purging the cache with the purge command

A mechanism to purge the tooltool cache has been implemented as a separate command. Let's see some examples of purge commands:

 python  tooltool.py purge -c ~/cache
 python  tooltool.py purge -c ~/cache -s 34
 python  tooltool.py purge -c ~/cache --size 34

In the first example, we are cleaning up the tooltool cache completely, i.e.: all the content will be wiped out

In the second and third example the extra parameter -s (or, in extended form, --size) is provided, which specifies the number of Gigs we want to be free at the end of the purging process. Tooltool will delete files in the cache folder, starting with the oldest ones, until the specified amount of gigs is free on disk. When there is more free space that the specified gigs when the command is invokes, no files will be deleted at all.

Enabling automated tooltool cache purge within purge_build.py

Since the purge_builds.py script (within the build-tools repo) is already responsible of clean-up operations, the option of cleaning the tooltool cache has been added to that script as well.

In order for the automated cleanup to take place, two environment variables need to be setup when purge_builds.py is invoked, and namely:

 TOOLTOOL_HOME: the folder containing tooltool.py
 TOOLTOOL_CACHE: the local folder being used as tooltool cache (which will be cleaned up)

How Tooltool is invoked in build/test jobs

Two wrapper scripts exist which are usually being used to invoke tooltool in build/test jobs:

• tooltool_wrapper.sh: http://hg.mozilla.org/build/tools/file/327616201e98/scripts/tooltool/tooltool_wrapper.sh
• TooltoolMixin class for mozharness calls: https://hg.mozilla.org/mozilla-central/file/default/testing/mozharness/mozharness/mozilla/tooltool.py (despite being an Homonym of tooltool itself, this is a python module defining TooltoolMixin class to be used in mozharness calls).

Tooltool uploads: the old school

This is the old-school, deprecated, manual way to upload artifacts to tooltool. Please use the official procedure described above whenever possible, in order to keep information about what is uploaded, by whom, and why (the manual procedure doesn't allow to store any metadata about uploads).

If you don't want to upload from your own laptop (because, eg, you have a slow uplink) you can do this from cruncher.

Access cruncher with your credentials:

ME="your_short_ldap_username" # or `whoami`
ssh -A $ME@cruncher.build.mozilla.org

Download the file and then:

scp filename.tar.xz $ME@relengwebadm.private.scl3.mozilla.com:

Login to relengwebadmn:

ssh $ME@relengwebadm.private.scl3.mozilla.com

And deploy the file to tooltool:

FILE=~/emulator.zip # or whatever you're uploading
export SHA512=`openssl sha512 $FILE | cut -d' ' -f2`
sudo mv -i $FILE /mnt/netapp/relengweb/tooltool/pvt/build/sha512/${SHA512}
sudo chmod 644  /mnt/netapp/relengweb/tooltool/pvt/build/sha512/${SHA512}
ls -l  /mnt/netapp/relengweb/tooltool/pvt/build/sha512/${SHA512}

• Add the filename, filesize, and sha512 digest to the bug you are working on. These can be added to the tooltool manifests later.