the Chromium logo

The Chromium Projects
Chromium OS > Reference >

Autoserv Packaging

Autoserv at a glance

Autoserv -- as described on its documentation page -- is a framework for automating machine control. It can do things like power cycle, install kernels, run arbitrary commands, transfer files, and run autotest tests. While a “machine” can be virtual, local, or remote, autoserv is largely used on chromeOS for remotely controlling chromebooks for automated testing purposes.

Autoserv includes specific support for Autotest. It’s able to install Autotest on host devices, run control files on them, and then fetch results back to the server. To install the appropriate files onto a host device, autoserv’s packaging system is used. Since autotest is it's own package it can change at a rate independent of the image, and changes to one do not require changes to the other. Using packages also allows us continue with our suite scheduling logic while non-essential packages are staged asynchronously.

Autoserv packaging high-level overview

Autoserv’s packaging system is essentially a set of functions to fetch, upload, install, and remove packages that are necessary to control a device under test (DUT). Each package is a collection of files that is tarred and bzipped (into a *.tar.bz2 file) to facilitate the transferring of files onto the DUT. When transferring packages to a DUT, the system computes an MD5 checksum for each package to determine whether or not any previously-installed package has changed and needs to be re-installed. A package is only installed if it has changed according to the checksum (or, of course, if the package is not already on the DUT).

There are 3 types of packages: those associated with test code (“test”); with profiler code (“profiler”); and with dependencies (“dep”). These packages are all generally handled the same by the packaging system, but they’re installed into different subfolders on the DUT. There is also a special package that is used to bundle the autotest client code (“client”).

Implementation details

Much of the packaging system is implemented in client/common_lib/base_packages.py. It contains the logic for fetching packages from either HTTP or from a local filesystem (class HttpFetcher and class LocalFilesystemFetcher, respectively). Class BasePackageManager in that file also contains the logic for creating package files (tar_package), fetching and installing (i.e., untarring) package files (fetch_pkg and install_pkg), and working with checksums.

The autotest client code is installed via the packaging system from the server/autotest.py file. Autotest deps are installed via setup_dep in client/bin/job.py. The utils/packager.py file also calls into the packaging system to create packages.

For test code, packages are generally created at emerge time (this is specific to chromeOS). They are simply bzipped versions of corresponding test code located in the source directory at client/site_tests. If a previously-built package is missing from both a DUT and the server, autoserv will bundle a temporary package using local source and send it to the DUT. Otherwise, if the server finds a previously-built package, it will send it to the DUT only if it’s changed from what’s already there. In this case, local changes to source code are ignored because the previously-built package is used instead. autotest_quickmerge works around this by deleting the package files for altered tests to force autoserv to build and use temporary packages created from local source.

The interesting package fetching work for test code occurs as part of the autotest harness, which can be found in client/bin/harness_autoserv.py. This contains the AutoservFetcher (if you see log messages about fetching from an autoserv:// repo, this is what's being used). The fetching logic then ends up in server/autotest.py in _process_line and more interestingly _send_tarball, which sends the .tar.bz2 package file, or generates one using PackageManager's tar_package function if it doesn't exist.

Currently, fetching packages via autoserv is only used in non-prod situations, such as with run_remote_tests.sh. In prod, all of the packages are instead pulled from devservers via the HttpFetcher using the job_repo_url. This is a result of crbug.com/212641 and crbug.com/216918; we were hitting a case where fetching would hang (it creates a fifo that gets written to, once the package has been pushed down to the DUT). A few CLs were subsequently pushed so that fetching from autoserv could be disabled in prod (https://gerrit.chromium.org/gerrit/#/c/33804/2 and https://gerrit.chromium.org/gerrit/#/c/34665/3).

At a high level, this is what happens:

  1. In the simplest case, a reimage job runs, after which a server jobs runs. The reimage job stages autotest packages and sets the job_repo_url as part of

the autoupdate 'test'. These packages contain the tests (one bz file for each test), the dependencies (like chrome_test) and client-autotest.

  1. job_repo_url is set in add_cros_version_labels_and_job_repo_url, and points to the devserver where the required packages are staged.

  2. Server job kicks off autotest.py/site_autotest.py, which are the classes responsible for installing autotest on the client.

  3. autotest.py gets the job_repo_url through site_autotest's get_fetch_location, and creates a PackageManager in _install_using_packaging

  4. This package manager is the aforementioned HttpFetcher, which does a wget from the given url to fetch packages; This happens through

client/common_lib/base_packages install_pkg.

Deps:

An example of a dep is pyauto. The actual installation of this dep happens through bin/job.install_packages on the DUT, though the steps are pretty general for package installation and are as follows:

  1. Installs packages using packages.PackageManager
  1. The installation happens through PackageManager.fetch_pkg

The name of the tarball to get from the repo_url is constructed as pkg_type-name.tar.bz2. In the case of pyauto this is dep-pyauto.tar.bz2

  1. Check if the package has already been fetched

If not, check if we need to fetch:

a. Check if a checksum exists on the DUT

b. Fetch checksum from package url if it doesn't, through fetch_pkg_file

c. Parse the contents of the checksum file into a per package dictionary

d. Create an md5 checksum of the package present at /usr/local/autotest/packages/client-autotest.tar.bz2 on the DUT

e. Update the checksum file on the DUT so we don't pull the same package again through _save_checksum_dict

f. If the checksums don't match untar contents of the package via untar_pkg

g. This will essentially run a tar command on the DUT, recompute the checksums, and rewrite them to the DUT.

  1. If fetching is needed, invoke base_packages.fetch_pkg_file which will lead to a wget in the case of pyauto:

wget --connect-timeout=15 --retry-connrefused --wait=5 -nv http://devserver:port/static/archive/build/autotest/packages/dep-pyauto_dep.tar.bz2 -O /usr/local/autotest/packages/dep-pyauto_dep.tar.bz2

  1. Place the package in /usr/local/autotest/packages

  2. Update the checksum

  3. Install the package through base_packages.install_pkg

  4. Check to see if the install dir exists: ls /usr/local/autotest/packages/install_dir

  5. The actual untarring happens in base_packages.untar_pkg, before this we mkdir install_dir untar_pkg runs tar xjf from_path -C dest_dir

In an end to end run of a pyauto test, the following packages will get installed through the process described above:

On the server, install client-autotest on the DUT:

  1. mkdir /usr/local/autotest/packages

  2. Copy autotest client packages to /usr/local/autotest/packages/client-autotest.tar.bz2

  3. Untar the package if the checksum doesn't match, as described above.

On the DUT, install the tests and pyauto dep:

  1. Execute a command like: ./bin/autotest -H autoserv --verbose --user=beeps /usr/local/autotest/control.autoserv

Actually that's a lie, it executes autotestd_monitor which executes autotestd which leads to that command, but that is irrelevant for this discussion.

  1. Installing the test:

a. The control file Invokes job.run_test with the test name, which leads to common_lib/test.runtest

b. bin/job.install_pkg installs the test using common_lib.packages.PackageManager leading to a line in the logs like (instead of the wget):

22:26:48 DEBUG| Successfully fetched test-login_LoginSuccess.tar.bz2 from http://devserver/static/archive/lumpy-release/R29-

4279.0.0/autotest/packages/test-login_LoginSuccess.tar.bz2

Again this installation happens through a process as described above.

  1. Installing deps, cros_ui_test:

a. We then go on to executing the initialize function of the test via common_lib/test._cherry_pick_call

b. Since pyauto tests inherit from cros_ui_test, we execute the initialize of cros_ui_test, which does things like

start an authserver, remove cryptohome valuts, restart cros_ui etc. PyAutoTest.initialize finally install deps through

cros/pyauto_test._install_deps and then proceeds to import pyauto.

c. PyAutoTest passes in the install_dir (/usr/local/autotest/deps/pyauto_dep) and the package_dir (/usr/local/packages) which calls into bin/job

General autoserv documentation:
<https://github.com/autotest/autotest/wiki/Autoserv>

Published May 13, 2013