Build and Run Environments
Setting up your environment is crucial for running and building tests, and Pavilion gives you several options for doing so.
Assumptions
Pavilion assumes that it runs under a relatively clean, default login environment; ie the login environment a new user might get when they log into the machine for the first time, including any default modules or environment variables. This is not required, but simply means that when you run Pavilion, it will work the same as when your co-worker does.
That aside, most basic changes won’t have a significant impact on Pavilion. However, a few things will: - Changing from the default Python3 or PYTHONPATH - Modifying LD_LIBRARY_PATH or similar variables that affect compilation.
Lastly, Pavilion writes and runs BASH scripts. It assumes that whatever your environment is, the module system will work under BASH just as well as your native environment.
Environment Variables
The env
attribute allows you to set environment variables in either
the run or build scripts. They are configured as a YAML
mapping/dict, and (unlike the rest of Pavilion) can have upper-case keys
(but no dashes). Like with the run/build commands, the values can
contain any bash shell syntax without issue.
env_example:
run:
env:
PYTHONPATH: $(pwd)/libs
TEST_PARAM1: 37
# The starting { means this has to be quoted.
AN_ARRAY: "{hello world}"
cmds:
- for value in ${AN_ARRAY[@]}; do echo $value; done
- python3 mytest.py
Each set variable is set (and _exported) in the order given.
#!/bin/bash
export PYTHONPATH=$(pwd)/libs
export TEST_PARAM1=37
export AN_ARRAY={hello world}
for value in ${AN_ARRAY[@]}; do echo $value; done
python3 mytest.py
Escaping
Values are not quoted. If they need to be, you’ll have to quote them twice, once for YAML and once for the quotes you actually need.
quote_example:
run:
env:
DQUOTED: '"This will be in double quotes. It is a literal string as far
as YAML is concerned."'
SQUOTED: "'This $VAR will not be resolved in bash, because this is single
quoted.'"
DDQUOTED: """Double quotes to escape them."""
SSQUOTED: '"That goes for single quotes '' too."'
NO_QUOTES: $(echo "YAML only quotes things if the first character
is a quote. These are safe.")
#/bin/bash
export DQUOTED="This will be in double quotes. It is a literal string as far as YAML is concerned."
export SQUOTED='This $VAR will not be resolved in bash, because this is single quoted.'
export DDQUOTED="Double quotes to escape them."
export SSQUOTED="That goes for single quotes '' too."
export NO_QUOTES=$(echo "YAML only quotes things if the first character is a quote. These are safe.")
Modules
Many clusters employ module systems to allow for easy switching between build environments. Pavilion supports both the environment (TCL) and the LMOD module systems, but other module systems can be supported by overriding the base Module Wrapper Plugins.
Loading modules
In either run or build configs, you can have Pavilion import modules by listing them (in the order needed) under the modules attribute.
module_example:
build:
modules: [gcc, openmpi/2.1.2]
In the generated build script, each of these modules will be both loaded and checked to see if they were actually loaded.
#/bin/bash
TEST_ID=$1
module load gcc
# This checks to make sure the module was loaded. If it isn't the script
# exits and updates the test status.
is_module_loaded gcc $TEST_ID
module load openmpi/2.1.2
is_module_loaded openmpi/2.1.2 $TEST_ID
Other Module Manipulations
You can also unload and swap modules.
module_example2:
build:
source_location: test_code.xz
run:
# This assumes gcc and openmpi are already loaded by default.
modules: [gcc->intel/18.0.4, -openmpi, intel-mpi]
cmds:
- $MPICC -o test_code test_code.c
Module Wrappers
Module wrappers allow you to change how Pavilion loads specific modules, module versions, and even modules in general. The default module wrapper provides support for lmod and tmod, generates the source to load modules within run and build scripts, and checks to see if they’ve been successfully loaded (or unloaded).
Module wrappers are added, typically in Host Configs, via the module_wrappers
sections:
# This would be in a a 'host' file, typically
module_wrappers:
gcc:
# When gcc is asked for on this system, load these modules instead.
modules:
# This assumes PrgEnv-cray is the default on this machine.
- PrgEnv-cray->PrgEnv-gnu
# Swap the default gcc (that comes with the PrgEnv) for the requested one
# You shouldn't specify versions (unless you want to force a version), Pavilion
# will automatically ask for the version asked for by
- gcc->gcc
env:
# You can also add environment variables to automatically be exported after
# the module is loaded.
PAV_CC: '$CC'
PATH: '$PATH:$(dirname $(which gcc))'
So now, we can write our tests to generically ask for ‘gcc’:
mytest:
run:
modules: 'gcc'
Wildcards
The modules specified can be written as file system globs, to match a wider range of modules and to support module naming (particularly under lmods) that is less generic.
module_wrappers:
# On this system, the modules for MPI layers have the compiler as part of the name.
# This will match 'openmpi-gcc', 'openmpi-intel', etc.
openmpi-*:
modules:
# This will be auto-converted into the mpi requested.
- 'openmpi-*'
env:
PAV_MPICC: '$(which mpicc)'
Wildcards work on the left side of module swaps (modA->modB
) as well. Pavilion will look for a
loaded package that matches the left side, and swap it for the right side.
Version Variable
If you need the version of the loaded module, it’s available in the ‘<mod_name>_VERSION’
environment variable. If the the mod_name contains wildcards, ‘*’ is replaced with ‘any’, and
other characters are replaced with underscores. So gcc-[f]-?-*
gets a ‘gcc-_-_-any_VERSION’
environment variable.
Module Wrapper Plugins
For more complicated use cases, you can also write module wrapper plugins. For more information on writing these, see Module Wrapper Plugins.
Spack Packages
Pavilion supports both the installation and loading of Spack packages inside of test scripts. This is not enabled by default as it requires an external Spack instance.
Once configured, Spack packages can be installed and loaded in Pavilion test scripts using the ‘spack’ section inside both the ‘build’ and ‘run’ sections of a test config. This section has two keys, ‘install’ and ‘load’, that take a list of package names with optional spec and dependency options.
build:
spack:
install:
- ember
- mpich@3.0.4
- mpileaks @1.2:1.4 %gcc@4.7.5 +debug
load:
- gcc
run:
spack:
load:
- ember
- mpich
- mpileaks
Pavilion will also allow for Spack specific configuration changes to be added inside test configs under the ‘spack’ section. The following Spack specific options are currently supported:
build_jobs - The max number of jobs to use when running make in parallel.
repos - Paths to package repositories.
mirrors - URLs that point to a directories that contain Spack packages.
upstreams - Other Spack instances.
These are directly inserted into the Spack build environment’s spack.yaml file. Refer to Spack documentation on usage for these.
base:
spack:
build_jobs: 4
mirrors:
MIRROR1: https://a_spack_mirror.com
repos:
- /a/path/to/package/repo
- /a/different/path/to/package/repo
upstreams:
Upstream1:
install_tree: /path/to/other/spack/instance
Enabling Spack Features
Spack features can be added by providing a Spack instance’s path
under the spack_path
key in the Pavilion config file (pavilion.yaml
).
For more Pavilion configuration information, see
Configuring Pavilion.
Once Spack is enabled globally for Pavilion, it can be enabled for individual tests simply by including a spack.load or spack.install key under the run or build sections of a test config. Trying to use Spack in a test when it is not globally enabled first results in an error.
How Pavilion Uses Spack
When Spack is enabled inside of a test config, Pavilion generates an anonymous
Spack environment file that is activated at the beginning of both the build and
run scripts. The generated environment file, spack.yaml
, is placed in the
respective build directory so that it can be reactivated when a build is reused.
The Spack environment file is modified so that Spack packages are installed
inside their respective build directory in a directory named spack_installs
,
as seen below:
# SPACK: Spack environment configuration file.
spack:
config:
install_tree: ~/.pavilion/builds/7a3986a56e7c04a7/spack_installs
This means any installs that are not in the global Spack instance will only be in the scope of this build.
Global Spack packages or packages in upstreams will still require to be listed under the install section for both the build and run sections of a test config so that those packages can be added to the Spack environment correctly.