flux-environment(7)
DESCRIPTION
The following environment variables are set by Flux or influence Flux.
JOB ENVIRONMENT
The following are set in the environment of each task spawned by flux-shell(1) as part of a Flux job.
- FLUX_JOB_ID
The current jobid in F58 form. F58 is a compact, non-numeric representation of Flux's 64-bit integer jobid. If the numeric form is required, use e.g.:
NUMERIC_JOB_ID=$(flux job id $FLUX_JOB_ID)
- FLUX_JOB_SIZE
The number of tasks in the current job.
- FLUX_JOB_NNODES
The total number of nodes hosting tasks on behalf of the current job.
Note
FLUX_JOB_NNODES
is more precisely defined as the total number of
flux-shell(1) processes running tasks on behalf of the current job.
Normally one shell is started per broker, and one broker is started per
node. However, in rare test setups, a large Flux instance is mocked by
running multiple brokers per node. In that case, this variable may not
represent the physical node count.
- FLUX_TASK_RANK
The zero-origin, global rank for this task. Tasks are assigned ranks using a "block" algorithm by default, although
flux submit --taskmap
may select other mapping algorithms.Example: 8 tasks on 2 nodes with block and cyclic task mapping:
Mapping
Node 0
Node 1
block
0, 1, 2, 3
4, 5, 6, 7
cyclic
0, 2, 4, 6
1, 3, 5, 7
- FLUX_TASK_LOCAL_ID
The zero-origin, local (to the node) rank for this task.
Example: 8 tasks on 2 nodes:
Node 0
Node 1
0, 1, 2, 3
0, 1, 2, 3
- FLUX_JOB_CC
When
flux submit --cc
orflux bulksubmit --cc
is used to submit a set of jobs,FLUX_JOB_CC
is set to the the integer id of the current job in the set.
- FLUX_JOB_TMPDIR
The path of a per-job temporary directory that is created on each host node before any tasks are started, and cleaned up after all tasks have exited. All a job's tasks on a given node share the same directory.
- FLUX_KVS_NAMESPACE
Each job is assigned a unique, job-owner-writable Flux KVS key space that is independent of the default (primary) one and persists as such while the job is in the RUNNING state. This environment variable is interpreted by the Flux KVS API and therefore flux-kvs(1) as a directive to treat all operations as rooted in that space. The job exec service and the job shell record the job's input, output, and a log of events in this space.
After the job completes, the job's namespace is added to the primary namespace and becomes part of the read-only job record.
- PMI_RANK
- PMI_SIZE
- PMI_FD
- PMI_SPAWNED
- FLUX_PMI_LIBRARY_PATH
The
pmi
shell plugin sets these variables in the job environment to aid in the bootstrap of parallel programs. They are not set when the simple PMI server is disabled, e.g. withflux run -opmi=none
.The
PMI_*
variables are standard for PMI-1 and are described in Flux RFC 13.FLUX_PMI_LIBRARY_PATH
is set to the full path of Flux'slibpmi.so
shared library, which is normally not installed to standard system paths. This exists as an aid to the pre-v5 OpenMPI Flux MCA plugins so that an MPI program running under Flux knows where todlopen()
the library for bootstrap.
INITIAL PROGRAM ENVIRONMENT
The flux-alloc(1) interactive shell and the flux-batch(1) batch script are examples of Flux initial programs. Flux does not set many environment variables for the initial program. In fact, the following are actively unset to avoid confusion when they are set by the enclosing instance:
I_MPI_PMI_LIBRARY
PMI_*
SLURM_*
The FLUX_URI
variable is set, however, so Flux commands can be used
as needed from the initial program to obtain information they might get via the
environment in other workload managers, for example:
BATCH_NNODES=$(flux resource list -n -o {nnodes})
BATCH_NCORES=$(flux resource list -n -o {ncores})
BATCH_NGPUS=$(flux resource list -n -o {ngpus})
BATCH_HOSTLIST=$(flux getattr hostlist)
BATCH_JOBID=$(flux getattr jobid)
PMI CLIENT
The flux-broker(1) is capable of bootstrapping from configuration or using a PMI client, similar to the way an MPI program bootstraps. The broker's PMI client is separate from the flux-shell(1) PMI server offered to parallel programs launched by Flux. The following environment variables affect the broker's PMI client.
- FLUX_PMI_DEBUG
When set (to any value) in the broker's environment, PMI client tracing is enabled, causing PMI operations that occur during broker bootstrap to be logged to standard error.
- FLUX_PMI_CLIENT_METHODS
Flux iterates through a list of PMI client implementations to find one that works. By default the list is
simple libpmi2 libpmi single
. The sequence can be altered by setting this variable to a space-delimited list of client implementations. The built-in ones are:- simple
Use the PMI-1 simple wire protocol.
- libpmi2[:PATH]
dlopen()
libpmi2.so
and use the PMI-2 API, optionally at a specific PATH.- libpmi[:PATH]
dlopen()
libpmi.so
and use the PMI-1 API, optionally at a specific PATH.- single
Become a singleton. This always succeeds so should be the last method.
- FLUX_PMI_CLIENT_SEARCHPATH
A colon-separated list of directories to search for PMI client plugins. Client plugins can be packaged separately from flux-core.
- FLUX_IPADDR_HOSTNAME
When bootstrapping with PMI, the broker dynamically selects an TCP address to bind to for overlay network communication, which it then exchanges with peers using PMI. By default, it tries to use the address associated with the default route. Setting this variable to any value in the broker's environment directs it to prefer the address associated with the system hostname(1) instead.
- FLUX_IPADDR_V6
When dynamically selecting an address to use with PMI, the broker prefers IP version 4 addresses. Setting this variable to any value in the broker's environment causes it to prefer version 6 addresses.
- FLUX_IPADDR_HOSTNAME
Force PMI bootstrap to assign the broker an address associated with a particular network interface, like
eth0
.
CUSTOM OUTPUT FORMATS
Sites and individual users may create custom output formats for some Flux
commands. The formats are expressed in configuration files with a base name
of the command name plus a .toml
, .yaml
, or .json
extension,
stored in directories that follow the XDG Base Directory Specification.
Named formats are merged and/or overridden in the following order:
internal defaults
config files found in a
flux
sub-directory of theXDG_CONFIG_DIRS
directoriesconfig files found in a
flux
sub-directory ofXDG_CONFIG_HOME
For more information on named formats see the individual command documentation and the CONFIGURATION section of flux-jobs(1).
- XDG_CONFIG_DIRS
A colon-separated, preference-ordered list of base directories to search for configuration files in addition to the
XDG_CONFIG_HOME
base directory. If unset,/etc/xdg
is used.
- XDG_CONFIG_HOME
The base directory for user-specific configuration files. If unset,
$HOME/.config
is used.
- FLUX_JOBS_FORMAT_DEFAULT
- FLUX_RESOURCE_STATUS_FORMAT
- FLUX_RESOURCE_LIST_FORMAT_DEFAULT
- FLUX_QUEUE_LIST_FORMAT_DEFAULT
- FLUX_PGREP_FORMAT_DEFAULT
In addition to registering custom named formats, users and sites can change the default output format to one of the named formats by setting an environment variable to the format name. The above variables affect the default output of flux-jobs(1), flux-resource(1), flux-queue(1), and flux-pgrep(1).
TESTING
The following environment variables are primarily useful when debugging Flux components or writing tests.
- FLUX_HANDLE_TRACE
If set in the environment of a Flux component, the
FLUX_O_TRACE
flag is automatically set in any call to flux_open(3). This causes decoded messages passed over theflux_t
handle to be decoded and printed on standard error.
- FLUX_HANDLE_MATCHDEBUG
If set in the environment of a Flux component, the
FLUX_O_MATCHDEBUG
flag is automatically set in any call to flux_open(3). This causes a diagnostic to be printed to standard error if any matchtags are leaked when the broker connection is closed.
- FLUX_HANDLE_USERID
Mock a user. If set to a numerical user ID in the environment of a Flux component, all messages sent by the component appear to have been sent by this user. This is useful for testing code that authorizes actions based on the identity of the requesting user. This is restricted to the instance owner.
- FLUX_HANDLE_ROLEMASK
Mock a rolemask (capability set). If set to a decimal or hex (
0x
prefixed) value in the environment of a Flux component, all messages sent by the component are stamped with this rolemask. This is useful for testing code that authorizes actions based on the possession of particular roles. This is restricted to the instance owner.
- FLUX_FAKE_HOSTNAME
When Flux bootstraps from a configuration file as described in flux-config-bootstrap(5), a flux-broker(1) determines its rank by looking up its own hostname in a
hosts
array and using the array index as its rank. To allow this to be tested on a single node,FLUX_FAKE_HOSTNAME
may be set in the broker's environment to use the specified name instead of the result of gethostname(3). Use of this capability in test is simplified by theflux start --test-hosts
option.
- FLUX_HWLOC_XMLFILE
Flux discovers available resources dynamically using HWLOC. In some cases dynamic discovery is not desired, such as when it causes poor performance in parallel testing. Flux may be directed to read topology from an XML file instead by setting
FLUX_HWLOC_XMLFILE
to the file path.flux resource reload offers a related mechanism for loading a set of HWLOC xml files directly into the instance resource inventory for test scenarios.
- FLUX_URI_RESOLVE_LOCAL
If set, force flux-uri(1) and the URI resolver embedded in other commands to resolve URIs to local form. This is useful in test environments where the remote connector does not work.
- FLUX_RESOURCE_LIST_RPC
If set, flux-resource(1) uses the specified RPC topic string instead of
resource.sched-status
. This is used in test to verify that thesched.resource-status
RPC used in earlier releases still works for backwards compatibility.
MISCELLANEOUS
- FLUX_F58_FORCE_ASCII
A locale or terminal misconfiguration can cause the
ƒ
character used in Flux jobids to be rendered incorrectly. As a workaround, set this variable and ASCIIf
is used instead.
- FLUX_CONF_DIR
If set in in the flux-broker(1) environment, configuration files matching
*.toml
are loaded from the specified directory. Theflux broker --config-path
option does that too, and is more flexible in that it can also load single files in TOML or JSON format.
- FLUX_ATTACH_NONINTERACTIVE
If set, never show the status line in flux job attach output.
- FLUX_PROXY_REMOTE
When flux-proxy(1) connects to a remote instance, it sets this variable to the authority part of the remote URI. This serves as a hint to flux_attr_get(3) to transform the value of the
parent-uri
broker attribute into a remote URI so it can work from the remote proxy environment. For example:$ flux alloc -N1 f(s=1,d=1) $ flux getattr parent-uri local:///run/flux/local
$ flux proxy $(flux job last) ƒ(s=1,d=1) $ printenv FLUX_PROXY_REMOTE test0 ƒ(s=1,d=1) $ flux getattr parent-uri ssh://test0/run/flux/local
- FLUX_TERMINUS_SESSION
The current terminus session ID. A terminus session is started when the job has an interactive pseudo-terminal, which occurs when a job is run with
flux run -o pty.interactive
, or when a Flux instance is started with flux-alloc(1).
- FLUX_RC_EXTRA
If set to a colon-separated list of directories, the installed flux-broker(1) rc scripts search these directories for additional scripts to run during broker initialization and finalization.
Specifically the
rc1
script runsrc1.d/*
in each directory and therc3
script runsrc3.d/*
in each directory.
- FLUX_SHELL_RC_PATH
Set to a colon-separated list of directories to be added to the directories that flux-shell(1) searches for lua scripts to extend its initrc.
- FLUX_SSH
Override the compiled-in path to the ssh executable used by the ssh connector. The ssh connector is invoked when attempting to open a connection to Flux with a URI that begins with
ssh://
.
SUB-COMMAND ENVIRONMENT
flux(1) sets up the environment for sub-commands using a combination of compiled-in install paths and the environment.
Note
The PREPEND versions of environment variables below may
be necessary when developing and testing a new version
of a Flux command (FLUX_EXEC_PATH_PREPEND
),
module (FLUX_MODULE_PATH_PREPEND
), connector
(FLUX_CONNECTOR_PATH_PREPEND
), or Python module
(FLUX_PYTHONPATH_PREPEND
) when an existing version of that
component is already installed in the system default paths. Otherwise,
the installed component would always be used by the system Flux, since
the installed paths are always placed first in the subcommand environment
created by flux(1).
- FLUX_EXEC_PATH
- FLUX_EXEC_PATH_PREPEND
flux(1) finds sub-command executables by searching:
$FLUX_EXEC_PATH_PREPEND : install-path : $FLUX_EXEC_PATH
Values may include multiple directories separated by colons.
- FLUX_MODULE_PATH
- FLUX_MODULE_PATH_PREPEND
FLUX_MODULE_PATH
is set in the environment of the broker so that broker modules can be found and loaded when requested by flux-module(1):$FLUX_MODULE_PATH_PREPEND : install-path : $FLUX_MODULE_PATH
Values may include multiple directories separated by colons.
- FLUX_CONNECTOR_PATH
- FLUX_CONNECTOR_PATH_PREPEND
FLUX_CONNECTOR_PATH
is set in the environment of sub-commands so that flux_open(3) can find the connector corresponding to the URI scheme:$FLUX_CONNECTOR_PATH_PREPEND : install-path : $FLUX_CONNECTOR_PATH
Values may include multiple directories separated by colons.
- PYTHONPATH
- FLUX_PYTHONPATH_PREPEND
PYTHONPATH
is set so that sub-commands can find required Python libraries:$FLUX_PYTHONPATH_PREPEND : install-path : $PYTHONPATH
Values may include multiple directories separated by colons.
Note
Flux commands written in Python further modify Python's
sys.path
to ensure that interpreter default paths appear before any custom values
set in PYTHONPATH
. This is an attempt to avoid incompatible
modules interfering with the operation of Flux commands. If it becomes
necessary to force a non-standard module first in the search path (e.g.
for testing, instrumentation, etc.) then FLUX_PYTHONPATH_PREPEND
should be used.
- LUA_PATH
- LUA_CPATH
- FLUX_LUA_PATH_PREPEND
- FLUX_LUA_CPATH_PREPEND
LUA_PATH
andLUA_CPATH
are set so that sub-commands can find required Lua libraries. They are set, respectively, to$FLUX_LUA_PATH_PREPEND ; install-path ; $LUA_PATH ;;
$FLUX_LUA_CPATH_PREPEND ; install-path ; $LUA_CPATH ;;
Values may include multiple directories separated by semicolons.
RESOURCES
Flux: http://flux-framework.org
Flux RFC: https://flux-framework.readthedocs.io/projects/flux-rfc