flux-resource(1)
SYNOPSIS
DESCRIPTION
flux resource lists and manipulates Flux resources. The resource inventory is maintained and monitored by the resource service. The scheduler acquires a subset of resources from the resource service to allocate to jobs, and relies on the resource service to inform it of status changes that affect the usability of resources by jobs as described in RFC 27.
The flux resource list subcommand queries the resource module for the scheduler view of resources, including allocated/free status.
The other flux resource subcommands operate on the resource service and are primarily of interest to system administrators of a Flux system instance. For example, they can show whether or not a node is booted, and may be used to administratively drain and undrain nodes.
A few notes on drained nodes:
While a node is drained, the scheduler will not allocate it to jobs.
The act of draining a node does not affect running jobs.
When an instance is restarted, drained nodes remain drained.
The scheduler may determine that a job request is feasible if the total resource set, including drained nodes, would allow it to run.
In flux resource status and flux resource drain, the drain state of a node will be presented as "drained" if the node has no job allocations, and "draining" if there are still jobs running on the node.
If a node is drained and offline, then "drained*" will be displayed.
Some further background on resource service operation may be found in the RESOURCE INVENTORY section below.
COMMANDS
list
Show the scheduler view of resources.
In the scheduler view, excluded resources are always omitted, and unavailable resources are shown in the "down" state regardless of the reason (drained, offline, torpid, etc). Valid states in the scheduler view are:
- up
Available for use.
- down
Unavailable for use.
- allocated
Allocated to jobs.
- free
Not allocated to jobs.
- all
Wildcard matching all resources.
- -s, --states=STATE,...
Restrict displayed resource states to a comma separated list of the states listed above.
If unspecified,
free,allocated,downis used.
- -q, --queue=QUEUE,...
Filter results to only include resources in the specified QUEUE. Multiple queues may be separated by a comma.
- -i, --include=TARGETS
Filter results to only include resources matching TARGETS, which may be specified either as an idset of broker ranks or list of hosts in hostlist form. It is not an error to specify ranks or hosts which do not exist.
- -o, --format=FORMAT
Customize the output format (See the OUTPUT FORMAT section below).
- -n, --no-header
Suppress header from output,
info
Show a brief, single line summary of scheduler view of resources, for example:
8 Nodes, 32 Cores, 0 GPUs
- -s, --states=STATE,...
Limit the output to specified resource states as described above for the list command.
If unspecified,
allis used.
- -q, --queue=QUEUE,...
Filter results to only include resources in the specified QUEUE. Multiple queues may be separated by a comma.
- -i, --include=TARGETS
Filter results to only include resources matching TARGETS, which may be specified either as an idset of broker ranks or list of hosts in hostlist form. It is not an error to specify ranks or hosts which do not exist.
R
Emit an RFC 20 Resource Set based on the scheduler view of resources.
- -s, --states=STATE,...
Limit the output to specified resource states as described above for the list command.
If unspecified,
allis used.
- -q, --queue=QUEUE,...
Filter results to only include resources in the specified QUEUE. Multiple queues may be separated by a comma.
- -i, --include=TARGETS
Filter results to only include resources matching TARGETS, which may be specified either as an idset of broker ranks or list of hosts in hostlist form. It is not an error to specify ranks or hosts which do not exist.
status
Show system view of resources. Valid states in the system view are:
- avail
available for scheduling when up. This includes all nodes that are not excluded, drained, or torpid.
- exclude
excluded by configuration
- draining
drained but still allocated
- drained
drained and unallocated
- drain
shorthand for
drained,draining- allocated
node is currently allocated to a job or housekeeping
- torpid
node has been unresponsive for a period of time and is temporarily unavailable for scheduling
- housekeeping
node is currently running housekeeping
- offline
node has not joined the Flux instance (e.g. turned off or has not started the flux broker).
- online
node has joined the Flux instance
flux resource status displays a line of output for each set of resources that share a state and online/offline state.
Note
flux resource status queries both the administrative and scheduler view of resources to identify resources that are available, excluded by configuration, torpid, administratively drained or draining, or currently executing housekeeping.
- -s, --states=STATE,...
Restrict the set of resource states a comma-separated list.
If unspecified,
avail,exclude,draining,drained,torpid,housekeepingis used.
- -q, --queue=QUEUE,...
Filter results to only include resources in the specified QUEUE. Multiple queues may be separated by a comma.
- -i, --include=TARGETS
Filter the results to only include resources matching TARGETS, which may be specified either as an idset of broker ranks or list of hosts in hostlist form. It is not an error to specify ranks or hosts which do not exist.
- -o, --format=FORMAT
Customize output formatting. See the OUTPUT FORMAT section below for details.
- -n,--no-header
Suppress header from output,
- --skip-empty
Force suppression of empty lines.
Normally, flux resource status skips lines with no resources, unless the
-s, --statesoption is used.
drain
If specified without targets, list the drained nodes. In this mode, the following options are available:
- -o, --format=FORMAT
Customize output formatting. See the OUTPUT FORMAT section below for details.
- -n,--no-header
Suppress header from output,
- -q, --queue=QUEUE,...
Filter results to only include resources in the specified QUEUE. Multiple queues may be separated by a comma.
- -i, --include=TARGETS
Filter the results to only include resources matching TARGETS, which may be specified either as an idset of broker ranks or list of hosts in hostlist form. It is not an error to specify ranks or hosts which do not exist.
If specified with targets (IDSET or HOSTLIST), drain the specified nodes. Any remaining free arguments are recorded as a reason for the drain event. By default, flux resource drain fails if any of the targets are already drained.
Resources cannot be both excluded and drained, so flux resource drain will also fail if any targets are currently excluded by configuration. There is no option to force an excluded node into the drain state.
This command, when run with arguments, is restricted to the Flux instance owner.
- -f, --force
If any of targets are already drained, do not fail. Overwrite the original drain reason. When
--forceis specified twice, the original drain timestamp is also overwritten.
- -u, --update
If any of targets are already drained, do not fail and do not overwrite the existing drain reason or timestamp.
undrain
Undrain the nodes specified by the targets argument (IDSET or HOSTLIST).
This command is restricted to the Flux instance owner.
- -f, --force
Do not fail if any of the targets are not drained.
reload
Reload the resource inventory from path. By default, path refers to a file in RFC 20 format.
This command is primarily used in test.
- -x, --xml
Interpret path as a directory of hwloc
<rank>.xmlfiles.
- -f, --force
Do not fail if resource contain invalid ranks.
acquire-mute
Tell the resource module to stop sending RFC 28 resource.acquire responses
to the scheduler. This is used during Flux instance shutdown to avoid asking
the scheduler to needlessly process OFFLINE updates.
OUTPUT FORMAT
The --format option can be used to specify an output format using
Python's string format syntax or a defined format by name. For a list of
built-in and configured formats use -o help. An alternate default
format can be set via the FLUX_RESOURCE_STATUS_FORMAT_DEFAULT,
FLUX_RESOURCE_DRAIN_FORMAT_DEFAULT, and
FLUX_RESOURCE_LIST_FORMAT_DEFAULT environment variables (for
flux resource status, flux resource drain, and
flux resource list respectively). A configuration snippet for an
existing named format may be generated with --format=get-config=NAME.
See flux-jobs(1) OUTPUT FORMAT section for a detailed
description of this syntax.
Resources are combined into a single line of output when possible depending on the supplied output format. Resource counts are not included in the determination of uniqueness. Therefore, certain output formats will alter the number of lines of output. For example:
$ flux resource list -no {nnodes}
Would simply output a single of output containing the total number of nodes. The actual state of the nodes would not matter in the output.
The following field names can be specified for the status and drain subcommands:
- state
State of node(s): "avail", "exclude", "drain", "draining", "drained", "torpid", "allocated". If the set of resources is offline, an asterisk suffix is appended to the state, e.g. "avail*".
- statex
Like state, but exclude the asterisk for offline resources.
- status
Current online/offline status of nodes(s): "online", "offline"
- up
Displays a ✔ if the node is online, or ✗ if offline. An ascii y or n may be used instead with up.ascii.
- nnodes
number of nodes
- ranks
ranks of nodes
- nodelist
node names
- timestamp
If node(s) in drain/draining/drained state, timestamp of node(s) set to drain.
- reason
If node(s) in drain/draining/drained state, reason node(s) set to drain.
The following field names can be specified for the list subcommand:
- state
State of node(s): "up", "down", "allocated", "free", "all"
- queue
queue(s) associated with resources.
- properties
Properties associated with resources.
- propertiesx
Properties associated with resources, but with queue names removed.
- nnodes
number of nodes
- ncores
number of cores
- ngpus
number of gpus
- ranks
ranks of nodes
- nodelist
node names
- rlist
Short form string of all resources.
CONFIGURATION
Similar to flux-jobs(1), the flux resource command supports
loading a set of config files for customizing utility output formats. Currently
this can be used to register named format strings for the status,
list, and drain subcommands.
Configuration for each flux resource subcommand is defined in a
separate table, so to add a new format myformat for flux resource list,
the following config file could be used:
# $HOME/.config/flux/flux-resource.toml
[list.formats.myformat]
description = "My flux resource list format"
format = "{state} {nodelist}"
See flux-jobs(1) CONFIGURATION section for more information about the order of precedence for loading these config files.
RESOURCE INVENTORY
The Flux instance's inventory of resources is managed by the resource service, which determines the set of available resources through one of three mechanisms:
- configuration
Resources are read from a config file in RFC 20 (R version 1) format. This mechanism is typically used in a system instance of Flux.
- enclosing instance
Resources are assigned by the enclosing Flux instance. The assigned resources are read from the job's
Rkey in the enclosing instance KVS.- dynamic discovery
Resources are aggregated from the set of resources reported by hwloc on each broker.
Once the inventory has been determined, it is stored the KVS resource.R
key, in RFC 20 (R version 1) format.
Events that affect the availability of resources and should persist across a Flux restart are posted to the KVS resource.eventlog. Such events include:
- resource-define
The instance resource set is known (posted each time the resource module is loaded).
- drain
One or more nodes are administratively removed from scheduling.
- undrain
One or more nodes are no longer drained.
RESOURCES
Flux: http://flux-framework.org
Flux RFC: https://flux-framework.readthedocs.io/projects/flux-rfc
Issue Tracker: https://github.com/flux-framework/flux-core/issues