flux-batch(1)

SYNOPSIS

flux batch [OPTIONS] SCRIPT ...

flux batch [OPTIONS] --wrap COMMAND ...

DESCRIPTION

flux-batch submits SCRIPT to run as the initial program of a Flux subinstance. SCRIPT refers to a file that is copied at the time of submission. Once resources are allocated, SCRIPT executes on the first node of the allocation, with any remaining free arguments supplied as SCRIPT arguments. Once SCRIPT exits, the Flux subinstance exits and resources are released to the enclosing Flux instance.

If there are no free arguments, the script is read from standard input.

If the --wrap option is used, the script is created by wrapping the free arguments or standard input in a shell script prefixed with #!/bin/sh.

If the job request is accepted, its jobid is printed on standard output and the command returns. The job runs when the Flux scheduler fulfills its resource allocation request. flux-jobs(1) may be used to display the job status.

Flux commands that are run from the batch script refer to the subinstance. For example, flux-run(1) would launch work there. A Flux command run from the script can be forced to refer to the enclosing instance by supplying the flux --parent option.

Flux commands outside of the batch script refer to their enclosing instance, often a system instance. flux-proxy(1) establishes a connection to a running subinstance by jobid, then spawns a shell in which Flux commands refer to the subinstance. For example:

$ flux uptime
 07:48:42 run 2.1d,  owner flux,  depth 0,  size 8
$ flux batch -N 2 --queue=batch mybatch.sh
ƒM7Zq9AKHno
$ flux proxy ƒM7Zq9AKHno
$ flux uptime
 07:47:18 run 1.6m,  owner user42,  depth 1,  size 2
$ flux top
...
$ exit
$

Other commands accept a jobid argument and establish the connection automatically. For example:

$ flux batch -N 2 --queue=batch mybatch.sh
ƒM7Zq9AKHno
$ flux top ƒM7Zq9AKHno
...
$

Batch scripts may contain submission directives denoted by flux: as described in RFC 36. See SUBMISSION DIRECTIVES below.

The available OPTIONS are detailed below.

JOB PARAMETERS

flux-batch(1) and flux-alloc(1) do not launch tasks directly, and therefore job parameters are specified in terms of resource slot size and number of slots. A resource slot can be thought of as the minimal resources required for a virtual task. The default slot size is 1 core.

-n, --nslots=N

Set the number of slots requested. This parameter is required unless --nodes is specified.

-c, --cores-per-slot=N

Set the number of cores to assign to each slot (default 1).

-g, --gpus-per-slot=N

Set the number of GPU devices to assign to each slot (default none).

-N, --nodes=N

Distribute allocated resource slots across N individual nodes.

-x, --exclusive

With --nodes, allocate nodes exclusively.

-q, --queue=NAME

Submit a job to a specific named queue. If a queue is not specified and queues are configured, then the jobspec will be modified at ingest to specify the default queue. If queues are not configured, then this option is ignored, though flux-jobs(1) may display the queue name in its rendering of the {queue} attribute.

-B, --bank=NAME

Set the bank name for this job to NAME. This option is equivalent to --setattr=bank=NAME, and results in the bank attribute being set to NAME in the submitted jobspec. However, besides the bank name appearing in job listing output, this option may have no effect if no plugin or package that supports it (such as flux-accounting) is installed and configured.

-t, --time-limit=MINUTES|FSD

Set a time limit for the job in either minutes or Flux standard duration (RFC 23). FSD is a floating point number with a single character units suffix ("s", "m", "h", or "d"). The default unit for the --time-limit option is minutes when no units are otherwise specified. If the time limit is unspecified, the job is subject to the system default time limit.

--job-name=NAME

Set an alternate job name for the job. If not specified, the job name will default to the command or script executed for the job.

--flags=FLAGS

Set comma separated list of job submission flags. The possible flags are waitable, novalidate, and debug. The waitable flag will allow the job to be waited on via flux job wait and similar API calls. The novalidate flag will inform flux to skip validation of a job's specification. This may be useful for high throughput ingest of a large number of jobs. Both waitable and novalidate require instance owner privileges. debug will output additional debugging into the job eventlog.

STANDARD I/O

For flux-batch(1) the default --output TEMPLATE is flux-{{id}}.out. To force output to KVS so it is available with flux job attach or flux-watch(1) , set the --output TEMPLATE to none or kvs.

Note

Paths specified in the options --output, --error, and --input will be opened relative to the working directory of the job on the nodes on which the job is running, not on the node from which the job is submitted.

--input=FILENAME|RANKS

Redirect stdin to the specified filename, bypassing the KVS. As a special case for flux run, the argument may specify an idset of task ranks in to which to direct standard input.

--output=TEMPLATE

Specify the filename TEMPLATE for stdout redirection, bypassing the KVS. TEMPLATE may be a mustache template. If the mustache template contains node or task-specific tags, then a different output file will be opened per node or task, respectively. See MUSTACHE TEMPLATES below for more information.

--error=TEMPLATE

Redirect stderr to the specified filename TEMPLATE, bypassing the KVS. TEMPLATE is expanded as described above.

-u, --unbuffered

Disable buffering of standard input and output as much as practical. Normally, stdout from job tasks is line buffered, as is stdin when running a job in the foreground via flux-run(1). Additionally, job output may experience a delay due to batching of output events by the job shell. With the --unbuffered option, output.*.buffer.type=none is set in jobspec to request no buffering of output, and the default output batch period is reduced greatly, to make output appear in the KVS and printed to the standard output of flux-run(1) as soon as possible. The --unbuffered option is also passed to flux job attach, which makes stdin likewise unbuffered. Note that the application and/or terminal may have additional input and output buffering which this option will not affect. For instance, by default an interactive terminal in canonical input mode will process input by line only. Likewise, stdout of a process run without a terminal may be fully buffered when using libc standard I/O streams (See NOTES in stdout(3)).

-l, --label-io

Add task rank prefixes to each line of output.

CONSTRAINTS

--requires=CONSTRAINT

Specify a set of allowable properties and other attributes to consider when matching resources for a job. The CONSTRAINT is expressed in a simple syntax described in RFC 35 (Constraint Query Syntax) which is then converted into a JSON object compliant with RFC 31 (Job Constraints Specification).

A constraint query string is formed by a series of terms.

A term has the form operator:operand, e.g. hosts:compute[1-10].

Terms may optionally be joined with boolean operators and parenthesis to allow the formation of more complex constraints or queries.

Boolean operators include logical AND (&, &&, or and), logical OR (|, ||, or or), and logical negation (not).

Terms separated by whitespace are joined with logical AND by default.

Quoting of terms is supported to allow whitespace and other reserved characters in operand, e.g. foo:'this is args'.

Negation is supported in front of terms such that -op:operand is equivalent to not op:operand. Negation is not supported in front of general expressions, e.g. -(a|b) is a syntax error.

The full specification of Constraint Query Syntax can be found in RFC 35.

Currently, --requires supports the following operators:

properties

Require the set of specified properties. Properties may be comma-separated, in which case all specified properties are required. As a convenience, if a property starts with ^ then a matching resource must not have the specified property. In these commands, the properties operator is the default, so that a,b is equivalent to properties:a,b.

hostlist

Require matching resources to be in the specified hostlist (in RFC 29 format). host or hosts is also accepted.

ranks

Require matching resources to be on the specified broker ranks in RFC 22 Idset String Representation.

Examples:

a b c, a&b&c, or a,b,c

Require properties a and b and c.

a|b|c, or a or b or c

Require property a or b or c.

(a and b) or (b and c)

Require properties a and b or b and c.

b|-c, b or not c

Require property b or not c.

host:fluke[1-5]

Require host in fluke1 through fluke5.

-host:fluke7

Exclude host fluke7.

rank:0

Require broker rank 0.

DEPENDENCIES

Note

Flux supports a simple but powerful job dependency specification in jobspec. See Flux Framework RFC 26 for more detailed information about the generic dependency specification.

Dependencies may be specified on the command line using the following options:

--dependency=URI

Specify a dependency of the submitted job using RFC 26 dependency URI format. The URI format is SCHEME:VALUE[?key=val[&key=val...]]. The URI will be converted into RFC 26 JSON object form and appended to the jobspec attributes.system.dependencies array. If the current Flux instance does not support dependency scheme SCHEME, then the submitted job will be rejected with an error message indicating this fact.

The --dependency option may be specified multiple times. Each use appends a new dependency object to the attributes.system.dependencies array.

The following dependency schemes are built-in:

Note

The after* dependency schemes listed below all require that the target JOBID be currently active or in the job manager's inactive job cache. If a target JOBID has been purged by the time the dependent job has been submitted, then the submission will be rejected with an error that the target job cannot be found.

Note

Most after* dependency schemes require that the target job enters the RUN state. Only afterany can be satisfied by a job that is canceled while pending, since it is satisfied after ANY outcome.

A helpful analogy: dependency chains behave like shell command separators. afterany is like ; (run regardless of result), while afternotok and afterexcept are like || (run only on failure). Just as a chain of || commands terminates when one succeeds, a chain of afternotok jobs should terminate when canceled, not continue. Similarly afterok behaves like &&: once any job fails the rest of the chain is canceled.

Note

The after scheme is deprecated. Use afterstart instead.

afterstart:JOBID

This dependency is satisfied after JOBID starts. The target job must have entered the RUN state.

after:JOBID

Deprecated alias for afterstart. Use afterstart instead.

afterany:JOBID

This dependency is satisfied after JOBID enters the INACTIVE state, regardless of the result. This is the only after* dependency that can be satisfied even if the target job is canceled before starting.

afterok:JOBID

This dependency is satisfied after JOBID enters the INACTIVE state with a successful result. The target job must have entered the RUN state.

afternotok:JOBID

This dependency is satisfied after JOBID enters the INACTIVE state with an unsuccessful result. The target job must have entered the RUN state.

afterexcept:JOBID

This dependency is satisfied when JOBID enters the INACTIVE state and a fatal job exception caused the transition to CLEANUP (e.g., node failure, timeout). The target job must have entered the RUN state.

singleton

This dependency is satisfied when there are no other active jobs of the same userid and job name which are not already held with a singleton dependency. That is, a singleton can be the only job with the same userid and job name in the SCHED state or later. The singleton scheme requires an explicit job name using the --job-name option.

begin-time:TIMESTAMP

This dependency is satisfied after TIMESTAMP, which is specified in floating point seconds since the UNIX epoch. See the --begin-time option below for a more user-friendly interface to the begin-time dependency.

In any of the above after* cases, if it is determined that the dependency cannot be satisfied (e.g. a job fails due to an exception with afterok), then a fatal exception of type=dependency is raised on the current job. An example of submitting a job, getting the Flux job identifier, and then submitting a job that depends on it (meaning it will wait for completion before starting) is provided below:

# Do some long work
jobid=$(flux submit -N2 sleep 200)

# Submit the dependent job
flux submit --dependency=afterany:$jobid /bin/bash my-script.sh

# Look at your queue
flux jobs -a

# Block and watch output
flux job attach $(flux job last)

ENVIRONMENT

By default, flux-batch(1) duplicates the current environment when submitting jobs. However, a set of environment manipulation options are provided to give fine control over the requested environment submitted with the job.

Note

The actual environment of the initial program is subject to the caveats described in the INITIAL PROGRAM ENVIRONMENT section of flux-environment(7).

--env=RULE

Control how environment variables are exported with RULE. See the ENV RULES section below for more information. Rules are applied in the order in which they are used on the command line. This option may be specified multiple times.

--env-remove=PATTERN

Remove all environment variables matching PATTERN from the current generated environment. If PATTERN starts with a / character, then it is considered a regex(7), otherwise PATTERN is treated as a shell glob(7). This option is equivalent to --env=-PATTERN and may be used multiple times.

--env-file=FILE

Read a set of environment RULES from a FILE. This option is equivalent to --env=^FILE and may be used multiple times.

ENV RULES

The ENVIRONMENT options allow control of the environment exported to jobs via a set of RULE expressions. The currently supported rules are

• If a rule begins with -, then the rest of the rule is a pattern which removes matching environment variables. If the pattern starts with /, it is a regex(7), optionally ending with /, otherwise the pattern is considered a shell glob(7) expression.

  Examples:

  -* or -/.*/ filter all environment variables creating an empty environment.

• If a rule begins with ^ then the rest of the rule is a filename from which to read more rules, one per line. The ~ character is expanded to the user's home directory.

  Examples:

  ~/envfile reads rules from file $HOME/envfile

• If a rule is of the form VAR=VAL, the variable VAR is set to VAL. Before being set, however, VAL will undergo simple variable substitution using the Python string.Template class. This simple substitution supports the following syntax:

  • $$ is an escape; it is replaced with $

  • $var will substitute var from the current environment, falling back to the process environment. An error will be thrown if environment variable var is not set.

  • ${var} is equivalent to $var

  • Advanced parameter substitution is not allowed, e.g. ${var:-foo} will raise an error.

  VAL may also contain a mustache template, in which case the template will be substituted in the job shell with the corresponding value before launching job tasks. See MUSTACHE TEMPLATES for more information.

  Examples:

  PATH=/bin, PATH=$PATH:/bin, FOO=${BAR}something, PATH=${PATH}:/{{tmpdir}}/bin

• Otherwise, the rule is considered a pattern from which to match variables from the process environment if they do not exist in the generated environment. E.g. PATH will export PATH from the current environment (if it has not already been set in the generated environment), and OMP* would copy all environment variables that start with OMP and are not already set in the generated environment. It is important to note that if the pattern does not match any variables, then the rule is a no-op, i.e. an error is not generated.

  Examples:

  PATH, FLUX_*_PATH, /^OMP.*/

Since we always starts with a copy of the current environment, the default implicit rule is * (or --env=*). To start with an empty environment instead, the -* rule or --env-remove=* option should be used. For example, the following will only export the current PATH to a job:

flux run --env-remove=* --env=PATH ...

Since variables can be expanded from the currently built environment, and --env options are applied in the order they are used, variables can be composed on the command line by multiple invocations of --env, e.g.:

flux run --env-remove=* \
              --env=PATH=/bin --env='PATH=$PATH:/usr/bin' ...

Note that care must be taken to quote arguments so that $PATH is not expanded by the shell.

This works particularly well when specifying rules in a file:

-*
OMP*
FOO=bar
BAR=${FOO}/baz

The above file would first clear the environment, then copy all variables starting with OMP from the current environment, set FOO=bar, and then set BAR=bar/baz.

PROCESS RESOURCE LIMITS

By default these commands propagate some common resource limits (as described in getrlimit(2)) to the job by setting the rlimit job shell option in jobspec. The set of resource limits propagated can be controlled via the --rlimit=RULE option:

--rlimit=RULE

Control how process resource limits are propagated with RULE. Rules are applied in the order in which they are used on the command line. This option may be used multiple times.

The --rlimit rules work similar to the --env option rules:

• If a rule begins with -, then the rest of the rule is a name or glob(7) pattern which removes matching resource limits from the set to propagate.

  Example:

  -* disables propagation of all resource limits.

• If a rule is of the form LIMIT=VALUE then LIMIT is explicitly set to VALUE. If VALUE is unlimited, infinity or inf, then the value is set to RLIM_INFINITY (no limit).

  Example:

  nofile=1024 overrides the current RLIMIT_NOFILE limit to 1024.

• Otherwise, RULE is considered a pattern from which to match resource limits and propagate the current limit to the job, e.g.

  --rlimit=memlock

  will propagate RLIMIT_MEMLOCK (which is not in the list of limits that are propagated by default).

We start with a default list of resource limits to propagate, then applies all rules specified via --rlimit on the command line. Therefore, to propagate only one limit, -* should first be used to start with an empty set, e.g. --rlimit=-*,core will only propagate the core resource limit.

The set of resource limits propagated by default includes all those except memlock, ofile, msgqueue, nice, rtprio, rttime, and sigpending. To propagate all possible resource limits, use --rlimit=*.

EXIT STATUS

The job exit status is normally the batch script exit status. This result is stored in the KVS.

OTHER OPTIONS

--cwd=DIRECTORY

Set job working directory.

--urgency=N

Specify job urgency. N has a range of 0 to 16 for guest users, 0 to 31 for instance owners, and a default value of 16. In addition to numerical values, the following special names are accepted:

hold (0)

Hold the job until the urgency is raised with flux job urgency.

default (16)

The default urgency for all users.

expedite (31)

Assign the highest possible priority to the job (restricted to instance owner).

Urgency is one factor used to calculate job priority, which affects the order in which the scheduler considers jobs. By default, priority is calculated from the urgency and the time elapsed since job submission. This calculation may be overridden by configuration. For example, in a multi-user Flux instance with the Flux accounting priority plugin loaded, the calculation includes other factors such as past usage and bank allocations.

A job with an urgency value of 0 is treated specially: it is never considered by the scheduler and is effectively held. Similarly, a job with an urgency of 31 is always assigned the maximum priority, regardless of other factors and is considered expedited.

flux jobs -o deps lists jobs with urgency and priority fields.

-v, --verbose

Increase verbosity on stderr. For example, currently flux run -v displays jobid, -vv displays job events, and -vvv displays exec events. flux alloc -v forces the command to print the submitted jobid on stderr. The specific output may change in the future.

-o, --setopt=KEY[=VAL]

Set shell option. Keys may include periods to denote hierarchy. VAL is optional and may be valid JSON (bare values, objects, or arrays), otherwise VAL is interpreted as a string. If VAL is not set, then the default value is 1. See SHELL OPTIONS below.

-S, --setattr=KEY[=VAL]

Set jobspec attribute. Keys may include periods to denote hierarchy. If KEY does not begin with system., user., or ., then system. is assumed. VAL is optional and may be valid JSON (bare values, objects, or arrays), otherwise VAL is interpreted as a string. If VAL is not set, then the default value is 1. If KEY starts with a ^ character, then VAL is interpreted as a file, which must be valid JSON, to use as the attribute value.

--add-file=[NAME[:PERMS]=]ARG

Add a file to the RFC 37 file archive in jobspec before submission. Both the file metadata and content are stored in the archive, so modification or deletion of a file after being processed by this option will have no effect on the job. If no NAME is provided, then ARG is assumed to be the path to a local file and the basename of the file will be used as NAME. Otherwise, if ARG contains a newline, then it is assumed to be the raw file data to encode. If PERMS is provided and is a valid octal integer, then this will override the default file permissions of 0600. The file will be extracted by the job shell into the job temporary directory and may be referenced as {{tmpdir}}/NAME on the command line, or $FLUX_JOB_TMPDIR/NAME in a batch script. This option may be specified multiple times to encode multiple files. Note: As documented in RFC 14, the file names script and conf.json are both reserved.

Note

This option should only be used for small files such as program input parameters, configuration, scripts, and so on. For broadcast of large files, binaries, and directories, the flux-shell(1) stage-in plugin will be more appropriate.

--begin-time=+FSD|DATETIME

Convenience option for setting a begin-time dependency for a job. The job is guaranteed to start after the specified date and time. If argument begins with a + character, then the remainder is considered to be an offset in Flux standard duration (RFC 23), otherwise, any datetime expression accepted by the Python parsedatetime module is accepted, e.g. 2021-06-21 8am, in an hour, tomorrow morning, etc.

--signal=SIG@TIME

Send signal SIG to job TIME before the job time limit. SIG can specify either an integer signal number or a full or abbreviated signal name, e.g. SIGUSR1 or USR1 or 10. TIME is specified in Flux Standard Duration, e.g. 30 for 30s or 1h for 1 hour. Either parameter may be omitted, with defaults of SIGUSR1 and 60s. For example, --signal=USR2 will send SIGUSR2 to the job 60 seconds before expiration, and --signal=@3m will send SIGUSR1 3 minutes before expiration. Note that if TIME is greater than the remaining time of a job as it starts, the job will be signaled immediately.

Note

A signal sent to a batch job is delivered to both the batch script and any running jobs. If the script does not handle the signal it may exit first, terminating the batch job early. It is recommended to use a trap in the script to allow running jobs to handle the signal and exit cleanly.

The default behavior is to not send any warning signal to jobs.

--dry-run

Don't actually submit job. Just emit jobspec on stdout and exit for run, submit, alloc, and batch. For bulksubmit, emit a line of output including relevant options for each job which would have been submitted,

--debug

Enable job debug events, primarily for debugging Flux itself. The specific effects of this option may change in the future.

--conf=FILE|KEY=VAL|STRING|NAME

The --conf option allows configuration for a Flux instance started via flux-batch(1) or flux-alloc(1) to be iteratively built on the command line. On first use, a conf.json entry is added to the internal jobspec file archive, and -c{{tmpdir}}/conf.json is added to the flux broker command line. Each subsequent use of the --conf option updates this configuration.

The argument to --conf may be in one of several forms:

• A multiline string, e.g. from a batch directive. In this case the string is parsed as JSON or TOML:

  # flux: --conf="""
  # flux: [resource]
  # flux: exclude = "0"
  # flux: """
  

• A string containing a = character, in which case the argument is parsed as KEY=VAL, where VAL is parsed as JSON, e.g.:

  --conf=resource.exclude=\"0\"
  

• A string ending in .json or .toml, in which case configuration is loaded from a JSON or TOML file.

• If none of the above conditions match, then the argument NAME is assumed to refer to a "named" config file NAME.toml or NAME.json within the following search path, in order of precedence:

  • XDG_CONFIG_HOME/flux/config or $HOME/.config/flux/config if XDG_CONFIG_HOME is not set

  • $XDG_CONFIG_DIRS/flux/config or /etc/xdg/flux/config if XDG_CONFIG_DIRS is not set. Note that XDG_CONFIG_DIRS may be a colon-separated path.

--bg

(alloc only) Do not interactively attach to the instance. Instead, print jobid on stdout once the instance is ready to accept jobs. The instance will run indefinitely until a time limit is reached, the job is canceled, or it is shutdown with flux shutdown JOBID (preferred). If a COMMAND is given then the job will run until COMMAND completes. Note that flux job attach JOBID cannot be used to interactively attach to the job (though it will print any errors or output).

--broker-opts=OPT

Pass specified options to the Flux brokers of the new instance. This option may be specified multiple times.

--wrap

(batch only) The --wrap option wraps the specified COMMAND and ARGS in a shell script, by prefixing with #!/bin/sh. If no COMMAND is present, then a SCRIPT is read on stdin and wrapped in a /bin/sh script.

--dump=[FILE]

When the job script is complete, archive the Flux instance's KVS content to FILE, which should have a suffix known to libarchive(3), and may be a mustache template as described above for --output. The content may be unarchived directly or examined within a test instance started with the flux-start --recovery option. If FILE is unspecified, flux-{{jobid}}-dump.tgz is used.

--quiet

Suppress logging of jobids to stdout

SHELL OPTIONS

Some options that affect the parallel runtime environment of the Flux instance started by flux batch are provided by the Flux shell. These options are described in detail in flux-shell-options(7). A list of the most commonly needed options follows.

Usage: flux batch -o NAME[=ARG].

Name	Description
cpu-affinity	Set task affinity to cores ([verbose,dry-run,]off|per-task|map:LIST|on)
gpu-affinity	Set task affinity to GPUs (off|per-task|map:LIST|on)
verbose	Increase shell log verbosity (1 or 2).
nosetpgrp	Don't run each task in its own process group.
pmi	Set PMI service(s) for launched programs (off|simple|LIST)
stage-in	Copy files previously mapped with flux-archive(1) to FLUX_JOB_TMPDIR.
pty.interactive	Enable a pty on rank 0 for flux job attach.
exit-timeout	Start fatal job exception timer after first task exits (none|FSD)
exit-on-error	Raise a fatal job exception immediately if first task exits with nonzero exit code.
hwloc.xmlfile	Write hwloc XML gathered by job to a file and set HWLOC_XMLFILE. Note that this option will also unset HWLOC_COMPONENTS if set, since presence of this environment variable may cause hwloc to ignore HWLOC_XMLFILE.
hwloc.restrict	With hwloc.xmlfile, restrict the exported topology XML to only those resources assigned to the current job. By default, the XML is not restricted.
input.limit	Set KVS input limit to SIZE bytes, where SIZE may be a floating point value including optional SI units: k, K, M, G. If this limit is exceeded, a fatal job exception is raised. This value is ignored if input is directed from a file with --input.
output.limit	Set KVS output limit to SIZE bytes, where SIZE may be a floating point value including optional SI units: k, K, M, G. This value is ignored if output is directed to a file with --output.
output.mode	Set the open mode for output files to either "truncate" or "append". The default is "truncate".

SUBMISSION DIRECTIVES

The flux batch command supports submission directives mixed within the submission script. The submission directive specification is fully detailed in RFC 36, but is summarized here for convenience:

• A submission directive is indicated by a line that starts with a prefix of zero or more non-alphanumeric characters followed by a tag FLUX: or flux:. The prefix plus tag is called the directive sentinel. E.g., in the example below the sentinel is # flux::

  #!/bin/sh
  # flux: -N4 -n16
  flux run -n16 hostname
  

• All directives in a file must use the same sentinel pattern, otherwise an error will be raised.

• Directives must be grouped together - it is an error to include a directive after any non-blank line that doesn't start with the common prefix.

• The directive starts after the sentinel to the end of the line.

• The # character is supported as a comment character within directives. For example:

  # flux: -N4 # this is a comment
  

• UNIX shell quoting is supported in directives.

• Triple quoted strings can be used to include newlines and quotes without further escaping. If a triple quoted string is used across multiple lines, then the opening and closing triple quotes must appear at the end of the line. For example

  # flux: --setattr=user.conf="""
  # flux: [config]
  # flux:   item = "foo"
  # flux: """
  

Language-Agnostic Design

Unlike batch systems that embed the directive marker within the comment syntax (e.g., #SBATCH), Flux separates these concerns. The prefix (#, --, %, // etc.) is part of your script's native comment syntax, while flux: marks the directive itself. This allows Flux submission directives to work in any scripting language or shell (Python, Lua, etc.) without a language-specific parser. This approach also allows submission directives to be used within multi-line comments or quotes, for example:

#!/usr/bin/env python3
#
"""
flux: -N2
flux: -n2
"""

Note that within multi-line comments or quoted strings, no comment prefix is needed since you're already in a comment context. Here the sentinel is simply flux: (with an empty prefix).

Because the prefix is script-language syntax (not part of the directive), changing the prefix breaks the sentinel pattern. For example, if your first directive is # flux:, you cannot use ## flux: later, as these are considered different sentinels.

Commenting Out Directives

To disable a directive without removing it, place # after the sentinel to comment out the arguments:

# flux: -N4              # active directive
# flux: # -N4            # disabled (commented out)
# flux: # testing nodes  # disabled with explanation

The # character comments out everything that follows on the line, making this a no-op directive while maintaining the same sentinel pattern. This works regardless of your script's native comment syntax because the # is interpreted by the directive parser, not the script language.

Examples in Different Languages

Shell script:

#!/bin/sh
# flux: -N4
# flux: # -N8  (disabled)

Python script:

#!/usr/bin/env python3
# flux: -N4
# flux: # -N8  (disabled)

Lua script:

#!/usr/bin/env lua
-- flux: -N4
-- flux: # -N8  (disabled)

Command Line Override

Submission directives set default command line options for flux batch. Options given on the command line override those in the submission script, e.g.:

$ flux batch --job-name=test-name --wrap <<-EOF
> #flux: -N4
> #flux: --job-name=name
> flux run -N4 hostname
> EOF
ƒ112345
$ flux jobs -no {name} ƒ112345
test-name

MUSTACHE TEMPLATES

Submission commands support a simplified version of mustache templates in select options (e.g., --output, --error, --dump), and job commands/arguments. These templates are replaced by job details once available. A mustache template is a tag in double braces, like {{tag}}.

The following is a list of currently supported tags:

Note

The template system in the job shell allows plugins to extend the set of supported mustache tags. Check with your installation of Flux to determine if extra tags are available locally.

Job mustache tags:

{{id}} or {{jobid}}

Expands to the current jobid in F58 encoding. If needed, an alternate encoding may be selected by using a subkey with the name of the desired encoding, e.g. {{id.dec}}. Supported encodings include f58 (the default), dec, hex, dothex, and words.

{{name}}

Expands to the current job name. If a name is not set for the job, then the basename of the command will be used.

{{nnodes}}

Expands to the number of nodes allocated to the job.

{{size}} or {{ntasks}}

Expands to the job size, or total number of tasks in the job.

Node-specific tags are prefixed with node., and support any of the keys present in the object returned by flux_shell_get_rank_info(3), these include:

{{node.id}}

Expands to the current node index within the job (0 - nnodes-1).

{{node.name}}

Expands to the hostname of the current node.

{{node.broker_rank}}

Expands to the broker rank of the enclosing instance on this node.

{{node.cores}}

Expands to the core id list assigned to the current node in idset form. For example, 0-3 or 1

{{node.gpus}}

Expands to the GPU id list assigned to the job on this node in idset form, e.g. 0-1 or 4

{{node.ncores}}

Expands to the number of cores allocated to the job on this node.

Task-specific tags are prefixed with task., with the special case of {{taskid}}, which is an alias for {{task.id}}.

{{task.id}}, {{task.rank}}, or {{taskid}}

Expands to the global rank for the current task.

{{task.index}} or {{task.localid}}

Expands to the local rank for the current task.

Other tags:

{{tmpdir}}

Expands to the path to the job temporary directory on the local system.

If an unknown mustache tag is encountered, an error message may be displayed, and the unknown template will appear in the result.

RESOURCES

Flux: http://flux-framework.org

Flux RFC: https://flux-framework.readthedocs.io/projects/flux-rfc

Issue Tracker: https://github.com/flux-framework/flux-core/issues

SEE ALSO

flux-submit(1), flux-run(1), flux-alloc(1), flux-bulksubmit(1) flux-environment(7)