flux-submit(1)
SYNOPSIS
flux submit [OPTIONS] [--ntasks=N] COMMAND...
DESCRIPTION
flux submit enqueues a job to run under Flux and prints its numerical jobid on standard output. The job consists of N copies of COMMAND launched together as a parallel job.
If --ntasks
is unspecified, a value of N=1 is assumed.
The available OPTIONS are detailed below.
JOB PARAMETERS
These commands accept only the simplest parameters for expressing the size of the parallel program and the geometry of its task slots:
Common resource options
These commands take the following common resource allocation options:
- -N, --nodes=N
Set the number of nodes to assign to the job. Tasks will be distributed evenly across the allocated nodes, unless the per-resource options (noted below) are used with submit, run, or bulksubmit. It is an error to request more nodes than there are tasks. If unspecified, the number of nodes will be chosen by the scheduler.
- -x, --exclusive
Indicate to the scheduler that nodes should be exclusively allocated to this job. It is an error to specify this option without also using
--nodes
. If--nodes
is specified without--nslots
or--ntasks
, then this option will be enabled by default and the number of tasks or slots will be set to the number of requested nodes.
Per-task options
flux-run(1), flux-submit(1) and flux-bulksubmit(1) take two sets of mutually exclusive options to specify the size of the job request. The most common form uses the total number of tasks to run along with the amount of resources required per task to specify the resources for the entire job:
- -n, --ntasks=N
Set the number of tasks to launch (default 1).
- -c, --cores-per-task=N
Set the number of cores to assign to each task (default 1).
- -g, --gpus-per-task=N
Set the number of GPU devices to assign to each task (default none).
Per-resource options
The second set of options allows an amount of resources to be specified with the number of tasks per core or node set on the command line. It is an error to specify any of these options when using any per-task option listed above:
- --cores=N
Set the total number of cores.
- --tasks-per-node=N
Set the number of tasks per node to run.
- --tasks-per-core=N
Force a number of tasks per core. Note that this will run N tasks per allocated core. If nodes are exclusively scheduled by configuration or use of the
--exclusive
flag, then this option could result in many more tasks than expected. The default for this option is effectively 1, so it is useful only for oversubscribing tasks to cores for testing purposes. You probably don't want to use this option.
Additional job options
These commands also take following job parameters:
- -q, --queue=NAME
Submit a job to a specific named queue. If a queue is not specified and queues are configured, then the jobspec will be modified at ingest to specify the default queue. If queues are not configured, then this option is ignored, though flux-jobs(1) may display the queue name in its rendering of the
{queue}
attribute.
- -B, --bank=NAME
Set the bank name for this job to
NAME
. This option is equivalent to --setattr=bank=NAME, and results in thebank
attribute being set toNAME
in the submitted jobspec. However, besides the bank name appearing in job listing output, this option may have no effect if no plugin or package that supports it (such as flux-accounting) is installed and configured.
- -t, --time-limit=MINUTES|FSD
Set a time limit for the job in either minutes or Flux standard duration (RFC 23). FSD is a floating point number with a single character units suffix ("s", "m", "h", or "d"). The default unit for the
--time-limit
option is minutes when no units are otherwise specified. If the time limit is unspecified, the job is subject to the system default time limit.
- --job-name=NAME
Set an alternate job name for the job. If not specified, the job name will default to the command or script executed for the job.
- --flags=FLAGS
Set comma separated list of job submission flags. The possible flags are
waitable
,novalidate
, anddebug
. Thewaitable
flag will allow the job to be waited on viaflux job wait
and similar API calls. Thenovalidate
flag will inform flux to skip validation of a job's specification. This may be useful for high throughput ingest of a large number of jobs. Bothwaitable
andnovalidate
require instance owner privileges.debug
will output additional debugging into the job eventlog.
STANDARD I/O
By default, task stdout and stderr streams are redirected to the
KVS, where they may be accessed with the flux job attach
command.
In addition, flux-run(1) processes standard I/O in real time, emitting the job's I/O to its stdout and stderr.
Note
Paths specified in the options --output
, --error
,
and --input
will be opened relative to the working directory
on the first node of the job allocation, not on the node from which
the job is submitted.
- --input=FILENAME|RANKS
Redirect stdin to the specified filename, bypassing the KVS. As a special case for
flux run
, the argument may specify an idset of task ranks in to which to direct standard input.
- --output=TEMPLATE
Specify the filename TEMPLATE for stdout redirection, bypassing the KVS. TEMPLATE may be a mustache template which supports the following tags:
- {{id}} or {{jobid}}
Expands to the current jobid in the F58 encoding. If needed, an alternate encoding may be selected by using a subkey with the name of the desired encoding, e.g. {{id.dec}}. Supported encodings include f58 (the default), dec, hex, dothex, and words.
- {{name}}
Expands to the current job name. If a name is not set for the job, then the basename of the command will be used.
- --error=TEMPLATE
Redirect stderr to the specified filename TEMPLATE, bypassing the KVS. TEMPLATE is expanded as described above.
- -u, --unbuffered
Disable buffering of standard input and output as much as practical. Normally, stdout from job tasks is line buffered, as is stdin when running a job in the foreground via flux-run(1). Additionally, job output may experience a delay due to batching of output events by the job shell. With the
--unbuffered
option,output.*.buffer.type=none
is set in jobspec to request no buffering of output, and the default output batch period is reduced greatly, to make output appear in the KVS and printed to the standard output of flux-run(1) as soon as possible. The--unbuffered
option is also passed toflux job attach
, which makes stdin likewise unbuffered. Note that the application and/or terminal may have additional input and output buffering which this option will not affect. For instance, by default an interactive terminal in canonical input mode will process input by line only. Likewise, stdout of a process run without a terminal may be fully buffered when using libc standard I/O streams (See NOTES in stdout(3)).
- -l, --label-io
Add task rank prefixes to each line of output.
CONSTRAINTS
- --requires=CONSTRAINT
Specify a set of allowable properties and other attributes to consider when matching resources for a job. The CONSTRAINT is expressed in a simple syntax described in RFC 35 (Constraint Query Syntax) which is then converted into a JSON object compliant with RFC 31 (Job Constraints Specification).
A constraint query string is formed by a series of terms.
A term has the form
operator:operand
, e.g.hosts:compute[1-10]
.Terms may optionally be joined with boolean operators and parenthesis to allow the formation of more complex constraints or queries.
Boolean operators include logical AND (
&
,&&
, orand
), logical OR (|
,||
, oror
), and logical negation (not
).Terms separated by whitespace are joined with logical AND by default.
Quoting of terms is supported to allow whitespace and other reserved characters in operand, e.g.
foo:'this is args'
.Negation is supported in front of terms such that
-op:operand
is equivalent tonot op:operand
. Negation is not supported in front of general expressions, e.g.-(a|b)
is a syntax error.The full specification of Constraint Query Syntax can be found in RFC 35.
Currently,
--requires
supports the following operators:- properties
Require the set of specified properties. Properties may be comma-separated, in which case all specified properties are required. As a convenience, if a property starts with
^
then a matching resource must not have the specified property. In these commands, theproperties
operator is the default, so thata,b
is equivalent toproperties:a,b
.- hostlist
Require matching resources to be in the specified hostlist (in RFC 29 format).
host
orhosts
is also accepted.- ranks
Require matching resources to be on the specified broker ranks in RFC 22 Idset String Representation.
Examples:
a b c
,a&b&c
, ora,b,c
Require properties a and b and c.
a|b|c
, ora or b or c
Require property a or b or c.
(a and b) or (b and c)
Require properties a and b or b and c.
b|-c
,b or not c
Require property b or not c.
host:fluke[1-5]
Require host in fluke1 through fluke5.
-host:fluke7
Exclude host fluke7.
rank:0
Require broker rank 0.
DEPENDENCIES
Note
Flux supports a simple but powerful job dependency specification in jobspec. See Flux Framework RFC 26 for more detailed information about the generic dependency specification.
Dependencies may be specified on the command line using the following options:
- --dependency=URI
Specify a dependency of the submitted job using RFC 26 dependency URI format. The URI format is SCHEME:VALUE[?key=val[&key=val...]]. The URI will be converted into RFC 26 JSON object form and appended to the jobspec
attributes.system.dependencies
array. If the current Flux instance does not support dependency scheme SCHEME, then the submitted job will be rejected with an error message indicating this fact.The
--dependency
option may be specified multiple times. Each use appends a new dependency object to theattributes.system.dependencies
array.
The following dependency schemes are built-in:
Note
The after*
dependency schemes listed below all require that the
target JOBID be currently active or in the job manager's inactive job
cache. If a target JOBID has been purged by the time the dependent job
has been submitted, then the submission will be rejected with an error
that the target job cannot be found.
- after:JOBID
This dependency is satisfied after JOBID starts.
- afterany:JOBID
This dependency is satisfied after JOBID enters the INACTIVE state, regardless of the result
- afterok:JOBID
This dependency is satisfied after JOBID enters the INACTIVE state with a successful result.
- afternotok:JOBID
This dependency is satisfied after JOBID enters the INACTIVE state with an unsuccessful result.
- begin-time:TIMESTAMP
This dependency is satisfied after TIMESTAMP, which is specified in floating point seconds since the UNIX epoch. See the
--begin-time
option below for a more user-friendly interface to thebegin-time
dependency.
In any of the above after*
cases, if it is determined that the
dependency cannot be satisfied (e.g. a job fails due to an exception
with afterok), then a fatal exception of type=dependency is raised
on the current job. An example of submitting a job, getting the Flux
job identifier, and then submitting a job that depends on it (meaning
it will wait for completion before starting) is provided below:
# Do some long work
jobid=$(flux submit -N2 sleep 200)
# Submit the dependent job
flux submit --dependency=afterany:$jobid /bin/bash my-script.sh
# Look at your queue
flux jobs -a
# Block and watch output
flux job attach $(flux job last)
ENVIRONMENT
By default, these commands duplicate the current environment when submitting jobs. However, a set of environment manipulation options are provided to give fine control over the requested environment submitted with the job.
- --env=RULE
Control how environment variables are exported with RULE. See the ENV RULES section below for more information. Rules are applied in the order in which they are used on the command line. This option may be specified multiple times.
- --env-remove=PATTERN
Remove all environment variables matching PATTERN from the current generated environment. If PATTERN starts with a
/
character, then it is considered a regex(7), otherwise PATTERN is treated as a shell glob(7). This option is equivalent to--env=-PATTERN
and may be used multiple times.
- --env-file=FILE
Read a set of environment RULES from a FILE. This option is equivalent to
--env=^FILE
and may be used multiple times.
ENV RULES
The ENVIRONMENT options allow control of the environment exported to jobs via a set of RULE expressions. The currently supported rules are
If a rule begins with
-
, then the rest of the rule is a pattern which removes matching environment variables. If the pattern starts with/
, it is a regex(7), optionally ending with/
, otherwise the pattern is considered a shell glob(7) expression.
- Examples:
-*
or-/.*/
filter all environment variables creating an empty environment.If a rule begins with
^
then the rest of the rule is a filename from which to read more rules, one per line. The~
character is expanded to the user's home directory.
- Examples:
~/envfile
reads rules from file$HOME/envfile
If a rule is of the form
VAR=VAL
, the variableVAR
is set toVAL
. Before being set, however,VAL
will undergo simple variable substitution using the Pythonstring.Template
class. This simple substitution supports the following syntax:
$$
is an escape; it is replaced with$
$var
will substitutevar
from the current environment, falling back to the process environment. An error will be thrown if environment variablevar
is not set.
${var}
is equivalent to$var
Advanced parameter substitution is not allowed, e.g.
${var:-foo}
will raise an error.
- Examples:
PATH=/bin
,PATH=$PATH:/bin
,FOO=${BAR}something
Otherwise, the rule is considered a pattern from which to match variables from the process environment if they do not exist in the generated environment. E.g.
PATH
will exportPATH
from the current environment (if it has not already been set in the generated environment), andOMP*
would copy all environment variables that start withOMP
and are not already set in the generated environment. It is important to note that if the pattern does not match any variables, then the rule is a no-op, i.e. an error is not generated.
- Examples:
PATH
,FLUX_*_PATH
,/^OMP.*/
Since we always starts with a copy of the current environment,
the default implicit rule is *
(or --env=*
). To start with an
empty environment instead, the -*
rule or --env-remove=*
option
should be used. For example, the following will only export the current
PATH
to a job:
flux run --env-remove=* --env=PATH ...
Since variables can be expanded from the currently built environment, and
--env
options are applied in the order they are used, variables can
be composed on the command line by multiple invocations of --env
,
e.g.:
flux run --env-remove=* \
--env=PATH=/bin --env='PATH=$PATH:/usr/bin' ...
Note that care must be taken to quote arguments so that $PATH
is not
expanded by the shell.
This works particularly well when specifying rules in a file:
-*
OMP*
FOO=bar
BAR=${FOO}/baz
The above file would first clear the environment, then copy all variables
starting with OMP
from the current environment, set FOO=bar
,
and then set BAR=bar/baz
.
PROCESS RESOURCE LIMITS
By default these commands propagate some common resource limits (as described
in getrlimit(2)) to the job by setting the rlimit
job shell
option in jobspec. The set of resource limits propagated can be controlled
via the --rlimit=RULE
option:
- --rlimit=RULE
Control how process resource limits are propagated with RULE. Rules are applied in the order in which they are used on the command line. This option may be used multiple times.
The --rlimit
rules work similar to the --env
option rules:
If a rule begins with
-
, then the rest of the rule is a name or glob(7) pattern which removes matching resource limits from the set to propagate.
- Example:
-*
disables propagation of all resource limits.If a rule is of the form
LIMIT=VALUE
then LIMIT is explicitly set to VALUE. If VALUE isunlimited
,infinity
orinf
, then the value is set toRLIM_INFINITY
(no limit).
- Example:
nofile=1024
overrides the currentRLIMIT_NOFILE
limit to 1024.Otherwise, RULE is considered a pattern from which to match resource limits and propagate the current limit to the job, e.g.
--rlimit=memlock
will propagate
RLIMIT_MEMLOCK
(which is not in the list of limits that are propagated by default).
We start with a default list of resource limits to propagate,
then applies all rules specified via --rlimit
on the command line.
Therefore, to propagate only one limit, -*
should first be used to
start with an empty set, e.g. --rlimit=-*,core
will only propagate
the core
resource limit.
The set of resource limits propagated by default includes all those except
memlock
, ofile
, msgqueue
, nice
, rtprio
, rttime
,
and sigpending
. To propagate all possible resource limits, use
--rlimit=*
.
JOB ENVIRONMENT VARIABLES
The job environment is described in more detail in the flux-environment(7) JOB ENVIRONMENT section. A summary of the most commonly used variables is provided below:
Name |
Description |
---|---|
the current jobid |
|
the number of tasks in the current job |
|
the total number of nodes hosting tasks on behalf of the job |
|
the global task rank (zero origin) |
|
the local task rank (zero origin) |
|
path to temporary, per-job directory, usually in |
|
the URI of the enclosing Flux instance |
EXIT STATUS
The job exit status, normally the largest task exit status, is stored in the KVS. If one or more tasks are terminated with a signal, the job exit status is 128+signo.
The flux-job attach
command exits with the job exit status.
In addition, flux-run(1) runs until the job completes and exits with the job exit status.
OTHER OPTIONS
- --cwd=DIRECTORY
Set job working directory.
- --urgency=N
Specify job urgency. N has a range of 0 to 16 for guest users, 0 to 31 for instance owners, and a default value of 16. In addition to numerical values, the following special names are accepted:
- hold (0)
Hold the job until the urgency is raised with
flux job urgency
.- default (16)
The default urgency for all users.
- expedite (31)
Assign the highest possible priority to the job (restricted to instance owner).
Urgency is one factor used to calculate job priority, which affects the order in which the scheduler considers jobs. By default, priority is calculated from the urgency and the time elapsed since job submission. This calculation may be overridden by configuration. For example, in a multi-user Flux instance with the Flux accounting priority plugin loaded, the calculation includes other factors such as past usage and bank allocations.
A job with an urgency value of 0 is treated specially: it is never considered by the scheduler and is effectively held. Similarly, a job with an urgency of 31 is always assigned the maximum priority, regardless of other factors and is considered expedited.
flux jobs -o deps
lists jobs with urgency and priority fields.
- -v, --verbose
Increase verbosity on stderr. For example, currently
flux run -v
displays jobid,-vv
displays job events, and-vvv
displays exec events.flux alloc -v
forces the command to print the submitted jobid on stderr. The specific output may change in the future.
- -o, --setopt=KEY[=VAL]
Set shell option. Keys may include periods to denote hierarchy. VAL is optional and may be valid JSON (bare values, objects, or arrays), otherwise VAL is interpreted as a string. If VAL is not set, then the default value is 1. See SHELL OPTIONS below.
- -S, --setattr=KEY[=VAL]
Set jobspec attribute. Keys may include periods to denote hierarchy. If KEY does not begin with
system.
,user.
, or.
, thensystem.
is assumed. VAL is optional and may be valid JSON (bare values, objects, or arrays), otherwise VAL is interpreted as a string. If VAL is not set, then the default value is 1. If KEY starts with a^
character, then VAL is interpreted as a file, which must be valid JSON, to use as the attribute value.
- --add-file=[NAME=]ARG
Add a file to the RFC 37 file archive in jobspec before submission. Both the file metadata and content are stored in the archive, so modification or deletion of a file after being processed by this option will have no effect on the job. If no
NAME
is provided, thenARG
is assumed to be the path to a local file and the basename of the file will be used asNAME
. Otherwise, ifARG
contains a newline, then it is assumed to be the raw file data to encode. The file will be extracted by the job shell into the job temporary directory and may be referenced as{{tmpdir}}/NAME
on the command line, or$FLUX_JOB_TMPDIR/NAME
in a batch script. This option may be specified multiple times to encode multiple files. Note: As documented in RFC 14, the file namesscript
andconf.json
are both reserved.Note
This option should only be used for small files such as program input parameters, configuration, scripts, and so on. For broadcast of large files, binaries, and directories, the flux-shell(1)
stage-in
plugin will be more appropriate.
- --begin-time=+FSD|DATETIME
Convenience option for setting a
begin-time
dependency for a job. The job is guaranteed to start after the specified date and time. If argument begins with a+
character, then the remainder is considered to be an offset in Flux standard duration (RFC 23), otherwise, any datetime expression accepted by the Python parsedatetime module is accepted, e.g.2021-06-21 8am
,in an hour
,tomorrow morning
, etc.
- --signal=SIG@TIME
Send signal
SIG
to jobTIME
before the job time limit.SIG
can specify either an integer signal number or a full or abbreviated signal name, e.g.SIGUSR1
orUSR1
or10
.TIME
is specified in Flux Standard Duration, e.g.30
for 30s or1h
for 1 hour. Either parameter may be omitted, with defaults ofSIGUSR1
and 60s. For example,--signal=USR2
will sendSIGUSR2
to the job 60 seconds before expiration, and--signal=@3m
will sendSIGUSR1
3 minutes before expiration. Note that ifTIME
is greater than the remaining time of a job as it starts, the job will be signaled immediately.The default behavior is to not send any warning signal to jobs.
- --dry-run
Don't actually submit job. Just emit jobspec on stdout and exit for
run
,submit
,alloc
, andbatch
. Forbulksubmit
, emit a line of output including relevant options for each job which would have been submitted,
- --debug
Enable job debug events, primarily for debugging Flux itself. The specific effects of this option may change in the future.
- --taskmap=SCHEME[:VALUE]
Choose an alternate method for mapping job task IDs to nodes of the job. The job shell maps tasks using a "block" distribution scheme by default (consecutive tasks share nodes) This option allows the activation of alternate schemes by name, including an optional VALUE. Supported schemes which are built in to the job shell include
- cyclic[:N]
Tasks are distributed over consecutive nodes with a stride of N (where N=1 by default).
- manual:TASKMAP
An explicit RFC 34 taskmap is provided and used to manually map task ids to nodes. The provided TASKMAP must match the total number of tasks in the job and the number of tasks per node assigned by the job shell, so this option is not useful unless the total number of nodes and tasks per node are known at job submission time.
- hostfile:FILE
Assign tasks in order to hosts as they appear in FILE. FILE should have one or more lines each of which contains a host name or RFC 29 Hostlist string. Each host assigned to the job must appear in the hostfile and be assigned the same number of tasks as the default taskmap from the shell. If there are less hosts in the hostfile than tasks in the job, then the list of hosts will be reused.
However, shell plugins may provide other task mapping schemes, so check the current job shell configuration for a full list of supported taskmap schemes.
- --cc=IDSET
(submit,bulksubmit) Replicate the job for each
id
inIDSET
.FLUX_JOB_CC
will be set toid
in the environment of each submitted job to allow the job to alter its execution based on the submission index. (e.g. for reading from a different input file). When using--cc
, the substitution string{cc}
may be used in options and commands and will be replaced by the currentid
.
- --bcc=IDSET
(submit,bulksubmit) Identical to
--cc
, but do not setFLUX_JOB_CC
in each job. All jobs will be identical copies. As with--cc
,{cc}
in option arguments and commands will be replaced with the currentid
.
- --quiet
Suppress logging of jobids to stdout
- --log=FILE
(submit,bulksubmit) Log command output and stderr to
FILE
instead of the terminal. If a replacement (e.g.{}
or{cc}
) appears inFILE
, then one or more output files may be opened. For example, to save all submitted jobids into separate files, use:flux submit --cc=1-4 --log=job{cc}.id hostname
- --log-stderr=FILE
(submit,bulksubmit) Separate stderr into
FILE
instead of sending it to the terminal or aFILE
specified by--log
.
- --wait
(submit,bulksubmit) Wait on completion of all jobs before exiting. This is equivalent to
--wait-event=clean
.
- --wait-event=NAME
Wait until job or jobs have received event
NAME
before exiting. E.g. to submit a job and block until the job begins running, use--wait-event=start
. (submit,bulksubmit only) IfNAME
begins withexec.
, then wait for an event in the exec eventlog, e.g.exec.shell.init
. Forflux run
the argument to this option when used is passed directly toflux job attach
.
- --progress
(submit,bulksubmit) With
--wait
, display a progress bar showing the progress of job completion. Without--wait
, the progress bar will show progress of job submission.
- --jps
(submit,bulksubmit) With
--progress
, display throughput statistics (jobs/s) in the progress bar.
- --define=NAME=CODE
(bulksubmit) Define a named method that will be made available as an attribute during command and option replacement. The string being processed is available as
x
. For example:$ seq 1 8 | flux bulksubmit --define=pow="2**int(x)" -n {.pow} ...
- --shuffle
(bulksubmit) Shuffle the list of commands before submission.
- --sep=STRING
(bulksubmit) Change the separator for file input. The default is to separate files (including stdin) by newline. To separate by consecutive whitespace, specify
--sep=none
.
SHELL OPTIONS
Some options that affect the parallel runtime environment are provided by the Flux shell. These options are described in detail in the SHELL OPTIONS section of flux-shell(1). A list of the most commonly needed options follows.
Usage: flux submit -o NAME[=ARG]
.
Name |
Description |
---|---|
Set task affinity to cores ( |
|
Set task affinity to GPUs ( |
|
Increase shell log verbosity (1 or 2). |
|
Don't run each task in its own process group. |
|
Set PMI service(s) for launched programs ( |
|
Copy files previously mapped with flux-archive(1) to
|
|
Enable a pty on rank 0 for flux job attach. |
|
Start fatal job exception timer after first task exits
( |
|
Raise a fatal job exception immediately if first task exits with nonzero exit code. |
|
Write hwloc XML gathered by job to a file and set |
|
With |
|
Set KVS output limit to SIZE bytes, where SIZE may be a floating point
value including optional SI units: k, K, M, G. This value is ignored
if output is directed to a file with |
|
Set the open mode for output files to either "truncate" or "append". The default is "truncate". |
RESOURCES
Flux: http://flux-framework.org
Flux RFC: https://flux-framework.readthedocs.io/projects/flux-rfc
Issue Tracker: https://github.com/flux-framework/flux-core/issues
SEE ALSO
flux-run(1), flux-alloc(1), flux-batch(1), flux-bulksubmit(1), flux-environment(7)