Batch Jobs¶

In Flux, a batch job is a script submitted along with a request for resources. When resources become available, the script is run as the initial program of a new single-user instance of Flux, with the batch user as instance owner. Thus, in contrast to a traditional batch scheduler, a Flux batch job has access to fully featured resource manager instance, and may not only serially execute work on allocated resources, but load custom modules or customize parameters of existing modules, monitor and update status of allocated resources, and even submit more batch jobs to run further subinstances of Flux.

Batch Command¶

The high-level Flux command that users can use to submit a batch script is flux batch.

sleep_batch.sh:

#!/bin/bash

echo "Starting my batch job"
echo "Print the resources allocated to this batch job"
flux resource info

echo "Use sleep to emulate a parallel program"
echo "Run the program at a total of 4 processes each requiring"
echo "1 core. These processes are equally spread across 2 nodes."
flux run -N 2 -n 4 sleep 120
flux run -N 2 -n 4 sleep 120

The name of your batch script is sleep_batch.sh. As shown from the flux run lines in this script, you can see the the overall resource requirement is 4 cores equally spread across two nodes as the default numbers of cores assigned to each task (specified by the -n option) is just one.

We now submit this script interactively three times using flux batch.

$ flux batch --nslots=2 --cores-per-slot=2 --nodes=2 ./sleep_batch.sh
$ flux batch --nslots=2 --cores-per-slot=2 --nodes=2 ./sleep_batch.sh
$ flux batch --nslots=2 --cores-per-slot=2 --nodes=2 ./sleep_batch.sh

Users must specify the overall resource requirement of each job using --nslots, which is the only required option, along with other optional options including those describing the resource shape of each slot and how the slots are distributed across different nodes. In this example, each job requests two slots each with 2 cores (--cores-per-slot=2) and these slots must be equally spread across two nodes (--nodes=4).

You can check the status of these batch jobs:

$ flux jobs -a
   JOBID USER     NAME       ST NTASKS NNODES  RUNTIME RANKS
ƒWZEQa8X dahn     sleep_batc  R      2      2   1.931s [0-1]
ƒW8eCV2o dahn     sleep_batc  R      2      2   2.896s [0-1]
ƒVhYLeJB dahn     sleep_batc  R      2      2   3.878s [0-1]

By default, the stdout/stderr of each batch job will be redirected to the flux-${JOBID}.out file and you can easily change the name of this file by passing --output=<FILE NAME1> and --error=<FILE NAME2>.

Checking the output file of one of the batch job:

$ flux job attach ƒWZEQa8X
0: stdout redirected to flux-ƒWZEQa8X.out
0: stderr redirected to flux-ƒWZEQa8X.out

$ cat flux-ƒWZEQa8X.out
Print the resources allocated to this batch job
2 Nodes, 4 Cores, 0 GPUs
Use sleep to emulate a parallel program
Run the program at a total of 4 processes each requiring
1 core. These processes are equally spread across 2 nodes.

Launching Flux in Slurm’s Batch Mode¶

Users may want to script the above procedures within a script to submit to another resource manager such Slurm.

An example sbatch script:

#!/bin/sh
#SBATCH -N 4

srun -N ${SLURM_NNODES} -n ${SLURM_NNODES} --mpi=none --mpibind=off flux start flux_batch.sh

flux_batch.sh:

#!/bin/sh
flux batch --nslots=2 --cores-per-slot=2 --nodes=2 ./sleep_batch.sh
flux batch --nslots=2 --cores-per-slot=2 --nodes=2 ./sleep_batch.sh
flux batch --nslots=2 --cores-per-slot=2 --nodes=2 ./sleep_batch.sh
flux queue drain

Blocking and Non-blocking Commands¶

It is important to note that some of the Flux commands used above are blocking and some of them are non-blocking.

Both flux submit and flux batch have submit semantics and as such they submit a parallel program or batch script and return shortly after. To avoid the exiting of the containing script, you can use flux queue drain which drains the queue such that no job can be submitted and then waits until all submitted jobs complete. Thus, it is recommended not to run those commands in background.

By contrast, flux run blocks until the target program completes.

Fluxion Scheduler¶

With our Fluxion graph-based scheduler, users can easily specialize their scheduling behaviors tailored to the characteristics of their workloads.

As an example, we will describe how you can set the queue and backfill policies of the submitting Flux to a simple policy named EASY while still keeping the policy of the nested Flux instances default: First Come First Served (FCFS).

$ salloc -N4 -ppdebug
salloc: Granted job allocation 5620626
$ cat sched-fluxion-qmanager.toml
[sched-fluxion-qmanager]

  queue-policy = "easy"

$ srun -N ${SLURM_NNODES} -n ${SLURM_NNODES} --pty --mpi=none --mpibind=off flux start -o,--config-path=./

sched-fluxion-qmanager is the one of the modules from Fluxion and sched-fluxion-qmanager.toml in the current working directory is our TOML configuration file that changes the queue/backfilling policy to EASY-backfilling. This backfill-capable queue policy can significantly increase the makespan of batch jobs.