flux.job.Jobspec module

class flux.job.Jobspec.Jobspec(resources, tasks, **kwargs)

Bases: object

add_file(path, data, perms=384, encoding=None)

Add a file to the RFC 14 "files" dictionary in Jobspec. If data contains newlines or an encoding is explicitly provided, then it is presumed to be the file content. Otherwise, data is a local filesystem path, the contents of which are to be loaded into jobspec. For filesystem

Parameters

path (str) -- path or file name to encode data as in Jobspec
data (dict, str) -- content of file or a local path to load
perms (int) -- file pemissions, default 0o0600 (octal). If data is a file system path, then permissions of the local file system object will be used.
encoding (str) -- RFC 37 compatible encoding for data. None if data is a dict or to determine encoding from a file when data specifies a filesystem path. O/w, if encoding set, data is a string encoded in specified encoding.

property attributes: Jobspec attributes section

property cwd: Working directory of job.

dumps(**kwargs)

property duration

Job's time limit.

The duration may be:

an int or float in seconds
a string in Flux Standard Duration (see RFC 23)
a python datetime.timedelta

A duration of zero is interpreted as "not set".

property environment: Environment of job. Defaults to None.

classmethod from_yaml_file(filename): Create a jobspec from a path to a YAML file.

classmethod from_yaml_stream(yaml_stream): Create a jobspec from a YAML file-like object.

getattr(key)

get attribute from jobspec using dotted key notation, e.g. system.duration or optionally attributes.system.duration.

Raises KeyError if a component of key does not exist.

property queue: Target queue of job submission

resource_counts()

Compute the counts of each resource type in the jobspec

The following jobspec would return { "slot": 12, "core": 18, "memory": 242 }

- type: slot
  count: 2
  with:
    - type: core
      count: 4
    - type: memory
      count: 1
      unit: GB
- type: slot
  count: 10
  with:
    - type: core
      count: 1
    - type: memory
      count: 24
      unit: GB

Note

the current implementation ignores the unit label and assumes they are consist across resources

resource_walk()

Traverse the resources in the resources section of the jobspec.

Performs a depth-first, pre-order traversal. Yields a tuple containing (parent, resource, count). parent is None when resource is a top-level resource. count is the number of that resource including the multiplicative effects of the with clause in ancestor resources. For example, the following resource section, will yield a count of 2 for the slot and a count of 8 for the core resource:

- type: slot
  count: 2
  with:
    - type: core
      count: 4

property resources: Jobspec resources section

setattr(key, val): set job attribute

setattr_shell_option(key, val): set job attribute: shell option

property stderr: Path to use for stderr.

property stdin: Path to use for stdin.

property stdout: Path to use for stdout.

property tasks: Jobspec tasks section

top_level_keys = {'attributes', 'resources', 'tasks', 'version'}

property version: Jobspec version section

class flux.job.Jobspec.JobspecV1(resources, tasks, **kwargs)

Bases: Jobspec

classmethod from_batch_command(script, jobname, args=None, num_slots=1, cores_per_slot=1, gpus_per_slot=None, num_nodes=None, broker_opts=None, exclusive=False, conf=None)

Create a Jobspec describing a nested Flux instance controlled by a script.

The nested Flux instance will execute the script with the given command-line arguments after copying it and setting the executable bit. Conceptually, this differs from the from_nest_command, which also creates a nested Flux instance, in that it a) requires the initial program of the new instance to be an executable text file and b) creates the initial program from a string rather than using an executable existing somewhere on the filesystem.

Use setters to assign additional properties.

Parameters

script (str) -- contents of the script to execute, as a string. The script should have a shebang (e.g. #!/bin/sh) at the top.
jobname (str) -- name to use for system.job.name attribute This will be the default job name reported by Flux.
args (iterable of str) -- arguments to pass to script
num_slots (int) -- number of resource slots to create. Slots are an abstraction, and are only used (along with cores_per_slot and gpus_per_slot) to determine the nested instance's allocation size and layout.
cores_per_slot (int) -- number of cores to allocate per slot
gpus_per_slot (int) -- number of GPUs to allocate per slot
num_nodes (int) -- distribute allocated resource slots across N individual nodes
broker_opts (iterable of str) -- options to pass to the new Flux broker
conf (dict) -- optional broker configuration to pass to the child instance brokers. If set, conf will be set in the jobspec 'files' (RFC 37 File Archive) attribute as conf.json, and broker_opts will be extended to add -c{{tmpdir}}/conf.json

classmethod from_command(command, num_tasks=1, cores_per_task=1, gpus_per_task=None, num_nodes=None, exclusive=False)

Factory function that builds the minimum legal v1 jobspec.

Use setters to assign additional properties.

Parameters

command (iterable of str) -- command to execute
num_tasks -- number of MPI tasks to create
cores_per_task -- number of cores to allocate per task
gpus_per_task -- number of GPUs to allocate per task
num_nodes -- distribute allocated tasks across N individual nodes

classmethod from_nest_command(command, num_slots=1, cores_per_slot=1, gpus_per_slot=None, num_nodes=None, broker_opts=None, exclusive=False, conf=None)

Create a Jobspec describing a nested Flux instance controlled by command.

Conceptually, this differs from the from_batch_command method in that a) the initial program of the nested Flux instance can be any executable on the file system, not just a text file and b) the executable is not copied at submission time.

Use setters to assign additional properties.

Parameters

command (iterable of str) -- initial program for the nested Flux
instance --
num_slots (int) -- number of resource slots to create. Slots are an abstraction, and are only used (along with cores_per_slot and gpus_per_slot) to determine the nested instance's allocation size and layout.
cores_per_slot (int) -- number of cores to allocate per slot
gpus_per_slot (int) -- number of GPUs to allocate per slot
num_nodes (int) -- distribute allocated resource slots across N individual nodes
broker_opts (iterable of str) -- options to pass to the new Flux broker
conf (dict) -- optional broker configuration to pass to the child instance brokers. If set, conf will be set in the jobspec 'files' (RFC 37 File Archive) attribute as conf.json, and broker_opts will be extended to add -c{{tmpdir}}/conf.json

classmethod per_resource(command, ncores=None, nnodes=None, per_resource_type=None, per_resource_count=None, gpus_per_node=None, exclusive=False)

Factory function that builds a v1 jobspec from an explicit count of nodes or cores and a number of tasks per one of these resources.

Use setters to assign additional properties.

Parameters

ncores -- Total number of cores to allocate
nnodes -- Total number of nodes to allocate
per_resource_type -- (optional) Type of resource over which to schedule a count of tasks. Only "node" or "core" are currently supported.
per_resource_count -- (optional) Count of tasks per per_resource_type
gpus_per_node -- With nnodes, request a number of gpus per node
exclusive -- with nnodes, request whole nodes exclusively

flux.job.Jobspec.validate_jobspec(jobspec, require_version=None)

Validates the jobspec by attempting to construct a Jobspec object. If no exceptions are thrown during construction, then the jobspec is assumed to be valid and this function returns True. If the jobspec is invalid, the relevant exception is thrown (i.e., TypeError, ValueError, EnvironmentError)

By default, the validation function will read the version key in the jobspec to determine which Jobspec object to instantiate. An optional require_version is included to override this behavior and force a particular class to be used.

Parameters

jobspec -- a Jobspec object or JSON string
require_version -- jobspec version to use, if not provided, the value of jobspec['version'] is used

Raises

ValueError --
TypeError --
EnvironmentError --