42/Subprocess Server Protocol

The subprocess server protocol is used for execution, monitoring, and standard I/O management of remote processes.

Name	github.com/flux-framework/rfc/spec_42.rst
Editor	Jim Garlick <garlick@llnl.gov>
State	raw

Language

The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in RFC 2119.

Background

The subprocess server protocol is implemented in two distinct Flux components:

Name	Service Name	Notes
Flux broker	rexec	Always available
sdexec broker module	sdexec	If systemd support is configured

The primary use cases are:

The job execution service runs job shells on job nodes.
The job manager perilog plugin runs prolog/epilog scripts on job nodes.
The instance owner runs arbitrary processes with flux exec.

In a multi-user Flux instance where a user transition is necessary in order for the instance owner to run commands with the credentials of a guest user, the subprocess server delegates this to the IMP. On its own, the subprocess server can only run jobs with the credentials of the process it is embedded within (the broker, for example). For more detail, refer to RFC 15.

Goals

Run a command with configurable path, arguments, environment, and working directory.
Launch the command directly, without a remote shell.
Monitor the command for completion or error.
Forward standard I/O.
Optionally forward additional I/O channels.
Provide signal delivery capability.
Protect against unauthorized use.

Implementation

exec

The streaming exec RPC creates a new subprocess. Payloads are are defined as follows:

exec request

The request SHALL consist of a JSON object with the following keys:

cmd

(object, REQUIRED) An object that defines the command.

See Command Object below.

flags

(integer, REQUIRED) A bitfield comprised of zero or more flags:

stdout (1): Forward standard output to the client.
stderr (2): Forward standard error to the client.
channel (4): Forward auxiliary channel output to the client.

Several response types are distinguished by the type key:

exec started response

The remote process has been started.

The response SHALL consist of a JSON object with the following keys:

type: (string, REQUIRED) The response type with a value of started.

pid: (integer, REQUIRED) The remote process ID returned from the UNIX fork() system call.

exec stopped response

The remote process has been stopped due to a SIGSTOP signal. The response SHALL consist of a JSON object with the following keys:

type: (string, REQUIRED) The response type with a value of stopped.

Note

No response is generated if the process continues with SIGCONT.

exec finished response

The remote process is no longer running. The response SHALL consist of a JSON object with the following keys:

type: (string, REQUIRED) The response type with a value of finished.

status: (integer, REQUIRED) The UNIX wait() status value.

exec output response

The remote process has produced output. The response SHALL consist of a JSON object with the following keys:

type: (string, REQUIRED) The response type with a value of output.

io

(object, REQUIRED) Output data from the process.

See I/O Object below.

exec error response

The exec response stream SHALL be terminated by an error response per RFC 6, with ENODATA (61) indicating success.

Failure of the remote command SHALL be indicated in finished response and SHALL NOT result in an error response. Other errors, such as an ENOENT error from the execvp() system call SHALL result in an error response.

write

The write RPC sends data to an I/O channel of a remote process. Valid I/O channel names MAY include stdin and auxiliary channel names specified in the exec request command object.

write request

The request SHALL consist of a JSON object with the following keys:

pid: (integer, REQUIRED) The process ID of the remote process.

io

(object, REQUIRED) Input data for the process.

See I/O Object below.

This request receives no response, thus the request message SHOULD set FLUX_MSGFLAG_NORESPONSE. Write Requests to invalid channel names MAY be ignored by the subprocess server.

kill

The kill RPC sends a signal to a remote process.

kill request

The request SHALL consist of a JSON object with the following keys:

pid: (integer, REQUIRED) The process ID of the remote process.

signum: (integer, REQUIRED) The signal number.

kill response: The successful response SHALL contain no payload.

Command Object

The subprocess server command object SHALL consist of a JSON object with the following keys:

cwd

(string, OPTIONAL) The current working directory.

If unspecified, the server working directory SHALL be used.

cmdline

(array of string, REQUIRED) The command and its arguments.

The array SHALL have at least one element.

env

(object, REQUIRED) A set of key-value pairs that define the command’s environment.

All values SHALL be of type string.

opts

(object, REQUIRED) A set of key-value pairs that set subprocess options.

All values SHALL be of type string.

Options are implementation dependent and are not specified here.

channels

(array of string, REQUIRED) A list of I/O channel names.

A socketpair SHALL be created for each channel and one end passed to the subprocess in an environment variable whose name is the same as the channel name.

I/O Object

The subprocess server io object is identical to the RFC 24 Data Event context.

It SHALL consist of a JSON object with the following keys:

stream: (string, REQUIRED) The stream name such as stdout, stderr.

rank: (string, REQUIRED) An RFC 22 idset describing the source rank(s).

data: (string, OPTIONAL) Output data, encoded per encoding.

encoding

(string, OPTIONAL) Encoding type for data.

Possible values:

UTF-8: Encode as a UTF-8 string.
base64: Encode as a base64 string

If not present, UTF-8 is assumed.

eof: (boolean, OPTIONAL) EOF indicator for stream.

Example

exec request

{
  "cmd": {
    "cwd": "/home/test",
    "cmdline": [
      "hostname"
    ],
    "env": {
      "PATH": "/bin:/usr/bin:/home/test/bin",
    },
    "opts": {},
    "channels": []
  },
  "flags": 3
}

exec responses

{
  "type": "started",
  "pid": 1848495
}

{
  "type": "output",
  "pid": 1848495,
  "io": {
    "stream": "stdout",
    "rank": "0",
    "data": "system76-pc\n"
  }
}

{
  "type": "output",
  "pid": 1848495,
  "io": {
    "stream": "stderr",
    "rank": "0",
    "eof": true
  }
}

{
  "type": "output",
  "pid": 1848495,
  "io": {
    "stream": "stdout",
    "rank": "0",
    "eof": true
  }
}

{
  "type": "finished",
  "status": 0
}

42/Subprocess Server Protocol

Language

Related Standards

Background

Goals

Implementation

exec

write

kill

Command Object

I/O Object

Example

exec request

exec responses