5/Flux Broker Modules
This specification describes the broker extension modules used to implement Flux services.
Name: github.com/flux-framework/rfc/spec_5.rst
Editor: Jim Garlick <garlick@llnl.gov>
State: raw
Language
The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in RFC 2119.
Background
Flux services are implemented as dynamically loaded broker plugins called “broker modules”. They are “actors” in the sense that they have their own thread of control, and interact with the broker and the rest of Flux exclusively via messages.
The flux-module
front-end utility loads, unloads, and lists broker modules
by exchanging RPC messages with a module management component of the broker.
A broker module exports two symbols: a mod_main()
function, and
a mod_name
NULL-terminated string.
The broker starts a module by calling its mod_main()
function in
a new thread. The broker provides a broker handle and argv vector
style arguments to the via mod_main()
arguments. The arguments originate
on the flux-module load
command line.
Prior to calling mod_main()
, the broker registers a service for the
module based on the value of mod_name
. Request messages with
topic strings starting with this service name are diverted by the broker
to the module as described in RFC 3. The portion of the topic string
following the service name is called a “service method”. A module
may register many service methods.
The broker also pre-registers handlers for service methods that all modules are expected to provide, such as “ping” and “shutdown”. These handlers may be overridden by the module if desired.
The broker module implementing a new service is expected to register message handlers for its methods, then run the flux reactor. It should use event driven (reactive) programming techniques to remain responsive while juggling work from multiple clients.
Status messages are sent to the broker via pre-registered reactor
watchers to indicate when the module is initializing, running, finalizing,
or exited. At initialization, a module MAY also manually send a status
message to indicate to the broker when initialization is complete. This
provides synchronization to the broker module loader as well as useful
runtime debug information that can be reported by flux module list
.
Implementation
Well known Symbols
A broker module SHALL export the following global symbols:
const char \*mod_name;
A null-terminated C string defining the module name.
int mod_main (void \*context, int argc, char \**argv);
A C function that SHALL serve as the entry point for a thread of control. This function SHALL return zero to indicate success or -1 to indicate failure. The POSIX
errno
thread-specific variable SHOULD be set to indicate the type of error on failure.
Status Messages
A broker module SHALL be considered to be in one of the following states, represented by the integer values shown in parenthesis:
FLUX_MODSTATE_INIT (0) - initializing
FLUX_MODSTATE_RUNNING (1) - running
FLUX_MODSTATE_FINALIZING (2) - finalizing
FLUX_MODSTATE_EXITED (3) -
mod_main()
exited
Upon loading the module, the broker SHALL initialize the broker state
to FLUX_MODSTATE_INIT
.
After initialization is complete, a module SHALL send an RPC to the
broker.module-status
service with the FLUX_RPC_NORESPONSE flag to
notify the broker that the module has started successfully. In order to
ensure this happens for all modules, the RPC SHALL be sent via a
pre-registered reactor watcher upon a module’s first entry to the reactor
if the module has not already sent the message.
Example payload:
{
"status":1
}
After exiting the reactor and before exiting the module thread, the module
SHALL send an RPC to broker.module-status
indicating that it intends to
exit. The module SHALL wait for a response to this message before exiting
mod_main()
.
Example payload:
{
"status":2
}
Finally once mod_main()
has exited, the module thread SHALL send an RPC
to `broker.module-status
with the FLUX_RPC_NORESPONSE flag including
the error status of the module: zero if mod_main()
exited with a return
code greater than or equal to zero, otherwise the value of errno
.
{
"status":2,
"errnum":0
}
Load Sequence
The broker module loader SHALL launch the module’s mod_main()
in a
new thread. The broker.insmod
response is deferred until the module
state transitions out of FLUX_MODSTATE_INIT. If it transitions immediately to
FLUX_MODSTATE_EXITED, and the errnum
value is nonzero, an error response
SHALL be returned as described in RFC 3.
Unload Sequence
The broker module loader SHALL send a <service>.shutdown
request to the
module when the module loader receives a broker.rmmod
request for the
module. In response, the broker module SHALL exit mod_main()
, sending
state transition messages as described above, and exit the module’s thread
or process. The final state transition indicates to the broker that it MAY clean up the module thread.
Built-in Request Handlers
All broker modules receive default handlers for the following methods:
<service>.shutdown
The default handler immediately stops the reactor. This handler may be overridden if a broker module requires a more complex shutdown sequence.
<service>.stats.get
The default handler returns a JSON object containing message counts. This handler may be overridden if module-specific stats are available. The
flux-module stats
command sends this request and reports the result.<service>.stats.clear
The default handler zeroes message counts. This handler may be overridden if module-specific stats are available. The
flux-module stats --clear
sends this request.<service>.rusage
The default handler reports the result of
getrusage(RUSAGE_THREAD)
. Theflux-module rusage
sends this request and reports the result.<service>.ping
The default handler responds to the ping request. The
flux-ping
command performs ping RPCs.<service>.debug
The default handler manipulates the value of an integer stored in the module’s broker handle aux hash, under the key “flux::debug_flags”. The
flux-module debug
sends this request.
Built-in Event Handlers
In addition, all broker modules subscribe to and register a handler for the following events:
<service>.stats.clear
The default handler zeroes message counts. A custom handler may be registered for this event if module-specific stats are available. The
flux-module stats --clear-all
publishes this event.
Module Management Message Definitions
Module management messages SHALL follow the Flux message rules described in RFC 3 for requests and responses with JSON payloads.
The broker module loader SHALL implement the broker.insmod
,
broker.rmmod
, and broker.lsmod
methods.
Module management messages are described in detail by the following ABNF grammar:
MODULE = C:insmod-req S:insmod-rep
/ C:rmmod-req S:rmmod-rep
/ C:lsmod-req S:lsmod-rep
; Multi-part zeromq messages
C:insmod-req = [routing] insmod-topic insmod-json PROTO ; see below for JSON
S:insmod-rep = [routing] insmod-topic PROTO
C:rmmod-req = [routing] rmmod-topic rmmod-json PROTO ; see below for JSON
S:rmmod-rep = [routing] rmmod-topic PROTO
C:lsmod-req = [routing] lsmod-topic PROTO
S:lsmod-rep = [routing] lsmod-topic lsmod-json PROTO ; see below for JSON
; topic strings are optional service + module operation
insmod-topic = "broker.insmod"
rmmod-topic = "broker.rmmod"
lsmod-topic = "broker.lsmod"
; PROTO and [routing] are as defined in RFC 3.
JSON payloads for the above messages are as follows, described using JSON Content Rules
insmod-json {
"path" : string, ; path to module file
"args" : [ *: string ] ; argv array (first element is not special)
}
rmmod-json {
"name" : string, ; module name
}
lsmod-obj {
"name" : string ; module name
"size" : integer 0.. ; module file size
"digest" : string ; SHA1 digest of module file
"idle" : integer 0.. ; idle time in heartbeats
"status" : integer 0.. ; module state (enumerated above)
}
lsmod-json {
"mods" : [ *lsmod-obj ]
}