5/Flux Broker Modules
This specification describes the broker extension modules used to implement Flux services.
Name |
github.com/flux-framework/rfc/spec_5.rst |
Editor |
Jim Garlick <garlick@llnl.gov> |
State |
raw |
Language
The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in RFC 2119.
Background
Flux services are implemented as dynamically loaded broker plugins called “broker modules”. They are “actors” in the sense that they have their own thread of control, and interact with the broker and the rest of Flux exclusively via messages.
The flux-module
front-end utility loads, unloads, and lists broker modules
by exchanging RPC messages with a module management component of the broker.
A broker module exports a function named mod_main()
that is the starting
point for a new broker thread. The function accepts a flux_t
handle and
an argc, argv style argument list. The flux_t
handle provides direct
communication with the broker over shared memory. The argument list is derived
from the free arguments on the flux module load
command line. When
mod_main()
returns, the thread is terminated and the module is unloaded.
Prior to calling the function entry point, the broker registers the module
name as a service and arranges for request messages with topics matching the
service to be routed to the module, as described in RFC 3. Importantly,
messages may be sent to this service immediately upon completion of the
flux module load
command. Handlers for a few standard methods such as
“ping” and “shutdown” are automatically registered for the module. These
handlers may be overridden by the module if desired.
The broker module implementing a new service is expected to register message handlers for its methods, then run the flux reactor. It should use event driven (reactive) programming techniques to remain responsive while juggling work from multiple clients.
Status messages are sent to the broker via pre-registered reactor
watchers to indicate when the module is initializing, running, finalizing,
or exited. At initialization, a module MAY also manually send a status
message to indicate to the broker when initialization is complete. This
provides synchronization to the broker module loader as well as useful
runtime debug information that can be reported by flux module list
.
Implementation
Well known Symbols
A broker module SHALL export the following global symbol:
int mod_main (flux_t *h, int argc, char **argv);
A C function that SHALL serve as the entry point for a thread of control. This function SHALL return zero to indicate success or -1 to indicate failure. The POSIX
errno
thread-specific variable SHOULD be set to indicate the type of error on failure.
Status Messages
A broker module SHALL be considered to be in one of the following states, represented by the integer values shown in parenthesis:
FLUX_MODSTATE_INIT (0) - initializing
FLUX_MODSTATE_RUNNING (1) - running
FLUX_MODSTATE_FINALIZING (2) - finalizing
FLUX_MODSTATE_EXITED (3) -
mod_main()
exited
Upon loading the module, the broker SHALL initialize the broker state
to FLUX_MODSTATE_INIT
.
After initialization is complete, a module SHALL send an RPC to the
module.status
service with the FLUX_RPC_NORESPONSE flag to
notify the broker that the module has started successfully. In order to
ensure this happens for all modules, the RPC SHALL be sent via a
pre-registered reactor watcher upon a module’s first entry to the reactor
if the module has not already sent the message.
Example payload:
{
"status":1
}
After exiting the reactor and before exiting the module thread, the module
SHALL send an RPC to module.status
indicating that it intends to
exit. The module SHALL wait for a response to this message before exiting
mod_main()
.
Example payload:
{
"status":2
}
Finally once mod_main()
has exited, the module thread SHALL send an RPC
to `module.status
with the FLUX_RPC_NORESPONSE flag including
the error status of the module: zero if mod_main()
exited with a return
code greater than or equal to zero, otherwise the value of errno
.
{
"status":2,
"errnum":0
}
Load Sequence
The broker module loader SHALL launch the module’s mod_main()
in a
new thread. The module.load
response is deferred until the module
state transitions out of FLUX_MODSTATE_INIT. If it transitions immediately to
FLUX_MODSTATE_EXITED, and the errnum
value is nonzero, an error response
SHALL be returned as described in RFC 3.
Unload Sequence
The broker module loader SHALL send a <service>.shutdown
request to the
module when the module loader receives a module.remove
request for the
module. In response, the broker module SHALL exit mod_main()
, sending
state transition messages as described above, and exit the module’s thread
or process. The final state transition indicates to the broker that it MAY
clean up the module thread.
Built-in Request Handlers
All broker modules receive default handlers for the following methods:
<service>.shutdown
The default handler immediately stops the reactor. This handler may be overridden if a broker module requires a more complex shutdown sequence.
<service>.stats-get
The default handler returns a JSON object containing message counts. This handler may be overridden if module-specific stats are available. The
flux-module stats
command sends this request and reports the result.<service>.stats-clear
The default handler zeroes message counts. This handler may be overridden if module-specific stats are available. The
flux-module stats --clear
sends this request.<service>.rusage
The default handler reports the result of
getrusage(RUSAGE_THREAD)
. Theflux-module rusage
sends this request and reports the result.<service>.ping
The default handler responds to the ping request. The
flux-ping
command performs ping RPCs.<service>.debug
The default handler manipulates the value of an integer stored in the module’s broker handle aux hash, under the key “flux::debug_flags”. The
flux-module debug
sends this request.
Built-in Event Handlers
In addition, all broker modules subscribe to and register a handler for the following pub/sub events:
<service>.stats-clear
The default handler zeroes message counts. A custom handler may be registered for this event if module-specific stats are available. The
flux-module stats --clear-all
publishes this event.
Module Attributes
The following key-value pairs SHALL be provided to broker modules via the
flux_t
handle AUX container:
- flux::uuid
The UUID assigned to the module which is used for message routing, in string form.
- flux::name
The module name. This is usually derived from basename of the module’s shared object file, minus the
.so
extension. However it may also be overridden by request at module load time.
Multiple Loading
A properly conditioned broker module MAY be loaded more than once under different names. The following constraints SHOULD be considered:
The service registered on behalf of the module is based on its name, therefore any message handlers for the module’s default service MUST be registered with a matching topic string. This may be accomplished by using the
flux::name
attribute to build matching topic strings, or using topic string wildcards.There are no safeguards against loading improperly conditioned modules multiple times. A module MAY prevent multiple loading by checking for an expected value of
flux::name
.
Service-specific constraints SHOULD be considered as well.