Python Broker Modules

Flux broker modules extend the broker with new services. Traditionally modules are written in C as dynamic shared objects, but Flux also supports modules written in Python. A Python broker module is a .py file that defines either a mod_main() entry function or a single BrokerModule subclass.

Loading a Python module

Python modules are loaded and removed with the same flux-module(1) commands used for C modules:

$ flux module load /path/to/mymod.py
$ flux ping mymod
$ flux module remove mymod

The module name defaults to the file's basename without the .py suffix and may be overridden with --name:

$ flux module load --name myservice /path/to/mymod.py

The mod_main entry point

A Python broker module may define a top-level function named mod_main():

def mod_main(h, *args):
    ...

h is a flux.Flux handle connected to the broker. args is a (possibly empty) tuple of strings passed after the module path on the flux module load command line. The function must run the reactor (directly or via BrokerModule) and return only after the reactor exits.

Defining mod_main() explicitly is optional when using BrokerModule — see Using BrokerModule below.

Using BrokerModule

BrokerModule is a base class that handles the boilerplate of registering message handlers, managing the reactor, and looking up the module name. Subclass it and use the request_handler() and event_handler() decorators to declare handlers:

from flux.brokermod import BrokerModule, event_handler, request_handler


class MyModule(BrokerModule):

    @request_handler("info")
    def info(self, msg):
        self.handle.respond(msg, {"name": self.name})

    @event_handler("shutdown", prefix=False)
    def on_shutdown(self, msg):
        self.stop()

When a module file contains exactly one BrokerModule subclass and no mod_main(), the loader synthesizes the entry point automatically as MyModule(h, *args).run(), so no explicit mod_main() is required. If multiple subclasses are present, mod_main() must be defined to resolve the ambiguity:

def mod_main(h, *args):
    MyModule(h, *args).run()

Handler methods receive a Message object. For request handlers, call flux.Flux.respond() on the handle to send a response.

By default each handler's topic is prefixed with the module name, so @request_handler("info") registers mymod.info. Pass prefix=False to use the topic string as-is.

By default, only requests from the instance owner are delivered to a handler. Pass allow_guest=True to also accept requests from non-owner users:

@request_handler("status", allow_guest=True)
def status(self, msg):
    self.handle.respond(msg, {"running": True})

The handle property returns the flux.Flux handle, giving full access to the Flux Python API for RPCs, KVS operations, timers, and other watchers.

Logging

BrokerModule provides a BrokerLogger instance as self.log. Use the convenience methods to emit log messages without importing syslog:

self.log.info("module started")
self.log.error(f"handler failed: {exc}")

The full set of methods mirrors syslog severity levels: debug, info, notice, warning, error, critical, alert, and emerg. Log messages appear in flux-dmesg(1) and the broker's stderr.

By default all messages are forwarded to the ring buffer and the broker's global log level acts as the filter — consistent with C module behaviour. Subclasses may tighten the threshold by setting self.log.level:

self.log.level = "info"   # suppress debug messages

Error handling

If run() raises OSError, or if mod_main() raises any exception, the broker treats the module as having exited with an error and logs a module runtime failure message. Use stop_error() to signal an error from within a handler.

Module arguments

Arguments passed after the module path at load time are available as args in mod_main() and as args on a BrokerModule instance:

$ flux module load /path/to/mymod.py --verbose --count=4

def mod_main(h, *args):
    # args == ('--verbose', '--count=4')
    MyModule(h, *args).run()

Arguments are split on whitespace; quoting is not supported.

API reference

class flux.brokermod.BrokerModule(h, *args)

Base class for Python broker modules.

Subclass this and use the request_handler() and event_handler() decorators to register message handlers, then call run() to start the reactor. Handler topics are automatically prefixed with the module name unless prefix=False is passed to the decorator.

Example:

from flux.brokermod import BrokerModule, event_handler, request_handler

class MyModule(BrokerModule):

    @request_handler("info")
    def info(self, msg):
        self.handle.respond(msg, self.name)

    @event_handler("panic")
    def panic(self, msg):
        self.stop_error()

When this is the only BrokerModule subclass in the file and no mod_main() is defined, the loader synthesizes the entry point automatically. An explicit mod_main() is only needed to resolve ambiguity when multiple subclasses are present.

property args

Tuple of module arguments passed at load time.

debug_test(flag, clear=False)

Test a module debug bit, optionally clearing it.

Returns True if the bit identified by flag is set. If clear is True the bit is atomically cleared after being read. Bits are set externally via flux module debug --setbit N <module-name>.

property handle

The underlying flux.Flux handle.

property name

The module name as assigned by the broker.

run()

Run the reactor until stopped.

Raises OSError if the reactor was stopped with an error (e.g. via stop_error()) or if a Python exception was set on the handle during a callback.

set_running()

Signal to the broker that the module is now running.

Call this before performing synchronous initialization in run() to allow flux module load to return instead of waiting for the reactor to start. See flux_module_set_running(3).

stop()

Stop the reactor and exit run() normally.

stop_error()

Stop the reactor and cause run() to raise OSError.

flux.brokermod.request_handler(topic, prefix=True, allow_guest=False)

Decorator to register a method as a request handler.

The handler is invoked with (self, msg) when a request matching <module_name>.<topic> is received. If prefix=False, the topic is used as-is without prepending the module name. If allow_guest=True, requests from non-owner users (FLUX_ROLE_USER) are also accepted.

flux.brokermod.event_handler(topic, prefix=True)

Decorator to register a method as an event handler.

The handler is invoked with (self, msg) when an event matching <module_name>.<topic> is received. If prefix=False, the topic is used as-is without prepending the module name.