flux.sdexec.map module

Resource ID to systemd unit property mapping for sdexec.

This module maps Flux resource IDs (logical core and GPU indices) to systemd transient unit properties such as AllowedCPUs, AllowedMemoryNodes, and DeviceAllow.

Property values must be strings: the sdexec module rejects a mapper result whose values are not strings, failing the job before anything reaches systemd. Properties that take a list in systemd (such as DeviceAllow) are encoded as a comma-separated string.

Overview

ResourceMapper is the base class. Its map() method dispatches on resource type: for each key in the input dict it calls self.map_<type>(value) and merges the results. Supported resource types are defined by the methods present on the mapper class.

HwlocMapper is the built-in implementation. It uses hwloc topology data (passed as XML) to translate logical core IDs to CPU and NUMA-node sets, and logical GPU IDs to PCI addresses which are then resolved to /dev paths via sysfs.

GPU Device Discovery

HwlocMapper discovers GPU device nodes via sysfs and includes all device paths needed for compute workloads:

NVIDIA GPUs:

  • /dev/nvidia<N> — Main GPU device (N is the kernel Device Minor, read from /proc/driver/nvidia/gpus/<pci_addr>/information)

  • /dev/nvidiactl — Control device (shared across all GPUs)

  • /dev/nvidia-uvm — Unified Virtual Memory (required for CUDA)

  • /dev/nvidia-uvm-tools — UVM tools (optional)

  • /dev/dri/renderD<M> — DRM render node (alternative interface)

AMD GPUs:

  • /dev/dri/renderD<M> — DRM render node (primary ROCm interface)

  • /dev/dri/card<N> — DRM card device

  • /dev/kfd — Kernel Fusion Driver (required for ROCm/HIP, shared)

Discovery is opportunistic: if a device path doesn't exist, it's skipped. This makes the mapper work across different drivers and configurations. Shared devices like /dev/kfd and /dev/nvidiactl are automatically deduplicated when multiple GPUs are allocated to a job.

Extension

Override the GPU mapping only — the most common need. Subclass HwlocMapper and override map_gpus(). The core and NUMA-node mapping is inherited unchanged:

class MyGpuMapper(HwlocMapper):
    def map_gpus(self, gpus):
        # custom discovery, e.g. vendor-specific sysfs
        ...
        return {"DeviceAllow": "/dev/foo rw,/dev/bar rw"}

Add a new resource type — subclass HwlocMapper (or ResourceMapper) and add a map_<type> method. The dispatcher picks it up automatically when that key appears in the resources dict:

class FpgaMapper(HwlocMapper):
    def map_fpgas(self, fpgas):
        ...
        return {"DeviceAllow": "/dev/fpga0 rw,/dev/fpga1 rw"}

Replace the implementation entirely — subclass ResourceMapper directly and implement all required map_<type> methods. Point the sdexec broker module at your class via the [sdexec] mapper config key using the fully-qualified class name ("mypackage.mymodule.MyMapper").

Configuration

The sdexec broker module loads the mapper class named by the [sdexec] mapper TOML config key. The value must be a fully-qualified Python class name. When omitted, HwlocMapper is used.

The [sdexec] allowed-devices config key holds a list of device path globs under /dev that are unconditionally merged into DeviceAllow for every job, independent of its resource allocation. The sdexec-mapper module applies this allowlist to the mapper's output (see expand_allowed_devices() and merge_allowed_devices()), so it works with any mapper class regardless of its constructor.

class flux.sdexec.map.HwlocMapper(xml, rank=0)

Bases: ResourceMapper

hwloc-based resource mapper.

Uses hwloc topology XML to map logical core IDs to AllowedCPUs and AllowedMemoryNodes, and logical GPU IDs to DeviceAllow entries resolved via sysfs.

Parameters

  • xml -- hwloc topology XML string for the local node.

  • rank -- Local broker rank. Defaults to 0.

finalize_properties(properties, R, extra_properties=None)

Scale memory properties from extra_properties by the PU allocation ratio.

For each property in _SCALED_MEMORY_PROPS present in extra_properties, scales it by the ratio of allocated to total processing units on this node, provided AllowedCPUs was set by map_cores(). Absolute sizes and percentage values are both supported; "infinity" passes through unchanged.

map_cores(cores)

Map logical core IDs to AllowedCPUs and AllowedMemoryNodes.

Parameters

cores -- Idset string of logical core IDs, e.g. "0-3".

Returns

Dict with AllowedCPUs and AllowedMemoryNodes keys.

Raises

OSError -- If the mapping fails.

map_gpus(gpus)

Map logical GPU IDs to a DeviceAllow property.

An empty gpus string (no GPUs allocated) returns an empty dict. Device policy enforcement is handled by finalize_properties().

Parameters

gpus -- Idset string of logical GPU IDs, e.g. "0-1", or "" when no GPUs are allocated.

Returns

Dict with DeviceAllow key containing a comma-separated string of "<path> rw" entries, or empty dict when no GPUs are allocated.

Raises

OSError -- If the hwloc GPU PCI address lookup fails.

class flux.sdexec.map.ResourceMapper(rank=0)

Bases: object

Base class for resource ID to systemd unit property mappers.

Subclasses implement map_<type> methods for each resource type they support. map() extracts the local rank's resources from R and dispatches each resource type to the corresponding method.

Parameters

rank -- Local broker rank used to extract per-node resources from R. Defaults to 0.

finalize_properties(properties, R, extra_properties=None)

Hook to add or modify systemd unit properties.

Called after all resource-specific mapping is complete. Subclasses can override this to add arbitrary systemd unit properties not tied to specific resource types, such as resource accounting, limits, security settings, or conditional properties based on job characteristics.

The default implementation sets DevicePolicy=closed when properties are present (non-empty dict), ensuring device containment by allowing access to standard pseudo devices (/dev/null, /dev/zero, etc.) while blocking physical devices unless explicitly allowed via DeviceAllow (set by resource mappers like map_gpus()). The static sdexec.allowed-devices allowlist is applied separately by the sdexec-mapper module (see merge_allowed_devices()), not here.

An empty properties dict is left empty, preserving the ability for custom mappers to return {} to indicate unconstrained execution (no systemd unit properties applied). In practice, normal jobs will always have cores allocated and thus non-empty properties.

Parameters

  • properties -- Dict of properties generated by map_<type>() methods.

  • R -- The original ResourceSet object.

  • extra_properties -- Optional dict of additional systemd unit properties passed through from map(). Subclasses may inspect or scale these. The base implementation ignores it.

Returns

Dict of systemd unit property names to values.

Example

Override to add resource accounting:

class AccountingMapper(HwlocMapper):
    def finalize_properties(self, properties, R,
                            extra_properties=None):
        properties.update({
            "CPUAccounting": "true",
            "MemoryAccounting": "true",
        })
        return super().finalize_properties(
            properties, R, extra_properties=extra_properties
        )
map(R, extra_properties=None)

Map local resources from R to systemd unit properties.

Extracts the local rank's resources from R, then dispatches each resource type to map_<type>(idset_string), merging the results. Resource types with no corresponding method are silently skipped.

Parameters

  • R -- R JSON string, dict, or ResourceSet describing the job's allocated resources.

  • extra_properties -- Optional dict of additional systemd unit properties (e.g. from exec.sdexec-properties) that the mapper may use or scale. Passed through to finalize_properties(). None preserves existing behavior.

Returns

Dict of systemd unit property names to values.

flux.sdexec.map.expand_allowed_devices(patterns)

Expand configured device path globs to DeviceAllow entries.

Each entry in patterns is a path glob under /dev, optionally followed by whitespace and a systemd permissions field. Permissions default to "rw". Globs are expanded relative to /dev, so a pattern cannot escape upward, recursive "**" globs are rejected, and each result must be a char or block device node.

A pattern that matches nothing is silently ignored. This lets a site use one global allowed-devices policy across nodes with different hardware: a node simply grants the devices it has and skips the rest.

Parameters

patterns -- List of configured pattern strings.

Returns

Deduplicated list of "<path> <perms>" strings.

Raises

ValueError -- A pattern is malformed or not under /dev.

flux.sdexec.map.main(args=None)

CLI entry point: show systemd unit properties for given resource IDs.

flux.sdexec.map.merge_allowed_devices(properties, allowed_devices)

Merge a static device allowlist into a mapper's DeviceAllow property.

The allowlist is site policy applied by the sdexec-mapper module to the output of any mapper, so it lives here rather than in a mapper class: a custom ResourceMapper that never calls super().__init__ still gets the configured devices. Entries are merged with, not substituted for, any DeviceAllow set by the mapper's map_<type>() methods.

An empty properties dict is left untouched: a mapper returns {} to request unconstrained execution, and adding DeviceAllow would flip that to a constrained unit.

Parameters

  • properties -- Dict of systemd unit properties from a mapper's map().

  • allowed_devices -- List of pre-expanded "<path> <perms>" entries, as returned by expand_allowed_devices().

Returns

The properties dict, modified in place.