flux.sdexec.map module

Resource ID to systemd unit property mapping for sdexec.

This module maps Flux resource IDs (logical core and GPU indices) to systemd transient unit properties such as AllowedCPUs, AllowedMemoryNodes, and AllowedDevices.

Overview

ResourceMapper is the base class. Its map() method dispatches on resource type: for each key in the input dict it calls self.map_<type>(value) and merges the results. Supported resource types are defined by the methods present on the mapper class.

HwlocMapper is the built-in implementation. It uses hwloc topology data (passed as XML) to translate logical core IDs to CPU and NUMA-node sets, and logical GPU IDs to PCI addresses which are then resolved to /dev paths via sysfs.

GPU Device Discovery

HwlocMapper discovers GPU device nodes via sysfs and includes all device paths needed for compute workloads:

NVIDIA GPUs:

  • /dev/nvidia<N> — Main GPU device (N is the kernel Device Minor, read from /proc/driver/nvidia/gpus/<pci_addr>/information)

  • /dev/nvidiactl — Control device (shared across all GPUs)

  • /dev/nvidia-uvm — Unified Virtual Memory (required for CUDA)

  • /dev/nvidia-uvm-tools — UVM tools (optional)

  • /dev/dri/renderD<M> — DRM render node (alternative interface)

AMD GPUs:

  • /dev/dri/renderD<M> — DRM render node (primary ROCm interface)

  • /dev/dri/card<N> — DRM card device

  • /dev/kfd — Kernel Fusion Driver (required for ROCm/HIP, shared)

Discovery is opportunistic: if a device path doesn't exist, it's skipped. This makes the mapper work across different drivers and configurations. Shared devices like /dev/kfd and /dev/nvidiactl are automatically deduplicated when multiple GPUs are allocated to a job.

Extension

Override the GPU mapping only — the most common need. Subclass HwlocMapper and override map_gpus(). The core and NUMA-node mapping is inherited unchanged:

class MyGpuMapper(HwlocMapper):
    def map_gpus(self, gpus):
        # custom discovery, e.g. vendor-specific sysfs
        ...
        return {"AllowedDevices": [...]}

Add a new resource type — subclass HwlocMapper (or ResourceMapper) and add a map_<type> method. The dispatcher picks it up automatically when that key appears in the resources dict:

class FpgaMapper(HwlocMapper):
    def map_fpgas(self, fpgas):
        ...
        return {"AllowedDevices": [...]}

Replace the implementation entirely — subclass ResourceMapper directly and implement all required map_<type> methods. Point the sdexec broker module at your class via the [sdexec] mapper config key using the fully-qualified class name ("mypackage.mymodule.MyMapper").

Configuration

The sdexec broker module loads the mapper class named by the [sdexec] mapper TOML config key. The value must be a fully-qualified Python class name. When omitted, HwlocMapper is used.

class flux.sdexec.map.HwlocMapper(xml, rank=0)

Bases: ResourceMapper

hwloc-based resource mapper.

Uses hwloc topology XML to map logical core IDs to AllowedCPUs and AllowedMemoryNodes, and logical GPU IDs to AllowedDevices entries resolved via sysfs.

Parameters

  • xml -- hwloc topology XML string for the local node.

  • rank -- Local broker rank. Defaults to 0.

finalize_properties(properties, R, extra_properties=None)

Scale memory properties from extra_properties by the PU allocation ratio.

For each property in _SCALED_MEMORY_PROPS present in extra_properties, scales it by the ratio of allocated to total processing units on this node, provided AllowedCPUs was set by map_cores(). Absolute sizes and percentage values are both supported; "infinity" passes through unchanged.

map_cores(cores)

Map logical core IDs to AllowedCPUs and AllowedMemoryNodes.

Parameters

cores -- Idset string of logical core IDs, e.g. "0-3".

Returns

Dict with AllowedCPUs and AllowedMemoryNodes keys.

Raises

OSError -- If the mapping fails.

map_gpus(gpus)

Map logical GPU IDs to a DeviceAllow property.

An empty gpus string (no GPUs allocated) returns an empty dict. Device policy enforcement is handled by finalize_properties().

Parameters

gpus -- Idset string of logical GPU IDs, e.g. "0-1", or "" when no GPUs are allocated.

Returns

Dict with DeviceAllow key containing a comma-separated string of "<path> rw" entries, or empty dict when no GPUs are allocated.

Raises

OSError -- If the hwloc GPU PCI address lookup fails.

class flux.sdexec.map.ResourceMapper(rank=0)

Bases: object

Base class for resource ID to systemd unit property mappers.

Subclasses implement map_<type> methods for each resource type they support. map() extracts the local rank's resources from R and dispatches each resource type to the corresponding method.

Parameters

rank -- Local broker rank used to extract per-node resources from R. Defaults to 0.

finalize_properties(properties, R, extra_properties=None)

Hook to add or modify systemd unit properties.

Called after all resource-specific mapping is complete. Subclasses can override this to add arbitrary systemd unit properties not tied to specific resource types, such as resource accounting, limits, security settings, or conditional properties based on job characteristics.

The default implementation sets DevicePolicy=closed when properties are present (non-empty dict), ensuring device containment by allowing access to standard pseudo devices (/dev/null, /dev/zero, etc.) while blocking physical devices unless explicitly allowed via DeviceAllow (set by resource mappers like map_gpus()).

An empty properties dict is left empty, preserving the ability for custom mappers to return {} to indicate unconstrained execution (no systemd unit properties applied). In practice, normal jobs will always have cores allocated and thus non-empty properties.

Parameters

  • properties -- Dict of properties generated by map_<type>() methods.

  • R -- The original ResourceSet object.

  • extra_properties -- Optional dict of additional systemd unit properties passed through from map(). Subclasses may inspect or scale these. The base implementation ignores it.

Returns

Dict of systemd unit property names to values.

Example

Override to add resource accounting:

class AccountingMapper(HwlocMapper):
    def finalize_properties(self, properties, R,
                            extra_properties=None):
        properties.update({
            "CPUAccounting": "true",
            "MemoryAccounting": "true",
        })
        return super().finalize_properties(
            properties, R, extra_properties=extra_properties
        )
map(R, extra_properties=None)

Map local resources from R to systemd unit properties.

Extracts the local rank's resources from R, then dispatches each resource type to map_<type>(idset_string), merging the results. Resource types with no corresponding method are silently skipped.

Parameters

  • R -- R JSON string, dict, or ResourceSet describing the job's allocated resources.

  • extra_properties -- Optional dict of additional systemd unit properties (e.g. from exec.sdexec-properties) that the mapper may use or scale. Passed through to finalize_properties(). None preserves existing behavior.

Returns

Dict of systemd unit property names to values.

flux.sdexec.map.main(args=None)

CLI entry point: show systemd unit properties for given resource IDs.