Slurm, in depth.

§01What Slurm is, and why it dominates HPC

SLURM — Simple Linux Utility for Resource Management — began at Lawrence Livermore National Laboratory in 2002 as a lightweight replacement for proprietary HPC batch systems. It is now maintained primarily by SchedMD and has become the de facto standard for HPC and AI supercomputers. Of the TOP500 supercomputers, the large majority run Slurm. Every major sovereign-AI cluster, most academic and national-lab HPC systems, and many private AI labs use it as their workload manager.

What a workload manager does

A workload manager sits between users (who have programs to run) and the hardware (which is finite). Its job has four parts:

• Accept a description of what resources a job needs and what it should run.
• Schedule — decide when the job should run and on which nodes.
• Execute — launch the job's processes on the allocated nodes, with the right environment, isolation, and resource constraints.
• Account — track what ran, on what, for how long, by whom.

Everything else Slurm does — preemption, fair-share, QoS, reservations, topology, GRES — is in service of one of those four.

Why HPC adopted Slurm, not Kubernetes

This is the historical question the role keeps confronting. Slurm and Kubernetes are both schedulers; they grew up serving different worlds.

Kubernetes grew up serving web services: many independent containers, elastic scaling, partial failures absorbed by replicas, anonymous traffic, ops-as-platform. The two worlds had different needs; the schedulers diverged.

At the modern AI scale they are converging. Slurm now supports containers natively (via Pyxis, Apptainer, and the upcoming scrun). Kubernetes has grown gang scheduling, topology-aware placement, and GPU-aware operators (Volcano, Kueue, Run:ai, KAI Scheduler). The decision between them is still real — covered in depth in Part VII — but neither is obviously wrong for AI workloads anymore.

The two-sentence definition to remember

The cluster engineer's role

If you are an AI Performance and Efficiency Engineer, Slurm is not your job to operate (that is the cluster operations team) but it is your job to use fluently. You will:

• Submit large training jobs across hundreds or thousands of GPUs and need to know why one rank started 90 seconds after the others.
• Help researchers whose jobs sit in PENDING and they don't know why.
• Diagnose why an allreduce on this allocation is slower than the same allreduce on yesterday's.
• Influence the configuration (topology plugin, GRES weights, QoS) to make AI workloads efficient.
• Collaborate with ops on health checks, prologues, and epilogues that catch broken nodes before researchers run into them.

You don't need to write Slurm patches. You do need to read Slurm's logs and translate them into actionable diagnoses faster than anyone else in the room.

§02Architecture: daemons, plugins, control flow

Slurm is built from a small number of daemons that communicate over a custom RPC protocol, with the actual scheduling logic, allocation logic, accounting, and resource discovery factored into plugins. Knowing which daemon does what and where the plugin boundaries are is the foundation for reading the logs.

The daemons

slurmctld

The brain. There is one active slurmctld per cluster (with an optional standby for HA, covered in §23). It holds all cluster state — node states, the job queue, running jobs, reservations, partitions — in memory, with periodic state-save to disk for crash recovery. The scheduler runs inside this daemon. Every sbatch, squeue, and scontrol command ultimately talks to slurmctld.

slurmd

The node agent. One slurmd runs on each compute node. It registers itself with slurmctld on startup (telling it the node's CPUs, memory, GPUs, features), maintains a heartbeat, and when the controller wants to launch a job on this node, slurmd receives the RPC and spawns slurmstepd processes.

slurmstepd

The step process. When a job runs on a node, slurmd forks a slurmstepd per job-step (a "step" is an srun invocation within a job — more on this in §04). slurmstepd sets up cgroups, applies the user's process, opens stdio sockets back to the launching srun, and waits for the user processes to finish. Crucially: slurmstepd is the parent of the user's actual workload. Kill slurmstepd and you kill the job's processes.

slurmdbd

The accounting daemon. A buffering layer between slurmctld and a SQL backend (MariaDB or MySQL). Records every job's resource usage, plus the association tree (users, accounts, QoS, limits) that powers fair-share and access control. Running without slurmdbd is possible for tiny clusters but standard everywhere else.

slurmrestd

Optional. Provides a REST/OpenAPI interface to Slurm — useful for web UIs, dashboards, and integrations that don't want to shell out to squeue.

The plugin model — where Slurm's flexibility lives

Slurm is, structurally, a thin daemon framework with dozens of plugin interfaces. Each interface has multiple implementations; the operator picks one per cluster in slurm.conf. This is why two Slurm clusters can behave so differently while running the same code.

Control flow: what happens when you sbatch

Trace a single sbatch hello.sh from press-Enter to running:

1. sbatch parses the script (including #SBATCH directives), packages a JOB_DESC RPC, and sends it to slurmctld over MUNGE-authenticated TCP.
2. slurmctld validates the request (does the user have access to the requested partition? is the QoS valid? are the resources possible?), assigns a job ID, persists the job to its state files, and inserts it into the priority-ordered queue.
3. On the next scheduling cycle (every few seconds, plus event-driven), the scheduler considers the job. If resources are available and priority allows, the job is "started": the controller picks nodes, builds a job credential, and sends LAUNCH RPCs to the slurmd on each allocated node.
4. Each slurmd validates the credential, runs the configured prolog scripts (which can drain the node if a health check fails), and — for a batch job — launches one slurmstepd that runs the batch script on the head node of the allocation.
5. When the user script invokes srun, that creates a step: srun sends another RPC to slurmctld, which dispatches further launch RPCs to all nodes in the allocation, each spawning its own slurmstepd for the step.
6. The user's processes run. slurmd samples their resource usage. Output is streamed back through slurmstepd into the configured output file.
7. When the user processes exit (or the job hits its time limit, or someone scancels), slurmstepd reports completion to slurmd, which reports to slurmctld, which marks the job COMPLETED, runs epilog scripts, frees the resources, and writes the accounting record to slurmdbd.

Read the steps above twice. Almost every diagnostic question — why didn't my job start, why is my job stuck, why didn't my output appear — maps to "which of those seven steps did it fail at?"

Authentication: MUNGE

Every Slurm RPC is authenticated by MUNGE — a service that creates short-lived, signed credentials shared by all nodes in the cluster. MUNGE requires a shared key file (/etc/munge/munge.key) identical on every node, and time-synchronized clocks (NTP/chrony must be working). If MUNGE breaks on a node, every Slurm command on that node fails with "Invalid Credential" — a classic intermittent failure that turns out to be clock drift.

§03The data model: nodes, partitions, jobs, steps, GRES

Slurm has a small, carefully chosen vocabulary. Learn the seven terms below and most of the documentation will read naturally.

Node

A physical (or virtual) compute host. Each node has a name (node001), a set of features (haswell,broadwell,gpu), a count of consumable resources (CPUs, memory, GPUs, NICs), and a state. Node states are the alphabet of cluster health:

Partition

A named subset of nodes with policies attached. Think "queue" if it helps: gpu partition might be 100 H100 nodes with a 24-hour time limit, cpu partition might be 50 CPU-only nodes with a 7-day limit, debug partition might be 4 nodes with a 30-minute limit and high priority.

Partitions are the primary place admins enforce policy: time limits, QoS access, allowed users/accounts, default resources, preemption modes. A job specifies a partition (or gets the default); the partition's policies apply.

Job

An accepted resource request, assigned a unique JobID. A job has a state (PENDING, RUNNING, COMPLETED, CANCELLED, FAILED, TIMEOUT, OUT_OF_MEMORY, NODE_FAIL, PREEMPTED, etc.), an allocation (set of nodes + resources), a user/account, a partition, and a script that defines what to run.

Step

A unit of execution within a job. Each call to srun inside a batch script creates a step. A job can have many steps, sequentially or in parallel. Steps inherit the job's allocation but can use a subset of it. sstat reports per-step statistics; sacct shows them as JOBID.STEPID rows (e.g., 12345.0, 12345.1, 12345.batch, 12345.extern).

Two special pseudo-steps every batch job has:

• batch step (JOBID.batch) — the slurmstepd that ran the submitted shell script.
• extern step (JOBID.extern) — created to track any processes spawned outside srun (PAM sessions, ssh-into-node, etc.), so they get the right cgroups.

Task

A single process within a step. When you write srun -n 16 ./prog, you're saying "run 16 tasks of ./prog." Tasks are usually MPI ranks or distributed-training ranks. Each task gets its environment variables (SLURM_PROCID, SLURM_NODEID, etc.) so it knows its identity.

GRES — Generic Resources

Slurm's mechanism for tracking and allocating things other than CPUs and memory. GPUs are the most common: declared as Gres=gpu:8 on each node, requested via --gres=gpu:4 or --gpus=4 on a job. Other common GRES: NICs (nic), high-speed scratch storage (nvme), licenses (license), and custom resources operators want to ration.

GRES can be typed (gpu:a100:8 vs gpu:h100:8) so users can request a specific kind. GRES can also carry per-resource attributes — the cores binding for GPUs that says "GPU 0 is closest to cores 0-15" so Slurm can do NUMA-aware allocation.

Account & Association

Accounting concepts that drive fair-share. An account is a project/group (e.g., physics, nlp-team). A user can belong to multiple accounts and uses --account=name to charge a particular one. The combination (user, account, partition, QoS) is an association — and limits like maximum running jobs or fair-share shares live on associations. More in §22.

QoS — Quality of Service

A named bundle of policy: priority bias, preemption rules, limits, time limits, GRES caps. Users select a QoS via --qos=name. Common patterns: a "normal" QoS for daily work, a "high" QoS with priority bias for time-sensitive work (and a strict GPU-hour cap), a "preemptible" QoS for cheap-but-interruptible jobs that can fill the cluster.

Reservation

An advance booking of resources by an admin. Used for scheduled maintenance windows, dedicated time for a workshop or benchmark, holding nodes for an incoming job that needs everything. Jobs that don't match a reservation's parameters are excluded from those nodes during the window.

Master this seven-word vocabulary — node, partition, job, step, task, GRES, association — and three quarters of Slurm reads itself.

§04The submission trinity: sbatch, salloc, srun

Three commands, each with a distinct role. People confuse them frequently — the confusion is the source of a remarkable share of new-user problems.

sbatch — submit a batch job

The most common verb. sbatch myscript.sh submits a job described by the script. The script runs (eventually) on one of the allocated nodes, with the user's allocation reserved for the whole duration. Inside the script, the user runs whatever they want — most commonly, an srun command that fans the work out across the allocation.

# myscript.sh
#!/bin/bash
#SBATCH --job-name=train_llama
#SBATCH --partition=gpu
#SBATCH --nodes=8
#SBATCH --gpus-per-node=8
#SBATCH --ntasks-per-node=8
#SBATCH --cpus-per-task=12
#SBATCH --time=06:00:00
#SBATCH --output=logs/%x-%j.out

module load cuda/12.4 nccl/2.21

srun python train.py --config=configs/llama70b.yaml

# Submit it
$ sbatch myscript.sh
Submitted batch job 8472341

sbatch returns immediately; the job is now in the queue. The script will run when scheduled. Its stdout/stderr go to the configured output files.

salloc — interactive allocation

"Give me an allocation; I'll use it interactively." After salloc succeeds, the user gets a shell on the submit host (or, optionally, on a compute node) with the allocation reserved. They can then run srun within the allocation. Useful for debugging, interactive notebooks, multi-step experiments where they want to keep the allocation between commands.

$ salloc --partition=gpu --nodes=1 --gpus=8 --time=1:00:00
salloc: Granted job allocation 8472342
salloc: Nodes node-42 are ready for job

# Now you have a shell with the allocation held.
$ srun --pty bash         # interactive shell on the compute node
[node-42]$ nvidia-smi
[node-42]$ exit

$ exit                    # or scancel from elsewhere
salloc: Relinquishing job allocation 8472342

srun — launch a step inside an allocation

This is the verb that actually runs processes on nodes. Inside a sbatch script or after salloc, srun starts a step using the existing allocation. From outside any allocation, srun first creates an allocation and then launches the step — a one-shot interactive run.

# Inside a batch script: launch 64 ranks of train.py across the allocation
srun --ntasks=64 --ntasks-per-node=8 python train.py

# One-shot interactive: allocate AND launch, blocks until done
srun --partition=debug --nodes=1 --gpus=1 --time=00:30:00 nvidia-smi

# Within salloc: just runs the step using the existing alloc
$ salloc --nodes=2 --gpus=16 --time=02:00:00
$ srun --ntasks=16 python train.py

The mental model

The classic confusion: forgetting srun in a batch script

A surprising number of researchers write batch scripts like this and wonder why their distributed training runs only on one node:

#!/bin/bash
#SBATCH --nodes=8 --gpus-per-node=8

python train.py    # NO srun — runs on the head node only!
                   # The other 7 nodes sit idle for the whole 6h job.

Without srun, the python process only runs on the batch host. To launch on all eight nodes, you need either srun python train.py (which creates one task per CPU/GPU on each node) or a framework launcher (torchrun, accelerate) that uses srun under the hood or uses Slurm env vars (SLURM_*) to identify ranks. More in §16.

Per-job vs per-step resources

The #SBATCH directives request the job's total resources. The srun flags request the step's resources — by default, the same as the job, but you can request less.

# Job has 8 GPUs total. Run two steps in parallel, each using 4 GPUs.
#SBATCH --gpus=8 --ntasks=2

srun --gpus=4 --ntasks=1 python train_a.py &
srun --gpus=4 --ntasks=1 python train_b.py &
wait

Less common in practice, but worth knowing when you see it.

§05Submit script anatomy and idioms

The structure that every well-written Slurm batch script shares. Idioms here matter because researchers copy-paste from each other; getting these conventions right propagates through the cluster.

The skeleton

#!/bin/bash
#SBATCH --job-name=my-experiment           # readable name in squeue
#SBATCH --account=ai-research              # billing/fair-share account
#SBATCH --partition=h100                   # which queue
#SBATCH --qos=normal                       # quality of service
#SBATCH --nodes=16                         # node count
#SBATCH --ntasks-per-node=8                # one task per GPU
#SBATCH --gpus-per-node=8                  # GPUs per node
#SBATCH --cpus-per-task=14                 # CPUs per task (dataloader workers + overhead)
#SBATCH --mem=0                            # 0 = all available memory on the node
#SBATCH --time=12:00:00                    # wall clock limit (HH:MM:SS)
#SBATCH --output=logs/%x-%j.out            # %x = job name, %j = job ID
#SBATCH --error=logs/%x-%j.err
#SBATCH --signal=B:USR1@60                 # send SIGUSR1 to batch shell 60s before kill
#SBATCH --requeue                          # auto-requeue on node failure

# --- fail-fast ---
set -euo pipefail

# --- environment ---
module purge
module load cuda/12.4 nccl/2.21 openmpi/4.1.6
source /shared/envs/training/bin/activate

# --- diagnostics for postmortem ---
echo "=== job ${SLURM_JOB_ID} on ${SLURMD_NODENAME} ==="
echo "nodes: ${SLURM_JOB_NODELIST}"
echo "start: $(date)"
nvidia-smi --query-gpu=name,driver_version --format=csv,noheader

# --- the actual work ---
srun --kill-on-bad-exit=1 \
     --label \
     python -u train.py \
       --output_dir=/scratch/${SLURM_JOB_ID} \
       --resume_from_checkpoint=auto

echo "end: $(date)"

The idioms that matter

• set -euo pipefail — fail fast on errors, undefined variables, and pipe failures. Without this, a typo in module load sails through silently and your job runs with the wrong environment.
• --output=logs/%x-%j.out — separate output files per job, named for both the job name and the ID. Predictable paths matter when you have hundreds of past jobs.
• python -u — unbuffered stdout. Without it, Python buffers output and the log file appears to hang for minutes, which terrifies researchers.
• srun --kill-on-bad-exit=1 — if any task fails, kill the rest immediately. The default behavior in many Slurm configurations is to let other ranks continue while one has crashed, leading to NCCL hangs.
• srun --label — prefix each line of output with the task ID. Indispensable for multi-rank debugging.
• --signal=B:USR1@60 — Slurm sends SIGUSR1 to the batch shell 60 seconds before forced kill. The "B:" prefix means signal the batch shell (not the user processes). Researchers use this to trigger checkpoint-on-timeout. PyTorch Lightning, NeMo, and similar frameworks have signal handlers built in.
• --requeue — auto-requeue the job if a node fails or it's preempted. Combined with checkpoint/restart, this means a 3-day training run can survive a node failure without the user noticing.

Environment variables Slurm sets

Inside the job, dozens of SLURM_* variables describe the allocation. The ones every distributed-training launcher reads:

Output-file patterns

The pattern characters in --output= and --error=:

%j   job ID                  %J   job ID with array index (12345_3)
%x   job name                %u   user name
%N   master node name        %n   node ID within the job (0..N-1)
%t   task ID within the job  %A   master job ID for arrays
%a   array task ID

For an array job, use --output=logs/%x-%A_%a.out so each array task gets its own file.

Signals and graceful termination

The --signal flag is one of the most underused. The full syntax:

--signal=[B:]<sig>[@<seconds-before-timeout>]

• --signal=USR1@120 — send SIGUSR1 to user processes 120 seconds before time limit.
• --signal=B:USR1@120 — send SIGUSR1 to the batch shell (not the workload).
• --signal=R:USR1@120 — only for requeued jobs.

The receiving program installs a signal handler:

import signal, sys

def on_timeout(signum, frame):
    save_checkpoint()
    sys.exit(0)

signal.signal(signal.SIGUSR1, on_timeout)

Combined with --requeue, this is the canonical pattern for jobs that ride out time limits indefinitely.

§06Resource requests in depth

Most user confusion in Slurm centers on the resource-request flags. They are powerful and intersect in ways that are not obvious. This is the chapter to bookmark.

The four dimensions

Every Slurm resource request lives on four axes:

• Nodes — how many machines.
• Tasks — how many parallel processes total.
• CPUs (cores) — per-task or per-job count.
• Memory — per-task, per-CPU, or per-node.

Plus GPUs, which have their own dedicated flags but layer on top.

The node-task-cpu flags

The arithmetic

The three quantities — total tasks, nodes, tasks-per-node — are over-determined. Specify any two; Slurm derives the third. Common shapes:

# 8 nodes, 8 GPUs each, one task per GPU = 64 tasks total
--nodes=8 --ntasks-per-node=8 --gpus-per-node=8 --cpus-per-task=14

# Equivalent
--nodes=8 --ntasks=64      --gpus-per-node=8 --cpus-per-task=14

# Single node, multi-GPU (FSDP within one box)
--nodes=1 --ntasks=8        --gpus=8            --cpus-per-task=14

# CPU-only training or preprocessing
--nodes=1 --ntasks=1        --cpus-per-task=32  --mem=128G

Memory flags

For GPU AI workloads, --mem=0 on a dedicated node is usually right — you've already reserved the whole node via --exclusive or by requesting all the GPUs, so claim all the host RAM too.

GPU flags

Slurm has two generations of GPU-request flags. Both work; new code should prefer the modern ones (Slurm 19.05+).

Time

Always set --time. Without it, you get the partition default, which is usually too long or too short and may cost you in priority.

Format: MM, HH:MM:SS, D-HH:MM:SS, or D-HH.

--time=30           # 30 minutes
--time=04:00:00     # 4 hours
--time=2-12:00:00   # 2 days 12 hours

Shorter time limits help the backfill scheduler (§09) find slots for your job. Overestimating wastes opportunities; underestimating gets your job killed mid-run.

Constraints and features

Nodes can be tagged with arbitrary features in slurm.conf (Feature=h100,sxm5,rail-A). Users request features with --constraint=:

--constraint=h100             # nodes tagged h100
--constraint="h100&sxm5"      # both tags
--constraint="h100|a100"      # either
--constraint="[rail-A*4&rail-B*4]"  # 4 of rail-A, 4 of rail-B
                                       # (heterogeneous constraint syntax)

Use constraints judiciously. Each unique constraint set reduces the number of nodes available to your job and increases queue time.

§07Job arrays, dependencies, heterogeneous jobs

Three orthogonal mechanisms for jobs that have structure beyond "one allocation, one program." All three appear in real AI workflows.

Job arrays

Submit one description, get N jobs. Each job sees a different SLURM_ARRAY_TASK_ID and runs independently. Perfect for hyperparameter sweeps, evaluation runs, embarrassingly parallel data processing.

#!/bin/bash
#SBATCH --array=0-99%10        # 100 tasks, at most 10 running concurrently
#SBATCH --job-name=hp-sweep
#SBATCH --gpus=1 --time=2:00:00
#SBATCH --output=logs/%x-%A_%a.out

LR_VALUES=(1e-5 3e-5 1e-4 3e-4 1e-3)
WD_VALUES=(0 0.01 0.1)

lr=${LR_VALUES[$((SLURM_ARRAY_TASK_ID % 5))]}
wd=${WD_VALUES[$((SLURM_ARRAY_TASK_ID / 5 % 3))]}

srun python train.py --lr=$lr --wd=$wd --seed=$SLURM_ARRAY_TASK_ID

Array task IDs can be a range (0-99), a list (1,5,7,9), a step (0-100:10), or any combination (0-99,200,300-400:5).

The %N suffix throttles concurrent execution. --array=0-1000%50 means "1001 tasks, never more than 50 running at once" — essential when you don't want a single user's sweep to eat the whole cluster.

Dependencies

"Don't start until that other job finishes." Useful for staged pipelines: preprocess → train → evaluate.

# Submit the preprocess job
$ jid1=$(sbatch --parsable preprocess.sh)

# Train depends on preprocess completing successfully
$ jid2=$(sbatch --parsable --dependency=afterok:$jid1 train.sh)

# Eval depends on training (any exit status)
$ sbatch --dependency=afterany:$jid2 evaluate.sh

Dependencies can be chained, comma-separated (AND), or question-marked (OR). Watch for unsatisfiable dependencies — if the upstream job fails with afterok, the dependent job is automatically cancelled with reason DependencyNeverSatisfied.

Heterogeneous jobs (job packs)

A single job that combines different resource shapes. Useful for client/server topologies (database server + compute clients) and for the increasingly common pattern of training + inference + telemetry components in one allocation.

#!/bin/bash
#SBATCH --nodes=1 --gpus=1 --time=1:00:00   # pack 0 — coordinator
#SBATCH hetjob
#SBATCH --nodes=8 --gpus-per-node=8           # pack 1 — workers
#SBATCH hetjob
#SBATCH --nodes=1 --cpus-per-task=16          # pack 2 — telemetry sidecar

srun --pack-group=0 python coordinator.py &
srun --pack-group=1 python worker.py &
srun --pack-group=2 python telemetry.py &
wait

Each pack is a separate set of resources; they share a job ID but have different node-lists and resource shapes. The --pack-group selects which when launching.

Heterogeneous jobs are powerful but operationally awkward — they make queueing harder (the scheduler must find resources for all packs simultaneously) and confuse downstream tooling. Use when the workflow really demands it; otherwise stick to plain jobs with shaped resources.

§08Querying and introspection

Slurm exposes everything through six command-line tools. Knowing the useful invocations is a daily-driver skill.

squeue — what's running and what's queued

# Default format — too verbose, often not what you want
squeue

# Your jobs only
squeue -u $USER

# A specific partition, with priority
squeue -p gpu --sort=-p

# Custom format — this is what experienced operators use
squeue -o "%.10i %.9P %.20j %.8u %.2t %.10M %.6D %.4C %.4z %R"
#       jobid partit  name        user  state  time     nodes cpus gpu reason

# Show GPU/GRES request
squeue -o "%.10i %.20j %.8u %.10M %.6D %b %R" 
#                                                ^ GRES

# Why are pending jobs pending?
squeue --start                # estimated start times
squeue -t PENDING -o "%.10i %.20j %.8u %.10M %R"

The %R field is the most valuable for diagnostics: it shows the reason a job is in its current state. For PENDING jobs, that's the Reason — Priority, Resources, QOSMaxJobsPerUser, ReqNodeNotAvail, etc. (Full list in §27.)

sinfo — cluster state

# Default — partition summary
sinfo

# By node, with state and reason
sinfo -N -o "%.20n %.10T %.10P %.30G %.30E"
#              node    state     part      gres            reason

# Only nodes in non-healthy states
sinfo -N -t DOWN,DRAIN,FAIL,MAINT -o "%n %T %E"

# How much idle GPU capacity is in each partition right now?
sinfo -h -o "%P %D %t" | awk '{print}'

# Show topology if topology plugin is on
sinfo --topology

scontrol — control-plane queries and actions

The Swiss Army knife. Read state from anything; write to most things if you have permission.

# Full info on one job
scontrol show job 12345

# Full info on a node — its features, state, GRES, last reason
scontrol show node node-042

# Expand a node list to individual hostnames
scontrol show hostnames "node-[042-049,053]"

# What's in this partition?
scontrol show partition gpu

# Update a running job (admin or job owner with limits)
scontrol update jobid=12345 TimeLimit=12:00:00

# Drain a node (admin)
scontrol update NodeName=node-042 State=DRAIN Reason="investigating XID"

# Re-enable a drained node (admin)
scontrol update NodeName=node-042 State=RESUME

# Cancel future state changes (rare)
scontrol cancel_reboot node-042

sacct — accounting history

Slurm's view of jobs that have completed. Lives in the accounting database — queries can go back years. Use it for postmortems, capacity planning, anything historical.

# Your jobs from today
sacct -u $USER -S today

# Custom format with the things that matter
sacct -u $USER -S "2026-05-20" -o "JobID,JobName%30,State,Elapsed,NodeList,ExitCode,ReqMem,MaxRSS"

# All steps of one job, including .batch and .extern
sacct -j 12345 -o "JobID,JobName,State,Elapsed,ConsumedEnergy,MaxRSS"

# All FAILED jobs in the last week on a partition
sacct -S "now-1week" -X -s FAIL,TIMEOUT,NODE_FAIL,OUT_OF_MEMORY \
      -p gpu -o "JobID,User,JobName,State,Elapsed,Reason"

# Per-user GPU-hours consumed last month
sacct -S "2026-04-01" -E "2026-04-30" -X --format=User,AllocTRES%50

sstat — live stats on running steps

# Per-step max memory, CPU, etc., for a running job
sstat -j 12345.batch -o "JobID,MaxRSS,MaxVMSize,AveCPU,AveCPUFreq"

# For an srun-launched step
sstat -j 12345.0

Only running jobs. Once the step completes, use sacct instead.

scancel — stop jobs

scancel 12345                       # cancel one job
scancel 12345.0                     # cancel a single step (leaving job alive)
scancel -u $USER                     # cancel all your jobs (careful)
scancel -t PENDING -u $USER          # cancel only your pending jobs
scancel -p gpu --state=PENDING       # cancel all pending in a partition (admin)
scancel --signal=USR1 12345          # send SIGUSR1 instead of SIGTERM (for checkpoint)

Cancel with a custom signal (--signal=USR1) is the right way to ask a job to checkpoint without killing it abruptly. The job's signal handler does the right thing.

seff — quick efficiency check

A small wrapper on sacct that prints a one-screen summary: requested CPUs/memory vs used, efficiency percentages. Use it on every long job after it completes; gross overprovisioning shows up immediately.

$ seff 12345
Job ID: 12345
Cluster: ai-prod
User/Group: alice/researchers
State: COMPLETED (exit code 0)
Nodes: 8
Cores per node: 112
CPU Utilized: 13-14:22:17
CPU Efficiency: 12.4%  <-- huge over-request of CPUs
Job Wall-clock time: 02:11:04
Memory Utilized: 412.31 GB
Memory Efficiency: 18.7%   <-- requested too much memory

For an efficiency engineer, seff across the fleet is one of the simpler fleet-wide signals to mine.

§09The scheduling cycle & backfill

Slurm has two cooperating schedulers running on different cadences: the main scheduler and the backfill scheduler. They have different jobs.

The main scheduler

Wakes up every SchedulerParameters: sched_interval seconds (typical default: 60), plus on certain events (job submitted, job ended, node changed state). It walks the queue in priority order. For each PENDING job, in priority order:

1. Are required resources currently available?
2. Are all dependencies satisfied?
3. Is the user/account under their limits (running jobs, GPU-hours)?
4. Is the partition open and within its limits?
5. Does the QoS allow this job to run now?

If all yes, the job starts. If not, the scheduler moves on. The main scheduler does not look further than the highest-priority job that cannot be satisfied right now — unless backfill is enabled.

The backfill scheduler

The whole point of backfill: many small jobs would otherwise sit behind one big job that can't run yet. Backfill runs them — but only if they won't delay the big job.

Backfill runs every bf_interval seconds (typical default: 30) and is much more expensive than the main scheduler. It looks at every PENDING job, projects when it could run given the current allocation forecast, and starts any job that:

• Has all required resources available now;
• Has a time limit short enough that it will complete before the next reservation/higher-priority job needs the nodes.

This is why your time limit affects when your job runs. A job with --time=01:00:00 is much easier to backfill into a 1.5-hour gap than a job with --time=24:00:00 that doesn't fit anywhere. Realistic time limits get jobs started faster.

The scheduling cycle, drawn

The knobs operators tune

From SchedulerParameters in slurm.conf — these matter for cluster behavior at scale:

On a 5,000-node cluster with 50,000 pending jobs, default values are wrong. The controller spends all its time scheduling and stops responding to squeue. Tuning these is a recurring conversation between performance engineers and cluster ops.

Why your job starts (or doesn't)

Q: Is the job at the head of the priority-ordered queue (within main sched's depth)?
   no  -> wait for backfill to consider it.
   yes -> Are resources available right now?
          no  -> PENDING / Resources. Wait for jobs to finish.
          yes -> Dependencies satisfied?
                 no  -> PENDING / Dependency.
                 yes -> Under all limits?
                        no  -> PENDING / QOSMax... or AssocMax...
                        yes -> RUNNING.

Most "why isn't my job running" questions resolve to one of those branches. squeue -t PENDING -o "%i %j %R" shows the Reason in the rightmost field; matching it against the tree above is the first diagnostic move.

§10Priority and the multifactor plugin

Job priority is a single number, but it's computed from several weighted factors. Understanding the factors lets you predict, and influence, queue order.

The formula

priority = PriorityWeightAge           × age_factor
         + PriorityWeightFairshare     × fairshare_factor
         + PriorityWeightJobSize       × jobsize_factor
         + PriorityWeightPartition     × partition_factor
         + PriorityWeightQOS           × qos_factor
         + PriorityWeightTRES          × Σ tres_weights × tres_factors
         - nice                        (set by user with --nice)

Each ...factor is a number in [0, 1]. The ...weight coefficients are configured in slurm.conf — typically integers between 0 and 10,000,000. The sum is the job's priority value, visible in sprio output.

The factors

Fair-share in depth

Fair-share is the most important and least-understood factor. Operators allocate "shares" to accounts (an organizational construct) and to users within accounts. The cluster maintains a usage history; the factor is high when your historical usage is below your share, low when above.

Fair-share computes hierarchically. Alice's effective fair-share is the multiplication of nlp-team's standing within ai-research, ai-research's within the root, and Alice's within nlp-team. A user in an over-using team starts at a disadvantage even if they personally haven't used much.

Looking at priority components

# Show factor breakdown for a specific job
sprio -j 12345

# All pending jobs sorted by priority
sprio -S -y -o "%.10i %.10Y %.10A %.10F %.10J %.10P %.10Q"
#                          age  fairshare jobsize partition qos

# Show your fair-share standing
sshare -u $USER --format=Account,User,RawShares,NormShares,RawUsage,EffectvUsage,FairShare

Common fair-share questions

The PriorityFlags worth knowing

• SMALL_RELATIVE_TO_TIME — JobSize factor favors smaller (or shorter) jobs.
• FAIR_TREE — the modern fair-share algorithm (vs. older CLASSIC_FAIRSHARE). Almost universally used.
• DEPTH_OBLIVIOUS — fair-share ignores account-tree depth, treating each user as independent.
• NO_FAIR_TREE — disables fair-tree, fallback only.
• CALCULATE_RUNNING — recompute priorities of running jobs (affects preemption decisions).

§11Preemption, QoS, requeue

Preemption: high-priority jobs displacing lower-priority ones. Done well, it's how a cluster offers both reliable production jobs and cheap "scavenge" capacity for experimental work. Done badly, it shows up as random job kills with no clear pattern.

How preemption is configured

Two pieces in slurm.conf:

PreemptType = preempt/qos              # or preempt/partition_prio
PreemptMode = REQUEUE                  # what happens to preempted jobs
PreemptExemptTime = 5:00               # protect a job for the first 5 min

PreemptMode options:

• CANCEL — kill the preempted job. Harsh but simple.
• REQUEUE — kill and re-submit; will start again when resources are free.
• SUSPEND — pause the processes (cgroup freeze), keep memory; resume later when resources are available. Rarely used in AI clusters because GPU memory is held; cluster can't actually use the freed nodes.
• OFF — disable for this partition/QoS.

The QoS-based preemption model

Each QoS has a Priority value. With PreemptType=preempt/qos and PreemptMode=REQUEUE, the canonical setup is:

Researchers running speculative experiments use --qos=preemptible and accept the requeue cost; production fine-tunes use --qos=normal; time-critical work uses --qos=high within their cap.

Preemption-aware job design

A job in a preemptible QoS must handle being requeued at any time. Best practice:

#SBATCH --qos=preemptible
#SBATCH --requeue
#SBATCH --signal=B:USR1@90       # 90s warning before kill

# In the training code:
# - register a SIGUSR1 handler that saves checkpoint and exits 0
# - on startup, look for existing checkpoint and resume
# - the job auto-requeues; loop continues from checkpoint

For a researcher using NeMo, PyTorch Lightning, or any framework with built-in signal handling, the pattern is "free." For custom training loops, the signal handler is a five-line addition that pays for itself the first time a preemption happens.

Requeue without preemption

Preemption is only one trigger. Requeue also happens on:

• Node failure (NODE_FAIL state) — if --requeue is set, the job goes back to the queue.
• Manual requeue by an admin (scontrol requeue 12345) — for maintenance, fixing a misbehaving job, etc.
• Held-then-released — jobs held with scontrol hold and released with scontrol release are re-queued (subject to dependencies and priority).

QoS in depth: the limits

Beyond priority and preemption, a QoS carries limits that protect the cluster from any single user/account dominating:

When a user hits a QoS limit, jobs sit in PENDING with reason like QOSMaxJobsPerUserLimit or QOSGrpGRESMinutes. sacctmgr show qos dumps the configured values.

§12Reservations & maintenance windows

Reservations are advance bookings of resources. They appear in three patterns: maintenance, dedicated time, and "hold the resources for this incoming big job."

Creating a reservation

# Maintenance reservation: drain these nodes for 6 hours starting tomorrow 02:00
scontrol create reservation \
  ReservationName=maint_2026_05_26 \
  StartTime=2026-05-26T02:00:00 \
  Duration=06:00:00 \
  Nodes=node-[001-016] \
  Users=root \
  Flags=MAINT,IGNORE_JOBS

# Dedicated time for a benchmark run
scontrol create reservation \
  ReservationName=mlperf_run_q2 \
  StartTime=2026-06-15T14:00:00 \
  Duration=48:00:00 \
  Nodes=node-[100-355] \
  Accounts=ai-research

# Held-for-a-specific-user, takes effect immediately
scontrol create reservation \
  ReservationName=urgent_train \
  StartTime=now \
  Duration=12:00:00 \
  Nodes=node-[001-064] \
  Users=alice

Reservation flags worth knowing

Using a reservation

# Submit a job that runs inside a reservation
sbatch --reservation=mlperf_run_q2 mlperf_job.sh

# See active and upcoming reservations
scontrol show reservations

# Delete a reservation
scontrol delete ReservationName=maint_2026_05_26

The performance engineer's reservation usage

Three common needs:

• Benchmark windows — dedicated, contention-free time to measure baselines. Critical when results need to be reproducible across days.
• Suspected-fault investigation — drain suspect nodes via reservation, run diagnostics, return them to service.
• Multi-day large training — a 5,000-GPU run that takes a week. Without a reservation, you can't reliably gather all 5,000 GPUs at the right time. With one, ops aligns scheduling and maintenance around your window.

Reservations are politically sensitive — they explicitly take capacity from the queue. Most clusters have process around creating large ones. The performance engineer's job is to make the case with data: "this benchmark needs uncontended time on 100 nodes for 12 hours; here's what we'll learn and the cost of not learning it."

§13Topology-aware scheduling

The most important scheduler feature for AI workloads. Slurm's topology plugin places jobs on nodes that share switches, dramatically reducing inter-rack/inter-spine traffic for collectives. Without it, an 8-node training job might be spread across the cluster; with it, it lands on 8 nodes hanging off the same leaf.

The topology.conf format

The cluster's physical network is declared in topology.conf:

# Leaf switches: each holds nodes 0-63, 64-127, ...
SwitchName=leaf01 Nodes=node-[001-064]
SwitchName=leaf02 Nodes=node-[065-128]
SwitchName=leaf03 Nodes=node-[129-192]
SwitchName=leaf04 Nodes=node-[193-256]

# Spine switch: connects the leaves
SwitchName=spine01 Switches=leaf01,leaf02
SwitchName=spine02 Switches=leaf03,leaf04

# Root: connects spines
SwitchName=root01 Switches=spine01,spine02

With TopologyPlugin=topology/tree in slurm.conf, the scheduler treats this as a tree and prefers placements that minimize the highest switch level the job spans.

Topology in the job request

# Default: scheduler does its best with topology
sbatch --nodes=8 train.sh

# Explicit hint: keep within a single leaf if possible
sbatch --nodes=8 --switches=1 train.sh

# Stronger: keep within 1 switch level; wait up to 1 hour to find such a placement
sbatch --nodes=8 --switches=1@1:00:00 train.sh

# Match parallelism shape: 64 nodes split across 4 leaves, 16 each
sbatch --nodes=64 --switches=4 train.sh

The --switches=N@TIMEOUT syntax is the operationally important one: "I want topology, but I won't wait forever." Without the timeout, a job demanding strict topology can sit in PENDING indefinitely.

Block topology — for rail-optimized fabrics

For modern AI clusters with rail-optimized InfiniBand (each GPU pinned to a specific NIC, each NIC to a specific rail switch), Slurm's topology/block plugin is becoming the right choice. It models "blocks" of nodes that share the same fast-path connections to each rail, allowing the scheduler to place training jobs so that allreduce traffic stays on its own rail.

# topology.conf, block-style
BlockName=block01 Nodes=node-[001-032]
BlockName=block02 Nodes=node-[033-064]
BlockName=block03 Nodes=node-[065-096]
BlockSizes=32,64,128

BlockSizes defines allowed allocation granularities — the scheduler will allocate in multiples of these. For training jobs that need to fit "exactly one rack" or "exactly one pod," block topology gives precise placement guarantees.

Verifying placement

# What nodes did my job get?
$ scontrol show job 12345 | grep -E 'NodeList|BatchHost'
   NodeList=node-[041-048]
   BatchHost=node-041

# Are they on the same leaf?
$ scontrol show topology node-041
SwitchName=leaf01 Level=0 Nodes=node-[001-064] LinkSpeed=400000
...

# All 8 nodes on the same leaf? Compare against topology.conf.

The performance engineer's diagnostic

"My allreduce is slower today than yesterday" — first move is to look at placement. If yesterday's job had all 8 nodes on one leaf and today's is split across two leaves, the spine traffic explains the slowdown. The fix is either to retry with --switches=1@5:00 or to talk to ops about why the placement diverged.

§14GRES, GPUs, and the device plugin layer

Generic Resources (GRES) is how Slurm tracks anything that isn't a CPU or memory. The mechanism is general; in practice, 99% of GRES configuration on AI clusters is about GPUs.

GRES configuration on nodes

Each node declares its GRES in slurm.conf or via the autodetect mechanism:

# slurm.conf
NodeName=node-[001-100] Gres=gpu:h100:8,nic:8,nvme:8

# gres.conf - detailed per-resource info, on each node
Name=gpu Type=h100 File=/dev/nvidia0 Cores=0-13
Name=gpu Type=h100 File=/dev/nvidia1 Cores=14-27
Name=gpu Type=h100 File=/dev/nvidia2 Cores=28-41
Name=gpu Type=h100 File=/dev/nvidia3 Cores=42-55
Name=gpu Type=h100 File=/dev/nvidia4 Cores=56-69
Name=gpu Type=h100 File=/dev/nvidia5 Cores=70-83
Name=gpu Type=h100 File=/dev/nvidia6 Cores=84-97
Name=gpu Type=h100 File=/dev/nvidia7 Cores=98-111

The Cores= field is the magic. It tells Slurm which CPU cores are "closest" (PCIe-direct, same NUMA node) to each GPU. With this in place, Slurm can bind tasks to CPUs that match the GPU they're using — automatic NUMA-aware allocation.

The AutoDetect mechanism

Modern Slurm supports auto-detection so the operator doesn't have to write out every device by hand:

# gres.conf, autodetected
AutoDetect=nvml
Name=gpu Type=h100 File=/dev/nvidia[0-7]

With AutoDetect=nvml, Slurm queries NVML at startup, learns which GPUs are present, their model, NVLink topology, and PCIe affinity. Less to maintain by hand.

Requesting and binding GPUs

# Whole-node, all GPUs
--gpus-per-node=8

# One GPU per task (most common for distributed training)
--gpus-per-task=1 --ntasks-per-node=8

# Specific type
--gres=gpu:h100:4

# Bind each task to the closest GPU (NUMA-aware)
--gpu-bind=closest

# Bind explicitly: rank 0 -> GPU 0, rank 1 -> GPU 1, etc.
--gpu-bind=map_gpu:0,1,2,3,4,5,6,7

# Bind by bitmask: each task can see specific GPUs
--gpu-bind=mask_gpu:0x01,0x02,0x04,0x08

How GPU isolation actually works

When Slurm allocates a subset of a node's GPUs to a step, it sets CUDA_VISIBLE_DEVICES in the step's environment to limit which GPUs the user's process can see. The other GPUs on the host still physically exist but are hidden by the runtime.

This is a key fact to internalize: Slurm doesn't enforce GPU isolation at the kernel level — it relies on CUDA respecting CUDA_VISIBLE_DEVICES. A motivated user can override it. The cgroup device controller (ConstrainDevices=yes in cgroup.conf) adds a second layer that actually denies access at the device-node level, and that's the production-grade configuration.

# cgroup.conf
ConstrainCores=yes
ConstrainRAMSpace=yes
ConstrainDevices=yes

With ConstrainDevices=yes, Slurm uses cgroup device controllers to make non-allocated GPUs literally unreadable to the job's processes. nvidia-smi from inside the job sees only the allocated GPUs; opening /dev/nvidia2 directly fails with EPERM.

MIG support

NVIDIA Multi-Instance GPU (MIG) partitions an A100/H100 into multiple isolated slices. Slurm exposes MIG slices as typed GRES:

# gres.conf with MIG slices
AutoDetect=nvml
Name=gpu Type=h100 File=/dev/nvidia0
Name=mig Type=1g.10gb File=/proc/driver/nvidia/capabilities/gpu0/mig/gi0/access
Name=mig Type=2g.20gb File=/proc/driver/nvidia/capabilities/gpu0/mig/gi1/access

Users request slices: --gres=mig:1g.10gb:1. Common in inference fleets and shared-development clusters; less common in training, where whole-GPU allocation dominates.

Other GRES patterns worth knowing

• NICs — Name=nic File=/dev/infiniband/uverbs[0-7] in gres.conf, requested as --gres=nic:4. For workloads that want explicit RDMA HCA allocation.
• NVMe scratch — Name=nvme File=/dev/nvme1n1 Count=2T, allowing per-job claim of local scratch storage. Critical for clusters where node-local NVMe is a shared resource.
• Licenses — Licenses=ansys:10,matlab:50 in slurm.conf; jobs request --licenses=ansys:1. Centralized rationing of paid software seats.

GRES is Slurm's primary mechanism for "things that aren't CPUs and memory." Every device that gets allocated gets modeled here, and every per-device policy lives in this layer.

§15GPU jobs: from request to running kernel

Trace a single GPU training job through the layers. Understanding this trace converts vague performance hypotheses into testable ones.

The interesting bugs live at the boundaries: between step 3 and 4 (cgroup misconfigurations), between 6 and 7 (rank-to-GPU mapping mismatches), between 7 and 8 (CUDA_VISIBLE_DEVICES exposure), between 8 and 9 (NCCL not picking the right NICs).

The rank-to-GPU contract

The convention that almost every distributed training framework assumes:

• One Slurm task per GPU on the node.
• SLURM_LOCALID on the node corresponds to the GPU index in CUDA_VISIBLE_DEVICES.
• The framework reads SLURM_PROCID as global rank, SLURM_NTASKS as world size, SLURM_LOCALID as local rank.

For PyTorch:

import os, torch, torch.distributed as dist

rank        = int(os.environ["SLURM_PROCID"])
world_size  = int(os.environ["SLURM_NTASKS"])
local_rank  = int(os.environ["SLURM_LOCALID"])

# Talk to the right GPU
torch.cuda.set_device(local_rank)

# Coordinate
master_addr = os.environ["SLURM_LAUNCH_NODE_IPADDR"]
master_port = 29500

dist.init_process_group(
    backend="nccl",
    init_method=f"tcp://{master_addr}:{master_port}",
    rank=rank,
    world_size=world_size,
)

The diagnostic moves when "GPU jobs are weird"

§16Multi-node launchers: srun, pmix, torchrun

Three real launcher choices for distributed training. They have different ergonomics and different failure modes. A senior engineer can recognize all three in the wild.

Pattern A: srun directly

The Slurm-native way. srun launches one task per rank, sets SLURM_* env vars, and the training script reads them. PMIx coordinates the bootstrap if MPI is involved.

#!/bin/bash
#SBATCH --nodes=8 --ntasks-per-node=8 --gpus-per-node=8
#SBATCH --cpus-per-task=14

export MASTER_ADDR=$(scontrol show hostname $SLURM_NODELIST | head -n1)
export MASTER_PORT=29500

srun --mpi=pmix \
     --cpu-bind=cores \
     --gpu-bind=closest \
     --label \
     python train.py

In train.py, the rank info comes from SLURM_PROCID/SLURM_LOCALID. Cleanest. Requires that Slurm and PMIx are correctly built and that the user script is Slurm-aware.

Pattern B: srun + torchrun

The PyTorch-native pattern. srun launches one torchrun per node; torchrun fans out to one process per GPU within the node. PyTorch's rendezvous handles coordination.

#!/bin/bash
#SBATCH --nodes=8 --ntasks-per-node=1 --gpus-per-node=8
#SBATCH --cpus-per-task=112  # all CPUs on the node for the single Slurm task

export MASTER_ADDR=$(scontrol show hostname $SLURM_NODELIST | head -n1)
export MASTER_PORT=29500
export NNODES=$SLURM_JOB_NUM_NODES
export GPUS_PER_NODE=8

srun --label torchrun \
    --nnodes=$NNODES \
    --node_rank=$SLURM_NODEID \
    --nproc_per_node=$GPUS_PER_NODE \
    --rdzv_id=$SLURM_JOB_ID \
    --rdzv_backend=c10d \
    --rdzv_endpoint=$MASTER_ADDR:$MASTER_PORT \
    train.py

Note --ntasks-per-node=1: each Slurm task is one torchrun, which then spawns 8 Python processes. This gives torchrun control over per-GPU subprocesses, including failure handling and elastic scaling.

Pattern C: srun --multi-prog or accelerate

Variations on the above with extra abstraction. HuggingFace accelerate launch, DeepSpeed's launcher, NeMo's launcher, Mosaic Composer — all wrap one of patterns A or B with friendlier defaults.

Comparison: which pattern when

For pure performance work — benchmarking, profiling, MFU measurement — Pattern A is cleaner because Slurm has direct visibility into every rank. For research workloads that need elastic resizing or per-rank fault tolerance, Pattern B is the better fit.

PMIx — what it is and why it matters

PMIx (Process Management Interface — Exascale) is the protocol Slurm and MPI use to bootstrap distributed jobs. Without it, MPI processes can't find each other; with it, MPI_Init in any rank can ask "who else is in this job and where do I reach them?" and the answer comes from slurmstepd via the PMIx server.

The Slurm option --mpi=pmix enables it. For PyTorch's NCCL backend, PMIx isn't strictly required (PyTorch uses its own rendezvous), but having it correctly installed means MPI-using libraries (UCX, NCCL-with-MPI bootstrap, certain profilers) work without further configuration.

§17NCCL integration: env, plugins, topology

NCCL doesn't know about Slurm. NCCL knows about networks, GPUs, and ranks. The integration is in the user's hand: set the right environment variables so NCCL picks the right network path, the right topology, and the right number of channels.

The NCCL env-var battery

The canonical setting block for a multi-node GPU job in a Slurm script:

# --- NCCL essentials ---
export NCCL_DEBUG=INFO                              # at least once; remove after working
export NCCL_DEBUG_SUBSYS=INIT,NET,ENV               # filter to relevant subsystems

# --- Network selection ---
export NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_6,mlx5_7,mlx5_8,mlx5_9
export NCCL_IB_GID_INDEX=3                          # RoCE / IPoIB; check with show_gids
export NCCL_SOCKET_IFNAME=ib0                       # bootstrap interface
export NCCL_IB_DISABLE=0                            # 1 to force TCP for diagnostic
export NCCL_NET_GDR_LEVEL=PIX                       # GPUDirect level

# --- Stability/timeout ---
export TORCH_NCCL_ASYNC_ERROR_HANDLING=1
export TORCH_NCCL_BLOCKING_WAIT=1
export NCCL_TIMEOUT=600                              # seconds (override Python default)

# --- Algorithm/protocol (tune only with measurement) ---
export NCCL_MAX_NCHANNELS=16
export NCCL_BUFFSIZE=8388608

The principle: NCCL has many tuning knobs that interact with the fabric. For a healthy production cluster the operator has already established good defaults (often as cluster-wide environment loaded by module load nccl). The user's job is to not override them unless they have a measured reason.

Rail-aware HCA selection

The single most impactful NCCL setting on a rail-optimized fabric is NCCL_IB_HCA. On a DGX-class node with 8 GPUs and 8 IB HCAs (mlx5_0 through mlx5_9, sometimes with two for storage that aren't for fabric), specifying the right 8 HCAs ensures NCCL uses GPU-local NICs.

# Check NIC-to-GPU affinity
$ nvidia-smi topo -m
        GPU0    GPU1    ...    mlx5_0  mlx5_1
GPU0     X      NV18    ...    PIX     NODE
GPU1    NV18     X      ...    NODE    PIX
...
# PIX = direct PCIe switch path = fast. NODE = same NUMA, slower. SYS = crosses NUMA.
# Set NCCL_IB_HCA to the 8 HCAs with PIX affinity to GPUs
export NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_6,mlx5_7,mlx5_8,mlx5_9

Slurm-specific NCCL conventions

Three patterns that recur in well-run AI clusters:

• Cluster-wide NCCL env in a module. The cluster provides module load nccl/2.21 that exports all the NCCL_* variables the cluster expects. Users source it from their script; if it's wrong, ops fixes it once for everyone.
• NCCL_DEBUG=WARN by default, INFO opt-in. Verbose NCCL output is too noisy for normal use but invaluable for debugging. The pattern: export NCCL_DEBUG=${NCCL_DEBUG:-WARN} in the module, so users can override on demand.
• Per-rank NCCL log files. NCCL_DEBUG_FILE=/path/log.%h.%p so each rank's output goes to a separate file. Without this, an 8-node job interleaves all output into one stream and you can't tell whose log line you're reading.

The NCCL plugin layer

NCCL supports network plugins that replace its default transport. The most important is the AWS OFI plugin (and its EFA-specific variant), which lets NCCL use libfabric instead of NCCL's native TCP/IB transports. NVIDIA's own SHARP plugin enables in-network reduction on Mellanox switches.

# Tell NCCL where to find network plugins (some clusters set this in their module)
export NCCL_NET_PLUGIN=ofi
export LD_LIBRARY_PATH=/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH

# Or for SHARP
export NCCL_COLLNET_ENABLE=1

Plugin selection happens at NCCL_DEBUG=INFO time. The log line "Using network IB" vs "Using network AWS Libfabric" vs "Using network CollNet" tells you which transport you got.

The diagnostic loop for "NCCL is slow"

Q: Is bandwidth lower than expected (nccl-tests baseline)?
   |
   +-- Yes, dramatically so -> wrong transport (TCP fallback?)
   |     Check NCCL_DEBUG=INFO: "Using network ___"
   |     Check NCCL_IB_DISABLE accidentally =1
   |     Check IB link state (ibstatus)
   |
   +-- Yes, mildly (10-30%) -> rail/topology issue
   |     Check NCCL_IB_HCA matches node topology
   |     Check rank-to-host placement (scontrol show job)
   |     Check switch placement (sinfo --topology)
   |
   +-- Bandwidth ok but latency high -> protocol choice
   |     Check NCCL_PROTO; LL/LL128 vs Simple matters
   |     Check NCCL_ALGO; Tree vs Ring
   |
   +-- Bandwidth ok in tests, bad in framework -> overlap broken
         Profile with nsys; check stream serialization

§18Containers: enroot + pyxis, Apptainer

HPC clusters have always been allergic to root-required containers. The Docker-style daemon model doesn't fit shared multi-tenant systems where users can't be trusted with root. Two solutions have emerged: NVIDIA's enroot + pyxis pair, and Apptainer (formerly Singularity).

enroot + pyxis

enroot is a rootless container runtime — converts OCI images into chroot-style sandboxes, no daemon, no setuid binary beyond an unprivileged user mount helper. pyxis is a SPANK plugin (a Slurm extension point) that adds container-aware flags to srun and integrates with enroot.

# Pull and convert a container image to enroot's local format (cached)
$ enroot import docker://nvcr.io#nvidia/pytorch:25.04-py3

# Use it in a Slurm job
#!/bin/bash
#SBATCH --nodes=8 --gpus-per-node=8 --ntasks-per-node=8

srun \
  --container-image=nvcr.io/nvidia/pytorch:25.04-py3 \
  --container-mounts=/scratch:/scratch,/home/$USER:/home/$USER \
  --container-workdir=/scratch/$SLURM_JOB_ID \
  --container-env=NCCL_DEBUG,NCCL_IB_HCA,MASTER_ADDR,MASTER_PORT \
  python train.py

What pyxis does for you

• Each task runs inside the container, with Slurm's resource limits applied.
• CUDA_VISIBLE_DEVICES, NCCL env, and Slurm rank vars are passed through into the container.
• The container's userspace sees the host's GPU devices (via nvidia-container-runtime).
• Mounts are explicit; nothing else from the host is visible by default.
• Image caching is per-node; the same image is reused across jobs on the same node.

Apptainer (formerly Singularity)

The older HPC container runtime. Single-binary, no daemon, can run as a regular user. Uses a different image format (SIF) — a single file you can ship around like a script.

# Build an SIF from a Docker image
$ apptainer build pytorch.sif docker://nvcr.io/nvidia/pytorch:25.04-py3

# Run inside Slurm
srun apptainer exec --nv pytorch.sif python train.py
#                        ^-- --nv passes through GPU devices

The --nv flag enables NVIDIA GPU passthrough. Apptainer's appeal: SIF is one file, easy to share, no registry needed. The downside: no native Slurm integration like pyxis.

Comparing the two

For modern AI infrastructure, enroot + pyxis is the conventional choice — first-class NVIDIA support, tight Slurm integration. Apptainer remains important in legacy and academic settings.

The container-image idioms

• Pre-pull images at the cluster level. A first-pull on demand takes minutes; pre-cached images start instantly.
• Pin image tags. nvidia/pytorch:25.04-py3, not latest. Reproducibility depends on it.
• Mount the home directory and scratch explicitly; don't rely on defaults that might change.
• Use --container-env to pass through only what's needed.

§19Resilience: checkpoint/restart, requeue, hot spares

The reality of 1000+ GPU jobs: hardware fails, networks degrade, processes die. A multi-day training run survives only because it expects failure. The patterns:

Pattern: requeue on node failure

The simplest. With --requeue in the job script, Slurm automatically re-submits the job if a node fails (NODE_FAIL). If the training code resumes from the latest checkpoint, the run continues with at most a few minutes lost.

#!/bin/bash
#SBATCH --requeue
#SBATCH --signal=B:USR1@90       # 90s checkpoint warning before kill

# Resume from the latest checkpoint if present
srun python train.py \
    --output_dir=/checkpoints/$SLURM_JOB_NAME \
    --resume_from_checkpoint=auto

The script must be idempotent under requeue. Common pitfalls:

• Output directory named with timestamp at start — different name on requeue, breaks resume.
• Wandb/Tensorboard run created at start — second instance creates a new run, fragmenting metrics.
• Dataloader random state not in checkpoint — requeued job sees different data ordering.

Pattern: signal-triggered checkpoint

The signal handler is the key. Combined with --signal=B:USR1@SECS, the framework gets a heads-up before Slurm hard-kills:

import signal, sys, torch

def on_signal(signum, frame):
    if rank == 0:
        torch.save(checkpoint_state, f"{ckpt_dir}/latest.pt")
    dist.barrier()
    sys.exit(0)

signal.signal(signal.SIGUSR1, on_signal)

For PyTorch Lightning, NeMo, transformers Trainer, MosaicML Composer — all have built-in handlers that just need the right Slurm flag.

Pattern: hot spare nodes

For very large jobs, even with requeue the restart cost is large (cold disk caches, reloading weights, NCCL rebuild). Hot-spare patterns mitigate this:

• Over-allocate by 1-2 nodes. The job runs on N-2 nodes; the spares sit idle, ready to take over if a node fails.
• Elastic training (torchrun --max_restarts=N). The job continues with one fewer rank if a node fails, automatically rebalancing.
• Replica training. Two independent jobs train on the same data; if one fails the other has lost no progress.

Pattern: distributed checkpoint, async write

Checkpoint write time matters because it's pure overhead. For a 70B model, naive checkpoint can take 20+ minutes; distributed-async checkpoint drops it to under a minute.

In the Slurm context, what matters: the signal-to-actual-kill window must be larger than checkpoint write time. If checkpoint takes 5 minutes and --signal=B:USR1@60 only gives 60 seconds warning, the kill arrives mid-write. Match the warning to the actual checkpoint duration.

Pattern: prologue health checks

The other half of resilience: stop bad nodes from joining jobs in the first place. The Slurm Prolog script runs at job start on every node; if it exits non-zero, the node is drained and the job is requeued.

# /etc/slurm/prolog.d/10-gpu-check.sh, run as root before each job
#!/bin/bash
set -e

# Are all 8 GPUs present and healthy?
gpu_count=$(nvidia-smi --query-gpu=count --format=csv,noheader | head -n1)
if [[ "$gpu_count" != "8" ]]; then
  echo "GPU count is $gpu_count, expected 8" >&2
  exit 1
fi

# Any XID errors in recent dmesg?
if dmesg --since="1 minute ago" | grep -q NVRM:; then
  echo "Recent NVRM events found" >&2
  exit 1
fi

# Is IB up on all 8 HCAs?
for hca in mlx5_{0..9}; do
  state=$(cat /sys/class/infiniband/$hca/ports/1/state 2>/dev/null || echo MISSING)
  if [[ "$state" != *ACTIVE* ]]; then
    echo "HCA $hca state: $state" >&2
    exit 1
fi
done

exit 0

A failed prologue drains the node with the script's stderr as the drain reason — exactly what an operator needs for triage. The performance engineer's contribution here is recommending what checks to add based on patterns observed in past failures.

§20Cgroups, NUMA, CPU/GPU/NIC pinning

The last mile of performance for distributed training: making sure every task runs on the right CPU cores, with access to the right NUMA-local memory, the right GPU, and the right NIC. Done wrong, you pay 10-25% in throughput silently.

The cgroup hierarchy Slurm creates

With ProctrackType=proctrack/cgroup and TaskPlugin=task/cgroup, every job step lives in its own cgroup tree. Each task within the step has its own cgroup; resources are enforced at that level.

Each task_N directory has the standard cgroup v2 control files: cpu.max, memory.max, cpuset.cpus, cpuset.mems, devices.allow. Slurm writes these based on the job's allocation and the cgroup.conf settings.

The cgroup.conf knobs

# /etc/slurm/cgroup.conf
ConstrainCores=yes           # cpuset limits each task to allocated cores
ConstrainRAMSpace=yes        # memory.max enforces memory limit
ConstrainSwapSpace=yes       # prevent swap (kills before swap on GPU nodes)
ConstrainDevices=yes         # device cgroup limits /dev/nvidiaN access
MaxRAMPercent=98             # cap memory at 98% of node (leave room for kernel)
AllowedKmemSpace=             # empty = unlimited kernel memory
CgroupAutomount=yes

ConstrainDevices=yes is the one that matters most for AI clusters. Without it, GPU isolation is advisory (just CUDA_VISIBLE_DEVICES); with it, the kernel itself denies access to non-allocated GPUs.

CPU pinning options

srun has rich CPU-binding options:

For multi-GPU training, the right answer is almost always --cpu-bind=cores plus relying on the GRES Cores= directive to pin each task to the cores closest to its GPU. Verify with --cpu-bind=verbose,cores and look at the stderr.

The NUMA-locality story

On a modern dual-socket node, half the GPUs are PCIe-attached to socket 0, half to socket 1. If a task assigned to GPU 0 runs on a CPU in socket 1, every host-to-device transfer crosses the QPI/UPI link — adding latency and contention.

With GRES Cores=0-55 for GPU 0 (cores on socket 0) and Cores=56-111 for GPU 4 (cores on socket 1), Slurm's allocator picks CPU cores matching the GPU. The result: each task's CPUs are on the same socket as its GPU, and pinned memory lives in the local NUMA node.

# Verify NUMA placement inside a job
$ srun --cpu-bind=verbose,cores --label hostname
0: cpu-bind=MASK - node-041, task 0 0 [12345]: mask 0x00000000FFFFFFFF000000000000FFFF set
1: cpu-bind=MASK - node-041, task 1 0 [12346]: mask 0x0000FFFFFFFF00000000FFFF00000000 set
...

# Check memory NUMA from inside the task
$ srun -n 1 numactl --show
policy: bind
preferred node: 0
physcpubind: 0 1 2 3 ... 13
cpubind: 0
nodebind: 0
membind: 0

The NIC pinning question

NCCL picks NICs via NCCL_IB_HCA. There's no Slurm flag that maps "task N uses NIC X"; that decision lives in NCCL based on its topology view. The Slurm contribution: ensure rank-to-host placement is correct, and that the cluster's NCCL_IB_HCA module setting matches the hardware. NIC affinity then falls into place.

The diagnostic loop for "training is slow and I think it's affinity"

Q: Run --cpu-bind=verbose,cores and see the masks. Are they right?
   no  -> Slurm allocator didn't pick correctly.
          Check gres.conf has Cores= field per GPU.
          Check ConstrainCores=yes.
   yes -> Run numactl --show in a task. Is membind = the same NUMA node as cpubind?
          no  -> NUMA mismatch. Add --cpu-bind=ldoms or check task layout.
          yes -> Run nvidia-smi topo -m. Are GPU and CPU on same NUMA?
                 no  -> Wrong gres.conf Cores= mapping.
                 yes -> Affinity is fine. Look elsewhere (NCCL, dataloader, etc).

When affinity is wrong, the symptoms are subtle: 10-25% throughput loss, slightly higher H2D copy time, sometimes mysteriously variable step times. None of them look like "affinity is wrong" — until you check.

§21Configuration tuning for large clusters

A Slurm config that works fine for 100 nodes will choke on 5,000. The scheduling cycle, the RPC backlog, the state-save cadence, the database commit rate — all of these have knobs that need tuning as the cluster grows.

The configuration files

Changes require an scontrol reconfigure — which gracefully reloads. Some changes (plugin changes, port changes) require a controller restart. Compute nodes pick up most changes via the same mechanism.

Scheduler tuning at scale

The single most-tuned section in any large cluster's slurm.conf:

SchedulerType=sched/backfill
SchedulerParameters=\
  bf_continue,\
  bf_interval=30,\
  bf_max_job_test=2000,\
  bf_max_job_user=100,\
  bf_max_job_part=1000,\
  bf_resolution=60,\
  bf_window=10080,\
  bf_yield_interval=1000000,\
  bf_yield_sleep=200000,\
  default_queue_depth=1000,\
  sched_max_job_start=100,\
  sched_min_interval=2000000,\
  defer

DefMemPerCPU=3000
MaxJobCount=500000          # total jobs in queue + recently-completed
MaxArraySize=100001          # max array task index + 1
MaxTasksPerNode=512
TreeWidth=128                 # fanout for fwd controller->slurmd msgs

MessageTimeout=30
SlurmctldTimeout=600
SlurmdTimeout=600

• bf_max_job_test — total jobs backfill considers per cycle. Defaults to 100; large clusters with deep queues need 1000-5000.
• bf_yield_interval / bf_yield_sleep — how often backfill yields to other RPCs (microseconds). Critical: if backfill never yields, squeue hangs for users while the cycle runs.
• default_queue_depth — main scheduler depth. Same scale-up reasoning.
• defer — don't run the main scheduler on every job submission; coalesce. Hugely reduces controller load on bursty submission patterns (job arrays, workflow tools).
• TreeWidth — Slurm uses a tree topology to dispatch messages from controller to slurmds. With 128, the controller talks to 128 nodes directly; each of those forwards to up to 128 more. For 5000 nodes, 128 gives a 2-level tree; for 50000 a 3-level. Tune so depth stays at ≤3.

State-save and snapshots

StateSaveLocation=/var/spool/slurmctld
SlurmctldDebug=info
SlurmSchedLogLevel=0          # 0 off, 1 on; very expensive at scale

The state-save location is where slurmctld persists job/node state. It must be on fast, reliable storage — NVMe or strong network filesystem. State files are written every few seconds; slow storage backs up the controller. On 5,000-node clusters, this is non-negotiable: state-save on slow storage is the #1 cause of controller unresponsiveness.

The controller's resource needs

The controller is a single process. For a large cluster:

• CPU: enough cores for the scheduler thread + multiple RPC handler threads. 16-32 cores is typical.
• Memory: holds all job/node/reservation state. Plan ~1 KB per pending job + per node. 64 GB is generous for most clusters.
• Network: every node connects to the controller. Make sure NIC bandwidth and conntrack capacity scale.
• Storage: state-save dir on local NVMe with strict latency targets.

The "controller fell behind" syndrome

§22Accounting (slurmdbd), fair-share, limits

The accounting database is more than a record-keeping system. It's where the cluster's allocation policies live, where every job's resource use is captured, and where fair-share gets its inputs.

The slurmdbd architecture

slurmdbd sits between the controller and the SQL database. It buffers writes, batches them, and presents a single point of contact for accounting queries. Multiple controllers from federated clusters can talk to the same slurmdbd.

Associations and the hierarchy

An association is the tuple (cluster, account, user, partition, qos). Limits and fair-share shares are set per association. The full hierarchy is a tree:

# Create accounts
sacctmgr add account ai-research description="AI research org"
sacctmgr add account nlp parent=ai-research description="NLP team"
sacctmgr add account vision parent=ai-research description="Vision team"

# Assign shares (fair-share weights)
sacctmgr modify account ai-research set Fairshare=50
sacctmgr modify account nlp set Fairshare=30
sacctmgr modify account vision set Fairshare=20

# Add users
sacctmgr add user alice account=nlp Fairshare=parent
sacctmgr add user bob   account=vision Fairshare=parent

# Per-user limits
sacctmgr modify user alice account=nlp set MaxJobs=20 MaxSubmitJobs=200

# Per-account TRES (resource) limits
sacctmgr modify account vision set GrpTRES=cpu=5000,gres/gpu=256

# Budget — total GPU-minutes over a window
sacctmgr modify account nlp set GrpTRESMins=gres/gpu=100000

Common limits and what they enforce

When a limit is hit, jobs stay in PENDING with reason AssocGrpGRESMinutes or similar. The job will eventually run when usage decays below the cap (for cumulative limits) or finishes when other jobs complete (for concurrent limits).

The decay mechanism

PriorityDecayHalfLife in slurm.conf controls how fast historical usage "decays" out of fair-share calculations. Common values: 7 days (typical), 14 days (for slower-moving research clusters), 30 days (for very long-cycle workloads). A 7-day half-life means a job from 7 days ago counts half as much; a job from 14 days ago counts a quarter; etc.

Researchers ask: "When will my fair-share recover?" The answer is "with the half-life cadence, conditional on you not using more in the meantime." Set researcher expectations: bursts of usage will depress priority for ~2-3 half-lives.

Reporting from the database

# Per-user GPU-hours last month
sreport user TopUsage start=2026-04-01 end=2026-05-01 TopCount=20 -t hours

# Per-account utilization
sreport cluster AccountUtilizationByUser start=2026-04-01 end=2026-05-01

# Cluster utilization (idle vs allocated vs drained, over time)
sreport cluster Utilization start=2026-04-01 end=2026-05-01

# Direct SQL on slurm_acct_db (read-only user, please)
SELECT
  acct,
  SUM(time_end - time_start) / 3600 AS gpu_hours,
  COUNT(*) AS job_count
FROM cluster_job_table
WHERE time_end > UNIX_TIMESTAMP('2026-04-01')
  AND tres_req LIKE '%1001=8%'   -- 1001 is gres/gpu in TRES IDs
GROUP BY acct
ORDER BY gpu_hours DESC;

The performance engineer mines this database for fleet-wide signals: per-account efficiency, per-job-class MFU, the long-tail of users whose jobs all OOM, the distribution of queue times. The accounting DB is the single most valuable observability surface that isn't a profiler.

§23HA, failover, disaster patterns

Slurm has supported controller HA since early days. The pattern: primary + backup slurmctld sharing a state-save directory. If the primary dies, the backup takes over by reading the shared state.

The HA setup

# slurm.conf
SlurmctldHost=ctl-01(10.0.0.10)        # primary
SlurmctldHost=ctl-02(10.0.0.11)        # backup
StateSaveLocation=/shared/slurmctld   # on a shared FS, NFS or Lustre

Both hosts run slurmctld; only one is active at a time. The active one writes state to StateSaveLocation. The backup polls; if the primary stops responding, it takes over by reading the state and resuming.

The shared filesystem for state-save needs:

• Low latency (every state operation hits it).
• POSIX-compliant locking (Slurm uses file locks for coordination).
• Higher availability than the controllers (you can't lose state if the FS dies).

NFS works for clusters up to a few thousand nodes. Lustre or GPFS for larger. Some clusters use DRBD-replicated local NVMe with manual switchover.

Failover semantics

When the backup takes over:

• Running jobs continue uninterrupted. slurmd on the compute nodes doesn't need to talk to the controller for already-running jobs; it just reports state back and the backup picks up where the primary left off.
• Queued jobs continue queuing. The backup reads the same queue from the state files.
• In-flight RPCs (sbatch, scontrol, etc.) may fail and need retry. Most clients retry automatically.
• Failover detection takes some seconds to minutes depending on timeout values.

The slurmdbd HA story

The accounting database is also a single point. The pattern: MariaDB/MySQL replication with manual failover (active-passive), or Galera/InnoDB cluster for active-active. slurmdbd itself can be active-passive (the standby just sits there until the primary fails).

The implication: short outages of slurmdbd are tolerable. The controller buffers accounting writes; when the DB comes back, the buffer drains. Multi-hour DB outages eventually fill the buffer and start dropping accounting data, which is non-recoverable.

Disaster scenarios

Key principle: Slurm is robust against controller failure but not against state loss. Run the state-save directory on storage you trust. Snapshot it regularly.

§24Monitoring & observability

Three observability layers a healthy cluster has: Slurm-internal metrics, cluster-physical metrics, and per-job metrics. The performance engineer cares about all three but lives mostly in the third.

The Prometheus exporter

prometheus-slurm-exporter (or its modern variants) scrapes Slurm via squeue, sinfo, sacct and exposes them as metrics. The standard set:

The dashboards that matter

For an AI cluster, six dashboards are non-negotiable:

1. Cluster heatmap — every node shown as a tile, colored by state. One glance shows draining, down, busy.
2. Queue depth by partition — pending jobs over time. Spikes suggest scheduling issues or sudden demand.
3. Per-job GPU utilization — DCGM-sourced. Outliers below 30% are candidates for performance attention.
4. Scheduler health — cycle time, RPC backlog, controller CPU usage. Detects controller overload before users complain.
5. Per-account usage — GPU-hours, last 24h / 7d / 30d. Detects budget overruns and fair-share imbalances.
6. Job lifecycle — pending → running → completed durations. Detects scheduler problems and excessive failure rates.

The Slurm logs

Three logs every operator and performance engineer can grep:

Useful grep patterns:

# Find what happened to a specific job
grep "JobId=12345" /var/log/slurmctld.log

# Recent scheduling delays
grep "Warning: scheduler" /var/log/slurmctld.log

# Node failures and reasons
grep -E "(node_fail|DRAINED|NODE_FAIL)" /var/log/slurmctld.log

# Prolog failures (the root cause of mysterious requeues)
grep -A 3 "Prolog failed" /var/log/slurmd.log

# Communication issues from a controller's view
grep "unable to register" /var/log/slurmctld.log

# Failed RPC retries
grep "Failed to forward RPC" /var/log/slurmctld.log

Per-job metrics via DCGM

For GPU jobs, Slurm itself doesn't capture detailed GPU metrics. The pattern: DCGM exporter on every node, scraped per-GPU into Prometheus, labeled with the Slurm job ID via DCGM-Slurm integration. The result: time-series GPU metrics queryable by job.

# PromQL: GPU utilization for job 12345 over time
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE{slurm_job="12345"}

# MFU (approximated by tensor pipe active rate) averaged over the job
avg_over_time(DCGM_FI_PROF_PIPE_TENSOR_ACTIVE{slurm_job="12345"}[1h])

The integration is achieved either by tagging GPUs at job-start (via prolog scripts that write the job ID to a per-GPU label) or by querying DCGM-aware exporters that already correlate.

Slurm-web and friends

Several open-source web UIs sit on top of Slurm: Slurm-web, Open OnDemand, sview (Slurm's own X11 tool, mostly dead). For most users the CLI is enough; for casual users and managers, a web UI showing queue state and per-account usage justifies itself.

§25Health checks & node lifecycle

Bad nodes are the cluster's most persistent enemy. A single degraded node can ruin large training jobs by becoming the straggler. The defense: continuous health checks and aggressive draining.

The Slurm health-check hook

HealthCheckProgram in slurm.conf is a script run periodically on every node. Returns 0 = healthy; non-zero = drain the node with the script's output as the reason.

# slurm.conf
HealthCheckProgram=/etc/slurm/healthcheck.sh
HealthCheckInterval=300          # every 5 minutes
HealthCheckNodeState=ANY,CYCLE   # run on all nodes always

# /etc/slurm/healthcheck.sh
#!/bin/bash
set -e

# GPU count check
gpu_count=$(nvidia-smi --list-gpus | wc -l)
[[ $gpu_count -eq 8 ]] || { echo "gpu_count=$gpu_count"; exit 1; }

# GPU temperature check
while read -r temp; do
  [[ $temp -lt 85 ]] || { echo "gpu_temp=$temp"; exit 1; }
done < <(nvidia-smi --query-gpu=temperature.gpu --format=csv,noheader)

# ECC errors
ecc=$(nvidia-smi --query-gpu=ecc.errors.uncorrected.aggregate.total --format=csv,noheader,nounits | head -n1)
[[ $ecc -eq 0 ]] || { echo "ecc_errors=$ecc"; exit 1; }

# IB link state
for hca in /sys/class/infiniband/mlx5_*; do
  state=$(cat $hca/ports/1/state)
  [[ $state == *ACTIVE* ]] || { echo "ib_$(basename $hca)=$state"; exit 1; }
done

# NVLink errors
nvlink_errors=$(nvidia-smi nvlink -e 2>/dev/null | grep -i error | wc -l)
[[ $nvlink_errors -eq 0 ]] || { echo "nvlink_errors=$nvlink_errors"; exit 1; }

# File systems mounted
for mp in /scratch /home /shared; do
  mountpoint -q $mp || { echo "unmounted=$mp"; exit 1; }
done

exit 0

DCGM diagnostics

dcgmi diag runs structured GPU health checks. Three diagnostic levels:

dcgmi diag -r 1     # quick: ~10 seconds, basic memory and SM checks
dcgmi diag -r 2     # medium: ~2 minutes, more thorough
dcgmi diag -r 3     # long: ~30 minutes, includes pcie bandwidth, NVLink, heat

Run level 1 as part of health checks; level 2 nightly on drained nodes; level 3 on suspected-faulty nodes before returning to service.

Node lifecycle automation

A mature cluster has automation that closes the loop:

1. Health check fails → node DRAINED with reason.
2. Monitoring sees the drain → ticket created in the ops system.
3. Diagnostic suite runs automatically on the drained node.
4. If hardware fault confirmed → human triage.
5. If transient (e.g., one ECC retired) → auto-resume after diagnostic passes.
6. Persistent issues → node tagged for hardware replacement.

The performance engineer's role: when a job's straggler analysis points at a specific node, ensure that node enters this lifecycle and doesn't get re-allocated to the next unsuspecting job.

The drain culture

A cluster's culture about draining matters. Two failure modes:

• Too cautious: a node with one transient hiccup stays drained for days; capacity is wasted.
• Too aggressive: the same flaky node returns to service repeatedly and ruins multiple jobs before someone notices.

The right balance is "drain on first sign of trouble; require diagnostic-passing return-to-service; track per-node failure history; retire nodes with chronic issues." The performance engineer contributes to this by surfacing the per-node correlation: "this node has been a straggler in 4 of the last 10 jobs; here's the data."

A cluster's effective capacity isn't its node count; it's its node count minus bad nodes minus drained capacity. Healthy node lifecycle automation is what keeps that number close to the first.

§26The state machine: every job state explained

Slurm tracks a job through a state machine. Knowing every state and its valid transitions is the foundation for "why is my job stuck?"

The state-and-reason pair

Every job state, especially PENDING, comes with a Reason. The pair is the diagnostic key. squeue -o "%i %T %r" shows JobID, State, Reason.

The state transitions that surprise people

• RUNNING → REQUEUED: a node failed; with --requeue, the job goes back to the queue. To the user, the job "disappeared" briefly and re-emerged with the same ID.
• RUNNING → NODE_FAIL → COMPLETED: NODE_FAIL is the terminal state when --requeue is not set.
• COMPLETING for minutes: epilog is slow, or output flushing is, or filesystem is. Watch for this pattern; it indicates infrastructure issue.
• PENDING → CANCELLED with "DependencyNeverSatisfied": an upstream job failed and afterok can never trigger.

§27PENDING reasons decoded

The Reason field on a PENDING job is the single most important diagnostic in everyday Slurm use. Below: every common reason, what it means, and what to do about it.

"Priority" — the most common

If your job is PENDING with reason "Priority", it means: there are jobs ahead of you in priority order, and they together would consume the resources you want. Wait, raise priority (if you have a way), or change request to something easier (shorter, smaller).

"Resources" — the second most common

You're at the top of priority but the resources aren't physically available. Different from "Priority"; here, there's no contention from queued jobs, just no free hardware. Check sinfo to see how much is free in the partition.

The "I waited an hour and my job didn't start" diagnostic

Q: What's the Reason in squeue?
   |
   +-- Priority -> look at sprio. Job ahead of you?
   |     +-- yes -> wait, or lower request, or change QoS for priority bias.
   |     +-- no  -> backfill cycle isn't reaching you. Check bf_max_job_test.
   |
   +-- Resources -> sinfo -t IDLE | wc shows free nodes.
   |     +-- enough free -> probably constraint mismatch. Check --constraint.
   |     +-- not enough -> wait for jobs to finish. Estimate via squeue --start.
   |
   +-- ReqNodeNotAvail -> the specific nodes you named are down/drained.
   |                       Drop --nodelist or wait for the nodes.
   |
   +-- QOS/Assoc...Limit -> you're over a limit. sacctmgr show association.
   |                          Wait for usage to decay, or talk to the admin.
   |
   +-- Dependency -> upstream job. Check it with squeue.
   |
   +-- BadConstraints / InvalidQOS / etc. -> bug in submit script. Fix and resubmit.

Estimated start time

# Slurm's estimate of when each pending job will start
squeue --start -u $USER

# For a specific job
scontrol show job 12345 | grep StartTime

The estimate comes from the backfill scheduler's lookahead. It's an upper bound — jobs can start earlier if higher-priority jobs finish ahead of schedule — but rarely later. Useful for users planning their day.

§28Logs: what's where, what to grep for

Every Slurm bug is a log-grep problem. Knowing what to look for, in which file, on which host, is half the skill.

Log file map

Tracing one job

# 1. Get the full job record
scontrol show job 12345 > /tmp/job12345.txt

# 2. Controller log entries for this job
grep "JobId=12345" /var/log/slurmctld.log

# 3. Pull the allocated nodes and check their slurmd logs
NODES=$(scontrol show job 12345 -o | grep -oP 'NodeList=\S+' | cut -d= -f2)
for n in $(scontrol show hostnames $NODES); do
  ssh $n "grep '12345' /var/log/slurmd.log" > /tmp/slurmd-$n.log
done

# 4. Look at accounting for the final state
sacct -j 12345 -o "JobID,State,ExitCode,Reason,DerivedExitCode,Comment"

The grep patterns that pay off

# Recent prolog failures (mysterious requeues come from here)
grep "prolog" /var/log/slurmd.log | grep -i fail

# Communication failures between controller and slurmd
grep -E "(unable to connect|connection refused|timeout)" /var/log/slurmctld.log

# OOM events
grep -E "(out of memory|oom|memory limit)" /var/log/slurmd.log

# NCCL or torch errors that ended up in job stderr
grep -E "(NCCL|c10d|distributed)" /path/to/job.err

# Backfill scheduler taking too long
grep "backfill" /var/log/slurmctld.log | grep -E "seconds"

Increasing log verbosity

# slurm.conf
SlurmctldDebug=debug3       # or info, verbose, debug, debug2, debug3
SlurmdDebug=debug
DebugFlags=Backfill,Gres,Priority,Steps    # subsystem flags

Bumping SlurmctldDebug from info to debug generates much more output (hundreds of MB/day on a busy cluster). Use temporarily; turn it back down.

The DebugFlags approach is better for targeted debugging: DebugFlags=Backfill adds backfill-cycle details without flooding other categories.

sdiag — controller introspection

$ sdiag
*******************************************************
sdiag output at Sun May 25 14:23:19 2026
Data since      Sun May 25 00:00:02 2026
*******************************************************
Server thread count:  9
Agent queue size:      0      # RPCs waiting to fan out — >0 is congestion
Agent count:           0

Jobs submitted: 8472
Jobs started:   8311
Jobs completed: 8204

Main schedule statistics (microseconds):
  Last cycle:   12343            # last scheduling cycle duration
  Max cycle:    442181           # worst case in this period
  Total cycles: 1843
  Mean cycle:   28401            # average
  ...

Backfilling stats:
  Total backfilled jobs (since last slurm start): 3209
  Total backfilled jobs (since last stats cycle start): 187
  Last cycle:    94002            # last backfill duration
  Mean cycle:    88301
  ...

sdiag is the single best command for "is the controller healthy?" Agent queue size > 0 sustained means RPCs are backing up. Mean scheduling cycle > a few hundred ms suggests tuning is needed.

§29Decision trees for common failures

Six recurring user-reported issues, mapped to diagnostic paths. The senior engineer can run these without thinking.

"My job won't start"

Q: squeue -j JOBID -o "%T %r" — state and reason?
  PENDING / Priority      -> wait, or see §27
  PENDING / Resources     -> sinfo for capacity; estimate via squeue --start
  PENDING / Dependency    -> check upstream job
  PENDING / *Limit        -> hit a limit, see §22
  PENDING / BadConstraints-> fix submit script
  PENDING / NodeDown      -> nodes are down; scontrol show nodes
  CONFIGURING (long)      -> prolog is slow or hanging
  RUNNING                 -> it already started; user is confused

"My job ran but produced no output"

Q: Did the output file get created?
  no  -> path doesn't exist or no write permission. Check --output path.
  yes, empty -> job exited too fast OR stdout is buffered. Add python -u.
  yes, partial -> job died. Check job state.

Q: sacct -j JOBID -o "JobID,State,ExitCode,Reason"
  State=COMPLETED ExitCode=0:0 -> job ran clean, just produced nothing
  State=FAILED                 -> grep .err for the trace
  State=OUT_OF_MEMORY          -> increase --mem
  State=TIMEOUT                -> increase --time
  State=NODE_FAIL              -> hardware; check node status, requeue
  State=CANCELLED              -> someone (or something) ran scancel

"My distributed training hangs at startup"

Q: Did all ranks make it past process_group init?
  no  -> bootstrap problem
         -> grep "Connecting" in NCCL_DEBUG=INFO output
         -> verify MASTER_ADDR is reachable from all ranks
         -> firewall? scontrol show node | check
  yes, some ranks did, some didn't
         -> mismatched config; some ranks have different env
         -> check NCCL_IB_HCA, MASTER_PORT same everywhere
  yes all initialized, hangs at first collective
         -> network not actually working between some pairs
         -> ib_write_bw between the suspect pairs
         -> check NCCL channel construction in INFO log

"My job is using less GPU than expected"

Q: nvidia-smi inside the job shows what?
  fewer GPUs than allocated -> CUDA_VISIBLE_DEVICES wrong
                            -> torch.cuda.device_count() check
  all GPUs, low utilization -> framework issue, see AI perf manual §09
  GPUs showing wrong model  -> different gres.conf type than expected

Q: scontrol show job — what GRES was allocated?
  matches request -> Slurm did its job; investigate further in framework
  doesn't match    -> gres.conf miscount, or --gres typo, or partition limits

"Jobs from one user are failing on a specific node"

Q: Are the failures NODE_FAIL, FAILED, or something else?
  NODE_FAIL  -> the node itself died during the job
  OUT_OF_MEMORY -> per-job; user requested too little memory OR memory leak
  FAILED with non-zero exit -> the job's command exited
  TIMEOUT    -> wall clock hit; not node-related

Q: Is it always the same node?
  yes  -> investigate the node: dcgmi diag, XID errors, IB state
  no   -> distributed across nodes; likely user code issue, not infra

Q: Same user only, or others affected?
  same user only -> user-script issue (env, paths, output handling)
  others too     -> infra issue; drain node, escalate to ops

"Backfill seems broken — many jobs would fit but don't run"

Q: Check sdiag. Is backfill running successfully?
  Last cycle > 60s        -> backfill is stuck; controller may be overloaded
  Total backfilled = 0    -> bf_max_job_test or bf_max_job_user too restrictive
  Reasonable counts       -> the candidate jobs may not be eligible for backfill

Q: Are pending jobs in backfill-eligible states?
  Held, dependent, ineligible BeginTime, etc. -> backfill skips them
  Have explicit --time limits?               -> jobs without --time can't backfill
                                                because backfill needs to know
                                                they'll fit in the window

§30Architectural comparison

The two systems were designed for different problems, and their architectures still reflect that.

Side by side

Why etcd vs in-memory matters

Slurm's state model gives it speed: scheduling decisions don't hit a database. Slurm can schedule tens of thousands of jobs per minute on hardware where Kubernetes would struggle. The flip side: state loss is catastrophic (state-save FS failure = job state loss), and HA is limited to active-passive.

Kubernetes's etcd backend gives it durability and horizontal HA: kill any control-plane node and the cluster keeps working. The cost: every state change is a raft consensus write. Pod creation rates of 100/sec are healthy; 1000/sec needs careful tuning; 10000/sec is research territory. Slurm tops these numbers by orders of magnitude.

The declarative vs imperative split

Kubernetes: "I want a Deployment with 5 replicas of my container." The system figures out how to make that true and keeps it true. If a pod dies, a new one starts.

Slurm: "Run this script with these resources for up to 24 hours." Slurm runs it, and if it dies, it stays dead unless you submitted with --requeue.

The declarative model is a beautiful match for service workloads (you want 5 replicas, always). It is awkward for training (you don't want 5 replicas of a 70B training run; you want one run that, if it dies, restarts from checkpoint). Both systems have grown features to address the mismatch, but the underlying mental models differ.

§31Scheduling models compared

The schedulers do very different things by default.

The default scheduling unit

Slurm schedules a job: a set of resources allocated together for a bounded time. Either all resources arrive or none do (gang scheduling is the default). Time-limited.

Kubernetes schedules a pod: a set of containers running on one node. Pods are individual; a Deployment of 5 pods means 5 independent scheduling decisions. No time limit by default.

For AI training that needs all-or-nothing on N nodes, this is the largest gap. Kubernetes ecosystem closed it with gang schedulers: Volcano, Kueue, KAI Scheduler (NVIDIA), Run:ai. They group pods into PodGroups that schedule together. With these in place, Kubernetes can do gang scheduling — but the operator has to install and configure them.

Backfill, queueing, priority

The Kueue / Volcano / KAI ecosystem

The Kubernetes community has converged on a stack to bring HPC-style scheduling to k8s:

• Kueue (Kubernetes upstream): job-level queueing, cohorts (groups of teams), borrowing/lending of quotas, gang scheduling via PodGroups, dynamic resource reclamation.
• Volcano: full-featured batch scheduler with gang scheduling, fair-share, preemption, backfill, and topology awareness. Most "HPC-on-k8s" deployments use Volcano.
• KAI Scheduler (NVIDIA, open-sourced 2025): GPU-first batch scheduler designed for AI workloads — gang scheduling, fractional GPUs, topology-aware placement, hierarchical queues.
• Run:ai (acquired by NVIDIA): commercial product layered on Kubernetes, providing fair-share GPU scheduling, fractional GPUs, hierarchical queues. Very popular in enterprise AI.
• YuniKorn (Apache): a Kubernetes scheduler aimed at big-data + ML, supporting hierarchical queues and gang scheduling.

The trajectory: out-of-the-box Kubernetes is weak for batch ML; Kubernetes + Volcano/Kueue/KAI is competitive with Slurm; Kubernetes + Run:ai is a higher-cost, higher-feature commercial choice.

What this means in practice

For a single user submitting a 64-GPU training job:

• Slurm: sbatch --nodes=8 --gpus-per-node=8 train.sh. The 64 GPUs allocate or pend together. Done.
• Kubernetes (vanilla): write a Job with 64 pods, each requesting 1 GPU. Pods schedule independently; you may end up with 30 pods running and 34 waiting, blocking other workloads. Training can't start.
• Kubernetes + Volcano: PodGroup of 64, gang-scheduled. Behaves like Slurm.

The pattern recurs: anything Slurm does natively, Kubernetes can do with the right add-ons; without them, things break in surprising ways.

§32Resource isolation & cgroups

Both systems use cgroups to enforce resource limits, but apply them differently.

Slurm

Slurm sets cgroups per task: every srun task gets its own cgroup, with CPU set, memory limit, and device list. Containers (via pyxis) add another layer of namespacing, but the cgroup boundary is Slurm's.

Kubernetes

The kubelet sets cgroups per pod (and per-container within pod). The pod is the cgroup boundary. Multi-container pods share network and IPC namespaces but have their own cgroups.

The GPU-isolation story

Both systems rely on NVIDIA's container toolchain to expose GPUs into containers/cgroups. The mechanism is the same; the orchestration is different.

Slurm: when a job is allocated, slurmstepd writes the allowed GPU devices to the cgroup device controller. The job's CUDA_VISIBLE_DEVICES is also set. Both layers must agree.

Kubernetes: the NVIDIA Device Plugin advertises GPUs as extended resources (nvidia.com/gpu). When a pod requests them, the kubelet asks the device plugin which devices to provide; the device plugin returns mount specifications and environment variables (including NVIDIA_VISIBLE_DEVICES). The container runtime sets these up.

For MIG and shared GPUs (MPS), both systems have grown support, but Kubernetes has more momentum here: the NVIDIA Device Plugin supports MIG slices, time-slicing, and CUDA MPS as configurable resources.

§33Containers & packaging

The packaging stories differ.

Implications

• Iteration speed: Slurm wins here. Edit Python file, resubmit, done. Kubernetes: build image, push image, update manifest, apply, wait for pull, run.
• Reproducibility: Kubernetes wins. Image digests are immutable; "what software ran" is in the manifest. Slurm: depends on whatever the user's environment was at submit time.
• Distribution: Kubernetes wins. Pull an image, it's the same everywhere. Slurm: shared filesystem dependency, or per-user environment setup.
• Layering complexity: Kubernetes adds a registry + image-builder workflow that Slurm shops can skip.

The container-image cache problem

On a 1000-node Kubernetes cluster, pulling a 10 GB image on first use for a 1000-pod job means 1000 simultaneous registry hits. Modern Kubernetes deployments solve this with pre-pulled images, image streaming (estargz, zstd:chunked), or peer-to-peer (Dragonfly, Spegel). Slurm avoids the problem entirely because enroot caches per-node.

For AI workloads with 20+ GB containers (CUDA + PyTorch + NCCL + framework), image distribution is a real cluster-design question on Kubernetes that doesn't exist on Slurm.

§34Networking, storage, devices

Networking

Slurm: jobs see the host network. They use the cluster's IB/Ethernet fabric directly with whatever IP addresses the host has. NCCL picks NICs via topology. Firewall rules are operator-managed and stable.

Kubernetes: every pod gets a network namespace and a CNI-assigned IP. The CNI plugin (Calico, Cilium, Flannel, etc.) handles routing. For IB/RDMA, special CNI plugins (Multus + SR-IOV) expose the IB device into the pod alongside the standard pod network.

The Multus + SR-IOV pattern is the production approach for Kubernetes AI clusters: each pod gets a pod-network NIC (for the Kubernetes service mesh) plus one or more IB virtual functions (for NCCL traffic). It works, but it's significantly more complex than Slurm's "the network is the network."

Storage

The HPC-style storage model (one big parallel FS, mounted everywhere) is intrinsic to Slurm. Kubernetes can achieve the same with the right CSI drivers, but the cultural default is "pods don't share state outside their explicit volumes."

Devices

GPUs, NICs, NVMe scratch, FPGAs, custom accelerators — all are "devices" in both systems. Slurm uses GRES (§14); Kubernetes uses the Device Plugin framework + extended resources. Both mechanisms work; both require per-device-type plugins.

For GPUs specifically, NVIDIA maintains plugins for both systems, and the feature parity is close: MIG, time-slicing, MPS, vGPU all supported on both. The choice rarely turns on device support.

§35Operator & user ergonomics

The day-to-day feel of working with each system differs in ways that matter for adoption.

User ergonomics

Operator ergonomics

§36AI/ML fit: which wins where

The honest answer: depends on the workload mix. Below: the workload classes most AI orgs run, and which system fits each better.

The performance question

For a well-tuned, large-scale training job, is one faster than the other? Almost always: no. The bottleneck is GPU, NCCL, fabric — not the scheduler. Both systems can place a job on the right nodes, with the right cgroups, with NCCL using the right NICs. Once running, the workload is identical.

The schedulers do differ in scheduling overhead: Slurm starts jobs faster, has lower controller overhead per submission, and handles deep queues better at the upper end. For a cluster running 1000 jobs/day this doesn't matter. For one running 100,000 jobs/day, it does.

The fleet-wide efficiency question

Slurm's fair-share and accounting tooling is more mature for "how much did each project use last month" questions. Kubernetes is catching up via Kueue cohorts and per-namespace usage reporting, but the out-of-box experience favors Slurm. For an AI Performance and Efficiency Engineer, this matters: the cluster you're auditing should give you per-account, per-job, per-resource utilization without custom integration work.

§37Hybrid patterns & decision framework

Many real AI shops end up running both. Knowing the patterns saves you from re-inventing them.

Hybrid pattern A: Slurm for training, Kubernetes for inference

The most common split. Reasoning: training is HPC-shaped (gang-scheduled, big, periodic); inference is web-shaped (replicated, scaling, multi-tenant). Different clusters, different schedulers, sometimes different hardware (training on H100/B200, inference on a mix of H100, L40S, A10G).

Operational reality: two clusters, two on-call rotations, sometimes two storage tiers. The trade is paying for the duplication vs forcing one system to do both jobs poorly.

Hybrid pattern B: Kubernetes for everything, Slurm-on-Kubernetes for batch

Newer pattern. Kubernetes is the substrate; Slurm runs as Kubernetes pods (the controller as a Deployment, the slurmds as a DaemonSet). Tooling like "Slinky" (SchedMD's Kubernetes operator for Slurm) makes this real.

Why: you want one orchestration plane, but you also want Slurm's batch features for training. The cost: complexity, since now you have both schedulers' issues + an integration layer.

Hybrid pattern C: Slurm with Kubernetes-style features bolted on

Slurm clusters that adopt the JWT REST API (slurmrestd), pyxis containers, and prometheus-based monitoring — getting closer to Kubernetes ergonomics without leaving Slurm. Most "modernizing HPC clusters" land here.

Hybrid pattern D: Kubernetes with KAI Scheduler or Volcano

The other direction: Kubernetes, but with a batch-quality scheduler. KAI Scheduler (NVIDIA) or Volcano gives you gang scheduling, fair-share, topology awareness. For a Kubernetes-native team running AI workloads, this is increasingly the answer.

The decision framework

Q: What's your team's existing infrastructure expertise?
   HPC/MPI background     -> Slurm. Build on what they know.
   Cloud/k8s background   -> Kubernetes + KAI/Volcano. Same.
   Mixed                  -> Likelier to hybridize.

Q: What's the workload mix?
   Mostly large training  -> Slurm or Kubernetes + KAI. Both work.
   Mostly inference       -> Kubernetes. No competition.
   Mostly research, small -> Slurm. Simpler.
   Strict mix of all      -> Hybrid (training/inference split).

Q: Cluster scale and operational team?
   < 100 nodes, small team  -> Slurm. Simpler ops.
   100-1000 nodes           -> Either, with mature ops.
   > 1000 nodes             -> Slurm scales further, but k8s+KAI works.

Q: Multi-tenant isolation requirements?
   Loose (research)         -> Slurm.
   Strict (production+research mixed) -> Kubernetes RBAC + namespaces.

Q: Cloud-portability requirements?
   Cloud-native, multi-region -> Kubernetes (managed offerings everywhere).
   On-prem, single site       -> Either.

The pragmatic answer

The performance engineer's job is the same on both: profile the workload, find the bottleneck, fix it, prevent the next one. The orchestrator is plumbing; the work doesn't change.

§38Production debugging cases

Every case here turned on one specific log line or one specific command output. The senior signal is knowing which log to read, in which file, on which host, in the first minute.

§39Cheatsheets: commands, env, conf

§40The senior signals & closing

What distinguishes a senior cluster engineer from a strong individual contributor. Slurm-specific, but most of these generalize.

You read the log file, not the user's narration

"My job didn't run" is the user's story. The reason field in squeue and the lines around JobId= in slurmctld.log are the truth. The senior signal: ask for the JobID, run the three diagnostic commands (§29), conclude in 30 seconds.

You know the state machine cold

NODE_FAIL, OUT_OF_MEMORY, PREEMPTED, TIMEOUT, CANCELLED — each has different causes, different evidence, different fixes. Knowing the difference means you don't have to guess.

You can explain priority to a researcher in 30 seconds

"Your job is pending because you have higher fair-share usage this week than the team average. It will run after these three higher-priority jobs finish — about 90 minutes. Or you can use the preemptible QoS for an immediate start if you accept that the job can be killed."

You know the operator's knobs

The performance engineer doesn't change slurm.conf, but knows what's in it. When ops says "we're tuning backfill," you know what they mean. When a researcher's job has weird scheduling behavior, you can ask the right questions about the cluster configuration.

You write the prolog check, not the post-mortem

Once a problem has surfaced as a job failure, much of the cost has already been paid. The senior move is to add a prolog check, a Lua submit rule, or a default in the job template — so the next person hits the issue at submit time, not 4 hours into a 6-hour run.

You bridge HPC and modern

Slurm is an HPC tradition; modern AI workloads come from a different culture. Bridging "this is how HPC has always worked" with "but we need elastic training and Kubernetes-style observability" is increasingly the role. Both sides have legitimate reasons; the bridge is technical translation, not advocacy.

You ship dashboards, not just answers

Helping a researcher with one job is good. Helping the next 50 users avoid the same issue without your involvement is the leverage. The senior engineer ships dashboards, defaults, templates, alerts — the infrastructure that makes the cluster systematically more usable.

You respect the system

Slurm has been in production at every major HPC site for two decades. The conventions feel old-fashioned because they are; they also work. Resist the urge to "modernize" before understanding why things are the way they are. The reasons are usually load-bearing.

A great Slurm engineer is the person whose presence makes researchers' workflows smoother, whose telemetry catches problems before users feel them, and whose collaborations with ops make the cluster systematically better quarter after quarter.