The Slurm Scheduler
Slurm is a widely used workload manager that allocates cluster nodes to jobs, enforces fair sharing, and launches parallel programs.
What a scheduler does
A shared cluster has far more work requested than nodes available at any moment. A workload manager such as Slurm (Simple Linux Utility for Resource Management) accepts job requests, decides which run when and where, enforces policies for fair sharing among users, and launches the processes. Users describe what they need (nodes, cores, GPUs, memory, time limit) and Slurm fits those requests onto the machine.
The core commands
- sbatch: submit a batch script that runs when resources are granted.
- srun: launch a parallel step, often the MPI program, within an allocation.
- salloc: obtain an interactive allocation for exploratory work.
- squeue and sacct: inspect pending/running jobs and completed-job accounting.
- scancel: cancel a queued or running job.
A batch script
#!/bin/bash
#SBATCH --job-name=hyperion-solve
#SBATCH --nodes=64
#SBATCH --ntasks-per-node=4
#SBATCH --gpus-per-node=4
#SBATCH --time=02:00:00
srun ./transport_solver --input case.cfgScheduling policy
Slurm orders the queue by priority (a mix of fair-share history, job age, size, and partition weights) and uses backfill to start smaller jobs early when they fit in the gap before a large job's reserved start, as long as they do not delay it. Understanding backfill explains a useful tactic: an accurate, modest time limit lets a job squeeze into gaps, whereas an inflated limit makes it wait for a large contiguous window.
In practice
A Hyperion campaign is a set of sbatch scripts, each requesting the node and GPU count a case needs with a realistic wall-time. Requesting only what is used improves both queue turnaround (via backfill) and the site's overall utilization, and it keeps accounting honest for capacity planning.