Crusoe Cloud

Crusoe is an AI infrastructure and cloud-compute company providing compute, storage, networking, and GPU capacity for AI workloads. Its managed orchestration stack runs Slurm on Kubernetes: Slurm supplies gang scheduling, topology awareness, and familiar sbatch workflows, while Kubernetes provides dynamic resource management, node health handling, and observability. Crusoe's AutoClusters automates GPU-failure remediation by notifying, draining, replacing, requeuing, and resuming workloads, including checkpointed distributed PyTorch training; the platform also supports sharing GPUs between training and inference as demand changes. Crusoe describes its infrastructure as using an energy-first approach and offering 24/7 support.

Crusoe Managed Slurm is a managed high-performance computing cluster orchestration service on Crusoe Cloud. It provisions complete Slurm clusters on Crusoe's GPU-optimized infrastructure through a CLI command, UI form, or Kubernetes add-on, combining Slurm job scheduling with Crusoe Managed Kubernetes. A cluster includes the Slurm controller and database, SSH-accessible login nodes, shared storage mounted at `/home`, topology discovery, and GPU worker node sets. The Crusoe Slurm Operator manages the Slurm components as Kubernetes resources, while Slinky runs Slurm daemons in Kubernetes pods. The service supports topology-aware scheduling, multi-user access, shared storage, and management through Slurm workflows or Kubernetes CRDs. Managed Slurm also includes AutoClusters for hardware remediation. When a critical GPU or HCA failure is detected, the affected node is taken down, running jobs are cancelled and requeued to healthy nodes, and the failed node is replaced automatically.

Crusoe Managed Kubernetes (CMK) is a managed Kubernetes platform from Crusoe that provides the infrastructure layer for Crusoe's managed Slurm service. In the described architecture, Kubernetes supplements Slurm with dynamic resource management, node health handling, and observability while Slurm retains gang scheduling, topology awareness, and familiar sbatch workflows. The platform supports GPU-oriented workloads and underlies automation that can notify about a GPU failure, drain and replace the affected node, requeue the job, and resume training from a checkpoint.

AutoClusters is a Crusoe Cloud Kubernetes add-on for detecting and remediating hardware failures in GPU cluster workloads. It continuously monitors nodes for configured hardware errors; when a failure is detected, it can gracefully terminate workloads, restart or replace the unhealthy node, and reschedule pods on a healthy node. In Crusoe's Managed Slurm workflow, the remediation flow drains and replaces a failed node, requeues Slurm jobs, and allows checkpointed training to resume without manual intervention. Remediation is configured per issue type, including GPU and interconnect errors, with actions of REPLACE_NODE or OFF; all remediation actions default to OFF until explicitly enabled. Workloads can opt out with a pod label, and the documentation recommends preStop hooks so applications can save state before termination. Support is limited to specified Crusoe GPU instance types and compatible Crusoe Managed Kubernetes versions, and does not cover multi-tenant VM types where node replacement cannot be safely determined.

This is a Crusoe engineering blog guide to deploying fault-tolerant distributed PyTorch training on Crusoe Managed Kubernetes. It combines Slurm, deployed through Crusoe's Slurm operator, with AutoClusters for hardware-failure detection and node replacement, Command Center for cluster topology, GPU metrics, notifications, and remediation visibility, and checkpoint-based recovery. The example uses a Slurm batch script with torchrun and a dynamic rendezvous backend to coordinate one process per GPU across multiple nodes; when a GPU or node failure is detected, the workflow notifies operators, drains and replaces the node, requeues the Slurm job, and resumes training from its last checkpoint. The guide reports recovery in under 15 minutes and covers cluster setup, NCCL and InfiniBand configuration, distributed PyTorch code, checkpointing, and failure testing. It is published as a dated Crusoe blog resource; the page does not state a separate maintenance policy.