AutoClusters

This is a Crusoe engineering blog guide to deploying fault-tolerant distributed PyTorch training on Crusoe Managed Kubernetes. It combines Slurm, deployed through Crusoe's Slurm operator, with AutoClusters for hardware-failure detection and node replacement, Command Center for cluster topology, GPU metrics, notifications, and remediation visibility, and checkpoint-based recovery. The example uses a Slurm batch script with torchrun and a dynamic rendezvous backend to coordinate one process per GPU across multiple nodes; when a GPU or node failure is detected, the workflow notifies operators, drains and replaces the node, requeues the Slurm job, and resumes training from its last checkpoint. The guide reports recovery in under 15 minutes and covers cluster setup, NCCL and InfiniBand configuration, distributed PyTorch code, checkpointing, and failure testing. It is published as a dated Crusoe blog resource; the page does not state a separate maintenance policy.

Crusoe Managed Slurm is a managed high-performance computing cluster orchestration service on Crusoe Cloud. It provisions complete Slurm clusters on Crusoe's GPU-optimized infrastructure through a CLI command, UI form, or Kubernetes add-on, combining Slurm job scheduling with Crusoe Managed Kubernetes. A cluster includes the Slurm controller and database, SSH-accessible login nodes, shared storage mounted at `/home`, topology discovery, and GPU worker node sets. The Crusoe Slurm Operator manages the Slurm components as Kubernetes resources, while Slinky runs Slurm daemons in Kubernetes pods. The service supports topology-aware scheduling, multi-user access, shared storage, and management through Slurm workflows or Kubernetes CRDs. Managed Slurm also includes AutoClusters for hardware remediation. When a critical GPU or HCA failure is detected, the affected node is taken down, running jobs are cancelled and requeued to healthy nodes, and the failed node is replaced automatically.

Crusoe Cloud is Crusoe's cloud infrastructure platform for AI workloads, providing compute, storage, networking, GPU node pools, and a console for managing and observing GPU infrastructure. Crusoe describes its offering as AI infrastructure and cloud compute with an energy-first approach. Its Managed Slurm runs on Kubernetes: Slurm provides gang scheduling, topology awareness, and familiar sbatch workflows, while Kubernetes supplies dynamic resource management, node health handling, and observability. The platform's AutoClusters can notify operators, drain and replace a failed GPU node, requeue the workload, and resume training from a checkpoint without manual intervention; the demonstrations also cover sharing GPUs between training and inference as demand changes.

Crusoe is an AI infrastructure and cloud-compute company providing compute, storage, networking, and GPU capacity for AI workloads. Its managed orchestration stack runs Slurm on Kubernetes: Slurm supplies gang scheduling, topology awareness, and familiar sbatch workflows, while Kubernetes provides dynamic resource management, node health handling, and observability. Crusoe's AutoClusters automates GPU-failure remediation by notifying, draining, replacing, requeuing, and resuming workloads, including checkpointed distributed PyTorch training; the platform also supports sharing GPUs between training and inference as demand changes. Crusoe describes its infrastructure as using an energy-first approach and offering 24/7 support.

Crusoe Managed Kubernetes (CMK) is a managed Kubernetes platform from Crusoe that provides the infrastructure layer for Crusoe's managed Slurm service. In the described architecture, Kubernetes supplements Slurm with dynamic resource management, node health handling, and observability while Slurm retains gang scheduling, topology awareness, and familiar sbatch workflows. The platform supports GPU-oriented workloads and underlies automation that can notify about a GPU failure, drain and replace the affected node, requeue the job, and resume training from a checkpoint.