NextTransformer

Transformers is Hugging Face's open-source model-definition framework and Python library for inference and training with pretrained text, computer vision, audio, video, and multimodal machine-learning models. It centralizes model definitions behind a unified API and provides the Pipeline interface, which preprocesses inputs and returns task-specific outputs for applications such as text generation, automatic speech recognition, image classification, and visual question answering. Models can be downloaded and cached from the Hugging Face Hub, and the library supports moving models between PyTorch, JAX, and TensorFlow workflows while exposing model internals for customization. It requires Python 3.10 or later and PyTorch 2.5 or later for the documented installation.

τ₀-VLA is an open-source hierarchical robot foundation model and reference implementation for long-horizon manipulation. A memory-augmented high-level policy generates the next subtask and uses world-model-guided test-time computation to search over alternatives when additional reasoning is needed; a generalist low-level policy then executes the selected subtask across robot embodiments. The low-level policy combines a Qwen3.5 vision-language backbone with a Mixture-of-Transformers action expert trained through conditional flow matching, using a unified 40-dimensional state/action space and multimodal co-training on tens of thousands of hours of heterogeneous real-world robot data. The repository provides LeRobot-format data loading, prompting, masking and normalization, embodiment-specific adapters, training and post-training recipes, deployment contracts, a policy server, and open-loop evaluation tools. It includes an AgiBot World example subset and templates for other datasets and robots. Public v1 serving supports joint-control checkpoints; native end-effector data can be used for training but end-effector serving is not supported in that release. The reference environment uses Python 3.11, CUDA 12.8, and PyTorch 2.7.1. Code and model weights are released under the Apache License 2.0.

KTransformers is an open-source research framework for efficient large-language-model inference and fine-tuning through heterogeneous CPU-GPU computing. Its user-facing capabilities are provided through the kt-kernel source tree: high-performance inference and supervised fine-tuning (SFT). For mixture-of-experts models, it uses heterogeneous expert scheduling, placing frequently used or “hot” experts on GPUs and other experts on CPUs with optimized AMX, AVX512, or AVX2 kernels and NUMA-aware memory management. It supports CPU-side INT4/INT8 quantized weights, GPU-side GPTQ, native BF16 and FP8 precision, and multiple CPU, GPU, and NPU backends. The SFT capability integrates with LlamaFactory for hybrid CPU-GPU fine-tuning of large MoE models, including LoRA and full-parameter training, with INT4/INT8 quantization options.

NVIDIA Model Optimizer (ModelOpt) is an open-source library for optimizing deep-learning and generative-AI models for inference deployment. It accepts Hugging Face, PyTorch, or ONNX models and provides Python APIs for composing post-training quantization, quantization-aware training and distillation, pruning, neural architecture search, speculative decoding, and sparsity techniques, producing optimized or quantized checkpoints. The resulting checkpoints can be exported for inference frameworks including TensorRT-LLM, TensorRT, vLLM, and SGLang. The library also integrates with NVIDIA Megatron-Bridge, Megatron-LM, Hugging Face Accelerate, Transformers, and Diffusers for supported training and export workflows. It is distributed as the nvidia-modelopt package on PyPI and can also be installed from the NVIDIA GitHub repository or used through NVIDIA container images.

Megatron is a training framework that handles data-parallel ranks and model-weight optimization. In the described multimodal training pipeline, it acts as a downstream consumer of the data pipeline rather than managing the data-loading bottlenecks.