Tool
Vortex is an open-source columnar file format for streaming compressed analytical and machine-learning training data from object storage such as S3 to CPUs and GPUs. It separates logical and physical types and uses layouts, zone maps, cascading encodings, and byte coalescing to reduce unnecessary I/O, decompression, and memory copies; its encodings include dictionary arrays, run-end encoding, and bit-packing, allowing computation on compressed data. The format is designed for zero-copy execution with Apache Arrow and integration with DataFusion, DuckDB, and Polars, with scan paths that can use buffer pools, kTLS, and RDMA to move data toward CPU or GPU execution.
1 use taken from transcripts — each links to the moment in the video.
An open-source columnar file format designed to stream compressed analytical and training data directly from S3 to CPUs and GPUs. It uses layouts, zone maps, cascading encodings, and compute on compressed data to reduce I/O, decompression, and memory-copy overhead.
1 in the library.