Polars

Vortex is an open-source columnar file format for streaming compressed analytical and machine-learning training data from object storage such as S3 to CPUs and GPUs. It separates logical and physical types and uses layouts, zone maps, cascading encodings, and byte coalescing to reduce unnecessary I/O, decompression, and memory copies; its encodings include dictionary arrays, run-end encoding, and bit-packing, allowing computation on compressed data. The format is designed for zero-copy execution with Apache Arrow and integration with DataFusion, DuckDB, and Polars, with scan paths that can use buffer pools, kTLS, and RDMA to move data toward CPU or GPU execution.

Apache DataFusion is an extensible query engine written in Rust that uses Apache Arrow as its in-memory format. It provides SQL and DataFrame APIs, a query planner, and a columnar, streaming, multithreaded, vectorized execution engine for partitioned data sources. It supports CSV, Parquet, JSON, and Avro out of the box, and can be extended with custom data sources, query languages, functions, operators, and other workload-specific components. DataFusion is developed as an Apache Software Foundation project and is available as libraries and binaries for building database and analytics systems.

DuckDB is an in-process SQL OLAP database management system for analytical workloads. It uses a PostgreSQL-inspired SQL dialect, can run from edge devices to servers, and reads and writes formats including CSV, JSON, Parquet, Iceberg, and Vortex both locally and in object storage. Extensions add functions and format support, while native clients provide APIs for Python, Go, Java, Node.js, C, C++, R, Rust, and ODBC. DuckDB is open source and MIT-licensed, with governance by the independent DuckDB Foundation.

Apache Arrow is an open-source, language-independent universal columnar in-memory format and multi-language toolbox for fast data interchange and in-memory analytics. It provides a standard columnar representation for flat and nested data, an IPC format, Flight RPC, zero-copy reads, and libraries and development building blocks across languages including C, C++, .NET, Go, Java, JavaScript, Julia, MATLAB, Python, R, Ruby, Rust, and Swift. The project is an Apache Software Foundation effort developed by a distributed developer community, with readers and writers for formats such as Parquet and CSV.

Polar is a billing platform and Merchant of Record aimed at developers building AI-era software. It provides payments, subscriptions, and usage-based billing while handling global tax and compliance for its customers.