Apache Arrow

Apache DataFusion is an extensible query engine written in Rust that uses Apache Arrow as its in-memory format. It provides SQL and DataFrame APIs, a query planner, and a columnar, streaming, multithreaded, vectorized execution engine for partitioned data sources. It supports CSV, Parquet, JSON, and Avro out of the box, and can be extended with custom data sources, query languages, functions, operators, and other workload-specific components. DataFusion is developed as an Apache Software Foundation project and is available as libraries and binaries for building database and analytics systems.

Polars is a Rust-backed data-processing engine that uses Apache Arrow's columnar in-memory format, multithreaded execution, SIMD vectorization, and a lazy query optimizer. Its Python API supports DataFrames, declarative expressions, joins, aggregations, window functions, schema-aware ingestion, and Parquet scanning. The lazy API applies predicate and projection pushdown before execution, while eager operations load data immediately; the page also describes out-of-core processing and PySpark migration workflows. The video identifies Polars as a data-processing engine that can interoperate with Vortex and other columnar-data systems.

Vortex is an open-source columnar file format for streaming compressed analytical and machine-learning training data from object storage such as S3 to CPUs and GPUs. It separates logical and physical types and uses layouts, zone maps, cascading encodings, and byte coalescing to reduce unnecessary I/O, decompression, and memory copies; its encodings include dictionary arrays, run-end encoding, and bit-packing, allowing computation on compressed data. The format is designed for zero-copy execution with Apache Arrow and integration with DataFusion, DuckDB, and Polars, with scan paths that can use buffer pools, kTLS, and RDMA to move data toward CPU or GPU execution.

Apache Spark is a unified analytics engine for large-scale data processing, developed as an Apache project. It provides high-level APIs in Scala, Java, Python, and R, and executes general computation graphs for data analysis. Its higher-level components include Spark SQL and DataFrames, the pandas API on Spark, MLlib for machine learning, GraphX for graph processing, and Structured Streaming for stream processing. Spark can run locally or submit workloads to clusters through Spark, YARN, or Kubernetes integrations, and is used in large-scale data pipelines such as those underlying Poolside's model factory.

Dalaran is an Apache-2.0 robotics-focused observability and visualization stack for recording, replaying, and inspecting multimodal time-series data such as ROS 2 topics, LiDAR sweeps, camera streams, IMU readings, and TF transforms on a synchronized timeline. It is a hard fork of Rerun, uses an Arrow-backed columnar chunk store, renders synchronized 2D and 3D views, and exposes recorded data as dataframes for computation. The project supports Python, Rust, and C++ SDKs, a viewer and CLI, ROS 2 data, Rosbag 2 and MCAP replay, existing Rerun .rrd recordings, and Dalaran .dlr recording files with saved .dbl blueprints; recordings can be written to files or streamed to a viewer or remote catalog.