Tool Open source
Glyd is a Rust lossless-compression library and command-line tool with a C ABI, streaming interfaces, and Python and Go bindings. It targets object storage, data lakes, logs, telemetry, backups, versioned exports, RPC payloads, and caches as an alternative to LZ4, Snappy, and zstd.
Its record mode detects delimited records, SQL dumps, JSON lines, and varying-shape logs, converts fields or template variables into typed streams, and compresses them in parallel. Base mode compresses a new version against an older version by using regions of the old data as history. The format uses independently decodable blocks and parallel paths; higher levels use separated token, offset, length, and literal streams with SIMD-oriented decoding, while long-distance matching can find repeated content up to 128 MB back. The repository also provides shape dictionaries for small objects, packs for many small objects, container and JPEG handling at selected levels, and a cold context-mixing level for infrequently read data.
The separate glyd-store crate and CLI select similar stored objects as delta bases using fingerprints, cap delta chains, and support local directories and S3-compatible backends. The codec, CLI, C ABI, and bindings use the BSD 3-Clause License or GPL-2.0; glyd-store uses the Business Source License 1.1, with commercial production use requiring a license and conversion to Apache-2.0 four years after each release.
1 use taken from transcripts — each links to the moment in the video.
A Rust lossless compressor for logs, data dumps, and version backups. It supports structured-record compression, storage of new versions against older ones, and fast parallel reads in exchange for slower compression.
1 in the library.