zukomp is a codec registry that ships with
DEFLATE, not a DEFLATE package with room for extras. Everything
goes through two functions that take raw vectors and return raw vectors;
which codec does the work is a runtime lookup, so adding one — including
one that lives in someone else’s package — requires no change to the
public header and no ABI bump.
Compressing and decompressing
komp_compress() names a codec. level is
optional and codec-specific; the default is whatever that codec
considers balanced.
x <- charToRaw(strrep("the quick brown fox ", 500))
z <- komp_compress(x, "gzip")
c(input = length(x), output = length(z))
#> input output
#> 10000 81
identical(komp_decompress(z), x)
#> [1] TRUERaw vectors only. Character input is refused, because the encoding to compress is the caller’s decision and not one this package should make quietly.
komp_compress("the quick brown fox")
#> Error in `komp_compress()`:
#> ! `x` must be a raw vector, not character. zukomp is bytes in, bytes out; convert text with charToRaw() so the encoding is your decision.Concatenated gzip members are a single stream, as the format intends:
both <- c(komp_compress(charToRaw("hello "), "gzip"),
komp_compress(charToRaw("world"), "gzip"))
rawToChar(komp_decompress(both))
#> [1] "hello world"Detection, and refusing to guess
komp_decompress() defaults to
codec = "auto", which calls komp_detect().
Detection is a registry property rather than a table of magic bytes kept
in one place, and it refuses to guess: a format with no header to
recognise returns NA and has to be named.
komp_detect(z)
#> [1] "gzip"
komp_detect(komp_compress(x, "deflate-raw"))
#> [1] NAThat is deliberate. Guessing wrong on a headerless format does not raise an error — it returns wrong bytes.
komp_decompress(komp_compress(x, "deflate-raw"))
#> Error in `komp_decompress()`:
#> ! Could not identify a codec from these bytes. Headerless formats such as "deflate-raw" cannot be detected and must be named explicitly via `codec`.The registry
komp_codecs() is the capability table, and the reason
this package can honestly call itself extensible. Codecs it knows the
name of but does not implement still get a row, so “install the
satellite package” is a possible answer rather than an unrecognised
name.
komp_codecs()
#> id available can_encode can_decode level_min level_max
#> 1 identity TRUE TRUE TRUE NA NA
#> 2 deflate-raw TRUE TRUE TRUE 0 9
#> 3 zlib TRUE TRUE TRUE 0 9
#> 4 gzip TRUE TRUE TRUE 0 9
#> 5 brotli FALSE NA NA NA NA
#> 6 zstd FALSE NA NA NA NA
#> 7 lz4-frame FALSE NA NA NA NA
#> 8 lz4-block FALSE NA NA NA NA
#> 9 snappy-frame FALSE NA NA NA NA
#> 10 snappy-raw FALSE NA NA NA NA
#> level_default level_fast level_best detectable can_flush content_encoding
#> 1 NA NA NA FALSE TRUE identity
#> 2 6 1 9 FALSE TRUE <NA>
#> 3 6 1 9 TRUE TRUE deflate
#> 4 6 1 9 TRUE TRUE gzip
#> 5 NA NA NA NA NA br
#> 6 NA NA NA NA NA zstd
#> 7 NA NA NA NA NA <NA>
#> 8 NA NA NA NA NA <NA>
#> 9 NA NA NA NA NA <NA>
#> 10 NA NA NA NA NA <NA>
#> source
#> 1 zukomp
#> 2 zukomp
#> 3 zukomp
#> 4 zukomp
#> 5 <NA>
#> 6 <NA>
#> 7 <NA>
#> 8 <NA>
#> 9 <NA>
#> 10 <NA>content_encoding is the HTTP token for the codec, which
is what makes the table useful to an HTTP client: zlib is
deflate on the wire, and a codec with no token gets
NA.
komp_codec_available("gzip")
#> [1] TRUE
komp_codec_available("zstd")
#> [1] FALSEkomp_info() reports the build itself — ABI version,
which codecs are compiled in, and the vendored sources behind them.
komp_info()
#> $version
#> [1] "0.1.0"
#>
#> $abi_version
#> [1] 1
#>
#> $codecs
#> [1] "identity" "deflate-raw" "zlib" "gzip"
#>
#> $vendored
#> source version
#> 1 miniz 11.3.2
#>
#> $build_flags
#> [1] "MINIZ_NO_ARCHIVE_APIS" "MINIZ_NO_ARCHIVE_WRITING_APIS"
#> [3] "MINIZ_NO_STDIO" "MINIZ_NO_TIME"
#> [5] "MINIZ_NO_ZLIB_COMPATIBLE_NAMES" "MINIZ_NO_PNG_APIS"
#> [7] "MINIZ_NO_ASSERT"Untrusted input
Compressed input from the network can expand enormously.
Decompression is capped at 1 GiB by default, overridable per call with
max_output or globally with
options(zukomp.max_output = ). The cap is enforced by the
core stream driver, so no codec — including a third-party one — can
bypass it.
bomb <- komp_compress(raw(10e6), "gzip")
length(bomb)
#> [1] 9737
komp_decompress(bomb, max_output = 1024)
#> Error in `komp_decompress()`:
#> ! Decompressed output exceeded `max_output`.max_ratio is the second lever, off by default, bounding
output against input rather than in absolute terms:
komp_decompress(bomb, max_ratio = 10)
#> Error in `komp_decompress()`:
#> ! Compression ratio exceeded `max_ratio`.Errors are structured conditions
Failures are conditions with classes and data, not just messages, so
callers branch on what went wrong rather than matching on text.
zukomp_truncated, zukomp_checksum_error,
zukomp_invalid_data, zukomp_output_limit,
zukomp_ratio_limit and the rest all inherit from
zukomp_error.
e <- tryCatch(komp_decompress(z[1:10]), zukomp_error = function(e) e)
class(e)
#> [1] "zukomp_invalid_data" "zukomp_error" "error"
#> [4] "condition"
conditionMessage(e)
#> [1] "Invalid compressed data in gzip stream."Each carries codec, input_bytes,
output_bytes and native_status:
e[c("codec", "input_bytes", "output_bytes", "native_status")]
#> $codec
#> [1] "gzip"
#>
#> $input_bytes
#> [1] 10
#>
#> $output_bytes
#> [1] 0
#>
#> $native_status
#> [1] 6For package authors
zukomp exposes a versioned C ABI through R’s registered
C-callable mechanism, so another package can drive the codecs from C
without vendoring miniz itself. A consumer needs both
Imports: zukomp (to load the DLL) and
LinkingTo: zukomp (for the headers), plus a real
importFrom() directive in NAMESPACE. The
directive is what Writing R Extensions documents as loading an
imported namespace before yours; an Imports: entry alone
happens to do it on current R, but that is undocumented, so do not rely
on it.
# in DESCRIPTION
Imports: zukomp
LinkingTo: zukomp#include <zukomp-r.h>
const zukomp_api_v1 *api = zukomp_api(); // resolved lazily, cached
zu_codec c = api->codec_lookup("gzip");A package can also register a codec of its own at
ZU_CODEC_VENDOR_BASE, and it becomes a first-class citizen
of komp_codecs() — with zukomp’s output and ratio limits
applying to it automatically, because they live in the stream driver
rather than in each codec. tools/zukomptest does exactly
this, as a test.
The static archive, for ZIP containers
The registered table is the route to prefer, and it reaches every
codec. It does not reach the ZIP container layer — reading the
members of a .zip, an .xlsx or any other
ZIP-shaped format — because that layer is trimmed out of
zukomp.so deliberately and is not a codec.
For that, an installed zukomp also ships a static archive: a second
compilation of the same vendored miniz.c with the ZIP
reader left in, at
system.file("lib", .Platform$r_arch, package = "zukomp"),
with miniz’s own header beside zukomp.h on the
LinkingTo include path. It is linked into the consumer’s
own binary rather than resolved at runtime, so the consumer gets a
private copy of miniz that a zukomp update does not reach until they
rebuild.
The README
has the configure recipe and the compile flags, which are
not optional — most importantly that
MINIZ_NO_ZLIB_COMPATIBLE_NAMES must be defined and
MINIZ_NO_TIME must not.
Scope
In: identity, deflate-raw,
zlib, gzip; streaming; detection; concatenated
gzip members; decompression limits; the C ABI.
Not in: ZIP and other archive formats, tar, PNG, encryption, filesystem archive APIs, and a zlib-compatible ABI. Brotli, zstd, LZ4 and Snappy are planned as separate satellite packages, so their licences and source size stay out of this one.