Skip to contents

zubin

Every binary format, from an image header to a market data feed to a C struct dumped to disk, is specified the same way: a table of fields, each with a name, a type, a width and an offset. Base R reads such data one readBin() call at a time, loops over the records itself, has no unsigned 32-bit integer and no half-precision float, and reads an 8-byte integer silently wrong once it outgrows 32 bits.

zubin lets you write the table down once, as a layout, and then read or write any number of records in one call.

  • Describe once, read millions. A layout is one string, "<id:u32 ts:i64 price:f64 side:u8", and bin_unpack() turns a raw vector into a data frame of every record in it, one tight loop per field.
  • Every type a format uses. Integers from 8 to 128 bits, signed and unsigned; f16, bf16, f32 and f64; complex numbers; booleans; fixed bytes and strings; arrays; padding; either byte order, per field; and C struct alignment.
  • Exact, or an error. Nothing is silently rounded, wrapped or truncated. A 64-bit integer that a double cannot hold, or a value too large for its field, is an error that names the field and the record.
  • Writing too. bin_pack() and bin_encode() are the inverses, and a byte builder appends without quadratic copying, up to a hard cap. bin_serialize() writes an R object into one without first making a raw vector, and bin_hash_object() fingerprints an object without making its serialisation at all.
  • A C library underneath. The same buffers, cursors and kernels are a header-only C99 library that other packages use through LinkingTo alone.

Installation

zubin is not on CRAN yet. Once it is, install it with:

Until then, and for the development version, install it from GitHub:

# install.packages("pak")
pak::pak("pedrobtz/zubin")

Usage

Read a real header. A PNG file opens with an 8-byte signature and then an IHDR chunk, big-endian, as its specification lays it out. That table, written down, reads the image’s size from the file R just wrote:

png_file <- tempfile(fileext = ".png")
png(png_file, width = 640, height = 480)
plot(pressure)
invisible(dev.off())

ihdr <- bin_layout(">x8 length:u32 type:s4 width:u32 height:u32 depth:u8 colour:u8")
bin_unpack(readBin(png_file, "raw", 64), ihdr, n = 1)
#>   length type width height depth colour
#> 1     13 IHDR   640    480     8      6

The types base R lacks. readBin() has no unsigned 32-bit integers, so it warns and reads four ff bytes as -1, and it has no half floats. zubin decodes both, in either byte order:

x <- as.raw(c(0xff, 0xff, 0xff, 0xff))
readBin(x, "integer", size = 4, signed = FALSE)
#> Warning in readBin(x, "integer", size = 4, signed = FALSE): 'signed = FALSE' is
#> only valid for integers of sizes 1 and 2
#> [1] -1
bin_decode(x, "u32")
#> [1] 4294967295

# half-precision floats, as machine-learning weights are stored
bin_decode(as.raw(c(0x00, 0x3c, 0x00, 0xc0, 0x00, 0x7c)), "f16")
#> [1]   1  -2 Inf

Exact, or an error. readBin() reads the 8-byte integer 2^32 as 0, without a word. zubin reads it exactly, and refuses a value above 2^53, which has no exact double, naming the record rather than handing back a neighbouring number (int64 = "integer64" returns the exact bits instead):

counts <- bin_encode(c(2^32, 2^53 + 2), "u64")
readBin(counts, "integer", size = 8, n = 2)
#> [1] 0 2
bin_decode(counts[1:8], "u64")
#> [1] 4294967296
bin_decode(counts, "u64")
#> Error in `bin_decode()`:
#> ! The value of record 1 has no exact R value.

Take a number apart. A double is a sign bit, an 11-bit exponent and a 52-bit fraction. Encoded and read back as two unsigned words, its fields come out exactly, and 0.1 turns out to be 7205759403792794 / 2^56, a little more than a tenth:

w <- bin_unpack(bin_encode(c(1, 0.1, -2.5), "f64", endian = "big"), ">hi:u32 lo:u32")
bits <- function(v, from, width) (v %/% 2^from) %% 2^width
parts <- data.frame(sign = bits(w$hi, 31, 1), exponent = bits(w$hi, 20, 11) - 1023,
                    fraction = bits(w$hi, 0, 20) * 2^32 + w$lo)
format(parts, scientific = FALSE)
#>   sign exponent         fraction
#> 1    0        0                0
#> 2    0       -4 2702159776422298
#> 3    1        1 1125899906842624
(2^52 + parts$fraction[2]) / 2^56 == 0.1
#> [1] TRUE

A million records in one call. Packing a data frame and unpacking it again is one call each way, and the round trip is exact:

set.seed(1)
rec <- bin_layout("<id:i32 price:f64 qty:i16 x2")
trades <- data.frame(id = 1:1e6, price = round(runif(1e6, 90, 110), 2),
                     qty = sample(1:500, 1e6, replace = TRUE))
bytes <- bin_pack(rec, trades)
length(bytes)
#> [1] 16000000
identical(bin_unpack(bytes, rec), trades)
#> [1] TRUE

Mixed byte orders: a shapefile. The .shp part of a shapefile mixes byte orders inside one record: its file code and length are big-endian, everything after them little-endian. A .be suffix overrides the layout’s order for one field, so each record is still a single layout. Here the quakes data become a point shapefile, written with a builder, as the ESRI specification lays it out:

shp_header <- bin_layout("<code:i32.be x20 words:i32.be version:i32 type:i32
                          xmin:f64 ymin:f64 xmax:f64 ymax:f64 x32")
point <- bin_layout("<id:i32.be words:i32.be type:i32 x:f64 y:f64")

n <- nrow(quakes)
b <- bin_builder()
bin_put(b, bin_pack(shp_header, code = 9994, words = (100 + 28 * n) / 2,
                    version = 1000, type = 1,
                    xmin = min(quakes$long), ymin = min(quakes$lat),
                    xmax = max(quakes$long), ymax = max(quakes$lat)))
bin_put(b, bin_pack(point, id = seq_len(n), words = 10, type = 1,
                    x = quakes$long, y = quakes$lat))
shp <- bin_take(b)
bin_hexdump(shp, n = 48)
#> 00000000: 0000 270a 0000 0000 0000 0000 0000 0000  ..'.............
#> 00000010: 0000 0000 0000 0000 0000 36e2 e803 0000  ..........6.....
#> 00000020: 0100 0000 3d0a d7a3 70b5 6440 ec51 b81e  ....=...p.d@.Q..

The file code 9994 opens the file big-endian (0000 270a), and the version 1000 at offset 28 is little-endian (e803 0000). Reading it back is one call for the header and one for every point after it:

bin_unpack(shp, shp_header, n = 1)
#>   code words version type   xmin   ymin   xmax   ymax
#> 1 9994 14050    1000    1 165.67 -38.59 188.13 -10.72
pts <- bin_unpack(shp, point, offset = 100)
head(pts, 3)
#>   id words type      x      y
#> 1  1    10    1 181.62 -20.42
#> 2  2    10    1 181.03 -20.62
#> 3  3    10    1 184.10 -26.00
identical(pts$x, quakes$long)
#> [1] TRUE

Every function is prefixed bin_, and offsets are 0-based everywhere, as in every format specification.

Learn more

  • Structured binary data with zubin, the getting-started guide: layouts, a WAV file written and read back, C structs, arrays, 64-bit integers and errors.
  • The C API, for package authors: the LinkingTo recipe, the cursor, the record kernels, and buffers owned by R.
  • Reading R’s own object header, a layout over bytes zubin did not write: the fixed-size C header behind every R object, in native order, with a bitfield word, pointers and a body at a known offset.
  • The reference index, with the full type table under ?bin_layout and the error classes under ?zubin-conditions.

zubin’s integer loads and stores, half-float conversions and UTF-8 validation come from zufast. Python’s struct module and the structured dtypes of NumPy are the nearest equivalents in other languages.