Read the Arrow IPC formats

arrow_read() reads an Arrow IPC file, with its footer (.arrow, also called Feather version 2), or an IPC stream (.arrows). The framing is found from the first bytes, whatever the source is called.

Usage

arrow_read(
  x,
  columns = NULL,
  lazy = FALSE,
  to = NULL,
  lossy = zar_lossy_policies,
  limits = arrow_limits(),
  copy = TRUE
)

Arguments

x

A path, a raw vector, or a connection, which is read to its end.

columns

NULL for every column, or the names or positions of the top-level columns to read. Only those are decoded.

lazy

FALSE to read every batch into one data frame; TRUE to return a stream handle whose batches are decoded by arrow_next(). With lazy = TRUE, to, lossy and copy belong to that later conversion, and setting one here is a zuarrow_type_error.

to

NULL, or a named list of prototypes, one per column. For an array that is not a struct, a single prototype.

lossy

What to do with a column R cannot hold exactly.

limits

An arrow_limits() object.

copy

TRUE to create every R value during the conversion. FALSE to defer string columns: each becomes a character vector that keeps the Arrow buffers and creates an element's R string the first time it is used. Most of the cost of a string column is creating its R strings, so a result that is only partly read, or passed on, is much cheaper. The values are checked during the conversion either way, so copy = FALSE raises the same errors at the same time. The vector holds the batch's memory until every element has been used, and it is saved by saveRDS() as an ordinary character vector. Columns of other types, and strings read through a dictionary, a run-end encoding or a struct with nulls, are copied either way.

Value

A data frame, or with lazy = TRUE a stream handle.

Details

The input is checked before it is used: every message is verified, every buffer must lie inside its message, and every limit of arrow_limits() binds, so malformed or hostile bytes are a zuarrow_format_error or a zuarrow_limit_error, never a crash. A file whose footer repeats the schema differently is a zuarrow_format_error too. Each batch then passes the full validation of arrow_validate() before it is converted. Bodies compressed with LZ4 or zstd are decompressed through 'zukomp'. Big-endian input is a zuarrow_format_error.

Dictionary batches, and the deltas a stream may send between record batches, are applied as the format says; a file that replaces a dictionary is an error, as the format forbids it. A delta that extends a dictionary already holding values is not supported yet and is a zuarrow_format_error.

A raw vector is read where it is. With lazy = TRUE the handle holds a copy instead, because a handle can be passed to another consumer that outlives the vector.

See also

arrow_to_r() for the conversion and its lossy policy.

Examples

path <- tempfile(fileext = ".arrows")
stream <- arrow_stream(data.frame(x = 1:3, y = c("a", "b", "c")))
if (requireNamespace("nanoarrow", quietly = TRUE)) {
  nanoarrow::write_nanoarrow(stream, path)
  arrow_read(path)
}
#>   x y
#> 1 1 a
#> 2 2 b
#> 3 3 c