Convert Arrow data to R

arrow_to_r() converts an array to an R vector, and a stream or a struct array to a data frame with one column per field. The result carries no zuarrow class: it is a plain base R object.

Usage

arrow_to_r(x, to = NULL, lossy = zar_lossy_policies, copy = TRUE)

arrow_next(stream, to = NULL, lossy = zar_lossy_policies, copy = TRUE)

# S3 method for class 'zuarrow_array_stream'
as.data.frame(
  x,
  row.names = NULL,
  optional = FALSE,
  ...,
  to = NULL,
  lossy = zar_lossy_policies,
  copy = TRUE
)

# S3 method for class 'zuarrow_array'
as.data.frame(
  x,
  row.names = NULL,
  optional = FALSE,
  ...,
  to = NULL,
  lossy = zar_lossy_policies,
  copy = TRUE
)

# S3 method for class 'zuarrow_array'
as.vector(x, mode = "any")

Arguments

x

An array or stream handle.

to

NULL, or a named list of prototypes, one per column. For an array that is not a struct, a single prototype.

lossy

What to do with a column R cannot hold exactly.

copy

TRUE to create every R value during the conversion. FALSE to defer string columns: each becomes a character vector that keeps the Arrow buffers and creates an element's R string the first time it is used. Most of the cost of a string column is creating its R strings, so a result that is only partly read, or passed on, is much cheaper. The values are checked during the conversion either way, so copy = FALSE raises the same errors at the same time. The vector holds the batch's memory until every element has been used, and it is saved by saveRDS() as an ordinary character vector. Columns of other types, and strings read through a dictionary, a run-end encoding or a struct with nulls, are copied either way.

stream

A stream handle.

row.names

Must be NULL: the rows of a converted data frame have no names.

optional

Ignored. Column names always come from the schema.

...

Unused.

mode

Must be "any": the array converts to its own R form, as arrow_to_r() gives it, with the default lossy = "error".

Value

An R vector, or a data frame.

arrow_next() returns the next batch as a data frame (or a vector, for a stream that is not of struct), or NULL once the stream has ended.

The as.data.frame() methods return a data frame. An array or stream that is not of struct is a zuarrow_type_error.

Details

A conversion is exact or it fails. lossy decides what happens to a column that R cannot hold exactly, such as an int64 with a value past 2^53, a decimal128, or a nanosecond timestamp:

• "error", the default, raises zuarrow_lossy_error, naming the column, the Arrow type, the first offending row where there is one, and the package that would read the column exactly.

• "character" returns the values' exact text: integers as digits, decimals with their scale, timestamps in ISO 8601 at the type's precision. The column carries the Arrow format string in its arrow_type attribute.

• "warn" converts to the nearest R type and raises one zuarrow_lossy_warning per column.

• "allow" converts to the nearest R type silently. An int64 column becomes bit64's integer64, exactly, when bit64 is loaded.

Intervals and unions have no R form and always fail.

to names the R type a column must become: character(), double(), integer(), or arrow_matrix() for a fixed_size_list. A target that cannot take the column is a zuarrow_type_error, never a silent fallback.

Nested types follow the table too. A struct becomes a data frame column; list, large_list and the list views become list columns, NULL for a null list; a map becomes a list of data frames with columns key and value, and an arrow_keys_sorted attribute that writes it back as a map. A dictionary of strings becomes a factor, ordered when the dictionary is, with the union of every batch's levels; a dictionary of other values, and a run-end encoded array, decode to their values' R form. An extension type converts as its storage and keeps its name and metadata in the arrow_extension and arrow_extension_metadata attributes, so writing the column back restores it. Field metadata becomes the arrow_field_metadata attribute. A type registered with arrow_register() goes through its handler.

See also

arrow_array() for the way there, arrow_types() for the table.

Examples

arrow_to_r(arrow_array(1:3))
#> [1] 1 2 3

big <- arrow_array(2^53 + 2, schema = "l")
try(arrow_to_r(big))
#> Error in arrow_to_r(big) : 
#>   Column 1 (Arrow format `l`) can't be converted to R exactly: row 1 holds 9007199254740994. Load zuint to read it exactly, or use `lossy = "character"` for its exact text, or `lossy = "allow"` to accept the loss.
arrow_to_r(big, lossy = "character")
#> [1] "9007199254740994"
#> attr(,"arrow_type")
#> [1] "l"