Blog

Advent of Code with `data.table`: Week One

tutorials

community

First translation of R vignettes

grant

documentation

translation

Comparing `data.table` reshape to `duckdb` and `polars`

tips

tutorials

developer

benchmarks

Visualizing performance regression of `data.table` with `atime`

performance

testing

developer

Seal of Approval: mlr3

seal of approval

application package

Seal of Approval: collapse

seal of approval

partner package

Newly awarded translation projects

announcements

grant

translation

Seal of Approval: dtplyr

seal of approval

bridge package

Seal of Approval: nc

seal of approval

extension package

Seal of Approval: tidyfast

seal of approval

bridge package

Announcement: The ‘Seal of Approval’

announcements

grant

community

seal of approval

Announcement: Paola Corrales, data.table Ambassador

announcements

grant

ambassadors

travel

Two Roads Diverged

opinion

Testing infrastructure for data.table

grant

testing

developer

Community interviews about data.table

community

grant

Results of the 2023 survey

community

guest post

governance

Column assignment and reference semantics in data.table

tips

tutorials

developer

The Benefits of `data.table` Syntax

tips

tutorials

documentation

New governance, release with new features

governance

releases

Piping data.tables

tips

tutorials

documentation

guest post

Announcement: Jan Gorecki, data.table Ambassador

announcements

grant

ambassadors

Summary of LatinR conference

conferences

Announcement: The data.table Ambassadors Travel Grant

announcements

grant

funding opportunity

Announcement: data.table translation projects

announcements

grant

funding opportunity

Welcome to the data.table ecosystem project!

announcements

grant

Use of non-API entry points in data.table

Ivan Krylov — Mon, 13 Jan 2025 00:00:00 GMT

In the late 1970’s, people at Bell Laboratories designed the S programming language in order to facilitate interactive exploratory data analysis (Chambers 2016). Instead of writing, compiling, scheduling, and interpreting the output of individual Fortran programs, the goal of S was to conduct all the necessary steps of the analysis on the fly. S achieved this not by replacing the extensive collection of Fortran subroutines, but by providing a special interface language (Becker and Chambers 1985) through which S could communicate with compiled code.

Fast forward more than four decades and an increase by more than three orders of magnitude in storage and processing capability of computers around us. The dominant implementation of S is now R. It is now feasible to implement algorithms solely in R, recouping the potential performance losses by reducing the programmer effort spent debugging and maintaining the code (Nash and Bhattacharjee 2024). Still, the capability of R to be extended by special-purpose compiled code is as important as ever. As of 2024-11-29, 22% of CRAN packages use compiled code. Since the implementation language of R is C, not Fortran, the application programming interface (API) for R is mainly defined in terms of C.

A visual metaphor for something that one does fully knowing what they are doing, at their own peril, until one day it fails.

_{Image attribution: Wikipedia user ArnoldReinhold.}

What’s in an API?

Writing R Extensions (“WRE”) is the definitive guide for R package development. Together with the CRAN policy it forms the “rules as written” that the maintainers of CRAN packages must follow. A recent version of R exports 1202 symbols, including 1049 functions (“entry points”, not counting C preprocessor macros) and 153 variables. Not all of them are intended to be used by packages. Even back in R-3.3.0, the oldest version currently supported by data.table, WRE chapter 6, “The R API” classified R’s entry points into four categories:

API Entry points which are documented in this manual and declared in an installed header file. These can be used in distributed packages and will only be changed after deprecation.

public Entry points declared in an installed header file that are exported on all R platforms but are not documented and subject to change without notice.

private Entry points that are used when building R and exported on all R platforms but are not declared in the installed header files. Do not use these in distributed code.

hidden Entry points that are where possible (Windows and some modern Unix-alike compilers/loaders when using R as a shared library) not exported.

Although nobody objected to the use of the API entry points, and there was little point in trying to use the hidden entry points in a package that would fail to link almost everywhere, the public and the private entry points ended up being a point of contention. Those deemed too internal to use but not feasible to make hidden were (and still are) listed in the character vector tools:::nonAPI: R CMD check looks at the functions imported by the package and signals a NOTE if it finds any listed there.

The remaining public functions, neither documented as API nor explicitly forbidden by R CMD check, sat there, alluring the package developers with their offers. For example, the serialization interface is only documented in WRE since R-4.5, but it has been powering the RApiSerialize CRAN package since 2014 (building on ideas from the now-archived Rhpc, 2013) without any drastic changes. Some of the inclusions in tools:::nonAPI could have been historical mistakes: while WRE has been saying back in version 3.3.0 that wilcox_free should be called after a call to the (API) functions dwilcox, pwilcox or qwilcox, the function was only declared in the public headers and removed from tools:::nonAPI in R-4.2.0. Still, between R-3.3.3 and R-4.4.2, the #define USE_RINTERNALS escape hatch finally closed, tools:::nonAPI grew from 259 to 272 entries, and the package maintainers had to adapt or face archival of their packages.

A recent question on R-devel (whether the ALTREP interface should be considered “API” for the purpose of CRAN package development) sparked a series of events and an extensive discussion containing the highest count of occurrences of the word “API” per month ever seen on R-devel (234), topping October 2002 (package versioning and API breakage, 150), October 2005 (API for graphical interfaces and console output, 124), and May 2019 (discussions of the ALTREP interface and multi-threading, 121). As a result, Luke Tierney started work on programmatically describing the functions and other symbols exported by R (including variables and preprocessor and enumeration constants), giving a stronger definition to the interface. His changes add the currently unexported function tools:::funAPI() that lists entry points and two more of their categories:

experimental Entry points declared in an installed header file that are part of an experimental API, such as R_ext/Altrep.h. These are subject to change, so package authors wishing to use these should be prepared to adapt.

embedding Entry points intended primarily for embedding and creating new front-ends. It is not clear that this needs to be a separate category but it may be useful to keep it separate for now.

Additionally, WRE now spells out that entry points not explicitly documented or at least listed in the output of tools:::funAPI (or something that will replace it) are now off-limits, even if not currently present in tools:::nonAPI (emphasis added):

public Entry points declared in an installed header file that are exported on all R platforms but are not documented and subject to change without notice. Do not use these in distributed code. Their declarations will eventually be moved out of installed header files.

Correspondingly, the number of tools:::nonAPI entry points in the current development version of R rose to 322, prompting the blog post you are currently reading.

Non-API entry points marked by `R CMD check`

The first version of the data.table package in the CRAN archive dates back to April 2006 (which corresponds to R version 2.3.0). It has been evolving together with R and its API and thus has accumulated a number of uses of R internals that are now flagged by R CMD check as non-API:

Package: data.table 1.16.2
Flavor: r-devel-linux-x86_64-debian-clang
Check: compiled code, Result: NOTE
  File ‘data.table/libs/data_table.so’:
    Found non-API calls to R: ‘LEVELS’, ‘SETLENGTH’, ‘SET_GROWABLE_BIT’,
      ‘SET_TRUELENGTH’, ‘STRING_PTR’, ‘TRUELENGTH’
  
  Compiled code should not call non-API entry points in R.
  
  See ‘Writing portable packages’ in the ‘Writing R Extensions’ manual,
  and section ‘Moving into C API compliance’ for issues with the use of
  non-API entry points.

– R CMD check --as-cran on a released version of data.table

Operating on the S4 bit: `IS_S4_OBJECT`, `SET_S4_OBJECT`, `UNSET_S4_OBJECT`

In R’s “S4” OOP system, objects can have a primitive base type (e.g. setClass("PrimitiveBaseType", contains = "numeric") or no base type at all (e.g. setClass("NoBaseType")). In the former case, their SEXPTYPE code is that of their base class (e.g. REALSXP). In the latter case, their type code is OBJSXP (previously S4SXP, which is now an alias for OBJSXP). To make both cases work consistently, R uses a special “S4” bit in the header of the object.

The data.table class is registered with the S4 OOP system, making it possible to create S4 classes containing data.tables as members (setClass(slots = c(mytable = 'data.table'))) or even inheriting from data.table (and, in turn, from data.frame: setClass(contains = 'data.table')). Additionally, data.tables may contain columns that are themselves S4 objects, and both of these cases require care from the C code.

The undocumented functions IS_S4_OBJECT, SET_S4_OBJECT, UNSET_S4_OBJECT exist as bare interfaces to the internal macros of the same names and directly access the flag inside their argument. Writing R Extensions documents Rf_isS4 and Rf_asS4 as their replacements.

The Rf_isS4 function is a wrapper for IS_S4_OBJECT that follows the usual naming convention for remapped functions, has been part of the API for a long time, and could implement additional checks if they are needed by R. The Rf_asS4 function (experimental API) is more involved, trying to “deconstruct” S4 objects into S3 objects if possible and requested to. If the reference count of its argument is above 1, it will operate upon and return its shallow duplicate.

data.table used to directly operate on the S4 bit in two places, the shallow function in src/assign.c and the keepattr function in src/dogroups.c. In both cases, this was required after directly modifying attribute list using the undocumented function SET_ATTRIB. For shallow, the solution was to replace the manual operation of attributes with SHALLOW_DUPLICATE_ATTRIB (API, available since 3.3.0), which itself takes care of invariants like the object bit and the S4 bit.

The keepattr function is only used in growVector to transplant all attributes from a vector to its enlarged copy without duplicating them, for which no API exists. The solution is to use Rf_asS4 to control the S4 object bit, knowing that the new vector is freshly allocated and thus cannot be shared yet.

Status in data.table: fixed in #6183 and #6264.

Converting between calls and pairlists: `SET_TYPEOF`

In R, function calls are internally represented as Lisp-style pairlists where the first pair is of special type LANGSXP instead of LISTSXP. For example, the following diagram illustrates the data structure of the call print(x = 42L):

Here, every list item is a separate R object, a “cons cell”; each cell contains the value in its CAR field and a reference to the rest of the list in its CDR field. Argument names, if provided, are stored in the third field, TAG. The list is terminated by R_NilValue, which is of type NILSXP. These structures must be constructed every time C code wants to evaluate a function call (e.g.).

Previously, R API contained a function to allocate LISTSXP pairlists of arbitrary length, allocList(), but not function calls, so it became a somewhat common idiom to first allocate the list and then use SET_TYPEOF to change the type of the head pair to LANGSXP. This did not previously lead to problems, since the two types have the same internal memory layout.

The danger of SET_TYPEOF lies in the possibility to set the type of an R value to one with an incompatible memory layout. (For example, vector types REALSXP and INTSXP are built very differently from cons cells LISTSXP and LANGSXP.) Starting with R-4.4.1, R contains the allocLang function in addition to the allocList function that directly allocates a function call object with a head pair of type LANGSXP. In order to stay compatible with previous R versions, packages may allocate the LISTSXP tail first and then use lcons() to construct the LANGSXP head pair of the call.

Problem (the only instance in data.table):

   SEXP s = PROTECT(allocList(2));
   SET_TYPEOF(s, LANGSXP);
// ^^^^^^^^^^ unsafe operation, could be used to corrupt objects
   SETCAR(s, install("format.POSIXct"));
   SETCAR(CDR(s), column);

Solutions:

// for fixed-size calls with contents known ahead of time
SEXP s = lang2(install("format.POSIXct"), column);

or:

// partially pre-populate
SEXP s = lang2(install("format.POSIXct"), R_NilValue);
// later, when 'column' is known:
SETCAR(CDR(s), column);

or:

// allocate a call with 'n' elements
SEXP call = lcons(R_NilValue, allocList(n - 1));

or:

// in R >= 4.4.1 only:
SEXP call = allocLang(n);

Unfortunately, the LCONS macro didn’t work with #define R_NO_REMAP prior to R-4.4, because it expanded to lcons() instead of Rf_lcons().

Status in data.table: fixed in #6313.

Strings as C arrays of `CHARSXP` values: `STRING_PTR`

From the point of view of R code, strings are very simple things, much like numbers: they live in atomic vectors and can be directly compared with other objects. It is only natural to desire to work with them as easily from C code as it’s possible with other atomic types, where functions REAL(), INTEGER(), or COMPLEX() can be used to access the buffer containing the numbers.

The underlying reality of strings is more complicated: since they internally manage memory buffers containing text in a given encoding, they must be subject to garbage collection. Like other managed objects in R, they are represented as SEXP values of special type CHARSXP. R’s garbage collector is generational and requires the use of write barrier (1, 2) any time a SEXP value (such as an STRSXP vector) references another SEXP value (such as a CHARSXP string). In a generational garbage collector, “younger” generations are marked and sweeped more frequently than “older” ones, because in a typical R session, most objects are temporary (Jones et al. 2012, chap. 9). If package C code manually writes a reference to a “young” CHARSXP object into an “old” STRSXP vector without taking generations into account, a following collection of the “young” pool of objects will miss the CHARSXP being referenced by the “old” STRSXP and remove the CHARSXP as “garbage”. This makes the SEXP * pointers returned by STRING_PTR unsafe and requires the use of STRING_PTR_RO function, which returns a read-only const SEXP *.

Thankfully, data.table has already been using read-only const SEXP * pointers when working with STRSXP vectors, so the required changes to the code were not too substantial, limited to the name of the function:

Example of the problem:

const SEXP *sourceD = STRING_PTR(source);
//                    ^^^^^^^^^^
// returns a writeable SEXP * pointer, therefore unsafe

Solution:

#if R_VERSION < R_Version(3, 5, 0)
// STRING_PTR_RO only appeared in R-3.5
#define STRING_PTR_RO(x) (STRING_PTR(x))
#endif

// later:
const SEXP *sourceD = STRING_PTR_RO(source);
//                    ^^^^^^^^^^^^^
// returns a const SEXP * pointer, which prevents accidental writes

Status in data.table: fixed in #6312. See also: PR18775.

Reading the reference counts: `NAMED`

In plain R, all value types – numbers, strings, lists – have pass-by-value semantics. Without dark and disturbing things in play, such as non-standard evaluation or active bindings, R code can give a plain value (x <- 1:10) to a function (f(x)) or store it in a variable (y <- x), have the function modify its argument (f <- \(x) { x[1] <- 0 }) or change the duplicate variable (y[2] <- 3), and still have the original value intact (stopifnot(identical(x, 1:10))). Only the inherently mutable types, such as environments, external pointers and weak references, will stay shared between all assignments and function arguments; the value types behave as if R copies them every time.

And yet actually making these copies is wasteful when the code only reads the variable and does not alter it. (In fact, one of the original motivations of data.table was to reduce certain wasteful copying of data that happens during normal R computations.) Until version 4.0.0, NAMED was R’s mechanism to save memory and CPU time instead of creating and storing these copies. A temporary object such as the value of 1:10 was not bound to a symbol and thus could be modified right away. Assigning it to a variable, as in x <- 1:10, gave it a NAMED(x) count of 1, for which R had an internal optimisation in replacement function calls like foo(x) <- 3. Assigning the same value to yet another symbol (by copying y <- x or calling a function foo(x)) increased the NAMED() count to 2 or more, for which there was no optimisation: in order to modify one of the symbols, R was required to duplicate x first. NAMED() was not necessarily decreased after the bindings disappeared, and decreasing it after having reached NAMEDMAX was impossible. During the lifetime of R-3.x, NAMEDMAX was increased from 2 to 3 and later to 7.

Between R-3.1.0 and R-4.0.0, R migrated from NAMED to reference counting. Reference counts are easier to properly decrement than NAMED, thus preventing unneeded copies of objects that became unreferenced. R-3.5.0 documented the symbols MAYBE_REFERENCED(.) / NO_REFERENCES(.) for use instead of checking NAMED(.) == 0, MAYBE_SHARED(.) / NOT_SHARED(.) instead of checking NAMED(.) > 1, and MARK_NOT_MUTABLE(.) instead of setting NAMED(.) to NAMEDMAX, which later became part of the API instead of the NAMED(.) and REFCNT(.) functions. The hard rules are that a value is safe to modify in place if it has NO_REFERENCES() (reference count of 0), definitely unsafe to modify in place (requiring a call to duplicate or shallow_duplicate) if it is MAYBE_SHARED() (reference count above 1), and almost certainly unsafe to modify in place if it is MAYBE_REFERENCED() (reference count of 1).

data.table’s only uses of NAMED() were in the verbose output during assignment:

if (verbose) {
  Rprintf(_("RHS for item %d has been duplicated because NAMED==%d MAYBE_SHARED==%d, but then is being plonked. length(values)==%d; length(cols)==%d)\n"),
          i+1, NAMED(thisvalue), MAYBE_SHARED(thisvalue), length(values), length(cols));
            // ^^^^^ non-API function
}

Since the correctness of the modification operation hinges on the reference count being 0 (and it may be important whether it’s exactly 1 or above 1), the same amount of useful information can be conveyed by printing MAYBE_REFERENCED() and MAYBE_SHARED() instead of NAMED():

if (verbose) {
  Rprintf(_("RHS for item %d has been duplicated because MAYBE_REFERENCED==%d MAYBE_SHARED==%d, but then is being plonked. length(values)==%d; length(cols)==%d)\n"),
          i+1, MAYBE_REFERENCED(thisvalue), MAYBE_SHARED(thisvalue), length(values), length(cols));
            // ^^^^^^^^^^^^^^^^ API function
}

Status in data.table: fixed in #6420.

Encoding bits: `LEVELS`

LEVELS is the name of the internal R macro and an exported non-API function accessing a 16-bit field called gp (general-purpose) that is present in the header of every SEXP value. Not every access to this field is done using the LEVELS() macro; there are bits of R code that access (sexp)->sxpinfo.gp directly. R uses this field for many purposes:

matching given arguments against the formals of a function (1, 2, 3)
remembering the previous type of a garbage-collected value
finalizing the reference-semantics objects before garbage-collecting them
marking condition handlers as “calling” (executing on top of where the condition was signalled in the call stack), as opposed to “non-calling” (executing at the site of the tryCatch call)
marking objects in complex assignment calls
storing the S4 object bit
marking functions as (un)suitable for bytecode compilation
marking vectors as growable
marking provided (“actual”) function arguments as missing
marking the ..1, ..2, etc symbols as corresponding to the given element of the ... argument
marking environments as locked, or for caching the global variable lookup, or for looking up values in the base environment or the special functions (1, 2, 3, 4)
marking symbols naming environment contents for hash lookup
marking bindings inside environments as active
marking promise objects as already evaluated
marking CHARSXP values as present in the global cache or being in a certain encoding

Although the value of gp is directly stored in R’s serialized data stream, neither of these are part of the API. Out of all possible uses for this flag, data.table is only interested in string encodings. From the viewpoints of plain R and the C API, an individual string (CHARSXP value) can be marked with the following encodings:

R-level encoding name	C-level encoding constant	Meaning
`"latin1"`	`CE_LATIN1`	ISO/IEC 8859-1 or CP1252
`"UTF-8"`	`CE_UTF8`	ISO/IEC 10646
`"unknown"`	`CE_NATIVE`	Encoding of the current locale
`"bytes"`	`CE_BYTES`	Not necessarily text; `translateChar` will fail

Internally, R also marks strings as encoded in ASCII: since all three encodings are ASCII-compatible, an ASCII string will never need to be translated into a different encoding. Note that there is a subtle difference between a string marked in a certain encoding and actually being in a certain encoding: in an R session running with a UTF-8 locale (which includes most modern Unix-alikes and Windows ≥ 10, November 2019 update) a string marked as CE_NATIVE will also be in UTF-8. (Similarly, with an increasingly rare Latin-1 locale, a CE_NATIVE string will be in Latin-1.)

The data.table code is interested in knowing whether a string is marked as UTF-8, Latin-1, or ASCII. This is used to convert strings to UTF-8 when needed (also: output to native encoding or UTF-8 in fwrite, automatic conversion in forder). The getCharCE API function appeared in R-2.7.0 together with the encoding support, so switching the IS_UTF8 and IS_LATIN macros from LEVELS to API calls was relatively straightforward.

R-4.5.0 is expected to introduce the charIsASCII “experimental” API function that returns the value of the ASCII marker for a CHARSXP value, which will replace the use of LEVELS in the IS_ASCII macro. Curiously, while it looks like the code could benefit from switching from the getCharCE() tests (which only look at the value of the flags and so may needlessly translate strings from CE_NATIVE) to the new experimental charIs(UTF8|Latin1) functions that will also return TRUE for a matching native encoding, actually making the change breaks a number of unit tests.

Status in data.table: partially fixed in #6420, waiting for R-4.5.0 to be released with the new API in #6422.

`SETLENGTH`, `SET_GROWABLE_BIT`, `(SET_)TRUELENGTH`

Growable vectors

Since data.frames and data.tables are lists, and lists in R are value types with pass-by-value semantics, adding or removing a column to one normally involves allocating a new list referencing the rest of the columns (performing a “shallow duplicate”). By contrast, the over-allocated lists can be resized in place by gradually increasing their LENGTH (remembering their original length in the TRUELENGTH field), obviating the need for shallow duplicates at the cost of making data.tables shared, by-reference values. The change has been introduced in v1.7.3, November 2011, together with the := operator for changing the columns by reference (which has since become the defining feature of data.table).

R’s own use of TRUELENGTH is varied. The field itself appeared in R-0.63 together with the VECSXP lists (to replace the Lisp-style linked pairlists). R uses this field in CHARSXP strings to store the hash values belonging to symbols. R’s many VECSXP-based hash tables use it to count the primary slots in use: hashes are used for reference tracking during (un)serialization (1, 2) and looking up environment contents (1, 2). R-3.3 (May 2016) saw the inclusion of radix sort from data.table itself, which uses TRUELENGTH to sort strings. R-3.4 (April 2017) introduced over-allocation when growing vectors due to assignment outside their bounds. The growable bit was added to prevent the mismanagement of the allocated memory counter: without the bit set on the over-allocated vectors, the garbage collector only counted LENGTH(x) instead of TRUELENGTH(x) units as released when garbage-collecting the vector, inflating the counter over time. ALTREP objects introduced in R-3.5 (April 2018) don’t have a TRUELENGTH: it cannot be set and is returned as 0. In very old versions of R, TRUELENGTH wasn’t initialised, but it is nowadays set to 0, which data.table depends upon.

Nowadays, data.table uses vectors whose length is different from their allocated size in many places:

src/dogroups.c
- reuses the same memory for the data.table subset object .SD and for the virtual row number column .I by shortening the vectors to the size of the current group
- later restores their natural length
- extends the data.table for new columns as needed
src/freadR.c
- works with an over-estimated line count and so can truncate the columns after the value is known precisely
  - the columns are prepared to be truncated
- may also drop columns by reference
src/subset.c
- the subsetDT function prepares an over-allocated data.table together with its names.
src/assign.c
- the shallow function prepares an over-allocated data.table referencing the columns of an existing data.table
- assign creates or removes columns by reference
- finalizer causes an INTSXP vector with the fake length to be (eventually) garbage-collected to fix up a discrepancy in R’s vector size accounting caused by the existence of the over-allocated data.table

SETLENGTH presents many opportunities to create inconsistencies within R:

When copying shortened objects without the GROWABLE_BIT set, R allocates and copies only XLENGTH elements and duplicates the value of TRUELENGTH.
- For this and other reasons, data.tables have a special .internal.selfref attribute attached containing an EXTPTR back to itself. A copy of a table can be detected because it will have a different address.
  - The _selfrefok function tries to restore the correct TRUELENGTH if it detects a copy.
- Setting the GROWABLE_BIT on the data.table would make R keep the default TRUELENGTH (0) instead of copying it.
When deallocating shortened objects without the GROWABLE_BIT set, R accounts only for the XLENGTH elements being released, over-counting the total amount of allocated memory.
- data.table compensates for this using the finalizer on the .internal.selfrep attribute.
- Setting the GROWABLE_BIT on the data.table would make R account for TRUELENGTH elements instead of XLENGTH elements.

Unfortunately, GROWABLE_BIT is not part of the API and was only introduced in R-3.4, so it does not present a full solution to the problems. Moreover,

Setting LENGTH larger than the allocated length may cause R to access undefined or even unmapped memory.

For vectors containing other SEXP values (of type VECSXP, EXPRSXP, STRSXP): when reducing the LENGTH, having a non-persistent value (something unlike the persistent values R_NilValue or R_BlankString or R_NaString provided by R itself) in the newly inaccessible cells will also make them unreachable from the viewpoint of the garbage collector, potentially prompting it to reuse or unmap the pointed-to memory. Increasing the LENGTH again with invalid pointers in the newly accessible slots will make an invalid vector that cannot be safely altered or discarded:

#include 
#include 
void foo(void) {
  {
    SEXP list = PROTECT(allocVector(VECSXP, 100)), elt;
    SET_VECTOR_ELT(list, 99, elt = allocVector(REALSXP, 1000));

    double * p = REAL(elt); // initialise the vector
    for (R_xlen_t i = 0; i < xlength(elt); ++i) p[i] = i;

    SETLENGTH(list, 1); // elt is unreachable
    R_gc(); // elt is collected
    SETLENGTH(list, 100); // invalid elt is reachable again
    R_gc(); // invalid elt is accessed
    UNPROTECT(1);
  }
  R_gc(); // crash here if not above
}

Starting with R-4.3, R packages can implement their own VECSXP-like objects using the ALTREP framework; STRSXP objects have been supported since R-3.5. An ALTREP object is defined by its class (a collection of methods) and two arbitrary R values, data1 and data2. (Attributes are not a part of the ALTREP representation and exist the same way as on normal R objects.) A simple implementation of a shortened, expandable vector could hold a full-length vector in the data1 slot and its pretend-length as a one-element REALSXP value in the data2 slot. (Currently, R_xlen_t values are limited by the largest integer precisely representable in an IEEE double value, which is .) The over-allocated class will need to implement the following methods:

Common ALTREP methods:
- Length(), returning the pretend-length of the vector. Required.
- Duplicate(deep). If not implemented, R will create a copy as an ordinary object using the length and the data pointer provided by the class.
  - There is also DuplicateEX(deep), which is responsible for copying the value and the attributes, but it may be hard to implement within the API bounds (ATTRIB is not API), and R provides a default implementation that calls Duplicate above.
  - Shared mutable vectors can cause problems, so a decision to let the Duplicate() return value share the original vector will require a lot of thought and testing.
- Serialized_state(), Unserialize(state). If not implemented, R will serialize the value as an ordinary object, which is what currently happens for data.tables. Once an R package implements an ALTREP class with a Serialized_state method, the format is set in stone; any changes will have to introduce a new class.
  - Similarly, there is UnserializeEX(state, attr, objf, levs) responsible for setting LEVELS, the object bit, and the attributes; the default implementation should suffice.
- Inspect(pre, deep, pvec, inspect_subtree). May Rprintf some information from the ALTREP fields before returning FALSE to let R continue inspecting the object.
Common altvec methods required for most code to work with the class:
- Dataptr(writable), returning the pointer to the start of the array backing the underlying vector. For VECSXP or STRSXP vectors, writable should always be FALSE, so DATAPTR_RO can be used.
- Dataptr_or_null(). May delegate to Dataptr(FALSE) above.
- Extract_subset(indx, call). Must allocate and return x[indx] for 1-based numeric indx that may be outside the bounds of x.
Class-specific methods:
- altstring methods:
  - Elt(i). Must return x[[i]] for 0-based i or signal an error. Required.
  - Set_elt(i, v). Must perform x[[i]] <- v for 0-based i or signal an error. Required.
  - Is_sorted(). If not implemented, will always return UNKNOWN_SORTEDNESS.
  - No_NA(). If not implemented, will always return 0 (unknown whether contains missing values).
- altlist methods:
  - Elt(i) and Set_elt(i, v) like above.
- The rest of the atomic vector types (integer, logical, numeric, complex, raw) will each need a subset of the following methods:
  - Elt(i), Is_sorted(), No_NA(), as above.
  - Get_region(i, n, buf) to populate the buffer buf[n] of the corresponding C type with elements at the given 0-based indices i. The indices are not guaranteed to be within bounds; the number of actually copied elements must be returned. If not implemented, R will use the Elt(i) method, which is slower.
  - Sum(narm), Min(narm), Max(narm) to compute a summary of the vector, optionally ignoring the missing values. If not implemented, R will use Get_region to compute the summaries.

Additionally, data.table will need a function to create new ALTREP tables and to resize the vector in place. The resize function will need to check whether the given value R_altrep_inherits from the over-allocated class and then modify the ALTREP data slots as needed. The function may even reallocate the payload to enlarge the vector in place past the original allocation limit without requiring a setDT call from the user. Since a reallocation will invalidate the data pointer, it must be only used from inside data.table, not from the ALTREP methods.

The original implementation that uses SETLENGTH can be kept behind #if R_VERSION < R_Version(4, 3, 0) for backwards compatibility.

Replacing TRUELENGTH-based growable vectors with ALTREP-based ones will conform to the API, allow growing the vector in place, and avoid the various inconsistencies that happen when R duplicates or deallocates these vectors, but also has the following downsides:

Every place in data.table that uses growable vectors will have to be refactored to use the new abstraction layer (SETLENGTH in R < 4.3, ALTREP in R ≥ 4.3).
- Both implementations will have to be maintained as long as data.table supports R < 4.3.
- The current implementation in data.table re-creates ALTREP objects as ordinary ones because it’s impossible to SET_TRUELENGTH on ALTREP objects and because ALTREP objects may share other columns by reference. This will also need to be refactored.
The data pointer access is slower for ALTREP vectors than for ordinary vectors: having checked the ALTREP bit in the header, R will have to access the method table and call the method instead of adding a fixed offset to the original SEXP pointer. This shouldn’t be noticeable unless data.table puts data pointer access inside a “hot” loop.
For numeric ALTREP classes, ALTREP-aware operations that use *_GET_REGION instead of the data pointer will become slower unless the class implements a Get_region method.

Status in data.table: not fixed yet.

Fast string matching

data.table’s use of TRUELENGTH is not limited to growable buffers. A common idiom is to set the TRUELENGTHs of CHARSXP values from a vector to their negative 1-based indices in that vector, then look up other CHARSXPs in the original vector using -TRUELENGTH(s). This technique relies on R’s CHARSXP cache: for the given string contents and encoding, only one copy of a string created by mkCharLenCE (and related functions) will exist in the memory. As a result, if a string does exist in the original vector, it will be the same CHARSXP object whose TRUELENGTH had been set to its negative index. R does not currently set negative TRUELENGTHs by itself, so any positive TRUELENGTHs can be safely discarded as non-matches.

In the best case scenario, this lookup is very fast: for a table of size and strings to look up in it, it takes memory (the TRUELENGTH is already there, unused) and time for overhead plus time for the actual lookups.

Care must be taken for the technique to work properly:

The strings must be converted to UTF-8. Two copies of the same text in different encodings will be stored in different objects at different addresses, preventing the technique from working:

packageVersion('data.table')
# [1] ‘1.16.99’
x <- data.table(factor(rep(enc2utf8('ø'), 3)))
# memrecycle() forgot to account for encodings
x[1,V1 := iconv('ø', to='latin1')]
as.numeric(x$V1)
# [1] 2 1 1
levels(x$V1) # duplicated levels!
# [1] "ø" "ø"
identical(levels(x$V1)[[1]], levels(x$V1)[[2]])
# [1] TRUE
levels(x$V1) <- levels(x$V1)
levels(x$V1) # R restores unique levels
# [1] "ø"

Any non-zero TRUELENGTH values resulting from R-internal usage must be saved beforehand and restored afterwards.
The TRUELENGTHs are used to look up variables in hashed environments, so R code should not run while the values are disturbed.

The encoding conversions take time and space; the TRUELENGTH bookkeeping takes space and time (thanks to the exponential realloc trick).

The fast string lookup is used in the following places:

src/assign.c: factor level merging in memrecycle, savetl helper
src/rbindlist.c: matching column names, matching factor levels
src/forder.c: (different purpose, same technique) storing the group numbers, looking them up, restoring the original values
src/chmatch.c: saving the original TRUELENGTHs, remembering the positions of CHARSXPs in the table, cleaning up on error, looking up strings in the table, cleaning up before exit
src/fmelt.c: combining factor levels by merging their CHARSXPs in a common array with indices in TRUELENGTH

Since there doesn’t seem to be any intent to allow using R API to place arbitrary integer values inside unused SEXP fields, data.table will have to look up the CHARSXP values using the externally available information. Performing direct pointer comparisons would scale poorly, so for an individual lookup data.table could build a hash table of SEXP pointers. While pointer hashing isn’t strictly guaranteed by the C standard to work, it has been used in R itself. A hash table for CHARSXP pointers would need memory, time to initialise, and average time for lookups (Cormen et al. 2009, chap. 11).

Taking the savetl bookkeeping into account, the average asymptotic performance of TRUELENGTH and hashing for string lookup is the same in both time and space, but the constants are most likely lower for TRUELENGTH. Transitioning to a hash will probably involve a performance hit.

A truly lazy implementation could just use std::unordered_map (at the cost of requiring C++11, which was supported but far from required in R-3.3, and having to shield R from the C++ exceptions) or the permissively-licensed uthash. In cases where the upper bound on the size of the table is known ahead of time, a custom-made open-addressing hash table (Cormen et al. 2009, sec. 11.4) could be implemented with a fixed load factor, requiring only one allocation and no linked lists to walk. In cases where the upper bound is unrealistic (forder, rbindlist), a dynamically-growing hash table will need appropriate locking that won’t slow it down too much.

Status in data.table: not fixed yet.

Marking columns for copying

The use of TRUELENGTH in data.table to mark objects is not limited to CHARSXP strings. Individual columns are also marked in a similar manner for later copying:

In src/dogroups.c, the vectors allocated for the special symbols .BY, .I, .N, .GRP must not be returned by the grouping operations evaluated with dt[..., ..., by=...], so they are marked with a TRUELENGTH of -1, and the marked columns are later re-created.
In src/utils.c, columns share memory or are ALTREP must be copied. Memory sharing between columns may lead to confusing results when they are altered by reference, and ALTREP columns cannot have TRUELENGTH set. The code uses the same trick as with CHARSXP objects: if TRUELENGTH is set on an object, accessing it through a different pointer and seeing a non-zero value will prove that the object had been previously visited. The code first prepares zero TRUELENGTHs, then marks ALTREP, special, and already marked columns for copying, then marks columns not previously marked with their column number, then finally restores the TRUELENGTHs for the columns that won’t be overwritten.
- The SET_TRUELENGTH call in copySharedColumns would fail if it ever got an ALTREP column, but the only use of copySharedColumns in reorder guards against those.

The same solution as above can be used here, with the same downsides of having to allocate memory for the hash table and the potential to have worst-case time for lookups fundamental to hash tables.

Status in data.table: not fixed yet.

But there’s more

Using tools:::funAPI together with the lists of symbols exported from R and imported by data.table, we can find a number of non-API entry points which R CMD check doesn’t complain about yet: ATTRIB, findVar, GetOption, isFrame, OBJECT, SET_ATTRIB, SET_OBJECT.

`(SET_)ATTRIB`, `SET_OBJECT`

data.table performs some direct operations on the attribute pairlists. Accessing attributes directly requires manually maintaining the object bit.

Use getAttrib for individual attributes. To test whether there are any attributes use ANY_ATTRIB, added in R 4.5.0. Use setAttrib for individual attributes, DUPLICATE_ATTRIB or SHALLOW_DUPLICATE_ATTRIB for copying attributes from one object to another. Use CLEAR_ATTRIB for removing all attributes, added in R 4.5.0.

– WRE 6.21.1

Testing for presence of attributes

src/nafill.c checks whether the source object has any attributes before trying to copy them using copyMostAttrib.

Problem:

if (!isNull(ATTRIB(VECTOR_ELT(x, i))))
         // ^^^^^^ non-API entry point

Solution:

#if R_VERSION < R_Version(4, 5, 0)
#define ANY_ATTRIB(x) (!isNull(ATTRIB(x)))
#endif

if (ANY_ATTRIB(VECTOR_ELT(x, i)))
 // ^^^^^^^^^^ introduced in R-4.5

Status in data.table: not fixed yet. Will need to wait for R-4.5.0 to be released with the new interface.

Iterating over all attributes

The code in src/assign.c needs to iterate over all the attributes of attr(dt, 'index') in order to find indices that use the given column.
The code in src/dogroups.c needs to iterate over all attributes of a column in case a reference to the value of a special symbol has been stashed there and must be duplicated.

Without ATTRIB, this will only be possible using an R-level call to attributes(). While the indices could be changed to use a different data structure (a named VECSXP list?), necessitating an update step for data.tables loaded from storage, the code in src/dogroups.c cannot avoid having to see all the attributes.

Status in data.table: no idea how to fix yet.

Raw `c(NA, n)` row names

The code in src/dogroups.c needs to access the raw rownames attribute of a data.table, even if it’s in the compact form as a 2-element integer vector starting with NA. The getAttrib function has a special case for the R_RowNamesSymbol, which returns an ALTREP representation of this attribute.

data.table needs this access in order to temporarily overwrite the rownames attribute for the specially-prepared subset data.table named .SD (which has a different number of rows and therefore needs different rownames). Creating a full-sized rownames attribute instead of its compact form would take more time than desirable.

Status in data.table: no idea how to fix yet.

Direct transplantation of attributes

The code in src/dogroups.c needs to transplant the attributes from one object to another without duplicating them, even shallowly. SHALLOW_DUPLICATE_ATTRIB could work as a replacement, but with worse performance because it would waste time copying attributes from an object that is about to be discarded.

Status in data.table: no idea how to fix yet.

`findVar`

Used in dogroups to look up the pre-created variables corresponding to the special symbols .SDall, .SD, .N, .GRP, .iSD, .xSD in their environment.

The functions findVar and findVarInFrame have been used in a number of packages but are too low level to be part of the API. For most uses the functions R_getVar and R_getVarEx added in R 4.5.0 will be sufficient. These are analogous to the R functions get and get0.

– WRE 6.21.7

The new function R_getVar is different in that it will never return a PROMSXP (which are an internal implementation detail) or an R_UnboundValue, but the current code doesn’t try to care about either.

Example of the problem:

SEXP SD = PROTECT(findVar(install(".SD"), env));
               // ^^^^^^^ non-API function

Solution:

#if R_VERSION < R_Version(4, 5, 0)
#define R_getVar(sym, rho, inherits) \
 ((inherits) ? findVar((sym), (rho)) : findVarInFrame((rho), (sym)))
#endif

SEXP SD = PROTECT(R_getVar(install(".SD"), env, TRUE));
               // ^^^^^^^^ introduced in R-4.5

Status in data.table: not fixed yet. Will need to wait for R-4.5.0 to be released with the new interface.

`GetOption`

Used in src/rbindlist.c to read the datatable.rbindlist.check option, src/freadR.c to read the datatable.old.fread.datetime.character option, src/init.c to read the datatable.verbose option, src/forder.c to get the datatable.use.index and datatable.forder.auto.index options, and src/subset.c to read the datatable.alloccol option.

Use GetOption1.

– WRE 6.21.1

The difference is that GetOption1 doesn’t take a second argument rho, which GetOption has been ignoring anyway.

Example of the problem:

SEXP opt = GetOption(install("datatable.use.index"), R_NilValue);
        // ^^^^^^^^^ non-API function

Solution:

SEXP opt = GetOption1(install("datatable.use.index"));
        // ^^^^^^^^^^ API function introduced in R-2.13

Status in data.table: not fixed yet.

Testing for a `data.frame`: `isFrame`

Back in 2012, Matt Dowle needed to quickly test an object for being a data.frame, and the undocumented function isFrame seemed like it did the job. Since isFrame was not part of the documented interface, in 2024 Luke Tierney gave the function a better-fitting name, isDataFrame, and made it an experimental entry point, while retaining the original function as a wrapper.

Use of isFrame doesn’t give a NOTE yet, but when R-4.5.0 is released together with the new name for the function, data.table will be able to use it, falling back to isFrame on older versions of R. isDataFrame is documented among other replacement entry point names in Writing R Extensions.

Problem (the only instance in data.table):

if (!isVector(thiscol) || isFrame(thiscol))
                       // ^^^^^^^ may disappear in a future R version

Solution:

#if R_VERSION < R_Version(4, 5, 0)
// R versions older than 4.5.0 released use the old name of the function
#define isDataFrame(x) (isFrame(x))
#endif

// later:
if (!isVector(thiscol) || isDataFrame(thiscol))
                       // ^^^^^^^^^^^ introduced in R-4.5

Status in data.table: change reverted in #6244, waiting for R-4.5.0 to release with the new interface.

`OBJECT`

Used in src/assign.c to test whether S3 dispatch is possible on an object before spending CPU time on constructing and evaluating an R-level call to as.character instead of coerceVector.

Use isObject.

– WRE 6.21.1

Problem:

if (OBJECT(source) && getAttrib(source, R_ClassSymbol)!=R_NilValue) {
 // ^^^^^^ non-API entry point

Solution:

if (isObject(source)) {
 // ^^^^^^^^ API entry point

Most likely, the check for getAttrib(source, R_ClassSymbol) is superfluous, because when used correctly, R API maintains the object bit set only when the class attribute is non-empty.

Status in data.table: not fixed yet.

Conclusion

While data.table could get rid of most of its non-API use with relative ease, either using a different name for the function (STRING_PTR_RO, GetOption1) or adding a wrapper for R < 4.5 (ANY_ATTRIB, findVar), two interfaces will require a significant amount of work.

Replacing the use of TRUELENGTH and related functions will require implementing two features from scratch: a set of ALTREP classes for growable vectors (with the previous implementation hidden in #ifdef for R < 4.3) and pointer-keyed hash tables for string and column marking.

It is not currently clear how to replace the use of ATTRIB.

Corrections

Instead of digest, the API introduction now uses the earlier RApiSerialize package as an example.
A paragraph at the end of the growable vectors section used to say that data.table re-creates ALTREP columns because those cannot have TRUELENGTH set. That’s not the only reason: ALTREP columns may also end up referencing special objects like .SD.
A link in the fast string matching section pointed to the wrong lines in the data.table source code.
The fast string matching section used to say that a good upper bound on the number of unique entries is always known; actually, there are two cases where a dynamically growing hash table is required.

Happy birthday, data.table

tips

tutorials

developer

Advent of Code with `data.table`: Week One

tutorials

community

First translation of R vignettes

grant

documentation

translation

Comparing `data.table` reshape to `duckdb` and `polars`

tips

tutorials

developer

benchmarks

Visualizing performance regression of `data.table` with `atime`

performance

testing

developer

Seal of Approval: mlr3

seal of approval

application package

Seal of Approval: collapse

seal of approval

partner package

Newly awarded translation projects

announcements

grant

translation

Seal of Approval: dtplyr

seal of approval

bridge package

Seal of Approval: nc

seal of approval

extension package

Seal of Approval: tidyfast

seal of approval

bridge package

Announcement: The ‘Seal of Approval’

announcements

grant

community

seal of approval

Announcement: Paola Corrales, data.table Ambassador

announcements

grant

ambassadors

travel

Two Roads Diverged

opinion

Testing infrastructure for data.table

grant

testing

developer

Community interviews about data.table

community

grant

Results of the 2023 survey

community

guest post

governance

Column assignment and reference semantics in data.table

tips

tutorials

developer

The Benefits of `data.table` Syntax

tips

tutorials

documentation

New governance, release with new features

governance

releases

Piping data.tables

tips

tutorials

documentation

guest post

Announcement: Jan Gorecki, data.table Ambassador

announcements

grant

ambassadors

Summary of LatinR conference

conferences

Announcement: The data.table Ambassadors Travel Grant

announcements

grant

funding opportunity

Announcement: data.table translation projects

announcements

grant

funding opportunity

Welcome to the data.table ecosystem project!

announcements

grant

References

Becker, Richard A., and John M. Chambers. 1985. Extending the S System. The Wadsworth Statistics/Probability Series. Wadsworth.

Chambers, John M. 2016. Extending R. Chapman & Hall / CRC The R Series. CRC Press.

Cormen, Thomas H., Charles Eric Leiserson, Ronald Linn Rivest, and Clifford Stein. 2009. Introduction to Algorithms. Third edition. MIT Press.

Jones, Richard, Antony Hosking, and Eliot Moss. 2012. The Garbage Collection Handbook: The Art of Automatic Memory Management. Applied Algorithms and Data Structures Series. CRC Press.

Nash, John C., and Arkajyoti Bhattacharjee. 2024. “A Comparison of R Tools for Nonlinear Least Squares Modeling.” The R Journal 15: 198–215. https://doi.org/10.32614/RJ-2023-091.

Advent of Code with data.table: Week One

Kelly Bodwin — Sat, 07 Dec 2024 00:00:00 GMT

Happy December, R friends!

One of my favorite traditions in the R community is the Advent of Code, a series of puzzles released at midnight EST from December 1st through 25th, to be solved through programming in the language of your choosing. I usually do a few of them each year, and once tried to do every single one at the moment it released!

This year, I know I won’t be able to do it daily, but I’m going to do as many as I can using just data.table solutions.

I’ll allow myself to use other packages when there isn’t any data.table equivalent, but my solutions must be as data.table-y as possible.

I’m going to abuse the blog post structure and update this file throughout the week.

library(data.table)

December 1st

Part One

d1 <- fread("day1_dat1.txt")

d1[, V1 := sort(V1)]
d1[, V2 := sort(V2)]
d1[, diff := abs(V1-V2)]

sum(d1$diff)

[1] 2815556

Part Two

d1[, similarity := sum(V1 == d1$V2)*V1, by = V1]

sum(d1$similarity)

[1] 23927637

December 2nd

Part One

d1 <- fread("day2_dat1.txt", fill = TRUE)

check_report <- function(vec) {
  
  vec <- na.omit(vec)
  
  has_neg <- vec < 0
  has_pos <- vec > 0
  
  inc_dec <- sum(has_neg) == length(vec) | sum(has_pos) == length(vec)

  too_big <- max(abs(vec)) > 3
  
  return(inc_dec & !too_big)
}

d1t <- transpose(d1)
deltas <- d1t[-nrow(d1t)] - d1t[2:nrow(d1t)]

res <- apply(deltas, 2, "check_report")

sum(res)

[1] 479

Part Two

test_reports <- function(dat) {

  deltas <- dat[-nrow(dat)] - dat[2:nrow(dat)]

  res <- apply(deltas, 2, "check_report")

  res
}

res <- test_reports(d1t)

for (i in 1:nrow(d1t)) {
  
  res <- res | test_reports(d1t[-i,])
  
  
}

sum(res)

[1] 531

Just for fun

I found the use of apply deeply unsatisfying, even though it was fast, so just for fun:

d1t <- transpose(d1)
deltas <- d1t[-nrow(d1t)] - d1t[2:nrow(d1t)]

is_not_pos <- deltas <= 0
is_not_neg <- deltas >= 0
is_big <- abs(deltas) > 3

res_inc <- colSums(is_not_neg | is_big, na.rm = TRUE)

res_dec <- colSums(is_not_pos | is_big, na.rm = TRUE)

sum(res_inc == 0) + sum(res_dec == 0)

[1] 479

Yay. :)

December 3rd-7th

Nothing data.table-y in this one; you can see my solution here if you want.

Happy birthday, data.table

tips

tutorials

developer

Use of non-API entry points in `data.table`

developer

guest post

performance

First translation of R vignettes

grant

documentation

translation

Comparing `data.table` reshape to `duckdb` and `polars`

tips

tutorials

developer

benchmarks

Visualizing performance regression of `data.table` with `atime`

performance

testing

developer

Seal of Approval: mlr3

seal of approval

application package

Seal of Approval: collapse

seal of approval

partner package

Newly awarded translation projects

announcements

grant

translation

Seal of Approval: dtplyr

seal of approval

bridge package

Seal of Approval: nc

seal of approval

extension package

Seal of Approval: tidyfast

seal of approval

bridge package

Announcement: The ‘Seal of Approval’

announcements

grant

community

seal of approval

Announcement: Paola Corrales, data.table Ambassador

announcements

grant

ambassadors

travel

Two Roads Diverged

opinion

Testing infrastructure for data.table

grant

testing

developer

Community interviews about data.table

community

grant

Results of the 2023 survey

community

guest post

governance

Column assignment and reference semantics in data.table

tips

tutorials

developer

The Benefits of `data.table` Syntax

tips

tutorials

documentation

New governance, release with new features

governance

releases

Piping data.tables

tips

tutorials

documentation

guest post

Announcement: Jan Gorecki, data.table Ambassador

announcements

grant

ambassadors

Summary of LatinR conference

conferences

Announcement: The data.table Ambassadors Travel Grant

announcements

grant

funding opportunity

Announcement: data.table translation projects

announcements

grant

funding opportunity

Welcome to the data.table ecosystem project!

announcements

grant

Comparing data.table reshape to duckdb and polars

Toby Dylan Hocking — Thu, 17 Oct 2024 00:00:00 GMT

One element of the NSF POSE grant for data.table is to create benchmarks which can inform users about when data.table could be more performant than similar software. Two examples of similar software are duckdb and polars, which each provide in-memory database operations. This post explores the differences in computational requirements, and in functionality, for data reshaping operations.

Terminology and functions in R, `data.table`, and SQL

Data reshaping means changing the shape of the data, in order to get it into a more appropriate format, for learning/plotting/etc. In R we use the terms “wide” (many columns, few rows) and “long” (few columns, many rows) to describe the different data shapes (and these terms come from ?stats::reshape), whereas in SQL we use the terms “pivoted” and “unpivoted” to describe these two table types.

R table type	SQL table type	rows	columns
long	unpivoted	many	few
wide	pivoted	few	many

For the wide-to-long reshape operation, data.table has melt() and SQL has UNPIVOT; for the long-to-wide reshape operation, data.table has dcast() and SQL has PIVOT.

Reshape operation	`data.table` function	SQL function
Wide-to-long	`melt`	`UNPIVOT`
Long-to-wide	`dcast`	`PIVOT`

Wide-to-long operations

We begin with a discussion of wide-to-long reshape operations, also known as unpivot in SQL.

Wide-to-long data reshape (unpivot) using `data.table::melt`

Wide-to-long reshape is often necessary before plotting. It is perhaps best explained using a simple example. Here we consider the iris data, which has four numeric columns:

library(data.table)
(iris.wide <- data.table(iris))

     Sepal.Length Sepal.Width Petal.Length Petal.Width   Species
                                      
  1:          5.1         3.5          1.4         0.2    setosa
  2:          4.9         3.0          1.4         0.2    setosa
  3:          4.7         3.2          1.3         0.2    setosa
  4:          4.6         3.1          1.5         0.2    setosa
  5:          5.0         3.6          1.4         0.2    setosa
 ---                                                            
146:          6.7         3.0          5.2         2.3 virginica
147:          6.3         2.5          5.0         1.9 virginica
148:          6.5         3.0          5.2         2.0 virginica
149:          6.2         3.4          5.4         2.3 virginica
150:          5.9         3.0          5.1         1.8 virginica

What if we wanted to make a facetted histogram of the numeric iris data columns, with one panel/facet for each column? With ggplots we would use geom_histogram(aes(numeric_variable)), where numeric_variable would be the column name of a data table containing all of the numbers that we want to show in the histogram. To construct that table, we would have to first reshape to “long” (or unpivoted) format. To easily understand what the reshape operation does, we show a subset of the data (first and last rows) below:

(two.iris.wide <- iris.wide[c(1,.N)])

   Sepal.Length Sepal.Width Petal.Length Petal.Width   Species
                                    
1:          5.1         3.5          1.4         0.2    setosa
2:          5.9         3.0          5.1         1.8 virginica

Note the table above has 8 numbers, arranged into a table of 2 rows and 4 columns. To reshape these data to “long” (or unpivoted) format, we can use data.table::melt, as in the code below.

melt(two.iris.wide, measure.vars=measure(part, dim, sep="."))

     Species   part    dim value
         
1:    setosa  Sepal Length   5.1
2: virginica  Sepal Length   5.9
3:    setosa  Sepal  Width   3.5
4: virginica  Sepal  Width   3.0
5:    setosa  Petal Length   1.4
6: virginica  Petal Length   5.1
7:    setosa  Petal  Width   0.2
8: virginica  Petal  Width   1.8

Note the table above has the same 8 numbers, but arranged into 1 column in a table with 8 rows, which is the desired input for ggplots. Also note that the reshaped column names (Petal.Length, Sepal.Width, etc) each consist of two components, which become two different columns in the output: part (Sepal or Petal) and dim (Length or Width). In the code above, we used sep="." to specify that we want to split all of the iris column names using a dot, and then reshape all of the columns whose names split into the max number of items. The corresponding column names of the output are specified as the arguments of measure(), and for more info about this functionality, please read its man page.

Below we do the same reshape with the full iris data set, this time using a regular expression (instead of the sep argument used above),

(iris.long <- melt(iris.wide, measure.vars=measure(part, dim, pattern="(.*)[.](.*)")))

       Species   part    dim value
           
  1:    setosa  Sepal Length   5.1
  2:    setosa  Sepal Length   4.9
  3:    setosa  Sepal Length   4.7
  4:    setosa  Sepal Length   4.6
  5:    setosa  Sepal Length   5.0
 ---                              
596: virginica  Petal  Width   2.3
597: virginica  Petal  Width   1.9
598: virginica  Petal  Width   2.0
599: virginica  Petal  Width   2.3
600: virginica  Petal  Width   1.8

In the code above, the pattern argument is a Perl-compatible regular expression, and columns that match the pattern will be reshaped. The pattern must contain the same number of capture groups (parentheses) as the number of other arguments to melt (part and dim), which are used for output column names. After reshaping, we plot the data in a histogram:

library(ggplot2)
ggplot()+
  geom_histogram(aes(
    value),
    bins=50,
    data=iris.long)+
  facet_grid(part ~ dim, labeller=label_both)

We can see in the plot above that there is a top strip for each dim and a right strip for each part, and each facet/panel contains a histogram of the corresponding subset of data.

Wide-to-long reshape via unpivot in `polars`

We’re all friends here.

polars is an implementation of data frames in Rust, with bindings in R and Python. In polars, the wide-to-long data reshape operation is documented on the man page for unpivot, which explains that we must specify index and/or on (no support for separator, nor regex). In our case, we use the code below:

(iris.long.polars <- polars::as_polars_df(iris)$unpivot(
  index="Species",
  on=c("Sepal.Length","Petal.Length","Sepal.Width","Petal.Width"),
  variable_name="part.dim",
  value_name="cm"))

shape: (600, 3)
┌───────────┬──────────────┬─────┐
│ Species   ┆ part.dim     ┆ cm  │
│ ---       ┆ ---          ┆ --- │
│ cat       ┆ str          ┆ f64 │
╞═══════════╪══════════════╪═════╡
│ setosa    ┆ Sepal.Length ┆ 5.1 │
│ setosa    ┆ Sepal.Length ┆ 4.9 │
│ setosa    ┆ Sepal.Length ┆ 4.7 │
│ setosa    ┆ Sepal.Length ┆ 4.6 │
│ setosa    ┆ Sepal.Length ┆ 5.0 │
│ …         ┆ …            ┆ …   │
│ virginica ┆ Petal.Width  ┆ 2.3 │
│ virginica ┆ Petal.Width  ┆ 1.9 │
│ virginica ┆ Petal.Width  ┆ 2.0 │
│ virginica ┆ Petal.Width  ┆ 2.3 │
│ virginica ┆ Petal.Width  ┆ 1.8 │
└───────────┴──────────────┴─────┘

The output above is analogous to the result from data.table::melt, but with one column named part.dim instead of the two columns named part and dim, because polars does not support splitting the reshaped column names into more than one output column. So with polars, if we wanted separate part and dim columns, we would have to specify that in a separate step, after the reshape. Or we could just use facet_wrap instead of facet_grid, as in the code below:

ggplot()+
  geom_histogram(aes(
    cm),
    bins=50,
    data=iris.long.polars)+
  facet_wrap(. ~ part.dim, labeller=label_both)

We can see in the plot above that there is a facet for each of the variables, but only one part.dim strip for each, instead of two strips (part and dim), as was the case for the previous plot.

Wide-to-long reshape via UNPIVOT in `duckdb`

(Image generated with Adobe Firefly.)

duckdb is a column-oriented database implemented in C++, with an R package that supports a DBI-compliant SQL interface. That means that we use R functions like DBI::dbGetQuery to get results, just like we would with any other database (Postgres, MySQL, etc). This is documented in the duckdb R API docs, which explain how to create a database connection, and then copy data from R to the database, as in the code below,

con <- DBI::dbConnect(duckdb::duckdb(), dbdir = ":memory:")
DBI::dbWriteTable(con, "iris_wide", iris)

The duckdb unpivot man page explains how to do wide-to-long reshape operations, which requires specifying names of columns to reshape (no support for separator, nor regex). In our case, we use the code below:

iris.long.duckdb <- DBI::dbGetQuery(con, '
UNPIVOT iris_wide
ON "Sepal.Length", "Petal.Length", "Sepal.Width", "Petal.Width" 
INTO NAME part_dim 
VALUE cm')
str(iris.long.duckdb)

'data.frame':   600 obs. of  3 variables:
 $ Species : Factor w/ 3 levels "setosa","versicolor",..: 1 1 1 1 1 1 1 1 1 1 ...
 $ part_dim: chr  "Sepal.Length" "Petal.Length" "Sepal.Width" "Petal.Width" ...
 $ cm      : num  5.1 1.4 3.5 0.2 4.9 1.4 3 0.2 4.7 1.3 ...

Above we use str to show a brief summary of the structure of the output, which is a data.frame with 600 rows. With duckdb, the output has one column named part_dim (dots in column names are not allowed so we use an underscore here instead), because it does not support splitting the reshaped column names into more than one output column. So with duckdb, if we wanted separate part and dim columns, we would have to specify that in a separate step, after the reshape.

Creating `part` and `dim` columns

Both polars and duckdb are not capable of producing the separate part and dim columns during the reshape operation, but we can always do it as a post-processing step. One way to do that, by specifying a separator, would be via data.table::tstrsplit, as in the code below:

data.table(iris.long.duckdb)[
, c("part","dim") := tstrsplit(part_dim, split="[.]")
][]

       Species     part_dim    cm   part    dim
                  
  1:    setosa Sepal.Length   5.1  Sepal Length
  2:    setosa Petal.Length   1.4  Petal Length
  3:    setosa  Sepal.Width   3.5  Sepal  Width
  4:    setosa  Petal.Width   0.2  Petal  Width
  5:    setosa Sepal.Length   4.9  Sepal Length
 ---                                           
596: virginica  Petal.Width   2.3  Petal  Width
597: virginica Sepal.Length   5.9  Sepal Length
598: virginica Petal.Length   5.1  Petal Length
599: virginica  Sepal.Width   3.0  Sepal  Width
600: virginica  Petal.Width   1.8  Petal  Width

The code above first converts to data.table, then uses the square brackets to assign new columns. Inside the square brackets, there is a walrus assignment:

, comma because there is no first argument (no subset, use all rows)
c("part","dim") is the left side of the walrus := assignment, which specifies the new column names to create.
on the right side of the walrus, the result of tstrsplit(part_dim, split="[.]") is used as the value to assign to the new columns (part_dim is the column to split, and "[.]" is the regex to use for splitting).
Since tstrsplit returns a list of two character vectors, there will be two new columns.

Finally after the walrus square brackets, we use another empty square brackets [] to enable printing (there is no printing immediately after assigning new columns using the walrus operator).

Another way of doing that, by specifying a regex, would be via nc::capture_first_df (recently given the data.table Seal of Approval), as in the code below:

nc::capture_first_df(iris.long.duckdb, part_dim=list(
  part=".*",
  "[.]",
  dim=".*"))

       Species     part_dim    cm   part    dim
                  
  1:    setosa Sepal.Length   5.1  Sepal Length
  2:    setosa Petal.Length   1.4  Petal Length
  3:    setosa  Sepal.Width   3.5  Sepal  Width
  4:    setosa  Petal.Width   0.2  Petal  Width
  5:    setosa Sepal.Length   4.9  Sepal Length
 ---                                           
596: virginica  Petal.Width   2.3  Petal  Width
597: virginica Sepal.Length   5.9  Sepal Length
598: virginica Petal.Length   5.1  Petal Length
599: virginica  Sepal.Width   3.0  Sepal  Width
600: virginica  Petal.Width   1.8  Petal  Width

The code above specifies:

capture_first_df, a function for applying capturing regex to columns of a data frame;
iris.long.duckdb is the input data frame, in which there is the part_dim column to split;
part=".*", "[.]", dim=".*" makes the capturing regex; R argument names are used to define the new column names, based on the text captured in the corresponding regex (".*" means zero or more non-newline characters).

Both results above are data tables with extra cols part and dim. For visualization, these data tables could be used with either facet_grid or facet_wrap, similar to the examples above.

Reshape into multiple columns

Another kind of wide-to-long reshape involves reshaping into multiple columns. For example, in the iris data, we may wonder whether sepals are larger than petals (in terms of both length and width). To answer that question, we could make a scatterplot of y=Sepal versus x=Petal, with a facet/panel for each dimension (Length and Width). In the ggplot system, we would need to compute a data table with columns Sepal, Petal, and dim, and we can do that by specifying the value.name keyword to measure(), as in the code below:

(iris.long.parts <- melt(iris.wide, measure.vars=measure(value.name, dim, sep=".")))

       Species    dim Sepal Petal
           
  1:    setosa Length   5.1   1.4
  2:    setosa Length   4.9   1.4
  3:    setosa Length   4.7   1.3
  4:    setosa Length   4.6   1.5
  5:    setosa Length   5.0   1.4
 ---                             
296: virginica  Width   3.0   2.3
297: virginica  Width   2.5   1.9
298: virginica  Width   3.0   2.0
299: virginica  Width   3.4   2.3
300: virginica  Width   3.0   1.8

Again, the measure() function in the code above operates by splitting the input column names using sep, which results in two groups (Sepal.Width split into Sepal and Width, etc) for each of the measured columns. The value.name keyword indicates that each unique value in the first group (Sepal and Petal) should be used as the name of an output column. This functionality can be very convenient for some data reshaping tasks, but it is neither supported in polars, nor in duckdb. Going back to our original motivating problem, we can make the scatterplot using the code below,

ggplot()+
  theme_bw()+
  geom_abline(slope=1, intercept=0, color="grey")+
  geom_point(aes(
    Petal, Sepal),
    data=iris.long.parts)+
  facet_grid(. ~ dim, labeller=label_both)+
  coord_equal()

From the plot above, we see that all of the data points (black) are above the y=x line (grey), so we can conclude that sepals are indeed larger than petals, in terms of both length and width.

Wide-to-long performance comparison

We may also wonder which data reshaping functions work fastest for large data. To answer that question, we will use atime, which is an R package that allows us to see how much time/memory is required for computations in R, as a function of data size N. In the setup argument of the code below, we repeat the iris data for a certain number of rows N. The code in the other arguments is run for the time/memory measurement, and is very similar to the code presented in previous sections. One difference is that for data.table we use id.vars instead of measure(), to more closely match the arguments provided to the other unpivot functions (for a more fair comparison).

seconds.limit <- 0.1
unpivot.res <- atime::atime(
  N=2^seq(1,50),
  setup={
    (row.id.vec <- 1+(seq(0,N-1) %% nrow(iris)))
    N.df <- iris[row.id.vec,]
    N.dt <- data.table(N.df)
    polars_df <- polars::as_polars_df(N.df)
    duckdb::dbWriteTable(con, "iris_table", N.df, overwrite=TRUE)
  },
  seconds.limit=seconds.limit,
  "duckdb\nUNPIVOT"=DBI::dbGetQuery(con, 'UNPIVOT iris_table ON "Sepal.Length", "Petal.Length", "Sepal.Width", "Petal.Width" INTO NAME part_dim VALUE cm'),
  "polars\nunpivot"=polars_df$unpivot(index="Species", value_name="cm"),
  "data.table\nmelt"=melt(N.dt, id.vars="Species", value.name="cm"))
unpivot.refs <- atime::references_best(unpivot.res)
unpivot.pred <- predict(unpivot.refs)
plot(unpivot.pred)+coord_cartesian(xlim=c(1e1,1e7))

Loading required namespace: directlabels

Warning in ggplot2::scale_x_log10("N", breaks = meas[,
10^seq(ceiling(min(log10(N))), : log-10 transformation introduced infinite
values.

In the plot above, the computation time in seconds is plotted as a function of N, the number of input rows to reshape. The horizontal reference line is drawn at 0.1 seconds, and the N highlighted corresponds to the throughput given that time limit. When we compare the N values shown for the different methods, we see that data.table is comparable to polars (within 2x), and both are much faster than duckdb (about 10x).

Above there are several confounding factors in the comparison, most notably that data must be copied to duckdb and polars before and after processing. In contrast, data.table provides setDT and setDF functions, which can convert to/from data tables, without copying. So when data originates in R, or needs to come back to R, we should include the copy time for a more fair comparison. Below we run that comparison:

seconds.limit <- 0.1
unpivot.copy.res <- atime::atime(
  N=2^seq(1,50),
  setup={
    (row.id.vec <- 1+(seq(0,N-1) %% nrow(iris)))
    N.df <- iris[row.id.vec,]
  },
  seconds.limit=seconds.limit,
  "duckdb\ncopy+UNPIVOT"={
    duckdb::dbWriteTable(con, "iris_table", N.df, overwrite=TRUE)
    DBI::dbGetQuery(con, 'UNPIVOT iris_table ON "Sepal.Length", "Petal.Length", "Sepal.Width", "Petal.Width" INTO NAME part_dim VALUE cm')
  },
  "polars\ncopy+unpivot"={
    polars_df <- polars::as_polars_df(N.df)
    polars_unpivot <- polars_df$unpivot(index="Species", value_name="cm")
    as.data.frame(polars_unpivot)
  },
  "data.table\nset+melt"=setDF(melt(setDT(N.df), id.vars="Species", value.name="cm")))
unpivot.copy.refs <- atime::references_best(unpivot.copy.res)
unpivot.copy.pred <- predict(unpivot.copy.refs)
plot(unpivot.copy.pred)+coord_cartesian(xlim=c(1e1,1e7))

Warning in ggplot2::scale_x_log10("N", breaks = meas[,
10^seq(ceiling(min(log10(N))), : log-10 transformation introduced infinite
values.

The result above shows that data.table is most efficient in terms of computation time. In this comparison, data.table is clearly faster than polars (about 10x), and much faster than duckdb (about 100x).

Wide-to-long summary of functionality

Wide to long.

In this section, we showed that data.table provides an efficient and feature-rich implementation of wide-to-long data reshaping.

measure() allows specification of columns to reshape using either a separator or a regular expression pattern. In contrast, duckdb nor polars require specifying input column names (no support for separator, nor regex), and output column post-processing, which is less convenient.
The value.name keyword can be used to reshape into multiple output columns, which is required for some kinds of reshape operations (no way to do that in duckdb/polars).
setDT and setDF can be used to avoid un-necessary copies with data.table. In contrast, duckdb/polars require copies to/from regular R memory, which can add significant time/memory requirements.
data.table was fastest and most memory efficient in the comparisons we examined (both with and without consideration of copying).

The table below summarizes support for different features in each software package (dash - means no support).

how to specify	`data.table`	`polars`	`duckdb`
function	`melt`	`unpivot`	`UNPIVOT`
reshape cols	`measure.vars`	`on`	`ON`
other cols	`id.vars`	`index`	-
output name (data)	`value.name`	`value_name`	`VALUE`
output name (columns)	`variable.name`	`variable_name`	`INTO NAME`
separator	`sep`	-	-
regex	`pattern`	-	-
multiple outputs	`value.name`	-	-
avoid copies	`setDT`, `setDF`	-	-

Long-to-wide operations

Another kind of reshape operation starts with a long table (many rows, few cols), and creates a wide table (many cols, few rows). I frequently use this operation when comparing results of machine learning algorithms (computing mean/SD over folds, p-values, etc). For examples of those use cases, please read my blog about Visualizing prediction error.

Long-to-wide data reshape using `data.table::dcast`

Here we continue with the iris data example. We will present three different reshape operations involving the iris data. The code below adds a column flower which contains the row number.

iris.wide[, flower := .I][]

     Sepal.Length Sepal.Width Petal.Length Petal.Width   Species flower
                                        
  1:          5.1         3.5          1.4         0.2    setosa      1
  2:          4.9         3.0          1.4         0.2    setosa      2
  3:          4.7         3.2          1.3         0.2    setosa      3
  4:          4.6         3.1          1.5         0.2    setosa      4
  5:          5.0         3.6          1.4         0.2    setosa      5
 ---                                                                   
146:          6.7         3.0          5.2         2.3 virginica    146
147:          6.3         2.5          5.0         1.9 virginica    147
148:          6.5         3.0          5.2         2.0 virginica    148
149:          6.2         3.4          5.4         2.3 virginica    149
150:          5.9         3.0          5.1         1.8 virginica    150

Then we do a wide-to-long reshape using the code below (same as previous section),

(iris.long.i <- melt(iris.wide, measure.vars=measure(part, dim, sep=".")))

       Species flower   part    dim value
             
  1:    setosa      1  Sepal Length   5.1
  2:    setosa      2  Sepal Length   4.9
  3:    setosa      3  Sepal Length   4.7
  4:    setosa      4  Sepal Length   4.6
  5:    setosa      5  Sepal Length   5.0
 ---                                     
596: virginica    146  Petal  Width   2.3
597: virginica    147  Petal  Width   1.9
598: virginica    148  Petal  Width   2.0
599: virginica    149  Petal  Width   2.3
600: virginica    150  Petal  Width   1.8

The table above has an additional column for flower, which we use in the code below on the left side of the formula (used to define output rows), along with part + dim on the right side of the formula (used to define output columns). The code below can therefore be used to reshape the data back into their original wide format:

dcast(# wide reshape 1
  data=iris.long.i,
  formula=flower + Species ~ part + dim,
  sep=".")

Key: 
     flower   Species Petal.Length Petal.Width Sepal.Length Sepal.Width
                                        
  1:      1    setosa          1.4         0.2          5.1         3.5
  2:      2    setosa          1.4         0.2          4.9         3.0
  3:      3    setosa          1.3         0.2          4.7         3.2
  4:      4    setosa          1.5         0.2          4.6         3.1
  5:      5    setosa          1.4         0.2          5.0         3.6
 ---                                                                   
146:    146 virginica          5.2         2.3          6.7         3.0
147:    147 virginica          5.0         1.9          6.3         2.5
148:    148 virginica          5.2         2.0          6.5         3.0
149:    149 virginica          5.4         2.3          6.2         3.4
150:    150 virginica          5.1         1.8          5.9         3.0

We can see that the result above is almost the same as the original iris data (but with the columns in a different order). Another kind of reshape involves computing an aggregation function, such as mean. Note in the code below that . on the right side of the formula indicates a single output column.

dcast(# wide reshape 2
  data=iris.long.i,
  formula=Species + part + dim ~ .,
  fun.aggregate=mean,
  sep=".")

Key: 
       Species   part    dim     .
           
 1:     setosa  Petal Length 1.462
 2:     setosa  Petal  Width 0.246
 3:     setosa  Sepal Length 5.006
 4:     setosa  Sepal  Width 3.428
 5: versicolor  Petal Length 4.260
 6: versicolor  Petal  Width 1.326
 7: versicolor  Sepal Length 5.936
 8: versicolor  Sepal  Width 2.770
 9:  virginica  Petal Length 5.552
10:  virginica  Petal  Width 2.026
11:  virginica  Sepal Length 6.588
12:  virginica  Sepal  Width 2.974

The output above has a row for every unique combination of Species, part, and dim, and a column (.)` for the mean of the corresponding data. The more complex reshape below involves multiple aggregations, and multiple value variables.

options(width=100)
dcast(# wide reshape 3
  data=iris.long.parts,
  formula=dim ~ Species,
  fun.aggregate=list(mean,sd),
  value.var=c("Sepal","Petal"))

Key: 
      dim Sepal_mean_setosa Sepal_mean_versicolor Sepal_mean_virginica Petal_mean_setosa
                                                              
1: Length             5.006                 5.936                6.588             1.462
2:  Width             3.428                 2.770                2.974             0.246
   Petal_mean_versicolor Petal_mean_virginica Sepal_sd_setosa Sepal_sd_versicolor
                                                             
1:                 4.260                5.552       0.3524897           0.5161711
2:                 1.326                2.026       0.3790644           0.3137983
   Sepal_sd_virginica Petal_sd_setosa Petal_sd_versicolor Petal_sd_virginica
                                                        
1:          0.6358796       0.1736640           0.4699110          0.5518947
2:          0.3224966       0.1053856           0.1977527          0.2746501

The output above includes two rows, and a column for every unique combination of value.var (Sepal or Petal), of fun.aggregate (mean or sd), and of Species (setosa, versicolor, virginica).

Long-to-wide reshape in `polars`

polars supports long-to-wide reshape via the pivot method, as in the code below.

(polars.wide <- polars::as_polars_df(
  iris.long.i
)$pivot(# wide reshape 1
  on=c("part","dim"),
  index=c("flower","Species"),
  values="value"))

shape: (150, 6)
┌────────┬───────────┬───────────────────┬───────────────────┬──────────────────┬──────────────────┐
│ flower ┆ Species   ┆ {"Sepal","Length" ┆ {"Sepal","Width"} ┆ {"Petal","Length ┆ {"Petal","Width" │
│ ---    ┆ ---       ┆ }                 ┆ ---               ┆ "}               ┆ }                │
│ i32    ┆ cat       ┆ ---               ┆ f64               ┆ ---              ┆ ---              │
│        ┆           ┆ f64               ┆                   ┆ f64              ┆ f64              │
╞════════╪═══════════╪═══════════════════╪═══════════════════╪══════════════════╪══════════════════╡
│ 1      ┆ setosa    ┆ 5.1               ┆ 3.5               ┆ 1.4              ┆ 0.2              │
│ 2      ┆ setosa    ┆ 4.9               ┆ 3.0               ┆ 1.4              ┆ 0.2              │
│ 3      ┆ setosa    ┆ 4.7               ┆ 3.2               ┆ 1.3              ┆ 0.2              │
│ 4      ┆ setosa    ┆ 4.6               ┆ 3.1               ┆ 1.5              ┆ 0.2              │
│ 5      ┆ setosa    ┆ 5.0               ┆ 3.6               ┆ 1.4              ┆ 0.2              │
│ …      ┆ …         ┆ …                 ┆ …                 ┆ …                ┆ …                │
│ 146    ┆ virginica ┆ 6.7               ┆ 3.0               ┆ 5.2              ┆ 2.3              │
│ 147    ┆ virginica ┆ 6.3               ┆ 2.5               ┆ 5.0              ┆ 1.9              │
│ 148    ┆ virginica ┆ 6.5               ┆ 3.0               ┆ 5.2              ┆ 2.0              │
│ 149    ┆ virginica ┆ 6.2               ┆ 3.4               ┆ 5.4              ┆ 2.3              │
│ 150    ┆ virginica ┆ 5.9               ┆ 3.0               ┆ 5.1              ┆ 1.8              │
└────────┴───────────┴───────────────────┴───────────────────┴──────────────────┴──────────────────┘

names(polars.wide)

[1] "flower"                 "Species"                "{\"Sepal\",\"Length\"}"
[4] "{\"Sepal\",\"Width\"}"  "{\"Petal\",\"Length\"}" "{\"Petal\",\"Width\"}"

The output above is consistent with the results from data.table::dcast, and the original iris data, although the names are unusual (with curly braces and double quotes). The next reshape example below shows that we need to create a dummy variable to use as the on argument.

polars::as_polars_df(
  iris.long.i[, dummy := "."]
)$pivot(# wide reshape 2
  on="dummy", # have to create dummy var for on.
  index=c("Species","part","dim"),
  values="value",
  aggregate_function="mean")

shape: (12, 4)
┌────────────┬───────┬────────┬───────┐
│ Species    ┆ part  ┆ dim    ┆ .     │
│ ---        ┆ ---   ┆ ---    ┆ ---   │
│ cat        ┆ str   ┆ str    ┆ f64   │
╞════════════╪═══════╪════════╪═══════╡
│ setosa     ┆ Sepal ┆ Length ┆ 5.006 │
│ versicolor ┆ Sepal ┆ Length ┆ 5.936 │
│ virginica  ┆ Sepal ┆ Length ┆ 6.588 │
│ setosa     ┆ Sepal ┆ Width  ┆ 3.428 │
│ versicolor ┆ Sepal ┆ Width  ┆ 2.77  │
│ …          ┆ …     ┆ …      ┆ …     │
│ versicolor ┆ Petal ┆ Length ┆ 4.26  │
│ virginica  ┆ Petal ┆ Length ┆ 5.552 │
│ setosa     ┆ Petal ┆ Width  ┆ 0.246 │
│ versicolor ┆ Petal ┆ Width  ┆ 1.326 │
│ virginica  ┆ Petal ┆ Width  ┆ 2.026 │
└────────────┴───────┴────────┴───────┘

The output above is consistent with the results from data.table::dcast. Currently polars only supports a single aggregation function, so we can not calculate both mean and sd at the same time, but we can at least do the mean for multiple values in the code below:

polars::as_polars_df(
  iris.long.parts
)$pivot(# wide reshape 3
  on="Species",
  index="dim",
  values=c("Sepal","Petal"),
  aggregate_function="mean")#multiple agg not supported.

shape: (2, 7)
┌────────┬──────────────┬──────────────┬──────────────┬──────────────┬──────────────┬──────────────┐
│ dim    ┆ Sepal_setosa ┆ Sepal_versic ┆ Sepal_virgin ┆ Petal_setosa ┆ Petal_versic ┆ Petal_virgin │
│ ---    ┆ ---          ┆ olor         ┆ ica          ┆ ---          ┆ olor         ┆ ica          │
│ str    ┆ f64          ┆ ---          ┆ ---          ┆ f64          ┆ ---          ┆ ---          │
│        ┆              ┆ f64          ┆ f64          ┆              ┆ f64          ┆ f64          │
╞════════╪══════════════╪══════════════╪══════════════╪══════════════╪══════════════╪══════════════╡
│ Length ┆ 5.006        ┆ 5.936        ┆ 6.588        ┆ 1.462        ┆ 4.26         ┆ 5.552        │
│ Width  ┆ 3.428        ┆ 2.77         ┆ 2.974        ┆ 0.246        ┆ 1.326        ┆ 2.026        │
└────────┴──────────────┴──────────────┴──────────────┴──────────────┴──────────────┴──────────────┘

Above we see the result only has 6 columns (for mean), whereas the analogous result from data.table::dcast above had 12 columns (with additionally the sd).

Long-to-wide reshape in `duckdb`

duckdb supports long-to-wide reshape via the SQL PIVOT command, which can be used to recover the original iris data via the command below:

duckdb::dbWriteTable(con, "iris_long_i", iris.long.i, overwrite=TRUE)
iris.wide.again.duckdb <- DBI::dbGetQuery(# wide reshape 1
  con, '
PIVOT iris_long_i 
ON part,dim 
USING sum(value) 
GROUP BY flower,Species 
ORDER BY flower')
str(iris.wide.again.duckdb)

'data.frame':   150 obs. of  6 variables:
 $ flower      : int  1 2 3 4 5 6 7 8 9 10 ...
 $ Species     : Factor w/ 3 levels "setosa","versicolor",..: 1 1 1 1 1 1 1 1 1 1 ...
 $ Petal_Length: num  1.4 1.4 1.3 1.5 1.4 1.7 1.4 1.5 1.4 1.5 ...
 $ Petal_Width : num  0.2 0.2 0.2 0.2 0.2 0.4 0.3 0.2 0.2 0.1 ...
 $ Sepal_Length: num  5.1 4.9 4.7 4.6 5 5.4 4.6 5 4.4 4.9 ...
 $ Sepal_Width : num  3.5 3 3.2 3.1 3.6 3.9 3.4 3.4 2.9 3.1 ...

We can see that the result above is consistent with the previous sections. The code below uses mean as an aggregation function.

DBI::dbGetQuery(# wide reshape 2
  con, '
PIVOT iris_long_i 
USING mean(value) 
AS "." 
GROUP BY Species,part,dim')

      Species  part    dim     .
1  versicolor Petal Length 4.260
2  versicolor Petal  Width 1.326
3  versicolor Sepal Length 5.936
4   virginica Sepal Length 6.588
5  versicolor Sepal  Width 2.770
6   virginica Sepal  Width 2.974
7   virginica Petal Length 5.552
8   virginica Petal  Width 2.026
9      setosa Sepal Length 5.006
10     setosa Sepal  Width 3.428
11     setosa Petal Length 1.462
12     setosa Petal  Width 0.246

The result above is consistent with previous results. Finally, we can do multiple aggregations via the code below, which requires enumerating each combination of aggregation function and input column to aggregate.

duckdb::dbWriteTable(con, "iris_long_parts", iris.long.parts, overwrite=TRUE)
DBI::dbGetQuery(# wide reshape 3
  con, '
PIVOT iris_long_parts 
ON Species 
USING 
 mean(Sepal) AS Sepal_mean, 
 stddev(Sepal) AS Sepal_sd, 
 mean(Petal) AS Petal_mean, 
 stddev(Petal) AS Petal_sd 
GROUP BY dim')

     dim setosa_Sepal_mean setosa_Sepal_sd setosa_Petal_mean setosa_Petal_sd versicolor_Sepal_mean
1 Length             5.006       0.3524897             1.462       0.1736640                 5.936
2  Width             3.428       0.3790644             0.246       0.1053856                 2.770
  versicolor_Sepal_sd versicolor_Petal_mean versicolor_Petal_sd virginica_Sepal_mean
1           0.5161711                 4.260           0.4699110                6.588
2           0.3137983                 1.326           0.1977527                2.974
  virginica_Sepal_sd virginica_Petal_mean virginica_Petal_sd
1          0.6358796                5.552          0.5518947
2          0.3224966                2.026          0.2746501

The result above is consistent with the result from data.table::dcast. Because all combinations of aggregation/columns must be enumerated, the duckdb code is a bit more repetitive than the corresponding data.table code (which is more convenient).

Long-to-wide performance comparison

Below we conduct an atime benchmark to measure the computation time of the reshape operation (without controlling for the copy operation).

seconds.limit <- 0.1
pivot.res <- atime::atime(
  N=2^seq(1,50),
  setup={
    (row.id.vec <- 1+(seq(0,N-1) %% nrow(iris.long.i)))
    N.dt <- iris.long.i[row.id.vec]
    N.df <- data.frame(N.dt)
    N_polars <- polars::as_polars_df(N.df)
    duckdb::dbWriteTable(con, "iris_long_i", N.df, overwrite=TRUE)
  },
  seconds.limit=seconds.limit,
  "duckdb\nPIVOT"=DBI::dbGetQuery(con, 'PIVOT iris_long_i USING mean(value) AS "." GROUP BY Species,part,dim'),
  "polars\npivot"=N_polars$pivot(on="dummy", index=c("Species","part","dim"), values="value", aggregate_function="mean"),
  "data.table\ndcast"=dcast(N.dt, Species + part + dim ~ ., mean))
pivot.refs <- atime::references_best(pivot.res)
pivot.pred <- predict(pivot.refs)
plot(pivot.pred)+coord_cartesian(xlim=c(1e1,1e7))

Warning in ggplot2::scale_x_log10("N", breaks = meas[, 10^seq(ceiling(min(log10(N))), : log-10
transformation introduced infinite values.

The result above shows that data.table::dcast is about 10x slower than duckdb. Below we run a more complex benchmark which also measures computation time for the copy operation (in addition to the reshape).

seconds.limit <- 0.1
pivot.copy.res <- atime::atime(
  N=2^seq(1,50),
  setup={
    (row.id.vec <- 1+(seq(0,N-1) %% nrow(iris.long.i)))
    N.df <- data.frame(iris.long.i[row.id.vec])
  },
  seconds.limit=seconds.limit,
  "duckdb\ncopy+PIVOT"={
    duckdb::dbWriteTable(con, "iris_long_i", N.df, overwrite=TRUE)
    DBI::dbGetQuery(con, 'PIVOT iris_long_i USING mean(value) AS "." GROUP BY Species,part,dim')
  },
  "polars\ncopy+pivot"={
    polars_pivot <- polars::as_polars_df(
      N.df
    )$pivot(# wide reshape 2
      on="dummy", # have to create dummy var for on.
      index=c("Species","part","dim"),
      values="value",
      aggregate_function="mean")
    as.data.frame(polars_pivot)
  },
  "data.table\nset+dcast"=setDF(dcast(setDT(N.df), Species + part + dim ~ ., mean)))
pivot.copy.refs <- atime::references_best(pivot.copy.res)
pivot.copy.pred <- predict(pivot.copy.refs)
plot(pivot.copy.pred)+coord_cartesian(xlim=c(1e1,1e7))

Warning in ggplot2::scale_x_log10("N", breaks = meas[, 10^seq(ceiling(min(log10(N))), : log-10
transformation introduced infinite values.

The result above shows that data.table is a bit faster than the others (2x or more).

Summary of long-to-wide reshaping

In this section, we showed that data.table provides an efficient and feature-rich implementation of long-to-wide data reshaping.

The formula interface allows specifying a dot (.) which is a convenient way to specify output of only one row/column. In contrast, polars requires creating a dummy variable to do that.
The fun.aggregate argument may be a list of functions, each of which will be used on each of the value.var (a convenient way of specifying all combinations). In contrast, duckdb requires specifying each combination separately (more tedious/error-prone), and polars only supports one aggregation function (not a list).

how to specify	`data.table`	`polars`	`duckdb`
function	`dcast`	`pivot`	`PIVOT`
rows	LHS of formula	`index`	`GROUP BY`
columns	RHS of formula	`on`	`ON`
no columns	dot `.`	dummy variable	omit `ON`
values	`value.var`	`values`	`USING`
aggregation	`aggregate.fun`	`aggregate_function`	`USING`
multiple agg.	all combinations	one function	specified combinations

Conclusion

We have compared the reshaping functions in data.table to duckdb and polars. Although all three provide similar functionality for basic operations, we observed that data.table has several advantages for advanced/complex/efficient operations (reshaping columns which match a regex/separator, reshaping into multiple columns, avoiding copies, multiple aggregation). We also observed that the speed of data.table functions is comparable, if not faster, than the other packages.

Attribution

Parts of this blog post were copied from my more extensive comparison blog.

Session info

sessionInfo()

R version 4.5.3 (2026-03-11)
Platform: x86_64-pc-linux-gnu
Running under: Ubuntu 24.04.4 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.12.0 
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.12.0  LAPACK version 3.12.0

locale:
 [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8        LC_COLLATE=C.UTF-8    
 [5] LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8    LC_PAPER=C.UTF-8       LC_NAME=C             
 [9] LC_ADDRESS=C           LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   

time zone: Etc/UTC
tzcode source: system (glibc)

attached base packages:
[1] stats     graphics  grDevices datasets  utils     methods   base     

other attached packages:
[1] ggplot2_4.0.2       data.table_1.18.2.1

loaded via a namespace (and not attached):
 [1] gtable_0.3.6           jsonlite_2.0.0         dplyr_1.2.1            compiler_4.5.3        
 [5] tidyselect_1.2.1       bspm_0.5.8             blob_1.3.0             directlabels_2025.6.24
 [9] scales_1.4.0           yaml_2.3.12            fastmap_1.2.0          lattice_0.22-9        
[13] R6_2.6.1               labeling_0.4.3         generics_0.1.4         knitr_1.51            
[17] tibble_3.3.1           polars_1.10.0          atime_2026.4.2         DBI_1.3.0             
[21] pillar_1.11.1          RColorBrewer_1.1-3     rlang_1.2.0            xfun_0.57             
[25] quadprog_1.5-8         S7_0.2.1-1             cli_3.6.6              withr_3.0.2           
[29] magrittr_2.0.5         digest_0.6.39          grid_4.5.3             hms_1.1.4             
[33] nc_2026.2.20           lifecycle_1.0.5        vctrs_0.7.3            bench_1.1.4           
[37] evaluate_1.0.5         glue_1.8.1             farver_2.1.2           duckdb_1.5.2          
[41] codetools_0.2-20       profmem_0.7.0          rmarkdown_2.31         tools_4.5.3           
[45] pkgconfig_2.0.3        htmltools_0.5.9

Happy birthday, data.table

tips

tutorials

developer

Use of non-API entry points in `data.table`

developer

guest post

performance

Advent of Code with `data.table`: Week One

tutorials

community

First translation of R vignettes

grant

documentation

translation

Visualizing performance regression of `data.table` with `atime`

performance

testing

developer

Seal of Approval: mlr3

seal of approval

application package

Seal of Approval: collapse

seal of approval

partner package

Newly awarded translation projects

announcements

grant

translation

Seal of Approval: dtplyr

seal of approval

bridge package

Seal of Approval: nc

seal of approval

extension package

Seal of Approval: tidyfast

seal of approval

bridge package

Announcement: The ‘Seal of Approval’

announcements

grant

community

seal of approval

Announcement: Paola Corrales, data.table Ambassador

announcements

grant

ambassadors

travel

Two Roads Diverged

opinion

Testing infrastructure for data.table

grant

testing

developer

Community interviews about data.table

community

grant

Results of the 2023 survey

community

guest post

governance

Column assignment and reference semantics in data.table

tips

tutorials

developer

The Benefits of `data.table` Syntax

tips

tutorials

documentation

New governance, release with new features

governance

releases

Piping data.tables

tips

tutorials

documentation

guest post

Announcement: Jan Gorecki, data.table Ambassador

announcements

grant

ambassadors

Summary of LatinR conference

conferences

Announcement: The data.table Ambassadors Travel Grant

announcements

grant

funding opportunity

Announcement: data.table translation projects

announcements

grant

funding opportunity

Welcome to the data.table ecosystem project!

announcements

grant

Visualizing performance regression of data.table with atime

Doris Afriyie Amoakohene — Thu, 10 Oct 2024 00:00:00 GMT

Since August 2023, I have been working on performance testing, which could be useful for expanding the open-source ecosystem around data.table package in R. This could increase confidence in code contributions by ensuring the sustained efficiency of the data.table package.

In data.table, the term “performance regression” refers to a change to the data.table source code, or to the core R build, that causes an increase in either time metrics and memory metrics.

It is important that we prevent significant performance regression from reaching the current release of the data.table package. Slowness or big memory usage can be frustrating; and in fact, are the issues data.table is most used to solve. Any performance regression that makes it into a version release will degrade user experience.

In this blog post, I will demonstrate the use of benchmarking techniques to verify whether reported issues on data.table have been successfully resolved.

Overview

Understanding performance in `data.table`

data.table is an extension of R’s data.frame, designed to handle large datasets efficiently. It provides a syntax that is both concise and expressive, allowing users to perform complex data manipulations with ease. Its efficiency is particularly evident when dealing with tasks like filtering, grouping, aggregating, and joining data.

The development team behind data.table is committed to continuously improving its performance. Over the years, several major version changes have been introduced, aiming to enhance speed and efficiency. These changes include algorithmic optimizations, memory management improvements, and enhancements to parallel processing capabilities. Upgrading to the latest version ensures that users can leverage the most recent performance enhancements.

Why do we run performance tests on GitHub commits?

Running performance tests on GitHub commits helps maintain a high-performance standard for the package, detect and fix performance regressions, optimize code, validate performance improvements, ensure consistent performance over time and to encourage confidence in code contributions from new people.

It is an essential practice to deliver a performant and reliable package to end-users.

Benchmarking for performance evaluation

To evaluate data.table performance, it is essential to employ benchmarking methodologies. The approach I used utilizes the atime_versions function from the atime package, which measures the actual execution time of specific operations. This function allows for accurate comparisons between different versions of the data.table package, by benchmarking against time and memory usage and giving a graphical visualization of the results.

Details of the performance tests

The primary function atime_versions has six main arguments:

pkg.path: This argument specifies the location on your system where you have stored a git clone of the data.table package.
pkg.edit.fun: The default behavior of pkg.edit.fun is designed to work with Rcpp packages and involves replacing instances of “PKG” with “PKG.SHA” in the package code. Any occurrences of the string “PKG” within the package code will be replaced with “PKG.SHA”, where “SHA” represents the commit SHA/ids associated with the version being installed.
N: This argument determines the number of iterations for the benchmarking process. It is a sequence of numbers that define different data sizes to test the performance of the operation.
setup: This section contains the setup code for generating the dataset used in the benchmarking process, the setup is determined by the value of N.
expr: This section contains the expression that represents the operation being benchmarked. It uses the data.table::[.data.table`` syntax to perform the operation on the dataset.

In the given syntax data.table::`[.data.table`, the first part data.table:: installs and loads different versions of the data.table package based on the specified commit ids. Hence, data.table:: will be translated to data.table.SHA1:: for some version hash SHA1. Following that, the expression specified within `[.data.table `` is executed on each installed version. This process is repeated for all the specified commit IDs in the code.

For example:

data.table.ec1259af1bf13fc0c96a1d3f9e84d55d8106a9a4:::`[.data.table`(DT, , .(v3=mean(v3, na.rm=TRUE)), by=id3, verbose=TRUE)

In this example, the expression [.data.table is executed on the DT dataset using the specified commit ID (ec1259af1bf13fc0c96a1d3f9e84d55d8106a9a4) of the data.table package. The expression calculates the mean of the v3 column (ignoring missing values) grouped by id3, and the verbose=TRUE argument enables verbose output during the operation. This process is typically repeated for all commit IDs in your code to compare the performance of different versions of the data.table package.

... : This specifies the different versions of the data.table packages that will be tested. It includes three versions: “Before,” “Regression,” and “Fixed.” Each version is associated with a specific commit id.

Test procedure

We run the full performance regression with atime:

Before the change causing performance regression is made (Before)
When the change causing performance regression is first submitted (Regression)
After the Pull Request (PR) which fixes the performance regression (Fixed)

Overall workflow

When a fixing Pull Request is submitted, our procedure automatically takes the following steps:

Pass the hashes for different branches (Before, Regression, Fix) to atime_versions; along with various parameters for the test (number of simulations, code expression to run, etc.).
Use the atime_versions function to measure time and memory usage across different versions.
Generate a plot to showcase the test results, using the atime package built in plotting functions.
Display the plot and test results as a comment on the submitted Pull Request.

Here is an example of how to perform the atime test. More documentation of the atime package can be found here.

Example

The first example we will show is an issue reported on performing group computations, specifically when running R’s C eval: link to GitHub Issue that reported regression. This regression was caused by the inclusion of the certain code within the #if block. This PR discusses the specific C code in q7 and q8 in the “db-benchmark” which causes the regression.

This PR fixed the regression problem.

The details of the code problems and solutions are not required for the example; we link them only to share a map of the regression-and-fix process.

To produce performance test results, we first load package dependencies, as well as the current GitHub snapshot of data.table in development:

library(atime)
library(ggplot2)
library(data.table)

tdir <- tempfile()
dir.create(tdir)
git2r::clone("https://github.com/Rdatatable/data.table", tdir )

Next, we establish our performance test. Here, we will create a data.table object and then compute the range by group. We vary the size of the object by varying values of N across tests.

d <- data.table(
      id3 = sample(c(seq.int(N*0.9), sample(N*0.9, N*0.1, TRUE))),
      v1 = sample(5L, N, TRUE),
      v2 = sample(5L, N, TRUE)
      )

data.table:::`[.data.table`(d, , (max(v1)-min(v2)), by = id3)

This setup and expression is then passed to atime_versions, along with a bit of package management information, and hashes (a.k.a. “Commit ID” or “SHA”) for the commits before, during, and after the performance regression.

atime.list.4200 <- atime::atime_versions(
  pkg.path = tdir,
  pkg.edit.fun = pkg.edit.fun,
  N = 10^seq(1,20),
  setup = { 
    set.seed(108)
    d <- data.table(
      id3 = sample(c(seq.int(N*0.9), sample(N*0.9, N*0.1, TRUE))),
      v1 = sample(5L, N, TRUE),
      v2 = sample(5L, N, TRUE))
  },
  expr = data.table:::`[.data.table`(d, , (max(v1)-min(v2)), by = id3),
  "Before" = "793f8545c363d222de18ac892bc7abb80154e724", # commit hash in PR prior to regression
  "Regression" = "c152ced0e5799acee1589910c69c1a2c6586b95d", # commit hash in PR causing regression
  "Fixed" = "f750448a2efcd258b3aba57136ee6a95ce56b302" # commit hash in PR that fixes the regression
)

Note

The function pkg.edit.fun that is passed to atime_versions above is a custom function written to manage the packages and paths on the server running this test.

You can see the code below if you wish.

Code

Seal of Approval: mlr3

Maximilian Mücke — Tue, 01 Oct 2024 00:00:00 GMT

`mlr3`

Author(s): Michel Lang, Bernd Bischl, Jakob Richter, Patrick Schratz, Martin Binder, Florian Pfisterer, Raphael Sonabend, Marc Becker, Sebastian Fischer

Maintainer: Marc Becker (marcbecker@posteo.de)

Seal of Approval

mlr3 hex sticker

A modern object-oriented machine learning framework. Successor of mlr.

Relationship with `data.table`

mlr3 was designed to integrate closely with data.table for efficient data handling in machine learning workflows. There are two main ways mlr3 is related to data.table:

Data Backend: mlr3 uses data.table as the core data backend for all Task objects. This means that when you work with tasks in mlr3, the underlying data is stored and managed using data.table. Moreover, users can leverage data.table syntax directly within mlr3 workflows. Accessing task data via task$data() returns a data.table, enabling you to apply data.table operations for data preprocessing, feature engineering, and subsetting without any additional conversion or overhead.
Result Storage: mlr3 stores various results such as predictions, resampling outcomes, and benchmarking results as data.table objects.

Overview

Excerpted from the mlr3 book

The mlr3 universe includes a wide range of tools taking you from basic ML to complex experiments. To get started, here is an example of the simplest functionality – training a model and making predictions.

library(mlr3)

task = tsk("penguins")
split = partition(task)
learner = lrn("classif.rpart")

learner$train(task, row_ids = split$train)
learner$model

n= 230 

node), split, n, loss, yval, (yprob)
      * denotes terminal node

1) root 230 130 Adelie (0.43478261 0.17826087 0.38695652)  
  2) flipper_length< 206 136  38 Adelie (0.72058824 0.27941176 0.00000000)  
    4) bill_length< 43.35 99   3 Adelie (0.96969697 0.03030303 0.00000000) *
    5) bill_length>=43.35 37   2 Chinstrap (0.05405405 0.94594595 0.00000000) *
  3) flipper_length>=206 94   5 Gentoo (0.02127660 0.03191489 0.94680851) *

prediction = learner$predict(task, row_ids = split$test)
prediction


──  for 114 observations: ───────────────────────────────────
 row_ids     truth  response
       7    Adelie    Adelie
       8    Adelie    Adelie
      10    Adelie    Adelie
     ---       ---       ---
     340 Chinstrap    Gentoo
     342 Chinstrap Chinstrap
     344 Chinstrap Chinstrap

prediction$score(msr("classif.acc"))

classif.acc 
  0.9298246

In this example, we trained a decision tree on a subset of the penguins dataset, made predictions on the rest of the data and then evaluated these with the accuracy measure.

The mlr3 interface also lets you run more complicated experiments in just a few lines of code:

library(mlr3verse)

tasks = tsks(c("german_credit", "sonar"))

glrn_rf_tuned = as_learner(ppl("robustify") %>>% auto_tuner(
    tnr("grid_search", resolution = 5),
    lrn("classif.ranger", num.trees = to_tune(200, 500)),
    rsmp("holdout")
))
glrn_rf_tuned$id = "RF"

glrn_stack = as_learner(ppl("robustify") %>>% ppl("stacking",
    lrns(c("classif.rpart", "classif.kknn")),
    lrn("classif.log_reg")
))
glrn_stack$id = "Stack"

learners = c(glrn_rf_tuned, glrn_stack)
bmr = benchmark(benchmark_grid(tasks, learners, rsmp("cv", folds = 3)))

bmr$aggregate(msr("classif.acc"))

      nr       task_id learner_id resampling_id iters classif.acc
                                
1:     1 german_credit         RF            cv     3   0.7650225
2:     2 german_credit      Stack            cv     3   0.7370005
3:     3         sonar         RF            cv     3   0.8122153
4:     4         sonar      Stack            cv     3   0.7307108
Hidden columns: resample_result

In this more complex example, we selected two tasks and two learners, used automated tuning to optimize the number of trees in the random forest learner, and employed a machine learning pipeline that imputes missing data, consolidates factor levels, and stacks models. We also showed basic features like loading learners and choosing resampling strategies for benchmarking. Finally, we compared the performance of the models using the mean accuracy with three-fold cross-validation.

Happy birthday, data.table

tips

tutorials

developer

Use of non-API entry points in `data.table`

developer

guest post

performance

Advent of Code with `data.table`: Week One

tutorials

community

First translation of R vignettes

grant

documentation

translation

Comparing `data.table` reshape to `duckdb` and `polars`

tips

tutorials

developer

benchmarks

Visualizing performance regression of `data.table` with `atime`

performance

testing

developer

Seal of Approval: collapse

seal of approval

partner package

Newly awarded translation projects

announcements

grant

translation

Seal of Approval: dtplyr

seal of approval

bridge package

Seal of Approval: nc

seal of approval

extension package

Seal of Approval: tidyfast

seal of approval

bridge package

Announcement: The ‘Seal of Approval’

announcements

grant

community

seal of approval

Announcement: Paola Corrales, data.table Ambassador

announcements

grant

ambassadors

travel

Two Roads Diverged

opinion

Testing infrastructure for data.table

grant

testing

developer

Community interviews about data.table

community

grant

Results of the 2023 survey

community

guest post

governance

Column assignment and reference semantics in data.table

tips

tutorials

developer

The Benefits of `data.table` Syntax

tips

tutorials

documentation

New governance, release with new features

governance

releases

Piping data.tables

tips

tutorials

documentation

guest post

Announcement: Jan Gorecki, data.table Ambassador

announcements

grant

ambassadors

Summary of LatinR conference

conferences

Announcement: The data.table Ambassadors Travel Grant

announcements

grant

funding opportunity

Announcement: data.table translation projects

announcements

grant

funding opportunity

Welcome to the data.table ecosystem project!

announcements

grant

Seal of Approval: collapse

Sebastian Krantz — Sat, 21 Sep 2024 00:00:00 GMT

`collapse`

Author(s): Sebastian Krantz

Maintainer: Sebastian Krantz (sebastian.krantz@graduateinstitute.ch)

collapse hex sticker

collapse is a large C/C++-based infrastructure package facilitating complex statistical computing, data transformation, and exploration tasks in R - at outstanding levels of performance and memory efficiency. It also implements a class-agnostic approach to R programming supporting vector, matrix and data frame-like objects (including xts, tibble, data.table, and sf). It has a stable API, depends on Rcpp, and supports R versions >= 3.4.0.

Relationship with `data.table`

At the C-level, collapse took much inspiration from data.table, and leverages some of its core algorithms like radixsort, while adding significant statistical functionality and new algorithms within a class-agnostic programming framework that seamlessly supports data.table. Notably, collapse::qDT() is a highly efficient anything to data.table converter, and all manipulation functions in collapse return a valid data.table object when a data.table is passed, enabling subsequent reference operations (:=).

Its added functionality includes a rich set of Fast Statistical Functions supporting vectorized (grouped, weighted) statistical operations on matrix-like objects. These are integrated with fast data manipulation functions in a way that also more complex statistical expressions can be vectorized across groups. It also adds flexible time series functions and classes supporting irregular series and panels, (panel-)data transformations, vectorized hash-joins, fast aggregation and recast pivots, (internal) support for variable labels, powerful descriptive tools, memory efficient programming tools, and recursive tools for heterogeneous nested data.

It is highly and interactively configurable. A navigable internal documentation/overview facilitates its use.

Overview

The easiest way to load collapse and data.table together is via the fastverse package:

library(fastverse)

-- Attaching packages ------------------------------------------------------------------------------- fastverse 0.3.4 --

v data.table 1.18.2.1     v kit        0.0.21  
v magrittr   2.0.5        v collapse   2.1.6

This demonstrates collapse’s deep integration with data.table.

mtcarsDT <- qDT(mtcars)                # This creates a valid data.table (no deep copy)
mtcarsDT[, new := mean(mpg), by = cyl] # Proof: no warning here

There are many reasons to use collapse, e.g., to compute advanced statistics very fast:

# Fast tidyverse-like functions: one of the ways to code with collapse
mtcDTagg <- mtcarsDT |> 
  fgroup_by(cyl, vs, am) |> 
  fsummarise(mpg_wtd_median = fmedian(mpg, wt),             # Weighted median
             mpg_wtd_p90 = fnth(mpg, 0.9, wt, ties = "q8"), # Weighted 90% quantile type 8
             mpg_wtd_mode = fmode(mpg, wt, ties = "max"),   # Weighted maximum mode 
             mpg_range = fmax(mpg) %-=% fmin(mpg),          # Range: vectorized and memory efficient   
             lm_mpg_carb = fsum(mpg, W(carb)) %/=% fsum(W(carb)^2)) # coef(lm(mpg ~ carb)): vectorized
# Note: for increased parsimony, can abbreviate fgroup_by -> gby, fsummarise -> smr
mtcDTagg[, new2 := 1][1:3] # Still a data.table

     cyl    vs    am mpg_wtd_median mpg_wtd_p90 mpg_wtd_mode mpg_range lm_mpg_carb  new2
                                           
1:     4     0     1           26.0     26.0000         26.0       0.0         NaN     1
2:     4     1     0           22.8     24.4000         24.4       2.9         2.1     1
3:     4     1     1           30.4     33.6894         30.4      12.5        -1.7     1

Or simply, convenience functions like collap() for fast multi-type aggregation:

# World Development Dataset (see ?wlddev)
head(wlddev, 3)

      country iso3c       date year decade     region     income  OECD PCGDP LIFEEX GINI       ODA     POP
1 Afghanistan   AFG 1961-01-01 1960   1960 South Asia Low income FALSE    NA 32.446   NA 116769997 8996973
2 Afghanistan   AFG 1962-01-01 1961   1960 South Asia Low income FALSE    NA 32.962   NA 232080002 9169410
3 Afghanistan   AFG 1963-01-01 1962   1960 South Asia Low income FALSE    NA 33.471   NA 112839996 9351441

# Population weighted mean for numeric and mode for non-numeric columns (multithreaded and 
# vectorized across groups and columns, the default in statistical functions is na.rm = TRUE)
wlddev |> collap(~ year + income, fmean, fmode, w = ~ POP, nthreads = 4) |> ss(1:3)

        country iso3c       date year decade                region              income  OECD      PCGDP   LIFEEX GINI
1 United States   USA 1961-01-01 1960   1960 Europe & Central Asia         High income  TRUE 12768.7126 68.59372   NA
2      Ethiopia   ETH 1961-01-01 1960   1960    Sub-Saharan Africa          Low income FALSE   658.4778 38.33382   NA
3         India   IND 1961-01-01 1960   1960            South Asia Lower middle income FALSE   500.7932 45.26707   NA
         ODA       POP
1  911825661 749495030
2  160457982 147355735
3 3278899549 927990163

We can also use the low-level API for statistical programming:

# Grouped mean
fmean(mtcars$mpg, mtcars$g)

       3        4        5 
16.10667 24.53333 21.38000

# Grouping object from multiple columns
g <- GRP(mtcars, c("cyl", "vs", "am"))
fmean(mtcars$mpg, g)

   4.0.1    4.1.0    4.1.1    6.0.1    6.1.0    8.0.0    8.0.1 
26.00000 22.90000 28.37143 20.56667 19.12500 15.05000 15.40000

vars <- c("carb", "hp", "qsec") # columns to aggregate
# Aggregating: weighted mean - vectorized across groups and columns 
add_vars(g$groups, # Grouping columns
  fmean(get_vars(mtcars, vars), g, 
        w = mtcars$wt, use.g.names = FALSE)
)

  cyl vs am     carb        hp     qsec
1   4  0  1 2.000000  91.00000 16.70000
2   4  1  0 1.720045  83.60420 21.04028
3   4  1  1 1.416115  82.11819 18.75509
4   6  0  1 4.670296 131.78463 16.33306
5   6  1  0 2.522685 115.32202 19.21275
6   8  0  0 3.186582 196.74988 17.20449
7   8  0  1 6.118694 301.60682 14.55297

# Let's aggregate a matrix 
m <- matrix(abs(rnorm(32^2)), 32)
m |> fmean(g) |> t() |> fmean(g) |> t()

          4.0.1     4.1.0     4.1.1     6.0.1     6.1.0     8.0.0     8.0.1
4.0.1 1.1567383 0.4712974 0.7291093 0.4188384 0.7668799 0.5462991 0.4285371
4.1.0 1.6510326 0.3433476 0.8675363 0.8106850 1.1305622 0.8240946 0.9043461
4.1.1 0.9855701 0.9180443 0.8813610 0.7026620 0.5591381 0.8175456 1.0759388
6.0.1 1.0916912 0.5785965 0.5478041 0.6592419 0.9330974 0.8968921 1.2222674
6.1.0 0.7910620 0.5924379 0.8199174 0.8940494 0.8194461 0.7782545 0.6762188
8.0.0 0.9334271 0.8254035 0.7394789 0.8520958 0.6007892 0.7498614 0.5767340
8.0.1 0.5174626 1.1922535 0.6787865 0.7702279 0.9552730 0.7862647 0.6655360

# Normalizing the columns, by reference
fsum(m, TRA = "/", set = TRUE)
fsum(m) # Check

 [1] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

# Multiply the rows with a vector (by reference)
setop(m, "*", mtcars$mpg, rowwise = TRUE)
# Replace some elements with a number
setv(m, 3:40, 5.76) # Could also use a vector to copy from
whichv(m, 5.76) # get the indices back...

 [1]  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40

It is also fairly easy to do more involved data exploration and manipulation:

# Groningen Growth and Development Center 10 Sector Database (see ?GGDC10S)
namlab(GGDC10S, N = TRUE, Ndistinct = TRUE, class = TRUE)

     Variable     Class    N Ndist                                                 Label
1     Country character 5027    43                                               Country
2  Regioncode character 5027     6                                           Region code
3      Region character 5027     6                                                Region
4    Variable character 5027     2                                              Variable
5        Year   numeric 5027    67                                                  Year
6         AGR   numeric 4364  4353                                          Agriculture 
7         MIN   numeric 4355  4224                                                Mining
8         MAN   numeric 4355  4353                                         Manufacturing
9          PU   numeric 4354  4237                                             Utilities
10        CON   numeric 4355  4339                                          Construction
11        WRT   numeric 4355  4344                         Trade, restaurants and hotels
12        TRA   numeric 4355  4334                  Transport, storage and communication
13       FIRE   numeric 4355  4349 Finance, insurance, real estate and business services
14        GOV   numeric 3482  3470                                   Government services
15        OTH   numeric 4248  4238               Community, social and personal services
16        SUM   numeric 4364  4364                               Summation of sector GDP

# Describe total Employment and Value-Added
descr(GGDC10S, SUM ~ Variable)

Dataset: GGDC10S, 1 Variables, N = 5027
Grouped by: Variable [2]
        N   Perc
EMP  2516  50.05
VA   2511  49.95
------------------------------------------------------------------------------------------------------------------------
SUM (numeric): Summation of sector GDP
Statistics (N = 4364, 13.19% NAs)
        N   Perc  Ndist         Mean          SD     Min             Max   Skew    Kurt
EMP  2225  50.99   2225     36846.87    96318.65  173.88          764200   5.02   30.98
VA   2139  49.01   2139  43'961639.1  358'350627       0  8.06794210e+09  15.77  289.46

Quantiles
         1%      5%      10%      25%        50%          75%          90%         95%         99%
EMP  256.12  599.38  1599.27  3555.62    9593.98      24801.5     66975.01   152402.28    550909.6
VA        0   25.01   444.54    21302  243186.47  1'396139.11  15'926968.3  104'405351  692'993893
------------------------------------------------------------------------------------------------------------------------

# Compute growth rate (Employment and VA, all sectors)
GGDC10S_growth <- tfmv(GGDC10S, AGR:SUM, fgrowth, # tfmv = transform variables. Alternatively: fmutate(across(...))
                       g = list(Country, Variable), t = Year, # Internal grouping and ordering, passed to fgrowth()
                       apply = FALSE) # apply = FALSE ensures we call fgrowth.data.frame

# Recast the dataset, median growth rate across years, taking along variable labels 
GGDC_med_growth <- pivot(GGDC10S_growth,
  ids = c("Country", "Regioncode", "Region"),
  values = slt(GGDC10S, AGR:SUM, return = "names"), # slt = shorthand for fselect()
  names = list(from = "Variable", to = "Sectorcode"),
  labels = list(to = "Sector"), 
  FUN = fmedian,  # Fast function = vectorized
  how = "recast"  # Recast (transposition) method
) |> qDT()
GGDC_med_growth[1:3]

   Country Regioncode             Region Sectorcode       Sector        VA       EMP
                                            
1:     BWA        SSA Sub-saharan Africa        AGR Agriculture   8.790267 0.8921475
2:     ETH        SSA Sub-saharan Africa        AGR Agriculture   6.664964 2.5876142
3:     GHA        SSA Sub-saharan Africa        AGR Agriculture  28.215905 1.4045550

# Finally, lets just join this to wlddev, enabling multiple matches (cartesian product)
# -> on average 61 years x 11 sectors = 671 records per unique (country) match
join(wlddev, GGDC_med_growth, on = c("iso3c" = "Country"), 
     how = "inner", multiple = TRUE) |> ss(1:3)

inner join: wlddev[iso3c] 2379/13176 (18.1%) <61:11> GGDC_med_growth[Country] 429/473 (90.7%)

    country iso3c       date year decade                    region              income  OECD    PCGDP LIFEEX GINI
1 Argentina   ARG 1961-01-01 1960   1960 Latin America & Caribbean Upper middle income FALSE 5642.765 65.055   NA
2 Argentina   ARG 1961-01-01 1960   1960 Latin America & Caribbean Upper middle income FALSE 5642.765 65.055   NA
3 Argentina   ARG 1961-01-01 1960   1960 Latin America & Caribbean Upper middle income FALSE 5642.765 65.055   NA
        ODA      POP Regioncode        Region Sectorcode        Sector       VA        EMP
1 219809998 20481779        LAM Latin America        AGR  Agriculture  32.91968 -0.8646301
2 219809998 20481779        LAM Latin America        MIN        Mining 25.72799  1.5627293
3 219809998 20481779        LAM Latin America        MAN Manufacturing 26.66754  1.0801500

In summary: collapse provides flexible high-performance statistical and data manipulation tools, which extend and seamlessly integrate with data.table. The package follows a similar development philosophy emphasizing API stability, parsimonious syntax, and zero dependencies (apart from Rcpp). data.table users may wish to employ collapse for some of the advanced statistical and manipulation functionality showcased above, but also to efficiently manipulate other data frame-like objects, such as sf data frames.

Happy birthday, data.table

tips

tutorials

developer

Use of non-API entry points in `data.table`

developer

guest post

performance

Advent of Code with `data.table`: Week One

tutorials

community

First translation of R vignettes

grant

documentation

translation

Comparing `data.table` reshape to `duckdb` and `polars`

tips

tutorials

developer

benchmarks

Visualizing performance regression of `data.table` with `atime`

performance

testing

developer

Seal of Approval: mlr3

seal of approval

application package

Newly awarded translation projects

announcements

grant

translation

Seal of Approval: dtplyr

seal of approval

bridge package

Seal of Approval: nc

seal of approval

extension package

Seal of Approval: tidyfast

seal of approval

bridge package

Announcement: The ‘Seal of Approval’

announcements

grant

community

seal of approval

Announcement: Paola Corrales, data.table Ambassador

announcements

grant

ambassadors

travel

Two Roads Diverged

opinion

Testing infrastructure for data.table

grant

testing

developer

Community interviews about data.table

community

grant

Results of the 2023 survey

community

guest post

governance

Column assignment and reference semantics in data.table

tips

tutorials

developer

The Benefits of `data.table` Syntax

tips

tutorials

documentation

New governance, release with new features

governance

releases

Piping data.tables

tips

tutorials

documentation

guest post

Announcement: Jan Gorecki, data.table Ambassador

announcements

grant

ambassadors

Summary of LatinR conference

conferences

Announcement: The data.table Ambassadors Travel Grant

announcements

grant

funding opportunity

Announcement: data.table translation projects

announcements

grant

funding opportunity

Welcome to the data.table ecosystem project!

announcements

grant

Newly awarded translation projects

Toby Dylan Hocking — Tue, 20 Aug 2024 00:00:00 GMT

French translation project

We are pleased to fund a French translation project, led by Philippe Grosjean, who is also the leader of the base R French translation. Co-authors include Christian Wia, Vincent Rocher, Vincent Runge, and Elise Maigné. The project has been awarded US$2000 to create and maintain the following translations:

all messages from the R-data.table.pot and data.table.pot files (approximately 1300 errors, warnings, etc).
all 12 vignettes.
the cheatsheet, which was recently updated!

Portuguese translation project

We are please to fund a Portuguese translation project, led by Leonardo Ferreira Fontenelle. Co-authors include Rafael Ferreira Fontenelle and Italo de Oliveira Santos. The aim is to provide a pt_BR translation which would make data.table more accessible to the 200 million people in Brazil. This team has been awarded US$2000 to create and maintain translations of the messages (errors, warnings, etc).

Happy birthday, data.table

tips

tutorials

developer

Use of non-API entry points in `data.table`

developer

guest post

performance

Advent of Code with `data.table`: Week One

tutorials

community

First translation of R vignettes

grant

documentation

translation

Comparing `data.table` reshape to `duckdb` and `polars`

tips

tutorials

developer

benchmarks

Visualizing performance regression of `data.table` with `atime`

performance

testing

developer

Seal of Approval: mlr3

seal of approval

application package

Seal of Approval: collapse

seal of approval

partner package

Seal of Approval: dtplyr

seal of approval

bridge package

Seal of Approval: nc

seal of approval

extension package

Seal of Approval: tidyfast

seal of approval

bridge package

Announcement: The ‘Seal of Approval’

announcements

grant

community

seal of approval

Announcement: Paola Corrales, data.table Ambassador

announcements

grant

ambassadors

travel

Two Roads Diverged

opinion

Testing infrastructure for data.table

grant

testing

developer

Community interviews about data.table

community

grant

Results of the 2023 survey

community

guest post

governance

Column assignment and reference semantics in data.table

tips

tutorials

developer

The Benefits of `data.table` Syntax

tips

tutorials

documentation

New governance, release with new features

governance

releases

Piping data.tables

tips

tutorials

documentation

guest post

Announcement: Jan Gorecki, data.table Ambassador

announcements

grant

ambassadors

Summary of LatinR conference

conferences

Announcement: The data.table Ambassadors Travel Grant

announcements

grant

funding opportunity

Announcement: data.table translation projects

announcements

grant

funding opportunity

Welcome to the data.table ecosystem project!

announcements

grant

Seal of Approval: dtplyr

Kelly Bodwin — Thu, 01 Aug 2024 00:00:00 GMT

`dtplyr`

dtplyr hex sticker

Author(s): Hadley Wickham, Maximilian Girlich, Mark Fairbanks, Ryan Dickerson, Posit Software PBC

Maintainer: Hadley Wickham (hadley@posit.co)

Provides a data.table backend for dplyr. The goal of dtplyr is to allow you to write dplyr code that is automatically translated to the equivalent, but usually much faster, data.table code.

Relationship with `data.table`

dtplyr is a bridge for users who are more comfortable with the dplyr syntax, but who want to take advantage of the speed and efficiency benefits of data.table. This package exactly duplicates the core functions of dplyr, but replaces the back-end source code (originally in Base R) with data.table operations.

Overview

Excerpted from the dtplyr vignette

To use dtplyr, you must at least load dtplyr and dplyr. You may also want to load data.table so you can access the other goodies that it provides:

library(data.table)
library(dtplyr)
library(dplyr, warn.conflicts = FALSE)

Then use lazy_dt() to create a “lazy” data.table object that tracks the operations performed on it.

mtcars2 <- lazy_dt(mtcars)

You can preview the transformation (including the generated data.table code) by printing the result:

mtcars2 %>% 
  filter(wt < 5) %>% 
  mutate(l100k = 235.21 / mpg) %>% # liters / 100 km
  group_by(cyl) %>% 
  summarise(l100k = mean(l100k))

Source: local data table [3 x 2]
Call:   `_DT1`[wt < 5][, `:=`(l100k = 235.21/mpg)][, .(l100k = mean(l100k)), 
    keyby = .(cyl)]

    cyl l100k
   
1     4  9.05
2     6 12.0 
3     8 14.9 

# Use as.data.table()/as.data.frame()/as_tibble() to access results

But generally you should reserve this only for debugging, and use as.data.table(), as.data.frame(), or as_tibble() to indicate that you’re done with the transformation and want to access the results:

mtcars2 %>% 
  filter(wt < 5) %>% 
  mutate(l100k = 235.21 / mpg) %>% # liters / 100 km
  group_by(cyl) %>% 
  summarise(l100k = mean(l100k)) %>% 
  as_tibble()

# A tibble: 3 × 2
    cyl l100k
   
1     4  9.05
2     6 12.0 
3     8 14.9

Happy birthday, data.table

tips

tutorials

developer

Use of non-API entry points in `data.table`

developer

guest post

performance

Advent of Code with `data.table`: Week One

tutorials

community

First translation of R vignettes

grant

documentation

translation

Comparing `data.table` reshape to `duckdb` and `polars`

tips

tutorials

developer

benchmarks

Visualizing performance regression of `data.table` with `atime`

performance

testing

developer

Seal of Approval: mlr3

seal of approval

application package

Seal of Approval: collapse

seal of approval

partner package

Newly awarded translation projects

announcements

grant

translation

Seal of Approval: nc

seal of approval

extension package

Seal of Approval: tidyfast

seal of approval

bridge package

Announcement: The ‘Seal of Approval’

announcements

grant

community

seal of approval

Announcement: Paola Corrales, data.table Ambassador

announcements

grant

ambassadors

travel

Two Roads Diverged

opinion

Testing infrastructure for data.table

grant

testing

developer

Community interviews about data.table

community

grant

Results of the 2023 survey

community

guest post

governance

Column assignment and reference semantics in data.table

tips

tutorials

developer

The Benefits of `data.table` Syntax

tips

tutorials

documentation

New governance, release with new features

governance

releases

Piping data.tables

tips

tutorials

documentation

guest post

Announcement: Jan Gorecki, data.table Ambassador

announcements

grant

ambassadors

Summary of LatinR conference

conferences

Announcement: The data.table Ambassadors Travel Grant

announcements

grant

funding opportunity

Announcement: data.table translation projects

announcements

grant

funding opportunity

Welcome to the data.table ecosystem project!

announcements

grant

Seal of Approval: nc

Toby Dylan Hocking — Thu, 01 Aug 2024 00:00:00 GMT

`nc`

nc hex sticker

Maintainer: Toby Dylan Hocking (toby.hocking@r-project.org)

User-friendly functions for extracting a data table (row for each match, column for each group) from non-tabular text data using regular expressions, and for melting columns that match a regular expression. Patterns are defined using a readable syntax that makes it easy to build complex patterns in terms of simpler, re-usable sub-patterns. Named R arguments are translated to column names in the output, thereby providing a standard interface to three regular expression ‘C’ libraries (‘PCRE’, ‘RE2’, ‘ICU’). Output can also include numeric columns via user-specified type conversion functions.

Relationship with `data.table`

Whereas data.table provides several functions such as patterns() and measure() which support some regex engines (PCRE, TRE), nc interfaces with two other engines (RE2, ICU). nc imports data.table, and always returns regex match results as a data.table.

Overview

nc is useful for extracting numeric data from text, for example consider the following strings, which indicate genomic positions, in bases on a chromosome:

chr.pos.vec <- c(
  "chr10:213,054,000-213,055,000",
  "chrM:111,000",              # no end.
  "chr1:110-111 chr2:220-222") # two ranges.

The data above consist of a chromosome name (chr10), followed by a start position, and then optionally a dash and an end position. Using nc, we can extract these different pieces of information into a data table using the code below, which inputs the data to parse (first argument), along with a regular expression (subsequent arguments).

nc::capture_first_vec(
  chr.pos.vec,
  chrom="chr.*?",
  ":",
  start="[0-9,]+")

    chrom       start
         
1:  chr10 213,054,000
2:   chrM     111,000
3:   chr1         110

The code above uses chrom and start as argument names, which are therefore used for column names in the output data table (one row per input subject string, one column per named argument / capture group). However the code above only parses the start position (and not the optional end position). Below, we create a more complex regex to parse both the start and end, by first defining a common pattern to parse an integer,

keep.digits <- function(x) as.integer(gsub("[^0-9]", "", x))

int.pattern <- list("[0-9,]+", keep.digits)

In the code above, we use a list to group the regex "[0-9],]+" with the function keep.digits which will be used for parsing the text that is extracted by that regex. We use that pattern twice in the code below,

range.pattern <- list(
  chrom="chr.*?",
  ":",
  start=int.pattern,
  list( # un-named list becomes non-capturing group.
    "-",
    end=int.pattern
  ), "?") # chromEnd is optional.
nc::capture_first_vec(chr.pos.vec, range.pattern)

    chrom     start       end
             
1:  chr10 213054000 213055000
2:   chrM    111000        NA
3:   chr1       110       111

The result above is a data table containing the first match in each subject (three rows total). Note the second row has end=NA because that optional group did not match.

But the last subject has two potential matches (only the first is reported above). What if we wanted to get all matches in each subject? We can use another function, as in the code below.

nc::capture_all_str(chr.pos.vec, range.pattern)

    chrom     start       end
             
1:  chr10 213054000 213055000
2:   chrM    111000        NA
3:   chr1       110       111
4:   chr2       220       222

The output above includes all matches in each subject (four rows total), but does not include any information about which subject each row came from, because it treats the subject as a single string to parse. To get that info, we can use capture_all_str() for each row, using by=.I as in the code below.

library(data.table)
data.table(chr.pos.vec)[, nc::capture_all_str(
  chr.pos.vec, range.pattern), by=.I]

       I  chrom     start       end
              
1:     1  chr10 213054000 213055000
2:     2   chrM    111000        NA
3:     3   chr1       110       111
4:     3   chr2       220       222

The output above includes the additional I column which is the index of the subject that each match came from (two rows with I=3 because there are two matches in the third subject).

Finally, data.table::melt() is used to power the long-to-wide data reshaping functionality in nc. In data.table we could use measure() to specify a set of variables to reshape, as in the code below.

(iris.wide <- data.table(iris)[1])

   Sepal.Length Sepal.Width Petal.Length Petal.Width Species
                                  
1:          5.1         3.5          1.4         0.2  setosa

melt(iris.wide, measure.vars=measure(value.name, dim, pattern="(.*)[.](.*)"))

   Species    dim Sepal Petal
       
1:  setosa Length   5.1   1.4
2:  setosa  Width   3.5   0.2

The result above has reshaped the four numeric input columns into two numeric output columns (value.name is the sentinel/keyword indicating that we want to make a new column for each unique value captured in that group). The equivalent nc code would be as below, with the regex defined using a named argument for each capture group (instead of one long pattern string with parentheses for each capture group).

nc::capture_melt_multiple(
  iris.wide,
  column=".*",
  "[.]",
  dim=".*")

   Species    dim Petal Sepal
       
1:  setosa Length   1.4   5.1
2:  setosa  Width   0.2   3.5

The nc code above produces the same result, and in fact uses data.table::melt() internally.

For more info about the nc package, please read the vignettes on its CRAN page.

Happy birthday, data.table

tips

tutorials

developer

Use of non-API entry points in `data.table`

developer

guest post

performance

Advent of Code with `data.table`: Week One

tutorials

community

First translation of R vignettes

grant

documentation

translation

Comparing `data.table` reshape to `duckdb` and `polars`

tips

tutorials

developer

benchmarks

Visualizing performance regression of `data.table` with `atime`

performance

testing

developer

Seal of Approval: mlr3

seal of approval

application package

Seal of Approval: collapse

seal of approval

partner package

Newly awarded translation projects

announcements

grant

translation

Seal of Approval: dtplyr

seal of approval

bridge package

Seal of Approval: tidyfast

seal of approval

bridge package

Announcement: The ‘Seal of Approval’

announcements

grant

community

seal of approval

Announcement: Paola Corrales, data.table Ambassador

announcements

grant

ambassadors

travel

Two Roads Diverged

opinion

Testing infrastructure for data.table

grant

testing

developer

Community interviews about data.table

community

grant

Results of the 2023 survey

community

guest post

governance

Column assignment and reference semantics in data.table

tips

tutorials

developer

The Benefits of `data.table` Syntax

tips

tutorials

documentation

New governance, release with new features

governance

releases

Piping data.tables

tips

tutorials

documentation

guest post

Announcement: Jan Gorecki, data.table Ambassador

announcements

grant

ambassadors

Summary of LatinR conference

conferences

Announcement: The data.table Ambassadors Travel Grant

announcements

grant

funding opportunity

Announcement: data.table translation projects

announcements

grant

funding opportunity

Welcome to the data.table ecosystem project!

announcements

grant