DataFrames in a Box

Use a DataFrame when your data has columns you want to select, filter or transform. This page shows how to work with those columns in a box and how to keep their types across separate runs. For a first look at values and plots in Field, start with Looking at Data.

Field boxes can use Kotlin DataFrame with its compiler plugin switched on, so a frame's columns are typed members you can name directly:

:::static
import org.jetbrains.kotlinx.dataframe.api.*

val people = dataFrameOf("name", "age")(
    "Alice",
    15,
    "Bob",
    30,
    "Cara",
    45,
)

people.age // resolves. Not people["age"], not a cast, not a generated file.

You can write people.age directly. The plugin reads the dataFrameOf("name", "age")(...) call at compile time, works out the schema it produces, and gives the expression a type whose members are its columns. There is no @DataSchema in your project, no build step, and nothing on disk to keep in sync.

The typing does not survive from one execution to the next on its own; typed(), below, carries it across.

The shape that works: derive and consume in one execution

Within one execution, a chain of calls just works. Every step re-types the frame, so the next step can name the columns the previous one produced:

:::static
import org.jetbrains.kotlinx.dataframe.api.*

val people = dataFrameOf("name", "age")(
    "Alice",
    15,
    "Bob",
    30,
    "Cara",
    45,
)

// `age` inside the lambda resolves against the *input* schema.
// `decade` afterwards resolves against the schema `add` RETURNS.
val adults = people.filter { age >= 18 }.add("decade") { age / 10 }

println("names:  " + people.name.toList())
println("kept:   " + adults.name.toList())
println("decade: " + adults.decade.toList())

decade is not in the source data and is not declared anywhere. The plugin reads the add("decade") { … } call and adds the column to the frame's type.

Where the boundary falls

Typed columns do not survive a compile boundary. Declare val df = dataFrameOf("a","b")(1,2) in one snippet, read df.a in the next, and the frame comes back as DataFrame<Any> with an empty compile-time schema. In live coding you load data once and then work with it over many runs, so you need typed().

A @DataSchema interface shared between boxes with sharedTypes(...) does not help either: you can cast the frame to that type, but people.age still does not resolve. Use typed() in the box that needs the columns.

typed(): the way across

DataFrame<T>.typed(name) (DataFrameSchemas.kt:50) reads the frame's columns at run time, when the data has already been loaded, and writes a schema for them into the calling box's preamble:

:::static
val people =
    dataFrameOf("name", "age")(
            "Alice",
            15,
            "Bob",
            30,
            "Cara",
            45,
        )
        .typed("People")

/*
…and this appears in the box's preamble:
@org.jetbrains.kotlinx.dataframe.annotations.DataSchema
interface People { val `name`: kotlin.String; val `age`: kotlin.Int }
*/

From then on People is an ordinary source declaration in that box, so people.cast<People>().age resolves in every later execution.

Calling typed() on every run is cheap and safe. The preamble only re-evaluates lines that changed, so a re-run with unchanged columns costs nothing; a re-run with a changed schema rewrites the old declaration instead of accumulating a second one. Unlike sharedTypes, nothing is loaded into a classloader, so changing the columns and re-running simply redeclares the interface.

Flat value columns get their real element type. A column group or frame column would need a nested interface, so it is declared Any?: you still get the column, but not its shape. The log names which ones landed that way (DataFrameSchemas.kt:67). Calling typed() with no box in context logs a warning and does nothing (DataFrameSchemas.kt:58).

One execution, two halves: the //%- marker

typed() writes to the preamble, so it looks as though you have to run the box twice. You do not, if you split the box with a //%- marker. A typed() call in the first half has already published its schema by the time the second half is compiled:

:::static
import org.jetbrains.kotlinx.dataframe.api.*

val people =
    dataFrameOf("name", "age")(
            "Alice",
            15,
            "Bob",
            30,
            "Cara",
            45,
        )
        .typed("Peach")

// Run the first fragment to publish Peach into the preamble before compiling the cast.
//%-
val p = people.cast<Peach>()
println("ages:   " + p.age.toList())
println("adults: " + p.filter { age >= 18 }.name.toList())

Delete the //%- and the box stops compiling: with a single fragment, the schema is written only after that fragment has been compiled, so Peach does not exist yet.

Once it has run, the second half is independently re-runnable: select those lines, run them as often as you like, and the frame is never reloaded.

Looking at a frame: .h and .h2

A DataFrame renders through the value visualizer like any other value. Field registers it with ValuePrinterRegistry and H2ValueRegistry (DataFrameVisualizer.kt:62, DataFrameVisualizer.kt:69).

.h works on a frame, a column, a row and a GroupBy:

:::static
import org.jetbrains.kotlinx.dataframe.api.*

val people = dataFrameOf("name", "city", "age")(
    "Alice",
    "Berlin",
    15,
    "Bob",
    "Berlin",
    30,
    "Cara",
    "Lisbon",
    45,
    "Dev",
    "Lisbon",
    22,
)

display("frame", people.h) // a Table, plus a shape line
display("column", people.columns()[2].h) // one column, plus its element type
display("row", people[0].h) // label/value facts
display("grouped", people.groupBy("city").h) // how many groups, keyed by what

All four are drawn with the same table() and facts() that draw every other table in Field.

The compact form costs the same at any size

The compact .h form shows the first 10 rows (DataFrameVisualizer.kt:51) and says how many it is hiding. It reads them cell by cell, by index (DataFrameVisualizer.kt:119), so printing a very large frame costs what printing ten rows costs, and you can leave a large frame as a box's last expression.

:::static
import org.jetbrains.kotlinx.dataframe.api.*

// 200,000 rows. A spread argument is one the plugin cannot statically interpret,
// so this frame is schema-less — the point here is the RENDERING cost.
val big = dataFrameOf("i")(*Array(200_000) { it })

val compact = big.h
display("head", compact)
display(
    "cost",
    ("the compact form of a " + big.rowsCount() + "-row frame is " + compact.length + " characters").h,
)

.h2 scrolls the frame itself

.h2 opens the frame in the h2 explorer, which you can scroll through the whole frame. Field keeps the frame itself on the server and builds only the rows you are looking at (DataFrameVisualizer.kt:154), so a large frame scrolls as quickly as a small one.

:::static
import org.jetbrains.kotlinx.dataframe.api.*

/*
15,000 rows, and nothing below pages them: `.h2` hands the whole FRAME to the
explorer, which asks for the window it is showing and for nothing else. A spread
argument is one the plugin cannot statically interpret, so this frame is
schema-less — what is on show here is the rendering, not the typing.
*/
val n = 15_000
val big = dataFrameOf("i", "sq")(
    *Array<Any?>(n * 2) { k ->
        val r = k / 2
        if (k % 2 == 0) r else r.toLong() * r
    },
)

display("scroller", big.h2)

The header gives the frame's full shape, 15000 rows × 2 columns. The rows are shown as a table with one column per frame column, in the frame's order (h2explorer.kt:261).

A frame inside another value, such as a list of frames or a data class with a frame field, expands the same way.

Switches, and the version pin

The DataFrame runtime is pinned to 1.0.0-Beta4, the version the plugin was built against.

Where to go next

• dataframe_typed_columns in the Agent Transcript Gallery: a four-box example you can replay.
• Visualizers, Displays, and Little Instruments: everything else .h and .h2 can draw.
• Charts as FLines: DataFrame.chart() plots a frame.
• Kotlin in the Head and Running code in a box: snippets, fragments and preambles.

Known limits

• A frame's inferred type does not carry from one execution to the next. Use typed().
• A @DataSchema shared with sharedTypes gives you the type but no column accessors. Use typed() in the box that needs the columns.
• Column groups and frame columns come back from typed() as Any?.
• The plugin uses a compiler extension point that the Kotlin compiler documents as unstable, so a Kotlin upgrade can break it.