DataFrame: Kotlin's Innovative Approach to Data Structures | Roman Belov

We'll talk about Dataframe — a library and a data structure that can help to read, write, generate, transform and organize data for displaying or plotting. It supports not only flat data structures but also hierarchical ones and thus can represent CSV, JSON or even subgraphs of objects in memory. While dataframes are usually mentioned in the context of data analytics, Kotlin Dataframe is also focused on general data cases and perfect for both professional and personal projects. We'll start with how it all works in Kotlin Notebook — an environment where you can write and execute fragments of code. It’s a good fit for prototyping and trying out ideas. The IDE provides beautiful interactive tables that can display hierarchical data. Also, in the notebooks, the types of dataframe variables are updated after each fragment execution. It means that columns of the dataframe will appear in completion together with their content type. Kotlin Dataframe’s new compiler plugin takes this idea even further and infers on-the-fly the types of columns while data wrangling. Come over, and we’ll show it in action and share what Kotlin language features make it possible

[__J____ff]

this is how KOTLIN SCRIPT should act as well ! Convenience is key with scripting and kotlin should strive to be even better than python in convenience.

[skarloti]

Some time ago I tried to use this library for huge datasets (billion rows) like CSV and Json Lines, but despite being able to stream it it didn't work well. Has anyone done actual samples? However, this is important if you decide to train a large language model.

[Noxafurry]

It would be nice to see an example where one JSON field couldn't be typed. Let's say we have a collection of objects and some field in them while having the same name has different types in different objects. The solution with Any is obvious, but could this type be converted into a sealed class holder for example?

[Noxafurry]

On the production part: it looks cool, but wouldn't having a network request (I assume) lead to non-stable build, also does it mean that build will heavily rely on an internet connection? If I am right, it doesn't sound production-like :(

[skarloti]

42:19 In this part in production it is promising, but I also injected a scheme in Kotlin 1.x and classes are automatically generated!

[LarryGarfieldCrell]

The live demo gods were not kind today...