Scale By The Bay 2020: Yves Parès, Reproducible Data Pipelines Using Controlled Effects

In this talk, we present a new architecture for data pipelining frameworks which promotes reproducibility along with an implementation in Haskell, kernmantle. We discuss how this architecture allows you to explicitly control which effects a pipeline is allowed to execute and how they are executed. Such effects may include file and network access, random number generation, parallel execution, and more. This framework also opens the door for config-time interpretation, which allows a pipeline to be analyzed at load-time, prior to its execution.

Yves Parès
Tweag
Software Engineer
I'm Yves Parès, a 30 y.o. software engineer, and I've been on a strict FP-only diet for about 10 years, to the point I now have trouble processing imperative programming.
My go-to meal is some closures as a starter, a big stack of monad as the main course and usually a monoid for desert (And sometimes, I indulge myself to a little mutable reference, as a cheat meal).
Lately I've been spicing it all up with arrows and algebraic effects. At first it's hot and takes some getting used to, but then it's great.
I obtained a PhD in knowledge engineering in health sciences 4 years ago, after which I started working for Tweag as a Haskell dev, mainly on scientific computing and data engineering projects.
I'm the main developer of the Porcupine and Kernmantle Haskell libraries for building configurable and portable task workflows in Haskell, the former being used by one of our clients in production.
I'd like to present to you why I and other very sound of mind people at Tweag think that this kind of high-abstraction diet is promising for software engineers in the fields of data science and simulation.
When I'm not working, I ponder over the great questions of the universe, like are we all Boltzman brains which just fluctuated out of a high-entropy soup, does the arrow of time originate from the information loss induced by quantum wavefunctions collapsing, or why are people still playing Catan when there are much better games around. Also I like improv theater.