Reduce System Complexity with Data-Oriented Programming • Yehonathan Sharvit • GOTO 2023

This presentation was recorded at GOTO Aarhus 2023. #GOTOcon #GOTOaar

Yehonathan Sharvit - Author of Data-Oriented programming @viebel

RESOURCES

ABSTRACT
Complexity is one of the main difficulties in the development of successful software systems. Modern programming languages and frameworks make it easy to develop and deploy our code quickly, but as the code base grows, complexity makes it challenging to add new features.

Data-Oriented programming is a paradigm that aims at reducing the complexity of information systems such as back-end applications, web services, web workers, and front-end applications by rethinking data.

Data-Oriented programming treats data as an immutable value that is manipulated by general-purpose functions. Moreover, data is validated à la carte.

In this talk, we illustrate the principles of Data-Oriented programming in the context of a software production system.

After attending this talk, you will be able to apply Data-Oriented programming principles in your preferred programming language and reduce the complexity of the systems you build.

Takeaways
• Apply Data-Oriented Programming principles in your preferred programming language
• Apply data validation techniques without using static types
• Represent data with immutable data structures
• Manipulate data with generic functions [...]

TIMECODES
00:00 Intro
01:10 What is complexity?
03:00 Information systems
03:54 Principles of data-oriented programming
05:49 What makes a software system complex?
07:23 Principle No 1: Separate code from data
12:16 Principle No 2: Represent data with generic data structures
22:51 Principle No 3: Do not mutate data
28:42 Immutability in practice
29:42 What about data validation?
36:06 History of data-oriented programming
38:38 Summary
39:17 Outro

Read the full abstract here:

RECOMMENDED BOOKS

#Complexity #DataOriented #DataOrientedProgramming #Programming #SoftwareEngineering #YehonathanSharvit #Viebel #Immutability #Clojure

Looking for a unique learning experience?

SUBSCRIBE TO OUR CHANNEL - new videos posted almost daily.

[ribasenric]

I've actually read his book and it's quite good. Unfortunately the talk isn't great. Yes, nothing new if you are used to functional programming, immutable data, etc.

[madlep]

There were definitely ways of dealing with immutable data before Clojure implemented Hash Array Mapped Tries in the late 2000s. Functional programming languages had been doing that for years. Rich Hickey didn’t invent HAMTs, that was Phil Bagwell (he said Hickey “discovered” them. Not sure if that’s meaning Hickey used Bagwell’s paper, or not).

Structural sharing is different to HAMT, and is bread and butter for functional programming languages, but using the same techniques in non FP languages with mutable data has always ended up being fiddly and error prone if it’s not built in. Sure, you *can* do it, but without the guarantees of immutability, and without a language optimized for handling GC overhead as a result, it’s never great.

JSONSchema is great, BUT it’s really about validating data at the edge of a system where it’s receiving input. It’s quite expensive, and there’s NO WAY I’d want to be doing that every time an internal function is called in an app just because there are no guarantees about the shape of any data the app is using.

[krystofjakubek]

Its very interesting that his idea of what data oriented means is very different from what other people mean.

The game comunity sees data oriented design as "removing all thats not strictly necessary for the functioning of the program" or "getting the clowns out of the car" as formulated by Mike Acton. Of course the result is less code thus easier to understand and by almost coincidence better performance.

His idea of data oriented approach is a lot more opiniated and relies mostly on ideaological statements. It mostly boils down to "keep data around and transform it".

I find this in direct contradiction with data oriented design which begins with the question: what is it you are trying to do? Load an image process it write some logs and exit. Okay then do exactly that an no more. There is no reason to have classes, factories, singletons, concept of resposibility etc. because fundamentally they are nowhere in that description of needed steps. The steps are take this and do that then take the result and do the next thing.

I have seen his idea of data oriented design in the closure comunity before (or maybe its just an echo of his book) and find it very interesting how different comunities can interpret an idea totally differently (and wrong if you ask me).

[lwouisYT]

The presentation was unconvincing and shallow, in my opinion.

Nothing concrete was presented. It's all slide-ware, ideological claims (e.g. "this way is better/simpler"), non-sentences (e.g. "data is represented as data"), and a few minutes of random meditation (??).

The speaker argues that moving away from classes to describe data and instead of using untyped blobs of JSON + JSON schemas is a better approach. Of course there are no absolute "better" approaches in CS. Only approaches better suited for some problems. For instance, if compatibility is important, you could argue that having a server and a client be flexible about the data they share is important. This is why attributes are all optional in recent versions of Protobuff for instance. But the speaker doesn't share such nuances/trade-offs. They claim that's it's generally simpler/better without talking about any downsides.

It's essentially ideology we are served. In our field of science, we expect strong methodology and evidence. Nothing of the sort was shown here.

[Tekay]

11:00 I don't see how the diagrams are simpler. Instead of one mess I now have two messes.

[ChrisAthanas]

@11:00 I'm not seeing the value added here... why not just call this Procedural and leave it be?

[_FFFFFF_]

Why is there confusion that this is the same as Data Oriented Design ? Looks like a lot of people seem to assume its the same thing, does the presenter say DOD == DOP ?

[Swampdragon]

"Static type systems can not express that an integer is included in a range"
Yes, yes they can. Scala 3 can, and so do some proof based languages.

[ArifBasri]

what's the difference between this and dependency injection or declarative based programming
?

[romanzkv]

Totally nothing new here - and the performance is terrible: DOOM would run at 1FPS using HAMT

[chauchau]

Didn't watch the whole thing but I feel like the idea hs is promoting is treating everything as data instead of a model that modeling some concept in real world.
I feel like some type of people might like these idea and might be benefical to them but majority like me are only human. However, It is also similiar to "everything is a map" anti-pattern. Put everything in a map, pass the map to others and then receive the output as a map. You also have to constantly check whether a key-value pair is inside the map or not. And whenever you meed to do something like control flow or make a decision, you need to do all checking for all related properties all over agin in every layer. People have seen "everything is a map" in their life know how horrible is. this kimd of philosophy only guarantee you creating a dead-on-arrival system that the whole thing turns into an ununderstandable, unreliable, unreadable, unmaintainable, difficult to chnage, incapable to reason with and error-prone technical debt immediately.

[josda]

So suffice it to say, he is pro- manager classes and/or global functions and static methods. And that means you won't be able to reason about it as well, actually. Maybe you understand the system conceptually but if there are parallel calls going on, i think it'll be more difficult to debug. Singletons and globals are generally what i'd stay away from. Separate models are something i keep at the boundary but try to convert to domain specifications as true class definitions.