The 12 Factor App for Data • James Bowkett • GOTO 2023

This presentation was recorded at GOTO Copenhagen 2023. #GOTOcon #GOTOcph

James Bowkett - Technical Delivery Director at OpenCredo @OpencredoItd

RESOURCES

ABSTRACT
Data is everywhere, and so too are data-centric applications. As the world becomes increasingly data-centric, and the volumes of that data increase over time, data engineering will become more and more important. If we're going to be dealing with petabytes of data it will be better to get the fundamentals in place before you start, rather than trying to retrofit best practices onto mountains of data. This only makes a difficult job harder.

The 12-factor app helped to define how we think about and design cloud native applications. In this presentation, I will discuss 12 principles of designing data-centric applications that have helped me over the years across 4 categories : Architecture & Design, Quality & Validation (Observability), Audit & Explainability, Consumption.

This has ultimately led to our teams delivering data platforms that are both testable and well-tested. The 12 factors also enable them to be upgraded in a safe and controlled manner and will help them get deployed quickly, safely and repeatedly.

This talk will be filled with examples and counter examples from the course of my career and the projects that my teams have seen over the years. It will incorporate software engineering best practices and how these apply to data-centric engineering. We hope that you can benefit from some of our experience to create higher quality data-centric applications that scale better and get into production quicker. [...]

TIMECODES
00:00 Intro
01:51 What & why?
06:17 Architecture & design
06:27 No. 1: Data structures as code
10:07 No. 2: Append-only data structures
13:21 No. 3: Optimize for access & retrieval
15:51 No. 4: Separate data from logic
22:32 No. 5: Strongly type your data fields
25:03 Quality & validation
25:14 No. 6: Architect for regression testability
29:19 No. 7: Track changes in your test data
31:59 Audit & explainability
32:08 No. 8: Mind your metadata: Data cataloguing
34:37 No. 9: Mind your metadata: Code traceability
37:02 Consumption
37:28 No. 10: Defined APIs for accessing data
40:08 No. 11: Defined SLAs (& SLOs) for data
41:17 No. 12: Treat data as a product
42:28 Summary
45:32 Outro

Download slides and read the full abstract here:

RECOMMENDED BOOKS

#12FactorApp #Data #DataCentricApp #CloudNative #SoftwareArchitecture #DataPlatforms #DataStructuresAsCode #ContinuousDelivery #SLA #SLO #DataAsAProduct #JamesBowkett

CHANNEL MEMBERSHIP BONUS
Join this channel to get early access to videos & other perks:

Looking for a unique learning experience?

SUBSCRIBE TO OUR CHANNEL - new videos posted almost daily.