Data platform in a mesh architecture

Data Mesh is a paradigm shift in big analytical data management that addresses some of the limitations of the past paradigms, data warehousing and data lake. Data Mesh is founded in four principles: "domain-driven ownership of data", "data as a product", "self-serve data platform" and a "federated computational governance".

In this talk Emily and Zhamak will talk about data platforms in a mesh architecture. They will cover topics such as what does it take to make data sets independently understandable? How do we empower individual domain teams to deliver data products, while lowering their cognitive load? How do we build a platform that enables data generalists and reduces the need for specialization?

[Chris-esei]

I liked the video a lot! Our company is moving already into that direction. What is missing is a enterprise architecture team that has a working business model and the authority to apply certain governance rules to all systems, how they should export data. There are several data lakes and data warehouse in our company, the biggest ones have 150 source systems attached. There are 10 different ticketing systems sourced (ITIL with incidents, service requests, availability, outages, etc.) in at least two of those DWHs. Some of the tickets are synchronized to some other ticketing systems. The difficult part is to harmonize and combine the data without having duplicated data and integrate a business view on the data. If each ticketing system is providing their data in data mesh, everybody has to harmonize the data again and integrate a business view. The business view is use case dependent.
Another thing where I see an intermediate layer like a DWH is needed is the versioning and the anonymization and pseudonymization of data. Some of the data needs to be deleted after a while for different reasons, but it is allowed to use aggregated data for some use cases. You need to combine the data of different sources first in order to be able to aggregate or anonymize it. If it is anonymized already in the source, then you cannot combine the data anymore. I've seen many difficulties in operational systems when they should provide a history of their data in a consistent way. A generalized data hub which has a versioning per attribute in place (maybe in 5th NF) could help. So there would still be one data lake which contains all data, but the different data providers are responsible for their product.

[anttipikkusaari]

What an inspirational and informative presentation throughout! Thank You for that. Data Mesh is clearly the only game in town wrt achieving true Data Excellence - defined thru five strategic measures of Scale, Speed, Agility, Quality and Value. Why is this? Because no centralized architecture or organizational model can achieve Data Excellence in the context of Cambrian explosion of data sources and analytics use cases across products, services, processes, business functions and customer journey touchpoints. In other words, path to genuine AI Organization for maximum productivity and competitiveness goes thru Data Mesh whether we like it or not. Eventually, this will become Main Stream as companies choosing not to do this will be marginalized and their resources will be set free by Creative Destruction.

Data Mesh Platform has obviously critical role to play - more so the closer we get to main stream state. With that, the option to Buy instead of Build will become increasingly important. Given the circumstances - i.e. strong Data Mesh value proposition and the wide-spread excitement around it viz-a-viz fundamental need for Data Excellence - it is easy to predict explosion of product and solution offering around Data Mesh Platform in coming months and years. However, there will be numerous pitfalls emerging from unsubstantiated claims for "Data Mesh compliance" when the Buyer lacks the know-how to verify the claims. At worst, that leads to capabilities that do not decrease friction or cognitive load - on the contrary. It is this very reason that makes presentations like this so valuable!

[m_nouman_shahzad]

Video Timeline

|-24:44 Service Service Data Platform
|-25:22 Data Product Consumer Experience
|-30:30 Data Product Developer Experience
|-42:18 Example capabilities of a Mesh Experience Plane
|-49:50 Example capabilities of a Data Product Experience Plane
|-53:57 Accelerating Data Product Development
|-59:32 Utility Plane
|-1:06:12 How to Start Your Platform Implementation
|-1:11:47 Q&A

[techchanx]

Data mesh looks quite vague, never really understood what it is. Looks more like best practice/pattern than anything else.