Is Data Mesh the Future?

Data Mesh is growing in popularity. It attempts to apply the concepts of microservices to the modern data stack. So what is data mesh? Let's explain how does this decentralized data architecture work?

⏯RELATED VIDEOS⏯

________________________________________________

________________________________________________

🎓Data courses (Not Produced by nullQueries)🎓

________________________________________________

📷VIDEO GEAR📷

💻VIDEO SOFTWARE💻
________________________________________________

Some of the links in this description are affiliate links and support the channel. Thanks for the support!

________________________________________________
00:00 Intro
00:23 The Problem
02:27 The Solution
04:13 Example
05:05 Expectation

[JoshuaWilsonNZ]

love your content my man. The ownership piece really is the value add part here. As the businesses I deal with mostly want that to be "IT's problem" to deal with.

[michaelbarber]

The problem is your are looking at it from a technological viewpoint, especially where your are talking about parquet files and DataBricks. Those data pipelines have to be simplified to EL centric processes still performed by IT as their "infrastructure support". You need to also empower you end users with GUI-based Semantic layer tools so that IT perhaps builds the initial models but the end-users can make fine adjustments and maintain them.

[Oliv-B]

Thanks for this quick overview. I wonder what are the motivation for a domain to create and mantain a clean and entreprise wide data product ? The domain will be focused on its own needs, not the ones of the other domains. You gave the exemple of microservices, but the main usage is to build intra-domain service, when it comes to create an microservice for other domains, it becomes a lot harder to design and develop (priorities of the domain owner may not be the same as of consumer domains)...

[Roger-gbyy]

Architecting from domain level could lead to form another silo. No enterprise level business and data modeling means new chaos in very near future. The only way is doing enterprise data modeling, base one business data cohesion decides business object (or component), and then IT architecture like data architecture, application architecture, technical architecture etc. Thinking about this scenario, HR domain produces like employees data, Customers domain produces customer data, it seems that different domain produces different data, but what about if employees are customers? How to ensure data consistency like PII, individual addresses, contact information. Without enterprise level business modeling, you never know what is real Business Object. Another example, collaterals (such as house, bond, equity, vehicle etc) could be your assets. If architecting by domain, could be collateral management and asset management, which is very common mistake in most banking architecture.

[jplee]

This is a great overview of Data Mesh, but is missing what is probably the hardest part of Data Mesh - Federated Computational Governance. The solution mentioned in this video would be difficult enough of a mindshift. Layer on top of that Computational Governance and the journey gets about 10x-100x longer.

[alkiviadesgeorgakis]

So practically we are saying to Devs stop messing around and care for your product, as data are part of your product ... De Teams are not your maids... In the end again de team needs to know what it gets in order to give knowledge back to company. At least the data will be already cleansed

[bobbymbabu]

Amazingly simplified with your explanation. Thanks!

[ivani]

So, you invented Kimball's datamarts

[TheRyalBeast]

This really only applies to companies which sell data as a product? I don't see why you would ever need to do something like this with traditional organizational reporting as you showcased in your example (HR, Finance, SC Data). Or am I missing something? It seems like a lot of extra work compared to like a data vault architecture for example which accomplishes something similar with far less headcount and effort.

[kedharnatekar]

You said it rite. It would never work 😆. With no centralized source of truth and every domain team handling it's own data, it would increase manpower cost and communication between teams to share their data with correct schema for aggregation would be more hectic process than a single team managing everything.

[ponnuenigma]

can you please do a video on quantum computing and its scope?

[MuideenMuibi]

Sounds like going back to creating data silos

[ArnyTrezzi]

Great explanation, I am non technical and I got the ideas!

[adrianomiller]

Can you cover $dag constellation. They created a hgtp// protocol that acts as a decentralized data mesh/micro service, fully zero trust.