Don’t Build a Distributed Monolith - Jonathan 'J.' Tower - NDC London 2023

Don’t Build a Distributed Monolith: How to Avoid Doing Microservices Completely Wrong - Jonathan "J." Tower

As a consultant, I get to see many systems built by many different developers. Recently, I’ve seen an uptick in the number of systems built with a microservice architecture in mind, but those systems often include a lot of the same mistakes that keep them from working well.

In this session you’ll learn from my experiences and get pointers on what to avoid in your microservices implementations so that you don’t accidentally build something which has all the worst aspects of a monolithic application and the worst aspects of microservices These monsters are what I call “distributed monoliths”, and I can help you avoid building one accidentally.

Check out our new channel:
NDC Clips:
@ndcclips

Check out more of our featured speakers and talks at

[zimpoooooo]

I've used loadbalanced monolith with a single database for a pretty busy site. It is the simplest thing to build, and it works perfectly. Architecture astronauts is also a thing.

[xemtube]

0:00 Intro
2:31 About Me
3:53 Monolith Definition
4:56 Monolith Advantages
6:45 Monolith Disadvantages
8:07 Monolith Architecture
8:37 Scaling Monoliths - Increase memory, load balancing, data sharding
11:13 Microservice Definition
13:58 Microservice Architecture
14:48 Event Bus
16:00 Distributed Monolith - Bad guy
17:25 Distributed Monolith Architecture 1 - Not using event bus to communicate between different services
18:39 Distributed Monolith Architecture 2 - Having one shared database among all services
20:54 Good vs Bad. Good: modular monolith, microserice. Bad: ball of mud monolith, distributed monolith
24:01 10 Common Mistakes of Distributed Microservices
24:24 Problem 1: Assuming microservices are always better
25:27 Reasons not to use microservices: monoliths are easier and not unscalable, microservices are hard to implement
26:54 Good reasons to use microservices: more scalability options, independent deployability, isolate surface area of failure
27:32 Microservice vs Monolith Tradeoff: microservices have high availability and eventual consistency, monoliths have low availability and immediate consistency
28:44 Problem 2: Using shared data store in your microservices
32:20 Problem 3: Microservices that are too big
36:14 Problem 4: Microservices that are too small
37:36 Problem 5: Creating microservice from scratch
41:07 Migrating from monolith to microservices: Big bang, Evolution, Strangler Fig
44:23 Problem 6: Coupling microservices through cost-cutting, i.e. shared modules
46:50 Problem 7: Synchronous communication between microservices
50:31 Problem 8: Breaking changes to event contracts
52:49 Problem 9: Not automating build and release
54:12 Problem 10: Changes to microservice affect other modules (Iceberg anaology)
55:49 Bonus Problem: Mismatched teams
59:13 Summary
59:56 Further reading
1:00:24 Recap/Q&A
1:00:57: Q1: How to migrate to microservice at the database level?
1:02:21: Q2: Adding fields to event signatures?
1:03:32: Q3: Is there a use case for distributed monolith?

[Link-hoyq]

This is the best introduction to microservices I've found so far: It explains what microservices are, are not, and when to and not to use them.

[simonk]

Thanks very much J.! I started a big microservices project about 4 years ago, and spent a lot of time reading articles and watching presentations; I wish your presentation had existed then as you've summarized the most important parts of all that I learned the long way in just 50 minutes. As always, there is so much more, and I'd love to see a followup presentation with more details, but I'll definitely be passing this link on to anyone who says to me "I'm thinking about using microservices".

[vanivari]

A big cause for "distributed monoliths" is testing. People distribute everything, but then they try to test everything together, often in a "classic" acceptance test phase in a "release train".

[russellf]

This is a good distillation of a lot of wisdom around building microservices (or not!) without a lot of fluff. Some great diagrams in there too that do well to focus on the key points. I watched it at 1.5x which was about right for me given that it's not a new subject for me.

[georgebeierberkeley]

Sometimes I think microservices is all about politics / management. Everyone wants their own team, their own turf. You can operate independently and choose your tools, data structure, etc. But managing all the cross-talk on the data via the event bus seems like a nightmare. I'm sure I don't know enough about it but it sure seems complicated. Great talk.

[DragonRaider]

How'd one go about solving the 7th example (order service getting customer data from the customers service)? In my head even if the data is duplicated into the orders service, inserting a lot of customers would lead to the orders service also needing to duplicate them (by listening to some CustomerCreated event), meaning it'd still be impacted by this kind of load on the customers service?

[waitingforyou]

One of the most disadvantages of microservices that I don't see it mentioned enough when talking about about microservices is the hosting COST, its much much expensive to host microservices. To me thats a huge disadvantage that should be mentioned

[AlgoristHQ]

I’ve worked on both Greenfield and Brownfield products… The Brownfield products were always more difficult. I completely believe that Brownfield product would be easier if Software Engineering were a more robust field where Clean Code and BDD were understood and the default method of development but I can’t keep up with 1000 line methods in 150 different classes.

[yarmgl]

monolith could also scale horizontally by dividing it into "macro" services: api nodes, shared state manager, and background worker

[doBobro]

If the only issue of monolith (by author words) is deployment issues I take monolith approach 24/7. It's pretty solved class of problems by modern infra.

[arnoldhau]

I did one team many services before, it worked fine and did not at all push us to monolith. We choose that architecture for a reason after all. What I see as main issue, so many people do microservices just because they are cool but do not reflect on it properly (we all learn, after all, but you need to allow for that) and people cut microservcies not by functions, but by data. Sometimes they are treated just like extended database tables.

[andrewfigaroa]

This is great! I wish I was able to convey these concepts to teams where I have been part of. I was never able to convince team members to take a decoupled approach.

[WantMore-mcdx]

"if done correctly" -- and what big company does this?

[RobCoops]

Looking at the hotel room example I am instantly reminded why microservices can be a serious pain. Looking at this example I would expect that a maintenance and housekeeping service would also like to know if the room is occupied. At which point I am copying state of a room between a bunch of services that might or might not update when I ask them to (eventual consistency after all) adding a few more rooms means having to update many different services. Other reasons why this is not ideal, each service must have a way to add a room and potentially remove a room.

I would suggest to create a separate microservice that deals with the rooms and their states, each other microservice can still have their own details relevant to their service about each room but the shared state of the room would be held by the new microservice. This ensures that every service is aware of the shared state of the rooms without running the risk of all sorts of issues with data inconsistency as a result of duplication of data.

I totally disagree with the idea of duplicated data for the simple reason that after many, many years of supporting systems in various industries I can say with reasonable authority that duplicated data will always diverge over time. As a result your service becomes kind of useless and you end up spending inordinate amounts of time trying to keep things in sync at an ever increasing cost as usage of your solutions goes up.


I would suggest approaching this more from a object oriented point of view where you would not create 20 classes that all are basically operating on the same data as that becomes a nightmare in a very short order. You would also create a class that contains the shared data so that this is what everyone operates on and each instance of that class contains all shared for that instance.
You should in my humble opinion do the same in the microservices world. Of course that service should only deal with the shared data in order for everyone to have the most up to date information when it is needed. Now in the case of a hotel room this is not very rapidly changing data so it is not all that critical but I am sure we can all think of may other situations where data does change rapidly and having outdated information in your microservice because you have not processed all the events yet can lead to a disaster.

Staunch adherence to any architecture ideology is unhealthy and one should always consider the implications of such decisions. Again the hotel room example is probably not going to cause any serious issues, but that is not because it is a perfect solution it is just because a very slow rate of change and very small data volumes make even suboptimal solutions seem quite good.

[Wyvernnnn]

The thing about adding fields in event contracts, making sure the code expects unknown fields and forwards them is a good contender for some gnarly CI tests. Push out an event with a random new field, make sure it gets forwarded all the way to the end of all paths and that nothing crashes.

[FudgeYeahLinusLAN]

Hey, if my client wants me to build them a distributed monolith, I'm gonna build them a distributed monolith. Because when I tell them that what they want to build is a bad idea, they'll just refuse to listen, and they're just gonna get some other consultant to build them a distributed monolith. At least when I build it for them, I can ensure that their distributed monolith doesn't completely suck ash.

[TheEaglefield]

This is a great talk, and helped me to grok some more parts of EDA and microservices. However im uncertain how exactly changes to the data model is handled in a microservices setup. How can you get complete decoupling - if you can at all - when you sometimes need to introduce breaking changes to your data model it seems to me like you will still have to handle that breaking change in all microservices that are dependent on the service where you made a change

Do you just make sure you version your data with the data model that data adheres to?
*edit: ah okay saw there's a similar question in the Q&A, but still interested to hear how people would deal with this

[ora]

@jonathantower for problem #7 and slide on 47:00 how'd you do it without sync call if you need order service to fail when customer isn't found? Fire and forget will cause you to introduce some intermediate state on Orders side and then process the reply from the Customers, that might be breaking for the user experience.