Consistency 3/3 - Data Consistency and the "Theory of relativity"

 "If a tree falls in a forest and no one is around to hear it, does it make a sound?"

If a system has an inconsistency but no one is able to observe it, is it still an inconsistency?

Introduction

We need to scale data processing systems geographically to achieve lower Latency and (at least partial) Availability in case of network Partitioning. But CAP/PACELC tells us that we cannot achieve strong Consistency in this case. When we increase the Consistency requirements, we have to accept lower Availability and higher Latency.

What is the minimum consistency level that we need?

If eventual consistency is enough for your system, things are pretty clear and relatively simple. Most probably you want to achieve Strong Eventual Consistency that is relatively cheap and provides nice guaranties. For this you will have to you something like CRDT. Some theoretical results assure us that you cannot find something way cleverer than CRDT that achieves Strong Eventual Consistency.

Often, you have business flows where eventual consistency is just not enough. For example, after you update a data in one region, you expect a following read in another geographical node to return the updated value, like in "Read-Your-Writes Consistency". This sounds like a pretty simple requirement, but it is really hard to implement correctly and it will increase the Latency of the system and/or decrease Availability - compared with eventual consistency.

Consistency 2/3 - Flow consistency - read-your-writes consistency

  

2. Consistency, Availability and low Latency in Distributed system

(workarounding the CAP/PACELC theorems)

 

Introduction: "1. Cache and Data Consistency in Distributed systems (CAP/PACELC/CRDT)"

 

TL;DR

Full strong Consistency in geographically Distributed systems can only be achieved by sacrificing Availability (per CAP theorem) and with prohibitive Latency costs (per PACELC theorem).

However, we can still design consistent enough systems that continue to function when one geographical region is down and without paying the inter-region latency most of the time.

While eventual Consistency is OK many times, there are still cases when we want a strong read-after-write consistency for certain read-after-write flows. There is an optimum design that assures strong Consistency inside read-after-writes flows. Arguably, this is the highest Consistency level that can be assured without a prohibitive impact on Availability and Latency.

Consistency 1/3 - Cache and Data Consistency in Distributed systems (CAP/PACELC/CRDT)

 

Abstract

There is always a tension between data Consistency and system Availability when Partitioning a system across datacenters (think CAP). Especially data cache-ing poses interesting challenges. This tension becomes way more acute as soon you have 2 data centers separated by more than 10ms latency. I present below some of the problems along with possible solutions.

In the end I will present an elegant solution that maximizes Availability while providing the needed Consistency level for read-after-writes flows. The solution requires the client to carry a monotonic id along the flow. I would postulate that any solution where the client don't carry some consistency info will provide a higher latency that the presented solution (see chapter "Flow consistency").

The examples below are simplified to be intuitive and easy to understand, however these learnings also apply to N datacenters.


How it starts

Suppose you started with on single datacenter called Central. It has all your data, and all clients are sending requests there. You have some in-memory cache-ing (like Memcached, Redis), and you are managing to keep it consistent by making sure that each write to database is also put in the memory cache as it is written to database.

Reusable building blocks in software

When we design a building, we usually design it base on plane walls and right angles. Why is that?

Sometimes we design a round form, however that becomes way more complex to build. Still, we try to stick with simple forms like circle arcs.

The Sydney Opera House was architected with unusual round forms at the roof. That project "was completed ten years late and 1,357% over budget" (source). The project really gained speed when they reduced the complexity of sail-like round forms to reusable (smaller) building blocks.

In software you have the same problem. If you don't find reusable building blocks - as reusable code or reusable patterns - the complexity of the project grows exponentially, and you find yourself over budget (time and cost). 

In software, you rarely need something as sophisticated as Sydney Opera House. Therefore, try to reduce complexity as much as possible if you want to finish your project in realistic time.

Simplicity is the ultimate sophistication (Leonardo da Vinci)

The Grug Brained Developer teaches us that "complexity very, very bad". Then what is the antidote to complexity?

Simplicity should be the obvious answer. However, simplicity is tricky to define and even harder to achieve.

Is simplicity to express your intent with 50% less characters or words? Not if it reduces readability. The code should be simple to read, because you spend more time debugging a line of code than writing it.

Sometimes is preferrable to write longer code to achieve a different type of simplicity: the simplicity for the human brain. It is preferable to write 10 lines of verbose brain-dead code, and not compress it in 15 lines that are so "clever" that it takes 30 minutes to understand it - even for you, months later.

Now we are approaching the building blocks paradigm. It is not only about code reuse - sometimes code reuse introduces incidental complexity. It is about using reusable patterns that are simple to implement, understand and (re)use.

When higher level patterns are built on other simple to understand patterns, then you are approaching simplicity. But each pattern should be simple enough in itself, considering you understand the underlying patterns that are made of. Think about "code should read like good prose". This means that each word is easy to understand, and the phrase structure is simple enough.

Sometimes those patterns are methods and classes. Some other times they can be patterns of organizing code, like MVC. If code reuse makes you to write harder to understand code, maybe you are over-using it.

A brick is something simple: it has right angles and plane facets. A random rock is nothing like simple as shape. It is way easier to build something from bricks than random rocks.

Compression

This simplicity can be expressed in the terms of the ability to compress something complex (requirements) to something simple (implementation).

A brick is a highly compressible pattern, everyone can picture it. The shape of a random rock is almost impossible to be expressed concisely in words or code. Try to describe a random rock's shape in details...

A simple10-floors building block can be compressed by multiplying the plan of a floor by 10 times. This is a highly compressible design. It is not necessarily pretty, but easy to understand and execute. Most important, it is less likely to make serious mistakes in implementation. Often you want this in software - not something golden plated that don't pay off the extra implementation time. You should allow extra complexity only when the extra customization brings a really high business value.

Actually, pretty is somehow related to compression. A pretty face is often symmetrical left-to-right, so more compressible patterns feel pretty. A pretty fractal image is the absolute compressible form - it emerges from a simple rule applied recursively. You want something like this in your software - if possible. 

I prefer the work "elegant" to describe the analog of pretty in software. Sometimes elegant is only a feeling that "it looks right". Sometimes an elegant implementation even looks perfect. Sometimes you need to relax a bit the requirements to achieve such elegance and efficiency in implementation.

If we look deeper, an elegant implementation is a combination of simplicity with compressibility. It expressed requirements that look complex and messy in a concise way, that is also easy to understand. It solves a complex enough problem of the user in a simple and concise code. Once you read it, it gives you that Evrika feeling, however it was not obvious before finding the solution.

The "perfection" feeling that some code inspire must be related to compression. While simple code can be confused with short but unmaintainable code, the compression has a higher-level dimension of optimization. An elegant code should optimize higher level, including the effort of maintaining and extending that code later. Elegance normally provides a way to implement new requirements faster and with less code - and that is very related to compression.

An elegantly designed project normally achieves an economy of scale - as new features are implemented, it gets easier to implement a new one - because you can reuse some building blocks implemented before. Unfortunately, many projects achieve the opposite: as project gets bigger, it gets more costly to implement a relatively simple new feature. This is a sign that the architecture departed a long time ago from elegancy.

For a more abstract concept of compression see: Kolmogorov complexity - short, informal introduction

An iterative process

There is no up-front elegant or simple architecture for a project. You might have a good intuition about a fundamental structure that would simplify the project; however, you cannot have all the architecture details determined before starting to code.

Unlike a building, the software code is... soft - it can be refactored with relatively low cost. It is highly inefficient in software to determine the perfect architecture/blueprint of the project before starting to code. That was the waterfall paradigm, when refactoring code was expensive. Now we have great tools for refactoring.

Most often you get the requirements wrong and you need to throw away your blueprint later, anyway. Even worse, some people stick to the plan and continue to use an architecture that is highly inadequate for the actual business requirements. I call this an "impedance mismatch" between requirements and architecture.

The best way is to grow a software project organically. Start with a simple structure based on initial knowledge, then evolve that structure as new requirements arrive. The architecture should start to emerge a little later, when you implemented most patterns that your application need - ideally with reusable building blocks that are also simple to use.

Software building blocks

If you feel like you are re-implementing that same logic over and over for new requirements, this is a sign that you need some reusable building blocks in code. If the programming language does not allow to reuse the code, you can automatically generate the code for the same pattern - with a script. 

If you allow a complex pattern to be re-implemented multiple times in your project, you are really far from compressibility and elegance. The clear effect will be that: when you find a bug in one of the implementations, you don't know where you need to fix it additionally.

A word of caution

We were thought to use the DRY principle (don't repeat yourself). This does not mean to implement common code for anything that resembles a bit. It is not good to compress 2 methods of 10 lines of code in a common method of 15 lines of code - if those 15 lines have many if(case1){}, if(case2){}... Such 15 lines of code are harder to read than the 20 lines you started with. In that case, it is better to keep the two methods of 10 lines.

I say, duplicate code first if you need to change the code, and refactor after the third use. I have a theory about this based on skier's problem, but this is another subject.

Sometimes it is good to have some code duplication to achieve simplicity and compressibility for the human brain  (think WET). This is the fine grained situation of software decomposition. However, most software projects greatly benefit from identifying reusable building blocks that are elegantly designed and simple to use.

Bounded contexts or consistency contexts?

"Embrace modularity but beware of granularity"

 (Mark Richards)

 

While using the Microservice architecture in software can buy you some agility if done right, I often see architectures where microservices bring extra complexity that actually increases the implementation time. The microservice architecture often brings incidental complexity, that is often caused by the uninspired choice of our microservice boundaries.

Bounded context should guide the choice of microservice boundaries. However, I find bounded context to be a too ambiguous concept.

Any unit of software can be seen as a bounded context, even a class. A Payment sounds like a bounded context. What about a CreditCard, can it have its own bounded context?

We don't want to create a microservice for each class, for sure. Think about this when you think to create another microservice, are you going too close to the "microservice per class" anti-pattern?

On the other end, most real life software systems are complex enough that hardly makes any of the subsystems completely bounded. Any microservice you want to extract will have many wires (like API calls) that will need to be preserved with the rest. Ideally, we should cut where those wires are few rather than many, however this is not always easy.

Joins across microservices

A simple 2-tables join that works efficiently in a single database is way more complex to be computed between 2 microservices with their own database. If both tables are reasonably big and you cannot do it in the database, you need to query both microservices by (REST?) APIs and do that join in memory.

What about the consistency of data? While you make that join through API calls, data might change in each microservice, and you might end up with inconsistent results, like missing references or duplicated values.

The in-memory join might work if you only join with small tables from a remote service, like a table of names. However, in this case, why does it have to live in a separate microservice in the first place? Maybe you only needed a Java enum that is shared?

It is also doable to make a join between a selected list o local entities (like 100) with corresponding entities from another microservice. You can even have bulk APIs that serves all 100 entities from a list of kees. You might have some inconsistencies if you don't pay attention, however it can work.

But when you have complex joins between big tables of separate microservice, you might end up with a shared database in the end. But first, you will spend a lot of time trying to make it work via API calls :)

 

Transactions makes it even harder

Having too many calls between microservices can bring extra latency and serialization/deserialization overhead. However, the big monster of microservices is achieving transaction semantic across microservices. For example, you need to synchronize the buying order with the payment. You cannot have one without the other.

True microservices are designed to use separate databases. This forces you to not use the well-known database transactions and rely on complex sagas that are rarely achieving all the transactionality guaranties you get from a single database (like SQL). This brings extra design complexity. This makes the development time way longer, the debugging more difficult, and the resulting system way more fragile.

We can relax a bit the purity of microservices, and share the same database when the transactionality force is too strong between those "bounded contexts" that you previously identified. However, in this case you might want to consolidate those microservices that share a single database in a modular monolith or modular microservice. The main advantage of microservices comes through independent deploy-ability. If you cannot have independent deploy-ability, chances are that the gains don't justify the complexity cost of the extra microservices.

It is often desired to relax the transactional requirements and rely on eventual consistency when possible. A technique that assures a certain level of consistency is to construct a kind of immutable versions of entities, that refer each other - similar with how GIT works. As long as you don't delete referring entities, your image of the system will be consistent - however you might not see the most recent image.

However, often enough, business needs some functionalities to be more strongly consistent. As in the example above, you cannot have a shipment without a pay, or a pay without a shipment. And you are in big trouble when you need to implement such atomicity across separate microservices, with separate databases. Of course, you can try complex sagas that are doing compensation transactions to assure the system achieve consistency - at least eventually.

 

The consistency context

Transaction is a strong word. We often need various levels of consistency. Usually, the lower consistency let you distribute the system and have microservices. The higher consistency level is forcing you to rely on a single database, like aggregating some of the microservices to a modular microservice or modular monolith.

Then why not designing our microservices around consistency requirements? We still need to have bounded contexts to be able to name things. However, taking consistency into account prevents you to distribute strong consistency transactions across microservices and databases - that is really costly.

You can design you microservices around what data need to be consistently seen by the outside worlds.

If you conceive to have a customer's identity without a profile - like in a progressive profile approach, then you can safely distribute the customer's identity and profile in separate microservices.

If your business needs require for strong consistency between some properties, you might want to keep those entities in a single database, and therefore you might want to have a single service responsible for that data.

It is still possible to have separate services sharing the same database. Those are not real microservices for the purists. I could live with that compromise for the right reason. For example, if I have a module often takes too much memory and brings everything down, it might worth to extract it in a separate (micro)service - to protect the other modules. However, often there are not enough reasons to separate those services at all. They can be a modules micro-service, that is also pure for whoever care.

In the case of Payment vs Order, the data payment data is distributed by design, so you will need some compensation transactions from time to time anyway. However, it is much more simple to implement logical transaction with a database transaction whenever possible. In the payment case, it might still worth to have a local database transaction about the order and that payment transaction that happens in a remote place (a bank). It might save you from many race conditions you would need to program against.

 

The consistency force

If something is so coupled semantically, so that requires stronger consistency guaranties, then it might deserve to be a single responsibility (micro)Service, even modular. Even if a subsystem seems like a bounded context linguistically, it might not be a separate concern if it is bounded logically by transactions with some other concepts.

We should not artificially break functionalities that are logically cohesive in distributed microservices. Be like a good surgeon, do not section important blood vessels, or transactions for this case.

Splitting transactions across microservices only makes our life hard, and usually does not provide the expected benefits promised by the Microservice architecture.

We can still relax consistency requirements to achieve a highly distributed system with high availability. However, this is not trivial and requires a lot of compromises on business requirements. You can check my other articles to gain some insight on this approach:

Data Consistency in Distributed system - CAP, PACELC and more 

Dependency Inversion for Entities - software architecture

 Problem to solve:

• We create a core software entity, for example a Product - this will become root or parent entity
• Then we create multiple "child" entities that depend on Product. For example Order and Warranty.
• We have business rules for the "parent" entity that depend on "child" entities. For example you cannot remove a Product if there are in-flight Orders; or if the Warranty period is not over for all the instances of that Product
• We want to avoid putting all those business rules in Product, because such "tight coupling" would make the software harder to maintain as the system grows.

Solution:

TLDR:

It helps to imagine the entities Product, Order and Warranty as defined in separate software modules, either in a modular monolith or in a microservice architecture. This will highlight the loose coupling we want to achieve. However, the actual implementation might reside in a single application or microservice.

While traditionally these constraints between entities are often defined as database constraints, a more loose coupled approach is to move those dependencies at business code level (think Java code). It helps to imagine a microservice architecture because separate databases forces you to give up on database level constraints.

• we keep the Product module completely unaware of the existence of the child entities (Order, Warranty).
• Product entity offers generic interfaces to register software hooks from child entities, for various cases (like Create, Read, Update, Delete) - don't need to be actual CRUD operations. Actually it is easier to register those hooks between modules in the same application/microservice.
• At software bootstrap of parent module, any child entity module can register hooks that restrict operations on the "parent" entity or receive information about changes in the parent company. For microservices, it should be done by configuration or orchestration.
• Order entity can register a simple boolean hook/callback on Product's Delete, that will check if there are any pending Orders in flight. By returning false, the Order entity/module can block the delete of the parent Product entity instance.
• Warranty entity can register a hook at Product's create/update to capture and cache the Warranty periods of the Product.
• Another Warranty hook, on Delete, can block the Product's delete based on very complex rules that ere expressed in Warranty's business logic.
• The business logic of Product does not need to be updated each time a new business logic is added concerning a "child" entity like Order, Warranty.

Other developments

• We often need to provide the user with a reason for blocking a certain Product operation.
• Along with a boolean, a child's hooks can return a reason for blocking the operation, for example:
• a String with a message for the user that tried to do the blocked operation on Product, in the terminology of the "child" Order, Warranty, like "forbidden delete due to Order xxxx"
• an URL that permits the user to navigate to the "child" entity/entities that blocked the operation on parent
• the requirement is that the returned software object should be agnostic of the children's structure (think String, URL), while the containing text information can be expressed in the child's specific terminology but can also refer the Parent. 
• The parent must not need to includ the children as build dependency to understand the answer.

The full story

1. Introduction

Tight Coupling

When we write software, we see that tight coupling makes the software harder to evolve and maintain. Tight coupling increases the risk for a new feature to break existing features. As the software grows, it becomes much harder to reason about all the implication that a change might have. The observable effect is that it takes more time for a similar feature to be implemented as the software grow. At a certain point, the software becomes unmaintainable and needs to be re-written.

Dependency Inversion

Dependency Inversion Principle (DIP) is a solution to avoid some tight couplings that a naive implementation might bring. To have an intuition what inversion means: A module is calling B module to do the job, however, at build time, B's module should depend on A's module.

Note that Dependency Inversion is way different than Dependency Injection (as in Spring). However, in some cases, the former can be sometimes implemented using the later.

Plugin architecture

Dependency Inversion is used, for example, to create a plugin architecture, where the core application (A) does not need to know about all the potential plugins (like B) - that can be develop later. Only the plugins (like B) need to know about an interface published by the core application (A) and implement it.

The difficulty for a plugin architecture is: you cannot call from A something that is defined in B without knowing B's interface. We can make A to not depend on B at build time, by creating an interface in A and having B to implement that interface and register its implementation at runtime. In this way, only B (and other plugins) depends on A. When a new plugin B is created, the core application A does not need to be recompiled to work with plugin B.

It is rather contraintuitive principle: when a "parent" module needs to call many "children" modules to do the job, it is better to have the children to depend on the parent.

Children will need to build against a generic interface located in the "parent" module or a separate modules that both "parent" and "children" depend.

2. Philosophic interlude

What is the logic of a "child" entity to depend on a "parent"? Except that it happens with humans also?

The reason in software is that, a "parent" entity Product should be able to build and exist independent of additional modules that might depend on Product. We should not pollute the Product's logic and build with all the children that can be added later on Product.

However, the "child" entities like Order or Warranty can only exist in relation with the parent entity "Product". An Order can only exist if it refers a certain Product that is ordered. While we can code Order against an abstract Product interface, the child still needs to refer a certain Product entity.

In a way, the implementation of the "Order" concept must refer some kind of "Product" concept, concrete or abstract. However, a "Product" can exist independent of child concepts like "Order", "Warranty". 

But who's concern is the link between Product and Order/Warranty? While Product can exist without Order/Warranty, it is more natural to have that responsibility in the children's code - as Order need to refer a certain Product anyway.

When we only have Product, and we add a child Order, we are creating a third system based on the 2 modules Product and Order. For sure, we need a Product id for creating any Order. In the Order module we might have an Order management interface that is listing the Orders, and this must call specific functionalities from Product module (like display a referring Product).

However, any additional business rules are not inherently linked with Order either. We can imagine another integration where an Orders module display a "Product deleted" information when it is the case and still carry the job of managing Orders until they are delivered.

Therefore, these additional business rules are not inherently bound to any of the two modules: Product and Order. These module interaction business rules are actually part of the plumbing that united a Product module with and Order module. 

While it might be convenient to store all those constraints in the "child" (Order), it should be possible to have a third "plumbing" module that is outside of parent/child (Product/Order) definitions. This might require the 2 modules to provide hooks registrations and alternative hook implementations that can be "plumbed" by the third module. Depending of the plumbing that is used, the result might implement different business rules.

3. General case

When be build non-trivial software, we often end up with multiple entity trees, where some entities like Product are roots and we have multiple sub-trees of depending modules that can have a hard dependency on a parent like Product or on a chain of entities. Normally, all circular dependencies must be avoided, we need to have a tree-like structure for each parent.

The "holly grail" of modularity is to be able to start with a single module having a Product entity and add Order or Warranty modules as additional modules that seamlessly plug to Product module. It should be possible, in theory, for the client to only buy the Product module and later buy one or both of the additional modules Order and Warranty.

The addition of a child module (like Order) should not require modification on the parent module (like Product). Any additional business rules added by the new Order module should be ideally contained in the child module (Order) or in higher level module. The additional constraint of Product's service logic is something brought by the additional module (Order), and it should not pollute the Product's implementation. While codding the "child module" might bring the need for a new type of hook in Product, this should be the exception.

The Product's abilities should be defines in terms of abstract ways it can interact with future "child" module. Actually, it does not even need to be a child-parent relationship, we can imagine modules that are collaborating on equal base. However, we will prefer a tree-like structure if we can help it, for the sake of our brain's sanity.

4. Possible implementations

Lately there is a reflux from microservices to modular monoliths, and for good reasons. It is easier to imagine those hooks in a modular monolith or modular microservice that contain all the related entities. Why would you break a hierarchy of entities in multiple microservices anyway? Such tight hierarchy cannot be easily broken into multiple bounded contexts - see TDD. Maybe each separate hierarchy is a bounded context - that is possible.

So, we have a module named Product, in project. Then we add a new module, named Order. This Order depends on Product or on a higher level interface that Product implements.

As the Product and Order are modules in a single application, it is easier to enforce that Product will not start without the constraint hooks that Order module is adding. The hook registration can be a single call to the hook registration API that Product exposes. We can also imagine a more dynamic bindings by using Spring listeners on Product that can avoid the hook API all-together.

Then a new module Warranty is added, that will register it's own constraint hooks on Product and maybe also Order. The constraint code is part of the Warranty module. However, based on application configuration, there can be alternative hooks that can be registered from Warranty into Product. We can register a strong constraint on product, or we can decide at business level that we want a looser link with Product.

We observe that, having only the hooks API in Product, the parent module Product must not be touched when adding depending modules. Normally, the Product's tests should only test the hook APIs in abstract. 

While it is still possible for a child to register a hook that blocks Product's normal operation, this is a bug in the child's implementation, and must not touch Product to be fixed. The parent might add additional safeguards on hooks, like a timeout for hook to respond.

There is a steep learning curve to implement such hook system, so it only worth implementing it when you have many entities with complex intricated logic. However, in such cases, I predict such approach can bring high productivity gains on the medium and long term.

5. Conclusion

We can achieve looser coupling between depending entities, by making parent entities unaware of children entities. In order to implement constrains between parent entities and child entities,  we can design a hook registration system.

The parent entity should not be aware of all the child entities that might depend on it. Not should the parent module depend on any parts of the children code. Only children will naturally depend on the parent entities.

To implement constraints, each child entity module should register hooks on the parent module, using an abstract API in parent. This hooks will be called from the parent to verify various constraints that are related to each child entity. The constraint is codded in the children's code, and is automatically called from parent via this abstract interface.

This approach should create software systems that are more easy to maintain and evolve. Such software should be less brittle - it should not easily break functionalities that are already tested in a parent entity, just because we added a new child entity.

Microservices: Software decomposition is not for performance, it is for the human brain

You can actually prove that most software systems would work faster in a monolith than in a microservice architecture. Just put back together all the microservices in a monolith, mentally. Replace all the REST calls with direct Java calls, this should eliminate a lot of extra work to serialize/deserialize. 

The resulting system will consume less resources overall just by less serialization. The latency should be smaller without the network calls. If you eliminate all the workarounds added just to make the distributed transactions to work, the performance should be significantly higher - with a proper load balancing of course.

There are some edge cases, like if your resulting application does not fit into DRAM, however this is rarely the case. The horizontal scalability should still work is you deploy enough of those monoliths and correctly balance the work among them. Of course, the starting time might be higher with the monolith. The risk for Out Of Memory from one module to the other is higher. Compile time is higher with monoliths.

However, performance or resource utilization is almost never a gain when distributing your application. It is likely that you will consume significantly more hardware resources compared with the monolith.

Then why we decompose our software in separate systems, even microservices?

Decomposing a monolith is mainly because of the human brain's limitations. We need to manage the complexity, and decomposition is often the solution to have subsystems that can fit in the brains of the team that is handling that code.

If done well, decomposing should also reduce the need to synchronize with other teams that are handling other subsystems. Maybe you also get accountability - you know which team is responsible to keep a certain functionality working.

Sometimes you can address conflicting non-functional requirements, like keeping one region of your  software super-duper-stable-and-secure and have other parts where you can safely experiment with higher risk appetite. You can reduce the risk for a security breach to propagate from one microservice to the other, at least in theory.

What do you lose when decomposing a monolith?

Let's observe first that the monolith was a system because different parts need to communicate to each-other. If is absurd to have a monolith that is doing 2 completely unrelated activities, like a chess playing  platform and an online store. The monolith is there because it made sense to group all those parts to do a job.

Maybe some parts are less coupled with the rest and makes sense to separate them. However, there is no decomposition that will not need communication with the rest. And after decomposition, all this communication will have higher latency and it will have higher serialization overhead.

When choosing to extract a microservice from a monolith, always look at how many wires will remain between the extracted microservice to the rest. Each of these wires will need to be implemented as a REST API probably and it will add implementation and maintenance overhead.

If there are too many wires that cannot be cut on decomposition, the overhead of separation might not worth the advantages. You might end up with a distributed monolith, that is often worse than the original monolith.

Think also about transactions. Distributing a transaction over different systems is at least an order of magnitude more complex than doing it in a single system with a single database. Find a solution to work without distributed transactions or keep the transactions in the same system. There are solutions to distribute transactions, but it rarely worth the trouble.

Start with a problem to solve

Think about the outcome you want to achieve by decomposition. If you define the most important problem correctly, it is possible that there are lower hanging fruits that can be addressed first. If you don't have good indexes in your database, microservices will not solve that. If the people are not disciplined, microservices will not improve that.

Don't just decompose your software because it is trendy. Splitting a monolith is not trivial, almost always the effort is higher than originally estimated. Then the maintenance overhead is even harder to measure. At least, you should be able to measure how was the system improved by splitting the monolith. Sometimes you pay the cost of decomposing and the system gets even worse than before.

Be a good surgeon

When you really have a pain that is hard to be addressed in the existing monolith, maybe the solution is to extract that part first, in a microservice. Extracting a microservice is like removing an organ from a living being. It is not feasible to cut too many blood vessels, you need to find a way to only cut a few.

When deciding what to extract as a microservice, start "pulling" that problematic part from the whole and observe what is the highly cohesive part (a mesh of wires) and what is a border with few wires to the rest. The part with few wires is where you want to make your cut to separate the microservice.

You might want first to extract a module in the monolith, to see how cohesive things are. After you played with different ways to separate a module, you might be in the position to extract such module in a microservice. And if there are still other problems, you can do the process again.

But before doing the cut... thing again: 

• what is the problem are you trying to solve? 
• does it worth the cost of extra complexity?