Reusable building blocks in software

When we design a building, we usually design it base on plane walls and right angles. Why is that?

Sometimes we design a round form, however that becomes way more complex to build. Still, we try to stick with simple forms like circle arcs.

The Sydney Opera House was architected with unusual round forms at the roof. That project "was completed ten years late and 1,357% over budget" (source). The project really gained speed when they reduced the complexity of sail-like round forms to reusable (smaller) building blocks.

In software you have the same problem. If you don't find reusable building blocks - as reusable code or reusable patterns - the complexity of the project grows exponentially, and you find yourself over budget (time and cost). 

In software, you rarely need something as sophisticated as Sydney Opera House. Therefore, try to reduce complexity as much as possible if you want to finish your project in realistic time.

Simplicity is the ultimate sophistication (Leonardo da Vinci)

The Grug Brained Developer teaches us that "complexity very, very bad". Then what is the antidote to complexity?

Simplicity should be the obvious answer. However, simplicity is tricky to define and even harder to achieve.

Is simplicity to express your intent with 50% less characters or words? Not if it reduces readability. The code should be simple to read, because you spend more time debugging a line of code than writing it.

Sometimes is preferrable to write longer code to achieve a different type of simplicity: the simplicity for the human brain. It is preferable to write 10 lines of verbose brain-dead code, and not compress it in 15 lines that are so "clever" that it takes 30 minutes to understand it - even for you, months later.

Now we are approaching the building blocks paradigm. It is not only about code reuse - sometimes code reuse introduces incidental complexity. It is about using reusable patterns that are simple to implement, understand and (re)use.

When higher level patterns are built on other simple to understand patterns, then you are approaching simplicity. But each pattern should be simple enough in itself, considering you understand the underlying patterns that are made of. Think about "code should read like good prose". This means that each word is easy to understand, and the phrase structure is simple enough.

Sometimes those patterns are methods and classes. Some other times they can be patterns of organizing code, like MVC. If code reuse makes you to write harder to understand code, maybe you are over-using it.

A brick is something simple: it has right angles and plane facets. A random rock is nothing like simple as shape. It is way easier to build something from bricks than random rocks.

Compression

This simplicity can be expressed in the terms of the ability to compress something complex (requirements) to something simple (implementation).

A brick is a highly compressible pattern, everyone can picture it. The shape of a random rock is almost impossible to be expressed concisely in words or code. Try to describe a random rock's shape in details...

A simple10-floors building block can be compressed by multiplying the plan of a floor by 10 times. This is a highly compressible design. It is not necessarily pretty, but easy to understand and execute. Most important, it is less likely to make serious mistakes in implementation. Often you want this in software - not something golden plated that don't pay off the extra implementation time. You should allow extra complexity only when the extra customization brings a really high business value.

Actually, pretty is somehow related to compression. A pretty face is often symmetrical left-to-right, so more compressible patterns feel pretty. A pretty fractal image is the absolute compressible form - it emerges from a simple rule applied recursively. You want something like this in your software - if possible. 

I prefer the work "elegant" to describe the analog of pretty in software. Sometimes elegant is only a feeling that "it looks right". Sometimes an elegant implementation even looks perfect. Sometimes you need to relax a bit the requirements to achieve such elegance and efficiency in implementation.

If we look deeper, an elegant implementation is a combination of simplicity with compressibility. It expressed requirements that look complex and messy in a concise way, that is also easy to understand. It solves a complex enough problem of the user in a simple and concise code. Once you read it, it gives you that Evrika feeling, however it was not obvious before finding the solution.

The "perfection" feeling that some code inspire must be related to compression. While simple code can be confused with short but unmaintainable code, the compression has a higher-level dimension of optimization. An elegant code should optimize higher level, including the effort of maintaining and extending that code later. Elegance normally provides a way to implement new requirements faster and with less code - and that is very related to compression.

An elegantly designed project normally achieves an economy of scale - as new features are implemented, it gets easier to implement a new one - because you can reuse some building blocks implemented before. Unfortunately, many projects achieve the opposite: as project gets bigger, it gets more costly to implement a relatively simple new feature. This is a sign that the architecture departed a long time ago from elegancy.

For a more abstract concept of compression see: Kolmogorov complexity - short, informal introduction

An iterative process

There is no up-front elegant or simple architecture for a project. You might have a good intuition about a fundamental structure that would simplify the project; however, you cannot have all the architecture details determined before starting to code.

Unlike a building, the software code is... soft - it can be refactored with relatively low cost. It is highly inefficient in software to determine the perfect architecture/blueprint of the project before starting to code. That was the waterfall paradigm, when refactoring code was expensive. Now we have great tools for refactoring.

Most often you get the requirements wrong and you need to throw away your blueprint later, anyway. Even worse, some people stick to the plan and continue to use an architecture that is highly inadequate for the actual business requirements. I call this an "impedance mismatch" between requirements and architecture.

The best way is to grow a software project organically. Start with a simple structure based on initial knowledge, then evolve that structure as new requirements arrive. The architecture should start to emerge a little later, when you implemented most patterns that your application need - ideally with reusable building blocks that are also simple to use.

Software building blocks

If you feel like you are re-implementing that same logic over and over for new requirements, this is a sign that you need some reusable building blocks in code. If the programming language does not allow to reuse the code, you can automatically generate the code for the same pattern - with a script. 

If you allow a complex pattern to be re-implemented multiple times in your project, you are really far from compressibility and elegance. The clear effect will be that: when you find a bug in one of the implementations, you don't know where you need to fix it additionally.

A word of caution

We were thought to use the DRY principle (don't repeat yourself). This does not mean to implement common code for anything that resembles a bit. It is not good to compress 2 methods of 10 lines of code in a common method of 15 lines of code - if those 15 lines have many if(case1){}, if(case2){}... Such 15 lines of code are harder to read than the 20 lines you started with. In that case, it is better to keep the two methods of 10 lines.

I say, duplicate code first if you need to change the code, and refactor after the third use. I have a theory about this based on skier's problem, but this is another subject.

Sometimes it is good to have some code duplication to achieve simplicity and compressibility for the human brain  (think WET). This is the fine grained situation of software decomposition. However, most software projects greatly benefit from identifying reusable building blocks that are elegantly designed and simple to use.

Bounded contexts or consistency contexts?

"Embrace modularity but beware of granularity"

 (Mark Richards)

 

While using the Microservice architecture in software can buy you some agility if done right, I often see architectures where microservices bring extra complexity that actually increases the implementation time. The microservice architecture often brings incidental complexity, that is often caused by the uninspired choice of our microservice boundaries.

Bounded context should guide the choice of microservice boundaries. However, I find bounded context to be a too ambiguous concept.

Any unit of software can be seen as a bounded context, even a class. A Payment sounds like a bounded context. What about a CreditCard, can it have its own bounded context?

We don't want to create a microservice for each class, for sure. Think about this when you think to create another microservice, are you going too close to the "microservice per class" anti-pattern?

On the other end, most real life software systems are complex enough that hardly makes any of the subsystems completely bounded. Any microservice you want to extract will have many wires (like API calls) that will need to be preserved with the rest. Ideally, we should cut where those wires are few rather than many, however this is not always easy.

Joins across microservices

A simple 2-tables join that works efficiently in a single database is way more complex to be computed between 2 microservices with their own database. If both tables are reasonably big and you cannot do it in the database, you need to query both microservices by (REST?) APIs and do that join in memory.

What about the consistency of data? While you make that join through API calls, data might change in each microservice, and you might end up with inconsistent results, like missing references or duplicated values.

The in-memory join might work if you only join with small tables from a remote service, like a table of names. However, in this case, why does it have to live in a separate microservice in the first place? Maybe you only needed a Java enum that is shared?

It is also doable to make a join between a selected list o local entities (like 100) with corresponding entities from another microservice. You can even have bulk APIs that serves all 100 entities from a list of kees. You might have some inconsistencies if you don't pay attention, however it can work.

But when you have complex joins between big tables of separate microservice, you might end up with a shared database in the end. But first, you will spend a lot of time trying to make it work via API calls :)

 

Transactions makes it even harder

Having too many calls between microservices can bring extra latency and serialization/deserialization overhead. However, the big monster of microservices is achieving transaction semantic across microservices. For example, you need to synchronize the buying order with the payment. You cannot have one without the other.

True microservices are designed to use separate databases. This forces you to not use the well-known database transactions and rely on complex sagas that are rarely achieving all the transactionality guaranties you get from a single database (like SQL). This brings extra design complexity. This makes the development time way longer, the debugging more difficult, and the resulting system way more fragile.

We can relax a bit the purity of microservices, and share the same database when the transactionality force is too strong between those "bounded contexts" that you previously identified. However, in this case you might want to consolidate those microservices that share a single database in a modular monolith or modular microservice. The main advantage of microservices comes through independent deploy-ability. If you cannot have independent deploy-ability, chances are that the gains don't justify the complexity cost of the extra microservices.

It is often desired to relax the transactional requirements and rely on eventual consistency when possible. A technique that assures a certain level of consistency is to construct a kind of immutable versions of entities, that refer each other - similar with how GIT works. As long as you don't delete referring entities, your image of the system will be consistent - however you might not see the most recent image.

However, often enough, business needs some functionalities to be more strongly consistent. As in the example above, you cannot have a shipment without a pay, or a pay without a shipment. And you are in big trouble when you need to implement such atomicity across separate microservices, with separate databases. Of course, you can try complex sagas that are doing compensation transactions to assure the system achieve consistency - at least eventually.

 

The consistency context

Transaction is a strong word. We often need various levels of consistency. Usually, the lower consistency let you distribute the system and have microservices. The higher consistency level is forcing you to rely on a single database, like aggregating some of the microservices to a modular microservice or modular monolith.

Then why not designing our microservices around consistency requirements? We still need to have bounded contexts to be able to name things. However, taking consistency into account prevents you to distribute strong consistency transactions across microservices and databases - that is really costly.

You can design you microservices around what data need to be consistently seen by the outside worlds.

If you conceive to have a customer's identity without a profile - like in a progressive profile approach, then you can safely distribute the customer's identity and profile in separate microservices.

If your business needs require for strong consistency between some properties, you might want to keep those entities in a single database, and therefore you might want to have a single service responsible for that data.

It is still possible to have separate services sharing the same database. Those are not real microservices for the purists. I could live with that compromise for the right reason. For example, if I have a module often takes too much memory and brings everything down, it might worth to extract it in a separate (micro)service - to protect the other modules. However, often there are not enough reasons to separate those services at all. They can be a modules micro-service, that is also pure for whoever care.

In the case of Payment vs Order, the data payment data is distributed by design, so you will need some compensation transactions from time to time anyway. However, it is much more simple to implement logical transaction with a database transaction whenever possible. In the payment case, it might still worth to have a local database transaction about the order and that payment transaction that happens in a remote place (a bank). It might save you from many race conditions you would need to program against.

 

The consistency force

If something is so coupled semantically, so that requires stronger consistency guaranties, then it might deserve to be a single responsibility (micro)Service, even modular. Even if a subsystem seems like a bounded context linguistically, it might not be a separate concern if it is bounded logically by transactions with some other concepts.

We should not artificially break functionalities that are logically cohesive in distributed microservices. Be like a good surgeon, do not section important blood vessels, or transactions for this case.

Splitting transactions across microservices only makes our life hard, and usually does not provide the expected benefits promised by the Microservice architecture.

We can still relax consistency requirements to achieve a highly distributed system with high availability. However, this is not trivial and requires a lot of compromises on business requirements. You can check my other articles to gain some insight on this approach:

Data Consistency in Distributed system - CAP, PACELC and more