2016-09-06

Beauty of #microservices: part 7 breaking the monolith

1 Introduction


This blogpost is inspired by several blogposts about microservices and it is based on the blogpost [REF1] “Architecting #cloud-friendly application architecture #apparch (inspired by #microservices)” http://improving-bpm-systems.blogspot.ch/2015/04/architecting-cloud-friendly-application.html

See also the previous blogposts of “Beauty of #microservices” series

2 Breaking the monolith


This blogpost is a continuation of two previous ones:
Below, a series of steps to show how to remove from a monolith (actually a home-made ERP) some functionality around a particular Business Entity (BE) or a group of BEs.

At the AS-IS step, the monolith is the master of everything related to this BE (thisBE) – data, rules, processes and events (which are generated during the lifecycle of this BE and may affect other BEs.)

The first step is to externalise and make explicit the process to manage this BE in a BPM-suite tool. Keeping of thisBE (i.e. its data) is externalised as well. Also the associated rules must be externalised (as a copy) to reproduce the business logic spread in the monolith.

The monolith keeps its slave copy of data which are maintained via some stor-API. The associated business logic and event logic are still managed by the monolith. The data (as a slave) must always stay in the monolith because they may be used somewhere.

The second step is to externalise rules (when they are good enough to cover all the existing rules from the monolith).

At the TO-BE step, everything related to thisBE is externalised, but the associated events must be “injected” into the monolith by some func-API.


3 Anti-pattern DOUM


Avoid the "DOUble Master" (DOUM) anti-pattern (section 5.6 of my book "Improving enterprise BPM systems" http://www.improving-bpm-systems.com/book ).

As coordination can be carried out by an application or by a process engine, we have to be very careful to avoid the “double master” anti-pattern. At any moment in time there must be only one master responsible for the coordination of a particular process instance. (Of course, the coordination role may be delegated if appropriate.) This is analogous to a well-organised meeting where the chairperson decides who talks next.


The non-recognition of this anti-pattern can be very costly. We have observed a BPM solution which allowed the modification of data by a process engine, by an interactive application (i.e. by a human) and by a batch at the same time. The coordination of activities was based on data and, if necessary, the application or the batch could “correct” the process. The process engine was used mainly for the handling of three human activities, and the implementation of this solution (for a relatively simple business process) took several man-years.

4 Conclusion


How will you eat an elephant? Piece-by-piece, of course.

Thanks,
AS
Posted by Alex at 18:31 0 comments
Labels: , ,

Beauty of #microservices: part 6 managing state is a teamwork

1 Introduction


This blogpost is inspired by several blogposts about microservices and it is based on the blogpost [REF1] “Architecting #cloud-friendly application architecture #apparch (inspired by #microservices)” http://improving-bpm-systems.blogspot.ch/2015/04/architecting-cloud-friendly-application.html

See also the previous blogposts of “Beauty of #microservices” series.

2 Managing state is a teamwork


Obviously, a solution or an application implemented with microservices is a set or suite of stateful and stateless microservices. The chapter 7 of REF1 provides a classification of microservices. The stateful microservices are those which:
  1. manage some resources
  2. provide legacy functionality 
  3. assemble (implicitly and explicitly) other microservices
Each stateful microservice must be, ideally, idempotent to contribute to managing state.

Microservices, which manage some resources, may have a few impotency pitfalls to be avoided. For example, the read operation may be not idempotent if concurrent updates are possible. The update (or write) operation may be not idempotent if it can change some metadata, e.g. modification date. Also, idempotency may depend on a particular operation. The safest way is to create a small “shell” to guarantee the idempotency and a unique ID for each invocation.

Also, a small “shell” is only the option for microservices, which provide legacy functionality (they are considered as black boxes).

Microservices, which implicitly assemble other microservices, are a real pain because they have to be carefully reviewed about their idempotency. A possible approach for their idempotency is to re-execute again such a microservice. If all the microservices, which are invoked from it, are idempotent and don’t have any human involvement then such a re-execution will be idempotent as well.

Microservices, which explicitly assemble other microservices, may create some “check-points” (similar to mainframe batch systems) to start their re-execution from the last “passed” checkpoint. Of course, the data associated with checkpoints must be stored somewhere else as records. ( A similar approach can be found in http://www.theidentitycookbook.com/2016/06/blockchain-for-identity-access-request.html )

3 Error recovery (and distributed transactions)


As microservices form a distributed system, the error recovery is very difficult.

Explicit assembly of microservice, e.g. a business process in BPMN, can implement the error recovery in the following way.

Imagine a process fragment with three automated activities (A, B, and C) to be executed as a transaction. Each of those activities is an invocation of a microservice and the normal execution sequence is E2-A-B-C-E4.  Because any of those microservices may fail, this fragment contains the intermediate event E3 to intercept a failure and an activity for Error Recovery Procedure (ERP); the latter may be a human activity.

The first pass (with a failure of activity B ) has the following sequence:

E2-A(done)-B(failed)-E3-ERP


The second pass (with a failure of activity C) has the following trace:

E2-A(already done)-B(done)-C(failed)-E3-ERP


The third pass (with no failures) has the following trace:

E2-A(already done)-B(already done)-C(done)-E4

Activity A was executed 3 times, but it did the real work only at the first time – two other times were ignored because it is idempotent.

An extension with a timeout can be found in http://improving-bpm-systems.blogspot.ch/2014/08/bpm-for-digital-age-shifting.html

This way can be used also for implementing distributed transactions (please, note, some compensation activities may be necessary).

A similar approach was described at http://www.grahamlea.com/2016/08/distributed-transactions-microservices-icebergs/

4 Conclusion


Microservice architecture requires common efforts from microservices to achieve
  • state management, 
  • error recovery and 
  • distributed transactions.
Thanks,
AS
Posted by Alex at 09:28 0 comments
Labels: ,

2016-09-02

Beauty of #microservices: part 5 defragmentation of enterprise data model

1 Introduction


This blogpost is inspired by several blogposts about microservices and it is based on the blogpost [REF1] “Architecting #cloud-friendly application architecture #apparch (inspired by #microservices)” http://improving-bpm-systems.blogspot.ch/2015/04/architecting-cloud-friendly-application.html which uses other blogposts about microservices http://improving-bpm-systems.blogspot.ch/search/label/%23microservices

See also the previous blogposts of “Beauty of #microservices” series.

2 Unfortunate historical fragmentation of enterprise data


In modern enterprise computing environments, an enterprise data model (as a set of business-entities defined at the enterprise level) is, usually, an utopia. Enterprise data are spread among many existing applications that master some attributes of some business-entities. Also some business-entities are duplicated (usually, partially, by some fragments of them) among existing applications.

Because each of those applications (in-house developed, SaaS-based, PaaS-based, intact COTS, modified COTS) has its own lifecycle, the enterprise data model is far from the reality and complex data integration is mandatory to keep the integrity of the enterprise data.

Each new application has its own data and uses data from other existing applications. Thus, an application data model comprises some views on some business-entities from the enterprise data model. However, in such an application data model only some attributes are under control of the application.

If an application has to use data from several other applications then the evolution of some application may have a destructive effect on some other applications – typical “Château de cartes” anti-pattern.

3 Enterprise data model must be flexibility-driven not legacy-held


Ideally, an enterprise data model must
  • easy to evolve (by definition)
  • be service oriented so data can be accessed only via API (this its fragments can be joint up together even not sitting in the same database)
  • implement total versioning for some business-entities (to cover the enterprise lifecycle)
  • cover all the types of business-entities
    • transactional data (your business-specific and business-critical data)
    • reference data (taken from other sources and business-parties)
    • operational data (about how the work has been done)
    • reports
    • analytics data
    • records
    • documents
    • media
    • social

There are several techniques to move to this ideal.

Make some functionality enterprise-wide as a Corporate Unified Business Execution (CUBE) platform (see http://improving-bpm-systems.blogspot.ch/2015/10/enterprise-patterns-peas-example-cube.html ). Thus an individual application comprises only business-specific functionality. For example:

  • security management (identity, authentication and authorization)
  • content and knowledge management
  • software factory
  • reporting and analytics
  • business process management
  • records management
  • etc.

Make explicit data update processes – see the pattern “Practical process patterns: Synchronisation Of Sources (SOS)” http://www.slideshare.net/samarin/practical-process-pattern

Use some specific characteristics of your enterprise.

4 Adding flexibility


The classic “point-to-point” scenario is that a solution can read some data from some applications via storage-centric API or asset API or stor-API. Actually, those data are application-specific views on some enterprise business entities, e.g. “ClientERP BE” is the business entity CLIENT as it is defined in an ERP. Of course, in this sensation, the solution 1 strong depends on the evolution some other applications.


Another classic “data-access-layer” scenario is to use a microservice aggregate all the application-specific views in a read-only business entity.


The ability of microservices to have their own persistence store (which may be just a table in a common database) is very useful to implement a small extension of an business-entity. The same technique may help to avoid customisation of COTS products. This is a safe way to mode to an ideal enterprise data-model.


With some efforts (mainly for externalising some business logic), read-only APIs may be transformed into read-write APIs. If a business-entity uses more than one underlying applications then a “data update process” is necessary to implement a multi-phase commit.


If it is possible to externalise the business logic then some functionality-centric API or business API or func-API may be implemented.



5 Conclusion


With the use of microservices and other technologies, an enterprise data model may:

  • become always up-to-date and
  • evolve with the speed of agile development.

Thanks,
AS

Posted by Alex at 15:52 0 comments
Labels: , ,

2016-08-26

Beauty of #microservices: part 4 from agile development to agility of enterprise systems

1 Introduction


This blogpost is inspired by several blogposts about microservices and it is based on the blogpost [REF1] “Architecting #cloud-friendly application architecture #apparch (inspired by #microservices)” http://improving-bpm-systems.blogspot.ch/2015/04/architecting-cloud-friendly-application.html which uses other blogposts about microservices http://improving-bpm-systems.blogspot.ch/search/label/%23microservices

See also the previous blogposts of “Beauty of #microservices” topic.

2 Agility during the whole application lifecycle


This blogpost is named after my 10 years old presentation “From agile development to agile evolution of enterprise systems” ( see http://www.slideshare.net/samarin/from-agile-development-to-agile-evolution-of-enterprise-systems ) with a small change to introduce the word “agility”.

This done because of one of Matthew Kern’s blogpost “Fixing Agile with Architecture” (see https://www.linkedin.com/pulse/fixing-agile-architecture-part-4-matthew-kern-cea-cissp-issap-pmp- ). Obviously, architecture works with a longer timespan, e.g. the whole application lifecycle ( see also http://improving-bpm-systems.blogspot.ch/2014/03/enterprise-patterns-ear.html ) and application development covers only the delivery of the first production version. Thus architecture must guide the application development.

Considering that the TCO of a custom-developed application is only 20% for the implementation of its first production version then the rest 80% must be taken seriously. A real-life example: the cost for a small modification (changing an UI field from free-text to drop-box) in a custom-developed ERP is 20 KCHF and the implementation time is 3 months.

The agile development does guarantee that prototyping and implementation of a new application can be done rather quickly. But, the agile development does not guarantee that the production version of an application can be changed very quickly. Just compare – change a wheel of a parked car and change a wheel of the moving car.

The table below compares the level of frequency of changes for various application building blocks ( see the part 3 -- http://improving-bpm-systems.blogspot.ch/2016/08/beauty-of-microservices-part-3-employ.html ) during different phases of the application lifecycle.



Building blocks with the low level of changing frequency may be kept as microservices, services or even be embedded into a monolith. Building blocks with the medium or high level of changing frequency must be kept as microservices.

Thus the latter can be changed quickly and safe (without damaging the running application) because each new version of microservice assemble is a different process-template (a formal description of the process). Each process-template may be enacted as several process-instances.

The distinction between process template and process instance is very important. The lifecycle of process-template is controlled at design-time. The lifecycle of process instance is controlled at run-time. A process instance is created, maybe suspended & resumed and finally terminated. Many process instances from various templates may co-exist at the same time as shown below.



3 Conclusion


Microservice architecture, together with BPM, resolves two fundamental problems of the agile development:
  1. building blocks (which are discovered by BPM) are to be implemented à la agile development and 
  2. some building blocks are to be implemented as microservice thus enables agility of enterprise systems (quick and safe changing of production applications).
Thanks,
AS
Posted by Alex at 14:27 0 comments
Labels: , , ,

2016-08-24

Beauty of #microservices: part 3 employ #BPM to tame the service granularity beast

1 Introduction


This blogpost is inspired by several blogposts about microservices and it is based on the blogpost [REF1] “Architecting #cloud-friendly application architecture #apparch (inspired by #microservices)” http://improving-bpm-systems.blogspot.ch/2015/04/architecting-cloud-friendly-application.html which uses other blogposts about microservices http://improving-bpm-systems.blogspot.ch/search/label/%23microservices

See also the previous blogposts of “Beauty of #microservices” topic.

2 Service granularity


A recent article from Zhamak Dehghani of ThoughtWorks https://dzone.com/articles/taking-microservices-home says “Service Modeling is Difficult” and “Find your Granularity”.

Of course, the service granularity is difficult, because it is usually done bottom-up, i.e. in the solution-space without systematic understanding of the problem-space.

What could be a good enough description of a particular problem-space? Ideally, the following is necessary to analyse objectively a particular problem-space:
  1. An domain-of-interest (e.g. business unit, group of processes, etc.) whose unsatisfactory behaviour is qualified as a problem.
  2. An explicit and machine-executable model of this domain-of-interest.
  3. Data those are sufficient to simulate the behaviour of the domain-of-interest with its model.
To build an explicit and machine-executable model of a particular domain-of-interest, this domain-of-interest must be considered as a system. Considering the following definition of the concept ‘system’ “set of interacting discrete parts organized as a whole which exhibits (as the result of interaction between the parts) some emergent characteristics indispensable to achieve one or more stated purposes”, those “discrete parts” have to be identified first. In other words, it is mandatory to find something that will allow to state: “this domain-of-interest is a system of ”.

Typical business artefacts that are used to describe a particular business are: capabilities, functions, services, processes, roles, data, etc. Is it correct to state that a domain-of-interest is a system of capabilities? System of functions? System of services? Not sure – they (capabilities, functions or services) can be explicit, but they can’t form a machine-executable model to simulate the behaviour of domain-of-interest.

So far, it is correct to state that a particular domain-of-interest is a system of business processes ( see http://improving-bpm-systems.blogspot.ch/2014/03/enterprise-as-system-of-processes.html ).

Each business process is an assembly of the following artefacts: events, human activities, automated activities, services, processes, data, documents, rules, roles, records (audit trails), reports and KPIs. All those artefacts are designed in accordance with their domain because they are specific for a particular process or a group of processes. Thus, domain-driven design is automagically followed.

For example, data structures comprise some data necessary for a particular activity and those data are sub-sets of an enterprise data model (extra blogpost will be written on this). Roles are around a particular activity – who can execute this unit-of-work, who can supervise it, who can consult it, etc. Those roles are composed from the enterprise-wide organisational and functional roles.


All those artefacts are units-of-functionality with single responsibility.

All those artefacts have well-defined interfaces.

The majority of those artefacts are stateless.

Thus there is only one step away from the microservice architecture – just wrap those artefacts as services and deploy them separately.

Some of those artefacts in some BPM-suite tools are already microservices! In other BPM-suite tools, some of those artefacts are in the same JVM (see “programmable monolith” in http://improving-bpm-systems.blogspot.ch/2015/12/typology-of-platforms.html ).


3 Conclusion


The use of BPM (which is a top-down analysis of a domain-of-interest) resolve the following difficulties of the microservice architecture:
  • granularity of microservices is defined in accordance with the domain-of-interest, thus “domain-driven design” is achieved
  • relationships between microservices become explicit and they are known at the design-time
  • the number of relationships between microservices is reduced thus complexity is reduced as well

Thanks,
AS
Posted by Alex at 14:20 0 comments
Labels: , ,

Architecture is about WHY

Yet another definition of architecture which may be more understandable by some audiences.

MASTER definition

architecture, noun
totality of fundamental concepts or properties of a system in its environment embodied in its discrete parts and relationships, and in the principles of its design and evolution

Another definition

architecture of a system-of-interest, noun
description why this system-of-interest is designed as shown in its blueprint

Such a description must be
- explicit
- computer-executable (i.e. formalised)
- multi-viewpoints
- understandable by all the stakeholders of the system-of-interest

A blueprint of a system-of-interest is only one of the architecture description viewpoints.

Thanks,
AS


Posted by Alex at 08:49 0 comments
Labels:

2016-08-19

Better application architecture (#apparch) with #microservices and #BPM (as APaaS)

Please, see it below







Thanks,
AS
Posted by Alex at 15:40 0 comments
Labels: , , ,
Subscribe to: Posts (Atom)