Post-platform enterprise pattern: EEYORE

Post-platform enterprise pattern: Enterprise-Enterprise Yet Open Robotic Environment (EEYORE)

With thanks to John Morris (@JohnHMorris) and Michael Poulin (@m3poulin) for their valuable comments.

An updated version of the article is available at https://bpm.com/blogs/post-platform-enterprise-pattern-faster-and-cheaper-inter-enterprise-ecosystem-business

WHY

In the modern business world, the vast majority of work is done by joint efforts of two or many enterprises. Inter-enterprise “working together” is a norm right now. It may have different longevity: one-off or sporadic or permanent (as B2B partnership). Can such “working together” be used systemically? For example, a modern enterprise is trying to get competitive advantage by

1. perfecting, digitizing and innovating its core-business capabilities, and 
2. using the best “other” enterprises for provisioning “other” capabilities needed to achieve the particular goal (even its mission); note that, those “other” capabilities for the enterprise point of view are the core-business capabilities from “other” enterprises point of view. 

In other words, an enterprise may combine its own “internal” capabilities with “external” capabilities obtained from other enterprises to achieve the particular goal.

Thus every enterprise involved is carrying out only its core-business capabilities! This is a big difference with the current situation in which some of enterprise’s capabilities (i.e. core-business) must compete in the world-wide market and some of enterprise’s capabilities (supporting) have no competitor at all.

The economist Ronald Coase - said that “Firms exist when the transaction cost of doing something within the firm, even with all its overhead, is lower than cost of doing things through a marketplace of free agents”. At present, an enterprise possesses core-business capabilities and supporting capabilities. Historically, the cost of intra-enterprise transactions is lower than the cost of inter-enterprise transactions. At the digital age the costs of inter-enterprise transactions are constantly dropping as digital technology develops (the so-called "API economy" is a good example of this). Of course, the cost contracting out versus direct management is only one of many factors such as risks, time lag, security, etc. to be taken into account.

Thus, the famous classic representation of an enterprise may be redrawn as shown in the illustration below. Note: Also logistics, marketing and sales may be considered as non core-business capabilities and be provided by “other” enterprises.

Ability of several enterprises to work together with maximum synergy and minimum overhead (thus much better than a classic enterprise) is critically important. Potential list of common activities to achieve that is not long but daunting:

• formally define a work to be done ; 
• find the best other enterprises for this work to be done; 
• contract selected other enterprises; 
• activate and configure a trusted working environment for all participating enterprises; 
• carry out the work with secured sharing some data and information among participating enterprises and 
• complete all contracts related to this work to be one. 

Certainly if modern digital technologies are properly architected together then they can enable this new way of working by making it more efficient and more effective that the current way of working.

Explicit, formal, machine-readable and machine-executable processes can act as a neutral and natural referee for coordinating inter-enterprise work. A potential implementation is an BPM-suite tool. A target is to position a BPM-suite tool as an independent and trusted 3rd party (a referee or a coordinator) to execute legally-bound processes for 2 or more mutually non-trusted parties. See http://improving-bpm-systems.blogspot.com/2016/07/digital-contract-as-process-enables.html

Of course, when many enterprises contribute into some common work, records management must be centralized (i.e. accepted by all the participants) and suitable for digital way of working. A digital archive with good availability and integrity can act as an escrow for some transactions and documents. Potential implementation is the blockchain technology.

An example of such a “disaggregation” of a classic enterprise – https://news.cgtn.com/news/35636a4e33637a6333566d54/share_p.html

HOW

Working together for the same goal requires sharing some things (business and technical artefacts) and, probably, some agreed means. Such a sharing is lesser for cooperation (different enterprise for different activities) and greater for collaboration (different enterprises for same activities).

Imagine that 10 enterprises have decided to work together. Shall each of them implement business processes required for this engagement? Add complex EDI infrastructure for communicating between processes from different enterprise? Keep everything within each enterprise? It looks a bit difficult. If something that must be shared is done once for everyone then everyone will gain. Thus some processes related to this engagement may be implemented once at an independent provider and linked to existing processes in each enterprise.

 

WHAT to share	Examples	Reason(s)	Supporting means
Data (see the DIKW pattern)	hashes, cryptographic keys	Security	common immutable storage
Information (see the DIKW pattern)	business objects	Interoperability, e.g. in interfaces	common immutable storage
Facts	audit trails, signed documents, agreed goals, demonstrated KPIs, inputs & outputs of inter-enterprise transaction	Traceability	common immutable storage
Events	some facts important for business	synchronization of work	common event queue
Rules	agreed business logic	regulation of work	common decision mechanism
Activities	agreed procedures, agreed SLAs, agreed protocols	automation	common scripts in domain-specific languages
Processes	agreed testing methods	coordination of work	common coordination mechanism
KPI calculations	agreed formulas and results of calculations	common view on performance	common dashboard
Intelligence	agreed algorithms and results	common view on planning	common prediction mechanism
Improvements	some proposals for better working	better joint performance	common set of digital models and agreed scenarios
Estimations of uncertainty which are evolving along the time	participants’ opinions on some factors	non-biased corporate security for all the enterprises involved	common immutable storage
Estimations of risk which are evolving along processes	participants’ opinions on adverse impacts	non-biased corporate security for all the enterprises involved	common immutable storage
Finances	payment mechanism	minimizing transactions with banks	own “local” currency

Things, which are shared in the particular case, must be a) architected together and b) digital (explicit, formal, machine-readable and machine executable) to achieve desired values of the important emergent characteristics such as:

• the transactional cost for inter-enterprise transactions, 
• risks, 
• time lag, 
• security, 
• manageability, 
• etc. 

WHAT

New application architecture for inter-enterprise agile engagements can be built on the following pillars:

• Immutable common digital archive (FACTS) to store all versions of all the artefacts. 
• Microservices (ACTIONS), potentially in a serverless computing environment (because a digital archive is used as a common immutable storage). Please note that microservices are normal services (according to OASIS) except that one unit-of-functionality is one-unit-of-deployment is one-unit-of-execution. 
• Machine-executable processes (COORDINATION). 

These pillars perfectly work together (in a robotic way without high level of creativity) – process templates define the granularity and interfaces for microservices and digital archive keeps all the artifacts (of course, they must be versioned). Process instances define validity of various facts.
See also http://improving-bpm-systems.blogspot.com/2018/06/architecting-modern-digital-systems.html

Some hurdles – processes modelling

Modelling of inter-enterprise processes is not easy because such processes are distributed (each of them may be executed in its own computing environment) and they communicated by exchanging of messages thus it is necessary to handle exceptions in communicating processes. Good modelling styles and process patterns will help.

Some hurdles – process execution

Some BPM-suite tools are already connected to the blockchain which is used as a common immutable storage.

1) Ultimus can use blockchain as a data storage - "Ultimus uses Tierion to create an audit trail for business processes and prove the integrity and timestamp of documents." See https://medium.com/tierion/ultimus-integrates-tierion-for-blockchain-digital-process-automation-f8331d76216a

2) Bonita can use blockchain – https://www.youtube.com/watch?v=lkwvko2Uy24

3) ConsenSys uses Camunda BPM-suite to use blockchain as a data storage, creating new users and executing smart-contracts https://www.youtube.com/watch?v=oww8zMzxvZA&feature=youtu.be
Some hurdles – process execution as a smart contract

One of the option to bring process to existing blockchain implementations is to implement a BPMN-like interpreter as a smart contract in the blockchain technology.

Some hurdles – blockchain mentality

People from the blockchain domain follow the “if you have a hammer then everything is a nail” negative pattern. https://blog.apla.io/report-of-the-egaas-team-556e5e4717bd

See http://ipe-lab.com/publication/223/?sphrase_id=41 as a case of the blockchain for control of third-party services. This an attempt to change a group of processes (from a few B2B partners) into a blockchain application. On the bottom picture (TO-BE), actual processes from the top picture (AS-IS) were lost. And, “Erroneous behaviour is judged by voting among all participants”, not by thorough analysis!

Conclusion

This pattern opens new perspective for the traditional Business Process Management which becomes also Inter-Enterprise Ecosystem Business. In other words, processes will expand outside enterprises to enable faster and cheaper “working together” for enterprises. 

And a quote from Alberto Manuel‏ (@AlbertoManuel) "Architecting expanded Value chain in the era of digital ecosystems and shared capabilities."

Thanks,
AS

Architecting modern digital systems #entarch #bizarch #apparch #bpm #security #microservice

Mini-course at VFU

1 Title

Architecting modern digital systems

2 The problem to be addressed

At present, there are many IT-related methodologies, technologies, tools and schools of thoughts which overlap and contradict each other. The best practices are actually the best only in particular situations. Often decisions about software-intensive solutions are taken on the in-complete and subjective base. All of this tremendously complicates the modern digital systems thus reducing their potential effectiveness and efficiency. 

3 Objectives

The purpose of this course is to provide the basic knowledge and experience necessary to better understand how to deal with the increasing complexity of the information technologies to obtain the synergy between business needs and IT potentials. 

4 The approach

The course is based on the practical use of Enterprise Architecture (EA) which a methodology and practice for architecting solutions. EA provides an overarching guideline for understanding a “problem space” and take necessary decision about the “solution space” to deliver which addresses the problem. 

5 Learning outcomes

The trainees will

• learn a systematic approach for architecting digital solutions; 
• learn about some modern information technologies; 
• learn how those technologies are working together for a systematic architecting, design, implementation, operations and evolution of digital systems; 
• carry out a practical architecting exercise. 

6 Target audience

Bachelor and master level students specialising in IT. 

7 Requested knowledge

General knowledge of IS/IT. General programming experience. 

8 Layout of the teaching

Teaching will be given as six 1.5-hour lectures. The first 4 lectures will be about presenting some methodologies and technologies. Then the students will be asked to architect a solution based on a practical situation. The last session will be about presenting the student’s solution and wrapping up this mini-course. 

9 Agenda

• Introduction https://www.slideshare.net/samarin/minicourse-at-vfu-architecting-modern-digital-systems-0 
• Architecting digital systems https://www.slideshare.net/samarin/minicourse-at-vfu-architecting-modern-digital-systems-1 
• Business Process Management https://www.slideshare.net/samarin/minicourse-at-vfu-architecting-modern-digital-systems-2 
• Security https://www.slideshare.net/samarin/minicourse-at-vfu-architecting-modern-digital-systems-3
• Microservices https://www.slideshare.net/samarin/minicourse-at-vfu-architecting-modern-digital-systems-4
• Practical work – architecting a solution https://www.slideshare.net/samarin/minicourse-at-vfu-architecting-modern-digital-systems-5
• Review of the practical work 

Thanks,
AS

Beauty of #microservices - from #DevOps to #BizDevOps via #microservices first

As we all know, usage of MicroService Architecture (MSA) requires the very comprehensive operational practices and infrastructure. A microservice is a unit-of-functionality (or “class” in the informal IT terminology) within its own unit-of-deployment (or “component” in the informal IT terminology) acting as a unit-of-execution (or “computing process” in the informal IT terminology). Some applications may comprise a few hundred of microservices. This is certainly a serious barrier for exploiting MSA benefits such as easy to update and easy to scale to absorb heavy workloads.

Fortunately, as we know, various performance characteristics (e.g. easy to update, easy to scale) are not spread uniformly within applications. For example, 95% of CPU consumption is located in 5% of program code. Thus, it is not necessary to implement the whole application via microservices.

Let us ask a simple question, if a microservice is, actually, a service then can we use microservices and services together? Yes, and some functionality from platforms or monoliths may be used (via API) as well.

Now, let us reformulate the problem. Let us consider that any application is built from many units-of-functionality which must be deployed and then executed. What is the optimal arrangement of units-of-functionality into units-of-deployment and then units-of-execution? In other words,

• which units-of-functionality have to be implemented as microservices (microservices are agile and good for easy to update, but have some execution and management overhead);
• which units-of-functionality have to be implemented as monoliths (monoliths are not agile and not easy to update, but have no execution and management overhead);
• which units-of-functionality have to be implemented as services (classic services are something in between microservices and monoliths).

Thus, a few recommendations may be formulated.

• Units-of-functionality which are “often” updates must be implemented as microservices (so BizDevOps will be happy).
• Units-of-functionality which require to absorb heavy workloads must be implemented as microservices (so DevOps will be happy).
• Units-of-functionally which are “rarely” updated may be packed in a few units-of-deployment (different “packing” criteria may be used) and each unit-of-deployment has its own computing process (so DevOps will be happy). Another option is dynamic loading of those units-of-functionality.
• Units-of-functionality which are “never” updated may be packed as a monolith or platform, i.e. one unit-of-deployment and one unit-of-execution (so DevOps will be extremely happy).

Applying these recommendations to some phases of the whole application life cycle (conception, development, deployment, production, support, retirement and destruction) the following recommendations may be formulated:

• At the beginning of the application life cycle (concept, i.e. prototyping, and initial development), the majority of the units-of-functionality must be implemented as microservices, because easy to update characteristic is very important (especially for the business people) and, fortunately, performance characteristics are not an issue. 
• More close to the end of the development phase, it becomes clear which units-of-functionality have to changed more often than others; so those others may be considered as services and even monoliths or platforms.
• Also, the load tests (during the development and deployment phases) must show which units-of-functionality will require to absorb heavy workloads thus to be implemented as microservices.
• Other criteria may be considered as risk, security, etc. 

Obviously, that “moving” a unit-functionality from microservice-like implementation to service-like implementation and to platform-like implementation is much easier that “moving” a unit-of-functionality from monolith-like implementation to service-like implementation and to microservice-like implementation.

This confirms the primacy of the “microservices first” approach. This approach, actually, provides support for BizDevOps practices ( see http://improving-bpm-systems.blogspot.ch/2017/05/beauty-of-microservices-ebanliing.html ). Additionally, this approach enables interesting transformations such as automatic reconfiguration of applications to absorb the heavy workloads by moving temporarily some units-of-functionality from service-like implementation to microservice-like implementation.

Remember from prof. Knuth "Premature optimisation is the root of all evil".

Thanks,
AS

The collection of posts about microservices - http://improving-bpm-systems.blogspot.ch/search/label/%23microservice 

Beauty of #microservices - making them practical

The classic definition of the microservice architectural style “as an approach to developing a single application as a suite of small services, each running in its own process and communicating with lightweight mechanism” creates a lot of fears and misunderstandings:

• Application monoliths are evils, but having too many microservices sounds like creating an (unknown) evil as well.
• Everything has to be re-developed.
• Microservices will create a huge backlog for our agile team.
• Microservices? They are neither architecture nor architectural style – just a technical stack.

As usual in IT, any new technology or methodology (which pretends to revolutionized everything) must be used together with many existing ones. Let us “intermix” MSA with some existing and proven technologies and methodologies.

MicroService Architecture (MSA) bring two major concepts:

1. microservice as a unit-of-functionality, unit-of-deployment and unit-of execution with the same boundaries, and
2. assembling a whole application from microservices of different origins: off-the-shelf (commercial and FOSS), brought, rented, built, provided from SaaS, PaaS, APaaS, etc.

Using these two concepts, let us try to find a practical balance between monolith architecture and MSA.

Firstly, it is necessary to think about any application as a set of the following artefacts

• Events
• Roles (actually, access rights management)
• Rules (or decisions)
• Business objects – data structures
• Business objects – documents
• Human activities (or screens or interactive services)
• Automation activities (or scripting fragments or automation services)
• Coordination
• Audit trails
• KPIs
• Reports

Secondly, consider that each artefact must be, ideally, handled

• Explicitly
• As a set of microservices
• Via APIs
• With versioning 
• By a specialized OTS tool, e.g. data structures are handled by a database, processes are handled by a BPM-suite tool
• In a Domain Specific Language (DSL), e.g. BPMN for processes, DMN for rules
• Over its whole life cycle

Thirdly, understand specialised tools for that each artefacts:

• Coordination as explicit and machine-executable processes via a BPM-suite tool
• Roles via an access management tool
• Documents via an ECM product
• Automation fragments as scripts in an interpretive language and execution robots
• Audit trail and reports via BI tools
• etc.

Fourthly, prepare two common “pool” for future tools, services and microservices:

• technological pool for generic off-the-shelf products; their functionality is available via APIs
• enabling pool for services, microservices, tools which are a) specific for the particular organisation and b) potentially reusable within organisation; their functionality is available via APIs

For each monolith application, sort its functionality out into 2 common pools and an individual pool.

At the result, we got a corporate unified business execution platform which standardise and simplify core elements of the corporate-wide computing system. For any elements outside the platform, new opportunities should be explored using agile principles. These twin approaches should be mutually reinforcing:

• The platform frees up resource to focus on new opportunities while successful agile innovations are rapidly scaled up when incorporated into the platform.
• An agile approach requires coordination at a system level.
• To minimise duplication of effort in solving the same problems, there needs to be system-wide transparency of agile initiatives.
• Existing elements of the platform also need periodic challenge. Transparency, publishing feedback and the results of experiments openly, will help to keep the pressure on the platform for continual improvement as well as short-term cost savings.

Obviously, do not forget to the a good application architecture - http://improving-bpm-systems.blogspot.ch/2017/05/beauty-of-microservices-ebanliing.html and http://improving-bpm-systems.blogspot.ch/2016/08/better-application-architecture-apparch.html

Thanks,

AS

Other blogposts about microservices - http://improving-bpm-systems.blogspot.ch/search/label/%23microservices

Beauty of #microservices - enabling #BizDevOps culture

Everyone heard about the DevOps culture which refers to a set of practices that emphasize the collaboration and communication of both software developers and IT professionals while automating the process of software delivery and infrastructure changes.

Certainly, DevOps improves the time-to-market for digital solutions, but it spans only a down-stream part of the idea-to-solution value stream. To cover the whole value stream, any up-stream stumbling blocks must be removed.

An application architecture, which is built on microservices and their machine-executable coordination (e.g. by processes), enables a new BizDevOps culture by quick implementations of business ideas. (I think that ING has introduced the BizDevOps concept).

Microservice is a service with the same boundaries as

• a unit-of-functionality (for Biz)
• a unit-of-deployment (for Dev)
• a unit-of-execution (for Ops)

Thus, an implementation of a business idea as a group of microservices will have no unnatural complexity and, therefore, its time-to-market will be short.

See http://improving-bpm-systems.blogspot.ch/2016/08/better-application-architecture-apparch.html

Thanks,

AS

Other blogposts about microservices - http://improving-bpm-systems.blogspot.ch/search/label/%23microservices  

Beauty of #microservices - adding ASSEMBLE to BUILD or BUY or RENT

MicroServices Architecture (MSA) is changing the classic Buy or Build or Rent question.

With MSA, orgnisations may use an extra option - Assemble. This allows an organisation to select the best option for each service or microservices. For example:

• Buy services and microservices from organisation's business partners
• Build microservices for organisation's unique capabilities
• Rent services and microservices from the commodity markets
• Assemble all services and microservices into organisation's unique solutions

Thanks,
AS

Other blogposts about microservices - http://improving-bpm-systems.blogspot.ch/search/label/%23microservices

Beauty of #microservices - #microservice architecture maturity model

This blogpost is inspired by the discussion https://www.linkedin.com/feed/update/urn:li:activity:6266622261210411008/

I can imagine an MSA maturity matrix:

1. atomic microservices 
2. compound microservices (a microservice with wide single responsibility is built from microservices with narrow single responsibility) 
3. data-entry applications in MSA (interactive microservices, validation microservices, persistence microservices) 
4. process-driven applications in MSA (coordination microservices, management microservices, etc.) 
5. all former applications in a value-chain are in MSA(actually, no boundaries between applications)
6. all former applications within an organization are in MSA
7. all former applications in a supply-chain are in MSA
8. all former applications in an ecosystem are in MSA

Please, see my blogposts on #microservices http://improving-bpm-systems.blogspot.it/search/label/%23microservices 

Thanks,

AS

Beauty of #microservices: part 9 explicit coordination as a microservice

1 Introduction

This blogpost is inspired by several blogposts about microservices and it is based on the blogpost [REF1] “Architecting #cloud-friendly application architecture #apparch (inspired by #microservices)” http://improving-bpm-systems.blogspot.ch/2015/04/architecting-cloud-friendly-application.html

See also the previous blogposts of “Beauty of #microservices” series

• “Beauty of #microservices: part 1 #entarch #apparch” http://improving-bpm-systems.blogspot.ch/2015/05/beauty-of-microservices-part-1-entarch.html
• “Beauty of #microservices: part 2 architecting for change” http://improving-bpm-systems.blogspot.ch/2015/05/beauty-of-microservices-part-2.html
• “Beauty of #microservices: part 3 employ #BPM to tame the service granularly beast” http://improving-bpm-systems.blogspot.ch/2016/08/beauty-of-microservices-part-3-employ.html
• “Beauty of #microservices: part 4 from agile development to agility of enterprise systems” http://improving-bpm-systems.blogspot.ch/2016/08/beauty-of-microservices-part-4-from.html
• “Beauty of #microservices: part 5 defragmentation of enterprise data model” http://improving-bpm-systems.blogspot.ch/2016/09/beauty-of-microservices-part-5.html
• “Beauty of #microservices: part 6 managing state is a teamwork http://improving-bpm-systems.blogspot.ch/2016/09/beauty-of-microservices-part-6-managing.html
• “Beauty of #microservices: part 6 managing breaking the monolith” http://improving-bpm-systems.blogspot.ch/2016/09/beauty-of-microservices-part-7-breaking.html
• “Beauty of #microservices: part 8 dump-pipes & smart-containers & minimalistic-microservices” http://improving-bpm-systems.blogspot.ch/2016/09/beauty-of-microservices-part-8-dump.html

2 Things work better when they work together, on purpose (from www.tetradian.com )

To be efficient, their work must be explicitly coordinated. Certainly, this is strongly applied to all the microservices comprise an application. Of course, it is considered that an application is a several very loosely-coupled clusters of microservices to be coordinated (for example, each such a cluster is responsible for the lifecycle of a particular business entity).

Although there is an opinion that “Service is not comprised of other services due to the independence requirement” (see https://www2.opengroup.org/ogsys/catalog/W169), it is considered that some (with bigger responsibility) microservices can be assembled from other (with smaller responsivity) microservices.

There are several techniques to implement coordination.

Orchestration

• nature: centralised at design-time and centralised at run-time thus may be explicit
• specific: there is a misconception that it uses only synchronous communication (à la RPC) although it may use also asynchronous communication (à la message-passing)

Choreography

• nature: decentralised at design-time and decentralised at run-time thus implicit
• specific: uses only asynchronous communication (à la message-passing)

Reactive streams and runnable graphs

• nature: decentralised at design-time and centralised at run-time thus implicit
• specific: optimised for high volume event processing

Business-process-based

• nature: centralised at design time and decentralised at run-time thus explicit
• specific: each case is a completely separate instance with its own lifecycle; and the process may be another microservice

3 Implementation of business-process-based coordination

Of course, it should be a DSL to define an explicit coordination (e.g. BPEL, BPMN, etc.). Using the terminology from the section 7 of [REF1], a DSL-processor may act as a specialised container for DSL-scripts. Also, some microservices which are coordinated by a DSL-script may use some specialised containers. For example, a specialised container for human-operations, a specialised container for business rules, a few specialised containers for automated-operations.

4 Conclusion

Some advantages of the business-process-based technique:

• Assembled microservices have no routing logic (thus they follow SRP). 
• All the necessary microservices (assembled and dependent) can be instance-bound (help to predictive analytics).
• All the necessary microservices (assembled and dependent) can be instantiated on demand (this minimises DEVOPS).
• A particular instance may be stopped for the error-recovery without influencing other instances (operational isolation).
• A few versions of the same coordination (i.e. business process) may co-exist (versioning is easy).
• Different instances of the same process and their may be executed on different nodes (linear scaling out).
• Easy to visualise for the business people.

Thanks,
AS

Beauty of #microservices: part 8 dumb-pipes & smart-containers & minimalistic-microservices

1 Introduction

This blogpost is inspired by several blogposts about microservices and it is based on the blogpost [REF1] “Architecting #cloud-friendly application architecture #apparch (inspired by #microservices)” http://improving-bpm-systems.blogspot.ch/2015/04/architecting-cloud-friendly-application.html

See also the previous blogposts of “Beauty of #microservices” series

• “Beauty of #microservices: part 1 #entarch #apparch” http://improving-bpm-systems.blogspot.ch/2015/05/beauty-of-microservices-part-1-entarch.html
• “Beauty of #microservices: part 2 architecting for change” http://improving-bpm-systems.blogspot.ch/2015/05/beauty-of-microservices-part-2.html
• “Beauty of #microservices: part 3 employ #BPM to tame the service granularly beast” http://improving-bpm-systems.blogspot.ch/2016/08/beauty-of-microservices-part-3-employ.html
• “Beauty of #microservices: part 4 from agile development to agility of enterprise systems” http://improving-bpm-systems.blogspot.ch/2016/08/beauty-of-microservices-part-4-from.html
• “Beauty of #microservices: part 5 defragmentation of enterprise data model” http://improving-bpm-systems.blogspot.ch/2016/09/beauty-of-microservices-part-5.html
• “Beauty of #microservices: part 6 managing state is a teamwork http://improving-bpm-systems.blogspot.ch/2016/09/beauty-of-microservices-part-6-managing.html
• “Beauty of #microservices: part 6 managing breaking the monolith” http://improving-bpm-systems.blogspot.ch/2016/09/beauty-of-microservices-part-7-breaking.html 

2 Importance of containers for microservices

This blogpost is inspired by two comments to my blogpost about microservices:

• Igor Topalov “microservices shall be considered in conjunction with containers” 
• Bogdan Năforniţă “considering transactions moves us away from the concept of ‘smart endpoints, dumb pipes’” 

Considering the SRP is one of commonly-agreed characteristics of microservices, the “Smart endpoints and dumb pipes” characteristic is in the direct contradiction to the SRP. Making endpoints (i.e. microservices) “smart” requires that they have to have many various functionality in addition to their “core” functionality. Thus the question is how to allow simplify microservices and thus simplify the life of software developers.

I used several types of nested primitive containers:

generic – JVM on top of any popular OS (experience in programming of portable software helped);

language-specific – Jython on top of JVM to run small Python programs, and

specialised – particular environment on top of Jython on top of JVM; this environment considerably simplified the development of automation and integration functionality.

With each nested container, my microservices became more functional and easier to evolve. Finally, each of them was as small text fragments stored in a source version control tool; they were loaded into containers dynamically (at the run-time) and they could dynamically load some modules. Devops was minimal.

3 Conclusion

• Keep pipes dumb (no logic!).
• Create your own smart containers (maximum housekeeping and specialisation) from some standard ones.
• Help your microservices be functionally minimalistic (thus simplify the life of your software developers).

Thanks,
AS

Beauty of #microservices: part 7 breaking the monolith

1 Introduction

This blogpost is inspired by several blogposts about microservices and it is based on the blogpost [REF1] “Architecting #cloud-friendly application architecture #apparch (inspired by #microservices)” http://improving-bpm-systems.blogspot.ch/2015/04/architecting-cloud-friendly-application.html

See also the previous blogposts of “Beauty of #microservices” series

• “Beauty of #microservices: part 1 #entarch #apparch” http://improving-bpm-systems.blogspot.ch/2015/05/beauty-of-microservices-part-1-entarch.html
• “Beauty of #microservices: part 2 architecting for change” http://improving-bpm-systems.blogspot.ch/2015/05/beauty-of-microservices-part-2.html
• “Beauty of #microservices: part 3 employ #BPM to tame the service granularly beast” http://improving-bpm-systems.blogspot.ch/2016/08/beauty-of-microservices-part-3-employ.html
• “Beauty of #microservices: part 4 from agile development to agility of enterprise systems” http://improving-bpm-systems.blogspot.ch/2016/08/beauty-of-microservices-part-4-from.html
• “Beauty of #microservices: part 5 defragmentation of enterprise data model” http://improving-bpm-systems.blogspot.ch/2016/09/beauty-of-microservices-part-5.html
• “Beauty of #microservices: part 6 managing state is a teamwork http://improving-bpm-systems.blogspot.ch/2016/09/beauty-of-microservices-part-6-managing.html

2 Breaking the monolith

This blogpost is a continuation of two previous ones:

• "Enterprise patterns: eclipse" http://improving-bpm-systems.blogspot.ch/2013/06/enterprise-patterns-eclipse.html
• "How to modernise a legacy ERP" http://improving-bpm-systems.blogspot.ch/2014/04/ideas-for-bpmshift-delenda-est-vendor_27.html

Below, a series of steps to show how to remove from a monolith (actually a home-made ERP) some functionality around a particular Business Entity (BE) or a group of BEs.

At the AS-IS step, the monolith is the master of everything related to this BE (thisBE) – data, rules, processes and events (which are generated during the lifecycle of this BE and may affect other BEs.)

The first step is to externalise and make explicit the process to manage this BE in a BPM-suite tool. Keeping of thisBE (i.e. its data) is externalised as well. Also the associated rules must be externalised (as a copy) to reproduce the business logic spread in the monolith.

The monolith keeps its slave copy of data which are maintained via some stor-API. The associated business logic and event logic are still managed by the monolith. The data (as a slave) must always stay in the monolith because they may be used somewhere.

The second step is to externalise rules (when they are good enough to cover all the existing rules from the monolith).

At the TO-BE step, everything related to thisBE is externalised, but the associated events must be “injected” into the monolith by some func-API.

3 Anti-pattern DOUM

Avoid the "DOUble Master" (DOUM) anti-pattern (section 5.6 of my book "Improving enterprise BPM systems" http://www.improving-bpm-systems.com/book ).

As coordination can be carried out by an application or by a process engine, we have to be very careful to avoid the “double master” anti-pattern. At any moment in time there must be only one master responsible for the coordination of a particular process instance. (Of course, the coordination role may be delegated if appropriate.) This is analogous to a well-organised meeting where the chairperson decides who talks next.

The non-recognition of this anti-pattern can be very costly. We have observed a BPM solution which allowed the modification of data by a process engine, by an interactive application (i.e. by a human) and by a batch at the same time. The coordination of activities was based on data and, if necessary, the application or the batch could “correct” the process. The process engine was used mainly for the handling of three human activities, and the implementation of this solution (for a relatively simple business process) took several man-years.

4 Conclusion

How will you eat an elephant? Piece-by-piece, of course.

Thanks,
AS

Beauty of #microservices: part 6 managing state is a teamwork

1 Introduction

This blogpost is inspired by several blogposts about microservices and it is based on the blogpost [REF1] “Architecting #cloud-friendly application architecture #apparch (inspired by #microservices)” http://improving-bpm-systems.blogspot.ch/2015/04/architecting-cloud-friendly-application.html

See also the previous blogposts of “Beauty of #microservices” series.

• “Beauty of #microservices: part 1 #entarch #apparch” http://improving-bpm-systems.blogspot.ch/2015/05/beauty-of-microservices-part-1-entarch.html
• “Beauty of #microservices: part 2 architecting for change” http://improving-bpm-systems.blogspot.ch/2015/05/beauty-of-microservices-part-2.html
• “Beauty of #microservices: part 3 employ #BPM to tame the service granularly beast” http://improving-bpm-systems.blogspot.ch/2016/08/beauty-of-microservices-part-3-employ.html
• “Beauty of #microservices: part 4 from agile development to agility of enterprise systems” http://improving-bpm-systems.blogspot.ch/2016/08/beauty-of-microservices-part-4-from.html
• “Beauty of #microservices: part 5 defragmentation of enterprise data model” http://improving-bpm-systems.blogspot.ch/2016/09/beauty-of-microservices-part-5.html

2 Managing state is a teamwork

Obviously, a solution or an application implemented with microservices is a set or suite of stateful and stateless microservices. The chapter 7 of REF1 provides a classification of microservices. The stateful microservices are those which:

1. manage some resources
2. provide legacy functionality 
3. assemble (implicitly and explicitly) other microservices

Each stateful microservice must be, ideally, idempotent to contribute to managing state.

Microservices, which manage some resources, may have a few impotency pitfalls to be avoided. For example, the read operation may be not idempotent if concurrent updates are possible. The update (or write) operation may be not idempotent if it can change some metadata, e.g. modification date. Also, idempotency may depend on a particular operation. The safest way is to create a small “shell” to guarantee the idempotency and a unique ID for each invocation.

Also, a small “shell” is only the option for microservices, which provide legacy functionality (they are considered as black boxes).

Microservices, which implicitly assemble other microservices, are a real pain because they have to be carefully reviewed about their idempotency. A possible approach for their idempotency is to re-execute again such a microservice. If all the microservices, which are invoked from it, are idempotent and don’t have any human involvement then such a re-execution will be idempotent as well.

Microservices, which explicitly assemble other microservices, may create some “check-points” (similar to mainframe batch systems) to start their re-execution from the last “passed” checkpoint. Of course, the data associated with checkpoints must be stored somewhere else as records. ( A similar approach can be found in http://www.theidentitycookbook.com/2016/06/blockchain-for-identity-access-request.html )

3 Error recovery (and distributed transactions)

As microservices form a distributed system, the error recovery is very difficult.

Explicit assembly of microservice, e.g. a business process in BPMN, can implement the error recovery in the following way.

Imagine a process fragment with three automated activities (A, B, and C) to be executed as a transaction. Each of those activities is an invocation of a microservice and the normal execution sequence is E2-A-B-C-E4.  Because any of those microservices may fail, this fragment contains the intermediate event E3 to intercept a failure and an activity for Error Recovery Procedure (ERP); the latter may be a human activity.

The first pass (with a failure of activity B ) has the following sequence:

E2-A(done)-B(failed)-E3-ERP

The second pass (with a failure of activity C) has the following trace:

E2-A(already done)-B(done)-C(failed)-E3-ERP

The third pass (with no failures) has the following trace:

E2-A(already done)-B(already done)-C(done)-E4

Activity A was executed 3 times, but it did the real work only at the first time – two other times were ignored because it is idempotent.

An extension with a timeout can be found in http://improving-bpm-systems.blogspot.ch/2014/08/bpm-for-digital-age-shifting.html

This way can be used also for implementing distributed transactions (please, note, some compensation activities may be necessary).

A similar approach was described at http://www.grahamlea.com/2016/08/distributed-transactions-microservices-icebergs/

4 Conclusion

Microservice architecture requires common efforts from microservices to achieve

• state management, 
• error recovery and 
• distributed transactions.

Thanks,
AS

Beauty of #microservices: part 5 defragmentation of enterprise data model

1 Introduction

This blogpost is inspired by several blogposts about microservices and it is based on the blogpost [REF1] “Architecting #cloud-friendly application architecture #apparch (inspired by #microservices)” http://improving-bpm-systems.blogspot.ch/2015/04/architecting-cloud-friendly-application.html which uses other blogposts about microservices http://improving-bpm-systems.blogspot.ch/search/label/%23microservices

See also the previous blogposts of “Beauty of #microservices” series.

• “Beauty of #microservices: part 1 #entarch #apparch” http://improving-bpm-systems.blogspot.ch/2015/05/beauty-of-microservices-part-1-entarch.html
• “Beauty of #microservices: part 2 architecting for change” http://improving-bpm-systems.blogspot.ch/2015/05/beauty-of-microservices-part-2.html
• “Beauty of #microservices: part 3 employ #BPM to tame the service granularly beast” http://improving-bpm-systems.blogspot.ch/2016/08/beauty-of-microservices-part-3-employ.html
• “Beauty of #microservices: part 4 from agile development to agility of enterprise systems” http://improving-bpm-systems.blogspot.ch/2016/08/beauty-of-microservices-part-4-from.html 

2 Unfortunate historical fragmentation of enterprise data

In modern enterprise computing environments, an enterprise data model (as a set of business-entities defined at the enterprise level) is, usually, an utopia. Enterprise data are spread among many existing applications that master some attributes of some business-entities. Also some business-entities are duplicated (usually, partially, by some fragments of them) among existing applications.

Because each of those applications (in-house developed, SaaS-based, PaaS-based, intact COTS, modified COTS) has its own lifecycle, the enterprise data model is far from the reality and complex data integration is mandatory to keep the integrity of the enterprise data.

Each new application has its own data and uses data from other existing applications. Thus, an application data model comprises some views on some business-entities from the enterprise data model. However, in such an application data model only some attributes are under control of the application.

If an application has to use data from several other applications then the evolution of some application may have a destructive effect on some other applications – typical “Château de cartes” anti-pattern.

3 Enterprise data model must be flexibility-driven not legacy-held

Ideally, an enterprise data model must

• easy to evolve (by definition)
• be service oriented so data can be accessed only via API (this its fragments can be joint up together even not sitting in the same database)
• implement total versioning for some business-entities (to cover the enterprise lifecycle)
• cover all the types of business-entities
  • transactional data (your business-specific and business-critical data)
  • reference data (taken from other sources and business-parties)
  • operational data (about how the work has been done)
  • reports
  • analytics data
  • records
  • documents
  • media
  • social

There are several techniques to move to this ideal.

Make some functionality enterprise-wide as a Corporate Unified Business Execution (CUBE) platform (see http://improving-bpm-systems.blogspot.ch/2015/10/enterprise-patterns-peas-example-cube.html ). Thus an individual application comprises only business-specific functionality. For example:

• security management (identity, authentication and authorization)
• content and knowledge management
• software factory
• reporting and analytics
• business process management
• records management
• etc.

Make explicit data update processes – see the pattern “Practical process patterns: Synchronisation Of Sources (SOS)” http://www.slideshare.net/samarin/practical-process-pattern

Use some specific characteristics of your enterprise.

4 Adding flexibility

The classic “point-to-point” scenario is that a solution can read some data from some applications via storage-centric API or asset API or stor-API. Actually, those data are application-specific views on some enterprise business entities, e.g. “ClientERP BE” is the business entity CLIENT as it is defined in an ERP. Of course, in this sensation, the solution 1 strong depends on the evolution some other applications.

Another classic “data-access-layer” scenario is to use a microservice aggregate all the application-specific views in a read-only business entity.

The ability of microservices to have their own persistence store (which may be just a table in a common database) is very useful to implement a small extension of an business-entity. The same technique may help to avoid customisation of COTS products. This is a safe way to mode to an ideal enterprise data-model.

With some efforts (mainly for externalising some business logic), read-only APIs may be transformed into read-write APIs. If a business-entity uses more than one underlying applications then a “data update process” is necessary to implement a multi-phase commit.

If it is possible to externalise the business logic then some functionality-centric API or business API or func-API may be implemented.

5 Conclusion

With the use of microservices and other technologies, an enterprise data model may:

• become always up-to-date and
• evolve with the speed of agile development.

Thanks,
AS