Modern enterprise integration patterns on AWS
During my work, I often face situations when systems I integrate with are placed in a data center, the team that manage the system is outsourced but there’s a high number of features to deliver and as always, the time is limited. In some situations the answer is to use a cloud provider.
This post describes one of most common integration mechanisms at an enterprise although just an overview is provided, a thorough analysis would require a book (or many books).
A tale of enterprise systems optimization
What to do when a simple integration between two systems requires teams of developers that work in a waterfall approach? It may take half a year before any setup for testing (or even just production in some cases) will be established. Clearly, more agile approaches are necessary.
It’s an unfortunate characteristic of an enterprise that competencies are dispersed. Often, a team may perform work that has little to no value outside an organization, yet an enterprise created such a configuration as a mean to optimize their internal processes, an interesting anthropological curiosity.
To break the established norms, innovate and enable an organization to move forward while keeping potential risks low, the working teams must be supported with more means like well equipped technology that allows to successfully drive the integration efforts.
Enterprise architects more and more often are getting aware that the best process to perform the modernization is to reach out to offerings of cloud providers. That’s good but they usually limit themselves to tell “use [a cloud provider] for an integration”, that hopefully is something better that already forgotten ESB products popularized by salespeople in SOA systems era. To get to know more about that era, it’s good to watch timeless Does My Bus Look Big in This?.
How then to select the best possible approach, that would deliver value quickly and help the enterprise to grow?
The patterns below cover mostly ones defined in EIP or microservices.io, in those cases will make references there but usually the cases will be too detailed to combine in one pattern.
Let’s also consider for the simplicity the scenario when you have just two departments in an organization or two organizations that use their own AWS accounts (Account A and Account B) and our task is to setup a single integration point between them. We don’t consider whether those accounts are in the same or different organizations.
Because I’m proficient in AWS cloud, the patterns defined here will be only applicable to this provider although it doesn’t mean other providers don’t have similar capabilities.
Authentication and authorization
In order to perform a function, each and every API call to AWS must be signed with credentials that represent an identity and that a policy evaluation logic will allow to perform an action.
Let’s consider a scenario when a resource wants to make a call from Account B to Account A. Account A must configure proper IAM resources to accommodate that.
Delegation with a user
Let’s look at the first possible access delegation:
Operations from Account A configure an IAM User with a programmatic access and create an IAM Policy with permission to access a resource and associate it with a user. Account B simply uses the credentials provided by Account A.
Although this is simplest scenario, it is not recommended approach. The best practice is to use IAM Roles if the resources that perform programmatic access are running inside AWS. In the diagram above, Compute resource runs within an AWS Account B.
This approach works best when the compute resources aren’t running inside AWS network.
When using this approach, it’s recommended to perform key rotation.
Delegation with a role
In some cases, it is possible to delegate access to a role in another account. Let’s cover this scenario here as well.
In this scenario Operations in Account A configure a resource policy that allow a role created by Operations in Account B to perform an action on a resource in Account A.
The compute resource needs to be allowed to use a role within Account B but all the performed actions are towards a resource within Account A.
Although this is a simple scenario, it is applicable only to the following services:
As a bonus, it’s possible to define a policy principal as not only an IAM role but also whole AWS account, AWS services, and others.
Assuming a role across accounts
STS AssumeRole is undoubtedly one of most critical actions on AWS. It allows not only users to assume a role but also allow roles from one account to assume a role in another account.
An IAM Role can can be associated with two policies:
- Permission policy — to define access to a resource
- Trust policy — to define who you trust to assume the role
Although there is a lot of possible configurations of trust policies, let’s consider one usual scenario:
Operations in Account B needs to create a role and make a Compute resource to assume a role.
Once done, Operations in Account A needs to:
- Create a role with a permissions policy to allow access to a resource
- Receive a Role ARN in Account B and configure a trust policy in Account A’s Role to allow the other account to perform AssumeRole action
Once configured, Operations A need to send a Role ARN to Account B Operations, so that they can configure their compute resource, so that it will assume a role from another account to perform an action.
Same as before, a policy principal associated with a role doesn’t need to be another IAM Role.
Messages integration
What is messaging and why should I apply it at my project? As EIP mentions:
An enterprise has multiple applications that are being built independently, with different languages and platforms. The enterprise needs to share data and processes in a responsive way.
How can I integrate multiple applications so that they work together and can exchange information?
Use Messaging to transfer packets of data frequently, immediately, reliably, and asynchronously, using customizable formats.
Messaging has been used successfully for years at almost every enterprise to integrate systems. Nowadays, tools build on top like Kafka or Pulsar are used more often at modern organizations to support their processes.
Although AWS provides multiple services that enable enterprise integration, like Amazon MQ let’s consider cloud native services to help organizations integrate. Let’s also consider point-to point messaging, publish subscribe should be more considered as an internal integration.
Cloud native messaging
In order to provide an integration that ensures decoupled availability of the integrating parties, you can make use of SQS.
What if you want to enable one-way communication of compute resources located in separate accounts? Just fan out your messages. That can be done in multiple ways. Here’s one possible way:
Account A can setup a topic that can receive events from a Compute resource. Then, You can subscribe a queue in Account B to receive messages from that topic. This setup will allow having multiple subscriptions and with filter policies, you can configure what messages each subscriptions will receive.
You can apply any of the previously described authentication method along with this integration.
We all know that reality is more complicated, often Operations in Account A might be more efficient than in Account B. What to do in that case? It’s always possible to adjust the diagram and make most of the setup in Account A, like shown below:
This setup is still valid, Compute resources in Account B can make use of delegation with a role pattern to be able to work with the messages.
Files integration
Related services: S3, AWS Transfer Family
Plain old files
File exchange has been and still is one of most successful mechanisms of data exchange. Its simplicity and powerfulness comes with a fact that almost everyone is familiar with a concept of a computer file, which is simply a resource for recording data.
This popularity makes integration with this resource fairly simple as people with little to no computer expertise can design and often configure relatively powerful integrations.
That’s why it’s important to know how to design such an integration with a cloud and how it can actually be simpler to setup file integration in that way rather than with traditional approach — an (S)FTP server + cron.
Let’s consider now a scenario when a resource in a corporate data center wants to upload a file to an SFTP server.
Have you even faced a situation when you were trying to integrate with an SFTP server but it was configured in an uncommon manner? I did for one enterprise, the integration delayed over 4 months because of one configuration parameter that was not communicated to a team that was connecting to the server.
With cloud providers, such scenarios are no longer possible as you simply configure an SFTP service with well defined configuration parameters.
What’s more, the data can be saved directly on S3, where you can configure notifications when a new file arrives. You can configure an SQS queue, SNS topic or just go easiest way and make S3 invoke a Lambda function.
Modern objects
The scenario above can be simplified if Account A is in AWS. You can make use of many ways to grant access to S3 resources.
On top of that, Account B can have its notification logic, as shown below:
Why should you use S3 objects? It’s considered file system for the cloud, is designed for high durability and availability. Working with S3 is different than with regular file systems, as the data exchange protocol is HTTP and not DMA and SATA but enable you to share the data in an efficient fashion (see data lake).
Web integration
Related services: API Gateway, AppSync, ELB
Synchronous communication is often needed when reacting to client requests. because of that, below are examples of how this can be implemented.
Basic API Gateway
API Gateway or Backend for Frontend (BFF) is a common phrase for a software that acts as an entry point for all clients. In that way global functionalities like authentication can be performed. Because AWS Api Gateway is highly integrated with other AWS services, even IAM authentication can be configured.
In the example below, Compute resource in Account A doesn’t need to be within a cloud, it can be anywhere.
API Gateway helps with documentation efforts as well as it is possible to download Open API documentation and even generate skeleton of an SDK in several languages.
GraphQL API
It’s relatively new type of API, focuses on data and actions rather than endpoints but been for a while and the tooling works quite well to establish in longer living projects.
A think to stress is that, because GraphQL focuses on data structures, it is a good interface to define complex APIs in an efficient way, which would not require 500 pages of Word documents or MBs of WSDL/Open API files. It will result in decreased time to deliver an integration.
In the example below, Compute resource in Account A doesn’t need to be within a cloud, it can be anywhere.
On top of that, AWS provides Amplify Framework for the clients that use AppSync as GraphQL interface, which makes the integration simpler.
Plain load balancing
If you want to expose compute resources on the cloud, to ensure High Availability, the most common solution is to make use of load balancers. AWS provides three types of Elastic Load Balancers (ELBs):
- Application Load Balancer (ALB)
- Network Load Balancer (NLB)
- Gateway Load Balancer (GLB)
- Classic Load Balancer (deprecated)
Although NLB and GLB are powerful and useful tools, it will not be discussed further here as they’re being used in more advanced scenarios.
In the example below, Compute resource in Account A doesn’t need to be within a cloud, it can be anywhere.
Wrap up
That’s it when it comes to some of the most common integration patterns. With them, it’s possible to setup quite powerful integrations and with next posts, I will try to dive deeper into some of the integrations.
