The article is a brief introductory description. Kraken combines the ideas of two other projects — ALF and Octopus, so it might be difficult to understand all the details from the article. I’m focusing here more on the philosophical part of the idea. One can find more details in the corresponding GitHub repos: ALF, Octopus, and Kraken. Also, there is also a video from Berlin Elixir Meeetup.

Kraken

Kraken is a general framework for the orchestration of software systems.

It suggests the flow-based programming approach for organizing the interaction between the parts of the system (specific services).
It means that every interaction with the system is presented as an “input event” — the data structure that contains all the necessary information about the interaction. The event then goes through a predefined chain of “components” (“pipeline”), the components call underlying services and modify the event. The final state of the event represents the result of the interaction.

Kraken provides a simple declarative JSON DSL for the pipeline definitions as well as for the definitions of the “services” — clients to underlying services in the system. The interaction with the Kraken-based agent (orchestrator, or control panel) is also carried out via JSON API.

Kraken-based Control Panel in a system

When thinking about layered architecture (infrastructure, domain, application, and interface layer), Kraken is like Kubernetes for the application layer of the system. While K8s provides a declarative approach for defining the infrastructure layer (the one below the domain layer), Kraken does the same for the application layer of the system (orchestration layer).

In terms of the building blocks for the pipelines, Kraken has the same set of components as the ALF framework. Kraken depends on ALF and under the hood the JSON DSL definitions are being compiled into the ALF pipelines modules.

In terms of “services”, Kraken uses the Octopus library, which uses the simple “prepare-call-transform” approach for communication with external services. See the Octopus Readme for more details.

Example

Let’s say you have an online shop system.

There might be “user-service”, “products-service”, “billing-service”, and “notification-service” services.

There might be a dozen of “flows”, or “business-flow”, or “user-flows”, or “use-cases”, that are “pipelines” in Kraken (ALF) terminology:

  • User registration
  • User login
  • User lists products
  • User orders a product
  • etc., etc.

Each service is deployed to the system and exposes its API.

Now, the job of the “application engineer” is to design and build these flows in the system.

For each “use-case”, one should:

  • define the “services” — a uniform mapping to the underlying services API.
  • define the “event” that represents the interaction (and its result).
  • define the “pipeline” — a chain of components that will transform the event accordingly to the business logic.

Definitions

For example, for the “User order a product”, the “event” representing the interaction would be:

{
  "type": "order-event",
  "user_id": "user-123",
  "product_id": "product-456", 
}

The corresponding pipeline (simplified):

Simplified user-orders-product pipeline
{
  "name": "user-orders-product",
  "components": [
    {
      "name": "find-user",
      "service": {"name": "user-service", "function": "find"}
    },
    {
      "name": "find-product",
      "service": {"name": "product-service", "function": "find"}
    },
    {
      "name": "bill-user",
      "service": {"name": "billing-service", "function": "bill"}
    },
    {
      "name": "notify-user",
      "service": {"name": "notification-service", "function": "send-email"}
    }
  ]
}

So, the event goes through four components, each component calls underlying services.
After all the transformations, the output would be like:

{
  "type": "order-event",
  "success": "yes",
  "order_id": "789", 
}

A service definition is also presented as JSON objects. It specifies what should be called and how data are prepared for the call.

For example, the “user-service” might look like this:

{
  "name": "user-service",
  "client": {
    "module": "OctopusClientHttpFinch",
    "start": {
      "base_url": "https://user-service.online-shop",
      "headers": {
        "Accept": "application/vnd.github+json"
      }
    }
  },
  "interface": {
    "find_user": {
      "input": {
        "id": {
          "type": "string"
        }
      },
      "prepare": {
        "method": "GET",
        "path": "'/users/' <> args['id']"
      },
      "call": {
        "parse_json_body": true
      },
      "transform": {
        "user": "args['body']"
      },
      "output": {
        "user": {
          "type": "object"
        }
      }
    },
    ... other functions in the service
  }
}

The “client” section specifies what kind of “transport-layer client” should be used (OctopusClientHttpFinch) and configures the client with URL and headers.

The “interface” section contains the functions accessible for the service. Each function has 5 optional steps:

  • “input” — validates the input,
  • “prepare” — prepares data for the HTTP client,
  • “call” — configures the behavior of the HTTP client,
  • “transform” — gets necessary data from the response,
  • “output” — validates the output of the service

Deployment

Kraken provides HTTP JSON API for defining, starting, and calling services and pipelines. The “orchestrator” application is a simple Elixir app that has Kraken as the main dependency. It may contain also a set of low-level clients, helpers, templates, etc — all the reusable utility-like modules.

The orchestrator is then deployed to the system. The orchestrator must have access to all other services in the system and also be exposed to external communication via HTTP.

Then one can just post all the definitions to the “orchestrator”:

POST /services/define
POST /pipelines/define

Start the services and pipelines:

POST /services/start/service-name
POST /pipelines/start/pipeline-name

And call the pipelines directly:

POST /pipelines/call/user-orders-product with the event payload

Or, using event routing, where Kraken will route events to the corresponding pipeline.

See the Kraken page for more detailed examples of services and pipeline definitions.

Philosophy. Or why is it cool?

Flow-based programming

The approach applies the great engineering idea of an assembly line, where an initial workpiece is placed on the conveyor and goes through several processing stages until it becomes a ready-to-use product. The flow-based programming is a case of the general “event-driven” approach, where business logic is triggered by data.
The main distinctions are:

  • The topology of the “flow” is defined explicitly.
  • A single event represents the result of the interaction with the particular use-case in the software system.

This provides extraordinary observability and, therefore, understandability of the defined logic:

  • One can easily read “pipelines” (both in code and diagrams).
  • One can “sit” on an event and easily track its changes when passing through components in a pipeline.
  • One can “sit” on a component and observe transformations of different events passing through it.

Elixir GenStages, scaling, and monitoring

Under the hood, the ALF runs all components’ code inside Elixir GenServers. It means that all the components work in parallel.
When one pushes the stream of events to the pipeline, one component works with an n-th event, and the previous one work on (n-1)-th one.

Also, ALF (and therefore Kraken) allows having several copies of a component, which adds additional (and configurable) horizontal parallelism.

ALF uses the Elixir/Erlang [telemetry](https://github.com/beam-telemetry/telemetry) library, so it’s easy to track the event transformation on each step.

See the [ALF-monitor](https://github.com/antonmi/alf_monitor) project and video there.

Declarative JSON DSL

What can be simpler than JSON?

Kraken uses it for services and pipelines definitions, and events are also just JSON objects. So, Kraken provides a very simple DLS for modeling the whole application (orchestration) layer for a software system.

Moreover, services definitions can be considered as a specification for the underlying service:

  • services “names” and “interface functions” (together with input/output specifications) define a “language” for communicating with underlying domain services.
  • the “specifications” can (even must) be used by the engineers that work with domain-level services.

The pipeline definitions are comprehensible data structures that are consistent with visual representations of the flows defined using ALF diagrams.

The “design-, code-, and runtime-uniformity” buzz-phrase can be used for describing the advantages of the approach:

  • when one designs a “pipeline”, he/she thinks in terms of a chain of components and draws it on paper using the diagram language.
  • when one codes the pipeline, he/she codes the same structure as was on the paper.
  • and finally, when the program runs, it passes events throw the same chain of components.

Kraken-agent

One can for sure compile all the definitions in a build time and then upload the “orchestrator” into the system.

However, I see more advantages when the code is uploaded dynamically. So, first, one uploads “an empty” Kraken-based orchestrator, then uploads all the definitions and starts the pipelines. It is similar to the Kubernetes approach to creating an infrastructure layer of the system. The code then resides separately from the “orchestrator” and can be uploaded using a client application, something like kraken apply . (similar to kubectl apply .).

It will allow one to dynamically add new services and pipelines (or new versions of them). It also opens huge opportunities for developing (prototyping) pipelines on existing systems (better its development-purpose copies).

And also an IDE-like client can be implemented, providing a low-code platform for the orchestration layer.

Thanks for reading!

Kraken — Flow-based Service Orchestration Framework was originally published in Better Programming on Medium, where people are continuing the conversation by highlighting and responding to this story.

Previous Post
Next Post