Durable execution app

Overview

Durable execution applications are an approach to running long-lived workflows to maintain reliable and consistent stateful processes across distributed systems, ensuring seamless operation despite failures, restarts, or maintenance activities. These applications enable the continuous progression of complex workflows by employing persistent state management and asynchronous, durable messaging between stateful entities. This architecture is critical for building robust, fault-tolerant systems that uphold data integrity and operational continuity in modern software systems.

AI Apps Architecture

lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum

What is a durable execution app?

A durable execution application is an approach to running long-lived processes that are persisted and can ensure reliable execution across distributed systems. These durable processes interact through asynchronous, durable messaging triggered by persistently recorded state changes. This design allows the application to withstand failures, restarts, and maintenance without losing state or interrupting the execution flow.

In such applications, when an entity's state changes, it emits events that other durable processes consume to trigger subsequent actions. This event-driven architecture enables the creation of complex, long-running workflows and supports patterns like orchestration and choreography sagas. By guaranteeing that all state changes and inter-entity communications are durable, Durable Execution Apps ensure that processes can resume seamlessly after any disruption, maintaining data integrity and operational continuity throughout the system.

Key properties of durable execution apps

Durable execution applications possess several core properties that enable them to maintain consistent and reliable operations in distributed environments. These properties ensure that applications can handle complex workflows, recover from interruptions seamlessly, and scale effectively.

State persistence

Entities within the application persist their state changes to durable storage, ensuring that the current state can be recovered after failures or restarts. This persistence is crucial for maintaining the continuity of long-running processes and workflows.

Asynchronous, durable messaging

Communication between entities is handled through asynchronous messages that are durably stored. This guarantees that messages are not lost and that inter-entity communication can resume seamlessly after interruptions, ensuring reliable propagation of state changes throughout the system.

Fault tolerance and recovery

The application is designed to handle failures gracefully, with automatic recovery and state reconstruction mechanisms. Entities can resume operations from their last persisted state, minimizing downtime and preventing data loss.

Scalability and distribution

Durable Execution Apps are built to scale horizontally across distributed systems. Entities can be sharded and distributed across multiple nodes, allowing the application to handle increased load and maintain high availability.

Workflow orchestration & choreography

The application supports complex workflows through orchestration or choreography patterns. Entities can coordinate tasks, manage dependencies, and handle compensating actions, enabling robust transaction management across distributed components.

Event-driven architecture

The system is fundamentally event-driven, with entities reacting to state changes and events emitted by other entities. This promotes loose coupling, enhances responsiveness, and enables real-time business logic processing.

Akka components

The client sends a request to an endpoint component (API).
The endpoint component forwards the request to a workflow component.
Step 1 in the workflow requests data from a view and then sends a command to an entity.
In step 2 of the workflow, the result from step 1 is used to send a command to the second entity component.

How Akka enables durable execution apps

Akka provides comprehensive features and capabilities for building durable execution apps within distributed systems. The Akka SDK leverages advanced distributed systems technology to facilitate the creation of scalable, resilient, and high-performance applications. Two key components of the SDK, workflows, and consumers, are particularly crucial for implementing durable execution patterns.

Workflows

The Akka SDK provides a dedicated workflow component that supports orchestration sagas. Workflow components define and manage complex, long-running processes. The workflow component offers built-in mechanisms for coordinating tasks, managing state transitions, and handling compensating actions across distributed systems.

Workflow

Implementing Workflows

Workflow orchestration & choreography

Akka workflow, consumer, and entity components support flexible workflow designs. For orchestration, a central workflow component controls the sequence of operations, handling error recovery, retries, and compensations as needed. Meanwhile, Akka’s consumer and entity components enable event-driven choreography by allowing each step to react to events from the previous step. This decentralized approach promotes loose coupling and responsive processing flows.

Saga patterns

Choreography-Based Saga pattern

Orchestrator-Based Saga pattern

Brokerless pub/sub

Akka implements a high-performance, brokerless communication system. Instead of relying on external message brokers, Akka allows message consumers to read events directly from event stores, enhancing efficiency and reducing complexity in the messaging infrastructure.

Consuming and Producing

Consume from Event Sourced Entity

Consume from Key Value Entity

Scalable distributed processing

The Akka SDK provides a foundation for building scalable distributed systems. It supports a cluster architecture, allowing applications to distribute workflow processing across multiple nodes seamlessly.

Physical deployment

Akka Distributed Cluster

Architectural Overview

Reliable messaging

Akka ensures reliable messaging through durable event-driven communication. The platform guarantees that messages are not lost and inter-entity communication can resume seamlessly after interruptions, ensuring the system's reliable propagation of state changes.

Consuming and Producing

Configure message brokers

Durable state

Akka provides mechanisms for state persistence and recovery. Entities within Akka-based applications can persist their state changes to durable storage, ensuring that the current state can be recovered after failures or restarts. This feature is crucial for maintaining the continuity of long-running processes and workflows.

Entity state models