Microsoft Durable Task: Idempotency, Replay and External Side-Effect Design Guide

The five scenarios are invoice approvals, Azure tenant provisioning, subscription payment recovery, out-of-order webhooks, and an AI agent handling an incident. The author says the snippets are design sketches. They are not compiled, tested samples, and he warns against inferring exactly-once external effects from durable history.

The mental model: what durable history does and doesn't promise​

Microsoft's documentation explains the basics. When an orchestration has more work to do, such as a response arriving or a durable timer expiring, the orchestrator re-executes the whole function from the start to rebuild local state. During that replay, the framework checks the execution history. If an activity already ran and produced a result, the framework replays that result instead of running it again. That is re-execution against history. It does not restore a suspended stack.

Two consequences run through the whole article:

• Orchestrators must be deterministic, but activities can run any code, including nondeterministic or long-running operations.
• The runtime guarantees each activity runs at least once. If a failure occurs after the activity completes but before the result is recorded, the runtime might rerun it. Microsoft's guidance is to make activity logic idempotent whenever possible.

Durable Task remembers where the process got to. It does not make a payment, an Azure deployment or a model call happen exactly once. Dresher's boundary table captures this. In every scenario Durable Task coordinates part of the process, and the application stays authoritative for the external result.

On packaging, Microsoft describes the Durable Task SDKs as portable, open-source libraries that run on compute such as Azure Container Apps, Kubernetes or VMs. The .NET SDK is listed as GA and works with ASP.NET Core, console apps and worker services. The JavaScript/TypeScript SDK is still in preview, so don't assume every SDK has the same status.

1. Invoice approval: let the engine remember the wait​

The scenario is a service that extracts an invoice, validates it, and then posts it, rejects it or sends it to a human. In a batch, one invoice needing review shouldn't hold up the other seven. Each invoice therefore gets its own child orchestration. The limit of eight is an application admission rule, not a DTS quota.

Durable Task helps because the review wait can be a durable timer plus an external event. No thread is held while the invoice waits, and a replacement worker can replay to the same pending wait. The article is careful about what the event means:

• The event is only a hint. An authenticated endpoint must first commit a single terminal decision to the application's review store.
• An activity then reads or closes that authoritative ticket, including when the timer fires. If approval was committed just before the deadline but the hint arrived after it, the stored decision wins.
• Posting to accounting needs a stable posting key and an immutable request fingerprint. If the accounting system accepted the post but the acknowledgement was lost, the integration must reconcile the same identity.
• Task.WhenAll waits for every child. It does not undo a sibling's accounting entry if another child fails.

The author's own verdict is that if existing continuation handlers already do all this cleanly, you may not need another runtime.

2. Tenant provisioning: ARM may already be enough​

The example onboards a SaaS tenant by provisioning a network, PostgreSQL server, storage with private access and a Container App. It then configures DNS, checks readiness and activates the tenant.

Before reaching for DTS, the article asks whether Azure Resource Manager already solves the problem. For a single well-defined deployment it may, because ARM/Bicep already offers dependency ordering, parallel deployment, idempotent reapplication and server-side deployment state. A worker crash doesn't erase an ARM deployment.

The orchestration earns its place in the wider business process:

1. Admit an immutable onboarding request and return HTTP 202 with a request ID and status URL. Accepted does not mean ready.
2. Save the intent to start the workflow in an outbox alongside admission. The scheduler handoff is not atomic with the application database.
3. Provision in waves: network first, then database and storage in parallel, then the application, DNS and readiness.
4. Poll through activities that each make one bounded status query, with durable timers between polls. A 45-minute polling budget limits polling but does not cancel an Azure operation already accepted.
5. Activate the tenant as a separate idempotent step that revalidates ownership and actual readiness. If activation fails, the workflow fails visibly and does not auto-delete the resources.

Cleanup is not an undo button​

The article's most useful warning concerns a partial failure. Storage fails while Azure is still creating the database. If you delete everything you can see, the database may finish after cleanup has checked for it, leaving an orphan that keeps billing.

Microsoft's ARM documentation backs the caution. A deployment can be cancelled only while its provisioning state is Accepted or Running, and cancelling leaves the resource group partially deployed. Cancellation is not rollback.

The article's cleanup rules are:

• Stop the failed attempt from submitting new operations.
• Reconcile previously submitted operations, including those whose responses were lost.
• Delete only resources created exclusively for this attempt, and only before tenant activation or customer-data admission.
• Recheck ownership before each deletion, and delete in reverse dependency order: DNS, application, database/storage, then network.
• If safe ownership can't be established, keep the resources and surface NeedsIntervention.

Durable Task can coordinate this recovery. It can't decide which Azure resources are safe to delete.

3. Payments: a timeout is not a decline​

A subscription charge times out. The provider may have taken the money before the response vanished. A blind retry risks a double charge, and treating it as a decline risks a misleading reminder.

The article first defines a billing cycle. It is one billing period and its payment obligation. It is not a single attempt or the whole subscription. A €20 monthly plan has separate September and October cycles, each with its own CycleId, due date and initial PaymentKey.

A plain scheduled job that stores NextCheckUtc is described as a reasonable architecture. A durable orchestration becomes attractive when the policy also includes several elements:

• due dates
• callback hints
• a six-hour check cadence
• a bounded reminder
• a fixed grace deadline
• a handoff to the next cycle

The example loop is bounded to 16 inspected snapshots and to the grace deadline. After a validated paid receipt, an idempotent activity closes the cycle and obtains the authoritative next one. Only then does the flow call ContinueAsNew, which keeps the instance ID and starts a fresh history.

The ledger stays authoritative throughout. Neither a callback nor a payment-method update proves a charge succeeded. An unresolved cycle ends as NeedsReconciliation, and cancelling stops future collection but doesn't erase an accepted charge.

4. Webhooks: where a durable entity isn't enough​

This scenario shows a limit of the framework. A news CMS emits revisions 11, 12 and 13. Revision 12 can arrive before 11, arrive twice, or finish indexing after 13 and overwrite it with stale content.

A durable entity can remember the highest observed and applied revision, and it processes operations for one entity key serially. It does not lock the search index, and it does not stop separate orchestrations from writing concurrently. Two orchestrations can read the same counters and both attempt revision 13.

The fix has to live in the destination, which must atomically reject an older revision once a newer one is committed. Around that you still need a reliable inbox and dispatcher, plus periodic reconciliation for missed webhooks. A singleton orchestrator per article, using an instance ID derived from the generation-qualified key, can reduce concurrent controllers. That only holds if duplicate-start races are handled, and it still isn't exactly-once for external effects.

The article states its assumptions: strictly increasing positive revision numbers per article generation, immutable content per revision and versioned tombstones. If the CMS offers only opaque ETags or arrival timestamps, this algorithm doesn't apply. It also advises measuring actual indexing calls before claiming the design reduces work.

5. AI incident workflow: durable around the model, not inside it​

The workflow gathers logs and metrics in parallel and asks a model for a remediation proposal. It then waits for an engineer's approval, executes the approved action and verifies the result. Dresher's key design rule is to keep the model call inside an activity. The orchestrator should only ever see a stable reference to an immutable proposal. A second model call after a crash is a new decision, not a recovery.

The safeguards mirror the earlier examples:

• Generation uses a stable identity, so a saved proposal is reused or reconciled rather than silently replaced.
• The approval endpoint commits one terminal decision against the proposal ID, fingerprint and deadline. The event is a hint, never authorization.
• The remediation activity rechecks the stored decision and current safety policy. An approved plan that is no longer safe should go back for fresh review.
• Execution needs a stable key, an operation journal and a provider-specific reconciliation path. Blind retries of destructive tools are ruled out.

The article suggests testing by killing the worker at four points: after telemetry is recorded, after generation but before the result is recorded, while waiting for approval, and after remediation succeeds but before completion is recorded. The invariant to check is that only the specifically authorized proposal ran and uncertain effects were reconciled. Orchestrator completion alone doesn't prove that.

The author then points to Durable Agents and Durable Workflows for Microsoft Agent Framework, saying durability should be part of how agents run rather than something every app builds around them. The public Microsoft repository for this extension covers .NET durable agents and workflows, Azure Functions hosting, and Python Durable Task integration. That confirms the project exists and what it covers. It doesn't change the central caveat that the model stays nondeterministic.

Analysis: a good decision checklist​

This is a Microsoft engineering blog promoting a Microsoft runtime, so a skeptical read is reasonable. The piece is unusually willing to say when not to use it. It tells you to keep ARM for dependency graphs, a scheduled job for simple payment recovery, and an existing checkpoint-and-dispatch system for webhook ordering. That fits wider industry guidance. A Microsoft Durable Task veteran, writing in an earlier Medium post, noted that automatic retries mean a particular piece of code may be executed more than once. The advice is to write idempotent activities.

Before choosing a durable runtime, the article's closing advice is to pick one long-running workflow and write down three things:

1. Authoritative state: the system that decides what really happened.
2. Stable retry identity: the key that makes a repeated call recognizable.
3. Uncertain external side effects: the actions that may have happened without an acknowledgement.

If the code for remembering and resuming a process has grown into a system of its own, durable orchestration is a candidate. A few await calls alone are not a reason to adopt it.

One practical caution for teams: the code is illustrative only. Treat it as a design checklist, not production-ready scaffolding.